Generation device, generation method, and generation program

Abstract

To improve a detection accuracy of an object detection model.SOLUTION: A generation device includes: an acquisition part; a generation part; and a learning data generation part. The acquisition part acquires a first image and a second image as images of a predetermined object. The generation part generates a composite image in which the first image and the second image are pasted on a background image in a superimposed state. The learning data generation part associates information indicating the predetermined object with a composite image in the composite image to generate learning data used for learning an object detection model that detects the predetermined object from an input image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a generation device, a generation method, and a generation program. [Background technology]

[0002] Methods for generating training data used in training an object detection model that detects objects from input images have been proposed. For example, a method has been proposed for efficiently generating training data by extracting an object region from a given image containing the object and pasting the extracted object image onto a different background image, or a method has been proposed for pasting an object image onto a different background image to create a more natural appearance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-149086 [Patent Document 2] Japanese Patent Application Publication No. 2023-028298 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when an image of an object detected by an object detection model is pasted onto a background image as in the above-mentioned conventional technology, while it is possible to efficiently pad the training data image, it is not necessarily possible to generate a training data image that reproduces a realistic scene including the detected object.

[0005] For example, since photographed images of real space are generally used as input images for object detection models, an object detection model trained with unnatural training data images that do not fully reproduce the realistic scene surrounding the object to be detected cannot properly detect objects from the input photographed images, resulting in a problem of reduced detection accuracy.

[0006] For example, even if it is natural in a real situation for detection objects to overlap each other or for a detection object to be occluded by another object, such overlapping (occlusion) cannot be properly reproduced by simply pasting an image of the detection object onto a background image, as in conventional technology.

[0007] Therefore, with the above-mentioned conventional technology, for example, while it may be possible to accurately detect an object in an input image that shows the entire object without any overlapping, it is possible that an object detection model with low scalability may be obtained that detects multiple objects as a single object in an input image in which the objects overlap.

[0008] Therefore, the present invention provides a generation device, a generation method, and a generation program that can improve the detection accuracy of an object detection model. [Means for solving the problem]

[0009] In order to solve the above problem, one embodiment of the generating device according to the present invention includes an acquisition unit that acquires a first image and a second image as images of a predetermined object; a generation unit that generates a composite image in which the first image and the second image are superimposed and pasted onto a background image; and a training data generation unit that generates training data to be used in training an object detection model that detects the predetermined object from an input image by associating the composite image with information indicating the predetermined object in the composite image. [Effects of the Invention]

[0010] According to the present invention, it is possible to improve the detection accuracy of an object detection model. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating the conventional algorithm AL. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 3]FIG. 3 is a diagram illustrating an example of the configuration of a generating device according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the background image data storage unit. [Figure 5] FIG. 5 is a diagram illustrating an example of the object image data storage unit. [Figure 6] FIG. 6 is a diagram showing a specific example of the pasting method 1. In FIG. [Figure 7] FIG. 7 is a diagram showing a specific example of the pasting method 2. In FIG. [Figure 8] FIG. 8 is a diagram showing the results of verifying the accuracy of an existing object detection model M according to a conventional method and an object detection model Mx according to the method proposed by the present invention. [Figure 9] FIG. 9 is a flowchart illustrating an example of the operation of the generating device. [Figure 10] FIG. 10 is a diagram showing a specific example of a pasting technique based on depth information. [Figure 11] FIG. 11 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the generation device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0013] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from each other. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects from each other.

[0014] (Embodiment) 1. Introduction Challenges in building object detection models include the lack of large-scale labeled datasets and the need for data collection and annotation across a variety of environments. For these reasons, conventional techniques, such as those mentioned above, have been proposed to train object detection models using synthetically rendered scenes and objects.

[0015] For example, there is a Cut & Paste method that generates training data images to be used as training data for an object detection model by pasting a detection object image, which is an image of the detection object extracted from a predetermined image containing the detection object, onto a specific background image.In the following, the conventional training data generation method (conventional method) that uses the Cut & Paste method will be referred to as the "conventional algorithm AL," and the object detection model trained based on the training data generated by the conventional algorithm AL will be referred to as the "object detection model M."

[0016] First, the conventional algorithm AL will be explained using Figure 1. Figure 1 is a diagram for explaining the conventional algorithm AL. The conventional algorithm AL is divided into a Cut step and a Paste step, and Figure 1(a) shows the Cut step. Meanwhile, Figure 1(b) shows the Paste step.

[0017] In the Cut step, an image of the detected object is cut out from an input image containing the detected object (for example, a photographed image of the detected object (a dog in the example of Figure 1(a))) in which the area of ​​the detected object is extracted.

[0018] In the Paste step, for example, a composite image is generated in which the detection object image cut out in the Cut step is pasted onto a randomly acquired background image, thereby superimposing the detection object image on the background image. As shown in Figure 1(b), the composite image is associated with annotation information including the position information of the detection object on the composite image and the category of the detection object, etc., and this becomes the training data.

[0019] In the Paste step, data augmentation may be performed on each of the detection target image and the background image in order to increase the amount of training data and improve detection accuracy.

[0020] As such, the Cut & Paste method in the conventional algorithm AL has the advantage of being able to efficiently generate training data images (synthesized images).On the other hand, the object detection model M trained based on training data images in the conventional algorithm AL has room for improvement in terms of detection accuracy, such as low detection scalability and low recall for unknown objects of the same class.

[0021] One possible reason for the low detection accuracy of the object detection model M is that the conventional algorithm AL simply pastes the detection target image onto the background image, causing the object detection model M to learn by focusing only on the detection target. Therefore, to improve detection accuracy, it is necessary to position the detection target image relative to the background image so as to recreate a realistic scene that matches the actual scene in which the detection target is usually located. For example, it is necessary to position the detection target image relative to the background image, taking into consideration the positional relationship between the background and the detection target, the positional relationship between the detection target objects themselves, and the positional relationship between the detection target and other objects. In other words, it is necessary to position the detection target image relative to the background image while taking into consideration realism.

[0022] This point will be explained in more detail. For example, images captured in a real space are used as input images for the object detection model M, but in real space, the object to be detected appears in various ways depending on the situation and surrounding environment. For this reason, the images for learning data are also required to reproduce a realistic scene in which the object to be detected is placed.

[0023] However, with the conventional algorithm AL, since the detection target image is simply pasted onto the background image, it is difficult to properly reproduce a realistic scene, and even if scaling and rotation are performed using augmentation, there are limits to the degree to which realistic scenes can be reproduced. For example, when taking clothes hanging on a store shelf as an example, the actual appearance of the clothes may be affected by factors such as size, tilt, color, number, and overlap. Therefore, simply pasting images without considering these factors makes it impossible to obtain training data images that properly reproduce a realistic scene, such as clothes being sold in a store.

[0024] From the above, the object detection model M, which is trained based on images used as training data for the conventional algorithm AL, can accurately detect objects for input images that include the entire object, but its detection accuracy decreases for input images in which the entire object is not captured due to the influence of the surrounding environment, etc.

[0025] In view of the above-mentioned problems, the inventors of the present invention propose a training data generation method in which, when pasting an image of an object onto a background image, a synthetic image serving as training data is generated by arranging the image of the object while taking realism into consideration. As described above, to reproduce a realistic scene, i.e., to increase realism, it is necessary to consider factors such as the size, tilt (rotation), hue (color), number, and overlap (occlusion) of the objects. The inventors of the present invention focus particularly on overlap and propose a new algorithm to replace the conventional algorithm AL as a training data generation method. Hereinafter, the training data generation method proposed by the inventors of the present invention (proposed method) will be referred to as the "new algorithm ALx," and the object detection model trained based on the training data generated by the new algorithm ALx will be referred to as the "object detection model Mx."

[0026] Furthermore, a generation device (generation device 100 described below) that operates using a program based on a generation method called new algorithm ALx acquires a first image and a second image as images of a predetermined object, and generates a composite image by pasting the first image and the second image onto a background image in a state where they are superimposed (occluded).The generation device then associates information indicating the predetermined object in the composite image with the composite image, thereby generating training data used to train an object detection model Mx that detects the predetermined object from an input image.

[0027] The new algorithm ALx has two methods for overlaying images. Specifically, the new algorithm ALx has a method (hereinafter referred to as "method M1") in which images of objects detected by the object detection model Mx are superimposed on each other and pasted onto a background image, and a method (hereinafter referred to as "method M2") in which an image of an object detected by the object detection model Mx is superimposed on an image of an object not detected by the object detection model Mx and pasted onto a background image.

[0028] In method M1, the predetermined object is an object to be detected by the object detection model Mx, and both the first image and the second image are images of the object to be detected (images of the object to be detected). That is, in method M1, the images of the object to be detected are arranged on the background image in a state where they are superimposed on each other.

[0029] On the other hand, in method M2, the predetermined objects are objects to be detected by the object detection model Mx and non-detection objects that are not objects to be detected by the object detection model Mx, and the first image is an image of the detection object (detection object image) and the second image is an image of the non-detection object (non-detection object image). That is, in method M2, like method M1, detection object images are arranged on the background image in a state where they are superimposed on each other, but the detection object image and the non-detection object image are further superimposed on each other and arranged on the background image. The superimposition of the detection object image and the non-detection object image includes a pattern in which the non-detection object image is arranged in front of the detection object image and a pattern in which the non-detection object image is arranged behind the detection object image.

[0030] Furthermore, the non-detected object image plays the role of noise to more realistically reproduce the situation in which the detected object is placed. For example, taking a use case of the object detection model Mx as an example of detecting clothes from images taken inside an apparel store, the actual scene captured in the image may be a person selecting clothes in front of a display shelf, or a person passing in front of or behind the display shelf. Therefore, by superimposing an image of clothes as the detected object image and an image of a person as the non-detected object image, it becomes possible to reproduce a realistic scene that is close to the actual scene.

[0031] [2. System Configuration Overview] The configuration of the information processing system 1 will be described using Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing system 1 according to the embodiment. As shown in Fig. 2, the information processing system 1 includes an imaging system 2, a detection device 60, and a generation device 100. The imaging system 2, the detection device 60, and the generation device 100 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via wired or wireless communication. Note that the information processing system 1 shown in Fig. 2 may include a plurality of imaging systems 2, a plurality of detection devices 60, and a plurality of generation devices 100.

[0032] As shown in FIG. 2, the imaging system 2 may be configured with an imaging device 10, a display control device 11, and a display device 12.

[0033] The imaging device 10 is an imaging means (camera) installed in a predetermined location (for example, inside a store). The imaging device 10 may be, for example, an AI camera.

[0034] For example, the display control device 11 superimposes information on the object detection result on the captured image acquired by the imaging device 10, and controls the display device 12 to display the superimposed captured image.

[0035] The display device 12 has a screen using, for example, a liquid crystal display, an electroluminescence (EL) display, a cathode ray tube (CRT), etc. The display device 12 may be compatible with 4K or 8K, or may be formed by a plurality of display devices. The display device 12 displays the captured image controlled to be displayed by the display control device 11.

[0036] The detection device 60 inputs a captured image into the object detection model Mx generated by the generation device 100, causing the object detection model Mx to perform inference for detecting a predetermined object from the captured image. In this way, the detection device 60 corresponds to a detector that performs object detection using the object detection model Mx.

[0037] For example, the imaging device 10 takes continuous photographs, and when a photographed image is acquired, uploads the acquired photographed image to the detection device 60. When the detection device 60 accepts the upload of the photographed image, it uses the object detection model Mx to perform inference processing to detect an object from the currently acquired photographed image. Specifically, the detection device 60 inputs the photographed image to the object detection model Mx, thereby outputting the position coordinates of the object in the photographed image. The detection device 60 may also transmit the position coordinates of the object detected from the photographed image to the display control device 11. The display control device 11 depicts a rectangular area indicating the position coordinates of the object in the currently acquired photographed image. The display control device 11 also controls the display device 12 to display the photographed image including the rectangular area. The display device 12 displays the photographed image including the rectangular area in accordance with the control of the display control device 11. Through this series of processes, the user can check the detection results by the object detection model Mx, for example, via the display device 12.

[0038] The generating device 100 performs information processing related to the proposed technology of the present invention, including a process of generating training data using a new algorithm ALx and a process of generating an object detection model Mx with improved accuracy by training a model using the training data. As such, the generating device 100 corresponds to a learning device that trains the object detection model Mx. The generating device 100 may also provide the object detection model Mx to the detection device 60. The generating device 100 may be implemented as either a local server or a cloud server incorporating a learning function (AI software).

[0039] 3. Configuration of the Generation Device The generating device 100 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram illustrating an example of the configuration of the generating device 100 according to the embodiment. As shown in Fig. 3, the generating device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0040] <Communication Unit 110> The communication unit 110 is realized by, for example, a network interface card (NIC), etc. For example, the communication unit 110 transmits and receives information to and from the imaging system 2 and the detection device 60.

[0041] <Storage section 120> The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 may store, for example, data and programs related to the information processing according to the embodiment. As shown in FIG. 3 , the storage unit 120 may include a background image data storage unit 121 and an object image data storage unit 122.

[0042] (Background image data storage unit 121) The background image data storage unit 121 stores information related to background images. An example of the background image data storage unit 121 is shown in FIG. 4. In the example of FIG. 4, the background image data storage unit 121 has items such as "background image," "scene," "object," and "paste position information." The "paste position information" also includes items such as "detection target," "detection target object position," "non-detection target," and "non-detection target object position."

[0043] A "background image" is the background image itself, that is, background image data. A "scene" is the scene of the background image. An "object" indicates a characteristic object in the background image. FIG. 4 shows an example in which a background image "BG1", a scene "inside an apparel store", and an object "hanger box" are associated with each other. This example shows an example in which a "hanger box" is captured in background image BG1, which has a scene of the "inside an apparel store" as the background, as an object that characterizes this scene.

[0044] Furthermore, a pasting position, which is a position where an image of at least one of a detection target object or a non-detection target object is pasted as a predetermined object, is determined in advance in the background image. "Paste position information" is information indicating this paste position. As will be explained with reference to FIG. 6 etc., the paste position may be defined as a rectangular area with respect to the background image. Of course, the paste position may also be defined by other information (for example, the coordinate system of the background image).

[0045] A "detection object" is an object that can be detected by the object detection model Mx, and a "detection object position" is position information on a background image that defines the position on the background image where the detection object image is to be pasted. Fig. 4 shows an example in which "BG1", the detection object "clothes", and the detection object position "position #11" are associated with each other, and this example shows an example in which "position #11" is defined in the background image BG1 as the pasting position for pasting the image of "clothes", which is the detection object image.

[0046] An "undetected object" is an object that cannot be detected by the object detection model Mx, and a "undetected object position" is position information on a background image that defines at what position in the background image the image of the undetected object is pasted. Fig. 4 shows an example in which "BG1", an undetected object "person", and a detected object position "position #12" are associated with each other, and this example shows an example in which "position #12" is defined in the background image BG1 as the pasting position for pasting the image of the "person", which is the undetected object.

[0047] Note that an object detection model Mx may be generated for each detection target. In the example of Fig. 4, object detection models Mx may be generated individually to detect detection targets such as "clothes," "vehicles," and "fish." Furthermore, the position designation unit 132 of the generating device 100 may, for example, receive "paste position information" from the user and designate the received "paste position information" for the "background image."

[0048] (Object image data storage unit 122) The object image data storage unit 122 stores information about images of both detection targets and non-detection targets as predetermined objects. FIG. 5 shows an example of the object image data storage unit 122. In the example of FIG. 5, the object image data storage unit 122 has items such as "first category," "second category," "object image," and "setting information" for each "detection target" and "non-detection target." The "setting information" also includes items such as "enlargement / reduction," "rotation," "color," "number," and "degree of overlap."

[0049] A "detection target" is an object that can be detected by the object detection model Mx, and a "non-detection target" is an object that cannot be detected by the object detection model Mx. The "first category" and the "second category" are hierarchical categories to which predetermined objects (detection target, non-detection target) belong. An "object image" is an image of a predetermined object (detection target, non-detection target).

[0050] FIG. 5 shows an example in which a first category "clothing," a second category "jacket," and an object image "TG11" are associated with a "detection object." In this example, the detection object image TG11 is an image of a jacket. The detection object image TG11 may be, for example, an image in which the jacket area is extracted from an image of the jacket. The jacket area may be a rectangular area surrounding the jacket, or a contour area that molds the shape of the jacket.

[0051] FIG. 5 also shows an example in which the first category "person", the second category "woman", and the object image "NG11" are associated with an "undetected object". In this example, the undetected object image NG11 is an image of a woman. The undetected object image NG11 may be, for example, an image in which a woman's area is extracted from an image of the woman. The woman's area may be a rectangular area surrounding the woman's entire body, or a contour area that models the woman's entire body shape.

[0052] In this embodiment, the detection object image and the non-detection object image may be two-dimensional images or three-dimensional images.

[0053] In this embodiment, the user can specify, for example, a range of parameter values ​​(parameter values) for pasting the detection object image and the non-detection object image onto the background image, and the generating device 100 pastes the detection object image and the non-detection object image onto the background image based on parameter values ​​arbitrarily set within the specified parameter value range.

[0054] "Scale," "Rotation," "Color," "Number," and "Overlap" are examples of pasting parameters when pasting images of detected objects and non-detected objects. The user can specify the range for each of "Scale," "Rotation," "Color," "Number," and "Overlap."

[0055] "Scale" is information indicating how much the detected object image and non-detected object image are to be scaled when pasting the image, and an example is shown in which a parameter value range of "B1 to B3" is specified. The setting unit 133 of the generation device 100 may set a value randomly selected from the scale range "B1 to B3" for the detected object image and non-detected object image. In this case, the image generation unit 134 may paste each object image (detected object image, non-detected object image) onto the background image in a scaled state based on the scale value set for each object image.

[0056] "Rotation" is information indicating how much the detected object image and non-detected object image are to be rotated when pasting the image, and an example is shown in which a range of "B1 to B3" is specified. The setting unit 133 may set a value randomly selected from the rotation range "B1 to B3" for the detected object image and non-detected object image. In this case, the image generation unit 134 may paste each object image (detected object image, non-detected object image) onto the background image in a rotated state based on the rotation value set for each object image.

[0057] "Color" is information indicating how the color of the detection object image and the non-detection object image is changed when the detection object image and the non-detection object image are pasted, and an example is shown in which a range of "B1 to B3" is specified. The setting unit 133 may set a value randomly selected from the saturation range "B1 to B3" for the detection object image and the non-detection object image. In this case, the image generation unit 134 may paste the color of each object image (detection object image, non-detection object image) onto the background image in a state in which the color has been adjusted based on the saturation set for each object image.

[0058] "Number" is information indicating how to change the number of pasted images when pasting the detection object image and non-detection object image, and an example is shown in which a range of "B1 to B3" is specified. The setting unit 133 may set a value randomly selected from the number range "B1 to B3" for the detection object image and non-detection object image. In this case, the image generation unit 134 may paste the number of object images set for each object image (detection object image, non-detection object image) onto the background image.

[0059] The "degree of overlap" is information indicating the degree of overlap when detection object images are overlapped with each other, or information indicating the degree of overlap when a detection object image and a non-detection object image are overlapped with each other. For example, the setting unit 133 may calculate the "degree of overlap" using a method described below. For example, the setting unit 133 may calculate a random value of the "degree of overlap" between combinations of detection object images. Furthermore, the setting unit 133 may calculate a random value of the "degree of overlap" between combinations of detection object images and non-detection object images.

[0060] <Control unit 130> 3, the control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs (for example, the generation program according to the embodiment) stored in a storage device inside the generating device 100 using RAM as a work area. The control unit 130 is also realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0061] As shown in Fig. 3, control unit 130 has an acquisition unit 131, a position designation unit 132, a setting unit 133, an image generation unit 134, a learning data generation unit 135, and a learning unit 136, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of control unit 130 is not limited to the configuration shown in Fig. 3, and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationship between each processing unit included in control unit 130 is not limited to the connection relationship shown in Fig. 3, and may be other connection relationships.

[0062] In the control unit 130, the acquisition unit 131 acquires a first image and a second image as images of a predetermined object. The image generation unit 134 generates a composite image by pasting the first image and the second image onto a background image in a superimposed state. The training data generation unit 135 generates training data to be used for training an object detection model Mx that detects the predetermined object from an input image by associating the composite image with information indicating the predetermined object in the composite image. This will be explained in more detail below.

[0063] <Acquisition part 131> The acquisition unit 131 acquires a first image and a second image as images of a predetermined object. For example, the acquisition unit 131 acquires, as the first image and the second image, detection object images that are images of an object detected by the object detection model Mx. For example, the acquisition unit 131 may acquire a background image and also acquire a detection object image corresponding to the scene of the acquired background image.

[0064] Furthermore, the acquisition unit 131 may acquire, as the first image, a detected object image that is an image of an object to be detected by the object detection model Mx, and may acquire, as the second image, a non-detected object image that is an image of a non-detected object that is not a detection target by the object detection model Mx. For example, the acquisition unit 131 may acquire a background image and also acquire a detected object image and a non-detected object image according to the scene of the acquired background image.

[0065] For example, the acquisition unit 131 acquires a background image from the background image data storage unit 121 and acquires a detection target image and a non-detection target image from the object image data storage unit 122 .

[0066] <Position specification section 132> The position designation unit 132 designates a pasting position, which is a position at which an image of a predetermined object is pasted on the background image. Specifically, the position designation unit 132 designates a pasting position at which a detection object image is pasted on the background image as the pasting position. Furthermore, the position designation unit 132 designates a pasting position at which a non-detection object image is pasted on the background image as the pasting position. For example, the position designation unit 132 may accept a pasting position from a user and designate a pasting position on the background image based on the accepted pasting position.

[0067] <Settings section 133> The setting unit 133 sets pasting parameters when pasting the detection object image and the non-detection object. For example, when the user specifies a range of values ​​for the pasting parameters when pasting the detection object image and the non-detection object, the setting unit 133 may randomly set any value within the specified range of parameter values. As described above, the parameters include "enlargement / reduction" (enlargement / reduction rate), "rotation" (rotation rate), "color" (saturation), "number," "degree of overlap," etc.

[0068] <Image generation unit 134> The image generation unit 134 pastes the detection object image or the non-detection object onto the background image using either method 1 corresponding to the new algorithm ALx or method 2 corresponding to the new algorithm ALx. The image generation unit 134 pastes the detection object image or the non-detection object onto the paste position specified on the background image by the position specification unit 132 based on the parameters set by the setting unit 133. At this time, the image generation unit 134 can paste the image according to method 1 corresponding to the new algorithm ALx or method 2 corresponding to the new algorithm ALx.

[0069] In Method 1, the image generation unit 134 pastes the detection object images onto the background image in a state where they are superimposed on each other. In this case, the image generation unit 134 may superimpose the detection object images onto the background image using a degree of overlap that is randomly set for each combination of the detection object images. The process of pasting the detection object images onto the background image in a state where they are superimposed on each other includes the implication of arranging the detection object images onto the background image in a state where they are superimposed on each other.

[0070] Furthermore, in Method 2, the image generation unit 134 pastes the detection object images onto the background image in an overlapping state, but it also pastes the detection object image and the non-detection object image onto the background image in an overlapping state. In this case, the image generation unit 134 may superimpose the detection object image and the non-detection object image using an overlapping degree that is randomly set for each combination of the detection object image and the non-detection object image. The process of pasting the detection object image and the non-detection object image onto the background image in an overlapping state includes the implication of arranging the detection object image and the non-detection object image onto the background image in an overlapping state. Specific examples of Methods 1 and 2 will be described below with reference to FIGS. 6 and 7.

[0071] (An example of Method 1) 6 is a diagram showing a specific example of pasting method 1. In method 1, images of the detection object are superimposed on each other in order to increase the reproduction rate of a more realistic scene in which the detection object is placed.

[0072] In the example of Fig. 6(a), the acquisition unit 131 acquires a background image BG1 with a scene inside an apparel store as the background, and also acquires detected object images TG11, TG12, TG13, etc., from which clothing portions have been extracted, as detected object images corresponding to the scene. Furthermore, the position designation unit 132 designates a pasting position P11 on the background image BG1 as a pasting position for pasting the detected object images TG11, TG12, TG13, etc. onto the background image BG1. As shown in Fig. 6(a), the pasting position P11 may be designated as a rectangular area.

[0073] Also, Figure 6(b) shows an example in which the setting unit 133 sets "enlargement / reduction," "rotation," "color," and "number" as parameters when pasting the detection object image TG11 at the pasting position P11 on the background image BG1.

[0074] As shown in FIG. 6(b)(1), the setting unit 133 sets a scaling value (scaling ratio) for the detection object image TG11, and the image generation unit 134 enlarges or reduces the detection object image TG11 based on the scaling value.

[0075] 6(b)(2), the setting unit 133 sets a rotation value (rotation rate) for the detection object image TG11, and the image generating unit 134 rotates the detection object image TG11 based on the rotation value. Note that the setting unit 133 may set a rotation value for each axis of rotation (e.g., X-axis, Y-axis, Z-axis).

[0076] As shown in FIG. 6(b)(3), the setting unit 133 sets a color value (saturation) for the detection object image TG11, and the image generating unit 134 adjusts the color of the detection object image TG11 based on the color value.

[0077] 6(b)(4), the setting unit 133 sets the number of detection object images TG11 to be pasted onto the background image BG1, and the image generation unit 134 prepares that number of copies of the detection object images TG11. Furthermore, a scaling value, a rotation value, and a color value may be set for each copy image of the detection object image TG11. Furthermore, the scaling value, the rotation value, and the color value may be set randomly for each copy image.

[0078] 6(c) shows an example in which the setting unit 133 sets an "overlap degree" for overlapping the detection object image TG12 and the detection object image TG13 as a parameter when pasting the detection object images TG12 and TG13 at the pasting position P11 on the background image BG1. As an example, as shown in FIG. 6(c), the setting unit 133 may set the overlap degree α between the detection object image TG12 and the detection object image TG13 based on the center-to-center distance d in the X-axis direction between the detection object image TG12 and the detection object image TG13. In this case, the image generation unit 134 overlaps the detection object image TG12 and the detection object image TG13 based on the overlap degree α.

[0079] Here, the image generation unit 134 controls the appearance of the detection object images in accordance with the parameters set by the setting unit 133, and performs image synthesis by pasting the controlled detection object images so that they overlap each other at the pasting position P11. FIG. 6(d) shows an example of a composite image SG1 generated by the image generation unit 134 through image synthesis. In the composite image SG1, clothes with various appearances are arranged in various ways and overlapped in various ways at the pasting position P11 on the background image BG1. Therefore, the composite image SG1 reproduces a realistic situation that may occur in an apparel store, where clothes are displayed.

[0080] (An example of Method 2) Fig. 7 is a diagram showing a specific example of pasting method 2. In method 2, a detected object image and a non-detected object image are superimposed with the aim of increasing the reproduction rate of a more realistic scene in which a detected object is placed. In method 2, method 1, i.e., superimposing detected object images on each other, may also be performed, but this point has been explained in Fig. 6 and will not be described here.

[0081] In the example of Fig. 7(a), the acquisition unit 131 acquires a background image BG1 with a scene inside an apparel store as the background, and acquires non-detected object images NG11 and NG12 in which a woman's entire body is extracted as non-detected object images corresponding to the scene. The position designation unit 132 designates a pasting position P12 on the background image BG1 as a pasting position for pasting the non-detected object images NG11 and NG12 onto the background image BG1. As shown in Fig. 7(a), the pasting position P12 may be designated as a rectangular area.

[0082] Below, the control processing according to the parameters will be explained using the non-detection object image NG11 as an example, but the same processing is also performed on the non-detection object image NG12.

[0083] Figure 7(b) shows an example in which the setting unit 133 sets "enlargement / reduction," "rotation," "color," and "number" as parameters when pasting the non-detection object image NG11 at the pasting position P12 on the background image BG1.

[0084] As shown in Figure 7(b)(1), the setting unit 133 sets a scaling value (scaling ratio) for the non-detected object image NG11, and the image generation unit 134 enlarges or reduces the non-detected object image NG11 based on the scaling value.

[0085] 7(b)(2), the setting unit 133 sets a rotation value (rotation rate) for the non-detection object image NG11, and the image generating unit 134 rotates the non-detection object image NG11 based on the rotation value. Note that the setting unit 133 may set a rotation value for each axis of rotation (e.g., X-axis, Y-axis, Z-axis).

[0086] As shown in FIG. 7(b)(3), the setting unit 133 sets a color value (saturation) for the non-detection object image NG11, and the image generation unit 134 adjusts the color of the non-detection object image NG11 based on the color value.

[0087] 7(b)(4), the setting unit 133 sets the number of non-detection object images NG11 to be pasted onto the background image BG1, and the image generation unit 134 prepares that number of copies of the non-detection object images NG11. Furthermore, a scaling value, a rotation value, and a color value may be set for each copy image of the non-detection object image NG11. Furthermore, the scaling value, the rotation value, and the color value may be set randomly for each copy image.

[0088] Here, the image generation unit 134 controls the aspect of the non-detection object image NG11 in accordance with the parameters set by the setting unit 133, and performs image synthesis by pasting the controlled non-detection object image NG11 at the pasting position P12. Specifically, the image generation unit 134 performs image synthesis by pasting the non-detection object image NG11 so as to overlap the detected object image.

[0089] 7(c) shows an example of a composite image SG1 generated by the image generation unit 134 through image synthesis. In the composite image SG1, people with various appearances are arranged in various ways and overlaid in various ways at the attachment position P12 on the background image BG1. As a result, the composite image SG1 recreates a realistic situation that could occur in an apparel store, in which a person is choosing clothes.

[0090] The setting unit 133 may set the "degree of overlap" for overlapping the non-detection object image NG11 and the detection object image, following the method described in Fig. 6(c). In the example of Fig. 7, the setting unit 133 may set the degree of overlap β between the non-detection object image NG11 and the detection object image, for example, based on the center-to-center distance d in the Y-axis direction between the non-detection object image NG11 and the detection object image.

[0091] Furthermore, in the example of FIG. 7(c), the non-detection object image is placed in front of the detection object image, but the non-detection object image may also be placed behind the detection object image. For example, the image generation unit 134 may determine whether it is optimal to place the non-detection object image in front of or behind the detection object image based on the scene of the background image BG1. To enable such a determination, for example, the object image data storage unit 122 shown in FIG. 5 may previously associate at least one of the "front" and "back" sides with the non-detection object image as a superimposition relationship.

[0092] <Learning data generation unit 135> Returning to FIG. 3, the training data generation unit 135 generates training data used for training the object detection model Mx that detects objects from the input image by associating the synthetic image with information indicating a predetermined object in the synthetic image. Specifically, the training data generation unit 135 generates training data by associating the synthetic image with annotation information including position information and a type of the detection object on the synthetic image. In the example of FIG. 6, the training data generation unit 135 may generate training data by adding to the synthetic image SG1 annotation information including "position information of where the detection object image TG11 is pasted on the background image BG1" as position information of the detection object on the synthetic image and the type of the detection object "clothing."

[0093] <Study Section 136> The learning unit 136 generates the object detection model Mx by having the learning data generation unit 135 learn the model.

[0094] [4. Accuracy Comparison] FIG. 8 is a diagram showing the results of verifying the accuracy of an existing object detection model M according to a conventional method (conventional algorithm AL) and an object detection model Mx according to a proposed method of the present invention (new algorithm ALx).

[0095] Figure 8(a) shows the detection results when an input image containing clothing is input to object detection model M. Object detection model M is able to recognize each piece of clothing individually in the input image (left) where the clothing does not overlap, but in the input image (right) where the clothing overlaps, it erroneously recognizes the overlapping pieces of clothing as a single piece of clothing.

[0096] Figure 8(b) shows the detection results when an input image containing clothes is input to the object detection model Mx. The object detection model Mx is able to individually recognize each piece of clothing in both the input image where the clothes do not overlap (left) and the input image where the clothes overlap (right).

[0097] As such, it appears that object detection model Mx has improved accuracy compared to object detection model M. Therefore, the detection accuracy was evaluated between object detection model Mx and object detection model M. AP (Average Precision) was used as the index value for evaluating detection accuracy. As shown in FIG. 8, AP=40% for object detection model M, while AP=60% for object detection model Mx, proving that detection accuracy is improved.

[0098] [5. Example of operation of the generating device] Fig. 9 is a flowchart showing an example of the operation of the generation device 100. Fig. 9 shows an example of the operation when executing the method 2 corresponding to the new algorithm ALx.

[0099] First, the learning unit 136 determines whether or not the required number of pieces of learning data have been collected (step S901).

[0100] If the required number of pieces of learning data have not been collected (step S901; No), the acquiring unit 131 acquires a background image (step S902). The acquiring unit 131 acquires the background image from the background image data storage unit 121. The background image may be, for example, an image captured by the imaging device 10, or an image generated by any method.

[0101] Furthermore, the acquisition unit 131 acquires a detection object image and a non-detection object image as images of predetermined objects according to the scene of the background image (step S903).

[0102] The position designation unit 132 designates, on the background image, a paste position where the detection object image is to be pasted and a paste position where the non-detection object image is to be pasted (step S904).

[0103] The setting unit 133 sets a value for each pasting parameter for each detection object image and each non-detection object image (step S905).

[0104] The image generation unit 134 performs image composition based on the parameters (step S906). Specifically, the image generation unit 134 controls the aspect of each detection object image according to the parameters set for each detection object image, and performs image composition by pasting the controlled detection object images so as to overlap each other at the pasting position. Furthermore, the image generation unit 134 controls the aspect of each non-detection object image according to the parameters set for each non-detection object image, and performs image composition by pasting the controlled non-detection object image so as to overlap the detection object image.

[0105] The learning data generating unit 135 generates learning data by associating annotation information, including position information of the detection object on the composite image and the type of the detection object, with the composite image (step S907).

[0106] The process returns to step S901, and the learning unit 136 determines whether the required number of pieces of learning data have been collected. If the required number of pieces of learning data have been collected (step S901; Yes), the learning unit 136 trains the model using the learning data collected so far (step S908).

[0107] 6. Other Embodiments Other embodiments of the generating device 100 will be described below. The generating device 100 may extract depth information from a background image and set parameter values ​​according to the extracted depth information. The generating device 100 may then paste a detection object image onto the background image based on the parameter values ​​set according to the depth information. Specifically, the generating device 100 pastes the detection object image so as to create a sense of perspective in accordance with the depth of the background image according to the depth information.

[0108] 10 is a diagram showing a specific example of a depth information-based pasting technique, in which an image of a detected object is pasted onto a background image in order to increase the reproduction rate of a more realistic scene in which the detected object is placed.

[0109] 10(a), the acquisition unit 131 acquires a background image BG2 with a scene of a private room as the background, and also acquires a detected object image TG11 from which clothing parts have been extracted as a detected object image corresponding to the scene. Furthermore, the position designation unit 132 designates a pasting position P21 on the background image BG2 as a pasting position for pasting the detected object image TG11 onto the background image BG2. As shown in FIG. 10(a), the pasting position P21 may be designated as a rectangular area.

[0110] In this state, as shown in FIG. 10(b), for example, the setting unit 133 estimates depth information of the background image BG2. The setting unit 133 can estimate the depth information of the background image BG2 using any depth estimation model. The parameters described below are set based on the depth information. Note that the generating device 100 may also include an estimation unit that performs depth estimation.

[0111] FIG. 10(c) shows an example in which the setting unit 133 sets "enlargement / reduction," "rotation," and "color" as parameters when pasting the detection object image TG11 at the pasting position P21 on the background image BG2.

[0112] 10(c)(1), the setting unit 133 sets a scaling value (scaling ratio) for the detection object image TG11. For example, the setting unit 133 sets a scaling value for each detection object image TG11 based on depth information so that a sense of perspective is created between the detection object images TG11 when the detection object images TG11 are pasted onto the background image BG2. The image generation unit 134 enlarges or reduces the detection object image TG11 based on the scaling value.

[0113] 10(c)(2), the setting unit 133 sets a rotation value (rotation rate) for the detection object image TG11. For example, the setting unit 133 sets a rotation value for each detection object image TG11 based on depth information so that a sense of perspective is created between the detection object images TG11 when the detection object images TG11 are pasted onto the background image BG2.

[0114] 10(c)(3), the setting unit 133 sets a color value (saturation) for each detection object image TG11. For example, the setting unit 133 sets a color value for each detection object image TG11 based on depth information so that when the detection object image TG11 is pasted onto the background image BG2, it is possible to express how the amount of light hitting the image changes in accordance with the depth of the background image BG2.

[0115] Here, the image generation unit 134 controls the appearance of the detection object image TG11 in accordance with the parameters set by the setting unit 133, and performs image synthesis by pasting the controlled detection object image TG11 side by side at the pasting position P21. FIG. 10(d) shows an example of a synthetic image SG2 generated by the image generation unit 134 through image synthesis. The synthetic image SG2 reproduces a realistic situation in which clothes are hung on the wall at the pasting position P21 on the background image BG2, in accordance with the depth of the background image BG2. In other words, the pasting technique described in FIG. 10 can appropriately reproduce a realistic scene in which a detection object is placed.

[0116] [7. Hardware Configuration] The generating device 100 according to the embodiment may be realized, for example, by a computer 1000 configured as shown in Fig. 11. Fig. 11 is a hardware configuration diagram showing an example of a computer that realizes the functions of the generating device 100 according to the embodiment. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0117] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0118] The HDD 1400 stores programs executed by the CPU 1100, data used by these programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0119] The CPU 1100 controls an output device such as a display and an input device such as a keyboard via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. The CPU 1100 also outputs generated data to the output device via the input / output interface 1600.

[0120] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0121] For example, when the computer 1000 functions as the generating device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200, thereby realizing the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0122] [8. Other] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0123] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0124] Furthermore, the above-described embodiments can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0125] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the aspects described in the "present invention" section and that have been modified and improved in various ways based on the knowledge of those skilled in the art. [Explanation of symbols]

[0126] 1. Information Processing Systems 2. Imaging system 10. Imaging device 11 Display control device 12 Display device 60 Detection Device 100 generator 130 control section 131 Acquisition Department 132 Position specification part 133 Settings 134 Image Generation Unit 135 Learning Data Generation Unit 136 Learning Department

Claims

1. an acquisition unit that acquires a first image and a second image as images of a predetermined object; a generating unit that generates a composite image by pasting the first image and the second image onto a background image in a superimposed state; a learning data generation unit that generates learning data used for training an object detection model that detects a predetermined object from an input image by associating information indicating the predetermined object in the synthetic image with the synthetic image; A generating device comprising:

2. the predetermined object is an object to be detected by the object detection model, The acquisition unit acquires images of the detection object as the first image and the second image. The generating device of claim 1 .

3. the predetermined object is a detection target object by the object detection model and a non-detection target object that is not a detection target object by the object detection model, The acquisition unit acquires an image of the detection target object as the first image and an image of the non-detection target object as the second image. The generating device of claim 1 .

4. a position designation unit that designates a pasting position where the image of the predetermined object is pasted on the background image; Further preparation, The generation unit generates the composite image by pasting the first image and the second image onto the background image so as to overlap the first image and the second image based on the pasting position.

4. The generating device according to claim 2 or 3.

5. a setting unit that sets an overlapping degree between the first image and the second image; Further preparation, The generation unit superimposes the first image and the second image based on the degree of overlap.

4. The generating device according to claim 2 or 3.

6. the setting unit randomly sets the number of overlapping degrees corresponding to the number of combinations of the first image and the second image; The generation unit superimposes the first image and the second image using the overlapping degree that is randomly set for each combination of the first image and the second image. The generating device of claim 5 .

7. The generation unit superimposes the second image on the first image or superimposes the first image on the second image based on a superimposition relationship determined according to a scene of the background image. The generating device of claim 3 .

8. The acquisition unit acquires, as the second image, an image of the non-detection object corresponding to a scene in the background image. The generating device of claim 3 .

9. The learning data generation unit generates the learning data by associating annotation information, which includes position information of the predetermined object in the synthetic image and a type of the predetermined object, with the synthetic image as information indicating the predetermined object in the synthetic image. The generating device of claim 1 .

10. A generation method executed by a generation device, comprising: an acquisition step of acquiring a first image and a second image as images of a predetermined object; a generating step of generating a composite image in which the first image and the second image are pasted onto a background image in a superimposed state; a learning data generation step of generating learning data using information indicating the predetermined object in the synthetic image and information associating the synthetic image with each other, the learning data being used for training an object detection model for detecting the predetermined object from an input image; A generation method including:

11. A generation program executed by a generation device, an acquisition step of acquiring a first image and a second image as images of a predetermined object; a generating step of generating a composite image in which the first image and the second image are pasted onto a background image in a superimposed state; a learning data generation step of generating learning data, which is information that associates information indicating the predetermined object in the synthetic image with the synthetic image, to be used for training an object detection model that detects the predetermined object from an input image; A generation program that causes the generation device to execute the above.

Images