Generation apparatus, generation method, and generation program

Abstract

To improve the accuracy of an object detection model by generating appropriate teacher data from a small number of labeled data as learning data of the object detection model.SOLUTION: A generation device 10 includes: an acquisition section 11 that acquires teacher data to which a label indicating a class of an object included in an image and position information of the object are given; a mask generation section 12 that generates a mask for cutting out the object from the teacher data using a mask generation model; an object extraction section 13 that, on the basis of the mask, extracts an area in which the object is reflected from the teacher data as an object image, and gives the extracted object image the label of the object and output it; a pasting section 142 that randomly pastes the object image on a pasting destination image; and a smoothing processing section 143 that generates an image converted so that a boundary portion between background of the pasting destination image and the object image is smooth.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a generation device, a generation method, and a generation program.

Background Art

[0002] As one of the most basic image processing tasks using machine learning, there is an object detection model that detects the class and position of a specific object in an image or video.

[0003] In order to improve the detection accuracy of an object detection model, it is common to provide a large amount of teacher data to the object detection model as learning data. However, teacher data is often generated manually, and it is not easy to prepare a large amount of labeled data at once.

[0004] In particular, the labeling cost for setting class labels and bounding boxes required for object detection is extremely high. Furthermore, the method of attaching correct labels also varies depending on the individual who labels, and sometimes domain knowledge is required. Therefore, it is extremely difficult to create teacher data with a large number of correct labels at once.

[0005] Here, in order to solve these problems, data augmentation has been proposed. Data augmentation is a technique for improving the generalization performance of a model by augmenting images similar to an existing dataset and adding them to the learning data of an object detection model.

[0006] Data augmentation can generate a sufficient amount of teacher data for learning based on a small amount of teacher data without changing the architecture of the object detection model. As such data augmentation, various methods have been proposed, from simple rule-based ones to complex methods using neural networks.

Prior Art Documents

Non-Patent Documents

[0007]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

Non-Patent Document 5

Non-licensed Document 6

Non-licensed Document 7

Non-licensed literature 9

[0008] Traditionally, there is a data augmentation technique that involves cutting out labeled bounding boxes and pasting them onto a background image. However, this method can result in unnatural appearances due to differences between the background of the source image and the background of the destination image, and it may not be possible to sufficiently improve the accuracy of the object detection model.

[0009] The present invention has been made in view of the above, and aims to provide a generation device, generation method, and generation program that can improve the accuracy of an object detection model by generating appropriate training data from a small amount of labeled data as training data for an object detection model. [Means for solving the problem]

[0010] To solve the above-mentioned problems and achieve the objective, the generation apparatus according to the present invention is characterized by comprising: an acquisition unit that acquires a first image to which a label indicating the class of an object contained in the image and position information of the object are attached; a mask generation unit that generates a mask for cutting out the object from the first image using a mask generation model trained by unsupervised learning; an extraction unit that extracts the region in which the object is depicted from the first image as an object image based on the mask, and outputs the extracted object image with the label of the object attached; a pasting unit that randomly pastes the object image onto a second image which is the pasting destination; and a transformation unit that generates a third image by smoothly transforming the boundary portion between the background of the second image and the object image. [Effects of the Invention]

[0011] According to the present invention, the accuracy of an object detection model can be improved by generating appropriate training data from a small amount of labeled data as training data for the object detection model. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a schematic diagram showing an example of the configuration of the processing system in Embodiment 1. [Figure 2] Figure 2 is a schematic diagram showing an example of the configuration of the generating apparatus shown in Figure 1. [Figure 3] Figure 3 is a diagram illustrating the overview of the processing in the generation apparatus shown in Figure 2. [Figure 4] Figure 4 is a diagram illustrating the overview of the processing in the generation apparatus shown in Figure 2. [Figure 5-1] Figure 5-1 is a diagram illustrating the overview of the processing in the generation apparatus shown in Figure 2. [Figure 5-2] Figure 5-2 is a diagram illustrating the overview of the processing in the generating apparatus shown in Figure 2. [Figure 6] FIG. 6 is a flowchart showing a processing procedure of the generation process according to Embodiment 1. [Figure 7] FIG. 7 is a diagram for explaining the prior art. [Figure 8] FIG. 8 is a diagram for explaining object detection in Embodiment 1. [Figure 9] FIG. 9 is a diagram schematically showing an example of the configuration of the generation device according to Embodiment 2. [Figure 10] FIG. 10 is a diagram for explaining the processing of the generation device shown in FIG. 9. [Figure 11] FIG. 11 is a flowchart showing a processing procedure of the generation process according to Embodiment 2. [Figure 12] FIG. 12 is a diagram for explaining the processing of Embodiment 2. [Figure 13] FIG. 13 is a diagram showing a computer that executes a program.

MODE FOR CARRYING OUT THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited by this embodiment. In the description of the drawings, the same parts are denoted by the same reference numerals.

[0014] [Embodiment 1] FIG. 1 is a diagram schematically showing an example of the configuration of the processing system in Embodiment 1. The processing system in Embodiment 1 includes a learning device 20 and an object detection device 30.

[0015]

[0016] The learning device 20 executes learning of an object detection model that detects position information and labels of specific objects in images and videos. The object detection device 30 uses the object detection model learned by the learning device 20 to detect the labels and position information of objects appearing in an image or video that is test data.The processing system in Embodiment 1 has a generation device 10 in front of the learning device 20.

[0017] The generation device 10 outputs augmented data (third image), which is obtained by augmenting the training data (first image), to the training device 20 as training data for the object detection model. The training data is image data used for training the object detection model. Each image data in the training data is assigned a label indicating the class of the object contained in the image and the position information of the object.

[0018] [Generation device] Next, the generation apparatus 10 shown in Figure 1 will be described. Figure 2 is a schematic diagram showing an example of the configuration of the generation apparatus 10 shown in Figure 1.

[0019] The generation device 10 is realized, for example, by loading a predetermined program into a computer including ROM (Read Only Memory), RAM (Random Access Memory), CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The generation device 10 also has a communication interface for sending and receiving various information with other devices connected via a network or the like.

[0020] As shown in Figure 2, the generation device 10 includes an acquisition unit 11, a mask generation unit 12, an object extraction unit 13 (extraction unit), a generation unit 14, and an output unit 15.

[0021] The acquisition unit 11 acquires training data by accepting input of training data.

[0022] The mask generation unit 12 generates a mask from the training data using a mask generation model trained by unsupervised learning. The mask extracts objects from the training data by masking the areas of the training data that do not contain objects. The mask generation model can employ DNN (Deep Neural Network) architectures such as GAN (Generative Adversarial Network) or unsupervised segmentation mask generation models such as CP-GAN (Context Pyramid Generative Adversarial Network).

[0023] The object extraction unit 13 extracts the regions containing objects from the training data as object images, based on the mask generated by the mask generation unit 12. The object extraction unit 13 then adds object labels to the extracted object images and outputs them.

[0024] The generation unit 14 generates augmented data. The generation unit 14 includes an adhesive unit 142 and a smoothing processing unit 143 (conversion unit).

[0025] The pasting unit 142 acquires the destination image (second image) and randomly pastes the object images extracted by the object extraction unit 13 onto this destination image.

[0026] In this process, the pasting unit 142 sets the label (first label), number (first number), and size (first size) of the object images to be pasted onto the destination image, based on statistical information regarding the labels, number, and size of the object images in the training data. The pasting unit 142 then pastes the object images with the set labels onto the destination image in the set number and size.

[0027] The smoothing processing unit 143 generates an augmented image (third image) by smoothing the boundary between the background of the destination image and the object image to be pasted onto this destination image.

[0028] For example, if the imaging time is different (e.g., night and day) or the way the light source hits the image is different, an unnatural appearance may occur at the boundary between the background of the pasted image and the object image. The smoothing processing unit 143 adjusts the contrast between the pasted image and the object image, as well as the overall brightness of the image, to make the boundary between the background of the pasted image and the object image appear smooth.

[0029] The smoothing processing unit 143 employs Blur (for example, Gaussian Blur (References 1, 2)) and Poisson Blending (Non-Patent Literature 6) to smooth the boundary between the background of the destination image and the object image to be pasted, thereby converting it into a natural image. Reference 1: Blurred Borders in CSS, [online], [Retrieved May 24, 2022], Internet<URL:URL:https: / / css-tricks.com / blurred-borders-in-css / > Reference 2: Gaussian Blur, [online], [Retrieved May 24, 2022], Internet<URL:https: / / www.sciencedirect.com / topics / engineering / gaussian-blur>

[0030] The output unit 15 outputs the augmented data to the learning device 20 as training data for an object detection model that detects the location information and labels of specific objects in the image.

[0031] In the augmented data generated by the generation device 10, the boundary between the object image to be pasted onto the destination image is smoothly transformed. Therefore, the augmented data does not have any unnaturalness at the boundary between the background of the source image and the background of the destination image. For this reason, the learning device 20 can improve the accuracy of the object detection model by training the object detection model with this augmented data as training data.

[0032] [Overview of the generation device's processing] Next, the processing of the generating apparatus 10 will be described with reference to Figures 3, 4, 5-1, and 5-2. Figures 3, 4, 5-1, and 5-2 are diagrams illustrating the overview of the processing in the generating apparatus 10 shown in Figure 2.

[0033] In the generation device 10, the acquisition unit 11 acquires training data in which the object to be pasted is shown. The training data is, for example, an image of a dog, with the label "dog" and the dog's location information attached. The generation device 10 cuts out the labeled Bounding Box Gs from the training data (Figure 3 (1)).

[0034] The mask generation unit 12 uses an unsupervised segmentation mask generation model to generate a mask Ms that can mask areas of the Bounding Box Gs other than objects (Figure 3(2)).

[0035] Next, the object extraction unit 13 uses a mask Ms to extract the region containing only objects from the Bounding Box Gs cut out from the training data as an object image Ga (Figure 3 (3)). For example, the object extraction unit 13 removes the background from a Bounding Box Gs containing a dog, cat, or bird, and extracts only the region containing the object (dog, cat, bird, etc.) as an object image Ga. Then, the object extraction unit 13 assigns an object label to each object image.

[0036] The pasting unit 142 obtains the destination image G1, which is the image to which the object image Ga will be pasted (Figure 3 (4)). For example, the destination image G1 is an image showing a plain and the sky.

[0037] The pasting unit 142 randomly pastes object images Ga onto the destination image G1 (Figure 3 (5)). For example, the pasting unit 142 randomly pastes dog, cat, and bird object images Ga onto the destination image G1.

[0038] Then, the smoothing processing unit 143 performs boundary smoothing, which smooths the boundary between the background of the destination image G1 and each object image Ga to be pasted onto the destination image G1, thereby converting it into a natural image (Figure 3 (6)).

[0039] The generation unit 14 generates multiple composite images G2 in which the boundaries between the background of the destination image G1 and each object image Ga are smoothed (Figure 3 (7)). The output unit 15 outputs the generated multiple composite images G2 as augmented data to the learning device 20.

[0040] Here, as shown in Figure 4, the pasting unit 142 determines the size, number, labels, etc. of the object image Ga to be pasted onto the destination image G1 based on the statistical information of the training data Gt.

[0041] The pasting unit 142 extracts statistical information from the training data Gt regarding the labels of the object images contained in the training data Gt, the number of each label, and the size of the objects (Figure 4 (1)). For example, the pasting unit 142 extracts the number of objects corresponding to the labels dog, cat, and bird, and the size of each object (Figure 4 shows statistical information for the size of dogs).

[0042] The paste unit 142 then determines the type of probability distribution and the high parameters from this statistical information (Figure 4 (2), (3)). For example, the probability distribution may be a normal distribution, log-normal distribution, Poisson distribution, GMM (Gaussian Mixture Model), kernel density function, etc.

[0043] The pasting unit 142 then sets the label, number, and size of the object image Ga to be pasted based on the set probability distribution. The pasting unit 142 then pastes the object image Ga with the set label onto the destination image G1 in the set number and size (Figure 4 (4)) to generate the composite image G2 (Figure 4 (5)).

[0044] This section describes a case where we want to generate augmented data with the same trend as the training data Gt for object images labeled "dog". In this case, the pasting unit 142 adjusts the high parameters to create a probability distribution similar to the distribution of size and number of dog object images in the training data Gt (Figure 5-1). The pasting unit 142 sets the number and size of dog object images to be pasted according to the generated probability distribution.

[0045] In this way, the generation device 10 generates multiple composite images by pasting object images Ga onto the destination image G1 according to the statistical information of the training data Gt. Therefore, the generation device 10 can generate augmented data in which appropriate object images are placed in appropriate numbers and sizes from a small amount of labeled training data Gt. Since the object detection model can learn from a large amount of appropriate augmented data according to the statistical information of the training data Gt, an improvement in the accuracy of the object detection model can also be expected.

[0046] Furthermore, the generation device 10 automatically generates augmented data after receiving the training data Gt as input. Therefore, with the generation device 10, highly accurate training data can be easily obtained without the need for complicated processing such as labeling by an operator.

[0047] Next, we will explain the case where we want to generate augmented data by intentionally attaching larger objects to object images labeled as "dog". In this case, the pasting unit 142 increases the variance parameter for the probability distribution, widening the tails of the distribution as shown in Figure 5-2. The pasting unit 142 sets the number and size of dog object images to be pasted according to the generated probability distribution shown in Figure 5-2.

[0048] In other words, the pasting unit 142 sets a label (second label), a number (second number), and a size (second size) that deviate from the statistical information, and pastes the object image of this label onto the destination image G1 with a size and number that deviates from the statistical information. The degree of deviation from the statistical information is set in advance and updated as needed.

[0049] Thus, the pasting section 142 may paste object images onto the destination image G1 with labels, numbers, and sizes corresponding to outliers that deviate from the statistical information. By training with augmented data created in this way, the object detection model can learn about objects that appear with sizes and other characteristics that deviate from the statistical information, and thus an improvement in the accuracy of the object detection model can be expected.

[0050] [Processing steps for generation] Next, the processing procedure of the generation process performed by the generation device 10 will be described. Figure 6 is a flowchart showing the processing procedure of the generation process according to Embodiment 1.

[0051] As shown in Figure 6, in the generation device 10, once the acquisition unit 11 acquires training data (step S1), the mask generation unit 12 generates a mask from the training data using a mask generation model that has been trained by unsupervised learning (step S2).

[0052] The object extraction unit 13 extracts the region containing the object from the training data as an object image based on the mask generated by the mask generation unit 12 (step S3).

[0053] The pasting unit 142 acquires the destination image and randomly pastes the object images extracted by the object extraction unit 13 onto this destination image (step S4).

[0054] The smoothing processing unit 143 performs a smoothing process to generate an image in which the boundary between the background of the destination image and the object image to be pasted onto this destination image is smoothed (step S5). The output unit 15 outputs the smoothed image as augmented data to the learning device 20 (step S6).

[0055] [Effects of Embodiment 1] Figure 7 illustrates the conventional technique. Conventionally, when augmenting a small amount of labeled training data Gt with similar data (Figure 7(1)), the labeled bounding boxes were simply cut out from the training data Gt and pasted onto the target image.

[0056] When augmented data Gp' is used to train an object detection model (Figure 7(2)), the accuracy of the object detection model may not be sufficiently improved due to unnaturalness caused by the difference between the boundary of the source bounding box and the background of the destination image. For this reason, with conventional techniques, even when test data is input into a trained object detection model (Figure 7(3)), the accuracy of the object labels and location information output by the object detection model may not be sufficient (Figure 7(4)). For example, conventional object detection models may not be able to detect some objects.

[0057] Figure 8 illustrates object detection in Embodiment 1. When the generation device 10 augments training data from labeled training data Gt (Figure 8(1)), it extracts only object images from the training data. The generation device 10 then generates augmented data by smoothly transforming the boundary between the background of the destination image and the object image to be pasted onto this destination image (Figure 8(1)). The generation device 10 also generates augmented data by determining the size of the object image and the ratio of labels in the data to be generated from statistical information of the training data.

[0058] The learning device 20 trains the object detection model with a sufficient amount of augmented data Gp generated by the generation device 10 (Figure 8(2)). Because the augmented data Gp has smoothed boundaries between the object image boundaries and the background of the pasting image, the object detection model can learn appropriate augmented data Gp without unnaturalness, thereby significantly improving its accuracy.

[0059] Therefore, when test data is input into a trained object detection model (Figure 8 (3)), it is considered that sufficient detection accuracy can be ensured between the object labels and location information output by the object detection model (Figure 8 (4)).

[0060] Thus, the augmented data generated by the generation device 10 has a smooth transition at the boundary between the object image to be pasted onto the destination image, and there is no unnaturalness at the boundary between the background of the source image and the background of the destination image. For this reason, the learning device 20 can improve the accuracy of the object detection model by training the object detection model with this augmented data Gp as training data.

[0061] [Embodiment 2] Next, Embodiment 2 will be described. Figure 9 is a schematic diagram showing an example of the configuration of the generation apparatus according to Embodiment 2. Figure 10 is a diagram illustrating the processing of the generation apparatus shown in Figure 9.

[0062] As shown in Figure 9, the generation device 210 according to Embodiment 2 has a generation unit 214 instead of a generation unit 14, compared to the generation device 10 shown in Figure 2. The generation unit 214 has a detection unit 2141, an adhesive unit 2142, and a smoothing processing unit 143.

[0063] The detection unit 2141 detects the boundary between regions of different types in the destination image. The detection unit 2141 detects the horizon H1 that appears in the destination image G1 (Figure 10 (1)). The detection unit 2141 detects the horizon H1, which is the boundary between the sky and the ground, using, for example, a Hough transform or a line segment Hough transform.

[0064] The pasting unit 2142 pastes object images, each labeled with a corresponding region of the destination image, into each region of the destination image. The generation device 210 stores pasting rules that associate identification information of regions in the destination image with identification information of the corresponding labels. For example, in the pasting rules, the region "sky" is associated with the label "bird". Also, in the pasting rules, the region "ground" is associated with the labels "dog" and "cat".

[0065] The pasting unit 2142 places appropriate object images Ga above and below the horizon H1 in the pasting destination image G1' in which the horizon H1 has been detected (Figure 10 (2)).

[0066] For example, the pasting unit 2142 pastes the "dog" and "cat" object images from the object image Ga onto the destination image G1' such that the lower edge of the object image is located in the area "ground" below the horizon H1. Also, the pasting unit 2142 pastes the "bird" object image from the object image Ga onto the destination image G1' such that the lower edge of the object image is located in the area "sky" above the horizon H1.

[0067] The pasting unit 2142, like the pasting unit 142, determines the size, number, labels, etc., of the object images Ga to be pasted onto the destination image G1' based on the statistical information of the training data Gt. Alternatively, the pasting unit 2142 may set labels, numbers, and sizes that are outside the statistical information, and paste the object images with these labels onto the destination image G1' with sizes and numbers that are outside the statistical information.

[0068] Then, the generation device 210 performs boundary smoothing by the smoothing processing unit 143, which smooths the boundary between the background of the destination image G1 and each object image Ga to be pasted onto the destination image G1, thereby converting it into a natural image (Figure 10 (3)). The generation device 210 generates multiple composite images G3 (Figure 10 (4)).

[0069] [Processing steps for generation] Next, the processing procedure of the generation process performed by the generation device 210 will be described. Figure 11 is a flowchart showing the processing procedure of the generation process according to Embodiment 2.

[0070] Steps S11 to S13 shown in Figure 11 are the same processes as steps S1 to S3 shown in Figure 6.

[0071] In the generation device 210, the detection unit 2141 detects the horizon visible in the destination image (step S14). Then, the pasting unit 2142 pastes object images, each labeled with a corresponding region of the destination image, onto each region of the destination image (step S15).

[0072] Steps S16 and S17 shown in Figure 11 are the same processes as steps S5 and S6 shown in Figure 6.

[0073] [Effects of Embodiment 2] Figure 12 is a diagram illustrating the processing of Embodiment 2. As shown in Figure 12, if an object image Ga is placed on the destination image G1 without considering its attributes, the object may be placed in an area where it does not actually exist. For example, as in the composite image G4, the dog object image may be placed in mid-air. If such an unnatural composite image G4 is used as training data for an object detection model, the detection accuracy of the object detection model may decrease.

[0074] In contrast, the generation device 210 according to Embodiment 2 determines the boundaries between regions of different types in the destination image and generates an augmented image by appropriately pasting object images, each labeled with a corresponding label to the region of the destination image, into each region of the destination image. In other words, for any object, the generation device 210 pastes an object image in which the object is depicted into a region where the presence of that object is natural.

[0075] Therefore, the generation device 210 pastes object images containing objects that would naturally be present in each region of the destination image, and generates augmented data by smoothly transforming the boundary between the background of the destination image and the object images. As a result, the learning device 20 can improve the accuracy of the object detection model by training the object detection model with the augmented data generated by the generation device 210, which does not appear unnatural.

[0076] [System configuration, etc.] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those illustrated, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Moreover, each processing function performed by each device can be implemented, in whole or in any part, by a CPU or GPU and programs that are analyzed and executed by said CPU or GPU, or by hardware using wired logic.

[0077] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0078] [program] Furthermore, it is also possible to create a program that describes the processes performed by the generation devices 10 and 210 described in the above embodiment in a computer-executable language. For example, it is possible to create a program that describes the processes performed by the generation devices 10 and 210 in the embodiment in a computer-executable language. In this case, the same effects as in the above embodiment can be obtained by having a computer execute the program. Moreover, the same processes as in the above embodiment may be realized by recording such a program on a computer-readable recording medium and having a computer read and execute the program recorded on this recording medium.

[0079] Figure 13 shows a computer running a program. As illustrated in Figure 13, computer 1000 includes, for example, memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070, all of which are connected by bus 1080.

[0080] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012, as illustrated in Figure 13. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090, as illustrated in Figure 13. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0081] Here, as illustrated in Figure 13, the hard disk drive 1090 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. That is, the above-mentioned program is stored in the hard disk drive 1090, for example, as a program module in which instructions to be executed by the computer 1000 are described.

[0082] Furthermore, the various data described in the above embodiment are stored as program data in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes various processing procedures.

[0083] Furthermore, the program module 1093 and program data 1094 related to the program are not limited to being stored on the hard disk drive 1090, but may also be stored on a removable storage medium, for example, and read by the CPU 1020 via a disk drive or the like. Alternatively, the program module 1093 and program data 1094 related to the program may be stored on another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.) and read by the CPU 1020 via the network interface 1070.

[0084] The embodiments described above and their variations are included in the art disclosed herein, as well as in the scope of the invention described in the claims and its equivalents. [Explanation of Symbols]

[0085] 10,210 generator 11 Acquisition Department 12 Mask generation unit 13. Object Extraction Section 14,214 generator 15 Output section 20 Learning device 30 Object detection device 142,2142 Paste area 143 Smoothing Processing Unit 2141 Detection unit

Claims

1. An acquisition unit acquires a first image to which labels indicating the class of objects contained in the image and position information of the objects are assigned; A mask generation unit generates a mask for cutting out an object from the first image using a mask generation model trained by unsupervised learning, Based on the mask, the extraction unit extracts the region containing the object from the first image as an object image, and outputs the extracted object image with a label for the object. A storage unit that stores a paste rule in which identification information of a region of the image to be pasted and identification information of a label corresponding to the said region are associated, A detection unit detects the boundary between regions of different types in the second image to which the image is pasted, A pasting unit that pastes the object images, each labeled according to the pasting rules, onto each region of the second image such that the lower end of each object image, each labeled according to the pasting rules, is located in the corresponding region of the second image. A transformation unit generates a third image by smoothly transforming the boundary between the background of the second image and the object image, An output unit outputs the third image as training data for an object detection model that detects the position information and labels of specific objects in the image, A generating apparatus characterized by having the following features.

2. The pasting unit sets a first label, a first number, and a first size for the object images to be pasted onto the second image based on statistical information relating to the label, number, and size of the object images for the first image, and pastes the object images to which the first label has been assigned onto the second image in the first number and first size, as described in claim 1.

3. The generating apparatus according to claim 2, characterized in that the pasting section sets a second label, a second number, and a second size that are excluded from the statistical information, and pastes the object image to which the second label is assigned onto the second image with the second number and the second size.

4. The generation apparatus according to claim 1, characterized in that the detection unit detects the horizon that appears in the second image.

5. A generation method performed by a generation device, The generating device has a storage unit that stores pasting rules, which associate identification information of a region in the destination image with identification information of a label corresponding to the region. A step of acquiring a first image to which a label indicating the class of an object contained in the image and position information of the object are assigned; A step of generating a mask to cut out an object from the first image using a mask generation model trained by unsupervised learning, Based on the mask, the process involves extracting the region containing the object from the first image as an object image, assigning a label to the extracted object image, and outputting it. The process involves detecting the boundary between regions of different types in the second image to which the image will be pasted, The process involves pasting the object images, each labeled according to the pasting rules, onto each region of the second image such that the lower end of each object image, each labeled according to the pasting rules, is located in the corresponding region of the second image. A step of generating a third image by smoothly transforming the boundary between the background of the second image and the object image, The process involves outputting the third image as training data for an object detection model that detects the position information and labels of specific objects within the image, A method for generating a product that includes the following:

6. A step of obtaining a first image to which labels indicating the class of objects contained in the image and position information of the objects are assigned; A step of generating a mask to cut out an object from the first image using a mask generation model trained by unsupervised learning, Based on the mask, the region containing the object is extracted from the first image as an object image, and the extracted object image is labeled with the object's label and output. The second image, which is the destination for pasting, includes the step of detecting the boundary between regions of different types, The steps include: pasting the object images, each labeled with a label corresponding to a region of the second image, onto each region of the second image, in accordance with a pasting rule that associates identification information of a region of the destination image with identification information of a label corresponding to the region, such that the lower end of the object image, each labeled with a label corresponding to a region of the second image, is located in the corresponding region of the second image; The steps include generating a third image by smoothly transforming the boundary between the background of the second image and the object image, The third image is output as training data for an object detection model that detects the position information and labels of specific objects within the image. A generation program that causes a computer to execute something.

Images