A training data generation device, a training data generation method, and a training data generation program.

Abstract

The present invention provides a training data generation device comprising a first acquisition unit that acquires a plurality of first images obtained by imaging an object, a second acquisition unit that acquires a plurality of second images obtained by capturing an area that could be the background of the object, and a generation unit that generates training data by superimposing each of the plurality of first images with each of the plurality of second images.

Description

Technical Field

[0001] The disclosed technology relates to a learning data generation device, a learning data generation method, and a learning data generation program.

Background Art

[0002] In recent years, object detection means for identifying each object photographed in an image using an image has been established and utilized in various applications. However, in order to utilize it, it is necessary to generate a large number of learning data for learning about each object to be identified, and since the number of commercially available learning data sets is small, it has become an obstacle to introduction.

[0003] The generation of learning data mentioned here is also called annotation. Generally, for each image in a huge amount of image data sets, the pixel area of each object to be detected shown in the image is given as rectangular information and generated with a set of rectangular coordinates. Annotating each object from each image requires a great deal of labor.

[0004] For example, Non-Patent Document 1 describes existing research on annotation work. In this Non-Patent Document 1, in order to ensure accuracy, as shown in FIG. 16, not only does one person attach a plurality of Bounding Boxes (BBs, representing the rectangles of annotations) to one image, but first one person attaches BBs, then another person checks the quality, and still another person checks whether all categories of the image have been annotated. A technique for risk hedging is described.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] However, even with the technology described in Non-Patent Document 1, a certain number of annotations are still required initially. Therefore, there is a need to easily generate more versatile training data.

[0007] The disclosed technology was developed in view of the above points, and aims to provide a training data generation device, a training data generation method, and a training data generation program that can easily generate training data for use in machine learning. [Means for solving the problem]

[0008] A first aspect of the present disclosure is a learning data generation apparatus comprising: a first acquisition unit that acquires a plurality of first images including an object; a second acquisition unit that acquires a plurality of second images including a region that may be the background of the object; and a generation unit that generates learning data by superimposing each of the plurality of first images and each of the plurality of second images.

[0009] A second aspect of this disclosure is a method for generating training data, comprising: acquiring a plurality of first images including an object; acquiring a plurality of second images including a region that may be the background of the object; and generating training data by superimposing each of the plurality of first images and each of the plurality of second images.

[0010] A third aspect of this disclosure is a training data generation program that causes a computer to acquire a plurality of first images including an object, acquire a plurality of second images including a region that may be the background of the object, and generate training data by superimposing each of the plurality of first images and each of the plurality of second images. [Effects of the Invention]

[0011] The disclosed technology has the effect of making it possible to easily generate training data used for machine learning. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing an example of the hardware configuration of a learning data generation device according to the first embodiment. [Figure 2] This is a block diagram showing an example of the functional configuration of a learning data generation device according to the first embodiment. [Figure 3A] This is a diagram showing an example of the first image according to the embodiment. [Figure 3B] This figure shows an example of numbers that can be used in the first image. [Figure 4] This figure shows an example of images used for training data according to the embodiment. [Figure 5] This figure shows an example of the processed first image. [Figure 6] This figure shows an example of training data with annotation information. [Figure 7A] This figure shows another example of the first image according to the embodiment. [Figure 7B] This figure shows another example of the first image according to the embodiment. [Figure 8] This figure shows another example of training data images according to the embodiment. [Figure 9] This flowchart shows an example of the processing flow by the training data generation program according to the first embodiment. [Figure 10] This is a block diagram showing an example of the functional configuration of a learning data generation device according to the second embodiment. [Figure 11] This figure shows an example of an image to be estimated and an estimation result according to the embodiment. [Figure 12] This figure illustrates a region extraction process that extracts a first image from images of multiple different objects. [Figure 13] This figure illustrates the region extraction process applied to the target image. [Figure 14]It is a diagram showing yet another example of the first image according to the embodiment. [Figure 15] It is a diagram showing an example of a combination that can be assumed from a plurality of objects. [Figure 16] It is a diagram showing the prior art.

Mode for Carrying Out the Invention

[0013] Hereinafter, an example of an embodiment of the disclosed technology will be described while referring to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for the convenience of explanation and may be different from the actual ratios.

[0014] The learning data generation device according to the present embodiment provides a specific improvement over the conventional method of generating learning data used for machine learning using machine learning, and shows an improvement in the technical field of generating learning data used for machine learning.

[0015] [First Embodiment] In the learning data generation device according to the present embodiment, learning data used for machine learning is generated by superimposing an image of a target object on a background image. As a result, an annotated image is pseudo-reproduced, that is, an image with annotations pre-given can be obtained. Compared with the conventional method of annotating an object from an image, learning data can be generated easily.

[0016] FIG. 1 is a block diagram showing an example of the hardware configuration of the learning data generation device 10 according to the first embodiment.

[0017] As shown in Figure 1, the learning data generation device 10 includes a CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, storage 14, input unit 15, display unit 16, and communication interface (I / F) 17. Each component is connected to the others via a bus 18 so that they can communicate with each other.

[0018] The CPU 11 is a central processing unit that executes various programs and controls various components. Specifically, the CPU 11 reads a program from the ROM 12 or storage 14 and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculations according to the program stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores a training data generation program for executing the training data generation process. Note that a GPU (Graphics Processing Unit) may be used instead of the CPU.

[0019] ROM12 stores various programs and data. RAM13 temporarily stores programs or data as a working area. Storage14 consists of an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs, including the operating system, and various data.

[0020] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information to the device.

[0021] The display unit 16 is, for example, a liquid crystal display and displays various information. The display unit 16 may also function as an input unit 15 by employing a touch panel system.

[0022] The communication interface 17 is an interface for the device to communicate with other external devices. For this communication, a wired communication standard such as Ethernet (registered trademark) or FDDI (Fiber Distributed Data Interface), or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) may be used.

[0023] In this embodiment, the learning data generation device 10 can be a general-purpose computer device such as a server computer or a personal computer (PC).

[0024] Next, with reference to Figure 2, the functional configuration of the training data generation device 10 will be described.

[0025] Figure 2 is a block diagram showing an example of the functional configuration of the learning data generation device 10 according to the first embodiment.

[0026] As shown in Figure 2, the learning data generation device 10 has the following functional configuration: a first acquisition unit 101A, a first sorting unit 102A, a first processing unit 103A, a second acquisition unit 101B, a second sorting unit 102B, a second processing unit 103B, and a generation unit 104. Each functional configuration is realized when the CPU 11 reads a learning data generation program stored in the ROM 12 or storage 14, expands it into the RAM 13, and executes it.

[0027] The first acquisition unit 101A acquires a first image group containing multiple first images. The first image may be an image that has been captured or an image that has been generated. The first image is an image obtained by capturing or generating an object that is the target of object detection or the like.

[0028] The first selection unit 102A selects a first image to be used as training data from the first image group acquired by the first acquisition unit 101A.

[0029] The first processing unit 103A processes the first image selected by the first sorting unit 102A as needed. The method of processing the first image is not particularly limited, but examples include enlarging, reducing, rotating, changing the aspect ratio, and adding blur.

[0030] Meanwhile, the second acquisition unit 101B acquires a group of second images, which includes multiple second images. The second images may be captured images or generated images. The second images are images obtained by capturing or generating regions that can serve as the background of the object targeted in the first image.

[0031] The second selection unit 102B selects a second image to be used as training data from the second image group acquired by the second acquisition unit 101B.

[0032] The second processing unit 103B processes the second image selected by the second sorting unit 102B as needed. The method of processing the second image is not particularly limited, but as mentioned above, examples include enlarging, reducing, rotating, changing the aspect ratio, and adding blur.

[0033] The generation unit 104 generates training data by superimposing each of the multiple first images onto each of the multiple second images. The first and second images superimposed as training data may both be processed, one of them may be processed, or neither may be processed. The generation unit 104 stores the generated training data in the training data DB (Data Base) 141. The training data DB 141 is stored, for example, in the storage 14.

[0034] The generation unit 104 may also further add annotation information, which is information representing the object corresponding to the first image, to the first image in the training data.

[0035] Next, with reference to Figures 3A, 3B, 4, 5, and 6, we will specifically explain the training data generation process when a license plate is applied as an example of an object.

[0036] Figure 3A shows an example of the first image 21 according to this embodiment. In the example in Figure 3A, the first image 21 is shown with a license plate as the object.

[0037] As shown in Figure 3A, the first image 21 of the license plate may be automatically generated. The license plate image is an example, and in reality, other numbers, letters, background colors, letter colors, etc., will be mixed together. In the case of license plates, for example, the font, color scheme, and the placement of letters and numbers are standardized and fixed, so it is possible to generate them automatically. For example, it may be generated using software that automatically generates license plates (see, for example, https: / / carnumberplategenerator.firebaseapp.com / # / ).

[0038] Figure 3B shows an example of the numbers available in the first image 21.

[0039] As shown in Figure 3B, the data can be prepared on an individual digit basis rather than for the entire license plate. In this case, since the position of each digit on the license plate is fixed, the coordinates within the license plate can be automatically generated.

[0040] Figure 4 shows an example of training data images according to this embodiment. In the example in Figure 4, the second image 22 is shown as an image obtained from a drive recorder, in-vehicle camera, etc., which is highly likely to capture images of the vehicle and license plate.

[0041] The training data 20A and 20B shown in Figure 4 are each generated by superimposing a first image 21, which is an image of a license plate, and a second image 22, which is an image of a region that could be the background of the first image 21 (for example, an image from a dashcam).

[0042] Figure 5 shows an example of the processed first image 21.

[0043] The first image 21 shown in Figure 5 is an image that has been processed in such ways as randomly distorting the characters, changing the size or aspect ratio of the characters, blurring the characters, or rotating the characters. The first image 21 that has been processed in such a way may be superimposed on the second image 22.

[0044] Figure 6 shows an example of training data 20A and 20B with annotation information 23 attached.

[0045] Each of the training data sets 20A and 20B shown in Figure 6 is assigned annotation information 23. The annotation information 23 represents an object (for example, a license plate) corresponding to the first image 21. In this case, when the first image 21 and the second image 22 are superimposed, the coordinates of the superimposed position become the coordinates of the annotation, that is, the position of the rectangle surrounding the license plate. In other words, instead of extracting objects from images to generate training data, training data is generated in a pseudo-way by superimposing separately prepared images.

[0046] Next, with reference to Figures 7A, 7B, 8, and 9, we will specifically explain the data generation process for training when cooking is applied as an example of an object.

[0047] Figures 7A and 7B show another example of the first image 21 according to this embodiment.

[0048] As shown in Figure 7A, the system may be configured to automatically collect the first image 21 of the dish. The dish image is just an example; for example, a crawling tool can be used to collect images of a specific dish (e.g., beef with green peppers).

[0049] Note that image 24, shown in Figure 7B, includes dishes other than the specific dish (e.g., beef with green peppers). Therefore, it is desirable to delete image 24. It is also desirable to delete images of dishes other than the specific dish (e.g., beef with green peppers).

[0050] Figure 8 shows another example of training data images according to this embodiment. In the example in Figure 8, the second image 22 is shown as an image obtained by photographing a table, tray, etc., on which food is likely to be placed.

[0051] The training data 20 shown in Figure 8 is generated by superimposing a first image 21, which is an image of a specific dish (for example, beef with green peppers), and a second image 22, which is an image of a region that could serve as the background for the first image 21 (for example, an image of a tray).

[0052] Furthermore, annotation information 23 is added to the training data 20 shown in Figure 8. The annotation information 23 represents the object corresponding to the first image 21 (for example, beef with green peppers).

[0053] Next, with reference to Figure 9, the operation of the learning data generation device 10 according to the first embodiment will be described.

[0054] Figure 9 is a flowchart showing an example of the processing flow by the learning data generation program according to the first embodiment. Processing by the learning data generation program is achieved when the CPU 11 of the learning data generation device 10 writes the learning data generation program stored in the ROM 12 or storage 14 to the RAM 13 and executes it.

[0055] In step S101 of Figure 9, the CPU 11 acquires a group of first images, which are multiple first images containing the object to be detected, as shown in Figures 3A and 7A above as an example.

[0056] In step S102, the CPU 11 selects a first image to be used as training data from the first image group acquired in step S101, as shown in Figure 5 above as an example, and further processes the selected first image as necessary.

[0057] In step S103, the CPU 11 acquires a group of second images, which are multiple second images containing regions that could serve as the background for the object being targeted in the first image.

[0058] In step S104, the CPU 11 selects a second image to be used as training data from the second image group acquired in step S103, and further processes the selected second image as necessary.

[0059] In step S105, the CPU 11 generates training data by superimposing the first image obtained in step S102 and the second image obtained in step S104, as shown in Figures 4 and 8 above, for example.

[0060] In step S106, the CPU 11 adds annotation information to the training data generated in step S105, as shown in Figures 6 and 8 above, as an example.

[0061] In step S107, the CPU 11 stores the training data to which annotation information was added in step S106 into the training data DB 141, and terminates the series of processes performed by this training data generation program.

[0062] Thus, according to this embodiment, instead of extracting objects from an image to generate training data, training data is generated virtually by superimposing separately prepared images. This reduces the time, cost, and workload associated with generating training data.

[0063] Depending on the object in the first image, it is possible to duplicate the first image or overlay images with different content, assuming that a single second image contains multiple identical objects from the first image. Similarly, different images within the first image group may be processed with different content and overlaid simultaneously onto a single second image.

[0064] Furthermore, when using multiple images from the first image group, they may be superimposed so that the first images overlap each other. However, the overlap ratio should be kept low.

[0065] Furthermore, the upper limit of the overlap ratio and the method of overlapping may be varied depending on the object. For example, for traffic lights, the upper limit could be 5% of the area ratio, and for vehicles, it could be 30%, and the overlap could be limited to the horizontal direction.

[0066] For example, traffic lights and road signs may have multiple lights with the same color scheme (such as red lights) placed in close proximity. Also, depending on the arrangement, multiple objects may appear to be captured simultaneously or superimposed on each other in a dashcam recording. The above processing method is designed to handle such cases.

[0067] [Second Embodiment] This section describes a method in which a trained model is generated by performing machine learning using the training data generated in the first embodiment described above, and an object in an image is detected using the generated trained model.

[0068] Figure 10 is a block diagram showing an example of the functional configuration of the learning data generation device 10A according to the second embodiment. In this embodiment, the learning data generation process, learning process, and estimation process are shown to be implemented in a single device. However, these learning data generation process, learning process, and estimation process may be implemented in separate devices, or the learning data generation process, learning process, and estimation process may be implemented in separate devices.

[0069] As shown in Figure 10, the learning data generation device 10A has a functional configuration that includes a first acquisition unit 101A, a first sorting unit 102A, a first processing unit 103A, a second acquisition unit 101B, a second sorting unit 102B, a second processing unit 103B, a generation unit 104, as well as a learning unit 105, a receiving unit 106, a processing unit 107, and an estimation unit 108. The same reference numerals are used for components that are the same as those of the learning data generation device 10 described in the first embodiment above, and repeated explanations are omitted.

[0070] The storage 14 stores a training data DB 141 containing the training data generated by the generation unit 104 described in the first embodiment above.

[0071] The learning unit 105 uses the training data stored in the training data DB 141 to perform machine learning, thereby generating a trained model 142 that takes an image with an unknown object as input and outputs an estimated result for object detection. Here, the training model used for machine learning is, for example, a general object detection algorithm, such as a Convolutional Neural Network (CNN) or YOLO (You Only Look Once). The trained model 142 generated by the learning unit 105 is stored in, for example, the storage 14.

[0072] The reception unit 106 receives input of the estimated target image, which is the image to be used for object detection.

[0073] The processing unit 107 processes the estimated target image received by the receiving unit 106 as needed. The method of processing the estimated target image is not particularly limited, but as mentioned above, examples include enlarging, reducing, rotating, changing the aspect ratio, and adding blur.

[0074] The estimation unit 108 uses the trained model 142 to estimate the objects depicted in the target image obtained by processing the processing unit 107.

[0075] Figure 11 shows an example of the target images 30A and 30B and the estimation results 31A and 31B according to this embodiment.

[0076] As shown in Figure 11, the target image 30A shows "karaage" (Japanese fried chicken), which is one example of an object, and "karaage" is considered an unknown object. In the estimation result 31A obtained by the estimation unit 108, the object shown in the target image 30A is detected as "karaage". On the other hand, the target image 30B shows "gomoku ramen" (a type of mixed ramen), which is one example of an object, and "gomoku ramen" is considered an unknown object. In the estimation result 31B obtained by the estimation unit 108, the object shown in the target image 30B is detected as "gomoku ramen".

[0077] In the example shown in Figure 7B above, where the target of detection is the dishes, it was stated that it is desirable to delete images that include other dishes or are different from the original images. However, it is also possible to generate training data by performing region extraction processing on the images.

[0078] In other words, the first acquisition unit 101A may, for example, extract an image region representing each of several different objects from an image captured of multiple different objects, as shown in Figure 12, and acquire the extracted image region as the first image.

[0079] Figure 12 illustrates the region extraction process for extracting a first image from an image 25 containing images of multiple different objects.

[0080] Image 25, shown in Figure 12, is an image capturing multiple different objects. Various shape detection processes, such as rectangle detection, trapezoid detection, and ellipse detection, are performed on Image 25 to detect the plate region, which represents a plate containing food, an example of an object. From Image 25, plate regions R1 to R4 are detected. Then, rectangular regions that are circumscribing or inscribed around each of the detected plate regions R1 to R4 are extracted as images, and these extracted images are used as the first images of the training data.

[0081] Figure 13 is a diagram illustrating the region extraction process for the target image.

[0082] For the target image shown in Figure 13, dish region detection processing and image extraction processing may be performed in the same manner as in the example of image 25 shown in Figure 12 above, to carefully select the range to be estimated.

[0083] Figure 14 shows yet another example of the first image 21 according to this embodiment.

[0084] As shown in Figure 14, a portion of the first image 21 may be used as the annotation coordinates. In this case, the second image 22 does not need to be used. In the example in Figure 14, the region R21 obtained by cutting 5% from the top, bottom, left, and right of the entire image is used as the annotation coordinates, and annotation information 23 is added.

[0085] The training data generated by the method described in the above embodiment consists of one dish per image. However, known object detection methods often include a mechanism to improve learning accuracy by combining multiple training data to augment the data. Therefore, it is not a problem if multiple dishes (for example, rice and a hamburger) are shown in a single image of the target image.

[0086] Furthermore, taking into account cases where the object detection method does not support the augmentation described above, as an example, as shown in Figure 15, possible combinations from multiple dishes (for example, the combination of beef with green peppers and rice) may be created and the multiple images may be superimposed, or corrections may be made to the arrangement of the plates and the scaling ratio between dishes.

[0087] In the above embodiment, the training data generation process, which is executed by the CPU after it reads the program, may be executed by various processors other than the CPU. Examples of such processors include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits that are processors with circuit configurations specifically designed to execute specific processes, such as ASICs (Application Specific Integrated Circuits). Furthermore, the training data generation process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0088] Furthermore, although the above embodiment describes a configuration in which the learning data generation program is pre-stored (also referred to as "installed") in ROM or storage, the invention is not limited to this. The learning data generation program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the learning data generation program may be provided in a form that is downloaded from an external device via a network.

[0089] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0090] The following additional information is disclosed regarding the embodiments described above.

[0091] (Additional note 1) Memory and At least one processor connected to the memory, Includes, The aforementioned processor, Acquire multiple first images containing the object, Multiple second images are obtained that include regions that can serve as the background for the aforementioned object. Training data is generated by superimposing each of the plurality of first images and each of the plurality of second images. A training data generation device configured as follows.

[0092] (Additional note 2) A non-temporary storage medium that stores a program executable by a computer to perform a training data generation process, The aforementioned training data generation process is as follows: Acquire multiple first images containing the object, Multiple second images are obtained that include regions that can serve as the background for the aforementioned object. Training data is generated by superimposing each of the plurality of first images and each of the plurality of second images. Non-transitory storage medium. [Explanation of symbols]

[0093] 10, 10A Training Data Generation Device 11 CPU 12 ROM 13 RAM 14 Storage 15 Input section 16 Display section 17 Communication I / F 18 bus 101A 1st acquisition part 102A First Sorting Department 103A 1st processing section 101B 2nd acquisition part 102B Second Selection Department 103B 2nd processing section 104 Generation part 105 Learning Department 106 Reception Department 107 Processing Department 108 Estimation part 141 Training Data Database 142 pre-trained models

Claims

1. A first acquisition unit that acquires multiple first images including an object, A second acquisition unit acquires a plurality of second images including regions that may serve as the background of the aforementioned object, A generation unit that generates training data by superimposing each of the plurality of first images and each of the plurality of second images, Equipped with, The generation unit adds annotation information, which is information representing the object corresponding to the first image, to the first image in the training data. A single second image contains multiple identical objects from the first image. A device for generating training data.

2. A first processing unit for processing the first image, A second processing unit for processing the second image, Furthermore, The generation unit generates training data by superimposing the first image processed by the first processing unit and the second image processed by the second processing unit. The learning data generation device according to claim 1.

3. The first acquisition unit extracts image regions representing each of several different objects from an image of the objects captured, and acquires the extracted image regions as the first image. The learning data generation device according to claim 1.

4. The training data generation device, Acquire multiple first images containing an object, Multiple second images are obtained that include a region that can serve as the background of the aforementioned object. Training data is generated by superimposing each of the plurality of first images and each of the plurality of second images. Annotation information, which is information representing the object corresponding to the first image, is added to the first image in the training data. A single second image contains multiple identical objects from the first image. Method for generating training data.

5. Acquire multiple first images containing an object, Multiple second images are obtained that include a region that can serve as the background of the aforementioned object. Training data is generated by superimposing each of the plurality of first images and each of the plurality of second images. Annotation information, which is information representing the object corresponding to the first image, is added to the first image in the training data. A single second image contains multiple identical objects from the first image. A program for generating training data to be executed by a computer.

Images