Training data set generation device and training data set generation method

Abstract

To provide a technology for generating a learning data set contributing to improvement of performance of image recognition of an artificial intelligence using machine learning while suppressing costs.SOLUTION: A learning data set generation device creates a learning data set by using an image containing a whole of an object of a detection object in an object detection area, from among image data sets containing a plurality of images having a plurality of objects to be detected, and a plurality of images having the plurality of object detection areas for detecting the plurality of objects to be detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for generating a learning dataset for machine learning used in image recognition. [Background technology]

[0002] Image recognition, such as image classification and object detection, using machine learning such as deep learning requires a large number of images, ranging from thousands to millions. These images must then be labeled to identify the object type, object detection area, etc. The number of images and the accuracy of labeling have a significant impact on the performance of image recognition.

[0003] Hereinafter, artificial intelligence trained through machine learning may be referred to simply as "machine learning." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-164255 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 discloses an object detection device that performs image recognition using machine learning. When recognizing objects using images, the objects to be recognized are labeled. Here, the labeling process can also be automated using conventional machine learning. However, when the labeling process is automated using conventional machine learning, there is a high possibility that deviations in the object detection region will be larger than when the labeling process is performed manually. Therefore, when machine learning is performed on an object recognition artificial intelligence using images labeled using conventional machine learning, there is a risk that the image recognition performance will be reduced. On the other hand, manually labeling a large number of images would be extremely costly.

[0006] In view of the above-mentioned problems, the present invention aims to provide a technology for generating a training dataset that contributes to improving the image recognition performance of artificial intelligence using machine learning while suppressing costs. [Means for solving the problem]

[0007] The training dataset generation device according to the present invention is configured (first configuration) to create a training dataset using an image from an image dataset including a plurality of images each having a plurality of detection target objects and a plurality of object detection regions for detecting the plurality of detection target objects, in which the entire object to be detected is contained within the object detection region.

[0008] In the training dataset generation device of the first configuration described above, the plurality of detection target objects included in the plurality of images may be configured to be identified in advance by object type (second configuration).

[0009] The training dataset generation device of the second configuration may also be configured (third configuration) to include: a first creation unit that creates object detection regions for images that constitute the image dataset based on the classification results for each type of object in the image dataset; a labeling unit that labels a portion of the intermediate dataset constituted by the object detection regions as to whether or not the entire object is included; a learning unit that performs machine learning using the data labeled by the labeling unit as training data to generate a classifier; and a second creation unit that uses the classifier to create a training dataset by excluding images from the image dataset that have the object detection region that does not include the entire object.

[0010] In the learning dataset generation device of the third configuration, the labeling unit may be configured to perform labeling based on a user operation (fourth configuration).

[0011] In the training dataset generation device of the third or fourth configuration, the labeling unit may label a portion of the intermediate dataset constituted by the object detection region as to whether the entire object is included at an appropriate size, and the second creation unit may use the classifier to create the training dataset by excluding, from the image dataset, images having the object detection region that does not include the entire object at an appropriate size (fifth configuration).

[0012] In the training dataset generation device of any of the third to fifth configurations above, the second creation unit may be configured to use the classifier to extract from the image dataset the object detection region that does not include the entire object, correct the object detection region that does not include the entire object, and, after the correction, create a training dataset by excluding from the image dataset images that have the object detection region that does not include the entire object (sixth configuration).

[0013] The training dataset generation method according to the present invention is configured (seventh configuration) to create a training dataset using an image from an image dataset that includes a plurality of images having an object to be detected and an object detection region for detecting the object to be detected, the image including the entire object to be detected within the object detection region. [Effects of the Invention]

[0014] According to the present invention, it is possible to generate a training dataset that contributes to improving the image recognition performance of artificial intelligence using machine learning while keeping costs down. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing an example of a schematic configuration of an information processing apparatus according to an embodiment; [Figure 2] Flowchart showing an example of operation of the information processing device according to the embodiment [Figure 3] A diagram showing the first example of an image [Figure 4]A diagram showing a second example of an image [Figure 5] A diagram showing a third example of an image [Figure 6] A diagram showing a fourth example of an image [Figure 7] An example of an intermediate dataset DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0017] <1. Configuration of information processing device> FIG. 1 is a diagram illustrating an example of a schematic configuration of an information processing device according to an embodiment. The information processing device 1 is an example of a training dataset generation device. The information processing device 1 may be an information processing device installed in a single location, or may be a distributed information processing device in which components are installed in multiple locations. The information processing device 1 executes the training dataset generation method according to this embodiment.

[0018] The information processing device 1 includes a control unit 11 and a storage unit 12.

[0019] The control unit 11 is a computer including at least one processor. Specifically, the control unit 11 is a computer including a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory), which are not shown. The control unit 11 processes, transmits, and receives information based on programs stored in the storage unit 12, and controls the entire information processing device 1.

[0020] The control unit 11 includes a first creation unit 11a, a labeling unit 11b, and a learning unit 11c. The CPU executes calculations in accordance with the programs stored in the storage unit 12, thereby realizing various functions of the control unit 11, such as the first creation unit 11a.

[0021] The first creation unit 11a acquires an image dataset to which information for each type of object included in an image has been added in advance. The information for each type of object included in an image added to the image dataset has been added in advance by a first classifier.

[0022] The first classifier is provided separately from the information processing device 1. The first classifier classifies multiple types of objects included in an image for each type of object, and assigns information on the classification result to the image. For example, if the image includes a vehicle, the first classifier identifies the vehicle included in the image and assigns information indicating that the object included in the image is a vehicle to the image.

[0023] The first classifier is generated in advance by machine learning, and the data used for the machine learning is, for example, multi-class data such as Microsoft (registered trademark) COCO.

[0024] It is also possible that an image data set containing information for each type of object included in an image is available. In such a case, the first creation unit 11a may acquire such an image data set. Also, the first classifier may be omitted.

[0025] The first creation unit 11a creates an object detection region for an image constituting the image dataset based on information on the classification result for the image dataset. Specifically, the first creation unit 11a creates an object detection region for an object to be detected that is included in the image.

[0026] The first creating unit 11a has machine learning using, for example, R-CNN (Regions with Convolutional Neural Networks). The machine learning using R-CNN that the first creating unit 11a has creates an object detection region for a detection target object included in an image that constitutes an image dataset.

[0027] Furthermore, by using the first classifier as a classifier that classifies multiple types of objects by type, the accuracy of creating the object detection area in the first creation unit 11a can be improved compared to when the first classifier is a classifier that cannot classify multiple types of objects by type.

[0028] However, the object detection region generated by first creation unit 11a may be generated in a part of the image that is different from the object to be detected, or in a position that is different from the object to be detected. Hereinafter, an object detection region generated in this manner may be referred to as an "incorrectly generated object detection region." Furthermore, when an object detection region is generated in the image so as to match the object to be detected, that object detection region may be referred to as a "correctly generated object detection region."

[0029] Possible reasons for the occurrence of an erroneously generated object detection region include, for example, when a part of the object to be detected is hidden or when the image contains a background with similar features to the object to be detected. Performing machine learning using an image with an erroneously generated object detection region may impair the functionality of the artificial intelligence after machine learning. Therefore, the information processing device 1 uses a second classifier, which will be described later, to exclude images with an erroneously generated object detection region from the training dataset.

[0030] The training dataset is generated using the labeling unit 11b and a second classifier provided in the second creation unit 11d. The second classifier corresponds to an example of a classifier recited in the claims. The labeling unit 11b allows a user (person) to select images having a correctly generated object detection region from "some" of the images included in the image dataset. The learning unit 11c generates a second classifier for each object to be detected by machine learning using the selected images. The second classifier extracts images having a correctly generated object detection region from "all" of the images in the image dataset. A training dataset is generated based on the extracted images. This will be explained in detail below.

[0031] The labeling unit 11b labels a portion of the intermediate data set made up of the object detection region to indicate whether or not the entire object of the object to be detected is included. In this embodiment, this is performed based on the selection of a labeling user (human). The portion of the intermediate data set is, for example, less than 10% of the number of intermediate data sets, or approximately 1000 intermediate data sets. Since only a portion of the intermediate data set is selected by a human, the amount of human labor can be reduced compared to when the entire intermediate data set is selected by a human. Details of labeling will be described later.

[0032] The learning unit 11c performs machine learning using the data labeled by the labeling unit 11b as training data to generate a second classifier. In this embodiment, the learning unit 11c generates a second classifier for each type of object to be detected. The object to be detected is, for example, a vehicle or a person.

[0033] The second classifier classifies, for all images in the intermediate dataset, whether or not the image has a correctly generated object detection region. The second classifier includes, for example, a support vector machine (SVM). The SVM included in the second classifier classifies whether or not the image has a correctly generated object detection region.

[0034] The second creation unit 11d uses the second classifier to extract images having an object detection region that includes the entire object to be detected from all images in the image dataset, and creates a training dataset using the extracted images.

[0035] <2. Operation of the information processing device> FIG. 2 is a flowchart showing an example of the operation of the information processing device 1. When the information processing device 1 becomes operable and an image dataset is input, the information processing device 1 starts the operation of the flowchart shown in FIG. 2. The input form of the image dataset is not particularly limited. The information processing device 1 may receive the image dataset via wireless communication, may receive the image dataset via wired communication, or may input the image dataset to the information processing device 1 via a storage medium that is detachable from the information processing device 1. Here, for ease of explanation, it is assumed that an image dataset consisting of four images P1 to P4 shown in FIGS. 3 to 6 is input to the information processing device 1. However, an actual image dataset is composed of thousands to millions of images.

[0036] First, the first creation unit 11a creates object detection regions for images constituting the image dataset based on labeling of the image dataset using the first classifier generated by machine learning (step S10). Specifically, the first creation unit 11a recognizes vehicles 101 and 102 shown in image P1 shown in Fig. 3, and creates bounding boxes BB1 and BB2, which are object detection regions corresponding to vehicle 101 and vehicle 102, respectively.

[0037] The first creation unit 11a recognizes one vehicle 103 and one cone 201 that appear in image P2 shown in Fig. 4, and creates a bounding box BB3 that is an object detection area corresponding to the vehicle 103, and a bounding box BB4 that is an object detection area corresponding to the cone 201. The first creation unit 11a recognizes one vehicle 104 that appears in image P3 shown in Fig. 5, and creates a bounding box BB5 that is an object detection area corresponding to the vehicle 104. The first creation unit 11a also recognizes a cone 202 that appears in image P4 shown in Fig. 6, and creates a bounding box BB6 that is an object detection area corresponding to the cone 202.

[0038] See FIG. 2. Next, labeling unit 11b labels a part of the intermediate data set made up of object detection regions, indicating whether or not the whole object is included (step S20). Specifically, labeling unit 11b labels five object detection regions that are part of the intermediate data set made up of six object detection regions shown in FIG. 7, indicating whether or not the whole object is included. Labeling unit 11b labels object detection regions that include the whole object with "YES." Labeling unit 11b labels object detection regions that do not include the whole object with "NO."

[0039] By labeling a portion of the intermediate dataset consisting of object detection regions, rather than the entirety of the intermediate dataset, as to whether or not the entire object is included, the cost of the labeling process and, consequently, the cost of creating a training dataset can be reduced. To sufficiently reduce costs, it is desirable that the number of data items in the "portion of the intermediate dataset" be, for example, 10% or less of the number of data items in the "entire intermediate dataset." For example, if the image dataset includes 10,000 images and the total number of data items in the intermediate dataset is 10,000, the number of data items in the portion of the intermediate dataset is 1,000 or less.

[0040] In this embodiment, labeling unit 11b performs labeling based on user operations. Therefore, a user interface (not shown) for inputting the contents of user operations, such as a keyboard, pointing device, or microphone, is connected to information processing device 1. A user visually checks the object detection area to determine whether the entire object is included in the object detection area. Therefore, a display device (not shown) for displaying the object detection area is connected to information processing device 1. When labeling unit 11b is configured to perform labeling based on user operations, labeling unit 11b can be simplified and its costs can be reduced.

[0041] Note that labeling unit 11b may label "YES" or "NO" based on a method other than user operation. Any method may be used as long as it can determine whether the entire object is included in the object detection area created by first creation unit 11a. For example, after performing object edge detection processing on an image, labeling may be performed with "YES" if the edge of the detected object is included within the object detection area, and labeling may be performed with "NO" in other cases. Furthermore, after estimating the type of object included in the image, labeling may be performed with "YES" if an external model corresponding to the estimated object type is included within the object detection area, and labeling may be performed with "NO" in other cases.

[0042] In step S30 following step S20, the learning unit 11c performs machine learning using the data labeled with "YES" or "NO" by the labeling unit 11b as training data to generate a second classifier. In this embodiment, the learning unit 11c generates a second classifier for each type of object. That is, the learning unit 11c generates a second classifier for vehicles and a second classifier for cones. By generating a second classifier for each type of object in this way, improvement in the machine learning performance of the learning unit 11c can be expected.

[0043] However, the learning unit 11c may generate a single second classifier regardless of the type of object, and may generate a second classifier for vehicles and a second classifier for animals that detect dogs and cats when there are three types of objects, for example, vehicles, dogs, and cats. In other words, the number of second classifiers generated by the learning unit 11c may be less than the number of types of objects.

[0044] In step S40 following step S30, second creation unit 11d uses the second classifier to extract images having correctly generated object detection regions from the image dataset. Second creation unit 11d creates a training dataset using the extracted images. In other words, second creation unit 11d uses the second classifier to create the training dataset by excluding images having incorrectly generated object detection regions from the image dataset. Specifically, second creation unit 11d creates the training dataset by excluding images P2 and P4 from the image dataset made up of images P1 to P4. The training dataset is made up of images P1 and P3 having object detection regions that include the entire object to be detected. In other words, the images included in the training dataset have correctly generated object detection regions.

[0045] In this way, the training dataset created by the information processing device 1 is composed of images having correctly generated object detection areas, and therefore becomes a training dataset that contributes to improving the image recognition performance of artificial intelligence using machine learning.

[0046] <3. Variation 1> The above-described embodiments should be considered to be illustrative in all respects and not restrictive. The technical scope of the present invention is indicated by the claims, not by the description of the above-described embodiments, and should be understood to include all modifications that fall within the meaning and scope of the claims.

[0047] For example, labeling unit 11b may label a portion of the intermediate dataset formed by the object detection region, indicating whether the entire object is included at an appropriate size. In this case, labeling unit 11b may perform labeling using three labels: a "label indicating that only a portion of the object is included," a "label indicating that the entire object is included at an inappropriate size," and a "label indicating that the entire object is included at an appropriate size," or may perform labeling using two labels: a "label indicating that the entire object is not included at an appropriate size" and a "label indicating that the entire object is included at an appropriate size."

[0048] The "appropriate size" may be a size determined by a certain standard. For example, if a user visually determines that the area occupied by the object is equal to or greater than half the area of the object detection region, it may be considered that the "entire object is included at an appropriate size."

[0049] Then, the second creation unit 11d uses the second classifier to create a training dataset by excluding from the image dataset images having object detection regions that do not include the entire object in an appropriate size, thereby obtaining a training dataset that contributes to further improving the performance of image recognition using machine learning.

[0050] Furthermore, for example, second creation unit 11d may use the second classifier to extract, from the image dataset, object detection regions that do not include the entire object, correct the object detection regions that do not include the entire object, and, after the correction, create a training dataset by excluding from the image dataset images that have object detection regions that do not include the entire object. This increases the number of images that make up the training dataset, thereby obtaining a training dataset that contributes to further improving the performance of image recognition using machine learning.

[0051] As a correction method, for example, the center point of the object detection region may be fixed and the area of the object detection region may be enlarged by a certain factor. The correction may be performed only once, or may be repeated multiple times. When the correction is repeated multiple times, for example, an upper limit may be set on the number of corrections to prevent the corrections from continuing endlessly. Also, for example, if there is no change in the number of images constituting the training dataset before and after the second correction, the correction may be considered to have failed and no further corrections may be performed.

[0052] <4. Variation 2> In the above-described embodiment, the information processing device 1 is an information processing device installed in a single location, or a distributed information processing device in which components are installed in multiple locations. However, the present invention is not limited to this. The information processing device 1 may be a cloud server configured on a cloud. [Explanation of symbols]

[0053] 1. Information processing equipment 11 Control section 11a First Creation Section 11b Labeling section 11c Learning Department 11d Second Creation Section 12 Storage section

Claims

1. Creating an object detection region for images constituting an image dataset including a plurality of images having a plurality of objects to be detected; labeling a portion of the intermediate data set formed by the object detection region as including or not including the entire object; generating a classifier by performing machine learning on the labeled data as training data; creating a training dataset by using the classifier to exclude from the image dataset images having the object detection region that does not include the entire object; A training dataset generator.

2. The plurality of detection target objects included in the plurality of images are identified in advance by object type. The training dataset generation device according to claim 1 .

3. A learning dataset generation device as described in claim 1 or 2, which performs the labeling based on user operation.

4. Labeling a portion of the intermediate data set constituted by the object detection region as to whether or not the entire object is included at an appropriate size; The training dataset generation device according to any one of claims 1 to 3, wherein the training dataset is created by using the classifier to exclude from the image dataset images having the object detection region that does not include an entire object of an appropriate size.

5. A training dataset generation device as described in any one of claims 1 to 4, which uses the classifier to extract from the image dataset the object detection region that does not include the entire object, correct the object detection region that does not include the entire object, and after the correction, excludes from the image dataset images that have the object detection region that does not include the entire object, thereby creating a training dataset.

6. Creating an object detection region for images constituting an image dataset including a plurality of images having a plurality of objects to be detected; labeling a portion of the intermediate data set formed by the object detection region as including or not including the entire object; generating a classifier by performing machine learning on the labeled data as training data; creating a training dataset by using the classifier to exclude from the image dataset images having the object detection region that does not include the entire object; How to generate a training dataset.

Images