Learning data generation system, information processing apparatus, information processing method, and program

Abstract

To provide a learning data generation system for generating a data set rich in variation.SOLUTION: One or more sensors that acquire first information that is image information of an object, a plurality of object detection units that detect second information that includes type information of the object and position information of the object, based on the first information, a training data generation unit that generates training data that includes the first information and the second information, based on the second information detected by the object detection units, and a storage apparatus in which the training data is stored, in a case where there is a difference in the second information in the detection results of the plurality of object detection units based on the specific first information, the training data generation unit stores the specific first information and the second information including the type information in the storage device in association with each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system for generating a training dataset for an AI model, and more particularly to a system for generating an image dataset used for training an object recognition model. [Background technology]

[0002] In recent years, image processing techniques using deep learning models such as CNN (Convolutional Neural Network) and ViT (Vision Transformer) have become well known for image classification and specific object detection. Generating highly accurate deep learning models with high generalization performance requires a richly varied dataset. However, if the dataset becomes too large, it can increase management costs, increase the training time of the deep learning model, or even lead to overfitting. Therefore, there is a need to efficiently collect high-quality datasets.

[0003] Patent Document 1 discloses a method for efficiently generating a data set without overlapping features by filtering image data with similar features. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-112819 Summary of the Invention [Problem to be solved by the invention]

[0005] The above-mentioned Patent Document 1 discloses a technology for obtaining image classification results and feature values ​​using a deep learning model, and eliminating redundant image data by clustering the feature values. However, the method of Patent Document 1 is premised on classification using a pre-prepared deep learning model, and therefore has the problem that it is not possible to collect data including objects that cannot be classified by the deep learning model, and it is not possible to generate a dataset with a wide variety. In other words, it is not possible to prepare training data that enables the deep learning model to recognize objects that cannot be classified by the deep learning model.

[0006] Therefore, an object of the present invention is to provide a training data generation system that generates a variety of datasets by taking into account the output results of multiple object detection units and extracting scenes that the object detection units were unable to detect. [Means for solving the problem]

[0007] In order to solve the above problem, a training data generation system according to one embodiment of the present invention includes one or more sensors that acquire first information, which is either image information of an object or distance information to the object, or both; a plurality of object detection units that detect second information based on the first information, which includes at least one of object type information, object position information, and information on the likelihood of object detection; a training data generation unit that generates training data including the first information and the second information based on the second information detected by the object detection units; and a storage device in which the training data is stored; and when there is a difference in the second information in the detection results of the plurality of object detection units based on the specific first information, the training data generation unit associates the specific first information with the second information including the type information and stores them in the storage device. [Effects of the Invention]

[0008] According to the present invention, a wide variety of data sets can be generated. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of functional blocks of a learning data generation system according to a first embodiment; [Figure 2] FIG. 10 is a diagram showing another example of the functional blocks of the learning data generation system according to the first embodiment. [Figure 3] 1 is a flowchart showing the entire learning data generation process according to the first embodiment; [Figure 4] 1 is a flowchart showing details of the learning data generation process according to the first embodiment. [Figure 5] Schematic diagram of a specific shooting scene used to explain the first embodiment (specific example 1) [Figure 6] Schematic diagram explaining background subtraction processing [Figure 7] FIG. 10 is a diagram showing an example of an object detection result in Example 1. [Figure 8] Schematic diagram of a specific shooting scene used to explain the first embodiment (specific example 2) [Figure 9A] FIG. 10 is a diagram showing an example of an object detection result in specific example 2. [Figure 9B] FIG. 9B is a diagram showing an example of a data set generated from the object detection results of FIG. 9A. [Figure 10] Schematic diagram of a specific shooting scene used to explain the first embodiment (specific example 3) [Figure 11A] FIG. 10 is a diagram showing an example of an object detection result in Example 3. [Figure 11B] FIG. 11B is a diagram showing an example of a data set generated from the object detection result of FIG. 11A. [Figure 12] Schematic diagram of a specific shooting scene used to explain the first embodiment (specific example 4) [Figure 13A] FIG. 10 is a diagram showing an example of an object detection result in Example 4. [Figure 13B] FIG. 13B is a diagram showing an example of a data set generated from the object detection result of FIG. 13A. [Figure 14] FIG. 1 is a diagram showing an example of a functional block in an application example of the first embodiment. [Figure 15] FIG. 10 is a diagram showing an example of a display screen for prompting correction of data in an application example of the first embodiment. [Figure 16] FIG. 10 is a diagram showing an example of functional blocks of a learning data generation system according to a second embodiment. [Figure 17] FIG. 10 is a diagram showing an example of an object tracking processing result according to the second embodiment. [Figure 18] FIG. 10 is a diagram showing an example of functional blocks of a learning data generation system according to a third embodiment. [Figure 19] FIG. 10 is a diagram showing an example of functional blocks of an information processing apparatus according to a fourth embodiment; [Figure 20] Overall flowchart of the information processing method according to the fifth embodiment DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same members or elements are designated by the same reference numerals, and duplicate descriptions will be omitted or simplified. <Embodiment 1>

[0011] 1 is a functional block diagram showing an example of the configuration of a training data generation device 100 according to a first embodiment of the present invention. Reference numeral 110 denotes an information processing device that performs image processing including object detection, 120 denotes a sensor for acquiring first information that is either image information of an object or distance information to the object, or both, and 130 denotes a storage device (saving device).

[0012] 1 are realized by executing a computer program stored in a memory (not shown) included in the information processing device 110. However, some or all of these may be realized by hardware. The hardware may be a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like.

[0013] Furthermore, the functional blocks shown in Fig. 1 do not have to be built into the same housing, but may be configured as separate devices connected to each other via signal paths. For example, they may be configured as devices located in remote locations via a network. The above description of Fig. 1 also applies to Figs. 14, 16, 18, and 19, which will be described later.

[0014] The information processing device 110 includes object detection units 111a to 111c and a learning data generation unit 112. The object detection units 111a to 111c detect second information based on first information obtained from the sensor 120. Here, the first information is either image information of the object or information on the distance to the object, or both. The second information is information including at least one of object type information, object position information, and information on the likelihood of object detection.

[0015] Here, the object detection units 111a to 111c may be deep learning models such as CNN (Convolutional Neural Network; hereinafter referred to as CNN) and ViT (Vision Transformer; hereinafter referred to as ViT). Here, "deep learning" is included in "machine learning."

[0016] Furthermore, the object detection units 111a to 111c may use a group of techniques based on differences between frames, such as background subtraction or optical flow, or may use a group of techniques based on pattern matching that compares with predefined object features.

[0017] Alternatively, a group of methods based on clustering of distance information made up of point cloud data may be used. Any other method may be used as long as it is capable of detecting second information including at least one of object type information, object position information, and object detection probability information.

[0018] Furthermore, among the above-mentioned methods and methods based on clustering of distance information made up of point cloud data, a combination of methods based on machine learning, or a combination of methods based on machine learning and other methods may be used. Note that, in this embodiment, the number of object detection units is three, and the respective detection methods are described as CNN, ViT, and background subtraction, but the number of object detection units is not limited to three and may be any plural number, and any combination of detection methods may be used.

[0019] In this embodiment, the position information of an object is the position information of the object in the image coordinate system, which is a two-dimensional coordinate system, but when object detection is performed based on the distance information of the object, the position information of the object may also be the position information of the object in the world coordinate system, which is a three-dimensional coordinate system, for example.

[0020] The object location information represents the coordinates of a rectangular or rectangular parallelepiped subregion surrounding the object in the image, or it may be the result of labeling the object region on a pixel-by-pixel basis, as is typically done in semantic segmentation or instance segmentation.

[0021] The learning data generation unit 112 generates learning data including the first information and the second information based on the second information detected by the object detection units 111a to 111c. In detail, the learning data is data consisting of a pair of first information, which is either image information of an object or distance information to the object, or both, obtained from the sensor 120, and annotation data, which is second information including type information. A specific learning data generation flow will be described in detail later.

[0022] The sensor 120 is a device for acquiring first information, which is either image information or distance information, or both, and may be any imaging device such as a monocular camera, an IR camera (infrared camera), or a stereo camera. A distance measurement device such as a LiDAR (Laser Imaging Detection and Ranging) may also be used. As shown by the sensors 120 and 121 in FIG. 2, not only one sensor but also multiple sensors may be provided, and any combination of connections between one or more sensors and the object detection unit 111 described above may be used.

[0023] The storage 130 is a storage terminal for storing learning data. For example, it is a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage 130 may be built into the information processing device 110 or may be external. Alternatively, a storage system located remotely via a network may be used.

[0024] 3 and 4 are flowcharts showing an example of processing in generating learning data according to the first embodiment, and an example of the operation of the information processing device 110 of this embodiment will be described using Fig. 3 and Fig. 4. Note that the operation of each step in the flowcharts of Fig. 3 and Fig. 4 is performed by a CPU serving as a computer of the information processing device 110 executing a computer program stored in a memory (not shown). Furthermore, in this embodiment, a system for detecting the positions of vehicles and pedestrians on a road will be described as a use case as shown in Fig. 5, but the use case and type of subject may be any type.

[0025] The flow in FIG. 3 is started by a CPU (not shown) of the information processing device 110 when the information processing device 110 and the sensor 120 in FIG. 2 are powered on, for example.

[0026] In step S300, the CPU of the information processing device 110 acquires first information such as image information obtained from the sensor 120, and develops it in its own memory.

[0027] In steps S301, S302, and S303, each of the object detection units 111a, 111b, and 111c performs object detection processing using the first information expanded on the memory of the information processing device 110 as input. In this embodiment, the object detection unit 111a in Fig. 2 uses an object detection method (algorithm A) based on a CNN-based deep learning model.

[0028] The object detection unit 111b uses an object detection method (algorithm B) based on a ViT-based deep learning model (machine learning). The object detection unit 111c uses an object detection method (algorithm C) based on background subtraction (inter-frame difference). In this embodiment, in order to improve the accuracy of the deep learning models used by the object detection units 111a and 111b, it is intended to generate data sets specialized for each of the deep learning models.

[0029] The object detection units 111a and 111b output position information of the detected objects, object type information, and information on the likelihood of object detection (confidence score). The object detection unit 111c performs background subtraction processing (inter-frame subtraction processing) as shown in Fig. 6, and can obtain position information of each object by applying a labeling process to the background image, for example.

[0030] On the other hand, background subtraction processing cannot acquire object type information or object detection probability. In this way, the multiple object detection units detect second information including at least one of object type information, object position information, and object detection probability information based on the first information.

[0031] Here, the object position information is, for example, the upper left coordinates and lower right coordinates of a rectangular partial area surrounding the object in the image. As long as the position of the object in the image can be identified, other information may be, for example, the center coordinates of the rectangular partial area surrounding the object in the image, and the width and height of the partial area. The object type information is data indicating the classification name of the subject, such as car or person. The confidence score is an index indicating the likelihood of object detection (information on the likelihood of object detection), and the higher the score, the higher the likelihood of the object detection result.

[0032] 701, 702, and 703 in Fig. 7 are examples showing the object detection results output by object detection unit 111a, object detection unit 111b, and object detection unit 111c for the traffic scene shown in Fig. 5. Here, object detection unit 111c performs background subtraction processing and cannot detect object types, so the object type column of object detection result 703 is all expressed as "unknown."

[0033] Steps 301, 302, and 303 may be executed sequentially by the CPU of the information processing device 110, or may be executed in parallel to increase speed. When all of the processes in steps S301, S302, and S303 are completed, the process proceeds to step S304.

[0034] A detailed flow of step S304 will be described with reference to Fig. 4. In step S401, the learning data generation unit 112 expands the second information detected by the object detection units 111a, 111b, and 111c based on the acquired first information in an accessible memory area of ​​the information processing device 110.

[0035] In step S402, the second information detected by the object detection units 111a, 111b, and 111c is compared to determine whether there is a difference in the number of detected objects.

[0036] In step S403, the object detection units 111a, 111b, and 111c calculate the difference between the position information (position coordinates) of each object detected, and determine whether the difference is equal to or greater than a threshold. For example, the Euclidean distance between the position coordinates of each object is calculated, and a determination is made as to whether the distance is equal to or greater than a threshold.

[0037] Alternatively, the calculation may be based on the degree of overlap between rectangular partial regions surrounding the object. In this case, a known index such as IoU (Intersection over Union) may be used. In step S403, this process associates the objects output by the object detection units 111a, 111b, and 111c.

[0038] In step S404, the second information detected by the object detection units 111a, 111b, and 111c is compared to determine whether the type information corresponding to each object is different.

[0039] If none of steps S402, S403, or S404 applies, the results of each object detection process are the same, and it can be assumed that existing object detection processes are sufficient for detection. Therefore, since improvement in accuracy through learning using a dataset that includes the scene in question is not expected, the learning data generation unit 112 does not create a dataset and ends step S304.

[0040] If any of steps S402, S403, and S404 applies, the process proceeds to step S405.

[0041] In step S405, the learning data generation unit 112 identifies the object detection units that failed to detect the object based on the second information detected by each of the object detection units 111a, 111b, and 111c. Three methods for identifying the object detection units that failed to detect the object are listed below. The object detection units that failed to detect the object can be identified using a combination of one or more of these methods. Any combination and execution order can be defined depending on the characteristics of the learning data to be generated.

[0042] Identification method 1: When the number of object detection units is three or more, the failed object detection unit is identified by majority vote. For example, if the object detection results of object detection units 111a and 111c are the same and only the object detection result of object detection unit 111b differs, it is determined that object detection unit 111b failed to detect the object.

[0043] Identification method 2: The reliability scores of the second information detected by the object detection units are compared, and the object detection unit with the lower reliability score is deemed to have failed to detect the object.

[0044] Identification method 3: In cases where an object that is not detected by the deep learning model is detected by other methods (e.g., background subtraction), the object can be considered to be an unknown object that has not been learned by the deep learning model, and the object detection unit using the deep learning model is considered to have failed.

[0045] In step S406, the learning data generation unit 112 checks whether or not it has been able to identify the object detection unit that failed in object detection in step S405. If it has been able to identify the failed object detection unit, the process proceeds to step S407, and if it has not been able to identify the failed object detection unit, the process proceeds to step S408. If it has been able to identify the failed object detection unit, the corresponding first information is set as the specified first information.

[0046] In step S407, the learning data generation unit 112 generates annotation data from the second information detected by each object detection unit in steps S301, S302, and S303 for the specific first information, and then associates the specific first information acquired in S300 with the annotation data.

[0047] Here, annotation data is second information including type information, and is in text or binary format. In the case of text format, an existing format may be used. Examples include CSV (Comma Separated Values) and JSON (JavaScript Object Notation). Alternatively, YAML (Yet Another Markup Language) and XML (eXtensible Markup Language) may be used. Alternatively, a proprietary text format may be used.

[0048] Specific first information can be linked (associated) with second information including type information by adding a common character string such as a date or a sequential number to the file name. Examples include "annotation_202311151302_012345.csv" and "image_202311151302_012345.jpg."

[0049] Alternatively, the file name of the first information may be embedded in the annotation data. Alternatively, the annotation data and the first information may be generated as a single binary file. Any other method may be used as long as it is capable of acquiring the association between the annotation data and the first information at a later data utilization stage. This concludes the explanation of step S304 using Figure 4.

[0050] 3, step S305 will be described. The CPU (not shown) of the information processing device 110 stores the learning data, which is a pair of the annotation data and the first information generated in step S304, in the storage 130. At that time, the CPU (not shown) of the information processing device 110 associates the learning data with the object detection unit that failed to detect the object and stores the learning data.

[0051] For example, if the object detection unit A111a fails to detect an object, a directory called "dataset for object detection unit A" may be created and the learning data may be stored in that directory. Alternatively, the storage device for storing the data may be switched for each object detection unit.

[0052] When the process of step S305 is completed, the process returns from step S306 to step S300, and the process is executed again from step S300.

[0053] This concludes the explanation of the flows in Figures 3 and 4. Next, the learning data generation process (step S304) of this embodiment will be explained again using four specific examples.

[0054] Specific example 1 will be described with reference to Fig. 5 and Fig. 7. Fig. 5 shows an image captured in specific example 1, in which there are five cars (car001 to 005) and two people (person001, person002). 701 to 703 in Fig. 7 show the detection results of object detection units A to C, respectively.

[0055] 7 are shown in a table format for convenience of explanation, but in reality they may be expanded as a data structure of a program in a memory (not shown) of the information processing device 110, or may be a file in any format. The above explanation of Fig. 7 also applies to Figs. 9A, 9B, 11A, 11B, 13A, and 13B described below.

[0056] In specific example 1, since there is no difference between the detection results output by each object detection unit by more than a predetermined amount, the flow in FIG. 4 transitions in the order of S401 → S402 (No) → S403 (No) → S404 (No) → S304END.

[0057] Here, in S404, 701, 702, and 703 are different, with "Car" and "unknown", but since object type information cannot be acquired in the background subtraction process of object detection unit C, it can be assumed that the type information of all objects was the same. If there is no difference in the detection results of each object detection unit beyond a predetermined level, as in the scene in specific example 1, learning data is not generated or saved.

[0058] Specific Example 2 will be described with reference to Figures 8, 9A, and 9B. Figure 8 shows an image captured in Specific Example 2, in which there are five cars (car011-015) and two people (person011, person012). The difference from Specific Example 1 is the presence of a car (car012) with a special shape.

[0059] 9A, 901 to 903 respectively show the detection results of object detection units A to C. Focusing on detection result 901 of object detection unit A, an object that should be detected as a "Car" has been erroneously detected as a "Building."

[0060] At this time, the flow in FIG. 4 transitions in the order of S401 → S402 (No) → S403 (No) → S404 (Yes) → S405.

[0061] At this time, in S405 (identification method 2), the learning data generation unit 112 compares the detection results of each of the object detection units A to C. Next, the learning data generation unit 112 detects that the type information of a detected object is different for each of the object detection units, i.e., "Building," "Car," and "unknown."

[0062] As described above, since "unknown" is the output of object detection unit C, which is unable to detect object type information, learning data generation unit 112 ignores it when matching object type information. Therefore, learning data generation unit 112 determines which object detection unit failed: object detection unit A, which output "Building," or object detection unit B, which output "Car."

[0063] The reliability score of object detection unit A is 0.31, and the reliability score of object detection unit B is 0.91. Since the output result of object detection unit B is more likely, the learning data generation unit 112 can consider that the detection by object detection unit B was successful and the detection by object detection unit A was unsuccessful.

[0064] Next, in S406, S407, and S305, the learning data generation unit 112 generates learning data for the object detection unit A that failed to detect the object. The learning data generation unit 112 saves the annotation data 911 and the image data 912 in a folder called “dataset for object detection unit A” 913 in the storage 130.

[0065] At this time, since it is known that there is an error in the object type information in the second line of the detection result 901, which is the basis of the annotation data 911, the learning data generation unit 112 overwrites it with the result of the detection result 902 (overwriting "Building" to "Car"). Alternatively, the learning data generation unit 112 may use the entire detection result 902 as it is, as the annotation data 911.

[0066] An object that is falsely detected by the object detection unit A may be an object with special features that are not included in the learning source data of the object detection unit A. Therefore, data that can be used for future additional learning of the object detection unit A can be efficiently acquired.

[0067] Specific Example 3 will be described with reference to Fig. 10 and Fig. 11A. Fig. 10 shows an image captured in Specific Example 3, in which there are five cars (car021 to 025) and two people (person021, person022). The difference from Specific Example 1 is the presence of a roadside tree 1001, which is an obstruction.

[0068] 11A, 1101 to 1103 respectively indicate the detection results of object detection units A to C. Here, when attention is focused on the number of objects detected in detection result 1102 of object detection unit B, it can be seen that the number of objects is one less than the number of objects detected in the other detection results 1101 and 1103, and one object was not detected.

[0069] At this time, the flow in FIG. 4 transitions in the order of S401 → S402 (Yes) → S405. Next, in S405 (identification method 1), learning data generation unit 112 identifies the object detection units that failed to detect an object. Learning data generation unit 112 determines by majority vote that the number of object detections by object detection units A, B, and C is "7," "6," and "7," respectively, and that object detection unit B failed to detect an object.

[0070] Next, in S406, S407, and S305, the learning data generation unit 112 generates learning data for the object detection unit B that failed to detect the object. The learning data generation unit 112 saves the annotation data 1121 and the image data 1122 in a folder in the storage 130 called "dataset for object detection unit B" 1123 in Fig. 11B.

[0071] Here, the number of object detections by object detection unit B is one less than the other detection results. In other words, because object detection unit B missed a detection, learning data generation unit 112 extracts the missing detection results in 1102 from other detection results, for example, 1101, and generates annotation data 1121. Note that the detection results of object detection units that successfully detected objects, for example, detection result 1101 of object detection unit A, may all be reused as annotation data 1121.

[0072] The reason for the missed detection in object detection unit B may be that the learning source data for object detection unit B does not contain enough scenes in which the target object is partially hidden. Therefore, data that can be used for future additional learning by object detection unit B can be efficiently acquired.

[0073] Specific example 4 will be described with reference to Fig. 12 and Fig. 13A. Fig. 12 shows an image captured in specific example 4, in which there are four cars (car031, car033, car034, car035), two people (person011, person012), and an unknown object (unknown object031).

[0074] The difference from Example 1 is that an unknown object (unknown object031) exists.

[0075] 13A, 1301 to 1303 respectively show the detection results of object detection units A to C. Comparing detection result 1301 of object detection unit A, detection result 1302 of object detection unit B, and detection result 1303 of object detection unit C, it can be seen that object detection unit A and object detection unit B were unable to detect an unknown object (unknown object 031). This is because information about the unknown object is not included in the learning source data for object detection unit A and object detection unit B.

[0076] At this time, the flow in FIG. 4 transitions in the order of S401 → S402 (Yes) → S405. Next, in S405 (identification method 3), the training data generation unit 112 identifies the object detection unit that failed to detect the object. Since there is an object that could only be detected by object detection unit C (background subtraction processing) and not by object detection units A and B (deep learning-based), the training data generation unit 112 determines that the object is an unknown object not included in the training data.

[0077] Then, the learning data generation unit 112 determines that the object detection units A and B have failed in their detection.

[0078] Next, in S406 and S407, the learning data generation unit 112 generates learning data for the object detection units A and B that failed to detect the object. In S305, the learning data generation unit 112 saves the annotation data 1311 and the image data 1312 in a folder called "dataset for object detection unit A" 1313 in the storage 130.

[0079] In addition, in S305, the learning data generation unit 112 saves the annotation data 1321 and the image data 1322 in a folder called “dataset for object detection unit B” 1323 in the storage 130.

[0080] Here, the learning data generation unit 112 extracts information about the unknown object from the detection result 1303 of the object detection unit C in order to include information about the unknown object that could not be detected by the object detection unit A and the object detection unit B in the annotation data 1311, 1321. At this time, since the type of the object is undetectable, the learning data generation unit 112 sets the type of the object to "unknown."

[0081] In this manner, in this embodiment, learning data regarding unknown objects can be efficiently acquired.

[0082] As described above, according to the first embodiment, it is possible to efficiently collect training data containing a variety of variations, for example, unknown objects and known objects with unknown features. Next, an application example of the first embodiment will be shown. <Application example of embodiment 1>

[0083] Fig. 14 shows an application example of the training data generation system 100 of embodiment 1. The configuration and processing flow of the training data generation system 100 in Fig. 14 are the same as those in Fig. 1 shown in the description of embodiment 1.

[0084] The deep learning model generation device 200 is a computer for training a deep learning model based on the training data generated by the training data generation system 100.

[0085] The deep learning model generation device 200 includes a learning data collection unit 201, a learning data cleansing unit 202, a deep learning model learning unit 203, a deep learning model evaluation unit 204, a deep learning model deployment unit 205, and a storage unit 206 for saving the learned model.

[0086] The learning data collection unit 201 acquires the learning data generated by the learning data generation system 100 from the storage 130 and stores it in a memory or storage (not shown) provided in the deep learning model generation device 200.

[0087] The training data cleansing unit 202 corrects inaccurate training data, for example, data that includes unknown objects and has no label name, from among the training data acquired by the training data collection unit 201. For example, a graphical user interface such as that shown in Fig. 15 is provided via a display device (not shown) to prompt the user to correct the data.

[0088] In Figure 15, 1500 is an application window that prompts the user to correct the data. 1501 and 1502 are learning data, which are image data and annotation data, respectively. 1503 is a text box for entering the correct label name.

[0089] Buttons 1504 and 1505 are used to operate the application. For example, if the learning data includes an unknown object, the user enters an appropriate label name, in this example "Flying car," in text box 1503. Then, by pressing the modify button 1504, the learning data cleansing unit 202 updates the annotation data.

[0090] Alternatively, the user can set the learning data not to be used in subsequent learning by pressing a reject button 1505. This graphical user interface is merely an example, and any form of interface may be used as long as it allows correction of learning data.

[0091] The deep learning model learning unit 203 learns the deep learning model based on the learning data collected by the learning data collection unit 201 and corrected by the learning data cleansing unit 202, and updates the weight data of the deep learning model, etc.

[0092] Here, additional learning (transfer learning or fine tuning) may be performed using only the data obtained by the learning data collection unit 201 and the cleansing unit 202, or the learning data may be added to existing learning data for learning.

[0093] It is desirable to classify the learning data into data used for the deep learning learning process and data used for evaluating the learning results, but this is not limited to this. The deep learning model learning unit 203 stores the generated trained model data in the trained model storage 206.

[0094] The deep learning model evaluation unit 204 compares the accuracy or processing speed of an existing trained model with that of a newly trained model. A known index such as mAP (Mean Average Precision) may be used as the index for the accuracy comparison, but a unique index may also be used.

[0095] The deep learning model evaluation unit 204 stores the comparison results in a memory or storage (not shown) of the deep learning model generation device 200.

[0096] If the comparison by the deep learning model evaluation unit 204 shows that the newly trained model is superior in terms of accuracy or speed, the deep learning model deployment unit 205 transmits the newly trained model to the object detection unit 111a or the object detection unit 111b.

[0097] By providing the deep learning model generation device 200 in this manner, the recognition accuracy of the object detection units 111a and 112b and the rate of response to unknown objects can be gradually improved over time. <Embodiment 2>

[0098] A functional block diagram of a system according to the second embodiment is shown in Fig. 16. The only difference from the functional block diagram shown in Fig. 1 is that an object tracking unit 113 has been newly added, and the other configurations are the same as those in Fig. 1.

[0099] The object tracking unit 113 performs tracking processing for each object based on the output results of the object detections 111a, 111b, and 111c or image information or distance information obtained from the sensor 120. The tracking processing is a process of comparing the previous frame with the current frame and assigning the same label (for example, a unique sequential number) to objects determined to be the same object.

[0100] As a tracking algorithm, for example, a known method based on a Kalman filter or deep learning may be used.

[0101] An example of the tracking processing result is shown in Fig. 17. 1701 to 1703 in Fig. 17 are the object detection results to which the tracking processing result (tracking ID string) has been added.

[0102] Adding a tracking process can reduce false positives. For example, focusing on tracking ID=0103 in Fig. 17, "Car" is detected in frames N-2 and N-1, but "Track" is detected in frame N. Therefore, the CPU of the information processing device 110 determines that false positives occurred in frame N, and can correct the detection result 1703.

[0103] Furthermore, in order to reduce overlearning of the deep learning model, it is desirable to avoid duplication of learning data relating to the same object (such as the same vehicle or person) as much as possible. For example, if the learning data generation unit 112 checks the tracking ID and a certain amount of learning data relating to the tracking ID has already been created, a process may be added to prevent new learning data from being created (this example is not shown).

[0104] In this way, by adding the object tracking unit 113 to the training data generation system 100, it is possible to further reduce the number of inappropriate training data due to false detections and to reduce over-training by suppressing the collection of similar training data. <Embodiment 3>

[0105] Fig. 18 shows functional blocks of a system according to the third embodiment. The difference from the functional block shown in Fig. 1 is that the sensor 120 and the object detection units 111a to 111c are not directly connected to each other, but are connected via the sensor data storage 140.

[0106] The following description will focus on the differences from the first and second embodiments. In the third embodiment, an acquisition unit 114, a sensor data storage 140, and a communication unit 115 are newly provided to the system described in the first embodiment.

[0107] The acquisition unit 114 may receive image information or distance information directly from the sensor 120, or may acquire image information or distance information from a sensor data storage 140 in which pre-captured image information or distance information is stored.

[0108] Here, the sensor data storage 140 is a storage device such as an HDD, SSD, or flash memory, and may be built into the information processing device. Alternatively, the sensor data storage 140 may be image information or distance information received from an external terminal (not shown) via a communication unit 115 that performs network communication with the external terminal, for example, via an interface such as a WebAPI.

[0109] Any communication interface and protocol may be used. When the acquisition unit 114 acquires image information or distance information through any system, the acquisition unit 114 expands the data into a memory (not shown) of the information processing device 110.

[0110] To reduce the processing load, the acquisition unit 114 may thin out the image information or distance information.

[0111] The object detection units 111a, 111b, and 111c perform object detection processing on the image information or distance information expanded in a memory (not shown) of the information processing device 110 by the acquisition unit 114.

[0112] In the configuration of embodiment 1 or embodiment 2, the sensor and object detection unit are directly connected, so object detection and learning data generation processing must be completed before new frame image information and distance information arrive from the sensor. For example, if the sensor operates at 30 fps, the total time for object detection and learning data generation processing must be within approximately 33 ms.

[0113] To achieve this, the information processing device 110 is required to have a high-performance CPU, GPU, and memory (not shown), which increases costs.

[0114] As shown in the third embodiment, an acquisition unit 114 is provided, and by acquiring image information or distance information from the sensor data storage 140, for example, or by thinning out the data, learning data can be generated without using a high-performance information processing device 110. <Embodiment 4>

[0115] Fig. 19 shows functional blocks of an information processing device 160 according to the fourth embodiment. Comparing Fig. 19 with the information processing device 110 of the learning data generation system in Fig. 18, the difference is that an output unit 116 that outputs learning data is provided and is connected to storage or the like via a communication unit 115. That is, the information processing device 160 is configured such that the learning data generated from the learning data generation unit 112 can be output to external storage via the output unit 116 and the communication unit 115.

[0116] Therefore, in the information processing device 160 of the fourth embodiment, when there is a difference in the second information among the detection results of a plurality of object detection units based on specific first information, the learning data generation unit 112 performs the following process.

[0117] The learning data generation unit 112 associates the specific first information with the second information including the type information, and outputs the learning data via the output unit 116.

[0118] The configuration of the fourth embodiment provides the same effects as those of the third embodiment. <Embodiment 5>

[0119] The information processing method according to the fifth embodiment is described below. The information processing method according to the fifth embodiment is composed of an acquisition step, a detection step, and a generation step, and is executed by a processor or the like.

[0120] In the acquisition step, first information, which is either image information of an object or distance information to the object, or both, is acquired (S500 in FIG. 20).

[0121] In the detection process, second information including at least one of object type information, object position information, and object detection probability information is detected by a plurality of object detection units based on the first information (S301, S302, S303 in FIG. 20).

[0122] In the generating step, learning data including the first information and the second information is generated based on the second information (S304 in FIG. 20, S401 to S407 in FIG. 4).

[0123] In the output step, the learning data is output (S505 in FIG. 20).

[0124] The configuration of the fifth embodiment provides the same effects as those of the third embodiment. <Embodiment 6>

[0125] The program according to the sixth embodiment is a program for causing an information processing device (a computer that performs control using a processor, etc.) or a system (a system including a computer that performs control using a processor, etc.) to execute the information processing method according to the fifth embodiment.

[0126] The configuration of the sixth embodiment provides the same effects as those of the third embodiment. <Other>

[0127] The disclosure of the above-described embodiment includes the following configurations. (Configuration 1) one or more sensors that acquire first information, which is either image information of an object or distance information to the object, or both; a plurality of object detection units that detect second information including at least one of object type information, object position information, and object detection probability information based on the first information; a learning data generation unit that generates learning data including the first information and the second information based on the second information detected by the object detection unit; a storage device in which the learning data is stored; Equipped with The learning data generation system is characterized in that, when there is a difference in the second information in the detection results of the plurality of object detection units based on the specific first information, the learning data generation unit associates the specific first information with the second information including the type information and stores them in the storage device. (Configuration 2) The learning data generation system according to configuration 1, wherein the one or more sensors are one or a combination of a monocular camera, an IR camera, a stereo camera, and a LiDAR. (Configuration 3) The training data generation system according to configuration 1, wherein the plurality of object detection units are a combination of machine learning-based techniques from among a group of techniques based on machine learning, a group of techniques based on differences between frames, a group of techniques based on pattern matching, and a group of techniques based on clustering of distance information made up of point cloud data, or a combination of a group of machine learning-based techniques and another group of techniques. (Configuration 4) The learning data generation system described in configuration 1, characterized in that when the number of objects detected by each of the plurality of object detection units is different, the learning data generation unit associates the first information with the second information including the type information and stores them. (Configuration 5) The learning data generation system according to configuration 1, wherein the learning data generation unit associates the first information with the second information including the type information and stores the first information when there is a difference of a threshold or more between the position information of the same object detected by each of the plurality of object detection units. (Configuration 6) The learning data generation system according to configuration 1, wherein when there is a difference in the type information of the same object detected by each of the plurality of object detection units in the detection results, the learning data generation unit associates the first information with the second information including the type information and stores the information. (Configuration 7) The learning data generation system described in configuration 1 is characterized in that the learning data generation unit associates and stores the first information with the second information, which is annotation data including the type information detected by any of the plurality of object detection units. (Configuration 8) When there is a first object detection unit that fails to detect the type of a predetermined object and a second object detection unit that succeeds in detecting the type of the predetermined object, the learning data generation unit the type information detected by the first object detection unit is replaced with the type information detected by the second object detection unit, and the second information including the type information is stored in association with the first information; or The learning data generation system according to configuration 7, characterized in that the annotation data, which is the second information including the type information detected by the second object detection unit, and the first information are associated and saved. (Configuration 9) The plurality of object detection units output reliability scores that are information on the likelihood of detecting the object, and the learning data generation unit, when there is a difference in the type information of the same object detected by each of the plurality of object detection units or when there is a difference in the position information of the same object that is equal to or greater than a threshold, considers the detection by the object detection unit with the lowest reliability score to have failed, and considers the detection by the object detection unit with the highest reliability score to have succeeded. (Configuration 10) The training data generation system according to configuration 1, further comprising an object tracking unit that tracks objects for each of the detection results of the plurality of object detection units and assigns a tracking ID, and the training data generation unit generates training data based on the detection results of each of the plurality of object detection units and the tracking ID. (Configuration 11) an acquisition unit that acquires first information, which is either image information of an object or distance information to the object, or both; a plurality of object detection units that detect second information including at least one of object type information, object position information, and object detection probability information based on the first information; a learning data generation unit that generates learning data including the first information and the second information based on the second information detected by the object detection unit; a storage device in which the learning data is stored; Equipped with The learning data generation system is characterized in that, when there is a difference in the second information in the detection results of the plurality of object detection units based on the specific first information, the learning data generation unit associates the specific first information with the second information including the type information and stores them in the storage device. (Configuration 12) an acquisition unit that acquires first information, which is either image information of an object or distance information to the object, or both; a plurality of object detection units that detect second information including at least one of object type information, object position information, and object detection probability information based on the first information; a learning data generation unit that generates learning data including the first information and the second information based on the second information detected by the object detection unit; an output unit that outputs the learning data; Equipped with The information processing device is characterized in that, when there is a difference in the second information in the detection results of the plurality of object detection units based on the specific first information, the learning data generation unit associates the specific first information with the second information including the type information, and the output unit outputs the association. (Configuration 13) 1. An information processing method executed by a processor, comprising: an acquisition step of acquiring first information, which is either image information of an object or distance information to the object, or both; a detection step of detecting second information including at least one of object type information, object position information, and object detection probability information based on the first information using a plurality of object detection units; a generating step of generating learning data including the first information and the second information based on the second information; an output step of outputting the learning data; An information processing method comprising: (Configuration 14) 14. A program for causing a computer to execute the information processing method according to claim 13.

[0128] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. In addition, parts of the above embodiments may be combined with each other as appropriate. [Explanation of symbols]

[0129] 100: Learning data generation system 110: Information processing device 111a, 111b, 111c: object detection unit 112: Learning data generation unit 120: Sensor 130: Storage

Claims

1. one or more sensors that acquire first information, which is either image information of an object or distance information to the object, or both; a plurality of object detection units that detect second information including at least one of object type information, object position information, and object detection probability information based on the first information; a learning data generation unit that generates learning data including the first information and the second information based on the second information detected by the object detection unit; a storage device in which the learning data is stored; Equipped with The learning data generation system is characterized in that, when there is a difference in the second information in the detection results of the plurality of object detection units based on the specific first information, the learning data generation unit associates the specific first information with the second information including the type information and stores them in the storage device.

2. The system for generating training data according to claim 1, wherein the one or more sensors are one or a combination of a monocular camera, an IR camera, a stereo camera, and a LiDAR.

3. The training data generation system according to claim 1, characterized in that the plurality of object detection units are a combination of machine learning-based techniques from among a group of techniques based on machine learning, a group of techniques based on differences between frames, a group of techniques based on pattern matching, and a group of techniques based on clustering of distance information consisting of point cloud data, or a combination of a group of machine learning-based techniques and another group of techniques.

4. The learning data generation system of claim 1, characterized in that when the number of objects detected by each of the plurality of object detection units is different, the learning data generation unit associates and stores the first information with the second information including the type information.

5. The learning data generation system of claim 1, characterized in that when there is a difference in the position information of the same object detected by each of the multiple object detection units that is greater than or equal to a threshold, the learning data generation unit associates and stores the first information with the second information including the type information.

6. 2. The learning data generation system of claim 1, wherein when there is a difference in the type information of the same object detected by each of the plurality of object detection units in the detection results, the learning data generation unit associates and stores the first information with the second information including the type information.

7. The learning data generation system according to claim 1, characterized in that the learning data generation unit associates and stores the first information with the annotation data, which is the second information including the type information detected by any of the plurality of object detection units.

8. the learning data generation unit, when there is a first object detection unit that has failed to detect the type of a predetermined object and a second object detection unit that has succeeded in detecting the type of the predetermined object, the type information detected by the first object detection unit is replaced with the type information detected by the second object detection unit, and the second information including the type information is stored in association with the first information; or The learning data generation system according to claim 7, characterized in that the annotation data, which is the second information including the type information detected by the second object detection unit, and the first information are associated and saved.

9. The plurality of object detection units output reliability scores, which are information on the likelihood of detecting the object, and the learning data generation unit, when there is a difference in the type information of the same object detected by each of the plurality of object detection units or when there is a difference in the position information of the same object that is equal to or greater than a threshold, considers the detection of the object detection unit with the lowest reliability score to have failed, and the detection of the object detection unit with the highest reliability score to have been successful.

10. 2. The training data generation system according to claim 1, further comprising an object tracking unit that tracks objects for each of the detection results of the plurality of object detection units and assigns a tracking ID, and wherein the training data generation unit generates training data based on the detection results and tracking ID of each of the plurality of object detection units.

11. an acquisition unit that acquires first information, which is either image information of an object or distance information to the object, or both; a plurality of object detection units that detect second information including at least one of object type information, object position information, and object detection probability information based on the first information; a learning data generation unit that generates learning data including the first information and the second information based on the second information detected by the object detection unit; a storage device in which the learning data is stored; Equipped with The learning data generation system is characterized in that, when there is a difference in the second information in the detection results of the plurality of object detection units based on the specific first information, the learning data generation unit associates the specific first information with the second information including the type information and stores them in the storage device.

12. an acquisition unit that acquires first information, which is either image information of an object or distance information to the object, or both; a plurality of object detection units that detect second information including at least one of object type information, object position information, and object detection probability information based on the first information; a learning data generation unit that generates learning data including the first information and the second information based on the second information detected by the object detection unit; an output unit that outputs the learning data generated by the learning data generation unit; Equipped with The information processing device is characterized in that, when there is a difference in the second information in the detection results of the plurality of object detection units based on the specific first information, the learning data generation unit associates the specific first information with the second information including the type information, and the output unit outputs the association.

13. 1. An information processing method executed by a processor, comprising: an acquisition step of acquiring first information, which is either image information of an object or distance information to the object, or both; a detection step of detecting second information including at least one of object type information, object position information, and object detection probability information based on the first information using a plurality of object detection units; a generating step of generating learning data including the first information and the second information based on the second information; an output step of outputting the learning data; An information processing method comprising:

14. A program for causing a computer to execute the information processing method according to claim 13.

Images