Methods and systems for generating correct answer data for machine learning in a recognition system.

Abstract

This disclosure relates to a method and system for generating correct answer data for machine learning in a recognition device. A first recognition device that recognizes ambient environment data acquired by a first sensor is fully trained, while a second recognition device that recognizes ambient environment data acquired by a second sensor is untrained. The second sensor and the first sensor are mounted together in a vehicle, and the vehicle is in motion to acquire first ambient environment data from the first sensor and second ambient environment data from the second sensor. The first ambient environment data is then input to the first recognition device to obtain a first recognition result. Furthermore, the first recognition result is projected onto the second ambient environment data by transforming from the coordinate system of the first sensor to the coordinate system of the second sensor. Finally, based on the second ambient environment data projected with the first recognition result, correct answer data for machine learning in the second recognition device is generated.

Description

Technical Field

[0001] This disclosure relates to methods and systems for generating correct answer data for machine learning in a recognizer. Background Technology

[0002] The prior art disclosed in Patent Document 1 relates to a system for constructing a database for machine learning. In this prior art, a subset of points is projected onto the camera image by matching the spatial coordinates of the image of the object to be identified within the camera image with the spatial coordinates of data representing a subset of points. Then, a subset of points whose proportion of points contained within the range of the image of the object to be identified within the camera image is above a threshold is associated with the object to be identified and stored in the database as supervised learning data for machine learning.

[0003] In addition to Patent Document 1, Patent Document 2 can also be shown as a reference to the level of technology in the technical field related to this disclosure.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2017-102838

[0007] Patent Document 2: Japanese Patent Application Publication No. 2020-144755

[0008] To improve the performance of sensor recognition in autonomous vehicles, repeated machine learning is necessary. Machine learning requires preparing correct answer data, which in turn requires annotating the data acquired by the sensors. Furthermore, sensor configurations are sometimes updated due to replacement or new installations, requiring the collection and annotation of large amounts of data each time the sensor configuration is updated. However, annotation is typically a manual task, presenting problems in terms of time and cost. Summary of the Invention

[0009] This disclosure was made in view of the problems mentioned above, and its purpose is to provide a technique that can easily generate correct answer data for machine learning of a recognizer by automatically annotating it.

[0010] This disclosure provides a method for achieving the above-described objectives. The method of this disclosure is a method for generating correct answer data for machine learning in a recognizer. In the method of this disclosure, first ambient environment data is acquired from a first sensor, and at the same time as the acquisition of the first ambient environment data, in the same area as the acquisition area of ​​the first ambient environment data, second ambient environment data is acquired from a second sensor. The first ambient environment data is input to a first recognizer, thereby obtaining a first recognition result. The first recognizer uses the ambient environment data acquired from the first sensor to perform machine learning. The acquired first recognition result is projected onto the second ambient environment data through a transformation from the coordinate system of the first sensor to the coordinate system of the second sensor. That is, annotation is performed on the second ambient environment data. Then, based on the second ambient environment data projected with the first recognition result, correct answer data for machine learning in a second recognizer is generated.

[0011] According to the method of this disclosure, second ambient environment data acquired by a second sensor can be automatically annotated using a first recognition result obtained by a first recognizer using a first sensor that has already undergone machine learning. This allows for the easy generation of correct answer data for machine learning using a second recognizer using the second sensor.

[0012] In the method disclosed herein, the second sensor may also be a sensor having a different modality than the first sensor. Accordingly, the performance of a recognizer with sensors having different modalities can be improved by using the recognition results of a sensor having a certain modality. Sometimes, autonomous vehicles are equipped with multiple sensors with different modalities, such as cameras, LiDAR, and radar.

[0013] In the method disclosed herein, the first and second recognizers may also be configured as object detectors for object detection. Alternatively, the first recognition result may be represented by a bounding box. When the first recognizer is an object detector and the first recognition result is represented by a bounding box, the bounding box is projected onto second surrounding environment data acquired by the second sensor. The second recognizer is also an object detector, and therefore sets the second surrounding environment data with the projected bounding box as the correct answer data, thereby improving the learning efficiency of machine learning.

[0014] In the method disclosed herein, second ambient environment data can also be input into a third recognizer to obtain a second recognition result. The third recognizer uses the ambient environment data acquired from the second sensor to perform machine learning. Alternatively, the acquired second recognition result can be projected onto second ambient environment data that has been projected with the first recognition result. In this case, correct answer data is generated based on the second ambient environment data that has been projected with both the first and second recognition results. By also projecting the second recognition result obtained through the third recognizer onto the second ambient environment data, the accuracy of the annotation can be improved.

[0015] In the method disclosed herein, the second sensor may be a camera, the third recognizer may be an image discriminator performing semantic segmentation, and the second recognition result may be displayed as classification information in pixels. The accuracy of the annotation can be improved by also projecting the output of the image discriminator onto the second surrounding environment data.

[0016] In the method disclosed herein, if the first sensor is LiDAR, first ambient environment data can be input into a velocity estimator that estimates the velocity of the point cluster data acquired from the first sensor, thereby obtaining a velocity estimation result. Alternatively, the obtained velocity estimation result can be projected onto second ambient environment data that has been projected with the first recognition result. In this case, correct answer data is generated based on the second ambient environment data that has been projected with both the first recognition result and the velocity estimation result. By also projecting the output of the velocity estimator onto the second ambient environment data, the accuracy of the annotation can be improved.

[0017] In the method disclosed herein, if the first sensor is LiDAR, the contour shape of the object can be extracted from the point cluster data acquired by the first sensor. Alternatively, the extracted contour shape can be projected onto second surrounding environment data projected with the first recognition result. In this case, correct answer data is generated based on the second surrounding environment data projected with the first recognition result and the contour shape. By also projecting the contour shape of the object extracted from the point cluster data onto the second surrounding environment data, the accuracy of the annotation can be improved.

[0018] This disclosure provides a system for achieving the above-described objectives. The system comprises: a first storage device for storing first ambient environment data acquired by a first sensor; and a second storage device for storing second ambient environment data acquired by a second sensor. The second ambient environment data is ambient environment data acquired at the same time as the acquisition of the first ambient environment data, and in the same area as the acquisition area of ​​the first ambient environment data. Alternatively, the second sensor may be a sensor having a different mode than the first sensor. The first storage device and the second storage device may be different hardware, or they may be different storage areas of the same hardware.

[0019] The system disclosed herein includes: at least one processor; and a program memory coupled to the at least one processor, storing a plurality of executable instructions. The plurality of instructions are configured to cause the at least one processor to read first ambient data from a first storage device and input the first ambient data to a first recognizer to obtain a first recognition result. The first recognizer uses the ambient data acquired from a first sensor to perform machine learning. Furthermore, the plurality of instructions are configured to cause the at least one processor to project the first recognition result onto second ambient data by transforming from the coordinate system of the first sensor to the coordinate system of a second sensor. Moreover, the plurality of instructions are configured to cause the at least one processor to generate correct answer data for machine learning in a second recognizer based on the second ambient data projected with the first recognition result.

[0020] According to the system disclosed herein, second ambient environment data acquired by a second sensor can be automatically annotated using a first recognition result obtained by a first recognizer using a first sensor that has undergone machine learning. This allows for the easy generation of correct answer data for machine learning using a second recognizer using the second sensor.

[0021] Invention Effects

[0022] As described above, according to the method and system of this disclosure, correct answer data for machine learning of the recognizer can be easily generated by automatically annotating the data. Attached Figure Description

[0023] Figure 1 This is a diagram illustrating an outline of the first embodiment of this disclosure.

[0024] Figure 2 This is a diagram illustrating the configuration of a system according to the first embodiment of this disclosure.

[0025] Figure 3 This is a diagram illustrating a first specific example of a system according to a first embodiment of the present disclosure.

[0026] Figure 4 This is a schematic diagram illustrating the automatic annotation process for the first specific instance.

[0027] Figure 5 This is a diagram illustrating a second specific example of a system according to the first embodiment of this disclosure.

[0028] Figure 6 This is a schematic diagram illustrating the automatic annotation process for the second specific instance.

[0029] Figure 7 This is a diagram illustrating a third specific example of a system according to the first embodiment of this disclosure.

[0030] Figure 8 This is a schematic diagram illustrating the automatic annotation process for the third specific instance.

[0031] Figure 9 This is a diagram illustrating a fourth specific example of a system according to the first embodiment of this disclosure.

[0032] Figure 10 This is a schematic diagram illustrating the automatic annotation process for the fourth specific instance.

[0033] Figure 11 This is a diagram illustrating a fifth specific example of a system according to the first embodiment of this disclosure.

[0034] Figure 12 This is a schematic diagram illustrating the automatic annotation process for the fifth specific instance.

[0035] Figure 13 This is a diagram illustrating a sixth specific example of a system according to the first embodiment of this disclosure.

[0036] Figure 14 This is a schematic diagram illustrating the automatic annotation process for the sixth specific instance.

[0037] Figure 15 This is a diagram illustrating a seventh specific example of a system according to the first embodiment of this disclosure.

[0038] Figure 16 This is a diagram illustrating an outline of the second embodiment of this disclosure.

[0039] Explanation of reference numerals in the attached figures

[0040] 2, 2A, 2B: Vehicles

[0041] 4: Moving objects

[0042] 10: System

[0043] 11: Processor

[0044] 12: Program Memory

[0045] 13: Instructions

[0046] 14: Storage devices

[0047] 91: The recognition area of ​​the first sensor

[0048] 92: Recognition area of ​​the second sensor

[0049] 93: Overlapping area

[0050] 100: First sensor

[0051] 120: First Identifier

[0052] 200: Second sensor

[0053] 220: Annotator

[0054] 260: Second Recognizer. Detailed Implementation

[0055] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, when the number, quantity, quantity, range, etc., of each element are mentioned in the embodiments shown below, the technical concept of the present disclosure is not limited to the mentioned number, unless specifically stated or clearly determined in principle. Furthermore, the structures, etc., described in the embodiments shown below are not necessarily essential to the technical concept of the present disclosure, unless specifically stated or clearly determined in principle.

[0056] 1. First Implementation Method

[0057] 1-1. Overview of the First Embodiment

[0058] Figure 1 This is a diagram illustrating an outline of the first embodiment of this disclosure. Figure 1 The vehicle 2 shown is an autonomous vehicle that identifies its surroundings based on environmental data acquired by external sensors and drives itself accordingly. Vehicle 2 is equipped with two external sensors: a first sensor 100 and a second sensor 200. A first identifier is provided for the first sensor 100, and a second identifier is provided for the second sensor 200. Each identifier is configured, for example, as an application program of an onboard computer.

[0059] Here, we examine the case where the second sensor 200 is replaced with a new sensor. The new sensor may include, for example, a different type of sensor, a sensor of the same type from a different manufacturer, or a different model of the same type from the same manufacturer. When the second sensor 200 is replaced with these sensors, the second recognizer, which has undergone machine learning using the output of the previous sensor, cannot correspond to the new sensor. In this case, it is necessary to re-perform machine learning on the second recognizer to correspond to the new second sensor 200.

[0060] The machine learning required for the second recognizer is correct answer data to generate training and testing datasets. To create the correct answer data, annotation is needed, i.e., adding labels to the feature quantities contained in the data acquired by the second sensor 200. The method of the first embodiment also automatically performs this labeling task. In the method of the first embodiment, the surrounding environment data acquired by the existing first sensor 100 is used for annotation.

[0061] like Figure 1 As shown, there is an overlapping region 93 between region 91 identified by the first sensor 100 and region 92 identified by the second sensor 200. The moving object 4 present in the overlapping region 93 is included in the surrounding environment data (first surrounding environment data) acquired by the first sensor 100, and also in the surrounding environment data (second surrounding environment data) acquired by the second sensor 200.

[0062] The second recognizer needs to learn due to the replacement of the second sensor 200, but the first recognizer corresponding to the first sensor 100 will still function. The first ambient environment data acquired through the first sensor 100 is input to the first recognizer, thereby obtaining the recognition result obtained through the first sensor 100 (the first recognition result). The first ambient environment data and the second ambient environment data are data acquired at the same time in the same area; therefore, if the first recognition result contains the moving object 4, the second ambient environment data will also contain data on the moving object 4.

[0063] The first ambient environment data, containing the first recognition result, is data represented by the coordinate system of the first sensor 100, and the second ambient environment data is data represented by the coordinate system of the second sensor 200. Due to differences in the type of sensor and its mounting position and posture on the vehicle 2, there are differences between the coordinate systems of the first sensor 100 and the second sensor 200. In the method of the first embodiment, the first recognition result is projected onto the second ambient environment data by transforming from the coordinate system of the first sensor 100 to the coordinate system of the second sensor 200. To ensure accurate coordinate system transformation, calibration is performed beforehand between the first sensor 100 and the second sensor 200.

[0064] By projecting the first identification result onto the second surrounding environment data, the first identification result is superimposed on the feature values ​​of the second surrounding environment data. This determines which feature value of the second surrounding environment data corresponds to the moving object 4. Annotations using the first identification result are also applied to other feature values ​​contained in the second surrounding environment data, thereby generating correct answer data for the second surrounding environment data.

[0065] 1-2. System of the first embodiment

[0066] Figure 2 This diagram illustrates the configuration of the system according to the first embodiment. System 10 of the first embodiment uses the output of an existing first sensor 100 to generate correct answer data for machine learning of the recognizer in the second sensor 200. The first sensor 100 and the second sensor 200 are pre-calibrated for relative position and relative posture while mounted on the vehicle 2. It should be noted that if the mounting position and posture of the first sensor 100 and the second sensor 200 are the same as those of an actual autonomous vehicle, the vehicle 2 used to generate the correct answer data may not necessarily be an autonomous vehicle.

[0067] System 10 includes a storage device (first storage device) 110 for storing first ambient environment data acquired by the first sensor 100. The first ambient environment data includes a timecode indicating the acquisition time of the first sensor 100. Furthermore, system 10 includes a storage device (second storage device) 210 for storing second ambient environment data acquired by the second sensor 200. The second ambient environment data includes a timecode indicating the acquisition time of the second sensor 200. Both the first and second ambient environment data can be data acquired while the vehicle 2 is traveling on a street, or data acquired, for example, while the vehicle 2 is stopped near a road with heavy traffic.

[0068] System 10 includes a first recognizer 120. The first recognizer 120 is a learner that has undergone machine learning training using ambient environment data acquired from the first sensor 100. The first ambient environment data stored in storage device 110 is input to the first recognizer 120. The first recognizer 120 outputs a first recognition result obtained through the recognition processing of the first ambient environment data. The first recognition result is stored in storage device 130.

[0069] System 10 includes a transformer 140. Transformer 140 is configured to transform data from the coordinate system of the first sensor 100 to the coordinate system of the second sensor 200. A first identification result stored in storage device 130 is data represented by the coordinate system of the first sensor 100. The first identification result is input to transformer 140 and transformed from the coordinate system of the first sensor 100 to the coordinate system of the second sensor 200. The result of calibration between the first sensor 100 and the second sensor 200 is used in the coordinate system transformation.

[0070] System 10 includes an annotator 220 that automatically performs annotations. The annotator 220 reads second ambient data from storage device 210 and projects a first recognition result, transformed to the coordinate system of the second sensor 200, onto the second ambient data. Annotation is performed by the annotator 220 to ensure that the timecode of the second ambient data (the projection destination) matches the timecode of the first recognition result (the projection object). The annotator 220 outputs the tagged second ambient data obtained through annotation. The tagged second ambient data is stored in storage device 230.

[0071] System 10 includes a correct answer data generator 240. The correct answer data generator 240 generates correct answer data for machine learning in the second recognizer 260 based on labeled second surrounding environment data. For example, correct answer data is generated by changing the data format of the labeled second surrounding environment data to a data format used for machine learning. Furthermore, generating correct answer data also includes directly using the labeled second surrounding environment data as correct answer data. Correct answer data is stored in storage device 250 for machine learning in the second recognizer 260.

[0072] System 10 can consist of a single computer or multiple computers configured on a network. System 10 includes a processor 11, a program memory 12, and a storage device 14 as hardware. The processor 11 is coupled to the program memory 12 and the storage device 14 via the computer's internal bus or an inter-computer network.

[0073] The program memory 12 is a non-temporary memory that stores a plurality of executable instructions 13. The instructions 13 constitute a program for implementing the method of embodiment 1. The instructions 13 are executed by the processor 11, thereby enabling the processor 11 to function as a first recognizer 120, a converter 140, an annotator 220, and a correct answer data generator 240.

[0074] Storage device 14 can be, for example, flash memory, SSD (Solid State Disk), or HDD (Hard Disk Drive). It can be a single storage device 14 containing areas equivalent to storage units 110, 130, 210, 230, and 250, or it can be a single, independent storage device 14 for each of the storage units 110, 130, 210, 230, and 250.

[0075] 1-2-1. First specific example

[0076] Hereinafter, a specific example of the system 10 of the first embodiment will be described.

[0077] Figure 3This is a diagram illustrating system 10-1 as a first specific example. In this first specific example, the existing first sensor is LiDAR 101, and the new second sensor is camera 201. Camera 201 is a sensor with a different mode than LiDAR 101.

[0078] In the first specific example, point cluster data is obtained using LiDAR 101 as first ambient environment data. The point cluster data obtained by LiDAR 101 is stored in storage device 111. Furthermore, image data is obtained using camera 201 as second ambient environment data. The image data obtained by camera 201 is stored in storage device 211.

[0079] The first specific example, system 10-1, includes an object detector 121. The object detector 121 is a recognizer (first recognizer) that uses a deep learning model and performs machine learning to identify moving objects based on point cluster data acquired from LiDAR 101. Point cluster data stored in storage device 111 is input to the object detector 121. The object detector 121 outputs a recognition result obtained through the recognition processing of the point cluster data. The recognition result includes a three-dimensional bounding box representing the size of the identified moving object and the object name of the identified moving object. The identified three-dimensional bounding box and object name are stored in storage device 131.

[0080] In the first specific example, system 10-1, transformer 141 is configured to transform data from the three-dimensional coordinate system of LiDAR 101 to the two-dimensional coordinate system of camera 201. A three-dimensional bounding box is input to transformer 141 and transformed into a two-dimensional bounding box. The result of calibration between LiDAR 101 and camera 201 is used in the coordinate system transformation.

[0081] In the first specific example of system 10-1, the annotator 221 reads image data from storage device 211 and projects the two-dimensional bounding box and object name obtained by transformer 141 onto the image data. The annotator 221 annotates the image data at the projection destination with the timecode of the projected data. The annotator 221 outputs the labeled image data obtained through annotation. The labeled image data is stored in storage device 231.

[0082] Correct answer data generator 241 generates correct answer data for machine learning in object detector 261 based on labeled image data. Object detector 261 is a recognizer (second recognizer) that uses a deep learning model. Object detector 261 performs machine learning by recognizing moving objects based on image data acquired by camera 201. Correct answer data is stored in storage device 251 for machine learning by object detector 261.

[0083] Here, use Figure 4 The automatic annotation process implemented by the system 10-1 in the first specific example will be described in detail. Point group data 20 obtained by LiDAR 101 is input to object detector 121, thereby representing moving objects contained in the point group data 20 with three-dimensional bounding boxes 21 and 22. Furthermore, labels 21a and 22a representing object names are attached to each three-dimensional bounding box 21 and 22. Figure 4 In the example shown, 3D bounding box 21 represents a car, and 3D bounding box 22 represents a truck.

[0084] The three-dimensional bounding boxes 21 and 22 are transformed into two-dimensional bounding boxes 31 and 32 by transformer 141. The two-dimensional bounding boxes 31 and 32, together with labels 31a and 32a representing object names, are projected onto the image data 30 obtained by camera 201. Thus, the image of the car in the image data 30 is surrounded by the two-dimensional bounding box 31 and labeled with label 31a, and the image of the truck is surrounded by the two-dimensional bounding box 32 and labeled with label 32a.

[0085] As described above, according to the system 10-1 of the first specific example, image data 30 acquired by the new camera 201 can be automatically annotated using the recognition results obtained by the object detector 121 used by the existing LiDAR 101. Thus, correct answer data for machine learning using the object detector 261 used by the camera 201 can be easily generated.

[0086] 1-2-2. Second specific example

[0087] Figure 5 This is a diagram illustrating system 10-2 as a second specific example. Similar to the first specific example, the second specific example also uses the recognition results of a sensor with a certain modality to improve the performance of a recognizer with sensors having different modalities. In the second specific example, the existing first sensor is camera 102, and the new second sensor is LiDAR 202.

[0088] In the second specific example, image data acquired by camera 102 is used as first ambient environment data. The image data acquired by camera 102 is stored in storage device 112. Furthermore, point cluster data acquired by LiDAR 202 is used as second ambient environment data. The point cluster data acquired by LiDAR 202 is stored in storage device 212.

[0089] The second specific example, system 10-2, includes an object detector 122. The object detector 122 is a recognizer (first recognizer) that uses a deep learning model and performs machine learning to identify moving objects based on image data acquired by camera 102. Image data stored in storage device 112 is input to the object detector 122. The object detector 122 outputs a recognition result obtained through the recognition processing of the image data. The recognition result includes a two-dimensional bounding box representing the size of the identified moving object and the object name of the identified moving object. The identified two-dimensional bounding box and object name are stored in storage device 132.

[0090] In the second specific example, system 10-2, transformer 142 is configured to transform data from the two-dimensional coordinate system of camera 102 to the three-dimensional coordinate system of LiDAR 202. Two-dimensional bounding boxes are input to transformer 142 and transformed into three-dimensional objects. The results of calibration between camera 102 and LiDAR 202 are used in the coordinate system transformation.

[0091] In the second specific example, system 10-2, the annotator 222 reads point group data from storage device 212 and projects the 3D object and object name obtained by transformer 142 onto the point group data. The annotator 222 annotates the point group data at the projection destination with the timecode of the projected data. The annotator 222 outputs the annotated and labeled point group data. The labeled point group data is stored in storage device 232.

[0092] Correct answer data generator 242 generates correct answer data for machine learning in object detector 262 based on labeled dot cluster data. Object detector 262 is a recognizer (second recognizer) using a deep learning model. Object detector 262 performs machine learning by recognizing moving objects based on dot cluster data acquired from LiDAR 202. Correct answer data is stored in storage device 252 for machine learning by object detector 262.

[0093] Here, use Figure 6 The automatic annotation process implemented by system 10-2 in the second specific example will be described in detail. Image data 40 obtained by camera 102 is input to object detector 122, thereby representing the moving objects contained in image data 40 with two-dimensional bounding boxes 41 and 42. Furthermore, labels 41a and 42a indicating the object names are attached to each two-dimensional bounding box 41 and 42. Figure 6 In the example shown, 2D bounding box 41 represents a car, and 2D bounding box 42 represents a truck.

[0094] Based on the point group data 50 obtained by LiDAR 202, three-dimensional convex hulls 51, 52, 53, and 54 representing the contour shapes of objects are generated. The three-dimensional objects, transformed from two-dimensional bounding boxes 41 and 42, are projected onto the point group data 50, making the convex hull 51 representing the contour shape of a car and the convex hull 52 representing the contour shape of a truck apparent. Label 51a is attached to convex hull 51, and label 52a is attached to convex hull 52. Convex hulls 53 and 54 correspond to trees 43 and 44 in image data 40, respectively, but since trees 43 and 44 are stationary objects, they are not detected as moving objects by the object detector 122. Therefore, no additional labels are applied to convex hulls 53 and 54.

[0095] As described above, according to the system 10-2 of the second specific example, the point cluster data 50 acquired by the new LiDAR 202 can be automatically annotated using the recognition results obtained by the object detector 122 of the existing camera 102. Thus, correct answer data for machine learning using the object detector 262 of the LiDAR 202 can be easily generated.

[0096] 1-2-3. Third specific example

[0097] Figure 7 This is a diagram illustrating system 10-3 as a third specific example. The third specific example is an example of using the recognition results of a sensor with a certain modality to improve the performance of a recognizer with a sensor having the same modality. In the third specific example, the existing first sensor is camera 103, and the new second sensor is also camera 203.

[0098] In the third specific example, image data is acquired by camera 103 as first ambient environment data. The image data acquired by camera 103 is stored in storage device 113. Furthermore, image data is acquired by camera 203 as second ambient environment data. The image data acquired by camera 203 is stored in storage device 213.

[0099] The third specific example, system 10-3, includes an object detector 123. The object detector 123 is a recognizer (first recognizer) that uses a deep learning model and performs machine learning to identify moving objects based on image data acquired by camera 103. Image data stored in storage device 113 is input to the object detector 123. The object detector 123 outputs a recognition result obtained through the recognition processing of the image data. The recognition result includes a two-dimensional bounding box representing the size of the identified moving object and the object name of the identified moving object. The identified two-dimensional bounding box and object name are stored in storage device 133.

[0100] In the third specific example, system 10-3, transformer 143 is configured to transform data from the existing two-dimensional coordinate system of camera 103 to the two-dimensional coordinate system of new camera 203. Two-dimensional bounding boxes are input to transformer 143 and subjected to transformations such as parallel translation, rotation, and scaling. The results of calibration between the existing camera 103 and the new camera 203 are used in the coordinate system transformation.

[0101] In the third specific example, system 10-3, the annotator 223 reads image data from storage device 213 and projects the two-dimensional bounding box and object name obtained by transformer 143 onto the image data. The annotator 223 annotates the image data at the projection destination with the timecode of the projected data. The annotator 223 outputs the labeled image data obtained through annotation. The labeled image data is stored in storage device 233.

[0102] Correct answer data generator 243 generates correct answer data based on labeled image data for machine learning in object detector 263. Object detector 263 is a recognizer (second recognizer) that uses a deep learning model. Object detector 263 performs machine learning by recognizing moving objects based on image data acquired by camera 203. Correct answer data is stored in storage device 253 for machine learning by object detector 263.

[0103] Here, use Figure 8 The automatic annotation process implemented by system 10-3 in the third specific example will be described in detail. Image data 60L obtained by camera 103 is input to object detector 123, thereby representing the moving objects contained in image data 60L with two-dimensional bounding boxes 61 and 62. Furthermore, labels 61a and 62a representing the object names are attached to each two-dimensional bounding box 61 and 62. Figure 8 In the example shown, 2D bounding box 61 represents a car, and 2D bounding box 62 represents a truck.

[0104] Two-dimensional bounding boxes 61 and 62 are transformed into two-dimensional bounding boxes 63 and 64 by transformer 143. Two-dimensional bounding boxes 63 and 64, along with labels 63a and 64a representing object names, are projected onto image data 60R obtained by camera 203. Thus, the image of the car in image data 60R is bounded by two-dimensional bounding box 63 and labeled with label 63a, while the image of the truck is bounded by two-dimensional bounding box 64 and labeled with label 64a.

[0105] As described above, according to the third specific example of system 10-3, the image data 60R acquired by the new camera 203 can be automatically annotated using the recognition results obtained by the object detector 123 of the existing camera 103. Thus, correct answer data for machine learning on the object detector 263 of the camera 203 can be easily generated.

[0106] 1-2-4. Fourth Specific Example

[0107] Figure 9 This is a diagram representing system 10-4 as the fourth specific example. System 10-4, the fourth specific example, is a variation of system 10-1, the first specific example. Figure 9 The same reference numerals are used for elements shared with the first specific example system 10-1.

[0108] The fourth specific example, system 10-4, includes an image discriminator 314. The image discriminator 314 is a deep learning model-based recognizer (third recognizer) that performs semantic segmentation on image data acquired by camera 201. Image data stored in storage device 211 is input to the image discriminator 314. The image discriminator 314 outputs pixel-level classification information as a recognition result (second recognition result). The classification information is stored in storage device 324.

[0109] In system 10-4 of the fourth specific example, annotator 224 reads image data from storage device 211. In addition to the two-dimensional bounding box and object name obtained by transformer 141, annotator 224 also projects classification information read from storage device 324 onto the image data. Annotator 224 annotates the image data at the projection destination with the timecode of the projected data. Labeled image data obtained through annotation is output from annotator 224. The labeled image data is stored in storage device 234.

[0110] Correct answer data generator 244 generates correct answer data based on labeled image data for machine learning in object detector 264. Object detector 264 is a recognizer (second recognizer) using a deep learning model. Object detector 264 performs machine learning by recognizing moving objects based on image data acquired by camera 201. Correct answer data is stored in storage device 254 for machine learning by object detector 264.

[0111] Here, use Figure 10 The automatic annotation process implemented by system 10-4 in the fourth specific example will be described in detail. Image data obtained by camera 201 is input to image discriminator 314, thereby obtaining image data 70 classified in pixels. Figure 10 In the example shown, image data 70 is classified into moving objects 71, stationary objects 72, and the ground 73. The classification information obtained by image discriminator 314 is projected onto image data 30.

[0112] Three 2D bounding boxes 31, 32, and 36 are projected onto image data 30. However, only 2D bounding boxes 31 and 32 are projected in an overlapping manner to represent the classification information of the moving object 71. The projection onto 2D bounding box 36 represents the classification information of the stationary object 72. Based on this result, it is clear that 2D bounding boxes 31 and 32 are undoubtedly moving objects, while 2D bounding box 36 is a false detection. The annotator 224 outputs the labeled image data with 2D bounding box 36 removed.

[0113] As described above, in the fourth specific example of system 10-4, in addition to the two-dimensional bounding box identified based on the output of LiDAR 101, the classification information of the image data obtained by the image discriminator 314 is also projected onto the image data. Thus, by doubly projecting different data, the accuracy of the annotation can be further improved.

[0114] 1-2-5. Fifth Specific Example

[0115] Figure 11 This is a diagram representing system 10-5 as the fifth specific example. System 10-5, the fifth specific example, is a variation of system 10-1, the first specific example. Figure 11 The same reference numerals are used for elements shared with the first specific example system 10-1.

[0116] The fifth specific example, system 10-5, includes a velocity estimator 315. The velocity estimator 315 is an estimator that uses a deep learning model to estimate the velocity of the point group data acquired by LiDAR 101. The point group data stored in storage device 111 is input to the velocity estimator 315. The velocity estimation result is output from the velocity estimator 315. The velocity estimation result is stored in storage device 325.

[0117] In system 10-5 of the fifth specific example, the annotator 225 reads image data from storage device 211. In addition to the two-dimensional bounding box and object name obtained by transformer 141, the annotator 225 also projects the velocity estimation result read from storage device 325 onto the image data. The annotator 225 annotates the image data at the projection destination with the timecode of the projected data. The annotator 225 outputs the labeled image data obtained through annotation. The labeled image data is stored in storage device 235.

[0118] Correct answer data generator 245 generates correct answer data based on labeled image data for machine learning in object detector 265. Object detector 265 is a recognizer (second recognizer) using a deep learning model. Object detector 265 performs machine learning by recognizing moving objects based on image data acquired by camera 201. Correct answer data is stored in storage device 255 for machine learning by object detector 265.

[0119] Here, use Figure 12 The automatic annotation process implemented by system 10-5 in the fifth specific example will be described in detail. Point group data obtained by LiDAR 101 is input to velocity estimator 315, thereby obtaining velocity estimation results. The velocity estimation results are represented by velocity vectors 37 and 38 and projected onto image data 30.

[0120] Three 2D bounding boxes 31, 32, and 36 are projected onto image data 30. However, only 2D bounding boxes 31 and 32 are overlaid with velocity vectors 37 and 38. The velocity vector is not projected onto 2D bounding box 36. Based on this result, it is clear that 2D bounding boxes 31 and 32 are undoubtedly the moving objects, while 2D bounding box 36 is a false detection. The annotator 225 outputs the labeled image data with 2D bounding box 36 removed.

[0121] As described above, in system 10-5 of the fifth specific example, in addition to the two-dimensional bounding box identified based on the output of LiDAR 101, the velocity estimation result obtained by the velocity estimator 315 is also projected onto the image data. Thus, by doubly projecting different data, the accuracy of annotation can be further improved.

[0122] 1-2-6. The sixth specific example

[0123] Figure 13 This is a diagram representing system 10-6 as the sixth specific example. System 10-6, the sixth specific example, is a variation of system 10-1, the first specific example. Figure 13 The same reference numerals are used for elements shared with the first specific example system 10-1.

[0124] The sixth specific example, system 10-6, includes a three-dimensional convex hull generator 316. The three-dimensional convex hull generator 316 extracts the contour shape of an object from point group data acquired by LiDAR 101. The extracted contour shape of the object is represented by a three-dimensional convex hull. Point group data stored in storage device 111 is input to the three-dimensional convex hull generator 316. The three-dimensional convex hull output from the three-dimensional convex hull generator 316 is stored in storage device 326.

[0125] The sixth specific example, system 10-6, includes a transformer 336. Transformer 336 is configured to transform a three-dimensional convex hull into a two-dimensional convex hull by projecting the three-dimensional convex hull onto a plane. The plane onto which the three-dimensional convex hull is projected is a plane established by camera 201. The result of calibration between LiDAR 101 and camera 201 is used in the transformation from the three-dimensional convex hull to the two-dimensional convex hull.

[0126] In system 10-6 of the sixth specific example, annotator 226 reads image data from storage device 211. In addition to the two-dimensional bounding box and object name obtained by transformer 141, annotator 226 also projects the two-dimensional convex hull obtained by transformer 336 onto the image data. Annotator 226 annotates the image data so that the timecode of the projection destination image data is consistent with the timecode of the projected data. Labeled image data obtained through annotation is output from annotator 226. The labeled image data is stored in storage device 236.

[0127] Correct answer data generator 246 generates correct answer data based on labeled image data for machine learning in object detector 266. Object detector 266 is a recognizer (second recognizer) that uses a deep learning model. Object detector 266 performs machine learning by recognizing moving objects based on image data acquired by camera 201. Correct answer data is stored in storage device 256 for machine learning by object detector 266.

[0128] Here, use Figure 14 The automatic annotation process implemented by system 10-6 in the sixth specific example will be described in detail. Point group data 20 obtained by LiDAR 101 is input into the three-dimensional convex hull generator 316, thereby obtaining three-dimensional convex hulls 23 and 24. The three-dimensional convex hulls 23 and 24 are transformed into two-dimensional convex hulls 33 and 34 by transformer 336, and the two-dimensional convex hulls 33 and 34 are projected onto image data 30.

[0129] Three 2D bounding boxes 31, 32, and 36 are projected onto image data 30. However, only 2D bounding boxes 31 and 32 are overlaid onto 2D convex hulls 33 and 34. The 2D convex hull is not projected onto 2D bounding box 36. Based on this result, it is clear that 2D bounding boxes 31 and 32 are undoubtedly the moving objects, while 2D bounding box 36 is a false detection. The annotator 226 outputs the labeled image data with 2D bounding box 36 removed.

[0130] As described above, in the sixth specific example, system 10-6, in addition to the two-dimensional bounding box identified based on the output of LiDAR 101, the two-dimensional convex hull obtained from the point group data is also projected onto the image data. Thus, by doubly projecting different data, the accuracy of the annotation can be further improved.

[0131] 1-2-7. Seventh Specific Example

[0132] Figure 15 This diagram illustrates system 10-7 as a seventh specific example. The seventh specific example is an image discriminator, where the recognizer needs to be relearned upon replacement of the second sensor. System 10-7 of the seventh specific example can be described as a variation of system 10-1 of the first specific example. Figure 15 The same reference numerals are used for elements shared with the first specific example system 10-1.

[0133] The seventh specific example, system 10-7, generates correct answer data for machine learning in image discriminator 267. Image discriminator 267 is a recognizer (second recognizer) that uses a deep learning model. Image discriminator 267 performs machine learning by semantically segmenting the image data acquired by camera 201. The method for generating correct answer data is shared with system 10-1 of the first specific example. Therefore, system 10-7 of the seventh specific example is equivalent to replacing the object detector 261 in the configuration of system 10-1 of the first specific example with image discriminator 267.

[0134] 2. Second Implementation Method

[0135] Figure 16 This is a diagram illustrating an outline of the second embodiment of this disclosure. Figure 16 The first vehicle 2A and the second vehicle 2B shown are autonomous vehicles. The first vehicle 2A is equipped with a first sensor 100 and a first identifier corresponding to the first sensor 100. The second vehicle 2B is equipped with a second sensor 200 and a second identifier corresponding to the second sensor 200.

[0136] In the case where the second sensor 200 in the second vehicle 2B is replaced with a new sensor, machine learning needs to be performed on the second recognizer in conjunction with the new second sensor 200. In order to generate the correct answer data required for this machine learning, in the method of the second embodiment, the surrounding environment data acquired by the first sensor 100 mounted on the first vehicle 2A is used for annotation.

[0137] like Figure 16 As shown, when the first vehicle 2A and the second vehicle 2B approach each other, an overlapping region 93 is generated between the region 91 identified by the first sensor 100 of the first vehicle 2A and the region 92 identified by the second sensor 200 of the second vehicle 2B. The moving object 4 present in the overlapping region 93 is included in the surrounding environment data (first surrounding environment data) acquired by the first sensor 100, and also in the surrounding environment data (second surrounding environment data) acquired by the second sensor 200.

[0138] In the second embodiment, when an overlapping region 93 is generated, first ambient environment data is acquired in the first vehicle 2A via the first sensor 100, and second ambient environment data is acquired in the second vehicle 2B via the second sensor 200. Then, the first ambient environment data acquired via the first sensor 100 is input to a first recognizer, thereby obtaining a recognition result obtained via the first sensor 100. The recognition result obtained via the first sensor 100 is projected onto the second ambient environment data through a transformation from the coordinate system of the first sensor 100 to the coordinate system of the second sensor 200.

[0139] In the coordinate transformation of the recognition result obtained by the first sensor 100, the position and orientation of each vehicle 2A and 2B in the global coordinate system at the time of data acquisition are referenced. In the method of the second embodiment, for each of the first vehicle 2A and the second vehicle 2B, the position and orientation in the global coordinate system are measured using GPS (Global Positioning System) and IMU (Inertial Measurement Unit). The position and orientation data of the first vehicle 2A in the global coordinate system, measured together with a time code, are associated with the first ambient environment data acquired by the first sensor 100. The position and orientation data of the second vehicle 2B in the global coordinate system, measured together with a time code, are associated with the second ambient environment data acquired by the second sensor 200.

[0140] The system and specific examples used to implement the method of the first embodiment can also be applied to the method of the second embodiment.

[0141] 3. Other implementation methods

[0142] In the above embodiments, a camera and a LiDAR are illustrated as the first and second sensors, but the first and second sensors can also be radar. Radar is a sensor with a different mode than a camera and a LiDAR. For example, if the first sensor is radar, the second sensor can be a camera or a LiDAR. Alternatively, if the first sensor is a camera or a LiDAR, the second sensor can be radar.

Claims

1. A method of generating correct answer data for machine learning of a recognizer, characterized by, Comprising: acquiring first surrounding environment data from a first sensor; acquiring second surrounding environment data from a second sensor in the same region as the acquisition region of the first surrounding environment data at the same time as the acquisition time of the first surrounding environment data; inputting the first surrounding environment data to a first recognizer that has been subjected to machine learning using surrounding environment data acquired from the first sensor to acquire a first recognition result; projecting the first recognition result to the second surrounding environment data by transformation from the coordinate system of the first sensor to the coordinate system of the second sensor; inputting the second surrounding environment data to a third recognizer that has been subjected to machine learning using surrounding environment data acquired from the second sensor to acquire a second recognition result; further projecting the second recognition result to the second surrounding environment data to which the first recognition result has been projected; and generating correct answer data for machine learning of a second recognizer that is a recognizer that recognizes surrounding environment data acquired by the second sensor, based on the second surrounding environment data to which the first recognition result and the second recognition result have been projected.

2. The method of generating correct answer data for machine learning of a recognizer according to claim 1, wherein the second sensor is a sensor having a different modality from the first sensor.

3. The method of generating correct answer data for machine learning of a recognizer according to claim 1 or 2, wherein the first recognizer and the second recognizer are object detectors configured to perform object detection, the first recognition result is shown with a bounding box.

4. The method of generating correct answer data for machine learning of a recognizer according to claim 1, wherein the second sensor is a camera, the third recognizer is an image discriminator that performs semantic segmentation, the second recognition result is shown with classification information in units of pixels.

5. The method of generating correct answer data for machine learning of a recognizer according to claim 1 or 2, wherein the first sensor is a LiDAR, the method further comprises: inputting the first surrounding environment data to a speed estimator that estimates a speed of point cloud data acquired from the first sensor to acquire a speed estimation result; and further projecting the speed estimation result to the second surrounding environment data to which the first recognition result has been projected, generating the correct answer data includes generating the correct answer data based on the second surrounding environment data to which the first recognition result and the speed estimation result have been projected.

6. The method of generating correct answer data for machine learning of a recognizer according to claim 1 or 2, wherein the first sensor is a LiDAR, the method further comprises: extracting an outline shape of an object from point cloud data acquired from the first sensor; and further projecting the outline shape to the second surrounding environment data to which the first recognition result has been projected, The generating the correct answer data includes generating the correct answer data based on the second surrounding environment data to which the first recognition result and the contour shape are projected.

7. A system for generating correct answer data for machine learning of an identifier, characterized by, Possessing: a first storage device that stores first surrounding environment data acquired by a first sensor; a second storage device that stores second surrounding environment data acquired by a second sensor in the same region as an acquisition region of the first surrounding environment data at the same time as an acquisition time of the first surrounding environment data; at least one processor; and a program memory coupled to the at least one processor and storing executable instructions, the instructions being configured to cause the at least one processor to perform: inputting, to a first recognizer that has been subjected to machine learning using surrounding environment data acquired from the first sensor, the first surrounding environment data read out from the first storage device to acquire a first recognition result; projecting the first recognition result to the second surrounding environment data read out from the second storage device by transformation from a coordinate system of the first sensor to a coordinate system of the second sensor; inputting, to a third recognizer that has been subjected to machine learning using surrounding environment data acquired from the second sensor, the second surrounding environment data read out from the second storage device to acquire a second recognition result; further projecting the second recognition result to the second surrounding environment data to which the first recognition result is projected; and generating, based on the second surrounding environment data to which the first recognition result and the second recognition result are projected, correct answer data for machine learning by a second recognizer that is a recognizer that recognizes surrounding environment data acquired by the second sensor. ​

Images