Apparatus and method for automatically labeling learning data to compensate for weaknesses in AI object detection function of autonomous vehicles
The apparatus and method automatically correct sensor misrecognitions by synchronizing data from multiple modules to generate training data, improving object recognition performance in autonomous vehicles without manual labor.
Patent Information
- Application Number
- KR1020220159955
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-11-25
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-11-25
AI Technical Summary
Autonomous vehicle sensors such as camera, LiDAR, and radar sensors often fail to recognize or misidentify objects, lacking a method for auto-labeling these unrecognized or misrecognized objects, which hinders the continuous improvement of object recognition performance.
An apparatus and method that utilizes multiple sensor modules to recognize objects, determines recognition errors, and automatically generates training data by labeling misrecognized objects using a processor to synchronize and correct sensor data, eliminating the need for manual labor.
Enables continuous improvement of object recognition performance by automatically constructing training data for unrecognized and misrecognized objects, reducing costs and labor, and enhancing sensor accuracy.
Smart Images

Figure 112022126271108-PAT00001_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to an apparatus and method for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle, and more specifically, to an apparatus and method for automatically constructing training data for objects that are not recognized or are misrecognized by sensors (e.g., camera sensors, LiDAR sensors, radar sensors) installed in an autonomous vehicle. Background Technology
[0002] Automatic driving refers to a system in which a vehicle can make its own decisions and drive.
[0003] Data used for machine learning of artificial intelligence (AI) applicable to autonomous driving of vehicles is collected by various types of sensors installed in the vehicle. For example, the data used for machine learning of artificial intelligence (AI) applicable to autonomous driving of vehicles may be, but is not limited to, data acquired, captured, or detected by lidar, cameras, radar, and ultrasonic sensors fixedly installed in the vehicle.
[0004] However, camera sensors, lidar sensors, radar sensors, etc., in autonomous vehicles may fail to recognize or misidentify objects, and there is no way to identify such objects that have been failed to recognize or misidentified by these sensors.
[0005] Furthermore, there is no device that performs auto-labeling for objects that are not recognized or are misrecognized by the camera sensors, LiDAR sensors, radar sensors, etc., of autonomous vehicles.
[0006] As such, in order to continuously improve the object recognition performance of camera, lidar, and radar sensors installed in autonomous vehicles to a high level, it is important to detect unrecognized or misrecognized objects and to continuously learn from them.
[0007] To achieve this, camera sensors, LiDAR sensors, and radar sensors must be able to reliably detect unrecognized or misrecognized objects, and training data for those objects must be built quickly and at low cost.
[0008] The matters described in the background technology above are intended to aid in understanding the background of the invention and may include matters that are not disclosed prior art. Prior art literature
[0009] Prior Art 1: Korean Registered Patent No. 10-2310608 (Oct. 01, 2021) Prior Art 2: Korean Registered Patent No. 10-2264152 (June 7, 2021) The problem to be solved
[0010] The present invention is proposed in consideration of the aforementioned conventional circumstances, and aims to provide an apparatus and method for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle, wherein camera sensors, lidar sensors, and radar sensors installed in the autonomous vehicle detect objects that are not recognized or are misrecognized in real time, and automatically perform data labeling for the detected objects to construct training data. means of solving the problem
[0011] To achieve the above objectives, an automatic learning data construction device for supplementing vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle according to a preferred embodiment of the present invention comprises: a plurality of sensor modules that recognize external objects of the autonomous vehicle and output object recognition data including object detection data in which object attribute data is labeled according to the recognition result; a storage module configured to store a data labeling method associated with an object recognition method of each of the plurality of sensor modules; and a processor that determines a target sensor module that caused an object recognition error and a normal sensor module that did not cause an object recognition error among the plurality of sensor modules based on the object recognition data of each of the plurality of sensor modules, determines alternative object recognition data for an object area corresponding to the object recognition error among the object recognition data of the normal sensor module, generates alternative object detection data and alternative object attribute data for an object area corresponding to the object recognition error of the target sensor module using the alternative object recognition data, and generates alternative object detection data in which the alternative object attribute data is labeled based on the data labeling method of the target sensor module.
[0012] Meanwhile, a method for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle according to a preferred embodiment of the present invention is a method for automatically constructing training data performed in an autonomous vehicle equipped with a plurality of sensor modules, comprising: a step of storing a data labeling method associated with an object recognition method of each of the plurality of sensor modules; a step of recognizing an object outside the autonomous vehicle; a step of outputting object recognition data including object detection data labeled with object attribute data according to the recognition result in the recognition step; a step of determining a target sensor module that caused an object recognition error and a normal sensor module that did not cause an object recognition error among the plurality of sensor modules based on the object recognition data of each of the plurality of sensor modules; a step of determining replacement object recognition data for an object region corresponding to the object recognition error among the object recognition data of the normal sensor module; a step of generating replacement object detection data and replacement object attribute data for an object region corresponding to the object recognition error of the target sensor module using the replacement object recognition data; and a step of generating replacement object detection data labeled with the replacement object attribute data based on the data labeling method of the target sensor module. Effects of the invention
[0013] According to the present invention with such a configuration, training data can be automatically constructed by performing auto-labeling based on information (position, scope or scale, class, metadata, etc.) accurately recognized by other sensors regarding an object that is not recognized or misrecognized by a specific sensor.
[0014] Since a database of training data is automatically and continuously built for objects not recognized or misrecognized by camera sensors, lidar sensors, and radar sensors, the object recognition performance of camera sensors, lidar sensors, and radar sensors can be continuously improved through the training data on unrecognized and misrecognized objects.
[0015] In addition, since it does not require the labor of crowd workers to label objects that are not recognized or misrecognized by the camera sensors, lidar sensors, and radar sensors of autonomous vehicles, training data can be built quickly and at low cost. Brief explanation of the drawing
[0016] FIG. 1 is a configuration diagram of an automatic training data construction device for supplementing vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an auto-labeling method for an object not recognized by one sensor in an embodiment of the present invention. Figure 3 is a diagram illustrating the result of auto-labeling an unrecognized object of Figure 2. FIG. 4 is a diagram illustrating an auto-labeling method for an object misrecognized by one sensor in an embodiment of the present invention. Figure 5 is an internal configuration diagram of the processor shown in Figure 1. FIG. 6 is a flowchart illustrating a method for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle according to an embodiment of the present invention. Specific details for implementing the invention
[0017] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail.
[0018] However, this is not intended to limit the invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.
[0019] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0020] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.
[0021] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. In order to facilitate an overall understanding of the present invention, the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.
[0022] FIG. 1 is a configuration diagram of an automatic training data construction device for supplementing vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle according to an embodiment of the present invention.
[0023] An automatic learning data construction device for supplementing the vulnerability of an artificial intelligence object recognition function of an autonomous vehicle according to an embodiment of the present invention may include a camera sensor module (10), a LiDAR sensor module (20), a RaDAR sensor module (30), a processor (40), a storage module (50), a camera sensor learning data DB (60), a LiDAR sensor learning data DB (70), and a RaDAR sensor learning data DB (80).
[0024] The camera sensor module (10) is installed in the autonomous vehicle and can recognize external objects (e.g., roads, buildings, trees, other vehicles, pedestrians, etc.) in real time.
[0025] The camera sensor module (10) may include an object recognition data extraction module (10a).
[0026] For example, the object recognition data extraction module (10a) includes an artificial intelligence supervised learning algorithm for a camera sensor, and can extract object recognition data using the artificial intelligence supervised learning algorithm for a camera sensor.
[0027] The object recognition data extraction module (10a) can extract object recognition data (e.g., position, scope, scale, class, etc.) from the raw data of the camera sensor module (10). That is, the object recognition data extraction module (10a) can recognize an object from the raw data (e.g., 2D data) of the camera sensor module (10) through an artificial intelligence supervised learning algorithm for the camera sensor, and can extract data about the recognized object (i.e., object recognition data).
[0028] The camera sensor module (10) described above can photograph the exterior of an autonomous vehicle in real time while driving or stopped, recognize various objects in real time, and output object recognition data corresponding to the recognized objects.
[0029] For example, the camera sensor module (10) can perform AI object recognition by utilizing a bounding box object detection method and can output a predetermined object recognition data (which can be called labeling data) corresponding to the recognized object. That is, when the camera sensor module (10) performs AI object recognition by utilizing a bounding box object detection method, the camera sensor module (10) can output object recognition data including the type (or category), center position, and size information for each object (object). Of course, if necessary, the object recognition data described above may be recorded in text (txt) format in the raw data (e.g., 2D data) of the camera sensor module (10). If the camera sensor module (10) recognizes an object called a car, it can output object recognition data in a labeling data format such as "Car 400 300 120 60".
[0030] A LiDAR sensor module (20) is installed in an autonomous vehicle and can recognize external objects (e.g., road images, buildings, trees, other vehicles, pedestrians, etc.) in real time.
[0031] The LiDAR sensor module (20) may include an object recognition data extraction module (20a).
[0032] For example, the object recognition data extraction module (20a) includes an artificial intelligence supervised learning algorithm for a lidar sensor, and can extract object recognition data using the artificial intelligence supervised learning algorithm for a lidar sensor.
[0033] The object recognition data extraction module (20a) can extract object recognition data (e.g., position, scope or scale, class, etc.) from the raw data of the lidar sensor module (20). That is, the object recognition data extraction module (20a) can recognize an object from the raw data (e.g., 3D data) of the lidar sensor module (20) through an artificial intelligence supervised learning algorithm for the lidar sensor, and can extract data about the recognized object (i.e., object recognition data).
[0034] The above-described LiDAR sensor module (20) can rotate and photograph the exterior of an autonomous vehicle in real time in 360 degrees while driving or stopped to recognize various objects in real time and output object recognition data corresponding to the recognized objects.
[0035] A radar sensor module (30) is installed in an autonomous vehicle and can recognize external objects (e.g., road images, buildings, trees, other vehicles, pedestrians, etc.) in real time.
[0036] The radar sensor module (30) may include an object recognition data extraction module (30a).
[0037] For example, the object recognition data extraction module (30a) includes an artificial intelligence supervised learning algorithm for a radar sensor, and can extract object recognition data using the artificial intelligence supervised learning algorithm for a radar sensor.
[0038] The object recognition data extraction module (30a) can extract object recognition data (e.g., position, scope, scale, class, etc.) from the raw data of the radar sensor module (30). That is, the object recognition data extraction module (30a) can recognize an object from the raw data (e.g., 3D data) of the radar sensor module (30) through an artificial intelligence supervised learning algorithm for the radar sensor, and can extract data about the recognized object (i.e., object recognition data).
[0039] The radar sensor module (30) described above can photograph the exterior of an autonomous vehicle in real time while driving or stopped, recognize various objects in real time, and output object recognition data corresponding to the recognized objects.
[0040] The aforementioned camera sensor module (10), lidar sensor module (20), and radar sensor module (30) are pre-calibrated, so that the coordinate systems of each sensor module are synchronized with one another. It is believed that the method of synchronizing the coordinate systems of the camera sensor module (10), lidar sensor module (20), and radar sensor module (30) can be easily understood as a well-known technology by anyone working in the same industry.
[0041] In an embodiment of the present invention, the lidar sensor module (20) and the radar sensor module (30) can perform AI object recognition by utilizing a 3D object detection method and can output predetermined object recognition data (which may be labeled data) corresponding to the recognized object. That is, when the lidar sensor module (20) and the radar sensor module (30) perform AI object recognition by utilizing a 3D object detection method, the lidar sensor module (20) and the radar sensor module (30) can output the type (or category), location, size, and rotation information of each object (object) as object recognition data. Of course, if necessary, the object recognition data described above may be recorded in text (txt) format in the raw data (e.g., 3D data) of the lidar sensor module (20) and the radar sensor module (30). If the lidar sensor module (20) and the radar sensor module (30) recognize an object called a car, they can output object recognition data in a labeling data format such as "Car 15.41 1.84 1.47 3.64 1.65 1.67 0.01".
[0042] Of course, the camera sensor module (10), lidar sensor module (20), and radar sensor module (30) can perform AI object recognition using a semantic segmentation method and output predetermined object recognition data (which can be called labeling data) corresponding to the recognized object.
[0043] The object recognition data extraction unit (10a, 20a, 30a) described above is included in each sensor module (10, 20, 30), but if necessary, it may be included in the processor (40) described later.
[0044] In FIG. 1, the object recognition data extraction unit (10a, 20a, 30a) is shown as a separate hardware configuration, but may be implemented as a single hardware form if necessary.
[0045] The storage module (50) can store information about the learning model for each sensor (e.g., data labeling method associated with the object recognition method of each of the multiple sensor modules (10, 20, 30)). Here, the object recognition methods may include bounding box object detection, 3D object detection, semantic segmentation, etc.
[0046] Of course, the storage module (50) may also store version information of the learning model for each sensor.
[0047] Additionally, the storage module (50) can store a calibrated common coordinate system for the camera sensor module (10), the lidar sensor module (20), and the radar sensor module (30), a coordinate system conversion function between the camera sensor module (10) and the lidar sensor module (20), a coordinate system conversion function between the camera sensor module (10) and the radar sensor module (30), a coordinate system conversion function between the lidar sensor module (20) and the radar sensor module (30), etc.
[0048] If necessary, if the object recognition data extraction unit (10a, 20a, 30a) is changed to be included in the processor (40) described later, the storage module (50) can store information about the learning model for each sensor module (10, 20, 30) (e.g., an AI supervised learning algorithm model).
[0049] The processor (40) can compare the object recognition data of each of the plurality of sensor modules (10, 20, 30) with each other and determine the target sensor module that caused the object recognition error and the normal sensor module that did not cause the object recognition error according to the majority rule.
[0050] In other words, the processor (40) counts the number of first sensor modules that output first object recognition data for an object and the number of second sensor modules that output second object recognition data for the object among the plurality of sensor modules (10, 20, 30), and determines the sensor modules with a larger number among the counted number of first sensor modules and the number of second sensor modules as normal sensor modules and determines the sensor modules with a smaller number as target sensor modules. That is, the processor (40) can determine the normal sensor modules and target sensor modules by comparing and voting on the results of the objects recognized by each camera sensor module (10), lidar sensor module (20), and radar sensor module (30).
[0051] For example, assume a case where the camera sensor module (10) and the radar sensor module (30) recognize (detect) object A and output object recognition data for object A, whereas the lidar sensor module (20) does not recognize object A. In this case, the processor (40) can classify that the lidar sensor module (20) did not recognize (not detect) object A. In such an example, since the number of sensor modules that recognized object A is greater than the number of sensor modules that did not recognize object A, the processor (40) can determine the camera sensor module (10) and the radar sensor module (30) as normal sensor modules and the lidar sensor module (20) as a target sensor module according to the majority rule principle. Here, recognition is a concept that encompasses detection or identification, and may mean sensing that an object exists. Identification may mean distinguishing the type of the object. Non-detection may mean failing to determine even the existence of a specific object.
[0052] As another example, assume a case where the camera sensor module (10) and the radar sensor module (30) recognize object A as a car (classify type) and output object recognition data for object A, whereas the lidar sensor module (20) recognizes object A as a motorcycle (classify type) and outputs object recognition data for object A. In this case, the processor (40) may classify that the lidar sensor module (20) has misidentified object A. In such an example, since the number of sensor modules that recognize object A as a car is greater than the number of sensor modules that recognize object A as a motorcycle, the processor (40) may determine the camera sensor module (10) and the radar sensor module (30) as normal sensor modules and the lidar sensor module (20) as a target sensor module according to the majority rule principle. Here, misidentification may mean failing to determine the type of object or incorrectly classifying (identifying) the type of object.
[0053] As mentioned above, the failure to recognize (or detect) an object or the misrecognition of an object can be collectively referred to as an object recognition error.
[0054] The processor (40) may determine object recognition data that is different from the object recognition data of the target sensor module among the object recognition data of the normal sensor module as replacement object recognition data for the object area corresponding to the object recognition error in the target sensor module. For example, if, as a result of comparing the object recognition data of the normal sensor module and the object recognition data of the target sensor module, the object recognition data for object A is different from each other, the processor (40) may determine the object recognition data of the normal sensor module for the object A as replacement object recognition data for the object area (i.e., the area where object A is located) corresponding to the object recognition error in the target sensor module. Here, the replacement object recognition data may include replacement object detection data labeled with replacement object attribute data. Examples of cases where the object recognition data for object A is different from each other include cases where the normal sensor module has object recognition data for object A but the target sensor module does not, or cases where the object attribute data for object A in the normal sensor module and the object attribute data for object A in the target sensor module are different.
[0055] As such, since the object area corresponding to the object recognition error in the target sensor module is an area where there is an unrecognized (undetected) or misrecognized object, the processor (40) can determine which object is unrecognized (undetected) or misrecognized in each sensor module (10, 20, 30).
[0056] The processor (40) can generate replacement object detection data and replacement object attribute data for an object area corresponding to an object recognition error of the target sensor module using replacement object recognition data for an object area corresponding to an object recognition error. For example, replacement object attribute data among the replacement object recognition data can be converted (e.g., replaced) into object attribute data for an object area corresponding to an object recognition error of the target sensor module, thereby generating replacement object attribute data to be obtained. Meanwhile, replacement object detection data among the replacement object recognition data can be converted (e.g., replaced) into object detection data for an object area corresponding to an object recognition error of the target sensor module, thereby generating replacement object detection data to be obtained.
[0057] For example, in the case of non-recognition (non-detection) as exemplified in the description of the sensor modules (10, 20, 30) above, the processor (40) can generate the replacement object detection data and replacement object attribute data to be obtained by converting the replacement object attribute data and replacement object detection data of the generated replacement object recognition data into object detection data and object attribute data of the object area corresponding to the object recognition error of the target sensor module.
[0058] For example, in the case of a misrecognition (i.e., misrecognition of the type of object) as exemplified in the description of the sensor modules (10, 20, 30) above, the processor (40) can generate the desired replacement object attribute data by converting the replacement object attribute data of the generated replacement object recognition data into object attribute data of the object area corresponding to the object recognition error of the target sensor module.
[0059] And, the processor (40) can generate alternative object detection data in which alternative object attribute data is labeled based on the data labeling method of the target sensor module.
[0060] The processor (40) can auto-label according to the training data format (e.g., data labeling method) of the AI technology used by the target sensor module. For example, if the processor (40) has generated both replacement object attribute data and replacement object detection data, it can convert the generated replacement object detection data and replacement object attribute data according to the data labeling method of the target sensor module to generate replacement object detection data with the replacement object attribute data labeled. As another example, if the processor (40) has generated only the replacement object attribute data among the replacement object attribute data and replacement object detection data, it can convert the generated replacement object attribute data according to the data labeling method of the target sensor module to generate replacement object detection data with the replacement object attribute data labeled.
[0061] The processor (40) can store alternative object recognition data, which includes alternative object detection data labeled with alternative object attribute data converted to match the data labeling method of the target sensor module, as object training data in a training data DB (one of 60, 70, 80).
[0062] For example, if the target sensor module is determined to be the camera sensor module (10) and alternative object detection data with labeled alternative object attribute data for the camera sensor module (10) is generated, the processor (40) can store alternative object recognition data containing alternative object detection data with labeled alternative object attribute data converted to match the data labeling method of the camera sensor module (10) as object learning data in the camera sensor learning data DB (60).
[0063] Meanwhile, if the target sensor module is determined to be the LiDAR sensor module (20) and alternative object detection data with labeled alternative object attribute data for the LiDAR sensor module (20) is generated, the processor (40) can store alternative object recognition data containing alternative object detection data with labeled alternative object attribute data converted to the data labeling method of the LiDAR sensor module (20) as object learning data in the LiDAR sensor learning data DB (70).
[0064] Alternatively, if the target sensor module is determined to be the radar sensor module (30) and alternative object detection data with labeled alternative object attribute data for the radar sensor module (30) is generated, the processor (40) can store alternative object recognition data containing alternative object detection data with labeled alternative object attribute data converted to the data labeling method of the radar sensor module (30) as object learning data in the radar sensor learning data DB (80).
[0065] Meanwhile, if necessary, the processor (40) may additionally use object recognition data from a past point in time of the target sensor module to generate alternative object detection data and alternative object attribute data for the object area corresponding to the object recognition error of the target sensor module. At this time, the object recognition data from a past point in time of the target sensor module may be data regarding objects such as buildings or structures whose location does not change, and may have been pre-auto-labeled and stored in the storage module (50). Accordingly, the storage module (50) may store various geographical information (including the location of buildings and structures) according to the vehicle location and driving direction. In this way, the buildings or structures measured by the target sensor module at the location and driving direction of the autonomous vehicle can be identified.
[0066] That is, the processor (40) combines the replacement object recognition data of the normal sensor module at the current time and the object recognition data of the target sensor module at the past time with respect to the object area corresponding to the object recognition error. This is because the object attribute data among the replacement object recognition data of the normal sensor module at the current time and the object attribute data among the object recognition data of the target sensor module at the past time will be identical. Therefore, the processor (40) can use the replacement object attribute data of the normal sensor module at the current time as the replacement object attribute data of the object area corresponding to the object recognition error of the target sensor module. Also, since there is no change in the location of buildings and structures, the processor (40) can use the object detection data (including the location and size of the object) among the object recognition data of the target sensor module at the past time as the replacement detection data. Through such combination, the processor (40) can generate replacement object detection data and replacement object attribute data for the object area corresponding to the object recognition error of the target sensor module.
[0067] In the manner described above, the processor (40) can identify an object that is not recognized (not detected) or misrecognized in each sensor module (10, 20, 30), and each sensor module (10, 20, 30) can perform auto-labeling on the object that is not recognized or misrecognized.
[0068] The camera sensor training data DB (60) can store alternative object recognition data for objects that are not recognized (not detected) or misrecognized by the camera sensor module (10) as training data.
[0069] The lidar sensor training data DB (70) can store alternative object recognition data for objects that are not recognized (not detected) or misrecognized by the lidar sensor module (20) as training data.
[0070] The radar sensor training data DB (80) can store alternative object recognition data for objects that are not recognized (not detected) or misrecognized by the radar sensor module (30) as training data.
[0071] According to the configuration of the embodiment of the present invention as described above, a learning data DB can be automatically and continuously built for objects that are not recognized or are misrecognized by the camera sensor module (10), lidar sensor module (20), and radar sensor module (30). By doing so, through the learning data for objects that are not recognized or are misrecognized built in the learning data DB (60, 70, 80), the object recognition performance of the camera sensor module (10), lidar sensor module (20), and radar sensor module (30) in vulnerable autonomous driving scenes can be supplemented and continuously improved quickly and at low cost.
[0072] In addition, since the camera sensor module (10), lidar sensor module (20), and radar sensor module (30) do not require the labor of a crowd worker to perform labeling of unrecognized or misrecognized objects, training data can be built quickly and at low cost.
[0073] FIG. 2 is a diagram illustrating an auto-labeling method for an object not recognized by one sensor in an embodiment of the present invention, and FIG. 3 is a diagram illustrating the result of auto-labeling the object not recognized in FIG. 2.
[0074] First, the camera sensor module (10), the lidar sensor module (20), and the radar sensor module (30) are pre-calibrated, so that the coordinate systems of each sensor module are synchronized with each other.
[0075] In FIG. 2, it is assumed that the lidar sensor module (20) displays a cube for the object (24) in the raw data (22) and recognizes it as a car, the radar sensor module (30) displays a cube for the object (34) at the same coordinates as the object (24) in the raw data (32) and recognizes it as a car, and the camera sensor module (10) does not recognize (does not detect) the object at the same coordinates as the object (24, 34) in the raw data (12).
[0076] In such a case, the processor (40) determines the target sensor module that caused the object recognition error and the normal sensor module that did not cause the object recognition error based on the object recognition data of each of the multiple sensor modules (10, 20, 30). That is, since the LiDAR sensor module (20) and the radar sensor module (30) recognized the same object (24, 34) at the same location, but the camera sensor module (10) did not recognize (not detect), the processor (40) determines the LiDAR sensor module (20) and the radar sensor module (30) as normal sensor modules and determines the camera sensor module (10) as the target sensor module.
[0077] Then, the processor (40) determines the object recognition data of the corresponding object (24, 34) of the normal sensor module as replacement object recognition data for the object area corresponding to the object recognition error in the target sensor module (i.e., the area corresponding to the area where the object (24, 34) is located). According to the above example, the object recognition data of the corresponding object of the normal sensor module (20, 30) may be two, such as the object recognition data of the object (24) and the object recognition data of the object (34). In this case, since the location area of the object (24) and the location area of the object (34) will overlap by more than a predetermined standard, the processor (40) arbitrarily selects one of the sensor modules among the LiDAR sensor module (20) and the radar sensor module (30). Then, the processor (40) determines the object recognition data of the corresponding object (24 or 34) of the selected sensor module as replacement object recognition data for the object area corresponding to the object recognition error in the target sensor module.
[0078] Next, the processor (40) generates replacement object detection data and replacement object attribute data for an object area corresponding to an object recognition error of a target sensor module (e.g., camera sensor module (10)) using replacement object recognition data for an object area corresponding to an object recognition error.
[0079] Subsequently, the processor (40) performs auto-labeling on the object (14; see FIG. 3) that the camera sensor module (10) did not recognize based on the alternative object detection data and alternative object attribute data. By doing so, the processor (40) can generate labeling data (i.e., object recognition data) such as "Car 400 300 120 60" for the object (14) that the camera sensor module (10) did not recognize.
[0080] FIG. 4 is a diagram illustrating an auto-labeling method for an object misrecognized by one sensor in an embodiment of the present invention.
[0081] First, the camera sensor module (10), the lidar sensor module (20), and the radar sensor module (30) are pre-calibrated, so that the coordinate systems of each sensor module are synchronized with each other.
[0082] In FIG. 4, it is assumed that the LiDAR sensor module (20) displays a cube for the object (24) in the raw data (22) and recognizes it as a car, the radar sensor module (30) displays a cube for the object (34) at the same coordinates as the object (24) in the raw data (32) and recognizes it as a car, and the camera sensor module (10) displays a rectangle for the object (16) at the same coordinates as the objects (24, 34) in the raw data (12) and misidentifies it as a motorcycle. Here, misidentification may mean that the existence of the object (16) was properly detected, but the type (attribute) of the object (16) was incorrectly identified.
[0083] In such a case, the processor (40) determines the target sensor module that caused the object recognition error and the normal sensor module that did not cause the object recognition error based on the object recognition data of each of the multiple sensor modules (10, 20, 30). That is, the LiDAR sensor module (20) and the radar sensor module (30) recognized the same object (24, 34) at the same location, but the camera sensor module (10) recognized it differently (i.e., misrecognized it). Therefore, the processor (40) determines the LiDAR sensor module (20) and the radar sensor module (30) as normal sensor modules and determines the camera sensor module (10) as the target sensor module.
[0084] Then, the processor (40) determines the object recognition data of the corresponding object (24, 34) of the normal sensor module as replacement object recognition data for the object area corresponding to the object recognition error in the target sensor module (i.e., the area corresponding to the area where the object (24, 34) is located). According to the above example, the object recognition data of the corresponding object of the normal sensor module (20, 30) may be two, such as the object recognition data of the object (24) and the object recognition data of the object (34). In this case, since the location area of the object (24) and the location area of the object (34) will overlap by more than a predetermined standard, the processor (40) arbitrarily selects one of the sensor modules among the LiDAR sensor module (20) and the radar sensor module (30). Then, the processor (40) determines the object recognition data of the corresponding object (24 or 34) of the selected sensor module as replacement object recognition data for the object area corresponding to the object recognition error in the target sensor module. According to FIG. 4, among the object recognition data of the target sensor module (i.e., camera sensor module (10)), the object attribute data (including the type of object) is different from the object attribute data of the object recognition data of the normal sensor module (2, 30). Based on this difference, the processor (40) determines (converts) the object attribute data of the object recognition data of the selected normal sensor module into alternative object attribute data of the object area (16) corresponding to the object recognition error of the target sensor module. Since the object detection data of the object area (16) corresponding to the object recognition error of the target sensor module is normal, the object detection data of the object area (16) corresponding to the object recognition error of the target sensor module is used as is as alternative detection data.
[0085] Subsequently, the processor (40) performs auto-labeling on the object (16) misidentified by the camera sensor module (10) based on alternative object detection data and alternative object attribute data. By doing so, the processor (40) can generate labeling data (i.e., object recognition data) such as "Car 400 300 120 60" for the object (16) misidentified by the camera sensor module (10).
[0086] Figure 5 is an internal configuration diagram of the processor (40) shown in Figure 1.
[0087] The processor (40) may include an object recognition result comparison module (40a) and an object learning data generation module (40b).
[0088] The object recognition result comparison module (40a) receives object recognition data (including location, range or size, type, etc.) from each sensor module (10, 20, 30). Here, the object recognition data from the camera sensor module (10) may be referred to as the first object recognition data, the object recognition data from the lidar sensor module (20) as the second object recognition data, and the object recognition data from the radar sensor module (30) as the third object recognition data.
[0089] The object recognition result comparison module (40a) can compare the results of objects recognized by the camera sensor module (10), the lidar sensor module (20), and the radar sensor module (30) (i.e., first object recognition data, second object recognition data, third object recognition data) with each other.
[0090] The object recognition result comparison module (40a) can determine the target sensor module that caused the object recognition error (e.g., non-recognition or misrecognition) and the normal sensor module that did not cause the object recognition error based on the comparison result and the majority rule.
[0091] The object learning data generation module (40b) can determine object recognition data that is different from the object recognition data of the target sensor module among the object recognition data of the normal sensor module as object recognition data for a replacement object area corresponding to an object recognition error in the target sensor module.
[0092] The object learning data generation module (40b) can generate alternative object detection data and alternative object attribute data for the object area corresponding to the object recognition error of the target sensor module using alternative object recognition data for the object area corresponding to the object recognition error.
[0093] In addition, the object learning data generation module (40b) can generate alternative object detection data with labeled alternative object attribute data by converting alternative object attribute data and alternative object detection data according to the data labeling method of the target sensor module. That is, the object learning data generation module (40b) can de-identify and annotate according to the training data format of the AI technology (e.g., Bounding Box Object Detection, 3D Object Detection, Semantic Segmentation, etc.) used by the camera sensor module (10), lidar sensor module (20), and radar sensor module (30).
[0094] And, the object learning data generation module (40b) can store the alternative object recognition data, which includes alternative object detection data labeled with alternative object attribute data converted to match the data labeling method of the target sensor module, as object learning data in the corresponding learning data DB (one of 60, 70, 80).
[0095] FIG. 6 is a flowchart illustrating a method for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle according to an embodiment of the present invention.
[0096] First, a data labeling method associated with the object recognition method of each of the multiple sensor modules (camera sensor module (10), lidar sensor module (20), and radar sensor module (30)) is stored in a storage module (50) (S10).
[0097] Subsequently, when the autonomous vehicle is driving or stopped, the camera sensor module (10), the lidar sensor module (20), and the radar sensor module (30) recognize external objects (e.g., roads, buildings, trees, other vehicles, pedestrians, etc.) of the autonomous vehicle in real time (S20). At this time, the camera sensor module (10), the lidar sensor module (20), and the radar sensor module (30) are pre-calibrated so that the coordinate systems of each sensor module are synchronized with each other.
[0098] Accordingly, the camera sensor module (10), the lidar sensor module (20), and the radar sensor module (30) each output real-time object recognition data (e.g., position, scope or scale, class, etc.) (S30).
[0099] Next, the processor (40) compares the object recognition data of each of the plurality of sensor modules (10, 20, 30) with each other and determines the target sensor module that caused the object recognition error and the normal sensor module that did not cause the object recognition error according to the majority rule (S40).
[0100] For example, if the camera sensor module (10) and the radar sensor module (30) recognize (detect) object A, and the lidar sensor module (20) does not recognize object A, the number of sensor modules that recognized object A is greater than the number of sensor modules that did not recognize object A. Accordingly, the processor (40) determines the camera sensor module (10) and the radar sensor module (30) as normal sensor modules and the lidar sensor module (20) as target sensor modules according to the majority rule principle.
[0101] As another example, if the camera sensor module (10) and the radar sensor module (30) recognize object A as a car (type discrimination) and the lidar sensor module (20) recognize object A as a motorcycle (type discrimination), the number of sensor modules that recognize object A as a car is greater than the number of sensor modules that recognize object A as a motorcycle. Accordingly, the processor (40) determines the camera sensor module (10) and the radar sensor module (30) as normal sensor modules and the lidar sensor module (20) as target sensor modules according to the majority rule principle.
[0102] Then, the processor (40) determines, among the object recognition data of the normal sensor module, object recognition data that is different from the object recognition data of the target sensor module as replacement object recognition data for the object area corresponding to the object recognition error in the target sensor module (S50). For example, it is assumed that the object recognition data for object A of the normal sensor module and the object recognition data for object A of the target sensor module are different from each other. In this case, the processor (40) determines the object recognition data for object A of the normal sensor module as replacement object recognition data for the object area (i.e., the area where object A is located) corresponding to the object recognition error in the target sensor module.
[0103] Afterwards, the processor (40) generates replacement object detection data and replacement object attribute data for the object area corresponding to the object recognition error of the target sensor module using replacement object recognition data for the object area corresponding to the object recognition error (S60).
[0104] Then, the processor (40) generates alternative object detection data with labeled alternative object attribute data based on the data labeling method of the target sensor module (S70). That is, the processor (40) generates alternative object detection data with labeled alternative object attribute data by converting the generated alternative object detection data and alternative object attribute data to the data labeling method of the target sensor module.
[0105] Next, the processor (40) stores the alternative object recognition data, which includes alternative object detection data labeled with alternative object attribute data converted according to the data labeling method of the target sensor module, as object training data in the training data DB (one of 60, 70, 80).
[0106] Furthermore, the method for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle according to the present invention described above can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Additionally, the computer-readable recording medium may be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. Furthermore, the functional program, code, and code segments for implementing the above method can be easily inferred by programmers skilled in the art to which the present invention belongs.
[0107] The foregoing description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols
[0108] 10: Camera sensor module 20: LiDAR sensor module 30: Radar sensor module 40: Processor 50: Storage Module 60: Camera Sensor Training Data DB 70: LiDAR sensor training data DB 80: Radar sensor training data DB
Claims
Claim 1 A device for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle, comprising: a plurality of sensor modules that recognize external objects of an autonomous vehicle and output object recognition data including object detection data labeled with object attribute data according to the recognition result; a storage module configured to store a data labeling method associated with an object recognition method of each of the plurality of sensor modules; and a processor that determines a target sensor module that caused an object recognition error and a normal sensor module that did not cause an object recognition error among the plurality of sensor modules based on the object recognition data of each of the plurality of sensor modules, determines substitute object recognition data for an object area corresponding to the object recognition error among the object recognition data of the normal sensor module, generates substitute object detection data and substitute object attribute data for an object area corresponding to the object recognition error of the target sensor module using the substitute object recognition data, and generates substitute object detection data labeled with the substitute object attribute data by converting one or more of the substitute object detection data and substitute object attribute data to match the data labeling method of the target sensor module. Claim 2 A device for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle, wherein, in claim 1, the plurality of sensor modules includes a camera sensor module, a lidar sensor module, and a radar sensor module, and the coordinate systems of the camera sensor module, lidar sensor module, and radar sensor module are synchronized with each other. Claim 3 A device for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle, wherein, in claim 1, the object attribute data includes the type of object, the object detection data includes the location and size of the object, and the processor compares the object recognition data of each of the plurality of sensor modules with each other and determines the target sensor module and the normal sensor module according to the majority rule principle. Claim 4 In claim 1, the processor counts the number of first sensor modules that output first object recognition data for a first object among the plurality of sensor modules and the number of second sensor modules that output second object recognition data for the first object, respectively, and determines the larger number of sensor modules among the counted number of first sensor modules and the number of second sensor modules as normal sensor modules and determines the smaller number of sensor modules as target sensor modules, thereby forming an automatic training data construction device for supplementing vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle. Claim 5 A device for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle, wherein, in claim 1, the processor determines object recognition data that is different from the object recognition data of the target sensor module among the object recognition data of the normal sensor module as object recognition data for a substitute object region corresponding to the object recognition error. Claim 6 delete Claim 7 In claim 1, the processor generates alternative object detection data and alternative object attribute data for an object region corresponding to an object recognition error of the target sensor module by additionally utilizing object recognition data from a past point in time of the target sensor module, an automatic training data construction device for supplementing vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle. Claim 8 In claim 7, the processor combines object recognition data for replacing a normal sensor module at a current time point and object recognition data of the target sensor module at a past time point to generate replacement object detection data and replacement object attribute data for an object region corresponding to an object recognition error of the target sensor module, thereby forming an automatic training data construction device for supplementing vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle. Claim 9 A method for automatically constructing training data for supplementing vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle, wherein the method comprises: a step of storing a data labeling method associated with an object recognition method of each of the plurality of sensor modules; a step of recognizing an object outside the autonomous vehicle; a step of outputting object recognition data including object detection data labeled with object attribute data according to the recognition result in the recognition step; a step of determining a target sensor module that caused an object recognition error and a normal sensor module that did not cause an object recognition error among the plurality of sensor modules based on the object recognition data of each of the plurality of sensor modules; a step of determining substitute object recognition data for an object region corresponding to the object recognition error among the object recognition data of the normal sensor module; a step of generating substitute object detection data and substitute object attribute data for an object region corresponding to the object recognition error of the target sensor module using the substitute object recognition data; and a step of generating substitute object detection data labeled with the substitute object attribute data by converting one or more of the substitute object detection data and substitute object attribute data to match the data labeling method of the target sensor module. Claim 10 In claim 9, the object attribute data includes the type of object, the object detection data includes the location and size of the object, and the step of determining the target sensor module and the normal sensor module involves determining the target sensor module and the normal sensor module according to the majority rule by comparing the object recognition data of each of the plurality of sensor modules. A method for automatically constructing training data to compensate for vulnerabilities in the artificial intelligence object recognition function of an autonomous vehicle.
Citation Information
Patent Citations
Cross-validating sensors of an autonomous vehicle
US9221396B1
Electronic device and control method thereof
KR1020200084949A
Deep learning system and method using automatic input of correct answer data
KR1020210061092A
Labelling method
KR1020220077082A
Method and system for ground truth auto labeling advanced sensor data and image by camera
KR102264152B1
