Automatic labeling method and automatic labeling device

Abstract

In this automatic labeling method, position information and identification information relating to an object in dictionary data are acquired by a computer (S202). A point cloud of measurement points measured by a sensor in an area (110) that includes the position indicated by the acquired position information is acquired (S203). The object indicated by the acquired identification information is labeled for the point cloud on the basis of the number of measurement points corresponding to the position information relating to the object out of the acquired point cloud (S205-S209). When the number of measurement points of the acquired point cloud is greater than a predetermined threshold value (YES in S207) and a measurement range of the measurement points by the sensor includes the area (YES in S211), the acquired identification information may be labeled as a correct answer for the point cloud by the computer.

Description

Automatic labeling method and automatic labeling device

[0001] The present invention relates to an automatic labeling method and an automatic labeling device.

[0002] Patent Literature 1 describes a system and method for automatic image labeling. The technology in Patent Literature 1 labels images of roadside objects acquired by a vehicle camera based on the position and orientation of the vehicle and position information of the roadside objects.

[0003] Special Publication No. 2022-514891

[0004] The technology of Patent Document 1 does not take into consideration the possibility of measuring roadside objects using a camera. With this technology, if there is an obstruction between the vehicle and the roadside object that prevents measurement of the roadside object, erroneous automatic labeling is likely to occur for input data measuring the obstruction, with the roadside object being identified as the correct answer rather than the obstruction. Incorrect automatic labeling may result in a decrease in learning performance through machine learning. The present invention aims to prevent erroneous automatic labeling due to the presence of an obstruction that prevents measurement of the object to be labeled.

[0005] In order to solve the above-mentioned problems, an automatic labeling method and an automatic labeling device according to one aspect of the present invention use a computer to acquire position information and identification information of an object from pre-stored dictionary data. A point cloud of measurement points measured by a sensor in an area including the position of the acquired position information is acquired. Based on the number of measurement points in the acquired point cloud that correspond to the position information of the object, the acquired identification information is labeled to the point cloud.

[0006] According to the present invention, it is possible to prevent erroneous automatic labeling from being performed.

[0007] FIG. 1 is a configuration diagram of an automatic labeling device according to a first embodiment. FIG. 2 is a diagram of a vehicle's travel location. FIG. 3 is a diagram of a measurement range measured by a vehicle sensor. FIG. 4 is a diagram of an example of a processing procedure of the automatic labeling device. FIG. 5 is a diagram of an area set by a measurement point acquisition unit. FIG. 6 is a diagram of an example of a processing procedure of the automatic labeling device. FIG. 7 is a diagram of a measurement range measured by a vehicle sensor. FIG. 8 is a diagram of a measurement range measured by a vehicle sensor. FIG. 9 is a diagram of a measurement range measured by a vehicle sensor. FIG. 10 is a diagram of a measurement range measured by a vehicle sensor.

[0008] [First Embodiment] The automatic labeling device 100 of the first embodiment shown in FIG. 1 is mounted on a vehicle 200 shown in FIG. 2. The automatic labeling device 100 performs annotation by associating input data acquired by measuring the periphery of the vehicle 200 using a sensor (not shown) of the vehicle 200 with, for example, an object in dictionary data defining objects actually present around the vehicle 200. If the input data satisfies certain conditions, the automatic labeling device 100 labels the object associated with the input data as a correct answer. The object labeled by the automatic labeling device 100 can be used, in pairs with the input data, as a data set for machine learning of an object detection and recognition algorithm included in an object detection device (not shown) of the vehicle 200. In the example of FIG. 3, a crosswalk 310 and a pedestrian traffic light 320 at an intersection 300 are present within the measurement range of the sensor. For example, if there is an obstruction, such as a pillar, in front of the pedestrian traffic light 320 that blocks measurement of the pedestrian traffic light 320, the sensor will not correctly measure the pedestrian traffic light 320 but will instead measure the obstruction in front of the pedestrian traffic light 320. If the input data at this time is labeled as "pedestrian traffic light" defined in the dictionary data of the sensor's measurement range as the correct answer, an inappropriate dataset will be created. Machine learning using this dataset may result in a decrease in the learning performance of the object detection device. The automatic labeling device 100 of this embodiment shown in FIG. 1 prevents incorrect labeling of the input data. The automatic labeling device 100 includes, for example, a general-purpose microcontroller as a computer. The microcontroller includes a central processing unit (CPU) and memory. The memory includes read-only memory (ROM) and random access memory (RAM). The microcontroller can virtually construct multiple information processing circuits by having the CPU execute programs stored in the memory. The plurality of information processing circuits can constitute each of the units 101 to 107 (described later) of the automatic labeling device 100. In this embodiment, an example is shown in which the plurality of information processing circuits built in a microcontroller are realized by software.An information processing circuit may be configured by providing dedicated hardware for executing the information processing of each of the units 101 to 107 described below. Multiple information processing circuits may be configured as separate pieces of hardware. The dedicated hardware may include devices such as application specific integrated circuits (ASICs) and conventional circuit components arranged to execute the functions of each of the units 101 to 107.

[0009] The input data acquisition unit 101 acquires input data based on sensor data measured by a sensor of the vehicle 200. The input data includes position information of a target measured by the sensor. The position information of the target may be position coordinates in an absolute coordinate system or position coordinates in a relative coordinate system. The relative coordinate system may be, for example, a vehicle coordinate system with the position of the sensor of the vehicle 200 as its origin. The sensor outputs sensor data related to the target measured by the sensor at each measurement time point. The measurement time point may be specified by one or more times, or may be specified by a period. The sensor data can be used as input data either directly or after being preprocessed. The preprocessing may be, for example, a process of extracting feature points from the sensor data. The process of extracting feature points may be, for example, a filter process for edge detection. The sensor data may be data that can be used to machine-train an object detection and recognition algorithm so that the object detection device can output a correct answer for the input data. For example, a camera, LiDAR (Light Detection and Ranging), or laser radar can be used as the sensor. The camera may be, for example, a digital camera having an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), or a time-of-flight (ToF) camera. For example, if the sensor is a camera, the sensor data is image data captured by the camera from the vehicle 200 and output. The image data includes information on the position and pixel value of each pixel in the image. The input data acquisition unit 101 acquires the image data as input data, either directly or after processing. For example, if the sensor is a LiDAR or laser radar, the sensor data is, for example, a point cloud. The point cloud is also called point cloud data. The point cloud contains three-dimensional position information of targets. This position information can be acquired, for example, from distance information obtained by the LiDAR or laser radar and position information of the vehicle 200. The position information is obtained, for example, by a Global Navigation Satellite System (GNSS) sensor of the vehicle 200.The input data acquisition unit 101 can acquire the point cloud output by the LiDAR or laser radar as input data. The acquired input data becomes point cloud data of measurement points where the target is measured. In the above example, the measurement points are each pixel of the image data or each point of the point cloud.

[0010] The dictionary data acquisition unit 102 acquires dictionary data of objects corresponding to the input data acquired by the input data acquisition unit 101. The object corresponding to the input data may be, for example, an object that was present in the vicinity of the vehicle 200 equipped with a sensor at the time of measurement of the sensor data that is the source of the input data. The position of the object can be identified, for example, by position information of the vehicle 200 obtained by a GNSS sensor at the time of measurement of the sensor data. The dictionary data includes position information and identification information of objects on the ground. The objects on the ground include at least stationary objects. The position information of the objects on the ground may be, for example, position coordinates in an absolute coordinate system such as map coordinates, or position coordinates in a relative coordinate system based on a point such as a sensor on the vehicle 200. The position information of the objects on the ground includes position information of one or more locations on the object. When the position information of the object on the ground includes multiple locations, the set of each location has an outer shape that approximates the outline of the object as seen from the vehicle 200. The identification information of the objects on the ground may be, for example, information for distinguishing stationary objects, such as the crosswalk 310 and pedestrian traffic light 320 illustrated in FIG. 3 . The identification information of the object on the ground may include the name of a facility or building. The dictionary data acquisition unit 102 may, for example, acquire dictionary data of the object corresponding to the input data by extracting it from high-precision three-dimensional map data (HD map, High Definition MAP). The high-precision three-dimensional map data includes position coordinates in an absolute coordinate system and identification information of features existing on the ground. The crosswalk 310 and pedestrian traffic light 320 illustrated in FIG. 3 are examples of features. The feature may include a facility or a building. For example, the dictionary data acquisition unit 102 identifies the position of the vehicle 200 at the time of measurement of the sensor data corresponding to the input data using a GNSS sensor. The dictionary data acquisition unit 102 can extract position information and identification information of features around the identified position from the high-precision three-dimensional map data and acquire it as dictionary data of the object corresponding to the input data. For example, the dictionary data acquisition unit 102 may acquire the dictionary data of the object from dictionary data generated when the vehicle 200 previously traveled a predetermined route. The dictionary data generated in the past includes, for example, the position coordinates and identification information of objects present around the vehicle 200 on a predetermined travel route.The object position information can be calculated, for example, using the distance of the target from the sensor measured by the sensor and the position of the vehicle 200 on the travel route at the time of measurement of the target. The distance of the target from the sensor can be acquired, for example, by a method described in the description of the measurement point acquisition unit 104 below. The position of the vehicle 200 on the travel route can be identified by a GNSS sensor. The object identification information can be, for example, the content of the target recognized by the object detection device of the vehicle 200 based on input data acquired from sensor data of the target measured by the sensor of the vehicle 200 on the travel route. Dictionary data generated when the vehicle 200 previously traveled a predetermined travel route can be used to acquire dictionary data of objects corresponding to input data when the vehicle 200 is on the same travel route. The dictionary data acquisition unit 102 can acquire dictionary data of objects corresponding to input data acquired by the input data acquisition unit 101 when the vehicle 200 is traveling at a specific position from dictionary data generated when the vehicle 200 previously traveled at a specific position. The dictionary data from which the dictionary data corresponding to the input data is acquired may be stored, for example, in a database unit (not shown). The database unit may be present in the vehicle 200 or may be present outside the vehicle 200. The dictionary data acquisition unit 102 can acquire dictionary data from a database unit present outside the vehicle 200, for example, by wireless communication.

[0011] The coordinate conversion unit 103 performs coordinate conversion so that the position coordinates of the input data acquired by the input data acquisition unit 101 and the position coordinates of the dictionary data acquired by the dictionary data acquisition unit 102 are in the same coordinate system. The dictionary data corresponding to the input data can ultimately be registered as a dataset for machine learning in pair with the input data. The pair of input data and dictionary data needs to be expressed in position coordinates in the same coordinate system. For example, an absolute coordinate system or a vehicle coordinate system is used for the position coordinates of the dataset. The coordinate system converted by the coordinate conversion unit 103 does not necessarily have to be the coordinate system used in the dataset, but may be any coordinate system in which the measurement point acquisition unit 104 (described later) can process the input data and dictionary data. For example, if the sensor data corresponding to the input data is image data, the coordinate conversion unit 103 may convert the position coordinates of the dictionary data into the coordinate system of the image data to match the input data. For example, if the position coordinates of the input data and the dictionary data are originally in the same coordinate system, the coordinate conversion unit 103 does not need to perform coordinate conversion.

[0012] The measurement point acquisition unit 104 acquires, from the measurement points of the input data acquired by the input data acquisition unit 101, measurement points corresponding to the position information of objects in the dictionary data acquired by the dictionary data acquisition unit 102, and counts the number of acquired measurement points. The dictionary data contains position information of objects present in the vicinity of the vehicle 200 at the time of measurement of the sensor data corresponding to the input data acquired by the input data acquisition unit 101. The sensor data corresponding to the input data does not necessarily measure the same object as the object in the dictionary data. For example, if an obstruction exists between the sensor and an object in the vicinity of the vehicle 200, the sensor may measure the obstruction that blocks the object. The measurement point acquisition unit 104 acquires measurement points corresponding to objects in the dictionary data from the input data, and can identify, from among the measurement points, the measurement point at which the sensor measured the position of the object in the dictionary data. When identifying this measurement point, the measurement point acquisition unit 104 sets an area including the position of the object in the dictionary data in the input data. The position information of the object in the dictionary data includes position information of one or more locations of the object. For example, the measurement point acquisition unit 104 can set an area to a range that includes a set of location information in the dictionary data where adjacent locations have the same identification information. FIG. 3 illustrates areas 110 and 120 that the measurement point acquisition unit 104 can set when the vehicle 200 passes through an intersection 300. Area 110 includes the location of a crosswalk 310 in the dictionary data, and area 120 includes the location of a pedestrian traffic light 320 in the dictionary data. For example, if the location information for the crosswalk 310 in the dictionary data includes location information for multiple locations, the measurement point acquisition unit 104 can set, in the input data, area 110, which is a polygonal frame that corresponds to the perimeter of the multiple locations indicated by the location information. For example, if the location information for the crosswalk 310 in the dictionary data includes location information for only one location, the measurement point acquisition unit 104 can set, in the input data, area 110, which is a circle centered on the location indicated by the location information. The measurement point acquisition unit 104 can also set an area 120 for the pedestrian traffic light 320 in the input data based on the position information of the pedestrian traffic light 320 in the dictionary data, similar to the area 110 for the crosswalk 310. As in the example described above, the measurement point acquisition unit 104 can set an area including the position of the object in the dictionary data to a range of the minimum size that surrounds the object.The measurement point acquisition unit 104 acquires, from among the measurement points of the input data acquired by the input data acquisition unit 101, measurement points at which a target at an object's position within an area is measured as measurement points corresponding to the object's position information in the dictionary data. Among the measurement points of the input data at which a target at an object's position within an area is measured, the position of the target measured by the sensor matches the position of the object indicated by the position information in the dictionary data. A measurement point at which the position of the target in the input data matches the position of the object in the dictionary data is considered to be a measurement point at which the sensor correctly measured the object in the dictionary data. This measurement point can be identified, for example, using the distance from the sensor to the target in the input data and the distance from the sensor to the object in the dictionary data. Both distances may be distances on a common coordinate system. The common coordinate system may be, for example, a two-dimensional bird's-eye view plane coordinate system in which the vehicle is viewed from above, or a three-dimensional spatial coordinate system including height information. For example, if the sensor is a camera, the input data does not include position information of the target for each pixel in the image. The position information of the target can be identified, for example, based on the distance from the camera to the target. The measurement point acquisition unit 104 can calculate the distance from the camera to the target using, for example, image data acquired by the input data acquisition unit 101. For example, if the camera is a stereo camera, the distance from the camera to the target can be calculated from parallax information obtained from images captured by the stereo camera. For example, if the camera is a monocular camera, the distance from the camera to the target can be estimated from the focus state of the edge of the target in the image captured by the monocular camera. The distance from the camera to the target can be calculated using depth information obtained using, for example, motion graphics creation / editing software. The distance from the camera to the target can be calculated for each pixel, for example. For example, if the sensor is a LiDAR or laser radar, the point cloud output as sensor data includes position information of the target. The measurement point acquisition unit 104 can calculate the distance from the LiDAR or laser radar sensor to the target using, for example, the position information of the target in the point cloud acquired by the input data acquisition unit 101.When the position information of the point cloud is position coordinates in an absolute coordinate system, the measurement point acquisition unit 104 uses the position coordinates of the vehicle 200 measured by the GNSS sensor at the time of measurement of the target to calculate the distance from the LiDAR or laser radar to the target.

[0013] The threshold setting unit 105 sets a predetermined threshold used to determine whether or not a certain condition is satisfied for labeling the identification information of an object in the dictionary data as a correct answer for the input data, based on the number of measurement points counted by the measurement point acquisition unit 104. The predetermined threshold can be set based on a reference value determined from the number of measurement points counted by the measurement point acquisition unit 104. The number of measurement points used to determine the reference value may be, for example, the number of measurement points counted by the measurement point acquisition unit 104 for input data based on sensor data measured by a sensor of the vehicle 200 when the vehicle 200 traveled in the past. The threshold setting unit 105 may determine the reference value from the number of measurement points counted by the measurement point acquisition unit 104 at multiple points in time during a single travel of the vehicle 200, or may determine the reference value from the number of measurement points counted by the measurement point acquisition unit 104 during multiple travels of the vehicle 200. Even if there is an obstruction blocking a target corresponding to an object in the dictionary data, for example, the relative positional relationship between the vehicle 200 and the obstruction with respect to the sensor may change over time, potentially resulting in a scene in which the target is not obstructed by the obstruction. It is considered that, at least in such a scene, the object in the dictionary data is correctly measured by the sensor, and the number of measurement points counted by the measurement point acquisition unit 104 increases. The threshold setting unit 105 may, for example, use the maximum number of measurement points counted multiple times by the input data acquisition unit 101 as a reference value. For example, when the maximum number of measurement points counted multiple times by the measurement point acquisition unit 104 is used as a reference value, the threshold setting unit 105 may set a predetermined threshold based on the determined reference value so that the top number of measurement points is extracted using the predetermined threshold. The top number of measurement points extracted using the predetermined threshold may, for example, be measurement points that are 80% or 90% or more of the reference value, or measurement points that are distributed in the 90% range of the distribution of the multiple measurement points. The number of measurement points counted by the measurement point acquisition unit 104 varies depending on the distance between the sensor and the target. The number of measurement points of a target by the sensor increases as the target is closer to the sensor, and decreases as the target is farther away. When the target is closer to the sensor, the probability that an obstacle exists between the sensor and the target is lower than when the target is farther away from the sensor.When the probability of an obstruction between the sensor and the target decreases, the number of measurement points measuring the target at the object position in the dictionary data in the input data area is less likely to be reduced due to the presence of an obstruction. The threshold setting unit 105 may set a threshold, separate from the predetermined threshold, for example, based on the number of measurement points counted by the measurement point acquisition unit 104 to determine whether to exclude dictionary data corresponding to the input data from the label determination target to be assigned to the identification information of the dictionary data. The threshold for determining whether to exclude dictionary data from the label determination target based on the number of measurement points counted by the measurement point acquisition unit 104 represents a lower limit for the number of measurement points. For example, when setting a threshold representing a lower limit for the number of measurement points, the threshold setting unit 105 may vary the value of the threshold depending on the size of the area set in the input data by the measurement point acquisition unit 104. For example, the number of pixels in the portion of the camera image data in which the target is captured is greater when the distance between the camera and the target is closer than when the distance between the camera and the target is farther. Taking this into consideration, the threshold setting unit 105 may, for example, set the threshold representing the lower limit condition for the number of measurement points to a higher value when the area set by the measurement point acquisition unit 104 is larger than when the area is smaller. The threshold setting unit 105 may, for example, further calculate the maximum number of measurement points counted by the measurement point acquisition unit 104 at multiple points in time for one object in the dictionary data for other objects having the same identification information in the dictionary data. In this case, the threshold setting unit 105 may use the average or mode calculated from the multiple maximum values ​​of the number of measurement points calculated by the measurement point acquisition unit 104 for multiple objects having the same identification information as the reference value. By setting the threshold based on this reference value, the threshold representing the lower limit condition for the number of measurement points can be set in consideration of the distribution of the multiple maximum values ​​of the number of measurement points.

[0014] The predetermined threshold set by the threshold setting unit 105 does not have to be a threshold for measurement points that measure targets at object positions within an area and are counted by the measurement point acquisition unit 104 among the measurement points of the input data acquired by the input data acquisition unit 101. For example, the threshold setting unit 105 may set, as the predetermined threshold, a threshold for the ratio of the number of measurement points counted by the measurement point acquisition unit 104 to the number of measurement points within an area among the measurement points of the input data acquired by the input data acquisition unit 101. This ratio is unlikely to fluctuate due to the influence of the distance between the camera and the target. The threshold setting unit 105 may set the threshold for the ratio of the number of measurement points to a constant value regardless of the distance between the camera and the target.

[0015] The labeling unit 106 determines whether to label the input data with the identification information of the object in the dictionary data, depending on whether the number of measurement points counted by the measurement point acquisition unit 104 is greater than a predetermined threshold for the number of measurement points set by the threshold setting unit 105. By labeling the input data with the identification information, the input data and the identification information may be associated and stored in a storage device. Furthermore, the set of the associated input data and the identification information stored in the storage device may be read from the storage device and used to identify an object in another scene. The labeling unit 106 may determine whether to label the identification information of the object in the dictionary data as the correct answer for the input data, depending on whether the proportion of the above-mentioned number of measurement points is greater than a predetermined threshold set by the threshold setting unit 105. If the number of measurement points or the proportion of the measurement points is greater than the predetermined threshold, the labeling unit 106 labels the identification information of the object in the dictionary data as the correct answer. The labeling unit 106 does not label the object identification information of the dictionary data as a correct answer when the number of measurement points or the ratio of the measurement points is equal to or less than a predetermined threshold. The labeling unit 106 may label the object identification information of the dictionary data that is not labeled as a correct answer as an incorrect answer for the input data. The data registration unit 107 registers pairs of the dictionary data and the input data after labeling by the labeling unit 106 in a dataset used for machine learning of an object detection and recognition algorithm of the object detection device of the vehicle 200. The input data registered in the dataset may be limited to pairs of dictionary data and input data in which the object identification information of the dictionary data is labeled as a correct answer. The input data registered in the dataset may include pairs of dictionary data and input data that are not labeled as a correct answer. The pairs of input data and dictionary data that are not labeled as a correct answer may include pairs of dictionary data and input data that are labeled as an incorrect answer. The registered dataset may be stored in, for example, a memory of a microcontroller. The memory that stores the dataset can be configured by a storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0016] The microcontroller of the automatic labeling device 100 can implement the automatic labeling method according to one embodiment of the present invention by executing the processing procedure shown in Figure 4. The microcontroller acquires input data based on sensor data obtained by measuring a target at time t using the input data acquisition unit 101 (step S201). The microcontroller acquires the vehicle's position at time t (step S202). The vehicle's position at time t may be, for example, the position of the vehicle 200 at time t detected by the GNSS sensor of the vehicle 200. The microcontroller acquires dictionary data of an object corresponding to the input data using the dictionary data acquisition unit 102 (step S203). The microcontroller performs coordinate conversion to convert the position coordinates of the input data and the dictionary data into the same coordinate system (step S204). The microcontroller sets an area corresponding to the object's position information in the dictionary data to the input data using the measurement point acquisition unit 104, and acquires measurement points of the target corresponding to the object's position information in the set area. The microcontroller counts the number of acquired measurement points using the measurement point acquisition unit 104 (step S205). The microcontroller sets a predetermined threshold based on the number of measurement points counted by the measurement point acquisition unit 104 using the threshold setting unit 105 (step S206). The microcontroller checks whether the number of measurement points counted by the measurement point acquisition unit 104 is greater than the predetermined threshold using the labeling unit 106 (step S207). If the number of measurement points is greater than the predetermined threshold (YES in step S207), the microcontroller labels the object identification information in the dictionary data as correct using the labeling unit 106 (step S208). If the number of measurement points is equal to or less than the predetermined threshold (NO in step S207), the microcontroller does not label the object identification information in the dictionary data as correct using the labeling unit 106 (step S209). The microcontroller causes the data registration unit 107 to register pairs of the dictionary data and the input data after labeling by the labeling unit 106 in the dataset (step S210). After registering the pairs of the dictionary data and the input data in the dataset, the microcontroller ends the process.

[0017] Among the point cloud of measurement points of the target measured by the sensor in the area of ​​the input data, the number of measurement points where the target position matches the position of the object in the dictionary data is an index of the number of measurement points where the sensor measured the object of the position information acquired by the dictionary data acquisition unit 102 without being obstructed by an obstructing object. This number of measurement points is considered to be a numerical value representing the possibility of measuring the object by the sensor. By verifying the number of measurement points of the input data acquired and counted by the measurement point acquisition unit 104, the possibility of measuring the object in the dictionary data by the sensor can be evaluated. By determining whether or not the labeling unit 106 should label the identification information of the object in the dictionary data as correct based on the number of measurement points of the input data, it is possible to prevent erroneous automatic labeling due to the presence of an obstructing object that obstructs the measurement of the object to be labeled.

[0018] The reference value calculated by the threshold setting unit 105 from the number of measurement points counted by the measurement point acquisition unit 104 for multiple times in an area corresponding to an object with the same location information and identification information is a value that reflects the distribution of the number of measurement points at which the sensor correctly measured the same object multiple times. By setting a predetermined threshold based on this reference value, the threshold setting unit 105 can label the input data with the identification information of the object in the dictionary data, taking into account the circumstances under which the sensor previously measured the same object.

[0019] [First Modification of First Embodiment] In the automatic labeling device 100 of the first embodiment, the measurement point acquisition unit 104 sets the area of ​​the input data including the object position in the dictionary data to a range of the minimum size surrounding the object based on the object position information in the dictionary data. In the automatic labeling device 100 according to the first modification of the first embodiment, the measurement point acquisition unit 104 sets the area of ​​the input data including the object position in the dictionary data to a range including the periphery of the object. This area is larger than the range including the set of object position information in the dictionary data for adjacent locations with the same identification information. FIG. 5 shows the area set by the measurement point acquisition unit 104 on a two-dimensional bird's-eye view coordinate system. The measurement point acquisition unit 104 sets the area 130 of the input data including the position of the pedestrian crossing 310 to a range that is wider in all directions than the area 110 set by the measurement point acquisition unit 104 in the first embodiment. The area 130 is measured by a sensor within a measurement range 140 of the sensor of the vehicle 200. In this modification, the measurement point acquisition unit 104 acquires and counts, as measurement points corresponding to the position information of the object in the dictionary data, measurement points in the area 130 of the input data. The threshold setting unit 105 determines a reference value based on the number of measurement points counted by the measurement point acquisition unit 104 and sets a predetermined threshold based on the determined reference value. The labeling unit 106 determines whether to label the identification information of the object in the dictionary data as a correct answer for the input data based on whether the number of measurement points is greater than the predetermined threshold. The identification information of the object in the dictionary data to be determined is not identification information corresponding to all targets in the area 130. The area 130 includes the positions of both the object in the dictionary data and other objects existing around it. Only the identification information of the object in the dictionary data needs to be labeled as a correct answer for the input data in the area 130; the identification information of other objects existing around the object does not need to be labeled as a correct answer for the input data in the area 130. The measurement points of the input data in the area excluding area 110 from area 130 may be used to check for the presence or absence of an obstruction by comparing the distance from the sensor to the target with the distance from the sensor to the object.In a first modified example of the first embodiment, the microcontroller of the automatic labeling device 100 expands the area 130 set in the input data in step S205 in the processing procedure of Fig. 4 to a range that includes the object in the dictionary data and its surrounding area. In this modified example, the microcontroller changes the processing of step S205 as described above and executes the processing procedure of Fig. 4, thereby implementing the automatic labeling method according to the first modified example of the first embodiment.

[0020] In this variant, the measurement point acquisition unit 104 sets an area 130 in a range wider than the object in the dictionary data, thereby acquiring all measurement points where the sensor measured the position of the object from the input data, and reliably confirming that the sensor is measuring the object.

[0021] [Second Modification of First Embodiment] In the automatic labeling device 100 of the first embodiment, if the number of measurement points counted by the measurement point acquisition unit 104 is greater than a predetermined threshold set by the threshold setting unit 105, the object identification information of the dictionary data is labeled as the correct answer for the input data. In the automatic labeling device 100 according to the second modification of the first embodiment, the condition for determining whether to label the object identification information of the dictionary data as the correct answer includes whether the sensor measurement range 140 includes the area 110 containing the object position of the dictionary data. In this modification, if the number of measurement points is greater than the predetermined threshold (YES) in step S207 of FIG. 4, the microcontroller of the automatic labeling device 100 additionally performs the procedure of step S211 shown in FIG. 6 before step S208. In step S211, the microcontroller, using the labeling unit 106, checks whether the sensor measurement range 140 includes the area 110 containing the object position of the dictionary data and in which the measurement point acquisition unit 104 counted the number of measurement points. If the sensor measurement range 140 includes the area 110 containing the object position (YES in step S211), the process proceeds to step S208 in FIG. 4 . If the sensor measurement range 140 does not include the area 110 containing the object position (NO in step S211), the process proceeds to step S209 in FIG. 4 . In this modification, the microcontroller executes a processing procedure in which step S211 shown in FIG. 6 is added between steps S207 and S208 in FIG. 4 , thereby implementing an automatic labeling method according to a second modification of the first embodiment. In this modification, if the number of measurement points counted by the measurement point acquisition unit 104 is greater than a predetermined threshold and the sensor measurement range 140 includes the area 110 containing the object position, the object identification information in the dictionary data is labeled as correct. The sensor measurement range 140 refers to the range within which the sensor can theoretically measure a target. The measurement range 140 is, for example, the inside of the angle of view of a camera or the inside of the scanning range of the laser light of a LiDAR or laser radar. The outside of each is the non-measurement range of the sensor. Fig. 7 shows a state in which the measurement range 140 of the sensor includes the entire area 110. Fig. 8 shows a state in which the measurement range 140 of the sensor does not include part of the area 110.The sensor's non-measurement range also includes a blind spot due to an obstruction, preventing camera photography or scanning with LiDAR or laser radar laser light, and thus preventing the sensor from measuring in principle. Figure 9 shows a state in which the sensor's measurement range 140 does not include the area 110 due to obstruction by an obstruction 150 present within the sensor's measurement range 140. In the state of Figure 7, the sensor can measure the area 110 without being obstructed by the obstruction 150. In the states of Figures 8 and 9, the sensor cannot measure part or all of the area 110. In a second modification of the first embodiment, the labeling unit 106 checks whether the sensor's measurement range 140 includes the area 110. This check can be performed, for example, based on the position information in the dictionary data of an object located within the area 110 and the position information of the vehicle 200 obtained by the GNSS sensor. Alternatively, whether the sensor's measurement range 140 includes the area 110 can be determined based on whether the sensor has measured a target located in front of the object located within the area 110. When the measurement range 140 includes the area 110, it is considered that the sensor measured the target in a state where it could measure the entire portion including the position of the object in the dictionary data. Furthermore, when the number of measurement points among the point cloud of measurement points of the target measured by the sensor where the target position matches the position of the object in the dictionary data is greater than a predetermined threshold, it is considered that the sensor measured the object in the dictionary data with a certain degree of measurability or higher. For example, when the sensor's measurement range 140 changes whether or not it includes the area 110, the number of measurement points is more likely to be greater than the predetermined threshold when the measurement range 140 includes the area 110 than when it does not include part or all of the area 110. When the measurement range 140 includes the area 110 and the number of measurement points is greater than the predetermined threshold, the labeling unit 106 labels the identification information of the object in the dictionary data as correct.

[0022] In this modification, it is possible to confirm that the number of measurement points at which the sensor measures targets at positions that include the object position in the dictionary data is greater than a predetermined threshold value, while the sensor measures the entire area 110 that includes the object position. In this modification, it is possible to more reliably confirm that the sensor is measuring the object.

[0023] In this modification, when the sensor does not measure an obstructing object 150 in front of the object in the dictionary data, it is considered that the sensor is not blocked by the obstructing object 150 and is measuring a target at a position that includes the object in the dictionary data in the set area 110. When the sensor measures an obstructing object 150 in front of the object in the dictionary data, it is considered that the sensor is blocked by the obstructing object 150 and is not measuring at least a part of the target at a position that includes the object in the dictionary data in the set area 110. In this modification, it is possible to determine whether or not the measurement range 140 of the sensor includes the area 110 with a smaller amount of calculation than determining by actual calculation.

[0024] [Third Modification of First Embodiment] In the automatic labeling device 100 of the first embodiment, when the number of measurement points counted by the measurement point acquisition unit 104 is greater than a predetermined threshold set by the threshold setting unit 105, the object identification information in the dictionary data is labeled as a correct answer for the input data. In the automatic labeling device 100 according to the third modification of the first embodiment, the criteria for determining whether to label the object identification information in the dictionary data as a correct answer include whether the measurement range 140 of the sensor includes two or more representative points generated on the periphery of the area 110. For example, if the area 110 is rectangular, the two or more representative points may be set at each of the four corners of the area 110, or may be set at two points located at both ends of one of the long sides of the four corners. The two or more representative points may be set at two points located diagonally at the four corners of the rectangular area 110, or may be set at the remaining three points excluding one of the four corners. In this modification, as in the second modification, if the number of measurement points is greater than a predetermined threshold (YES) in step S207 of FIG. 4 , the microcontroller of the automatic labeling device 100 additionally performs step S211 shown in FIG. 6 before step S208. In step S211, the microcontroller determines whether the sensor's measurement range 140 includes the area 110 containing the object's position by determining whether the sensor's measurement range 140 includes two or more representative points generated on the periphery of the area 110. The representative points of the area 110 can be generated, for example, by the labeling unit 106. In this modification, the microcontroller can implement the automatic labeling method according to the third modification of the first embodiment by executing the processing procedure, which adds the above-described procedure of step S211 shown in FIG. 6 , between steps S207 and S208 of FIG. 4 . FIG. 10 illustrates an example in which the labeling unit 106 generates multiple representative points 160 on the periphery of the area 110. The representative points 160 may be, for example, points randomly generated on the periphery of the area 110, grid points generated by uniform sampling that include the corners of the area 110, or points generated by uniformly sampling the periphery of the area. If the measurement range 140 of the sensor includes all of the representative points 160 of the area 110, it is considered that the measurement range 140 of the sensor includes the area 110.If the sensor's coverage area 140 does not include a representative point 160 of at least a portion of the area 110 , the sensor's coverage area 140 is considered to not include at least a portion of the area 110 .

[0025] Whether or not the sensor measurement range 140 includes two or more representative points 160 generated on the periphery of the area 110 is considered to be an indicator of whether or not the sensor is measuring an obstructing object in front of the object in the dictionary data. Whether or not the sensor measurement range 140 includes two or more representative points 160 of the area 110 can be determined.

[0026] Second Embodiment Next, an automatic labeling device according to a second embodiment of the present invention will be described. The automatic labeling device according to the second embodiment has a hardware configuration similar to that of the automatic labeling device 100 according to the first embodiment shown in FIG. 1. The automatic labeling device according to the second embodiment differs from the automatic labeling device 100 according to the first embodiment in the processing performed by the measurement point acquisition unit 104. In the automatic labeling device according to the second embodiment, the measurement point acquisition unit 104 corrects the number of measurement points of the input data counted in accordance with position information in an area corresponding to an object in the dictionary data, in accordance with the distance from the sensor to the object. The following description of the automatic labeling device according to the second embodiment will focus on differences from the automatic labeling device 100 according to the first embodiment. Description of the parts of the automatic labeling device according to the second embodiment that are common to the automatic labeling device 100 according to the first embodiment will be omitted to avoid duplication. The number of measurement points counted by the measurement point acquisition unit 104 varies depending on the distance between the sensor and the object. Taking this into consideration, in the automatic labeling device of the second embodiment, the number of measurement points counted by the measurement point acquisition unit 104 is corrected according to the distance from the sensor to the object. A correction coefficient k may be used to correct the number of measurement points according to the distance from the sensor to the object. The correction coefficient k may be determined, for example, by the measurement point acquisition unit 104. The correction coefficient k can be determined, for example, as follows: The distance from the sensor to the object is divided into a maximum of n sections, with each section being a division distance r, and the number of measurement points previously counted by the measurement point acquisition unit 104 for one object is sorted into sections of the corresponding distance range according to the distance from the sensor to the object. The maximum distance D from the sensor to the object can be defined as D = r × n. The number of measurement points previously counted by the measurement point acquisition unit 104 may be for multiple input data based on sensor data measured at different times during a single travel of the vehicle 200. The number of measurement points previously counted by the measurement point acquisition unit 104 may be for multiple input data based on sensor data measured during multiple travels of the vehicle 200. For example, the maximum number of measurement points sorted into the category with the shortest distance range divided by the maximum number of measurement points sorted into each category other than the shortest distance range may be used as the correction coefficient k corresponding to the distance range of each category.The calculated correction coefficient k is a larger magnification when the distance from the sensor to the object is greater than when the distance from the sensor to the object is short. The threshold setting unit 105 determines a reference value based on the corrected number of measurement points and sets a predetermined threshold based on the determined reference value. The labeling unit 106 determines whether to label the object identification information in the dictionary data as the correct answer for the input data based on whether the corrected number of measurement points is greater than the predetermined threshold. In the second embodiment, the microcontroller corrects the number of measurement points counted by the measurement point acquisition unit 104 in step S205 of the processing procedure of FIG. 4 according to the distance from the sensor to the object. In the automatic labeling device of the second embodiment, the microcontroller executes the processing procedure of FIG. 4 by changing the processing of step S205 of FIG. 4 as described above, thereby implementing the automatic labeling method of the second embodiment.

[0027] When the distance from the sensor to the object is short, the number of measurement points of the sensor that measures the position of the object in the dictionary data is greater than when the distance from the sensor to the object is long. In the automatic labeling device of the second embodiment, when the distance from the sensor to the object is long, the number of measurement points is corrected by a larger magnification than when the distance from the sensor to the object is short. This correction makes it easy to determine the label for the input data by comparing the number of measurement points of the input data with a common threshold value, which is a predetermined threshold value, while taking into account the distance from the sensor to the object.

[0028] Third Embodiment Next, an automatic labeling device according to a third embodiment of the present invention will be described. The automatic labeling device according to the third embodiment has a hardware configuration similar to that of the automatic labeling device 100 according to the first embodiment shown in FIG. 1 . The automatic labeling device according to the third embodiment differs from the automatic labeling device 100 according to the first embodiment in the content of the processing performed by the threshold setting unit 105. In the automatic labeling device according to the third embodiment, the threshold setting unit 105 sets a predetermined threshold based on the number of measurement points counted by the measurement point acquisition unit 104. The threshold is set to a larger value when the distance from the sensor to the object is closer than when the distance from the sensor to the object is farther. The following description of the automatic labeling device according to the third embodiment will focus on differences from the automatic labeling device 100 according to the first embodiment. Description of the parts of the automatic labeling device according to the third embodiment that are common to the automatic labeling device 100 according to the first embodiment will be omitted to avoid duplication. When the distance from the sensor to the object is short, the number of measurement points of the sensor measuring the position of the object in the dictionary data is greater than when the distance from the sensor to the object is long. Taking this into consideration, in the automatic labeling device 100 of the third embodiment, the threshold setting unit 105 sets a predetermined threshold that is larger when the distance from the sensor to the object is short than when the distance from the sensor to the object is long. The threshold setting unit 105 can set a predetermined threshold that is larger when the distance from the sensor to the object is short than when the distance from the sensor to the object, for example, as follows: The number of measurement points counted by the measurement point acquisition unit 104 is sorted into distance range categories according to the distance from the sensor to the object. For the classification categories, for example, up to n distance range categories, each category being the division distance r, can be used, as used when determining the correction coefficient k for the number of measurement points counted by the measurement point acquisition unit 104 in the second embodiment. The threshold setting unit 105 can set a predetermined threshold for each category, for example, based on the number of measurement points sorted into each category, in the same way as the threshold setting unit 105 of the automatic labeling device 100 of the first embodiment sets a predetermined threshold for each category of the distance range related to the distance from the sensor to the object, by the threshold setting unit 105 in step S206 of the processing procedure of FIG.The threshold setting unit 105 sets a predetermined threshold for each category based on the number of measurement points counted by the measurement point acquisition unit 104 and the distance from the sensor to the object. The distance from the sensor to the object can be calculated in the same manner as the distance from the camera to the target or the distance from the LiDAR or laser radar to the target described in the first embodiment. In the automatic labeling device of the third embodiment, the microcontroller changes the processing of step S206 in Figure 4 as described above and executes the processing procedure of Figure 4, thereby implementing the automatic labeling method of the third embodiment.

[0029] When the distance from the sensor to the object is short, the number of measurement points of the sensor that measures the position of the object in the dictionary data is greater than when the distance from the sensor to the object is long. In the automatic labeling device of the third embodiment, the closer the distance from the sensor to the object, the greater the predetermined threshold value set than when the distance from the sensor to the object is long. By setting the predetermined threshold to a different value depending on the distance from the sensor to the object, the label determination for the input data can be more appropriately performed by comparing the predetermined threshold corresponding to the distance from the sensor to the object with the number of measurement points of the input data.

[0030] In this embodiment, the automatic labeling device 100 is mounted on the vehicle 200, which makes it possible to prevent erroneous automatic labeling from being performed on input data based on sensor data measuring objects present around the vehicle 200. Note that, although the automatic labeling device 100 is mounted on the vehicle 200 in this embodiment, the automatic labeling method may be performed using the automatic labeling device 100 without mounting it on the vehicle 200. The above-described embodiment is an example of the present invention. Therefore, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made depending on the design, etc., even in embodiments other than the above-described embodiment, as long as they do not deviate from the technical concept of the present invention.

[0031] For example, the measurement point acquisition unit 104 may set an area corresponding to a position including the object and its surroundings based on the position information of the object in the dictionary data acquired by the dictionary data acquisition unit 102. When the sensor measures an obstruction 150 that prevents measurement of the object of the position information acquired by the dictionary data acquisition unit 102, it may be determined that the measurement range 140 of the sensor does not include at least a part of the area 110.

[0032] 100 Automatic labeling device, 101 Input data acquisition unit, 102 Dictionary data acquisition unit, 104 Measurement point acquisition unit, 105 Threshold setting unit, 106 Labeling unit, 110 to 130 Area, 140 Measurement range, 150 Obstruction, 160 Representative point, 200 Vehicle, 310 Crosswalk (object), 320 Pedestrian traffic light (object)

Claims

1. By computer, Obtaining position information and identification information of an object from pre-stored dictionary data; Acquire a point cloud of measurement points measured by a sensor in an area including the position of the acquired position information; The acquired identification information is labeled for the point cloud based on the number of measurement points corresponding to the position information of the object among the acquired point cloud. Automatic labeling method.

2. 2. The automatic labeling method according to claim 1, wherein the computer labels the identification information acquired for the point cloud as correct if the number of measurement points of the acquired point cloud is greater than a predetermined threshold and the measurement range of the measurement points by the sensor includes the area.

3. 3. The automatic labeling method according to claim 2, wherein the computer determines that the measurement range includes the area when the sensor does not measure an obstruction that prevents measurement of the object of the acquired position information, and when it determines that the measurement range includes the area, labels the acquired identification information to the point cloud based on the number of measurement points in the acquired point cloud that correspond to the position information of the object.

4. 4. The automatic labeling method according to claim 2 or 3, wherein the computer generates two or more representative points on the periphery of the area, determines whether the measurement range includes the area depending on whether the measurement range includes the representative points, and if it is determined that the measurement range includes the area, labels the acquired identification information to the point cloud based on the number of measurement points in the acquired point cloud that correspond to the positional information of the object.

5. The automatic labeling method according to claim 2 or 3, wherein the computer sets the predetermined threshold value based on a reference value calculated from the number of measurement points measured by the sensor in the past in an area set each time the computer acquires the position information and identification information of the object with the same content.

6. The automatic labeling method according to claim 2 or 3, wherein the computer compares the number of measurement points of the acquired point cloud with the predetermined threshold value, which is larger when the distance from the sensor to the object is closer than when the distance is farther.

7. 4. The automatic labeling method according to claim 2 or 3, wherein the computer corrects the number of measurement points of the acquired point cloud by a larger magnification when the distance from the sensor to the object is greater than when the distance is short, and compares the corrected number of measurement points of the point cloud with the predetermined threshold value.

8. an input data acquisition unit that acquires input data based on sensor data measured by a sensor; a dictionary data acquisition unit that acquires, from pre-stored dictionary data, position information and identification information of an object present in the vicinity of the sensor at the time of measurement of the sensor data; and a labeling unit that labels the acquired identification information to the point cloud based on the number of measurement points corresponding to the position information of the object among the point cloud of measurement points measured by the sensor in an area including the position of the object's position information in the input data.