Data labeling method, device, electronic device and storage medium

Abstract

The embodiments of the present application disclose a data labeling method, device, electronic device, and storage medium. The method includes: identifying foreground data points corresponding to a target object and background data points unrelated to the target object from environmental point cloud data collected by a radar sensor; projecting the foreground data points and background data points onto the imaging plane of a visual sensor, respectively, to obtain a first projection result corresponding to the foreground data point and a second projection result corresponding to the background data point; determining the non-truncation rate of the target object based on the first projection result, and determining the non-occlusion rate of the target object based on the first projection result and the second projection result; and performing visibility labeling on the target object based on the non-truncation rate and non-occlusion rate of the target object. Implementation of the embodiments of the present application enables rapid and accurate visibility labeling of the collected obstacle data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and specifically to a data annotation method, device, electronic device and storage medium. Background Art

[0002] Three-dimensional (3D) visual perception is a key sensory capability in the field of autonomous driving. With the continuous optimization of autonomous driving processing model performance, the demand for 3D data annotation is increasing. To meet the rapidly growing demand for data annotation, the current mainstream annotation method uses lidar detection results as pre-annotation results, and then manually checks and adjusts the pre-annotation results. However, since the installation position of lidar generally cannot coincide with the installation position of the camera, it is easy for the lidar's perception range to be inconsistent with the camera's field of view, which leads to obstacle visibility issues.

[0003] To solve the visibility problem of obstacles, previous methods often use rules to define visibility standards. However, in practice, it is found that defining visibility standards by rules often leads to problems such as inaccurate visibility judgment and low labeling efficiency. Summary of the Invention

[0004] The embodiments of the present application disclose a data labeling method, device, electronic device and storage medium, which can quickly and accurately label the visibility of collected obstacle data.

[0005] The present application discloses a data annotation method, which includes:

[0006] Identifying foreground data points corresponding to a target object and background data points unrelated to the target object from environmental point cloud data collected by the radar sensor;

[0007] Projecting the foreground data points and the background data points onto an imaging plane of a visual sensor, respectively, to obtain a first projection result corresponding to the foreground data point and a second projection result corresponding to the background data point;

[0008] determining a non-truncation rate of the target object according to the first projection result, and determining a non-occlusion rate of the target object according to the first projection result and the second projection result;

[0009] The visibility of the target object is marked according to the non-truncation rate and the non-occlusion rate of the target object.

[0010] In one embodiment, determining the visibility of the target object according to the truncation rate and the occlusion rate of the target object includes:

[0011] Calculating a visibility score of the target object according to the product of a non-truncation rate and a non-occlusion rate of the target object;

[0012] If the visibility score of the target object is greater than or equal to a preset threshold, the visibility of the target object is marked as true.

[0013] In one embodiment, determining the non-occlusion rate of the target object according to the first projection result and the second projection result includes:

[0014] constructing a first foreground depth map of the target object according to the image coordinates of each foreground data point included in the first projection result on the imaging plane and the depth value of each foreground data point corresponding to the target object;

[0015] performing a resolution compression operation on the first foreground depth map to obtain a second foreground depth map;

[0016] constructing a first background depth map according to the image coordinates of each background data point on the imaging plane and the depth value of each background data point included in the second projection result;

[0017] performing a resolution compression operation on the first background depth map to obtain a second background depth map;

[0018] An occlusion rate of the target object is calculated according to the second foreground depth map and the second background depth map, and a non-occlusion rate of the target object is calculated according to the occlusion rate of the target object.

[0019] In one embodiment, performing a resolution compression operation on the first foreground depth map to obtain a second foreground depth map includes:

[0020] determining a two-dimensional bounding box corresponding to the target object according to the image coordinates of each foreground data point in the first foreground depth map;

[0021] Calculating a scaling coefficient corresponding to a resolution compression operation according to a preset number of rows and columns of the low-resolution depth map and a height and width of the two-dimensional bounding box;

[0022] A resolution compression operation is performed on the first foreground depth map according to the scale transformation coefficient to obtain a second foreground depth map.

[0023] In one embodiment, there are at least two data points in the first foreground depth map that are mapped to the same first data point in the second foreground depth map after a resolution compression operation; the depth value of the first data point in the second foreground depth map is the minimum value of the depth values ​​of the at least two data points in the first foreground depth map.

[0024] In one embodiment, calculating the occlusion rate of the target object according to the second foreground depth map and the second background depth map includes:

[0025] Comparing the depth values ​​corresponding to the foreground data points and the background data points with the same image coordinates in the second foreground depth map and the second background depth map, respectively, and counting the number of target data points in the second foreground depth map whose depth value difference exceeds a depth threshold;

[0026] The ratio between the number of target data points and the total number of foreground data points is determined as the occlusion rate of the target object.

[0027] In one embodiment, determining the non-truncation rate of the target object according to the first projection result includes:

[0028] Counting the number of foreground data points projected beyond the viewing angle of the visual sensor according to the first projection result;

[0029] determining a ratio between the number of foreground data points projected outside the viewing angle range and the total number of foreground data points as the truncation rate of the target object;

[0030] A non-truncation rate of the target object is determined according to the truncation rate of the target object.

[0031] The present application discloses a data annotation device, including:

[0032] An identification module is used to identify foreground data points corresponding to a target object and background data points unrelated to the target object from the environmental point cloud data collected by the radar sensor;

[0033] A projection module, configured to project the foreground data points and the background data points onto an imaging plane of a visual sensor, respectively, to obtain a first projection result corresponding to the foreground data point and a second projection result corresponding to the background data point;

[0034] a first determining module, configured to determine a non-truncation rate of the target object according to the first projection result, and to determine a non-occlusion rate of the target object according to the first projection result and the second projection result;

[0035] The second determining module is configured to perform visibility marking on the target object according to the non-truncation rate and the non-occlusion rate of the target object.

[0036] An embodiment of the present application discloses an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements any one of the data labeling methods disclosed in the embodiment of the present application.

[0037] An embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, any one of the data labeling methods disclosed in the embodiment of the present application is implemented.

[0038] Compared with the related art, the embodiments of the present application have the following beneficial effects:

[0039] After the radar sensor collects environmental point cloud data, it identifies and projects the environmental point cloud data to obtain a first projection result corresponding to the target object and a second projection result unrelated to the target object. The target object's non-truncation rate can be calculated based on the first projection result, and then the target object's non-occlusion rate can be calculated based on the second projection result. The target object's visibility can be calculated using the non-truncation rate and non-occlusion rate, reducing the need for complex rule-based design. By simply calculating the target object's non-truncation rate and non-occlusion rate, the target object can be quickly and accurately labeled for visibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 This is an example diagram of an application scenario of a data annotation method disclosed in an embodiment of the present application;

[0042] Figure 2 This is a flowchart of a data annotation method disclosed in an embodiment;

[0043] Figure 3 is a flowchart of another data annotation method disclosed in an embodiment;

[0044] Figure 4 This is an example diagram of a projection result of environmental point cloud data projected onto an imaging plane disclosed in an embodiment;

[0045] Figure 5 This is a schematic structural diagram of a data tagging device disclosed in an embodiment;

[0046] Figure 6The present invention is a schematic structural diagram of an electronic device disclosed in an embodiment. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] It should be noted that the terms "including," "having," and any variations thereof in the embodiments and drawings of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or apparatus.

[0049] The present application discloses a data annotation method, device, electronic device, and storage medium, which can quickly and accurately annotate the visibility of collected obstacle data. Detailed descriptions are provided below.

[0050] See also Figure 1 , Figure 1 This is an example diagram of collecting environmental point cloud data disclosed in an embodiment. Figure 1 As shown, the vehicle 10 may be provided with a radar sensor and a visual sensor.

[0051] Radar sensors may include laser radar, millimeter-wave radar, ultrasonic radar, etc., without specific limitation. Visual sensors may include visible light cameras, fisheye cameras, surround-view cameras, etc., without specific limitation.

[0052] The radar sensor can continuously collect environmental point cloud data of the environment in which the vehicle 10 is located. Figure 1 As shown, the environmental point cloud data collected by the radar sensor may include point cloud data corresponding to the vehicle 20, point cloud data corresponding to the tree 30, point cloud data corresponding to the building 40, etc., but is not limited thereto.

[0053] Depending on the purpose of labeling, different objects in the environment of the vehicle 10 can be selected as target objects corresponding to the visibility labeling. For example, if the obstacle that blocks the vehicle 10 from traveling needs to be labeled, the vehicle 20 can be selected as the target object.

[0054] Based on Figure 1 See the application scenario shown in Figure 2 , Figure 2 This is a flow chart of a method for data annotation disclosed in an embodiment. The method can be applied to various electronic devices, such as vehicle-mounted computers, electronic control units (ECUs), or computing devices such as personal computers, without limitation. Figure 2 As shown, the method may include the following steps:

[0055] 210. Identify foreground data points corresponding to the target object and background data points unrelated to the target object from the environmental point cloud data collected by the radar sensor.

[0056] Electronic devices can process the environmental point cloud data collected by radar sensors using target detection methods such as statistical classification recognition, fuzzy classification recognition, neural network classification recognition, and support vector machine classification recognition to identify foreground data points corresponding to target objects. In other words, they can identify foreground data points belonging to target objects.

[0057] After identifying the foreground data points, the electronic device may identify the remaining data points in the collected environmental point cloud data except for the foreground data points as background data points unrelated to the target object.

[0058] In some embodiments, in order to reduce the amount of calculation, the electronic device may also perform sparse processing on the remaining data points in the environmental point cloud data except for the foreground data points, eliminate some data points from the remaining data points according to rules, and identify the retained data points as background data points that are not related to the target object.

[0059] 220. Project the foreground data points and the background data points onto an imaging plane of a visual sensor to obtain a first projection result corresponding to the foreground data points and a second projection result corresponding to the background data points.

[0060] The size of a vision sensor's imaging plane is related to its field of view (FOV), which refers to the range covered by the sensor's lens, or the viewing angle. Therefore, when projecting foreground and background data points onto the vision sensor's imaging plane, data points within the FOV are projected onto the plane, while data points outside the FOV are generally not.

[0061] Therefore, the first projection result obtained after projecting the foreground data points may include one or more of the following parameters: the number of foreground data points that fall within the viewing angle range of the visual sensor, the image coordinates of each foreground data point that falls within the viewing angle range, and the number of foreground data points that are outside the viewing angle range of the visual sensor.

[0062] Accordingly, the second projection result obtained after projecting the background data points may include one or more of the following parameters: the number of background data points falling within the viewing angle of the visual sensor, the image coordinates of each background data point falling within the viewing angle, and the number of background data points outside the viewing angle of the visual sensor.

[0063] 230. Determine a non-truncation rate of the target object according to the first projection result, and determine a non-occlusion rate of the target object according to the first projection result and the second projection result.

[0064] The non-truncation rate of the target object can be used to indicate the probability that the foreground data points corresponding to the target object fall within the viewing angle range of the visual sensor. The more foreground data points fall within the viewing angle range, the higher the non-truncation rate of the target object.

[0065] In some embodiments, the first projection result may include the number of foreground data points falling within the viewing angle range, and the electronic device may directly calculate the non-truncation rate of the target object based on the number of foreground data points falling within the viewing angle range included in the first projection result and the total number of foreground data points.

[0066] In other embodiments, the electronic device may include the number of foreground data points that fall outside the viewing angle range. The electronic device may first calculate the target object's truncation rate based on the number of foreground data points that fall outside the viewing angle range and the total number of foreground data points, and then calculate the target object's non-truncation rate based on the target object's truncation rate. The fewer the number of foreground data points that fall outside the viewing angle range, the lower the target object's truncation rate and the higher the target object's non-truncation rate.

[0067] The number of foreground data points that fall outside the visual range is relatively small. Calculating the truncation rate first and then calculating the non-truncation rate based on the truncation rate can reduce the amount of calculation when counting the number of foreground data points, which is conducive to improving the calculation speed.

[0068] For example, the truncation rate of the target object can be calculated by the following formula:

[0069]

[0070] Among them, crop_ratio can represent the truncation rate of the target object, I crop It can represent the foreground data points projected outside the viewing angle range, I all It can represent the foreground data points belonging to the target object.

[0071] Therefore, the non-crop ratio of the target object can be expressed as (1-crop_ratio).

[0072] Furthermore, the target object's non-occlusion rate can be used to indicate the probability that a foreground data point corresponding to the target object is not occluded by background data points unrelated to the target object. Occlusion of a foreground data point by a background data point can mean that the physical distance between the foreground and background data points is too small for a vehicle to pass between the two points.

[0073] In some embodiments, the electronic device may first count the number of target data points in the foreground data points that are occluded by the background data points based on the image coordinates of each foreground data point included in the first projection result and the image coordinates of each background data point included in the second projection result, and then calculate the occlusion rate of the target object based on the number of target data points and the total number of foreground data points, and further calculate the non-occlusion rate of the target object based on the occlusion rate of the target object.

[0074] Optionally, if a foreground data point has the same image coordinates as another background data point, the foreground data point may be counted as an obscured target data point.

[0075] In other embodiments, the electronic device may also count the number of target data points in the foreground data points that are occluded by the background data points based on the image coordinates of each foreground data point included in the first projection result, the image coordinates of each background data point included in the second projection result, and the depth values ​​of each foreground data point and each background data point to calculate the occlusion rate of the target object, thereby calculating the non-occlusion rate of the target object based on the occlusion rate of the target object.

[0076] It should be noted that the depth values ​​of each foreground data point and each background data point may be measured by a radar sensor and included in the environmental point cloud data.

[0077] 240. The visibility of the target object is marked according to the non-truncation rate and non-occlusion rate of the target object.

[0078] In the embodiment of the present application, the visibility of the target object can be positively correlated with the non-truncation rate and non-occlusion rate of the target object. The higher the non-truncation rate of the target object, the higher the visibility; the higher the non-occlusion rate of the target object, the higher the visibility.

[0079] In some embodiments, the electronic device may set thresholds corresponding to the non-truncation rate and the non-occlusion rate, respectively, and mark the visibility of the target object according to the thresholds corresponding to the non-truncation rate and the non-occlusion rate, respectively.

[0080] In yet other embodiments, the electronic device may also calculate a visibility score for the target object based on the non-truncation rate and non-occlusion rate of the target object, and then label the target object as visible based on the visibility score. A preset threshold corresponding to the visibility score may be pre-set. If the calculated visibility score is greater than or equal to the preset threshold, the visibility of the target object may be labeled as true; otherwise, if the calculated visibility score is less than the preset threshold, the visibility of the target object may be labeled as false.

[0081] For example, the visibility annotation of the target object can be expressed with reference to the following formula:

[0082]

[0083] Among them, visibility can represent the visibility of the target object, (1-crop_ratio) can represent the non-truncation rate of the target object, (1-occ ratio ) may represent the non-occlusion rate of the target object, and Trheshold may represent a preset threshold.

[0084] That is, in the embodiments of the present application, the visibility of a target object can be automatically assessed based on its non-occlusion rate and non-truncation rate, enabling rapid and accurate visibility annotation of collected obstacle data. Furthermore, the visibility of a target object can be defined as the product of the non-occlusion rate and the non-truncation rate, thereby converting the binary classification problem of whether the target object is visible into a regression problem based on the product data. This allows machine learning, deep learning, and other artificial intelligence models to more flexibly perform subsequent data processing operations on the visibility of the target object.

[0085] Because radar sensors may have certain blind spots, objects closer to the radar sensor have sparse data points in the point cloud environment data, which can easily lead to misjudgments of occlusion relationships. In some possible embodiments, the electronic device can use pseudo-point cloud methods such as voxelization to complete the sparse environmental point cloud data. The completed environmental point cloud data is then processed using any of the aforementioned data annotation methods to reduce misjudgments of occlusion relationships, improve the accuracy of calculating the occlusion and non-occlusion rates of target objects, and thus improve the accuracy of visibility annotation.

[0086] In other possible embodiments, electronic devices can also perform low-resolution mapping on point cloud data, allowing sparse point cloud data to effectively express occlusion relationships. Compared to voxelized pseudo-point cloud filling methods, low-resolution mapping methods have lower computational cost and faster operation speed, and can also avoid misjudgment of occlusion relationships caused by voxel filling errors.

[0087] See also Figure 3 , Figure 3This is a flow chart of another data annotation method disclosed in an embodiment, which can be applied to the aforementioned electronic device. Figure 3 As shown, the data annotation method may include the following steps:

[0088] 310. Identify foreground data points corresponding to the target object and background data points unrelated to the target object from the environmental point cloud data collected by the radar sensor.

[0089] 320. Project the foreground data points and the background data points onto an imaging plane of a visual sensor to obtain a first projection result corresponding to the foreground data points and a second projection result corresponding to the background data points.

[0090] 330. Determine a non-truncation rate of the target object according to the first projection result.

[0091] The specific implementation of steps 310 to 330 can be found in the above embodiments, and will not be described in detail below.

[0092] 340 . Construct a first foreground depth map of the target object according to the image coordinates of each foreground data point included in the first projection result on the imaging plane and the depth values ​​of each foreground data point corresponding to the target object.

[0093] For example, the first foreground depth map may be represented as follows:

[0094]

[0095] Among them, depth map It can represent the first foreground depth map, u i and v i It can represent the image coordinates of the i-th foreground data point on the imaging plane, d i may represent the depth value of the i-th foreground data point, and n may represent the total number of foreground data points.

[0096] 350. Perform a resolution compression operation on the first foreground depth map to obtain a second foreground depth map.

[0097] In an embodiment of the present application, the image size of the compressed low-resolution depth map can be preset, that is, the number of rows and columns of the low-resolution depth map can be preset. The resolution compression operation can be used to compress the first foreground depth map with a larger image size into a low-resolution depth map with a preset number of rows and columns.

[0098] As an optional implementation, step 350 may include the following steps:

[0099] 3510. Determine a two-dimensional bounding box corresponding to the target object according to the image coordinates of each foreground data point in the first foreground depth map.

[0100] The two-dimensional bounding box is used to indicate the position of the target object in the first foreground depth map, and can be represented by the image coordinates of the foreground data points at the outermost periphery.

[0101] For example, the two-dimensional bounding box corresponding to the target object can be expressed as follows:

[0102]

[0103] Among them, u min It can represent the minimum value of the horizontal coordinate of the foreground data point, u max It can represent the maximum value of the horizontal coordinate of the foreground data point, v min It can represent the minimum value of the vertical coordinate of the foreground data point, v max It can represent the maximum vertical coordinate of the foreground data point.

[0104] 3520. Calculate a scaling coefficient corresponding to a resolution compression operation based on the preset number of rows and columns of the low-resolution depth map and the height and width of the two-dimensional bounding box.

[0105] The scale transformation coefficients can be used to guide the resolution compression operation. These scale transformation coefficients can include horizontal and vertical transformation coefficients. The horizontal transformation coefficients can be calculated using the preset number of columns in the low-resolution depth map and the width of the two-dimensional bounding box. The vertical transformation coefficients can be calculated using the preset number of rows in the low-resolution depth map and the height of the two-dimensional bounding box.

[0106] For example, the calculation method of the horizontal transform coefficient and the vertical transform coefficient can be expressed by the following formula:

[0107] scale u =N / (u max -u min ); Formula (5);

[0108] scale v =M / (v max -v min ); Formula (6);

[0109] Among them, scale u It can represent the horizontal transformation coefficient, N can represent the number of preset low-resolution depth map columns, (u max -u min ) can represent the width of the two-dimensional bounding box; scale v It can represent the vertical transformation coefficient, M can represent the number of rows of the preset low-resolution depth map, (v max -v min ) can represent the height of the two-dimensional bounding box.

[0110] 3530. Perform a resolution compression operation on the first foreground depth map according to the scale transformation coefficient to obtain a second foreground depth map.

[0111] In an embodiment of the present application, performing a resolution compression operation on the first foreground depth map according to the scale transformation coefficient may include: adjusting the image position of the foreground data points in the first foreground depth map using the scale transformation coefficient, and determining the depth value corresponding to each foreground data point after the transformation.

[0112] It should be noted that there may be two or more foreground data points in the first depth map that, after resolution compression, are mapped to the same first data point in the second foreground depth map. In this case, if the multiple foreground data points mapped to the same first data point have different corresponding depth values ​​in the first depth map, it is necessary to determine the final depth value of the mapped first data point in the second foreground depth map based on the different depth values ​​of the multiple foreground data points. Optionally, the minimum value among the depth values ​​corresponding to the multiple foreground data points may be taken as the depth value corresponding to the first data point in the mapped second depth map.

[0113] Exemplarily, the second foreground depth map may be represented as follows:

[0114]

[0115] Among them, depth map Can represent the second foreground depth map, u j and v j It can represent the image coordinates of the jth foreground data point on the second depth map, d j may represent the depth value of the j-th foreground data point in the second depth map.

[0116] Based on the aforementioned formula (7), the resolution compression operation of the first foreground depth map according to the scale transformation coefficient can be expressed by the following formula:

[0117] u j =(u i -u min )×sacle u +0.5; Formula (8);

[0118] v j =(v i -v min )×sacle v +0.5; Formula (9);

[0119] d j =min(d I ); Formula (10);

[0120] It should be noted that the jth foreground data point in the second depth map is obtained by performing a resolution compression operation on the i-th foreground data point in the first depth map. I It can represent the depth values ​​of all foreground data points in the first depth map that are mapped to the jth foreground data point in the second depth map, and min(·) is used to indicate the minimum value. In addition, the 0.5 in the above formulas (8) and (9) can be a quantitative parameter set by R&D personnel based on experience and can be set to other values ​​according to actual business needs. There is no specific limitation.

[0121] 360. Construct a first background depth map according to the image coordinates of each background data point on the imaging plane included in the second projection result and the depth value of each background data point.

[0122] 370. Perform a resolution compression operation on the first background depth map to obtain a second background depth map.

[0123] In the embodiment of the present application, the implementation of step 360 constructing the first background depth map and step 370 performing a resolution compression operation on the first background depth map can refer to the implementation of step 340 constructing the first foreground depth and step 350 performing a resolution compression operation on the first foreground depth map, and the following content will not be repeated.

[0124] It should be noted that multiple background data points may also correspond to the same background object. Therefore, if the electronic device needs to calculate a two-dimensional bounding box in step 370, the two-dimensional bounding box of the background object may be calculated.

[0125] 380. Calculate an occlusion rate of the target object according to the second foreground depth map and the second background depth map, and calculate a non-occlusion rate of the target object according to the occlusion rate of the target object.

[0126] In an embodiment of the present application, the electronic device can compare the depth values ​​corresponding to the foreground data points and background data points of the same image coordinates in the second foreground depth map and the second background depth map to calculate the depth value difference between the foreground data points and background data points of the same image coordinates.

[0127] According to the definition of occlusion, a depth threshold can be pre-set, for example, the depth threshold can be set to 1 meter.

[0128] The electronic device may count the number of target data points in the second foreground depth map whose depth value difference exceeds a depth threshold. For example, data points whose depth value difference exceeds 1 meter are target data points. After counting the number of target data points, the ratio between the number of target data points and the total number of foreground data points may be calculated as the occlusion rate of the target object.

[0129] For example, the occlusion rate of the target object can be calculated with reference to the following formula:

[0130]

[0131] Among them, occ_ratio can represent the occlusion rate, I occ The number of target data points that can be represented, I all It can represent the foreground data points belonging to the target object.

[0132] 390. Calculate the visibility score of the target object according to the product of the non-truncation rate and the non-occlusion rate of the target object, and perform visibility labeling on the target object according to the visibility score of the target object.

[0133] If the visibility score of the target object is greater than or equal to the preset threshold, the visibility of the target object is marked as true. If the visibility score of the target object is less than the preset threshold, the visibility of the target object can be marked as false.

[0134] For example, in order to more clearly explain the concepts of the aforementioned target object's truncation rate, non-truncation rate, occlusion rate, and non-occlusion rate, please refer to Figure 4 , Figure 4 The diagram is an example of a projection result of environmental point cloud data projected onto an imaging plane, as disclosed in an embodiment.

[0135] like Figure 4 As shown, the depth map 410 of the target object can indicate the image coordinates of each foreground data point corresponding to the target object after being projected onto the imaging plane. The visual sensor's viewing angle range 420 and the depth map 410 of the target object do not completely coincide with each other. The foreground data points in the depth map 410 of the target object that are outside the viewing angle range 420 are truncated foreground data points 410a, i.e., I in the aforementioned formula (1). crOp .

[0136] At the same time, among the multiple foreground data points included in the depth map 410 of the target object, there is a data point 410b that is blocked by a background data point, such as I in the above formula (11). OCc .

[0137] It can be seen that in the aforementioned embodiment, in addition to being able to quickly and accurately mark the visibility of the target object by calculating the non-occlusion rate and non-truncation rate, the electronic device can further reduce the calculation cost of the occlusion rate by introducing a low-resolution depth map, and improve the calculation accuracy of the occlusion rate, thereby further improving the speed and accuracy of visibility marking.

[0138] See also Figure 5 , Figure 5 This is a schematic diagram of a data tagging device disclosed in an embodiment, which can be applied to any of the aforementioned electronic devices. Figure 5As shown, the data tagging device 500 may include: an identification module 510 , a projection module 520 , a first determination module 530 , and a second determination module 540 .

[0139] Identification module 510, for identifying foreground data points corresponding to the target object and background data points unrelated to the target object from the environmental point cloud data collected by the radar sensor;

[0140] The projection module 520 is used to project the foreground data points and the background data points onto the imaging plane of the visual sensor, respectively, to obtain a first projection result corresponding to the foreground data point and a second projection result corresponding to the background data point;

[0141] A first determining module 530 is configured to determine a non-truncation rate of the target object according to the first projection result, and to determine a non-occlusion rate of the target object according to the first projection result and the second projection result;

[0142] The second determining module 540 is configured to perform visibility marking on the target object according to the non-truncation rate and the non-occlusion rate of the target object.

[0143] In one embodiment, the second determination module 540 may also be used to calculate the visibility score of the target object based on the product of the non-truncation rate and the non-occlusion rate of the target object; and if the visibility score of the target object is greater than or equal to a preset threshold, the visibility of the target object is marked as true.

[0144] In one embodiment, the first determining module 530 may include: a construction unit, a compression unit, and a calculation unit.

[0145] a construction unit, configured to construct a first foreground depth map of the target object based on the image coordinates of each foreground data point included in the first projection result on the imaging plane and the depth values ​​of each foreground data point corresponding to the target object;

[0146] a compression unit, configured to perform a resolution compression operation on the first foreground depth map to obtain a second foreground depth map;

[0147] The construction unit may be further configured to construct a first background depth map according to the image coordinates of each background data point included in the second projection result on the imaging plane and the depth value of each background data point;

[0148] The compression unit may be further configured to perform a resolution compression operation on the first background depth map to obtain a second background depth map;

[0149] The calculation unit can be used to calculate the occlusion rate of the target object according to the second foreground depth map and the second background depth map, and calculate the non-occlusion rate of the target object according to the occlusion rate of the target object.

[0150] In one embodiment, the compression unit can be used to determine the two-dimensional bounding box corresponding to the target object based on the image coordinates of each foreground data point in the first foreground depth map; and calculate the scale transformation coefficient corresponding to the resolution compression operation based on the preset number of rows and columns of the low-resolution depth map, and the height and width of the two-dimensional bounding box; and perform a resolution compression operation on the first foreground depth map according to the scale transformation coefficient to obtain a second foreground depth map.

[0151] Optionally, there are at least two data points in the first foreground depth map that are mapped to the same first data point in the second foreground depth map after resolution compression operation; the depth value of the first data point in the second foreground depth map is the minimum value of the depth values ​​of the at least two data points in the first foreground depth map.

[0152] In one embodiment, the computing unit can also be used to compare the depth values ​​corresponding to the foreground data points and background data points of the same image coordinates in the second foreground depth map and the second background depth map, respectively, and count the number of target data points in the second foreground depth map whose depth value difference exceeds the depth threshold; and determine the ratio between the number of target data points and the total number of foreground data points as the occlusion rate of the target object.

[0153] In one embodiment, the first determination module 530 can also be used to count the number of foreground data points projected outside the viewing angle range of the visual sensor based on the first projection result; and, determine the ratio between the number of foreground data points projected outside the viewing angle range and the total number of foreground data points as the truncation rate of the target object; and, determine the non-truncation rate of the target object based on the truncation rate of the target object.

[0154] Implementing the aforementioned data annotation device allows for rapid and accurate annotation of the visibility of target objects by calculating the non-occlusion rate and non-truncation rate. Visibility can be defined as the product of the non-occlusion rate and the non-truncation rate, facilitating the conversion of the binary classification problem of visibility into a regression problem based on the product data, thereby more flexibly supporting subsequent processing models. Furthermore, the computational cost of the occlusion rate can be reduced and the accuracy of the occlusion rate calculation improved by introducing a low-resolution depth map, further improving the speed and accuracy of visibility annotation.

[0155] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device disclosed in an embodiment. Figure 6 As shown, the electronic device may include:

[0156] A memory 610 storing executable program code;

[0157] a processor 620 coupled to the memory 610;

[0158] Among them, the processor 620 calls the executable program code stored in the memory 610 to execute any data labeling method disclosed in the embodiments of the present application.

[0159] It should be noted that Figure 6 The electronic device shown may also include a power supply, input buttons, a camera, a speaker, a screen, an RF circuit, a Wi-Fi module, a Bluetooth module, a sensor and other components not shown, which are not described in detail in this embodiment.

[0160] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein when the computer program is processed by a processor, the processor implements any one of the data labeling methods disclosed in the embodiment of the present application.

[0161] An embodiment of the present application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute any one of the data labeling methods disclosed in the embodiments of the present application.

[0162] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present application.

[0163] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0164] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of this embodiment.

[0165] In addition, the functional units in the embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0166] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a memory and includes several requests for a computer device (which can be a personal computer, server or network device, etc., specifically a processor in a computer device) to execute some or all of the steps of the above-mentioned methods of various embodiments of the present application.

[0167] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0168] The above is a detailed introduction to a data annotation method, device, electronic device, and storage medium disclosed in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present application.

Claims

1. A data annotation method, characterized in that: The method comprises: Identifying foreground data points corresponding to a target object and background data points unrelated to the target object from environmental point cloud data collected by the radar sensor; Projecting the foreground data points and the background data points onto an imaging plane of a visual sensor, respectively, to obtain a first projection result corresponding to the foreground data point and a second projection result corresponding to the background data point; Determining a non-truncation rate of the target object based on the first projection result, and determining a non-occlusion rate of the target object based on the first projection result and the second projection result, the non-truncation rate being used to indicate a probability that a foreground data point corresponding to the target object falls within a viewing angle range of the visual sensor, and the non-occlusion rate being used to indicate a probability that a foreground data point corresponding to the target object is not occluded by background data points unrelated to the target object; The visibility of the target object is marked according to the non-truncation rate and the non-occlusion rate of the target object.

2. The method according to claim 1, characterized in that The determining the visibility of the target object according to the truncation rate and the occlusion rate of the target object includes: Calculating a visibility score of the target object according to the product of a non-truncation rate and a non-occlusion rate of the target object; If the visibility score of the target object is greater than or equal to a preset threshold, the visibility of the target object is marked as true.

3. The method according to claim 1, characterized in that The determining the non-occlusion rate of the target object according to the first projection result and the second projection result includes: constructing a first foreground depth map of the target object according to the image coordinates of each foreground data point included in the first projection result on the imaging plane and the depth value of each foreground data point corresponding to the target object; performing a resolution compression operation on the first foreground depth map to obtain a second foreground depth map; constructing a first background depth map according to the image coordinates of each background data point on the imaging plane and the depth value of each background data point included in the second projection result; performing a resolution compression operation on the first background depth map to obtain a second background depth map; An occlusion rate of the target object is calculated according to the second foreground depth map and the second background depth map, and a non-occlusion rate of the target object is calculated according to the occlusion rate of the target object.

4. The method according to claim 3, characterized in that The performing a resolution compression operation on the first foreground depth map to obtain a second foreground depth map includes: determining a two-dimensional bounding box corresponding to the target object according to the image coordinates of each foreground data point in the first foreground depth map; Calculating a scaling coefficient corresponding to a resolution compression operation according to a preset number of rows and columns of the low-resolution depth map and a height and width of the two-dimensional bounding box; A resolution compression operation is performed on the first foreground depth map according to the scale transformation coefficient to obtain a second foreground depth map.

5. The method according to claim 3, characterized in that There are at least two data points in the first foreground depth map that are mapped to the same first data point in the second foreground depth map after resolution compression operation; the depth value of the first data point in the second foreground depth map is the minimum value of the depth values ​​of the at least two data points in the first foreground depth map.

6. The method according to claim 3, characterized in that The calculating the occlusion rate of the target object according to the second foreground depth map and the second background depth map includes: Comparing the depth values ​​corresponding to the foreground data points and the background data points with the same image coordinates in the second foreground depth map and the second background depth map, respectively, and counting the number of target data points in the second foreground depth map whose depth value difference exceeds a depth threshold; The ratio between the number of target data points and the total number of foreground data points is determined as the occlusion rate of the target object.

7. The method according to any one of claims 1 to 6, characterized in that The determining the non-truncation rate of the target object according to the first projection result includes: Counting the number of foreground data points projected beyond the viewing angle of the visual sensor according to the first projection result; determining a ratio between the number of foreground data points projected outside the viewing angle range and the total number of foreground data points as the truncation rate of the target object; A non-truncation rate of the target object is determined according to the truncation rate of the target object.

8. A data labeling device, characterized in that: include: An identification module is used to identify foreground data points corresponding to a target object and background data points unrelated to the target object from the environmental point cloud data collected by the radar sensor; A projection module, configured to project the foreground data points and the background data points onto an imaging plane of a visual sensor, respectively, to obtain a first projection result corresponding to the foreground data point and a second projection result corresponding to the background data point; a first determination module, configured to determine a non-truncation rate of the target object based on the first projection result, and to determine a non-occlusion rate of the target object based on the first projection result and the second projection result, wherein the non-truncation rate is used to indicate a probability that a foreground data point corresponding to the target object falls within a viewing angle range of the visual sensor, and the non-occlusion rate is used to indicate a probability that a foreground data point corresponding to the target object is not occluded by background data points unrelated to the target object; The second determining module is configured to perform visibility marking on the target object according to the non-truncation rate and the non-occlusion rate of the target object.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images