Data processing method and device, electronic equipment and storage medium

Abstract

The application provides a data processing method and device, electronic equipment and storage medium. The method comprises the following steps: acquiring first point cloud data collected by a radar on a target object under a first collection condition, wherein the first collection condition comprises that a first distance between the target object and the radar is less than a first threshold; acquiring second point cloud data collected by the radar on the target object under a second collection condition, wherein the second collection condition comprises that a number of data points in the second point cloud data is not greater than a second threshold, and / or a second distance between the target object and the radar is greater than a third threshold, and the third threshold is greater than or equal to the first threshold; fusing the first point cloud data and the second point cloud data, obtaining third point cloud data and a third detection box of the third point cloud data according to a fusion result and a perception algorithm; and marking the third detection box on the second point cloud data. The third detection box is marked based on the fused third point cloud data, and the marking accuracy and efficiency are considered.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] In road testing scenarios, test vehicles are equipped with radar, which collects data about the external environment, thus obtaining point cloud data of the external environment. When point cloud data needs to be used for training (including training, testing, or validation) of perception algorithms, it is usually necessary to label the point cloud data of the target object. For example, a detection box may be marked on the point cloud data of a vehicle. The detection box can be a cuboid that surrounds all the data points of the vehicle. Then, the detection box and the vehicle's point cloud data are injected back into the perception algorithm to achieve training, validation, or testing.

[0003] There are two main methods of labeling: one is entirely manual labeling, and the other is labeling using a combination of perception algorithms and manual labeling. However, while the former has higher accuracy, it is less efficient. The latter only has high accuracy when the target object is close to the radar. When the target object is far from the radar, the point cloud of the target object is often sparse, and the size and shape of the detection boxes obtained by the perception algorithm often do not match the actual situation, resulting in low accuracy. This will affect the training, verification, or testing results. If manual labeling is used for distant targets, it will result in excessively low efficiency.

[0004] Therefore, there is a technical problem that current point cloud data labeling methods cannot simultaneously achieve labeling accuracy and labeling efficiency, and improvements are needed. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and storage medium to alleviate the current technical problem that the accuracy and efficiency of point cloud data cannot be simultaneously achieved when labeling it.

[0006] To address the aforementioned technical problems, this application provides the following technical solutions:

[0007] This application provides a data processing method, including:

[0008] Acquire first point cloud data of the target object collected by the radar under the first acquisition condition, and mark the first detection box of the first point cloud data based on the perception algorithm. The first acquisition condition includes that the first distance between the target object and the radar is less than a first threshold.

[0009] The radar acquires second point cloud data of the target object under a second acquisition condition, wherein the second acquisition condition includes: the number of data points in the second point cloud data is not greater than a second threshold, and / or: the second distance between the target object and the radar is greater than a third threshold, wherein the third threshold is greater than or equal to the first threshold;

[0010] By fusing the first point cloud data and the second point cloud data, and based on the fusion result and the perception algorithm, a third point cloud data and a third detection box of the third point cloud data are obtained;

[0011] The third detection box is marked on the second point cloud data.

[0012] Meanwhile, this application also provides a data processing apparatus, including:

[0013] The first acquisition module is used to acquire first point cloud data of the target object collected by the radar under the first acquisition condition, and to mark the first detection box of the first point cloud data based on the perception algorithm. The first acquisition condition includes that the first distance between the target object and the radar is less than a first threshold.

[0014] The second acquisition module is used to acquire second point cloud data collected by the radar on the target object under second acquisition conditions. The second acquisition conditions include: the number of data points in the second point cloud data is not greater than a second threshold, and / or: the second distance between the target object and the radar is greater than a third threshold, and the third threshold is greater than or equal to the first threshold.

[0015] The fusion module is used to fuse the first point cloud data and the second point cloud data, and obtain the third point cloud data and the third detection box of the third point cloud data according to the fusion result and the perception algorithm.

[0016] A marking module is used to mark the third detection box on the second point cloud data.

[0017] This application also provides an electronic device, including a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to perform the steps in the data processing method described in any of the preceding claims.

[0018] This application provides a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the steps in the data processing method described above.

[0019] Beneficial effects: This application provides a data processing method, apparatus, electronic device, and storage medium. The method first acquires first point cloud data collected by radar on a target object under a first acquisition condition, the first acquisition condition including a first distance between the target object and the radar being less than a first threshold. Then, it acquires second point cloud data collected by radar on the target object under a second acquisition condition, the second acquisition condition including: the number of data points in the second point cloud data is not greater than a second threshold, and / or: the second distance between the target object and the radar is greater than a third threshold, the third threshold being greater than or equal to the first threshold. Then, it fuses the first point cloud data and the second point cloud data, and obtains third point cloud data and a third detection box of the third point cloud data based on the fusion result and a perception algorithm. Finally, it marks the third detection box on the second point cloud data. The first point cloud data obtained under the first acquisition condition is relatively dense and rich in information, while the second point cloud data obtained under the second acquisition condition is relatively sparse and lacks information. Therefore, after fusing the first and second point cloud data, the resulting third point cloud data contains information from both the first and second point cloud data, and its information is also relatively rich. Thus, compared to the scheme of directly labeling the sparse second point cloud data to obtain the second detection box, this application first obtains the third detection box based on the dense third point cloud data, and then labels the third detection box on the second point cloud data. This effectively improves the labeling accuracy of the third detection box, and the entire process is automatically calculated based on the perception algorithm, eliminating the need for manual labeling and improving labeling efficiency. In other words, this application achieves a balance between labeling accuracy and labeling efficiency. Attached Figure Description

[0020] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0021] Figure 1 This is a schematic diagram of a scenario for the data processing method provided in the embodiments of this application.

[0022] Figure 2 This is a flowchart illustrating the data processing method provided in an embodiment of this application.

[0023] Figure 3 This is a schematic diagram of the second point cloud data and the second detection box under the first fusion and labeling method.

[0024] Figure 4 This is a schematic diagram of the third point cloud data and the second detection box under the first fusion and labeling method.

[0025] Figure 5 This is a schematic diagram of the third point cloud data and the third detection box under the first fusion and labeling method.

[0026] Figure 6This is a schematic diagram of the second point cloud data and the third detection box under the first fusion and labeling method.

[0027] Figure 7 This is a schematic diagram of the second point cloud data and the second detection box under the second fusion and labeling method.

[0028] Figure 8 This is a schematic diagram of the third point cloud data and the second detection box under the second fusion and labeling method.

[0029] Figure 9 This is a schematic diagram of the third point cloud data and the third detection box under the second fusion and labeling method.

[0030] Figure 10 This is a schematic diagram of the second point cloud data and the third detection box under the second fusion and labeling method.

[0031] Figure 11 A schematic diagram of a data processing apparatus provided in an embodiment of this application.

[0032] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0034] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating a scenario in which the data processing method provided in this application embodiment is applied. The scenario may include a road test vehicle (or other vehicles not specifically designed for road testing), a target object, and a data processing device. The road test vehicle is equipped with radar. The target object is a vehicle, pedestrian, or other dynamic or static object traveling on the road. The data processing device may be a local data processing device or a remote data processing device.

[0035] When relative motion occurs between the target object and the road test vehicle, the radar acquires first point cloud data under the first acquisition condition and second point cloud data under the second acquisition condition. The first acquisition condition includes a first distance between the target object and the radar that is less than a first threshold. In this case, the first point cloud data is usually denser and contains more information. The second acquisition condition includes any one of the following three situations: (1) the number of data points in the second point cloud data is not greater than the second threshold; (2) the second distance between the target object and the radar is greater than a third threshold, and the third threshold is greater than or equal to the first threshold; (3) the second distance between the target object and the radar is greater than the third threshold, the third threshold is greater than or equal to the first threshold, and the number of data points in the second point cloud data is not greater than the second threshold. In this case, the second point cloud data is usually sparser and contains less information.

[0036] The first and second point cloud data are fused to obtain the third point cloud data. This third point cloud data is used as a marker source under the second acquisition condition. A perception algorithm is used to mark this marker source, resulting in a third detection box, which is a three-dimensional bounding box enclosing all data points in the third point cloud data. The third detection box is then used to mark the second point cloud data, and this third detection box serves as the marking result for the second point cloud data.

[0037] Since the third point cloud data contains information from both the first and second point cloud data, it is richer in information. The third detection box obtained by using the third point cloud data as the labeling source is closer to the actual size and position of the target object than the second detection box obtained by using the second point cloud data as the labeling source. This effectively improves the labeling accuracy of the second point cloud data. Moreover, the entire process is automatically calculated based on the perception algorithm, without the need for manual labeling, which improves the labeling efficiency. Therefore, it achieves a balance between labeling accuracy and labeling efficiency.

[0038] It should be noted that, Figure 1 The system scenario diagram shown is merely an example. The data processing device and scenario described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of systems and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. Detailed descriptions are provided below. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0039] Please see Figure 2 , Figure 2 This is a schematic diagram of the data processing flow provided in an embodiment of this application. The method specifically includes:

[0040] S1: Acquire the first point cloud data of the target object obtained by the radar under the first acquisition condition, the first acquisition condition including the first distance between the target object and the radar being less than the first threshold.

[0041] In this embodiment, the radar is mounted on a road test vehicle. In a road test scenario, the road test vehicle may include one or more vehicles, and each vehicle may be equipped with one or more radars. The radar can sense and collect external environmental information within its sensing range, obtaining point cloud data. The target object refers to an object in the external environment that needs to be tracked and marked, such as a vehicle. When the target object and the radar are in relative motion, the radar will periodically or continuously collect data on the target object, and acquire the corresponding point cloud data through a perception algorithm each time it collects data. The relative motion between the target object and the radar can be: the target object is stationary while the radar is moving, gradually approaching or moving away from the target object; the radar is stationary while the target object is moving, gradually approaching or moving away from the radar; or both the target object and the radar are moving simultaneously, gradually approaching or moving away from each other. This embodiment does not limit the type of relative motion between the target object and the radar; those skilled in the art can set the type of relative motion as needed.

[0042] Under the first acquisition condition, the radar acquires first point cloud data from the target object. This first point cloud data includes attribute information (coordinates, reflection intensity, color, etc.) of each data point on the target object, forming a three-dimensional first point cloud. The first acquisition condition includes the target object being less than a first threshold distance from the radar. This first threshold is specifically set after comprehensively considering factors such as the radar's acquisition capabilities (acquisition range, acquisition accuracy, etc.) and the target object's attribute information (shape, size, etc.). When the first distance is less than the first threshold, it indicates that the target object is relatively close to the radar, and the first point cloud data is usually denser and contains richer information.

[0043] Once the first point cloud data of the target object is obtained, it can be labeled using a perception algorithm to obtain the first detection box. The first detection box is a three-dimensional bounding box, such as a cuboid, that encloses all data points in the first point cloud data of the target object. Each data point in the first point cloud data of the target object serves as the labeling source for the first detection box. The perception algorithm can be the BEVFUSION algorithm, which encodes image data from the camera and point cloud data from the radar into features within the same BEV space and passes them to the downstream task architecture. The radar branch, which processes the point cloud data, can be used to perceive and label all data points of the target object to obtain the corresponding detection boxes.

[0044] When the point cloud data of the target object is relatively dense, the size and position of the detection boxes marked by the perception algorithm will closely match the actual situation of the target object, meaning the marking accuracy is high. Conversely, when the point cloud data of the target object is relatively sparse, the size and position of the detection boxes marked by the perception algorithm will deviate significantly from the actual situation of the target object, meaning the marking accuracy is low. In this embodiment, since the first point cloud data is relatively dense, the first detection boxes obtained by using the first point cloud data as the marking source can accurately depict the target object.

[0045] In one embodiment, S1 specifically includes: acquiring first point cloud data collected by the radar from the target angle. The acquired first point cloud data may be obtained by the radar in a single acquisition at a certain moment, where the acquisition angle of the radar relative to the target object at that moment is the target angle. In the following embodiments, the first point cloud data acquired by this scheme is denoted as A, and the corresponding first detection box is denoted as BOX_A.

[0046] In one embodiment, S1 specifically includes: acquiring at least two sets of sub-point cloud data collected by radar from at least two different angles of the target object, and fusing the at least two sets of sub-point cloud data to obtain first point cloud data. The acquired first point cloud data can also be obtained by separately collecting data from the target object from two or more different angles of the radar and then fusing the data. During collection, the same radar can collect data from different angles around the target object at different times, or different radars can collect data from different angles around the target object at different times or at the same time. In the following embodiments, the sub-point cloud data collected at each different angle in this scheme are respectively denoted as B1, B2, ..., Bn, the first point cloud data obtained after fusion is denoted as B, and the corresponding first detection box is denoted as BOX_B.

[0047] In one embodiment, the step of fusing at least two sets of sub-point cloud data to obtain the first point cloud data specifically includes: determining reference sub-point cloud data from at least two sets of sub-point cloud data, and transforming the other sub-point cloud data in the at least two sets of sub-point cloud data, excluding the reference sub-point cloud data, to the first reference coordinate system where the reference sub-point cloud data is located; performing feature matching between the other sub-point cloud data and the reference sub-point cloud data in the first reference coordinate system, and adjusting the coordinate information of the other sub-point cloud data according to the matching result; and fusing the adjusted other sub-point cloud data with the reference sub-point cloud data to obtain the first point cloud data.

[0048] When collecting data on a target object from different angles, the different coordinate systems corresponding to each sub-point cloud data make it impossible to directly compare the positions of the data points. Therefore, all sub-point cloud data need to be unified before fusion. Furthermore, although the sub-point cloud data are collected from the same target object from different angles, there needs to be a matching part between other sub-point cloud data and the reference sub-point cloud data during collection to achieve registration and fusion. These matching parts exist in the form of matching point pairs, which include feature point 'a' in other sub-point cloud data and feature point 'c' in the reference sub-point cloud data. Matching means that after registering other sub-point cloud data with the reference sub-point cloud data, feature point 'a' can express the attributes of other sub-point cloud data in the reference sub-point cloud data.

[0049] Typically, there are multiple matching point pairs between other sub-point cloud data and the reference sub-point cloud data. Feature matching involves finding these matching point pairs and solving for the coordinate transformation parameters between the other sub-point cloud data and the reference sub-point cloud data. This ensures that after coordinate adjustment based on these parameters, the cumulative coordinate difference between the two feature points a and c in all matching point pairs is minimized, thus maximizing the matching degree between the other sub-point cloud data and the reference sub-point cloud data. Feature matching can be based on the ICP algorithm or other point cloud registration algorithms. Based on the matching results, all matching point pairs can be determined, and the corresponding coordinate transformation parameters can be calculated. These parameters can be in matrix form. The coordinate information of all data points in the other sub-point cloud data is adjusted based on these parameters. Finally, the adjusted other sub-point cloud data is fused with the reference sub-point cloud data to obtain the first point cloud data B. When there are two or more sets of other sub-point cloud data, the above operation is performed once for each set.

[0050] Specifically, taking sub-point cloud data including B1, B2, and B3 as an example, we can first determine a set of reference sub-point cloud data, such as B1, from B1, B2, and B3. The coordinate system of B1 is then designated as the first reference coordinate system. B2 and B3 are then transformed to this first reference coordinate system. Next, feature matching is performed on B1 and B2 within the first reference coordinate system to find matching point pairs between them. The coordinate transformation parameters from B2 to B1 are calculated by combining the coordinate differences of each matching point pair. Then, the coordinate information of all data points in B2 is adjusted to complete the registration and fusion of B1 and B2. Then, feature matching is performed on B1 and B3 within the first reference coordinate system to find matching point pairs between them. The coordinate transformation parameters from B3 to B1 are calculated by combining the coordinate differences of each matching point pair. Then, the coordinate information of all data points in B3 is adjusted to complete the registration and fusion of B1 and B3. Combining the results of the two fusions, the final first point cloud data B is obtained.

[0051] S2: Acquire the second point cloud data of the target object collected by the radar under the second acquisition conditions. The second acquisition conditions include: the number of data points in the second point cloud data is not greater than the second threshold, and / or: the second distance between the target object and the radar is greater than the third threshold, and the third threshold is greater than or equal to the first threshold.

[0052] The radar can acquire second point cloud data by acquiring data from the target object under the second acquisition condition. The second acquisition condition includes any one of the following three situations: first, the number of data points in the second point cloud data is not greater than a second threshold; second, the second distance between the target object and the radar is greater than a third threshold, and the third threshold is greater than or equal to a first threshold; third, the second distance between the target object and the radar is greater than a third threshold, the third threshold is greater than or equal to a first threshold, and the number of data points in the second point cloud data is not greater than the second threshold. The first threshold is the same as the first threshold mentioned in step S1 above, the third threshold needs to be greater than or equal to the first threshold, and the second threshold can be specifically set after comprehensively considering factors such as the radar's acquisition capabilities (acquisition range, acquisition accuracy, etc.) and the target object's attribute information (shape, size, etc.). In the following embodiment, the acquired second point cloud data is denoted as C.

[0053] In the first scenario, when the number of data points is no greater than the second threshold, the second point cloud data can be directly considered sparse. Therefore, when this condition is met, it can also be considered unsuitable as a marker source for meeting the accuracy requirements of subsequent markers. In the second scenario, when the distance is greater than or equal to the third threshold, it indicates that the target object is far from the radar. In this case, the second point cloud data is usually sparse, and when this condition is met, it is considered unsuitable as a marker source for meeting the accuracy requirements of subsequent markers. In the third scenario, since both the distance and the point cloud sparseness conditions are met simultaneously, the accuracy of subsequent markers will be even lower, further failing to meet the marker requirements.

[0054] In the following embodiments, for ease of comparison and explanation, the second detection box obtained by directly labeling the second point cloud data C based on the perception algorithm is represented by BOX_C. In the above three cases, the labeling accuracy of BOX_C is low.

[0055] S3: Fuse the first point cloud data and the second point cloud data, and based on the fusion result and the perception algorithm, obtain the third point cloud data and the third detection box of the third point cloud data.

[0056] The first point cloud data A (or B) and the second point cloud data C are fused to obtain the third point cloud data C'. Using the third point cloud data C' as the label source, the label source is labeled based on the perception algorithm to obtain the third detection box BOX_C'. Since each data point in the first point cloud data A (or B) and the second point cloud data C is obtained by the perception algorithm from the same target object, each data point in the third point cloud data C' after fusing the first point cloud data A (or B) and the second point cloud data C is also a data point on the target object.

[0057] For the same target object, regardless of whether it is under the first acquisition condition or the second acquisition condition, its outline, size, etc. are the same. Therefore, theoretically, the orientation and size of the first detection box BOX_A (or BOX_B) and the second detection box BOX_C should be consistent. Moreover, when the accuracy of the perception algorithm is constant, the orientation and size of the first detection box BOX_A (or BOX_B) and the second detection box BOX_C are consistent with the difference between the actual detection box of the target object.

[0058] However, the second point cloud data C obtained under the second acquisition condition is relatively sparse. The second detection box BOX_C, labeled using the perception algorithm, differs significantly from the actual detection box, meaning its labeling accuracy is low. Conversely, the first point cloud data A (or B) obtained under the first acquisition condition is relatively dense. Therefore, the first detection box BOX_A (or BOX_B), labeled using the same accuracy perception algorithm, differs less from the actual detection box, meaning its labeling accuracy is high. The third point cloud data C' combines the first point cloud data A (or B) and the second point cloud data C, containing more data points. Therefore, the third detection box BOX_C', labeled using the same accuracy perception algorithm, differs less from the actual detection box, meaning its labeling accuracy is higher. Thus, among the first detection box BOX_A (or BOX_B), the second detection box BOX_C, and the third detection box BOX_C', the direction and size of the third detection box BOX_C' are closer to the direction and size of the actual detection box of the target object.

[0059] It should be noted that the first point cloud data A (or B) in this embodiment can be data collected by the radar at a single moment when the first distance between the radar and the target object is s1, or it can be data collected by the radar at the first distances between the radar and the target object sequentially s1, s2, ..., s n The data collected at different times are not limited in the number of first point cloud data A (or B) participating in the fusion in this embodiment of the application. Within a certain range, the more first point cloud data A (or B) participating in the fusion, the smaller the difference between the fused and labeled third detection box BOX_C' and the actual detection box.

[0060] There are two ways to obtain the third point cloud data and the third detection box.

[0061] In one embodiment, S3 specifically includes: using the second point cloud data as reference point cloud data, transforming the first point cloud data to the second reference coordinate system where the reference point cloud data is located; performing feature matching between the first point cloud data and the reference point cloud data in the second reference coordinate system, adjusting the coordinate information of the first point cloud data according to the matching result, and obtaining the fourth point cloud data; fusing the fourth point cloud data and the reference point cloud data to obtain the third point cloud data, and obtaining the third detection box of the third point cloud data based on the perception algorithm.

[0062] Using the second point cloud data C as reference point cloud data, the first point cloud data A (or B) and the first detection box BOX_A (or BOX_B) are first transformed to the second reference coordinate system where the second point cloud data C is located, achieving coordinate system unification. Then, in the second reference coordinate system, keeping all data points in the second point cloud data C unchanged, feature matching is performed between the first point cloud data A (or B) and the second point cloud data C. Based on the matching result, the coordinate information of all data points in the first point cloud data A (or B) is adjusted to obtain the fourth point cloud data. In the following embodiments, the adjusted fourth point cloud data is denoted as A' (or B'). The fourth point cloud data A' (or B') and the second point cloud data C are merged into a whole, which is the third point cloud data C'. Based on the perception algorithm, all data points in the third point cloud data C' are labeled to obtain the third detection box BOX_C'.

[0063] like Figures 3 to 5 As shown, the circle on the right side of the figure represents the same target object, while the left side of the figure presents the point cloud data of the target object and the detection box as a whole using a three-view drawing. Figure 3 The image shows the second point cloud data C of the target object and the second detection box BOX_C obtained by directly labeling the second point cloud data C. Figure 4 The image shows the third point cloud data C' obtained by fusing the first point cloud data A (or B) and the second point cloud data C of the target object in the manner described above, as well as the second detection box BOX_C. Figure 5 The image shows the third point cloud data C' and the third detection box BOX_C' obtained by labeling the third point cloud data C'.

[0064] Comparing the three figures, it can be seen that for the same second detection box BOX_C, Figure 3 The second point cloud data C is entirely located within the second detection box BOX_C, while Figure 4The third point cloud data C' is partially located within the second detection box BOX_C, with the remaining portion outside. Since all data points in the third point cloud data C' are points on the target object, the presence of points outside the second detection box BOX_C indicates that the labeled second detection box BOX_C is not the actual size of the target object and is too small. In this case, if the third point cloud data C' is labeled based on a perception algorithm, Figure 5 The third detection box BOX_C' obtained in the process will surround all the third point cloud data C', and will be larger in size and closer to the actual size and position of the target object.

[0065] In one embodiment, S3 specifically includes: labeling a first detection box of the first point cloud data based on a perception algorithm; transforming the first point cloud data and the first detection box into a second reference coordinate system where the reference point cloud data is located, using the second point cloud data as reference point cloud data; performing feature matching between the first point cloud data and the reference point cloud data in the second reference coordinate system, adjusting the coordinate information of the first point cloud data and the pose of the first detection box according to the matching result to obtain a fourth point cloud data and a fourth detection box; fusing the fourth point cloud data and the second point cloud data to obtain a third point cloud data, and adjusting the scaling ratio of the fourth detection box based on the third point cloud data to obtain a third detection box of the third point cloud data.

[0066] First, a first detection box BOX_A (or BOX_B) of the first point cloud data A (or B) is obtained based on a perception algorithm. Then, the second point cloud data C is used as reference point cloud data. Both the first point cloud data A (or B) and the first detection box BOX_A (or BOX_B) are transformed into the second reference coordinate system where the second point cloud data C is located, thus achieving coordinate system unification. Next, keeping all data points in the second point cloud data C unchanged, feature matching is performed between the first point cloud data A (or B) and the second point cloud data C. Based on the matching result, the coordinate information of all data points in the first point cloud data A (or B) is adjusted to obtain the fourth point cloud data A' (or B'). Simultaneously, the pose of the first detection box BOX_A (or BOX_B) is adjusted synchronously according to the matching result, that is, the first detection box BOX_A (or BOX_B) is translated and rotated to obtain the fourth detection box. In the following embodiments, the fourth detection box is denoted as BOX_A' (or BOX_B'). After the pose adjustment, the fourth detection box BOX_A' (or BOX_B') contains all the data points in the fourth point cloud data A' (or B').

[0067] The fourth point cloud data A' (or B') and the second point cloud data C are merged into a single unit, which is the third point cloud data C'. At this point, since the fourth detection box BOX_A' (or BOX_B') only contains all the data points from the fourth point cloud data A' (or B'), if some data points in the second point cloud data C are located outside the fourth detection box BOX_A' (or BOX_B'), it indicates that the size of the fourth detection box BOX_A' (or BOX_B') is too small. Directly using the fourth detection box BOX_A' (or BOX_B') as the third detection box BOX_C' would result in inaccurate labeling. Therefore, it is also necessary to adjust the scaling ratio of the fourth detection box BOX_A' (or BOX_B') based on the third point cloud data C', so that the adjusted fourth detection box BOX_A' (or BOX_B') can contain all the data points in the third point cloud data C', and finally use the adjusted fourth detection box BOX_A' (or BOX_B') as the third detection box BOX_C'.

[0068] like Figures 7 to 9 As shown, the circle on the right side of the figure represents the same target object, while the left side of the figure presents the point cloud data of the target object and the detection box as a whole using a three-view drawing. Figure 7 The image shows the second point cloud data C of the target object and the second detection box BOX_C obtained by directly labeling the second point cloud data C. Figure 8 The image shows the third point cloud data C' obtained by fusing the first point cloud data A (or B) and the second point cloud data C of the target object in the manner described above, as well as the second detection box BOX_C. Figure 9 The image shows the third point cloud data C' and the third detection box BOX_C'.

[0069] Comparing the three figures, it can be seen that for the same second detection box BOX_C, Figure 7 The second point cloud data C is entirely located within the second detection box BOX_C, while Figure 8 The third point cloud data C' is partially located within the second detection box BOX_C, with the remaining portion outside. Since all data points in the third point cloud data C' are points on the target object, the presence of points outside the second detection box BOX_C indicates that the labeled second detection box BOX_C is not the actual size of the target object and is too small. In this case, if the third point cloud data C' is labeled based on a perception algorithm, Figure 9 The third detection box BOX_C' obtained in the process will surround all the third point cloud data C', and will be larger in size and closer to the actual size and position of the target object.

[0070] In the two schemes mentioned above, the steps of performing feature matching between the first point cloud data and the reference point cloud data, and adjusting the coordinate information of the first point cloud data according to the matching result, specifically include: performing feature matching between the first point cloud data and the reference point cloud data in the second reference coordinate system, obtaining matching point pairs in the first point cloud data and the second point cloud data according to the matching result; and adjusting the coordinate information of all data points in the first point cloud data according to the coordinate difference of the matching point pairs.

[0071] For registration and fusion to be achieved, there needs to be a matching part between the first point cloud data A (or B) and the second point cloud data C. These matching parts exist in the form of matching point pairs. The matching point pairs include feature point d in the first point cloud data A (or B) and feature point e in the second point cloud data C. Matching means that after registering the first point cloud data A (or B) and the second point cloud data C, feature point d in the second point cloud data C can express the attributes in the first point cloud data A (or B).

[0072] Typically, there are multiple matching point pairs between the first point cloud data A (or B) and the second point cloud data C. Feature matching involves finding these matching point pairs and solving for the coordinate transformation parameters between the first point cloud data A (or B) and the second point cloud data C. The goal is to minimize the cumulative coordinate difference between the two feature points d and e in all matching point pairs after coordinate adjustment based on these transformation parameters, thus maximizing the matching degree between the first point cloud data A (or B) and the second point cloud data C. Feature matching can be performed based on the ICP algorithm or other point cloud registration algorithms. Based on the matching results, all matching point pairs can be determined, and the corresponding coordinate transformation parameters can be calculated. These parameters can be in matrix form. The coordinate information of all data points in the first point cloud data A (or B) is adjusted based on these transformation parameters. Finally, the adjusted first point cloud data A (or B) is fused with the second point cloud data C to obtain the third point cloud data C'. When there are two or more sets of the first point cloud data A (or B), perform the above operation once for each set of the first point cloud data A (or B).

[0073] Specifically, taking the first point cloud data, including A1 and A2, as an example, the coordinate system of the second point cloud data C is first determined as the second reference coordinate system. Then, both A1 and A2 are transformed into the second reference coordinate system. Next, feature matching is performed on A1 and C in the second reference coordinate system to find each matching point pair between A1 and C. The coordinate transformation parameters from A1 to C are calculated by combining the coordinate differences of each matching point pair. Then, the coordinate information of all data points in A1 is adjusted to complete the registration and fusion of A1 and C. Then, feature matching is performed on A2 and C in the second reference coordinate system to find each matching point pair between A2 and C. The coordinate transformation parameters from A2 to C are calculated by combining the coordinate differences of each matching point pair. Then, the coordinate information of all data points in A2 is adjusted to complete the registration and fusion of A2 and C. Combining the results of the two fusions, the final third point cloud data C' is obtained.

[0074] S4: Mark the third detection box on the second point cloud data.

[0075] Since the orientation and size of the third detection box BOX_C' are closer to the actual detection box of the target object than the first detection box BOX_A (or BOX_B), the second detection box BOX_C, and the third detection box BOX_C', the third detection box BOX_C' can be marked on the second point cloud data C, and the third detection box BOX_C' can be used as the marking result of the second point cloud data C. For example... Figure 6 and Figure 10 The figures show the effects of marking the third detection box BOX_C' obtained by the two fusion methods above onto the second point cloud data C.

[0076] It should be noted that the first point cloud data collected in step S1 includes two schemes, A and B. Since both schemes are collected under the first collection conditions, they are richer than the second point cloud data C collected under the second collection conditions. Therefore, both A and B can improve the accuracy of marking to a certain extent.

[0077] In one embodiment, after S4, the method further includes: training, testing, or validating the perception algorithm or another target algorithm based on the second point cloud data and the third detection box. The second point cloud data C and the third detection box BOX_C' can be used to reflect the perception algorithm's identification and labeling of distant target objects. Training, testing, or validating the perception algorithm based on this perception data, and optimizing the algorithm's performance based on the training, testing, or validation results, can improve the accuracy of the perception algorithm in identifying and labeling distant target objects. Furthermore, this scenario is not limited to this; the second point cloud data C and the third detection box BOX_C' can also be used to train, test, or validate another target algorithm. This other target algorithm is different from the perception algorithm used in the above steps. The relevant parameters and tasks to be performed by the two algorithms can be the same or not completely identical, but this target algorithm also requires relatively accurate point cloud data and corresponding detection boxes as the basis for algorithm optimization. This application does not limit the specific type of target algorithm; any algorithm that requires point cloud data and detection boxes of distant target objects can be used as the target algorithm. Training, testing, or validating another target algorithm based on the second point cloud data C and the third detection box BOX_C' can also help optimize the performance of that target algorithm.

[0078] As can be seen from the above embodiments, the first point cloud data obtained under the first acquisition condition is relatively dense and rich in information, while the second point cloud data obtained under the second acquisition condition is relatively sparse and lacks information. Therefore, after fusing the first and second point cloud data, the resulting third point cloud data simultaneously contains information from both the first and second point cloud data, and its information is also relatively rich. Thus, compared to the scheme of directly marking the sparse second point cloud data to obtain the second detection box, this application first obtains the third detection box based on the dense third point cloud data, and then marks the third detection box on the second point cloud data. This effectively improves the marking accuracy of the third detection box, and the entire process is automatically calculated based on the perception algorithm, eliminating the need for manual marking and improving marking efficiency. In other words, this application achieves a balance between marking accuracy and marking efficiency.

[0079] Based on the methods described in the above embodiments, this embodiment will be further described from the perspective of a data processing device, such as... Figure 11 As shown, the data processing device includes:

[0080] The first acquisition module 10 is used to acquire first point cloud data of the target object collected by the radar under the first acquisition condition, wherein the first acquisition condition includes that the first distance between the target object and the radar is less than a first threshold.

[0081] The second acquisition module 20 is used to acquire second point cloud data collected by the radar on the target object under second acquisition conditions. The second acquisition conditions include: the number of data points in the second point cloud data is not greater than a second threshold, and / or: the second distance between the target object and the radar is greater than a third threshold, and the third threshold is greater than or equal to the first threshold.

[0082] The fusion module 30 is used to fuse the first point cloud data and the second point cloud data, and obtain the third point cloud data and the third detection box of the third point cloud data according to the fusion result and the perception algorithm.

[0083] The marking module 40 is used to mark the third detection box on the second point cloud data.

[0084] In one embodiment, the first acquisition module 10 includes:

[0085] The first acquisition submodule is used to acquire the first point cloud data collected by the radar from the target object from the target angle;

[0086] Or include:

[0087] The second acquisition submodule is used to acquire at least two sets of sub-point cloud data collected by the radar from at least two different angles of the target object, and to fuse the at least two sets of sub-point cloud data to obtain the first point cloud data.

[0088] In one embodiment, the second acquisition submodule includes:

[0089] A transformation unit is configured to determine reference sub-point cloud data from the at least two sets of sub-point cloud data, and transform other sub-point cloud data in the at least two sets of sub-point cloud data, excluding the reference sub-point cloud data, to the first reference coordinate system where the reference sub-point cloud data is located.

[0090] The first matching unit is used to perform feature matching between the other sub-point cloud data and the reference sub-point cloud data in the first reference coordinate system, and adjust the coordinate information of the other sub-point cloud data according to the matching result;

[0091] The fusion unit is used to fuse the adjusted other sub-point cloud data with the reference sub-point cloud data to obtain the first point cloud data.

[0092] In one embodiment, the fusion module 30 includes:

[0093] The first transformation submodule is used to transform the first point cloud data to the second reference coordinate system where the reference point cloud data is located, using the second point cloud data as the reference point cloud data.

[0094] The first matching submodule is used to perform feature matching between the first point cloud data and the reference point cloud data in the second reference coordinate system, and adjust the coordinate information of the first point cloud data according to the matching result to obtain the fourth point cloud data.

[0095] The first fusion submodule is used to fuse the fourth point cloud data and the reference point cloud data to obtain the third point cloud data, and to obtain the third detection box of the third point cloud data based on the perception algorithm.

[0096] In one embodiment, the fusion module 30 includes:

[0097] The labeling submodule is used to label the first detection box of the first point cloud data based on the perception algorithm;

[0098] The second transformation submodule is used to transform the first point cloud data and the first detection box to the second reference coordinate system where the reference point cloud data is located, using the second point cloud data as the reference point cloud data.

[0099] The second matching submodule is used to perform feature matching between the first point cloud data and the reference point cloud data in the second reference coordinate system, and adjust the coordinate information of the first point cloud data and the pose of the first detection box according to the matching result to obtain the fourth point cloud data and the fourth detection box.

[0100] The second fusion submodule is used to fuse the fourth point cloud data and the second point cloud data to obtain the third point cloud data, and adjust the scaling ratio of the fourth detection box based on the third point cloud data to obtain the third detection box of the third point cloud data.

[0101] In one embodiment, the first matching submodule or the second matching submodule includes:

[0102] The second matching unit is used to perform feature matching between the first point cloud data and the reference point cloud data in the second reference coordinate system, and obtain matching point pairs in the first point cloud data and the reference point cloud data according to the matching result;

[0103] The adjustment unit is used to adjust the coordinate information of all data points in the first point cloud data according to the coordinate difference of the matching point pairs.

[0104] In one embodiment, the data processing apparatus further includes:

[0105] The training module is used to train, test, or verify the perception algorithm or another target algorithm based on the second point cloud data and the third detection box.

[0106] Unlike existing technologies, the data processing device provided in this application obtains relatively dense and information-rich first point cloud data under the first acquisition condition, and relatively sparse and information-scarce second point cloud data under the second acquisition condition. Therefore, after fusing the first and second point cloud data, the resulting third point cloud data simultaneously contains information from both the first and second point cloud data, and its information is also relatively rich. Thus, compared to the scheme of directly labeling the sparse second point cloud data to obtain the second detection box, this application first obtains the third detection box based on the dense third point cloud data, and then labels the third detection box onto the second point cloud data. This effectively improves the labeling accuracy of the third detection box, and the entire process is automatically calculated based on a perception algorithm, eliminating the need for manual labeling and improving labeling efficiency. In other words, this application achieves a balance between labeling accuracy and labeling efficiency.

[0107] Accordingly, embodiments of this application also provide an electronic device, such as... Figure 12 As shown, the electronic device may include a radio frequency (RF) circuit 101, a memory 102 including one or more computer-readable storage media, an input unit 103, a display unit 104, a sensor 105, an audio circuit 106, a WiFi module 107, a processor 108 including one or more processing cores, and a power supply 109, among other components. Those skilled in the art will understand that... Figure 12 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0108] The radio frequency circuit 101 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 108 for processing; additionally, it transmits uplink data to the base station. The memory 102 can be used to store software programs and modules. The processor 108 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The input unit 103 can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to customer settings and function control.

[0109] Display unit 104 can be used to display information input by the customer or information provided to the customer, as well as various graphical client interfaces of the server. These graphical client interfaces can be composed of graphics, text, icons, videos, and any combination thereof.

[0110] The electronic device may also include at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Audio circuitry 106 includes a speaker that provides an audio interface between the client and the electronic device.

[0111] WiFi is a short-range wireless transmission technology. Electronic devices using the WiFi module 107 can help customers send and receive emails, browse web pages, and access streaming media, providing customers with wireless broadband internet access. Although Figure 12 WiFi module 107 is shown, but it is understood that it is not a necessary component of the electronic device and can be omitted as needed without changing the nature of the application.

[0112] The processor 108 is the control center of the electronic device. It connects various parts of the phone through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 102, and calling data stored in the memory 102, it performs various functions of the electronic device and processes data, thereby monitoring the phone as a whole.

[0113] The electronic device also includes a power supply 109 (such as a battery) that supplies power to the various components. Preferably, the power supply can be logically connected to the processor 108 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0114] Although not shown, electronic devices may also include cameras, Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 108 in the server loads the executable files corresponding to the processes of one or more applications into the memory 102 according to the following instructions, and the processor 108 runs the applications stored in the memory 102 to achieve the following functions:

[0115] Acquire first point cloud data of a target object by radar under first acquisition conditions, wherein the first acquisition conditions include a first distance between the target object and the radar being less than a first threshold.

[0116] The radar acquires second point cloud data of the target object under a second acquisition condition, wherein the second acquisition condition includes: the number of data points in the second point cloud data is not greater than a second threshold, and / or: the second distance between the target object and the radar is greater than a third threshold, wherein the third threshold is greater than or equal to the first threshold;

[0117] By fusing the first point cloud data and the second point cloud data, and based on the fusion result and the perception algorithm, a third point cloud data and a third detection box of the third point cloud data are obtained;

[0118] The third detection box is marked on the second point cloud data.

[0119] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description above, and they will not be repeated here.

[0120] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0121] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to perform the following functions:

[0122] Acquire first point cloud data of a target object by radar under first acquisition conditions, wherein the first acquisition conditions include a first distance between the target object and the radar being less than a first threshold.

[0123] The radar acquires second point cloud data of the target object under a second acquisition condition, wherein the second acquisition condition includes: the number of data points in the second point cloud data is not greater than a second threshold, and / or: the second distance between the target object and the radar is greater than a third threshold, wherein the third threshold is greater than or equal to the first threshold;

[0124] By fusing the first point cloud data and the second point cloud data, and based on the fusion result and the perception algorithm, a third point cloud data and a third detection box of the third point cloud data are obtained;

[0125] The third detection box is marked on the second point cloud data.

[0126] The data processing method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of this application. Those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized by, include: Acquire first point cloud data of a target object by radar under first acquisition conditions, wherein the first acquisition conditions include a first distance between the target object and the radar being less than a first threshold. The radar acquires second point cloud data of the target object under a second acquisition condition, wherein the second acquisition condition includes: the number of data points in the second point cloud data is not greater than a second threshold, and / or: the second distance between the target object and the radar is greater than a third threshold, wherein the third threshold is greater than or equal to the first threshold; The first point cloud data and the second point cloud data are fused together, and based on the fusion result and the perception algorithm, a third point cloud data and a third detection box of the third point cloud data are obtained; the third point cloud data is in the second reference coordinate system where the second point cloud data is located; The third detection box is marked on the second point cloud data; The step of fusing the first point cloud data and the second point cloud data, and obtaining the third point cloud data and the third detection box of the third point cloud data based on the fusion result and the perception algorithm, includes: Using the second point cloud data as reference point cloud data, the first point cloud data is transformed into the second reference coordinate system where the reference point cloud data is located; In the second reference coordinate system, the first point cloud data and the reference point cloud data are matched for features, and the coordinate information of the first point cloud data is adjusted according to the matching result to obtain the fourth point cloud data; The fourth point cloud data and the reference point cloud data are fused to obtain the third point cloud data, and the third detection box of the third point cloud data is obtained based on the perception algorithm. The step of fusing the first point cloud data and the second point cloud data, and obtaining the third point cloud data and the third detection box of the third point cloud data based on the fusion result and the perception algorithm, further includes: The first detection box is labeled based on the perception algorithm to mark the first point cloud data; Using the second point cloud data as reference point cloud data, the first point cloud data and the first detection box are transformed to the second reference coordinate system where the reference point cloud data is located; In the second reference coordinate system, the first point cloud data and the reference point cloud data are feature matched, and the coordinate information of the first point cloud data and the pose of the first detection box are adjusted according to the matching result to obtain the fourth point cloud data and the fourth detection box. By fusing the fourth point cloud data and the second point cloud data, a third point cloud data is obtained. Based on the third point cloud data, the scaling ratio of the fourth detection box is adjusted to obtain a third detection box of the third point cloud data.

2. The data processing method according to claim 1, characterized in that, The steps for acquiring the first point cloud data of the target object obtained by the radar under the first acquisition condition include: Acquire the first point cloud data obtained by the radar from the target angle of the target object; Or include: At least two sets of sub-point cloud data are acquired by radar from at least two different angles of the target object, and the at least two sets of sub-point cloud data are fused to obtain the first point cloud data.

3. The data processing method according to claim 2, characterized in that, The steps of acquiring at least two sets of sub-point cloud data collected by radar from at least two different angles of the target object, and fusing the at least two sets of sub-point cloud data to obtain a first point cloud data, include: A reference sub-point cloud data is determined from the at least two sets of sub-point cloud data, and the other sub-point cloud data in the at least two sets of sub-point cloud data, excluding the reference sub-point cloud data, are transformed to the first reference coordinate system where the reference sub-point cloud data is located. In the first reference coordinate system, feature matching is performed between the other sub-point cloud data and the reference sub-point cloud data, and the coordinate information of the other sub-point cloud data is adjusted according to the matching result; The first point cloud data is obtained by fusing and adjusting the other sub-point cloud data with the reference sub-point cloud data.

4. The data processing method of claim 1, wherein, In the second reference coordinate system, the step of performing feature matching between the first point cloud data and the reference point cloud data, and adjusting the coordinate information of the first point cloud data according to the matching result, includes: In the second reference coordinate system, feature matching is performed between the first point cloud data and the reference point cloud data, and matching point pairs in the first point cloud data and the reference point cloud data are obtained based on the matching results; Based on the coordinate difference of the matched point pairs, adjust the coordinate information of all data points in the first point cloud data.

5. The data processing method of claim 1, wherein, After the step of marking the third detection box on the second point cloud data, the following steps are included: Based on the second point cloud data and the third detection box, the perception algorithm or another target algorithm is trained, tested, or verified.

6. A data processing apparatus, characterized in that, include: The first acquisition module is used to acquire first point cloud data of the target object collected by the radar under first acquisition conditions, wherein the first acquisition conditions include a first distance between the target object and the radar being less than a first threshold. The second acquisition module is used to acquire second point cloud data collected by the radar on the target object under second acquisition conditions. The second acquisition conditions include: the number of data points in the second point cloud data is not greater than a second threshold, and / or: the second distance between the target object and the radar is greater than a third threshold, and the third threshold is greater than or equal to the first threshold. The fusion module is used to fuse the first point cloud data and the second point cloud data, and obtain the third point cloud data and the third detection box of the third point cloud data based on the fusion result and the perception algorithm; the third point cloud data is in the second reference coordinate system where the second point cloud data is located; A marking module is used to mark the third detection box on the second point cloud data; The fusion module is further configured to: use the second point cloud data as reference point cloud data; transform the first point cloud data to the second reference coordinate system in which the reference point cloud data is located; perform feature matching between the first point cloud data and the reference point cloud data in the second reference coordinate system; adjust the coordinate information of the first point cloud data according to the matching result to obtain the fourth point cloud data; fuse the fourth point cloud data and the reference point cloud data to obtain the third point cloud data; and obtain the third detection box of the third point cloud data based on a perception algorithm. The fusion module is further configured to: label the first detection box of the first point cloud data based on a perception algorithm; transform the first point cloud data and the first detection box to the second reference coordinate system where the reference point cloud data is located, using the second point cloud data as reference point cloud data; perform feature matching between the first point cloud data and the reference point cloud data in the second reference coordinate system; adjust the coordinate information of the first point cloud data and the pose of the first detection box according to the matching result to obtain the fourth point cloud data and the fourth detection box; fuse the fourth point cloud data and the second point cloud data to obtain the third point cloud data; and adjust the scaling ratio of the fourth detection box based on the third point cloud data to obtain the third detection box of the third point cloud data.

7. An electronic device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor runs the application program within the memory to perform the steps of the data processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the steps of the data processing method according to any one of claims 1 to 5.

Images