Point cloud data fusion method, device, electronic device and storage medium

Abstract

The present disclosure provides a method, device, electronic device and storage medium for fusing point cloud data, the method comprising: acquiring point cloud data respectively collected by a main radar and a secondary radar arranged on a target vehicle; the main radar is one of the radars on the target vehicle, and the secondary radar is a radar other than the main radar among the radars on the target vehicle; based on a predetermined reflectivity calibration table of the secondary radar, adjusting the reflectivity in the point cloud data collected by the secondary radar to obtain adjusted point cloud data of the secondary radar; wherein the reflectivity calibration table represents target reflectivity information of the main radar matched by each reflectivity corresponding to each scanning line of the secondary radar; fusing the point cloud data collected by the main radar with the adjusted point cloud data corresponding to the secondary radar to obtain fused point cloud data.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method, device, electronic device and storage medium for fusing point cloud data. Background Art

[0002] LiDAR detects the position of a target by reflecting laser beams. It has the characteristics of long detection distance and high measurement accuracy, so it can be widely used in the field of autonomous driving.

[0003] Generally, in order to reduce the detection blind area and increase the detection distance, multiple laser radars can be installed on the vehicle. The manufacturers or models of the installed laser radars may be different, which leads to inconsistent standards for measuring reflectivity of the multiple laser radars, and inconsistent standards for measuring reflectivity corresponding to different fused point cloud data, resulting in distortion of the target represented by the fused point cloud data. When performing tasks such as target detection, target tracking, and high-precision map construction based on the fused point cloud data, the execution result accuracy is low. Summary of the invention

[0004] In view of this, the present disclosure at least provides a point cloud data fusion method, device, electronic device and storage medium.

[0005] In a first aspect, the present disclosure provides a method for fusing point cloud data, comprising:

[0006] Acquire point cloud data respectively collected by a main radar and a secondary radar installed on the target vehicle; the main radar is one of the radars on the target vehicle, and the secondary radar is a radar other than the main radar among the radars on the target vehicle;

[0007] Based on a predetermined reflectivity calibration table of the secondary radar, the reflectivity in the point cloud data collected by the secondary radar is adjusted to obtain the adjusted point cloud data of the secondary radar; wherein the reflectivity calibration table represents the target reflectivity information of the primary radar matched by each reflectivity corresponding to each scanning line of the secondary radar;

[0008] The point cloud data collected by the main radar is fused with the adjusted point cloud data corresponding to the secondary radar to obtain fused point cloud data.

[0009] By adopting the above method, a reflectivity calibration table is generated in advance, and the reflectivity calibration table represents the target reflectivity information of the main radar matched by each reflectivity corresponding to each scanning line of the secondary radar. Therefore, after obtaining the point cloud data collected by the secondary radar, the reflectivity in the point cloud data collected by the secondary radar can be adjusted according to the reflectivity calibration table, so that the measurement standards corresponding to the reflectivity in the point cloud data collected by the main radar and the adjusted point cloud data collected by the secondary radar are consistent, thereby alleviating the distortion problem of the fused point cloud data and improving the accuracy of target detection, etc.

[0010] In a possible implementation manner, the reflectivity calibration table is determined according to the following steps:

[0011] Acquire first sample point cloud data collected by the primary radar disposed on the sample vehicle, and second sample point cloud data collected by the secondary radar disposed on the sample vehicle;

[0012] Based on the first sample point cloud data, generating voxel map data, wherein the voxel map data includes data of a plurality of three-dimensional voxel grids, and the data of each three-dimensional voxel grid includes reflectivity information determined based on point cloud data of a plurality of scanning points in the three-dimensional voxel grid;

[0013] The reflectivity calibration table is generated based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids.

[0014] The above-mentioned embodiment provides a method for generating a reflectivity calibration table, by generating voxel map data based on the first sample point cloud data, obtaining the reflectivity information of the first sample point cloud data on each three-dimensional voxel grid, and then generating a reflectivity calibration table based on the second sample point cloud data and the voxel map data. The reflectivity calibration table can more accurately reflect the target reflectivity information of the main radar that matches each reflectivity of each scanning line of the secondary radar, that is, the generated reflectivity calibration table has a higher accuracy.

[0015] In a possible implementation manner, generating voxel map data based on the first sample point cloud data includes:

[0016] Acquire a plurality of position and posture data collected sequentially during the movement of the sample vehicle;

[0017] Performing dedistortion processing on the first sample point cloud data based on the multiple pose data to obtain processed first sample point cloud data;

[0018] Voxel map data is generated based on the processed first sample point cloud data.

[0019] In the above embodiment, the de-distortion processing process can eliminate the deviation caused by the different radar positions corresponding to the first sample point cloud data of different frames and the first sample point cloud data of different batches in each frame of the first sample point cloud data, so that the processed first sample point cloud data can be understood as the first sample point cloud data measured at the same radar position, so that when voxel map data is generated based on the first sample point cloud data obtained after the de-distortion processing, the accuracy of the generated voxel map data can be improved, and thus the accuracy of the generated reflectivity calibration table can be higher.

[0020] In a possible implementation manner, the reflectivity information includes a reflectivity average value, and the data of each three-dimensional voxel grid included in the voxel map data is determined according to the following steps:

[0021] For each of the three-dimensional voxel grids, based on the reflectivity in the point cloud data of each scanning point in the three-dimensional voxel grid, determine an average reflectivity value corresponding to the three-dimensional voxel grid;

[0022] Generating the reflectivity calibration table based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids includes:

[0023] For each reflectivity of each scanning line of the secondary radar, determine the position information of multiple target scanning points corresponding to the reflectivity from the second sample point cloud data, wherein the multiple target scanning points are scanning points obtained by scanning the scanning line; based on the position information of the multiple target scanning points, determine at least one three-dimensional voxel grid corresponding to the multiple target scanning points; based on the average values ​​of the reflectivities respectively corresponding to the at least one three-dimensional voxel grid, determine the target reflectivity information of the primary radar matched by the reflectivity of the scanning line;

[0024] The reflectivity calibration table is generated based on the target reflectivity information of the primary radar matched with each reflectivity of each scanning line of the determined secondary radar.

[0025] Generally, when the scan lines emitted by different radars hit the same object, the corresponding reflectivity should be consistent, that is, it can be considered that in the same three-dimensional voxel grid, the reflectivity of the scan point obtained by the main radar scan is consistent with the reflectivity of the scan point obtained by the secondary radar scan. Therefore, at least one three-dimensional voxel grid corresponding to each reflectivity of each scan line of the secondary radar can be determined, and based on the average reflectivity corresponding to at least one three-dimensional voxel grid, the target reflectivity information of the main radar that matches the reflectivity of the scan line can be more accurately determined, and then a more accurate reflectivity calibration table can be generated.

[0026] In a possible implementation manner, the data of the three-dimensional voxel grid includes the reflectivity average value and a weight influence factor, and the weight influence factor includes a reflectivity variance and / or a number of scanning points;

[0027] In a case where the at least one three-dimensional voxel grid is a plurality of three-dimensional voxel grids, determining the target reflectivity information of the primary radar that matches the reflectivity of the scan line based on the average reflectivity values ​​respectively corresponding to the at least one three-dimensional voxel grids, includes:

[0028] Determining a weight corresponding to each of the three-dimensional voxel grids in the at least one three-dimensional voxel grid based on the weight influencing factor;

[0029] Based on the weight corresponding to each three-dimensional voxel grid and the corresponding reflectivity average value, the target reflectivity information of the primary radar matched by the reflectivity of the scan line is determined.

[0030] Here, a weight can be determined for each three-dimensional voxel grid, and the weight of the three-dimensional voxel grid with higher credibility is set to be larger (for example, the three-dimensional voxel grid with smaller reflectivity variance and more scanning points has higher credibility), and the weight of the three-dimensional voxel grid with lower credibility is set to be smaller, so that based on the weight corresponding to each three-dimensional voxel grid and the average reflectivity, the target reflectivity information of the main radar matching the reflectivity of the scan line can be determined more accurately, and then the accuracy of the obtained reflectivity calibration table can be higher.

[0031] In a possible implementation, generating the reflectivity calibration table based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids includes:

[0032] Acquire a plurality of posture data sequentially collected during the movement of the sample vehicle, and perform dedistortion processing on the second sample point cloud data based on the plurality of posture data to obtain processed second sample point cloud data;

[0033] Determine relative position information between the first sample point cloud data and the second sample point cloud data based on position information of the primary radar on the sample vehicle and position information of the secondary radar on the sample vehicle;

[0034] Using the relative position information, coordinate transformation is performed on the processed second sample point cloud data to obtain second sample point cloud data in a target coordinate system; wherein the target coordinate system is a coordinate system corresponding to the first sample point cloud data;

[0035] The reflectivity calibration table is generated based on the second sample point cloud data in the target coordinate system and the data of the multiple three-dimensional voxel grids.

[0036] In the above implementation, the second sample point cloud data is first dedistorted to eliminate the deviation caused by the different radar positions corresponding to each batch of second sample point cloud data and each frame of second sample point cloud data in the second sample point cloud data; then the second sample point cloud data is converted to the target coordinate system corresponding to the first sample point cloud data to eliminate the deviation caused by the different radar positions corresponding to the second sample point cloud data and the first sample point cloud data, so that when a reflectivity calibration table is generated based on the second sample point cloud data obtained after the dedistortion processing and coordinate conversion, the accuracy of the generated reflectivity calibration table can be improved.

[0037] In a possible implementation manner, the first sample point cloud data and the second sample point cloud data are respectively used as target sample point cloud data; when the target sample point cloud data is the first sample point cloud data, the main radar is used as the target radar; when the target sample point cloud data is the second sample point cloud data, the secondary radar is used as the target radar; the target sample point cloud data has multiple frames, and each frame of target sample point cloud data includes target sample point cloud data collected by the target radar transmitting multiple scan lines; wherein the target radar transmits scan lines in batches according to a preset frequency, and each batch transmits multiple scan lines;

[0038] Dedistort the target sample point cloud data according to the following steps:

[0039] Based on the plurality of posture data, determining the posture information of the target radar when transmitting each batch of scan lines;

[0040] For the target sample point cloud data collected by the non-first batch of transmitted scan lines in each frame of target sample point cloud data, based on the posture information of the target radar when transmitting the batch of scan lines, the coordinates of the target sample point cloud data collected by transmitting the batch of scan lines are converted to the coordinate system of the target radar corresponding to the target sample point cloud data collected by transmitting the first batch of scan lines in the frame of target sample point cloud data, so as to obtain the target sample point cloud data corresponding to the frame of target sample point cloud data after the first dedistortion;

[0041] For any non-first frame target sample point cloud data in the multiple frames of target sample point cloud data after the first dedistortion, based on the posture information of the target radar when scanning to obtain the frame target sample point cloud data, the coordinates of the frame target sample point cloud data are converted to the coordinate system of the target radar corresponding to the first frame target sample point cloud data to obtain the target sample point cloud data after the second dedistortion corresponding to the frame target sample point cloud data.

[0042] In the above implementation, the target sample point cloud data collected by non-first batch scanning lines in each frame of target sample point cloud data and the non-first frame target sample point cloud data in different frames of target sample point cloud data are uniformly transformed into the coordinate system of the target radar corresponding to the first batch of target sample point cloud data in the first frame of target sample point cloud data, thereby improving the accuracy of the generated reflectivity calibration table.

[0043] In a possible implementation manner, after generating the reflectivity calibration table, the following steps are included:

[0044] In the reflectivity calibration table, determining the reflectivity of a scan line for which no matching target reflectivity information exists;

[0045] Based on the target reflectivity information of the primary radar in the reflectivity calibration table, determining the target reflectivity information of the primary radar corresponding to the reflectivity of the scanning line for which no matching target reflectivity information exists;

[0046] The reflectivity calibration table is updated based on the target reflectivity information of the primary radar corresponding to the reflectivity of the scan line for which no matching target reflectivity information is determined.

[0047] Under the above implementation mode, since there may be some grids in the generated reflectivity calibration table without corresponding target reflectivity information, that is, there may be a situation where the generated reflectivity calibration table is incomplete, in order to ensure the integrity of the reflectivity calibration table, the target reflectivity information missing in the reflectivity calibration table can be determined based on the target reflectivity information of the main radar in the reflectivity calibration table, and the reflectivity calibration table can be completed to generate an updated reflectivity calibration table, that is, a complete reflectivity calibration table is obtained.

[0048] The effects of the following devices, electronic devices, etc. are described in the description of the above method and will not be repeated here.

[0049] In a second aspect, the present disclosure provides a point cloud data fusion device, comprising:

[0050] An acquisition module, used to acquire point cloud data respectively collected by a main radar and a secondary radar installed on the target vehicle; the main radar is one of the radars on the target vehicle, and the secondary radar is a radar other than the main radar among the radars on the target vehicle;

[0051] an adjustment module, configured to adjust the reflectivity in the point cloud data collected by the secondary radar based on a predetermined reflectivity calibration table of the secondary radar, so as to obtain the adjusted point cloud data of the secondary radar; wherein the reflectivity calibration table represents the target reflectivity information of the primary radar matched by each reflectivity corresponding to each scanning line of the secondary radar;

[0052] The fusion module is used to fuse the point cloud data collected by the primary radar with the adjusted point cloud data of the secondary radar to obtain fused point cloud data.

[0053] In a possible implementation manner, the fusion device further includes: a reflectivity calibration and determination module;

[0054] The reflectivity calibration determination module is used to determine the reflectivity calibration table according to the following steps:

[0055] Acquire first sample point cloud data collected by the primary radar disposed on the sample vehicle, and second sample point cloud data collected by the secondary radar disposed on the sample vehicle;

[0056] Based on the first sample point cloud data, generating voxel map data, wherein the voxel map data includes data of a plurality of three-dimensional voxel grids, and the data of each three-dimensional voxel grid includes reflectivity information determined based on point cloud data of a plurality of scanning points in the three-dimensional voxel grid;

[0057] The reflectivity calibration table is generated based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids.

[0058] In a possible implementation manner, the reflectivity calibration determination module, when generating voxel map data based on the first sample point cloud data, is used to:

[0059] Acquire a plurality of position and posture data collected sequentially during the movement of the sample vehicle;

[0060] Performing dedistortion processing on the first sample point cloud data based on the multiple pose data to obtain processed first sample point cloud data;

[0061] Voxel map data is generated based on the processed first sample point cloud data.

[0062] In a possible implementation manner, the reflectivity information includes a reflectivity average value, and the reflectivity calibration determination module is used to determine the data of each three-dimensional voxel grid included in the voxel map data according to the following steps:

[0063] For each of the three-dimensional voxel grids, based on the reflectivity in the point cloud data of each scanning point in the three-dimensional voxel grid, determine an average reflectivity value corresponding to the three-dimensional voxel grid;

[0064] The reflectivity calibration determination module, when generating the reflectivity calibration table based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids, is used to:

[0065] For each reflectivity of each scanning line of the secondary radar, determine the position information of multiple target scanning points corresponding to the reflectivity from the second sample point cloud data, wherein the multiple target scanning points are scanning points obtained by scanning the scanning line; based on the position information of the multiple target scanning points, determine at least one three-dimensional voxel grid corresponding to the multiple target scanning points; based on the average values ​​of the reflectivities respectively corresponding to the at least one three-dimensional voxel grid, determine the target reflectivity information of the primary radar matched by the reflectivity of the scanning line;

[0066] The reflectivity calibration table is generated based on the target reflectivity information of the primary radar matched with each reflectivity of each scanning line of the determined secondary radar.

[0067] In a possible implementation manner, the data of the three-dimensional voxel grid includes the reflectivity average value and a weight influence factor, and the weight influence factor includes a reflectivity variance and / or a number of scanning points;

[0068] In the case where the at least one three-dimensional voxel grid is a plurality of three-dimensional voxel grids, the reflectivity calibration determination module, when determining the target reflectivity information of the primary radar that matches the reflectivity of the scan line based on the reflectivity average values ​​respectively corresponding to the at least one three-dimensional voxel grids, is used to:

[0069] Determining a weight corresponding to each of the three-dimensional voxel grids in the at least one three-dimensional voxel grid based on the weight influencing factor;

[0070] Based on the weight corresponding to each three-dimensional voxel grid and the corresponding reflectivity average value, the target reflectivity information of the primary radar matched by the reflectivity of the scan line is determined.

[0071] In a possible implementation manner, the reflectivity calibration determination module, when generating the reflectivity calibration table based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids, is used to:

[0072] Acquire a plurality of posture data sequentially collected during the movement of the sample vehicle, and perform dedistortion processing on the second sample point cloud data based on the plurality of posture data to obtain processed second sample point cloud data;

[0073] Determine relative position information between the first sample point cloud data and the second sample point cloud data based on position information of the primary radar on the sample vehicle and position information of the secondary radar on the sample vehicle;

[0074] Using the relative position information, coordinate transformation is performed on the processed second sample point cloud data to obtain second sample point cloud data in a target coordinate system; wherein the target coordinate system is a coordinate system corresponding to the first sample point cloud data;

[0075] The reflectivity calibration table is generated based on the second sample point cloud data in the target coordinate system and the data of the multiple three-dimensional voxel grids.

[0076] In a possible implementation manner, the first sample point cloud data and the second sample point cloud data are respectively used as target sample point cloud data; when the target sample point cloud data is the first sample point cloud data, the main radar is used as the target radar; when the target sample point cloud data is the second sample point cloud data, the secondary radar is used as the target radar; the target sample point cloud data has multiple frames, and each frame of target sample point cloud data includes target sample point cloud data collected by the target radar transmitting multiple scan lines; wherein the target radar transmits scan lines in batches according to a preset frequency, and each batch transmits multiple scan lines;

[0077] The reflectivity calibration determination module is used to perform dedistortion processing on the target sample point cloud data according to the following steps:

[0078] Based on the plurality of posture data, determining the posture information of the target radar when transmitting each batch of scan lines;

[0079] For the target sample point cloud data collected by the non-first batch of transmitted scan lines in each frame of target sample point cloud data, based on the posture information of the target radar when transmitting the batch of scan lines, the coordinates of the target sample point cloud data collected by transmitting the batch of scan lines are converted to the coordinate system of the target radar corresponding to the target sample point cloud data collected by transmitting the first batch of scan lines in the frame of target sample point cloud data, so as to obtain the target sample point cloud data after the first dedistortion of the frame of target sample point cloud data;

[0080] For any non-first frame target sample point cloud data in the multiple frames of target sample point cloud data after the first dedistortion, based on the posture information of the target radar when scanning to obtain the frame target sample point cloud data, the coordinates of the frame target sample point cloud data are converted to the coordinate system of the target radar corresponding to the first frame target sample point cloud data to obtain the target sample point cloud data after the second dedistortion corresponding to the frame target sample point cloud data.

[0081] In a possible implementation manner, after generating the reflectivity calibration table, the method further includes: an updating module, configured to:

[0082] In the reflectivity calibration table, determining the reflectivity of a scan line for which no matching target reflectivity information exists;

[0083] Based on the target reflectivity information of the primary radar in the reflectivity calibration table, determining the target reflectivity information of the primary radar corresponding to the reflectivity of the scanning line for which no matching target reflectivity information exists;

[0084] The reflectivity calibration table is updated based on the target reflectivity information of the primary radar corresponding to the reflectivity of the scan line for which no matching target reflectivity information is determined.

[0085] In a third aspect, the present disclosure provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the point cloud data fusion method described in the first aspect or any one of the embodiments are performed.

[0086] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the point cloud data fusion method as described in the first aspect or any one of the embodiments above are executed.

[0087] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0089] Figure 1 A schematic diagram of a process of a point cloud data fusion method provided by an embodiment of the present disclosure is shown;

[0090] Figure 2 A schematic flow chart of a method for determining a reflectivity calibration table in a point cloud data fusion method provided in an embodiment of the present disclosure is shown;

[0091] Figure 3 A schematic diagram of the architecture of a point cloud data fusion device provided by an embodiment of the present disclosure is shown;

[0092] Figure 4A schematic structural diagram of an electronic device 400 provided in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0094] Generally, multiple radars can be set up on the target vehicle, and each radar collects point cloud data separately. The point cloud data collected by multiple radars are fused to obtain richer fused point cloud data, and then target detection or target tracking can be performed based on the fused point cloud data. However, since the corresponding reflectivity between different radars may be inconsistent, the reflectivity is not uniform when the point cloud data from different sources are fused, and the fused point cloud data is distorted, which reduces the accuracy of the execution result.

[0095] The radars in the embodiments of the present disclosure include laser radars, millimeter wave radars, ultrasonic radars, etc. The radars for point cloud data fusion can be radars of the same type or radars of different types. The embodiments of the present disclosure are described by taking the example that the radars for point cloud data fusion are all laser radars.

[0096] In order to solve the problem of inconsistent reflectivity when fusing point cloud data from different sources, an embodiment of the present disclosure provides a method for fusing point cloud data.

[0097] To facilitate understanding of the embodiments of the present disclosure, a point cloud data fusion method disclosed in the embodiments of the present disclosure is first introduced in detail.

[0098] See also Figure 1 FIG. 1 is a flow chart of a method for fusing point cloud data provided by an embodiment of the present disclosure, the method comprising S101-S103, wherein:

[0099] S101, acquiring point cloud data respectively collected by a primary radar and a secondary radar installed on a target vehicle; the primary radar is one of the radars on the target vehicle, and the secondary radar is a radar other than the primary radar among the radars on the target vehicle;

[0100] S102, based on a predetermined reflectivity calibration table corresponding to the secondary radar, adjusting the reflectivity in the point cloud data collected by the secondary radar to obtain adjusted point cloud data of the secondary radar; wherein the reflectivity calibration table represents target reflectivity information of the primary radar matched by each reflectivity corresponding to each scanning line of the secondary radar;

[0101] S103, fusing the point cloud data collected by the primary radar with the adjusted point cloud data corresponding to the secondary radar to obtain fused point cloud data, so as to control the target vehicle according to the fused point cloud data.

[0102] In practical applications, target detection and tracking can be performed based on the fused point cloud data, and the target vehicle can be controlled based on the detection and tracking results.

[0103] In the above method, a reflectivity calibration table is generated in advance, and the reflectivity calibration table represents the target reflectivity information of the main radar matched by each reflectivity corresponding to each scanning line of the secondary radar. Therefore, after obtaining the point cloud data collected by the secondary radar, the reflectivity in the point cloud data collected by the secondary radar can be adjusted according to the reflectivity calibration table, so that the measurement standards corresponding to the reflectivity in the point cloud data collected by the main radar and the adjusted point cloud data collected by the secondary radar are consistent, thereby alleviating the distortion problem of the fused point cloud data and improving the accuracy of target detection, etc.

[0104] S101 to S103 are described in detail below.

[0105] For S101:

[0106] The main radar and the secondary radar may be radars set at different positions on the target vehicle, and the main radar and the secondary radar may be multi-line radars. The types and locations of the main radar and the secondary radar may be set according to actual needs, and the number of secondary radars may be multiple. For example, the main radar may be a laser radar set at the exact middle of the target vehicle, i.e., the main laser radar, and the two secondary radars may be laser radars set at both sides of the target vehicle, i.e., secondary laser radars; the main radar may be a laser radar with 16 lines, 32 lines, 64 lines, or 128 lines, etc., and the secondary radar may be a laser radar with 16 lines, 32 lines, 64 lines, or 128 lines, etc.

[0107] After the main radar and the secondary radar collect point cloud data, the point cloud data collected by the main radar and the secondary radar can be obtained. Generally, the point cloud data collected by the main radar includes data corresponding to multiple scanning points, and the data corresponding to each scanning point includes the position information and reflectivity of the scanning point in the rectangular coordinate system corresponding to the main radar; the point cloud data collected by the secondary radar includes the position information and reflectivity of the scanning point in the rectangular coordinate system corresponding to the secondary radar.

[0108] For S102 and S103:

[0109] After obtaining the point cloud data corresponding to the main radar and the secondary radar respectively, the point cloud data corresponding to the secondary radar is converted so that the converted point cloud data and the point cloud data collected by the main radar are located in the same coordinate system, that is, the converted point cloud data is located in the rectangular coordinate system corresponding to the main radar. Then, the reflectivity in the point cloud data collected by the secondary radar can be adjusted using the predetermined reflectivity calibration table of the secondary radar to obtain the adjusted point cloud data corresponding to the secondary radar. Then, the point cloud data collected by the main radar is fused with the adjusted point cloud data corresponding to the secondary radar to obtain the fused point cloud data.

[0110] If there are multiple secondary radars, a corresponding reflectivity calibration table can be generated for each secondary radar, and the reflectivity calibration table corresponding to each secondary radar can be used to adjust the point cloud data collected by the corresponding secondary radar to obtain the adjusted point cloud data corresponding to each secondary radar.

[0111] In practical applications, the reflectivity calibration table can be shown in Table 1 below, and the reflectivity calibration table can be a reflectivity calibration table corresponding to the 16-line secondary laser radar. Among them, Table 1 includes the target reflectivity information of the main laser radar matched with each reflectivity of each scanning line in the secondary laser radar, and 256 reflectivities corresponding to each scanning line (the 256 reflectivities can be a reflectivity of 0, a reflectivity of 1, ..., a reflectivity of 255), that is, the reflectivity calibration table includes target reflectivity information matched with each reflectivity of each scanning line in the 16 lines. The target reflectivity information may include the target reflectivity average, the target reflectivity variance, etc., wherein the target reflectivity average may be a positive integer, and the target reflectivity variance may be a positive real number. For example, the target emissivity information of the main laser radar with scanning line Ring0 and a reflectivity of 0 can be information X00; the target emissivity information of the main laser radar with scanning line Ring15 and a reflectivity of 255 can be information X15255.

[0112] Table 1 Reflectivity calibration table

[0113]

[0114] In an alternative embodiment, see Figure 2 As shown, determine the reflectance calibration table according to the following steps:

[0115] S201, obtaining first sample point cloud data collected by a primary radar disposed on a sample vehicle, and second sample point cloud data collected by a secondary radar disposed on the sample vehicle.

[0116] S202, generating voxel map data based on the first sample point cloud data, wherein the voxel map data includes data of a plurality of three-dimensional voxel grids, and the data of each three-dimensional voxel grid includes reflectivity information determined based on point cloud data of a plurality of scanning points in the three-dimensional voxel grid.

[0117] S203: Generate a reflectivity calibration table based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids.

[0118] S201 is explained. The sample vehicle can be the same vehicle as the target vehicle or a different vehicle. The sample vehicle equipped with a main radar and a secondary radar can be controlled to travel a preset distance on a preset road to obtain first sample point cloud data and second sample point cloud data. If there are multiple secondary radars, the second sample point cloud data corresponding to each secondary radar can be obtained.

[0119] S202 and S203 are explained. Voxel map data can be generated based on the first sample point cloud data. In specific implementation, the range of the voxel map can be determined according to the first sample point cloud data. For example, if the first sample point cloud data is sample point cloud data within the first distance range, the second distance range corresponding to the voxel map can be determined from the first distance range, wherein the second distance range corresponding to the voxel map is within the first distance range. The voxel map of the second distance range is then divided to obtain a plurality of three-dimensional voxel grids within the second distance range, and the initial data of each three-dimensional voxel grid is determined, that is, the initial data of each three-dimensional voxel grid is set to a preset initial value. For example, when the data of the three-dimensional voxel grid includes the reflectivity average value, the reflectivity variance and the number of scanning points, the initial data of each three-dimensional voxel grid can be a reflectivity average value of 0, a reflectivity variance of 0, and a scanning point number of 0. Then, the point cloud data of the plurality of scanning points in the first sample point cloud data is used to update the initial data of each three-dimensional voxel grid to obtain the updated data of each three-dimensional voxel grid.

[0120] The above-mentioned embodiment provides a method for generating a reflectivity calibration table, by generating voxel map data based on the first sample point cloud data, obtaining the reflectivity information of the first sample point cloud data on each three-dimensional voxel grid, and then generating a reflectivity calibration table based on the second sample point cloud data and the voxel map data. The reflectivity calibration table can more accurately reflect the target reflectivity information of the main radar that matches each reflectivity of each scanning line of the secondary radar, that is, the generated reflectivity calibration table has a higher accuracy.

[0121] In an optional implementation, generating voxel map data based on the first sample point cloud data includes:

[0122] Acquire multiple pose data collected sequentially during the movement of the sample vehicle.

[0123] The first sample point cloud data is subjected to a dedistortion process based on the plurality of position and posture data to obtain processed first sample point cloud data.

[0124] Voxel map data is generated based on the processed first sample point cloud data.

[0125] For example, the sample vehicle may be provided with a positioning device such as a global navigation satellite system-inertial navigation system GNSS-INS, and the sample vehicle may be positioned by the positioning device to obtain a plurality of posture data collected sequentially during the movement of the sample vehicle. Alternatively, the sample vehicle may be controlled to travel at a constant speed, and a plurality of posture data may be calculated based on the time when the main radar or the auxiliary radar transmits and receives the radio beam.

[0126] The first sample point cloud data can be dedistorted using multiple pose data to obtain processed first sample point cloud data. Since the radar obtains point cloud data by scanning the environment in a scanning cycle, when the radar is in motion, the generated point cloud data will be distorted, and the dedistortion method is to transform the obtained point cloud data to the same moment, that is, the dedistorted point cloud data can be considered as the point cloud data obtained at the same moment. Therefore, the processed first sample point cloud data can be understood as the first sample point cloud data obtained at the same moment. Then, voxel map data can be generated based on the processed first sample point cloud data.

[0127] In the above embodiment, the de-distortion processing process can eliminate the deviation caused by the different radar positions corresponding to the first sample point cloud data of different frames and the first sample point cloud data of different batches in each frame of the first sample point cloud data, so that the processed first sample point cloud data can be understood as the first sample point cloud data measured at the same radar position, so that when voxel map data is generated based on the first sample point cloud data obtained after the de-distortion processing, the accuracy of the generated voxel map data can be improved, and thus the accuracy of the generated reflectivity calibration table can be higher.

[0128] In an optional implementation, the reflectivity information includes a reflectivity average value, and the data of each three-dimensional voxel grid included in the voxel map data is determined according to the following steps:

[0129] For each three-dimensional voxel grid, based on the reflectivity in the point cloud data of each scanning point in the three-dimensional voxel grid, an average reflectivity value corresponding to the three-dimensional voxel grid is determined.

[0130] In the disclosed embodiment, the three-dimensional voxel grid where each scanning point is located can be determined according to the position information corresponding to each scanning point in the first sample point cloud data, and the scanning points included in each three-dimensional voxel grid can be obtained. For each three-dimensional voxel grid, the reflectivity of each scanning point in the three-dimensional voxel grid is averaged to obtain the average reflectivity corresponding to the three-dimensional voxel grid.

[0131] In a specific implementation, generating a reflectivity calibration table based on the second sample point cloud data and the data of a plurality of three-dimensional voxel grids may include:

[0132] For each reflectivity of each scanning line of the secondary radar, determine the position information of multiple target scanning points corresponding to the reflectivity from the second sample point cloud data, where the multiple target scanning points are scanning points obtained by scanning the scanning line; based on the position information of the multiple target scanning points, determine at least one three-dimensional voxel grid corresponding to the multiple target scanning points; based on the average reflectivity respectively corresponding to the at least one three-dimensional voxel grid, determine the target reflectivity information of the primary radar matched by the reflectivity of the scanning line;

[0133] A reflectivity calibration table is generated based on the target reflectivity information of the primary radar that is matched with each reflectivity of each scanning line of the determined secondary radar.

[0134] For example, for the scan line Ring1 of the secondary radar with a reflectivity of 1, the scan points obtained by scanning the scan line Ring1 are determined from the second sample point cloud data, and multiple target scan points with a reflectivity of 1 are determined from the scan points that can be scanned from Ring1; at least one three-dimensional voxel grid corresponding to the multiple target scan points is determined according to the position information of the multiple target scan points; based on the reflectivity average values ​​corresponding to at least one three-dimensional voxel grid, the target reflectivity average value and target reflectivity variance (the target reflectivity average value and target reflectivity variance are the target reflectivity information) of the primary radar matching the scan line Ring1 with a reflectivity of 1 can be calculated. Then, a reflectivity calibration table can be generated based on the target reflectivity information of the primary radar matched by each reflectivity of each scan line of the secondary radar.

[0135] In a specific implementation, by traversing the second sample point cloud data, a plurality of target scanning points corresponding to each reflectivity of each scanning line are determined; then, based on the position information of the plurality of target scanning points, at least one three-dimensional voxel grid corresponding to each reflectivity of each scanning line is determined; further, based on the average reflectivity values ​​corresponding to at least one three-dimensional voxel grid corresponding to each reflectivity of each scanning line, the target reflectivity information of the main radar that matches each reflectivity of each scanning line is determined; finally, based on the target reflectivity information of the main radar that matches each reflectivity of each scanning line, a reflectivity calibration table is generated.

[0136] By traversing the second sample point cloud data, multiple target scanning points corresponding to each reflectivity of each scanning line are determined; then based on the position information of the multiple target scanning points, at least one three-dimensional voxel grid corresponding to each reflectivity of each scanning line is determined, that is, at least one three-dimensional voxel grid corresponding to each grid in the reflection calibration table is determined; and then based on the average value of the reflectivity corresponding to at least one three-dimensional voxel grid corresponding to each grid, the target reflectivity information of each grid can be determined, and a reflectivity calibration table is generated.

[0137] Generally, when the radio beams generated by different radars hit the same object, the corresponding reflectivity should be consistent, that is, it can be considered that in the same three-dimensional voxel grid, the reflectivity of the scanning point obtained by the main radar scanning is consistent with the reflectivity of the scanning point obtained by the secondary radar scanning. Therefore, at least one three-dimensional voxel grid corresponding to each reflectivity of each scanning line of the secondary radar can be determined, and based on the average reflectivity corresponding to at least one three-dimensional voxel grid, the target reflectivity information of the main radar that matches the reflectivity of the scanning line can be more accurately determined, and then a more accurate reflectivity calibration table can be generated.

[0138] In an optional implementation, the data of the three-dimensional voxel grid includes a reflectivity average value and a weight influence factor, and the weight influence factor includes a reflectivity variance and / or a number of scanning points.

[0139] In a case where the at least one three-dimensional voxel grid is a plurality of three-dimensional voxel grids, determining the target reflectivity information of the primary radar matching the reflectivity of the scan line based on the average reflectivity values ​​respectively corresponding to the at least one three-dimensional voxel grids includes:

[0140] Based on the weight influencing factor, determining a weight corresponding to each three-dimensional voxel grid in at least one three-dimensional voxel grid;

[0141] Based on the weight corresponding to each three-dimensional voxel grid and the corresponding reflectivity average value, the target reflectivity information of the main radar matched by the reflectivity of the scan line is determined.

[0142] Here, after determining at least one three-dimensional voxel grid corresponding to each reflectivity of each scanning line of the secondary radar, the weight corresponding to each three-dimensional voxel grid in the at least one three-dimensional voxel grid may be determined according to the weight influence factor.

[0143] For example, when the weight influencing factor is the reflectivity variance value, the weight of the three-dimensional voxel grid with a large reflectivity variance can be set smaller, and the weight of the three-dimensional voxel grid with a small reflectivity variance can be set larger. When the weight influencing factor is the number of scanning points, the weight of the three-dimensional voxel grid with a large number of scanning points can be set larger, and the weight of the three-dimensional voxel grid with a small number of scanning points can be set smaller. When the weight influencing factor includes the reflectivity variance and the number of scanning points, the weight of the three-dimensional voxel grid with a small reflectivity variance and a large number of scanning points is set larger, and the weight of the three-dimensional voxel grid with a large reflectivity variance and a small number of scanning point data is set smaller, etc.

[0144] Then, based on the weight and reflectivity average value corresponding to each three-dimensional voxel grid, the target reflectivity average value can be obtained by weighted averaging, and the target reflectivity variance can be obtained by weighted variance, that is, the target reflectivity information of the main radar matched with each reflectivity of each scanning line can be obtained.

[0145] In the above implementation, a weight can be determined for each three-dimensional voxel grid, and the weight of the three-dimensional voxel grid with higher credibility is set to be larger (for example, the three-dimensional voxel grid with smaller reflectivity variance and more scanning points has higher credibility), and the weight of the three-dimensional voxel grid with lower credibility is set to be smaller, so that based on the weight corresponding to each three-dimensional voxel grid and the average reflectivity, the target reflectivity information of the main radar matching the reflectivity of the scan line can be determined more accurately, and thus the accuracy of the obtained reflectivity calibration table can be higher.

[0146] In an optional implementation, generating a reflectivity calibration table based on the second sample point cloud data and the data of a plurality of three-dimensional voxel grids includes:

[0147] 1. Acquire multiple posture data collected sequentially during the movement of the sample vehicle, and perform dedistortion processing on the second sample point cloud data based on the multiple posture data to obtain processed second sample point cloud data.

[0148] 2. Based on the position information of the main radar on the sample vehicle and the position information of the secondary radar on the sample vehicle, determine the relative position information between the first sample point cloud data and the second sample point cloud data.

[0149] 3. Use the relative position information to perform coordinate transformation on the processed second sample point cloud data to obtain the second sample point cloud data in a target coordinate system; wherein the target coordinate system is the coordinate system corresponding to the first sample point cloud data.

[0150] 4. Generate a reflectivity calibration table based on the second sample point cloud data in the target coordinate system and the data of multiple three-dimensional voxel grids.

[0151] Here, the second sample point cloud data can be processed by dedistorting the second sample point cloud data based on the obtained multiple posture data corresponding to the sample vehicle. The second sample point cloud data can be transformed by using the determined relative position information to obtain the second sample point cloud data in the target coordinate system, so that the second sample point cloud data obtained after the coordinate transformation is in the same coordinate system as the first sample point cloud data; finally, the reflectivity calibration table is generated using the second sample point cloud data in the target coordinate system and the data of multiple three-dimensional voxel grids.

[0152] In the above implementation, the second sample point cloud data is first dedistorted to eliminate the deviation caused by the different radar positions corresponding to each batch of sample point cloud data and each frame of sample point cloud data in the second sample point cloud data; then the second sample point cloud data is converted to the target coordinate system corresponding to the first sample point cloud data to eliminate the deviation caused by the different radar positions corresponding to the second sample point cloud data and the first sample point cloud data, so that when a reflectivity calibration table is generated based on the second sample point cloud data obtained after the dedistortion processing and coordinate conversion, the accuracy of the generated reflectivity calibration table can be improved.

[0153] In an optional implementation, the first sample point cloud data and the second sample point cloud data are respectively used as target sample point cloud data, when the target sample point cloud data is the first sample point cloud data, the main radar is used as the target radar, and when the target sample point cloud data is the second sample point cloud data, the secondary laser radar is used as the target radar; the target sample point cloud data has multiple frames, and each frame of target sample point cloud data includes target sample point cloud data collected by the target radar emitting multiple scan lines; wherein the target radar emits scan lines in batches according to a preset frequency, and each batch emits multiple scan lines;

[0154] Dedistort the target sample point cloud data according to the following steps:

[0155] Based on multiple pose data, determine the pose information of the target radar when transmitting each batch of scan lines;

[0156] For the target sample point cloud data collected by the non-first batch of transmitted scan lines in each frame of target sample point cloud data, based on the pose information of the target radar when transmitting the batch of scan lines, the coordinates of the target sample point cloud data collected by the batch of scan lines are converted to the coordinate system of the target radar corresponding to the target sample point cloud data collected by the first batch of scan lines in the frame of target sample point cloud data, so as to obtain the target sample point cloud data after the first dedistortion corresponding to the frame of target sample point cloud data;

[0157] For any non-first frame target sample point cloud data in the multi-frame target sample point cloud data after the first dedistortion, based on the pose information of the target radar when scanning to obtain the frame target sample point cloud data, the coordinates of the frame target sample point cloud data are converted to the coordinate system of the target radar corresponding to the first frame target sample point cloud data to obtain the target sample point cloud data after the second dedistortion corresponding to the frame target sample point cloud data.

[0158] Here, when the target sample point cloud data is the first sample point cloud data, the first sample point cloud data may include multiple frames of first sample point cloud data, and each frame of the first sample point cloud data includes multiple batches of first sample point cloud data. When performing dedistortion processing on the first sample point cloud data, the first sample point cloud data collected by the non-first batch emission scan lines in the frame of the first sample point cloud data may be converted to the coordinate system of the main radar corresponding to the emission time of the first batch of scan lines in the frame of the first sample point cloud data for each frame of the first sample point cloud data, to complete the first dedistortion processing. After the first dedistortion processing, the coordinates of any non-first frame of the first sample point cloud data in the multiple frames of the first sample point cloud data may be converted to the coordinate system of the main radar corresponding to the first frame of the first sample point cloud data, to complete the second dedistortion processing.

[0159] For example, if the first sample point cloud data includes 50 frames of first sample point cloud data, namely, the first frame of first sample point cloud data, the second frame of first sample point cloud data, ..., the fiftieth frame of first sample point cloud data, each frame of first sample point cloud data includes 10 batches of first sample point cloud data, namely, the first batch of first sample point cloud data, the second batch of first sample point cloud data, ..., the tenth batch of first sample point cloud data. For each batch of first sample point cloud data from the second batch of first sample point cloud data to the tenth batch of first sample point cloud data in each frame of first sample point cloud data, the position information when the main radar transmits the batch of scan lines is determined by interpolation, and the coordinates of the batch of first sample point cloud data (i.e., the first sample point cloud data collected by the batch of scan lines) are converted to the coordinate system of the main radar corresponding to the emission time of the first batch of scan lines in the frame of first sample point cloud data, that is, converted to the coordinate system of the main radar corresponding to the first batch of first sample point cloud data in the frame of first sample point cloud data, and then the first sample point cloud data after the first dedistortion corresponding to each frame of first sample point cloud data can be obtained.

[0160] For each frame of first sample point cloud data from the second frame to the fiftieth frame of first sample point cloud data, based on the posture information of the main radar when scanning and obtaining the first sample point cloud data of the frame, the coordinates of the first sample point cloud data of the frame are converted to the coordinate system of the main radar corresponding to the first frame of the first sample point cloud data, so as to obtain the first sample point cloud data after the second dedistortion corresponding to the first sample point cloud data.

[0161] The de-distortion processing process of the second sample point cloud data may refer to the de-distortion processing process of the first sample point cloud data, which will not be described in detail here.

[0162] Here, the target sample point cloud data collected by non-first batch scanning lines in each frame of target sample point cloud data and the non-first frame target sample point cloud data in different frames of target sample point cloud data are uniformly transformed into the coordinate system of the target radar corresponding to the first batch of target sample point cloud data in the first frame of target sample point cloud data, thereby improving the accuracy of the generated reflectivity calibration table.

[0163] In an optional implementation, after generating the reflectivity calibration table, the following steps are included:

[0164] In the reflectivity calibration table, determine the reflectivity of the scan lines for which there is no matching target reflectivity information.

[0165] Based on the target reflectivity information of the master radar in the reflectivity calibration table, the target reflectivity information of the master radar corresponding to the reflectivity of the scanning line for which there is no matching target reflectivity information is determined.

[0166] The reflectivity calibration table is updated based on the target reflectivity information of the primary radar corresponding to the reflectivity of the scan line for which no matching target reflectivity information is determined.

[0167] Here, in the generated reflectivity calibration table, when there is matching target reflectivity information for each reflectivity of each scan line, that is, when there is corresponding target reflectivity information in each grid of the generated reflectivity calibration table, there is no need to update the reflectivity calibration table.

[0168] In the generated reflectivity calibration table, when there is at least one reflectivity of the scan line that has no matching target reflectivity information (that is, when there is no corresponding target reflectivity information in some grids in the generated reflectivity calibration table), linear interpolation can be used to obtain at least one reflectivity matching target reflectivity information.

[0169] For example, if there is no matching target reflectivity information in the grid corresponding to Ring 1 and a reflectivity of 5, and at the same time, there is target reflectivity information in the grid corresponding to Ring 1 and a reflectivity of 4, and there is target reflectivity information in the grid corresponding to Ring 1 and a reflectivity of 6, then the target reflectivity information in the grid corresponding to Ring 1 and a reflectivity of 4 and the target reflectivity information in the grid corresponding to Ring 1 and a reflectivity of 6 in the reflectivity calibration table can be used to obtain the target reflectivity information in the grid corresponding to Ring 1 and a reflectivity of 5 through linear interpolation.

[0170] Alternatively, if there is no matching target reflectivity information in the grid corresponding to Ring1 and a reflectivity of 5, and at the same time, there is target reflectivity information in the grid corresponding to Ring0 and a reflectivity of 5, and there is target reflectivity information in the grid corresponding to Ring2 and a reflectivity of 5, then the target reflectivity information in the grid corresponding to Ring0 and a reflectivity of 5 and the target reflectivity information in the grid corresponding to Ring2 and a reflectivity of 5 in the reflectivity calibration table can be used to obtain the target reflectivity information in the grid corresponding to Ring1 and a reflectivity of 5 through linear interpolation.

[0171] Here, the reflectivity calibration table can be updated based on the target reflectivity information of the main radar corresponding to at least one determined reflectivity to generate an updated reflectivity calibration table, wherein in the updated reflectivity calibration table, the target reflectivity average value in the target reflectivity information can be a positive integer, that is, the target reflectivity average value corresponding to each grid in the reflectivity calibration table can be adjusted to a positive integer by rounding to generate an updated reflectivity calibration table.

[0172] There are multiple ways to determine the target reflectivity information of the primary radar corresponding to at least one reflectivity, which are only described here as examples.

[0173] Under the above implementation mode, since there may be some grids in the generated reflectivity calibration table without corresponding target reflectivity information, that is, there may be a situation where the generated reflectivity calibration table is incomplete, in order to ensure the integrity of the reflectivity calibration table, the target reflectivity information missing in the reflectivity calibration table can be determined based on the target reflectivity information of the main radar in the reflectivity calibration table, and the reflectivity calibration table can be completed to generate an updated reflectivity calibration table, that is, a complete reflectivity calibration table is obtained.

[0174] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0175] Based on the same concept, the present disclosure also provides a point cloud data fusion device, see Figure 3 , which is a schematic diagram of the architecture of a point cloud data fusion device provided by an embodiment of the present disclosure, including an acquisition module 301, an adjustment module 302, a fusion module 303, a reflectivity calibration determination module 304, and an update module 305. Specifically:

[0176] An acquisition module 301 is used to acquire point cloud data respectively collected by a primary radar and a secondary radar installed on a target vehicle; the primary radar is one of the radars on the target vehicle, and the secondary radar is a radar other than the primary radar among the radars on the target vehicle;

[0177] The adjustment module 302 is used to adjust the reflectivity in the point cloud data collected by the secondary radar based on a predetermined reflectivity calibration table of the secondary radar to obtain the adjusted point cloud data of the secondary radar; wherein the reflectivity calibration table represents the target reflectivity information of the primary radar matched by each reflectivity corresponding to each scanning line of the secondary radar;

[0178] The fusion module 303 is used to fuse the point cloud data collected by the primary radar with the adjusted point cloud data corresponding to the secondary radar to obtain fused point cloud data, so as to control the target vehicle according to the fused point cloud data.

[0179] In a possible implementation manner, the fusion device further includes: a reflectivity calibration determination module 304;

[0180] The reflectivity calibration determination module 304 is used to determine the reflectivity calibration table according to the following steps:

[0181] Acquire first sample point cloud data collected by the primary radar disposed on the sample vehicle, and second sample point cloud data collected by the secondary radar disposed on the sample vehicle;

[0182] Based on the first sample point cloud data, generating voxel map data, wherein the voxel map data includes data of a plurality of three-dimensional voxel grids, and the data of each three-dimensional voxel grid includes reflectivity information determined based on point cloud data of a plurality of scanning points in the three-dimensional voxel grid;

[0183] The reflectivity calibration table is generated based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids.

[0184] In a possible implementation manner, the reflectivity calibration determination module 304, when generating voxel map data based on the first sample point cloud data, is used to:

[0185] Acquire a plurality of position and posture data collected sequentially during the movement of the sample vehicle;

[0186] Performing dedistortion processing on the first sample point cloud data based on the multiple pose data to obtain processed first sample point cloud data;

[0187] Voxel map data is generated based on the processed first sample point cloud data.

[0188] In a possible implementation manner, the reflectivity information includes a reflectivity average value, and the reflectivity calibration determination module 304 is used to determine the data of each three-dimensional voxel grid included in the voxel map data according to the following steps:

[0189] For each of the three-dimensional voxel grids, based on the reflectivity in the point cloud data of each scanning point in the three-dimensional voxel grid, determine an average reflectivity value corresponding to the three-dimensional voxel grid;

[0190] The reflectivity calibration determination module 304, when generating the reflectivity calibration table based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids, is used to:

[0191] For each reflectivity of each scanning line of the secondary radar, determine the position information of multiple target scanning points corresponding to the reflectivity from the second sample point cloud data, wherein the multiple target scanning points are scanning points obtained by scanning the scanning line; based on the position information of the multiple target scanning points, determine at least one three-dimensional voxel grid corresponding to the multiple target scanning points; based on the average values ​​of the reflectivities respectively corresponding to the at least one three-dimensional voxel grid, determine the target reflectivity information of the primary radar matched by the reflectivity of the scanning line;

[0192] The reflectivity calibration table is generated based on the target reflectivity information of the primary radar matched with each reflectivity of each scanning line of the determined secondary radar.

[0193] In a possible implementation manner, the data of the three-dimensional voxel grid includes the reflectivity average value and a weight influence factor, and the weight influence factor includes a reflectivity variance and / or a number of scanning points;

[0194] In the case where the at least one three-dimensional voxel grid is a plurality of three-dimensional voxel grids, the reflectivity calibration determination module 304, when determining the target reflectivity information of the primary radar that matches the reflectivity of the scan line based on the reflectivity average values ​​respectively corresponding to the at least one three-dimensional voxel grids, is configured to:

[0195] Determining a weight corresponding to each of the three-dimensional voxel grids in the at least one three-dimensional voxel grid based on the weight influencing factor;

[0196] Based on the weight corresponding to each three-dimensional voxel grid and the corresponding reflectivity average value, the target reflectivity information of the primary radar matched by the reflectivity of the scan line is determined.

[0197] In a possible implementation manner, the reflectivity calibration determination module 304, when generating the reflectivity calibration table based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids, is configured to:

[0198] Acquire a plurality of posture data sequentially collected during the movement of the sample vehicle, and perform dedistortion processing on the second sample point cloud data based on the plurality of posture data to obtain processed second sample point cloud data;

[0199] Determine relative position information between the first sample point cloud data and the second sample point cloud data based on position information of the primary radar on the sample vehicle and position information of the secondary radar on the sample vehicle;

[0200] Using the relative position information, coordinate transformation is performed on the processed second sample point cloud data to obtain second sample point cloud data in a target coordinate system; wherein the target coordinate system is a coordinate system corresponding to the first sample point cloud data;

[0201] The reflectivity calibration table is generated based on the second sample point cloud data in the target coordinate system and the data of the multiple three-dimensional voxel grids.

[0202] In a possible implementation manner, the first sample point cloud data and the second sample point cloud data are respectively used as target sample point cloud data; when the target sample point cloud data is the first sample point cloud data, the main radar is used as the target radar; when the target sample point cloud data is the second sample point cloud data, the secondary laser radar is used as the target radar; the target sample point cloud data has multiple frames, and each frame of target sample point cloud data includes sample point cloud data collected by the target radar by transmitting multiple scan lines; wherein the target radar transmits scan lines in batches according to a preset frequency, and each batch transmits multiple scan lines;

[0203] The reflectivity calibration determination module 304 is used to perform dedistortion processing on the target sample point cloud data according to the following steps:

[0204] Based on the plurality of posture data, determining the posture information of the target radar when transmitting each batch of scan lines;

[0205] For the target sample point cloud data collected by the non-first batch of transmitted scan lines in each frame of target sample point cloud data, based on the posture information of the target radar when transmitting the batch of scan lines, the coordinates of the target sample point cloud data collected by transmitting the batch of scan lines are converted to the coordinate system of the target radar corresponding to the target sample point cloud data collected by transmitting the first batch of scan lines in the frame of target sample point cloud data, so as to obtain the target sample point cloud data after the first dedistortion of the frame of target sample point cloud data;

[0206] For any non-first frame target sample point cloud data in the multiple frames of target sample point cloud data after the first dedistortion, based on the posture information of the target radar when scanning to obtain the frame target sample point cloud data, the coordinates of the frame target sample point cloud data are converted to the coordinate system of the target radar corresponding to the first frame target sample point cloud data to obtain the target sample point cloud data after the second dedistortion corresponding to the frame target sample point cloud data.

[0207] In a possible implementation manner, after generating the reflectivity calibration table, the method further includes: an updating module 305, which is used to:

[0208] In the reflectivity calibration table, determining the reflectivity of a scan line for which no matching target reflectivity information exists;

[0209] Based on the target reflectivity information of the primary radar in the reflectivity calibration table, determining the target reflectivity information of the primary radar corresponding to the reflectivity of the scanning line for which no matching target reflectivity information exists;

[0210] The reflectivity calibration table is updated based on the target reflectivity information of the primary radar corresponding to the reflectivity of the scan line for which no matching target reflectivity information is determined.

[0211] In some embodiments, the functions or templates contained in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0212] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Figure 4As shown, it is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, including a processor 401, a memory 402, and a bus 403. Among them, the memory 402 is used to store execution instructions, including a memory 4021 and an external memory 4022; the memory 4021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 401, and the data exchanged with the external memory 4022 such as a hard disk. The processor 401 exchanges data with the external memory 4022 through the memory 4021. When the electronic device 400 is running, the processor 401 communicates with the memory 402 through the bus 403, so that the processor 401 executes the following instructions:

[0213] Acquire point cloud data respectively collected by a main radar and a secondary radar installed on the target vehicle; the main radar is one of the radars on the target vehicle, and the secondary radar is a radar other than the main radar among the radars on the target vehicle;

[0214] Based on a predetermined reflectivity calibration table corresponding to the secondary radar, the reflectivity in the point cloud data collected by the secondary radar is adjusted to obtain adjusted point cloud data corresponding to the secondary radar; wherein the reflectivity calibration table represents the target reflectivity information of the primary radar matched by each reflectivity corresponding to each scanning line of the secondary radar;

[0215] The point cloud data collected by the main radar is fused with the adjusted point cloud data corresponding to the secondary radar to obtain fused point cloud data.

[0216] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the point cloud data fusion method described in the above method embodiment are executed.

[0217] The computer program product of the point cloud data fusion method provided in the embodiment of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the point cloud data fusion method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0218] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0219] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0220] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0221] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0222] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for fusion of point cloud data, It is characterized in that include: Obtaining point cloud data collected by a primary radar and a secondary radar respectively installed on a target vehicle; The primary radar is one of the radars on the target vehicle, and the secondary radar is a radar other than the primary radar among the radars on the target vehicle; Based on a predetermined reflectivity calibration table of the secondary radar, the reflectivity in the point cloud data collected by the secondary radar is adjusted to obtain the adjusted point cloud data of the secondary radar; wherein the reflectivity calibration table represents the target reflectivity information of the primary radar matched by each reflectivity corresponding to each scanning line of the secondary radar; Fusing the point cloud data collected by the primary radar with the adjusted point cloud data of the secondary radar to obtain fused point cloud data; Determine the reflectivity calibration table according to the following steps: Acquire first sample point cloud data collected by the primary radar disposed on the sample vehicle, and second sample point cloud data collected by the secondary radar disposed on the sample vehicle; Based on the first sample point cloud data, generating voxel map data, wherein the voxel map data includes data of a plurality of three-dimensional voxel grids, and the data of each three-dimensional voxel grid includes reflectivity information determined based on point cloud data of a plurality of scanning points in the three-dimensional voxel grid; The reflectivity calibration table is generated based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids.

2. The method for fusing point cloud data according to claim 1, It is characterized in that The step of generating voxel map data based on the first sample point cloud data includes: Acquire a plurality of position and posture data collected sequentially during the movement of the sample vehicle; Performing dedistortion processing on the first sample point cloud data based on the multiple pose data to obtain processed first sample point cloud data; Voxel map data is generated based on the processed first sample point cloud data.

3. The point cloud data fusion method according to claim 1 or 2, It is characterized in that The reflectivity information includes a reflectivity average value, and the data of each three-dimensional voxel grid included in the voxel map data is determined according to the following steps: For each of the three-dimensional voxel grids, based on the reflectivity in the point cloud data of each scanning point in the three-dimensional voxel grid, determine an average reflectivity value corresponding to the three-dimensional voxel grid; Generating the reflectivity calibration table based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids includes: For each reflectivity of each scanning line of the secondary radar, determine the position information of multiple target scanning points corresponding to the reflectivity from the second sample point cloud data, wherein the multiple target scanning points are scanning points obtained by scanning the scanning line; based on the position information of the multiple target scanning points, determine at least one three-dimensional voxel grid corresponding to the multiple target scanning points; based on the average values ​​of the reflectivities respectively corresponding to the at least one three-dimensional voxel grid, determine the target reflectivity information of the primary radar matched by the reflectivity of the scanning line; The reflectivity calibration table is generated based on the target reflectivity information of the primary radar that matches each reflectivity of each scanning line of the determined secondary radar.

4. The method for fusing point cloud data according to claim 3, It is characterized in that The data of the three-dimensional voxel grid includes the reflectivity average value and a weight influence factor, and the weight influence factor includes the reflectivity variance and / or the number of scanning points; In a case where the at least one three-dimensional voxel grid is a plurality of three-dimensional voxel grids, determining the target reflectivity information of the primary radar that matches the reflectivity of the scan line based on the average reflectivity values ​​respectively corresponding to the at least one three-dimensional voxel grids, includes: Determining a weight corresponding to each of the three-dimensional voxel grids in the at least one three-dimensional voxel grid based on the weight influencing factor; Based on the weight corresponding to each three-dimensional voxel grid and the corresponding reflectivity average value, the target reflectivity information of the primary radar matched by the reflectivity of the scan line is determined.

5. The method for fusing point cloud data according to any one of claims 1 to 4, It is characterized in that Generating the reflectivity calibration table based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids includes: Acquire a plurality of posture data sequentially collected during the movement of the sample vehicle, and perform dedistortion processing on the second sample point cloud data based on the plurality of posture data to obtain processed second sample point cloud data; Determine relative position information between the first sample point cloud data and the second sample point cloud data based on position information of the primary radar on the sample vehicle and position information of the secondary radar on the sample vehicle; Using the relative position information, coordinate transformation is performed on the processed second sample point cloud data to obtain second sample point cloud data in a target coordinate system; wherein the target coordinate system is a coordinate system corresponding to the first sample point cloud data; The reflectivity calibration table is generated based on the second sample point cloud data in the target coordinate system and the data of the multiple three-dimensional voxel grids.

6. The point cloud data fusion method according to claim 2 or 5, It is characterized in that The first sample point cloud data and the second sample point cloud data are respectively used as target sample point cloud data; when the target sample point cloud data is the first sample point cloud data, the primary radar is used as the target radar; and when the target sample point cloud data is the second sample point cloud data, the secondary radar is used as the target radar; The target sample point cloud data includes multiple frames, and each frame of the target sample point cloud data includes target sample point cloud data collected by the target radar transmitting multiple scan lines; wherein the target radar transmits scan lines in batches according to a preset frequency, and each batch transmits multiple scan lines; Dedistort the target sample point cloud data according to the following steps: Based on the plurality of posture data, determining the posture information of the target radar when transmitting each batch of scan lines; For the target sample point cloud data collected by the non-first batch of transmitted scan lines in each frame of target sample point cloud data, based on the posture information of the target radar when transmitting the batch of scan lines, the coordinates of the target sample point cloud data collected by transmitting the batch of scan lines are converted to the coordinate system of the target radar corresponding to the target sample point cloud data collected by transmitting the first batch of scan lines in the frame of target sample point cloud data, so as to obtain the target sample point cloud data after the first dedistortion of the frame of target sample point cloud data; For any non-first frame target sample point cloud data in the multiple frames of target sample point cloud data after the first dedistortion, based on the posture information of the target radar when scanning to obtain the frame target sample point cloud data, the coordinates of the frame target sample point cloud data are converted to the coordinate system of the target radar corresponding to the first frame target sample point cloud data to obtain the target sample point cloud data after the second dedistortion corresponding to the frame target sample point cloud data.

7. The point cloud data fusion method according to any one of claims 1 to 6, It is characterized in that After generating the reflectivity calibration table, the method further includes: In the reflectivity calibration table, determining the reflectivity of a scan line for which no matching target reflectivity information exists; Based on the target reflectivity information of the primary radar in the reflectivity calibration table, determining the target reflectivity information of the primary radar corresponding to the reflectivity of the scanning line for which no matching target reflectivity information exists; The reflectivity calibration table is updated based on the target reflectivity information of the primary radar corresponding to the reflectivity of the scan line for which no matching target reflectivity information is determined.

8. A point cloud data fusion device, It is characterized in that include: An acquisition module is used to acquire point cloud data collected by a primary radar and a secondary radar respectively arranged on a target vehicle; The primary radar is one of the radars on the target vehicle, and the secondary radar is a radar other than the primary radar among the radars on the target vehicle; an adjustment module, configured to adjust the reflectivity in the point cloud data collected by the secondary radar based on a predetermined reflectivity calibration table of the secondary radar, so as to obtain the adjusted point cloud data of the secondary radar; wherein the reflectivity calibration table represents the target reflectivity information of the primary radar matched by each reflectivity corresponding to each scanning line of the secondary radar; A fusion module, used to fuse the point cloud data collected by the primary radar with the adjusted point cloud data of the secondary radar to obtain fused point cloud data; The reflectivity calibration determination module is used to determine the reflectivity calibration table according to the following steps: Acquire first sample point cloud data collected by the primary radar disposed on the sample vehicle, and second sample point cloud data collected by the secondary radar disposed on the sample vehicle; Based on the first sample point cloud data, generating voxel map data, wherein the voxel map data includes data of a plurality of three-dimensional voxel grids, and the data of each three-dimensional voxel grid includes reflectivity information determined based on point cloud data of a plurality of scanning points in the three-dimensional voxel grid; The reflectivity calibration table is generated based on the second sample point cloud data and the data of the plurality of three-dimensional voxel grids.

9. An electronic device, It is characterized in that include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the point cloud data fusion method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the point cloud data fusion method according to any one of claims 1 to 7 are executed.

Images