Leighting data dynamic fusion method and device, Leighting all-in-one machine and storage medium

Abstract

The invention provides a dynamic fusion method and device for thundersight data, a thundersight all-in-one machine and a storage medium. The system comprises a visual device and at least one group of detection radars, and each group of detection radars comprises a main radar and a blind compensation radar. The method comprises the following steps: determining a coordinate transformation matrix of each main radar based on pose change information of the main radar; based on the coordinate transformation matrix of each main radar, transforming the coordinates of the point cloud data collected by the main radar to a radar coordinate system where the blind compensation radar corresponding to the main radar is located, and obtaining the point cloud data after coordinate transformation; for each group of detection radars, clustering the point cloud data after coordinate transformation and the point cloud data collected by the blind compensation radar to obtain radar target information of the group of detection radars; and performing thunder-view data fusion on the radar target information of each group of detection radars and the visual target information collected by the visual equipment so as to perform target tracking. According to the invention, the fusion result of the thunder view data has better accuracy.

Description

Technical Field

[0001] The present invention relates to the field of perception data processing technology, and in particular to a radar and vision data dynamic fusion method, device, radar and vision integrated machine and storage medium. Background Art

[0002] The fusion of radar and video data is a complex task that requires ensuring the accuracy and effectiveness of the fusion. Due to the differences in the detection ranges of radar and visual sensors, in order to ensure the data fusion effect, a group of radars are usually set up in each direction in the radar-visual detection system. Each group of radars includes a main radar and a blind spot radar to achieve blind spot detection. After adding the blind spot radar, it is necessary to fuse the detection data of the main radar and the blind spot radar before performing the radar-visual data fusion.

[0003] However, the working position and detection angle of the radar equipment may be adjusted due to the actual application scenario or may change during use. Once the position or angle of the radar changes, the original fusion parameters are no longer applicable, and manual parameter adjustment and calibration are required to accurately detect the target, resulting in unstable radar data fusion effect. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for dynamic fusion of radar and vision data, a radar and vision integrated machine and a storage medium to solve the problem of unstable fusion effect of radar and vision data.

[0005] In a first aspect, an embodiment of the present invention provides a method for dynamic fusion of radar data, which is applied to a radar perception system, the system comprising a visual device and at least one group of detection radars, each group of detection radars comprising a main radar and a blind spot radar; the method comprises:

[0006] Determine the coordinate transformation matrix of each main radar based on the posture change information of the main radar;

[0007] Based on the coordinate conversion matrix of each main radar, the coordinates of the point cloud data collected by the main radar are converted to the radar coordinate system where the blind spot radar corresponding to the main radar is located, so as to obtain the point cloud data after the coordinate conversion;

[0008] For each group of detection radars, cluster the point cloud data after coordinate conversion and the point cloud data collected by the blind radar to obtain the radar target information of the group of detection radars;

[0009] The radar target information of each group of detection radars and the visual target information collected by the visual equipment are fused with radar data to perform target tracking.

[0010] In a possible implementation, the posture change information includes a pitch angle change, a horizontal angle change, and a position change;

[0011] The coordinate transformation matrix of each main radar is determined based on the posture change information of the main radar, including:

[0012] Obtain an initial coordinate transformation matrix corresponding to an initial posture of a first main radar; wherein the first main radar is any main radar;

[0013] The rotation angle of the initial coordinate conversion matrix is ​​adjusted based on the pitch angle change and the horizontal angle change of the first primary radar, and the translation amount of the initial coordinate conversion matrix is ​​adjusted based on the position change to obtain the coordinate conversion matrix of the first primary radar.

[0014] In a possible implementation, before clustering the point cloud data after coordinate transformation and the point cloud data collected by the blind spot compensation radar for each group of detection radars, the method further includes:

[0015] For each group of detection radars, based on the posture change information of the main radar in the group of detection radars, the overlapping part of the detection area of ​​the main radar and the blind radar in the group of detection radars is determined, and used as the overlapping area of ​​the group of detection radars;

[0016] The point cloud data in the overlapping area of ​​each group of detection radars are deduplicated to cluster the deduplicated point cloud data.

[0017] In a possible implementation, for each group of detection radars, based on the posture change information of the main radar in the group of detection radars, the overlapping part of the detection area of ​​the main radar and the blind radar in the group of detection radars is determined and used as the overlapping area of ​​the group of detection radars, including:

[0018] Obtaining the detection area corresponding to the initial posture of the first main radar;

[0019] Determining a detection area of ​​the first main radar after the posture change based on the posture change information of the first main radar;

[0020] Based on the detection area of ​​the first main radar after the posture change and the detection area of ​​the first blind radar, the overlapping part of the detection area of ​​the main radar and the blind radar in the first group of detection radars is determined; wherein the first main radar and the first blind radar are respectively the main radar and the blind radar in the first group of detection radars, and the first group of detection radars is any group of detection radars.

[0021] In a possible implementation, deduplication of point cloud data in the overlapping area of ​​each group of detection radars includes:

[0022] The point cloud data collected by the blind radar in the overlapping area of ​​each group of detection radars is removed.

[0023] In a possible implementation, radar target information of each group of detection radars and visual target information collected by visual equipment are fused with radar data, including:

[0024] Based on the radar coordinate system of each group of detection radars and the visual coordinate system of the visual device, the radar target information of each group of detection radars and the visual target information collected by the visual device are subjected to coordinate transformation, area splicing and identity merging to obtain fused target information.

[0025] In a possible implementation, before clustering the point cloud data after coordinate conversion and the point cloud data collected by the blind spot compensation radar for each group of detection radars to obtain radar target information of the group of detection radars, the method further includes:

[0026] Outliers are removed from each point cloud data, so as to perform coordinate transformation and clustering based on the point cloud data after outliers are removed.

[0027] In a second aspect, an embodiment of the present invention provides a dynamic fusion device for radar data, which is applied to a radar perception system. The system includes a visual device and at least one group of detection radars, each group of detection radars includes a main radar and a blind radar; the device includes:

[0028] A matrix construction module, used to determine the coordinate transformation matrix of each main radar based on the posture change information of the main radar;

[0029] A coordinate conversion module is used to convert the coordinates of the point cloud data collected by the main radar into the radar coordinate system where the blind spot radar corresponding to the main radar is located based on the coordinate conversion matrix of each main radar, so as to obtain the point cloud data after the coordinate conversion;

[0030] The first fusion module is used to cluster the point cloud data after coordinate conversion and the point cloud data collected by the blind spot radar for each group of detection radars to obtain radar target information of the group of detection radars;

[0031] The second fusion module is used to fuse the radar target information of each group of detection radars and the visual target information collected by the visual equipment to perform target tracking.

[0032] In a third aspect, an embodiment of the present invention provides a radar-vision integrated machine, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect or any possible implementation of the first aspect are implemented.

[0033] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect or any possible implementation method of the first aspect are implemented.

[0034] The embodiment of the present invention provides a method, device, integrated machine and storage medium for dynamic fusion of radar vision data. On the basis that only the posture of the main radar can be adjusted, the coordinate transformation matrix of the main radar is determined by the posture change information of the main radar, and the point cloud data of the main radar is converted to the radar coordinate system where the blind radar is located, so as to realize the splicing of the detection areas of the main radar and the blind radar, realize data fusion at the level of original data, and then cluster the point cloud data collected by each group of the main radar and the blind radar to obtain the structured information of the radar target. The clustering result is not affected by the change of the posture of the main radar, and has better accuracy, thereby improving the effect of subsequent fusion of radar vision data. Compared with clustering the point cloud data of the main radar and the blind radar separately to obtain the structured information of each radar target and then fusing them, it has a simpler and faster processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 This is an application scenario diagram of the method for dynamic fusion of radar vision data provided by an embodiment of the present invention;

[0037] Figure 2 is a flow chart of a method for dynamically fusing radar vision data provided by an embodiment of the present invention;

[0038] Figure 3 is a flow chart of a method for dynamically fusing radar vision data provided by another embodiment of the present invention;

[0039] Figure 4 It is a program function block diagram of a method for dynamic fusion of radar vision data provided by an embodiment of the present invention;

[0040] Figure 5 It is a structural schematic diagram of a radar data dynamic fusion device provided by an embodiment of the present invention;

[0041] Figure 6 It is a schematic diagram of a radar-vision integrated machine provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0042] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0044] Figure 1 The application scenario diagram of the method for dynamic fusion of radar data provided by the embodiment of the present invention. In the scenario of highways, the radar perception system cannot perceive and detect targets on the road in real time under various bad weather and lighting conditions only by using traditional video cameras. Only by using radar, it is impossible to obtain and distinguish the identity and appearance information of the target in real time, and the detection distance is limited. In order to extend the perception distance of its radar, a combined perception mode of radar and visual perception is adopted to fuse the data of the two types of equipment. Cameras with different focal lengths are used to achieve a larger range of overlap with the radar area, and then through data fusion, data fusion in a single direction can be achieved. In addition, there is a blind spot under the radar. Usually, the blind spot range of one side is 40 meters to 50 meters, and in the case of two sides, there is a blind spot of nearly 100 meters. Therefore, a blind spot radar is also set for the blind spot, and the detection data from different directions and sources are fused multiple times to achieve full coverage.

[0045] The radar perception system used in the embodiment of the present invention may include two sets of detection radars, each responsible for a different direction, such as Figure 1 As shown, in each group of detection radars, there is an overlapping area between the detection areas of the main radar and the blind radar. Figure 1 The figure shows the detection areas of the two main radars when they are in a certain position. In actual operation, the pitch angle, horizontal angle and position of the main radar may be adjusted according to changes in detection requirements, resulting in changes in the detection area of ​​the main radar and corresponding changes in the overlapping area with the blind radar.

[0046] See also Figure 2 , which shows a flow chart of the implementation of the method for dynamic fusion of radar vision data provided by an embodiment of the present invention, which is described in detail as follows:

[0047] The method is applied to a radar perception system, which includes a visual device and at least one group of detection radars, each group of detection radars including a main radar and a blind spot radar; the method includes:

[0048] Step 201: Determine the coordinate transformation matrix of each master radar based on the posture change information of the master radar.

[0049] In this embodiment, since the main radar and the blind spot radar have different installation positions, detection postures and coverage ranges, it is necessary to use a coordinate transformation matrix to convert the data collected by the main radar and the data collected by the blind spot radar into the same coordinate system, and then fuse the data collected by the two radars.

[0050] The initial pose of the main radar and the pose of the blind radar are known data, and a coordinate transformation matrix can be determined using the poses of the two. On this basis, when the pose of the main radar changes, the difference between the pose of the main radar before and after the change, that is, the pose change information of the main radar, can be used to adjust and update the coordinate transformation matrix, and continue to realize the conversion of the data collected by the main radar and the blind radar into the same coordinate system.

[0051] Step 202: based on the coordinate conversion matrix of each main radar, the coordinates of the point cloud data collected by the main radar are converted to the radar coordinate system where the blind spot radar corresponding to the main radar is located, so as to obtain the point cloud data after the coordinate conversion.

[0052] In this embodiment, the radar data of different areas are currently fused. Usually, each radar processes and analyzes the point cloud data collected by itself, extracts the structured data representing the identity of the target, and then the processor fuses and merges the structured data of multiple radars. This method requires setting the parameters of each radar to be implemented. When the posture of the main radar changes, the original parameters are no longer applicable when the main radar processes the point cloud data. The main radar needs to be manually debugged again, and real-time and dynamic data fusion cannot be achieved.

[0053] In this embodiment, the point cloud data of each radar is converted into the same coordinate system, and the point cloud data coordinates are aligned at the original data level. The subsequent target recognition and radar-vision fusion can be directly implemented based on the point cloud data, which has higher real-time performance.

[0054] Step 203: for each group of detection radars, cluster the point cloud data after coordinate conversion and the point cloud data collected by the blind spot radar to obtain radar target information of the group of detection radars.

[0055] In this embodiment, due to the difference in posture between the main radar and the blind spot radar, the point cloud data collected by the two for the same target may have different coordinates. After the coordinate conversion, the coordinates of each point cloud data can be unified. At this time, the point cloud data after the coordinate conversion and the point cloud data collected by the blind spot radar are clustered, and multiple point cloud data corresponding to the same target can be gathered together to obtain a point cloud fitting frame corresponding to each target. This avoids the situation where one target is detected as two targets by two radars.

[0056] Step 204 , radar target information of each group of detection radars and visual target information collected by visual equipment are fused to perform target tracking.

[0057] In this embodiment, after clustering is completed, the radar target information of the entire area can be obtained. At this time, the coverage area of ​​the radar target information overlaps with the coverage area of ​​the actual target information. After the radar and visual data are fused, the complete information of the target can be obtained to achieve target tracking. For example, for highway scenes, the track tracking algorithm can be used to identify and track the lane, speed, trajectory and other information of the target.

[0058] In the embodiment of the present invention, only the posture of the main radar can be adjusted. The coordinate transformation matrix of the main radar is determined through the posture change information of the main radar, and the point cloud data of the main radar is converted to the radar coordinate system where the blind radar is located, so as to realize the splicing of the detection areas of the main radar and the blind radar, and realize the data fusion at the level of the original data. Then, the point cloud data collected by each group of the main radar and the blind radar are clustered to obtain the structured information of the radar target. The clustering result is not affected by the posture change of the main radar and has better accuracy, thereby improving the effect of subsequent radar vision data fusion. Compared with clustering the point cloud data of the main radar and the blind radar separately to obtain the structured information of each radar target and then fusing them, it has a simpler and faster processing effect.

[0059] In a possible implementation, the posture change information includes a pitch angle change, a horizontal angle change, and a position change;

[0060] The coordinate transformation matrix of each main radar is determined based on the posture change information of the main radar, including:

[0061] Obtain an initial coordinate transformation matrix corresponding to an initial posture of a first main radar; wherein the first main radar is any main radar;

[0062] The rotation angle of the initial coordinate conversion matrix is ​​adjusted based on the pitch angle change and the horizontal angle change of the first primary radar, and the translation amount of the initial coordinate conversion matrix is ​​adjusted based on the position change to obtain the coordinate conversion matrix of the first primary radar.

[0063] In this embodiment, the initial coordinate conversion matrix is ​​determined based on the relationship between the initial posture of the main radar and the posture of the blind radar, and is used to rotate and translate the point cloud data to convert the point cloud data to the coordinates of the blind radar.

[0064] The posture adjustment of the main radar includes the adjustment of the pitch angle and the horizontal angle. The pitch angle is to adjust the front and rear direction of the radar, and the horizontal angle is to adjust the left and right direction of the radar. The position of the main radar may change due to environmental influences during use. After the posture changes, the original coordinate transformation matrix parameters of the main radar and the blind radar are no longer applicable. At this time, the rotation and translation parameters of the coordinate transformation matrix can be dynamically adjusted according to the specific posture change information of the main radar to continue to realize the coordinate transformation of the point cloud data.

[0065] In a possible implementation, before clustering the point cloud data after coordinate transformation and the point cloud data collected by the blind spot compensation radar for each group of detection radars, the method further includes:

[0066] For each group of detection radars, based on the posture change information of the main radar in the group of detection radars, the overlapping part of the detection area of ​​the main radar and the blind radar in the group of detection radars is determined, and used as the overlapping area of ​​the group of detection radars;

[0067] The point cloud data in the overlapping area of ​​each group of detection radars are deduplicated to cluster the deduplicated point cloud data.

[0068] In this embodiment, there is always an overlap between the detection areas of the main radar and the blind spot radar. For the overlapping area, there are two sets of point cloud data in the area, that is, the same target in the overlapping area corresponds to a set of point cloud data collected by the main radar and a set of point cloud data collected by the blind spot radar. Due to the difference in detection parameters between the main radar and the blind spot radar, the accuracy of the point cloud data collected by the two is also different. If the two sets of point cloud data are clustered directly, the clustering result will be affected by the point cloud data with lower accuracy in these point cloud data, resulting in a decrease in the accuracy of the clustering result compared to the point cloud data with higher accuracy, making the clustering result less than ideal.

[0069] Therefore, for overlapping areas, deduplication can be performed before clustering. Among the multiple sets of point cloud data corresponding to the same target, the point cloud data with higher accuracy can be retained, and the point cloud data with lower accuracy can be removed. By screening the data before clustering, the amount of data processing in the clustering process can be reduced and the effect of the clustering results can be improved.

[0070] In a possible implementation, for each group of detection radars, based on the posture change information of the main radar in the group of detection radars, the overlapping part of the detection area of ​​the main radar and the blind radar in the group of detection radars is determined and used as the overlapping area of ​​the group of detection radars, including:

[0071] Obtaining the detection area corresponding to the initial posture of the first main radar;

[0072] Determining a detection area of ​​the first main radar after the posture change based on the posture change information of the first main radar;

[0073] Based on the detection area of ​​the first main radar after the posture change and the detection area of ​​the first blind radar, the overlapping part of the detection area of ​​the main radar and the blind radar in the first group of detection radars is determined; wherein the first main radar and the first blind radar are respectively the main radar and the blind radar in the first group of detection radars, and the first group of detection radars is any group of detection radars.

[0074] In this embodiment, the initial detection area of ​​the main radar is known. The initial detection area can be adjusted in combination with the posture change of the main radar to determine the detection area after the posture change. The detection area of ​​the blind spot radar is fixed. At this time, the detection areas of the two radars are known quantities, and the overlapping area after the posture change can be determined based on this information.

[0075] In a possible implementation, deduplication of point cloud data in the overlapping area of ​​each group of detection radars includes:

[0076] The point cloud data collected by the blind radar in the overlapping area of ​​each group of detection radars is removed.

[0077] In this embodiment, generally speaking, the detection area of ​​the blind spot radar is mainly concentrated in the blind spot below the main radar, while the main radar is mainly concentrated in the long-distance area outside the blind spot. If the data of the blind spot radar is imported into the main radar, it will be found that the point cloud data detected by the blind spot radar is more divergent and the accuracy is not high enough. Clustering using the unstable data of the blind spot radar is prone to target point cloud clustering errors. Therefore, the point cloud data collected by the main radar is preferentially retained for the overlapping area to achieve a better clustering effect.

[0078] In a possible implementation, radar target information of each group of detection radars and visual target information collected by visual equipment are fused with radar data, including:

[0079] Based on the radar coordinate system of each group of detection radars and the visual coordinate system of the visual device, the radar target information of each group of detection radars and the visual target information collected by the visual device are subjected to coordinate transformation, area splicing and identity merging to obtain fused target information.

[0080] In this embodiment, the radar and camera data fusion first needs to be registered, and the radar and camera data are mapped to the same coordinate system through coordinate conversion, time synchronization and other steps. When the detection areas of the camera and radar are inconsistent, it involves regional splicing to achieve full coverage. Then the data is associated, and the nearest neighbor matching, Kalman filtering and other methods are used to determine whether the objects detected by the radar and camera correspond to the same target in the real world, and the same target is merged and tracked using the fused target information of the target.

[0081] In a possible implementation, before clustering the point cloud data after coordinate conversion and the point cloud data collected by the blind spot compensation radar for each group of detection radars to obtain radar target information of the group of detection radars, the method further includes:

[0082] Outliers are removed from each point cloud data, so as to perform coordinate transformation and clustering based on the point cloud data after outliers are removed.

[0083] In this embodiment, outliers refer to unqualified data. Point cloud data whose information such as the change in time, the change in amplitude, and the change in acceleration exceeds the normal parameter range of the target are mutation data or impossible data. They have no value for target tracking and will interfere with the tracking results of the target. Therefore, they should be eliminated as outliers to improve the stability and repeatability of target tracking.

[0084] Specifically, the features of each point in the point cloud, such as position, intensity, curvature, normal direction, etc., can be calculated, and outliers can be detected based on the threshold of each feature, and then eliminated, retaining only qualified point cloud data.

[0085] In a specific embodiment, for Figure 1 The overall steps of the radar and vision integrated machine for dynamic fusion of radar and vision data are as follows: Figure 3 As shown, this step is based on Figure 4 The thread architecture is implemented, and each thread corresponds to different work content. The radar-vision integrated machine includes two groups of radars in two directions. Threads 1, 2, 3, and 4 correspond to the main radar in direction 1, the blind radar in direction 1, the main radar in direction 2, and the blind radar in direction 2, respectively. The point cloud data collected by the radar is received and the static point cloud is filtered out. The coordinate transformation matrix is ​​determined according to the angle adjustment change value of the main radar, and the processed point cloud data and the coordinate transformation matrix are sent to thread 5 respectively. Thread 5 performs coordinate transformation and clustering on the point cloud data collected by the radar, removes wild values ​​from the clustering results, and aligns the time and space with the video data again, performs radar-vision data fusion and tracking, generates the target's track information, and can also include target feature type data processing output, target tracking type data processing output, target abnormal behavior type data processing output, and regional / section flow data processing output. Finally, thread 6 sends the track data to complete target tracking.

[0086] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0087] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0088] Figure 5 The structure diagram of the dynamic fusion device of radar data provided by the embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0089] like Figure 5 As shown, the dynamic fusion device 5 of the thunder vision data is applied to the thunder vision perception system, which includes a visual device and at least one group of detection radars, each group of detection radars includes a main radar and a blind radar; the device 5 includes:

[0090] A matrix construction module 51 is used to determine the coordinate transformation matrix of each master radar based on the posture change information of the master radar;

[0091] A coordinate conversion module 52 is used to convert the coordinates of the point cloud data collected by the main radar to the radar coordinate system where the blind spot radar corresponding to the main radar is located based on the coordinate conversion matrix of each main radar, so as to obtain the point cloud data after the coordinate conversion;

[0092] The first fusion module 53 is used to cluster the point cloud data after coordinate conversion and the point cloud data collected by the blind spot radar for each group of detection radars to obtain radar target information of the group of detection radars;

[0093] The second fusion module 54 is used to fuse the radar target information of each group of detection radars and the visual target information collected by the visual equipment to perform target tracking.

[0094] In a possible implementation, the posture change information includes a pitch angle change, a horizontal angle change, and a position change;

[0095] The matrix construction module 51 is specifically used for:

[0096] Obtain an initial coordinate transformation matrix corresponding to an initial posture of a first main radar; wherein the first main radar is any main radar;

[0097] The rotation angle of the initial coordinate conversion matrix is ​​adjusted based on the pitch angle change and the horizontal angle change of the first primary radar, and the translation amount of the initial coordinate conversion matrix is ​​adjusted based on the position change to obtain the coordinate conversion matrix of the first primary radar.

[0098] In a possible implementation, the matrix construction module 51 is further used for:

[0099] Before clustering the point cloud data after coordinate conversion and the point cloud data collected by the blind spot radar for each group of detection radars, for each group of detection radars, based on the posture change information of the main radar in the group of detection radars, determine the overlapping part of the detection area of ​​the main radar and the blind spot radar in the group of detection radars, and use it as the overlapping area of ​​the group of detection radars;

[0100] The point cloud data in the overlapping area of ​​each group of detection radars are deduplicated to cluster the deduplicated point cloud data.

[0101] In a possible implementation, the matrix construction module 51 is specifically used for:

[0102] Obtaining the detection area corresponding to the initial posture of the first main radar;

[0103] Determining a detection area of ​​the first main radar after the posture change based on the posture change information of the first main radar;

[0104] Based on the detection area of ​​the first main radar after the posture change and the detection area of ​​the first blind radar, the overlapping part of the detection area of ​​the main radar and the blind radar in the first group of detection radars is determined; wherein the first main radar and the first blind radar are respectively the main radar and the blind radar in the first group of detection radars, and the first group of detection radars is any group of detection radars.

[0105] In a possible implementation, the matrix construction module 51 is specifically used for:

[0106] The point cloud data collected by the blind radar in the overlapping area of ​​each group of detection radars is removed.

[0107] In a possible implementation, the second fusion module 54 is specifically configured to:

[0108] Based on the radar coordinate system of each group of detection radars and the visual coordinate system of the visual device, the radar target information of each group of detection radars and the visual target information collected by the visual device are subjected to coordinate transformation, area splicing and identity merging to obtain fused target information.

[0109] In a possible implementation, the first fusion module 53 is further configured to:

[0110] For each group of detection radars, the point cloud data after coordinate conversion and the point cloud data collected by the blind radar are clustered. Before obtaining the radar target information of the group of detection radars, outliers are eliminated for each point cloud data, so as to perform coordinate conversion and clustering based on the point cloud data after eliminating outliers.

[0111] In the embodiment of the present invention, only the posture of the main radar can be adjusted. The coordinate transformation matrix of the main radar is determined through the posture change information of the main radar, and the point cloud data of the main radar is converted to the radar coordinate system where the blind radar is located, so as to realize the splicing of the detection areas of the main radar and the blind radar, and realize the data fusion at the level of the original data. Then, the point cloud data collected by each group of the main radar and the blind radar are clustered to obtain the structured information of the radar target. The clustering result is not affected by the posture change of the main radar and has better accuracy, thereby improving the effect of subsequent radar vision data fusion. Compared with clustering the point cloud data of the main radar and the blind radar separately to obtain the structured information of each radar target and then fusing them, it has a simpler and faster processing effect.

[0112] Figure 6 Schematic diagram of the integrated radar and video machine provided by an embodiment of the present invention. Figure 6 As shown, the integrated radar and visual device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned embodiments of the dynamic fusion method of radar and visual data are implemented, such as Figure 2 Alternatively, when the processor 60 executes the computer program 62, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 Functionality of modules / units 51 to 54 shown.

[0113] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 62 in the integrated radar vision device 6. For example, the computer program 62 may be divided into Figure 5 Modules / units 51 to 54 are shown.

[0114] The integrated machine 6 for radar and video can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The integrated machine 6 for radar and video can include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art can understand that Figure 6 It is only an example of the integrated thunder and vision machine 6 and does not constitute a limitation of the integrated thunder and vision machine 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the integrated thunder and vision machine may also include input and output devices, network access devices, buses, etc.

[0115] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0116] The memory 61 may be an internal storage unit of the integrated thunder and vision machine 6, such as a hard disk or memory of the integrated thunder and vision machine 6. The memory 61 may also be an external storage device of the integrated thunder and vision machine 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the integrated thunder and vision machine 6. Further, the memory 61 may also include both an internal storage unit of the integrated thunder and vision machine 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the integrated thunder and vision machine. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0117] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0118] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0119] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0120] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0123] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned embodiments of the dynamic fusion method of radar data can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0124] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for dynamic fusion of radar data, characterized in that: Applied to a radar perception system, the system includes a visual device and at least one group of detection radars, each group of detection radars includes a main radar and a blind radar; the method includes: Determine the coordinate transformation matrix of each main radar based on the posture change information of the main radar; Based on the coordinate conversion matrix of each main radar, the coordinates of the point cloud data collected by the main radar are converted to the radar coordinate system where the blind spot radar corresponding to the main radar is located, so as to obtain the point cloud data after the coordinate conversion; For each group of detection radars, clustering the point cloud data after coordinate conversion and the point cloud data collected by the blind spot radar to obtain radar target information of the group of detection radars; The radar target information of each group of detection radars and the visual target information collected by the visual device are fused with radar data to perform target tracking.

2. The method for dynamic fusion of radar data according to claim 1, characterized in that: The posture change information includes the pitch angle change, horizontal angle change and position change; The step of determining the coordinate transformation matrix of each main radar based on the posture change information of the main radar includes: Obtaining an initial coordinate transformation matrix corresponding to an initial posture of a first primary radar; wherein the first primary radar is any primary radar; The rotation angle of the initial coordinate conversion matrix is ​​adjusted based on the pitch angle change and the horizontal angle change of the first primary radar, and the translation amount of the initial coordinate conversion matrix is ​​adjusted based on the position change to obtain the coordinate conversion matrix of the first primary radar.

3. The method for dynamic fusion of radar data according to claim 1, characterized in that: Before clustering the point cloud data after coordinate conversion and the point cloud data collected by the blind spot radar for each group of detection radars, the method further includes: For each group of detection radars, based on the posture change information of the main radar in the group of detection radars, the overlapping part of the detection area of ​​the main radar and the blind radar in the group of detection radars is determined, and used as the overlapping area of ​​the group of detection radars; The point cloud data in the overlapping area of ​​each group of detection radars are deduplicated to cluster the deduplicated point cloud data.

4. The method for dynamic fusion of radar data according to claim 3, characterized in that: For each group of detection radars, based on the posture change information of the main radar in the group of detection radars, determining the overlapping part of the detection area of ​​the main radar and the blind radar in the group of detection radars, and using it as the overlapping area of ​​the group of detection radars, includes: Obtaining the detection area corresponding to the initial posture of the first main radar; Determining a detection area of ​​the first main radar after the posture change based on the posture change information of the first main radar; Based on the detection area of ​​the first main radar after the posture change and the detection area of ​​the first blind radar, the overlapping part of the detection area of ​​the main radar and the blind radar in the first group of detection radars is determined; wherein the first main radar and the first blind radar are respectively the main radar and the blind radar in the first group of detection radars, and the first group of detection radars is any group of detection radars.

5. The method for dynamic fusion of radar data according to claim 3, characterized in that: Deduplication of point cloud data in the overlapping area of ​​each group of detection radars includes: The point cloud data collected by the blind radar in the overlapping area of ​​each group of detection radars is removed.

6. The method for dynamic fusion of radar data according to claim 1, characterized in that: The radar target information of each group of detection radars and the visual target information collected by the visual device are subjected to radar visual data fusion, including: Based on the radar coordinate system of each group of detection radars and the visual coordinate system of the visual device, the radar target information of each group of detection radars and the visual target information collected by the visual device are subjected to coordinate transformation, area splicing and identity merging to obtain fused target information.

7. The method for dynamic fusion of radar data according to claim 1, characterized in that: Before clustering the point cloud data after coordinate conversion and the point cloud data collected by the blind spot compensation radar for each group of detection radars to obtain radar target information of the group of detection radars, the method further includes: Outliers are removed from each point cloud data, so as to perform coordinate transformation and clustering based on the point cloud data after outliers are removed.

8. A dynamic fusion device for radar data, characterized in that: Applied to a radar perception system, the system includes a visual device and at least one set of detection radars, each set of detection radars includes a main radar and a blind radar; the device includes: A matrix construction module, used to determine the coordinate transformation matrix of each main radar based on the posture change information of the main radar; A coordinate conversion module is used to convert the coordinates of the point cloud data collected by the main radar into the radar coordinate system where the blind spot radar corresponding to the main radar is located based on the coordinate conversion matrix of each main radar, so as to obtain the point cloud data after the coordinate conversion; The first fusion module is used to cluster the point cloud data after coordinate conversion and the point cloud data collected by the blind spot radar for each group of detection radars to obtain radar target information of the group of detection radars; The second fusion module is used to perform radar data fusion on the radar target information of each group of detection radars and the visual target information collected by the visual device to perform target tracking.

9. A radar-visual integrated machine, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

Images