A multi-device radar data fusion method

Abstract

The application discloses a kind of multi-device radar data fusion methods, it is related to radar signal processing technical field, the method includes: time synchronization processing to multiple radar devices;Obtain the coordinates of the remaining radar devices when each radar device takes itself as coordinate origin;According to the orientation angle of radar device, adjust the coordinates of the remaining radar devices;Select one of the radar devices as reference device, the remaining radar devices as observed device, according to the first point cloud matrix formed by the coordinates of the remaining radar devices adjusted according to reference device, and the second point cloud matrix formed by the coordinates of the remaining radar devices adjusted according to observed device, construct the coordinate conversion matrix of observed device mapped to reference device, so as to fuse the data of multiple observed devices to a reference device.The method solves the problem that data from different radar devices exist time synchronization, coordinate system difference, effectively realizes seamless fusion between the data of multiple radar devices.

Description

Technical Field

[0001] This invention relates to the field of radar signal processing technology, and in particular to a method for multi-device radar data fusion. Background Technology

[0002] The increasingly complex road traffic environment is driving the demand for intelligent traffic management systems to become more intelligent, comprehensive, accurate, and real-time. Millimeter-wave radar, with its technological advantages, has extremely important applications in the field of intelligent transportation. Millimeter-wave radar measures the distance, speed, and angle of surrounding objects by emitting high-frequency electromagnetic waves and receiving echoes. Its advantages include rapid acquisition of speed information, enabling it to track vehicles ahead and control speed, and better robustness in different weather conditions. However, in practical applications, the performance and capabilities of a single radar are limited. To better understand and master the surrounding environment, fusing data from multiple radar devices is becoming increasingly important. Multi-device radar data fusion technology aims to combine information from different radar sensors to provide more comprehensive and accurate target information and environmental perception. By integrating the advantages of different radars, this technology can overcome the limitations of individual radar operation and improve the reliability of target detection and tracking. Summary of the Invention

[0003] To address the aforementioned problems and technical requirements, the inventors have proposed a multi-device radar data fusion method. This method resolves issues such as time synchronization and coordinate system differences in data from different radar devices, effectively achieving seamless fusion of data from multiple radar devices. The technical solution of this invention is as follows: A method for fusion of radar data from multiple devices includes the following steps: Time synchronization processing for multiple radar devices; Obtain the coordinates of the other radar devices when each radar device has itself as the origin; Adjust the coordinates of the remaining radar devices according to the orientation angle of the radar device; One radar device is selected as the reference device, and the remaining radar devices are the observed devices. Based on the first point cloud matrix formed by the adjusted coordinates of the remaining radar devices corresponding to the reference device, and the second point cloud matrix formed by the adjusted coordinates of the remaining radar devices corresponding to the observed devices, a coordinate transformation matrix is ​​constructed to map the observed devices to the reference device, thereby fusing the data of multiple observed devices into one reference device.

[0004] A further technical solution involves time synchronization processing for multiple radar devices, including: Specific synchronization signals are sent to multiple radar devices multiple times, and a series of timestamp data of the synchronization signals received by the multiple radar devices are recorded. By statistically analyzing a series of timestamp data, the timestamp offset and delay between multiple radar devices are estimated and corrected to achieve accurate alignment of the timestamps of multiple radar devices.

[0005] A further technical solution involves synchronizing multiple radar devices in time, including: Each radar device includes a radar module and a camera module. When the frame rates of the radar module and the camera module are different, a timestamp is added to the frames of each module, and the timestamp difference between the frames of the two modules is calculated and denoted as . ; Linear interpolation of the timestamps is performed based on the ratio of the frame rates of the two modules and the difference in their timestamps to align the timestamps of the two modules; where the linearly interpolated timestamps are... The expression is: ; In the formula, The timestamp of the radar frame. For radar frame rate, This refers to the video frame rate.

[0006] A further technical solution involves obtaining the coordinates of the other radar devices when each radar device uses itself as the origin, including: For each radar device, when it is regarded as the reference device, the other radar devices are regarded as the observed devices; Calculate the azimuth and straight-line distance of the reference device relative to the observed device; Using the location of the reference device as the origin of the coordinate system, the distance components of the x and y axes corresponding to all observed devices are calculated based on the azimuth angle and straight-line distance, and these are used as the coordinates of the observed devices relative to the reference device.

[0007] A further technical solution involves calculating the azimuth angle of the reference device relative to the observed device, including: The direction vector formed by the reference device relative to the observed device is calculated using the following expression: , ; In the formula: ; , For the reference equipment's latitude and longitude, , The latitude and longitude of the observed device; Let be the component of the direction vector in the east-west direction. This represents the north-south component of the direction vector; Calculate the azimuth angle of the reference device relative to the observed device based on the direction vector. The expression is: .

[0008] A further technical solution involves calculating the straight-line distance between the reference device and the observed device. The expressions include: ; ; ; In the formula, , For the reference equipment's latitude and longitude, , The latitude and longitude of the observed device; This represents the component of the direction vector in the latitudinal direction. This represents the component of the direction vector along the longitude direction. It is the length of the Earth's diameter.

[0009] A further technical solution involves adjusting the coordinates of other radar devices based on the orientation angle of the radar device, including: For each radar device, when it is regarded as the reference device, the other radar devices are regarded as the observed devices; Construct a translation matrix based on the translation of the observed device relative to the reference device; Construct a first rotation matrix based on the orientation angle of the observed device; Multiply the translation matrix by the first rotation matrix to obtain the transformation matrix; The coordinates of the other radar devices corresponding to the observed device are transformed using a transformation matrix to obtain the adjusted coordinates of the other radar devices.

[0010] A further technical solution involves constructing a coordinate transformation matrix mapping the observed device to the reference device, including: Calculate the covariance matrix of the first and second point cloud matrices after centering; Calculate the second rotation matrix and translation vector based on the covariance matrix; The second rotation matrix and translation vector are combined to form the coordinate transformation matrix mapping the observed device to the reference device, expressed as: ; In the formula, R This is the second rotation matrix. t It is a translation vector.

[0011] A further technical solution involves calculating the covariance matrix of the centered first and second point cloud matrices, including: Calculate the centroids of the first and second point cloud matrices, and subtract the corresponding centroid coordinates from the coordinates of each point in the point cloud matrix to obtain the centered first point cloud matrix. AA Second point cloud matrix BB In this matrix, each column of the first and second point cloud matrices represents the coordinates of a three-dimensional point. First point cloud matrix AA Second point cloud matrix BB The covariance matrix is ​​expressed as: .

[0012] A further technical solution involves calculating the second rotation matrix and translation vector based on the covariance matrix, including: Singular value decomposition of the covariance matrix yields the left singular vector matrix. U and right singular vector matrix V And based on this, the second rotation matrix is ​​calculated, expressed as: ; If the determinant of the second rotation matrix is ​​less than 0, then invert the elements of the third row of the second rotation matrix; Based on the centroid of the second rotation matrix and the first point cloud matrix centroid of the second point cloud matrix Calculate the translation vector, expressed as: .

[0013] The beneficial technical effects of this invention are: This method first performs time synchronization processing on multiple radar devices to facilitate subsequent data processing and fusion. Then, it acquires the coordinates of the observed device on each radar device and adjusts these coordinates based on the orientation angle. Finally, it selects a reference device and constructs a coordinate transformation matrix mapping the observed device to the reference device, thereby fusing the data from multiple observed devices onto a single reference device. This method can efficiently and accurately achieve registration and fusion of multiple radar devices, enabling accurate fusion of multiple radar point clouds in space. This technology can overcome the limitations of individual radar operation and improve the reliability of target detection and tracking. Attached Figure Description

[0014] Figure 1 This is a flowchart of the multi-device radar data fusion method provided in this application.

[0015] Figure 2 This is a schematic diagram showing the coordinate adjustment of the reference device and the observed device based on the device orientation angle provided in this application.

[0016] Figure 3 This is a diagram showing the effect of data fusion from the four radar devices provided in this application. Detailed Implementation

[0017] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0018] Please refer to Figure 1 As shown, this application provides a multi-device radar data fusion method, which specifically includes the following steps: Step 1: Perform time synchronization processing on multiple radar devices.

[0019] A specific synchronization signal is sent to a system consisting of multiple radar devices, and the timestamp of each radar device receiving the synchronization signal is recorded. By repeating the transmission and reception of the synchronization signal multiple times, a series of timestamp data can be obtained. Statistical analysis of this series of timestamp data is performed to estimate the timestamp offset and delay between the multiple radar devices, and corrections are made to achieve accurate alignment of the timestamps of the multiple radar devices.

[0020] Each radar device includes a radar module and a camera module. When the frame rates of the radar module and the camera module are different, a timestamp interpolation method is used for asynchronous processing and data alignment. This involves adding a timestamp to each module's frame and then using interpolation to achieve data alignment. Assume the timestamp of the radar frame from the radar module is... The timestamps of the video frames from the camera module are First, calculate the timestamp difference between the frames of the two modules. Then, based on the ratio of the frame rates of the two modules and the difference in timestamps, linear interpolation of the timestamps is performed to align the timestamps of the two modules. Specifically, assuming the radar frame rate is... The video frame rate is The timestamp after linear interpolation is The expression is: (1) In this way, radar frames can be aligned on the timeline for subsequent data processing and fusion.

[0021] Step 2: Obtain the coordinates of the other radar devices when each radar device is used as its own origin.

[0022] For each radar device, when it is regarded as a reference device, the other radar devices are regarded as the observed devices. This step specifically includes the following sub-steps: Step 2.1: Calculate the azimuth and straight-line distance of the reference device relative to the observed device.

[0023] The azimuth angle is the angle between the direction from the reference device, clockwise, to the observed device, and true north. The azimuth angle and straight-line distance are calculated by converting the latitude and longitude of the two devices to radians and using trigonometric and arcsine function formulas.

[0024] 1) Calculation process of azimuth: The direction vector formed by the reference device relative to the observed device is calculated using the following expression: (2) In the formula: ; , For the reference equipment's latitude and longitude, , The latitude and longitude of the observed device; Let be the component of the direction vector in the east-west direction. Let be the component of the direction vector in the north-south direction. The difference in longitude between the reference device and the observed device is expressed in radians.

[0025] Final azimuth By and Passed as a parameter The function converts the result into degrees for calculation. The expression is: (3) 2) Calculation process of straight-line distance: The length of the direction vector is calculated by representing the difference in latitude and longitude between the reference device and the observed device in radians. This vector is then multiplied by the diameter of the Earth to obtain the actual distance between the two locations.

[0026] (4) (5) (6) In the formula, This represents the component of the direction vector in the latitudinal direction. This represents the component of the direction vector along the longitude direction; This is the length of the Earth's diameter in meters. This value can be used to scale the distance to match the size of the Earth.

[0027] Step 2.2: Using the location of the reference device as the origin of the coordinate system, calculate the distance components of the x and y axes of all observed devices based on the azimuth and straight-line distance, and use these as the coordinates of the observed devices relative to the reference device.

[0028] Given that the distance components of the reference device relative to its own x and y axes are 0, the positions of multiple points can be determined from the perspective of the reference device. N Taking +1 radar device as an example, each device will receive the remaining... NCoordinates of each device These coordinate points lay the foundation for subsequent calculations of the coordinate transformation matrix.

[0029] Step 3: Adjust the coordinates of the remaining radar devices according to the orientation angle of the radar device.

[0030] like Figure 2 As shown, the device being observed Q Relative to reference device P Taking coordinate adjustment as an example, the specific steps include the following: Step 3.1: The observed device can be obtained using the same method as in Step 2. Q (Will Q When considered as a reference device, the remaining corresponding N Coordinates of the observed equipment It is called .

[0031] Step 3.2: Based on the observed device Q Relative to reference device P Translation amount ,structure The translation matrix is ​​expressed as: (7) Step 3.3: Based on the observed device Q Orientation angle structure The first rotation matrix is ​​represented as: (8) In the formula, .

[0032] Step 3.4: Multiply the translation matrix by the first rotation matrix to obtain the transformation matrix. Then use the transformation matrix T 1 pair The device under observation is transformed to obtain the adjusted device. Q The coordinates of the other corresponding radar devices .

[0033] Step 4: Select one radar device as the reference device and the remaining radar devices as the observed devices. Based on the first point cloud matrix formed by the adjusted coordinates of the remaining radar devices corresponding to the reference device, and the second point cloud matrix formed by the adjusted coordinates of the remaining radar devices corresponding to the observed devices, construct a coordinate transformation matrix mapping the observed devices to the reference device. This allows the data from multiple observed devices to be fused onto a single reference device. Specifically, this includes the following sub-steps: Step 4.1: Input two 3D matrices A andB ,in A Reference equipment P The first point cloud matrix formed by the coordinates of the remaining radar devices after corresponding adjustments. B The device being observed Q The second point cloud matrix is ​​formed by the coordinates of the remaining radar devices after adjustment. The purpose of calculating the coordinate transformation matrix is ​​to... B Matched A Among them, the first point cloud matrix A Second point cloud matrix B Each column represents the coordinates of a three-dimensional point. .

[0034] Step 4.2: Calculate the covariance matrix of the first and second point cloud matrices after centering.

[0035] Calculate the first point cloud matrix A Second point cloud matrix B Each's center of mass and Then, subtract the corresponding centroid coordinates from the coordinates of each point in the point cloud matrix to obtain the first centered point cloud matrix. AA Second point cloud matrix BB Then the first point cloud matrix AA Second point cloud matrix BB covariance matrix H Represented as: .

[0036] Step 4.3: Calculate the second rotation matrix and translation vector based on the covariance matrix.

[0037] 1) The calculation process of the second rotation matrix: Singular value decomposition of the covariance matrix yields the left singular vector matrix. U Singular value vectors S and right singular vector matrix V And calculate the second rotation matrix accordingly. R , represented as: .examine R If the determinant of the matrix is ​​less than 0, then rotate the second rotation matrix. R Invert the elements in the third row.

[0038] 2) Calculation process of translation vector: According to the second rotation matrix R The centroid of the first point cloud matrix centroid of the second point cloud matrix Calculate the translation vector t , represented as: .

[0039] Step 4.4: Combine the second rotation matrix and translation vector into coordinate transformation matrix T 2, represented as: (9) By employing the above method to integrate and optimize data from different radar devices, the advantages of each device can be fully utilized, improving the accuracy of target detection and the stability of target tracking. The fused data provides more comprehensive, multi-angle target information, helping users better understand the surrounding environment and target characteristics. This embodiment uses the data fusion of four radar devices at a certain intersection as an example, with the device in the north direction as the reference device. A schematic diagram of the fused data is shown below. Figure 3 As shown, this method can efficiently and accurately achieve registration and fusion of multiple radar devices.

[0040] The above descriptions are merely preferred embodiments of this application, and the present invention is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included within the protection scope of the present invention.

Claims

1. A method for fusion of radar data from multiple devices, characterized in that, The method includes: Time synchronization processing for multiple radar devices; Obtain the coordinates of the other radar devices when each radar device has itself as the origin; Adjust the coordinates of the remaining radar devices according to the orientation angle of the radar device; One radar device is selected as the reference device, and the remaining radar devices are the observed devices. Based on the first point cloud matrix formed by the adjusted coordinates of the remaining radar devices corresponding to the reference device, and the second point cloud matrix formed by the adjusted coordinates of the remaining radar devices corresponding to the observed devices, a coordinate transformation matrix is ​​constructed to map the observed devices to the reference device, thereby fusing the data of multiple observed devices into one reference device. The construction of the coordinate transformation matrix mapping the observed device to the reference device includes: Calculate the centroids of the first and second point cloud matrices, and subtract the corresponding centroid coordinates from the coordinates of each point in the point cloud matrix to obtain the centered first point cloud matrix. AA Second point cloud matrix BB In this matrix, each column of the first point cloud matrix and the second point cloud matrix represents the coordinates of a three-dimensional point. First point cloud matrix AA Second point cloud matrix BB The covariance matrix is ​​expressed as: ; Singular value decomposition is performed on the covariance matrix to obtain the left singular vector matrix. U and right singular vector matrix V And calculate the second rotation matrix accordingly. R , is represented as: ; If the determinant of the second rotation matrix is ​​less than 0, then invert the elements of the third row of the second rotation matrix; Based on the centroid of the second rotation matrix and the first point cloud matrix centroid of the second point cloud matrix Calculate the translation vector t , is represented as: ; The second rotation matrix and translation vector are combined to form the coordinate transformation matrix mapping the observed device to the reference device, expressed as: 。 2. The multi-device radar data fusion method according to claim 1, characterized in that, Time synchronization processing for multiple radar devices includes: A specific synchronization signal is sent to multiple radar devices multiple times, and a series of timestamp data of the synchronization signal received by the multiple radar devices is recorded. By statistically analyzing the series of timestamp data, the timestamp offset and delay between multiple radar devices are estimated and corrected to achieve accurate alignment of the timestamps of multiple radar devices.

3. The multi-device radar data fusion method according to claim 2, characterized in that, Synchronizing multiple radar devices in time also includes: Each radar device includes a radar module and a camera module. When the frame rates of the radar module and the camera module are different, a timestamp is added to the frames of each module, and the timestamp difference between the frames of the two modules is calculated and denoted as . ; Linear interpolation of the timestamps is performed based on the ratio of the frame rates of the two modules and the timestamp difference to align the timestamps of the two modules; wherein the linearly interpolated timestamps are... The expression is: ； In the formula, The timestamp of the radar frame. For radar frame rate, This refers to the video frame rate.

4. The multi-device radar data fusion method according to claim 1, characterized in that, Obtain the coordinates of the other radar devices when each radar device has its own origin as the coordinate system, including: For each radar device, when it is regarded as the reference device, the other radar devices are regarded as the observed devices; Calculate the azimuth angle and straight-line distance of the reference device relative to the observed device; Using the location of the reference device as the origin of the coordinate system, the distance components of the x and y axes corresponding to all observed devices are calculated based on the azimuth angle and straight-line distance, and used as the coordinates of the observed devices relative to the reference device.

5. The multi-device radar data fusion method according to claim 4, characterized in that, Calculating the azimuth angle of the reference device relative to the observed device includes: The direction vector formed by the reference device relative to the observed device is calculated as follows: ， ； In the formula: ; , The latitude and longitude of the reference device. , The latitude and longitude of the observed device; Let be the component of the direction vector in the east-west direction. The component of the direction vector in the north-south direction; Calculate the azimuth angle of the reference device relative to the observed device based on the direction vector. The expression is: 。 6. The multi-device radar data fusion method according to claim 4, characterized in that, Calculate the straight-line distance between the reference device and the observed device. The expressions include: ； ； ； In the formula, , The latitude and longitude of the reference device. , The latitude and longitude of the observed device; This represents the component of the direction vector in the latitudinal direction. This represents the component of the direction vector along the longitude direction. It is the length of the Earth's diameter.

7. The multi-device radar data fusion method according to claim 1, characterized in that, Adjusting the coordinates of the remaining radar devices according to the orientation angle of the radar device includes: For each radar device, when it is regarded as the reference device, the other radar devices are regarded as the observed devices; Construct a translation matrix based on the translation amount of the observed device relative to the reference device; Construct a first rotation matrix based on the orientation angle of the observed device; Multiply the translation matrix by the first rotation matrix to obtain the transformation matrix; The coordinates of the other radar devices corresponding to the observed device are transformed using the transformation matrix to obtain the adjusted coordinates of the other radar devices.

Images