Scene flow estimation method and system based on millimeter wave radar
By constructing a supervision strategy based on vision-inertial sensors and a sliding window graph optimization framework based on inertial neural networks, the accuracy and drift problems of scene flow estimation in millimeter-wave radar are solved, and high-precision and stable scene flow estimation is achieved.
Patent Information
- Application Number
- CN202511228795.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-29
- Publication Date
- 2025-12-12
AI Technical Summary
Existing scene flow estimation methods for millimeter-wave radar suffer from limited accuracy and drift, especially in dynamic target modeling. Furthermore, visual-inertial sensors cannot effectively assist radar modeling when closed-loop optimization conditions are insufficient.
A millimeter-wave radar scene flow estimation model based on a supervision strategy that integrates vision and inertial sensors is constructed. By acquiring relative pose information and IMU data, an inertial neural network is built, and combined with a sliding window graph optimization framework, accurate estimation of scene flow and anti-drift capability are achieved.
It improves the global and local accuracy of scene flow estimation for millimeter-wave radar, enhances the reliability and stability of the calculation results, and is suitable for robust scene flow estimation in complex environments.
Smart Images
Figure CN121120694A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent traffic perception and machine vision, and in particular to a scene flow estimation method and system based on millimeter-wave radar. Background Technology
[0002] Millimeter-wave radar (mmWave Radar) is widely used in smart cars, drones, and security applications due to its excellent environmental adaptability. Unlike visual or lidar, mmWave radar operates at millimeter-level wavelengths, making it robust to visually degraded environments such as fog, darkness, rain, and snow, and capable of continuously sensing target distance, speed, and direction. However, due to the signal characteristics of mmWave radar, the resulting point clouds are sparse and have low spatial resolution, making high-quality scene flow estimation based on mmWave radar particularly difficult.
[0003] Traditional radar scene flow estimation methods can be broadly classified into three categories: (1) LiDAR supervision: The dense scene flow provided by the LiDAR is used as the supervision signal to train the LiDAR-side model. Although methods like CMFlow have high accuracy, the equipment cost is high. (2) Pure radar self-supervision: such as RAFlow, which uses the radial velocity of the radar itself as implicit supervision to achieve end-to-end self-supervision learning. However, due to the lack of geometric information, the accuracy is limited. (3) Image / visual assistance: Using the optical flow information of the camera to assist radar modeling improves the scene flow estimation effect of dynamic targets, but cannot solve the problem of missing three-dimensional structural information.
[0004] Meanwhile, visual-inertial (VI) sensors are widely integrated into commercial platforms as a low-cost solution. Existing systems such as VINS and ORB-SLAM3 can estimate relative pose, but due to the lack of closed-loop optimization conditions in some scenarios, time drift exists and they cannot be directly used for supervisory radar modeling. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a scene flow estimation method and system based on millimeter-wave radar, thereby achieving high-quality scene flow estimation based on millimeter-wave radar.
[0006] Therefore, the technical solution adopted by the present invention is as follows: This invention provides a scene flow estimation method based on millimeter-wave radar, the method comprising: Acquire continuous frame point cloud data from millimeter-wave radar and perform preprocessing; The preprocessed point cloud data is input into the millimeter-wave radar scene flow estimation model, which outputs the three-dimensional motion vector of each point to obtain the scene flow estimation result. The millimeter-wave radar scene flow estimation model is established based on a supervision strategy that integrates vision and inertial sensors, and is trained with the goal of minimizing the supervision term loss. The supervision strategy that integrates vision and inertial sensors specifically obtains the relative pose information between adjacent frames of point cloud data, and combines it with IMU data to construct an inertial neural network to obtain the mapping relationship between IMU data and three-dimensional translation. The scene flow is then predicted based on the mapping relationship.
[0007] According to the above scheme, continuous frame point cloud data of millimeter-wave radar is obtained, where each frame of point cloud data includes three-dimensional spatial points and their velocity and energy; the preprocessing specifically includes using a point cloud feature extraction network to extract local geometric features of each frame of point cloud in the frame sequence, generating point-level feature tensors, and obtaining the initial rigid body transformation matrix and the initial scene flow.
[0008] According to the above scheme, the inertial neural network specifically acquires the relative pose information between adjacent frames of point cloud data through the vision-inertial SLAM system and constructs it in combination with IMU data, and outputs the translation vector of continuous frames of point cloud data and its uncertainty; the static point supervision term loss of point cloud data specifically optimizes the pre-integration residual of the inertial neural network output and the IMU data by tightly coupling, outputting the optimized rigid body transformation matrix, and using the optimized rigid body transformation matrix to map the point cloud data of the previous frame to the next frame, so as to calculate the static point supervision term loss of point cloud data.
[0009] According to the above scheme, the loss of the dynamic three-dimensional motion supervision item of point cloud data is specifically calculated based on the three-dimensional trajectory of each dynamic point in the point cloud data.
[0010] According to the above scheme, the loss of the optical flow information data supervision item is specifically calculated based on the set of all dynamic points in the point cloud data and the optical flow information.
[0011] According to the above scheme, the supervision item loss is obtained by weighting and summing the supervision item loss of static points in point cloud data, the supervision item loss of dynamic three-dimensional motion in point cloud data, the supervision item loss of optical flow information data, and the supervision item loss of chamfer distance information data.
[0012] According to the above scheme, the output of the inertial neural network is tightly coupled with the IMU pre-integration residual for optimization, which is specifically obtained by constructing a sliding window graph optimization framework. The nodes of the sliding window graph optimization framework are sequences of state variables containing position, velocity and attitude. The two types of factor edges of the sliding window graph optimization framework include IMU pre-integration factor edges and neural network translation supervision edges.
[0013] The present invention also provides a scene flow estimation system based on millimeter-wave radar, the system comprising: The data processing module is used to acquire continuous frame point cloud data from millimeter-wave radar and perform preprocessing. The scene flow estimation module is used to input preprocessed point cloud data into the millimeter-wave radar scene flow estimation model, output the three-dimensional motion vector of each point, and obtain the scene flow estimation result. The millimeter-wave radar scene flow estimation model is established based on a supervision strategy that integrates vision and inertial sensors, and is trained with the goal of minimizing the supervision term loss. The supervision strategy that integrates vision and inertial sensors specifically obtains the relative pose information between adjacent frames of point cloud data, and combines it with IMU data to construct an inertial neural network, obtain the mapping relationship between IMU data and three-dimensional translation, and predict the scene flow based on the mapping relationship.
[0014] The present invention also provides a computer storage medium storing a computer program executable by a processor, the computer program executing the scene flow estimation method based on millimeter-wave radar described above.
[0015] The beneficial effects of this invention are as follows: This invention constructs a millimeter-wave radar scene flow estimation model based on a supervision strategy that integrates vision and inertial sensors. It utilizes relative pose in conjunction with an inertial neural network to learn the mapping relationship between IMU sequences and 3D translation, thereby achieving precise guidance for millimeter-wave radar scene flow estimation. Simultaneously, it calculates the supervision term loss and trains the model with the goal of minimizing the supervision term loss. By fusing static and dynamic supervision signals through a unified loss function, it effectively improves the global and local accuracy of scene flow estimation, compensates for the shortcomings of radar itself in dynamic detail modeling, and enhances the accuracy and reliability of millimeter-wave radar-based scene flow estimation.
[0016] Furthermore, this invention constructs a sliding window graph optimization framework to tightly couple the output of the inertial neural network with the pre-integrated residual of the IMU, thereby achieving joint optimization of the state. This enables the millimeter-wave radar scene flow estimation model to have adaptive weighting capability and anti-drift performance, enhancing the reliability and stability of the calculation results. Attached Figure Description
[0017] Figure 1 This is a schematic diagram of the process flow of a scene flow estimation method based on millimeter-wave radar according to an embodiment of the present invention; Figure 2 This is a schematic diagram showing the installation positions of the millimeter-wave radar and the vision-inertial sensor according to an embodiment of the present invention; Figure 3 This is a diagram of the inertial neural network architecture according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the static point supervision term loss calculation process according to an embodiment of the present invention; Figure 5This is a schematic diagram of the optical flow projection supervision term loss calculation process according to an embodiment of the present invention; Figure 6(a) is a comparison of scene flow estimation results in an embodiment of the present invention; Figure 6(b) is a comparison diagram of the three-dimensional trajectory results of an embodiment of the present invention; Figure 7 This is a schematic diagram of the system structure of a scene flow estimation system based on millimeter-wave radar according to an embodiment of the present invention. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0019] To address the limitations of existing radar scene flow estimation methods, such as limited accuracy and susceptibility to time drift, this invention provides a scene flow estimation method based on millimeter-wave radar, such as... Figure 1 As shown, the method includes: Acquire continuous frame point cloud data from millimeter-wave radar and perform preprocessing; The preprocessed point cloud data is input into the millimeter-wave radar scene flow estimation model, which outputs the three-dimensional motion vector of each point to obtain the scene flow estimation result. The millimeter-wave radar scene flow estimation model is established based on a supervision strategy that integrates vision and inertial sensors, and is trained with the goal of minimizing the supervision term loss. The supervision strategy that integrates vision and inertial sensors specifically obtains the relative pose information between adjacent frames of point cloud data, and combines it with IMU data to construct an inertial neural network to obtain the mapping relationship between IMU data and three-dimensional translation. The scene flow is then predicted based on the mapping relationship.
[0020] Specifically, during the initial construction and training of the millimeter-wave radar scene flow estimation model, sparse point cloud data of consecutive frames are collected using millimeter-wave radar equipment to form a frame sequence; wherein, each frame of the frame sequence contains point cloud data... Specifically, this includes three-dimensional spatial point data as well as velocity and energy attribute data; Point cloud feature extraction networks (such as KPConv and PointNet++) are used to extract local geometric features from the point cloud of each frame in the frame sequence, generating point-level feature tensors. The point cloud feature extraction network is specifically KPConv or PointNet++; and utilizes and Prediction is performed to initially obtain the rigid body transformation matrix from the current frame to the next frame. and the initial scene flow prediction results at each point ,Right now:
[0021] in, Indicates the first The first frame The location of each point.
[0022] Specifically, the millimeter-wave radar is a 4D radar device capable of outputting five-dimensional information including target reflection intensity, range, velocity, azimuth angle, and elevation angle. The installation positions of the millimeter-wave radar and the visual-inertial sensor are as follows: Figure 2 As shown. The point cloud is output continuously in frames, denoted as... Each of them .
[0023] Specifically, such as Figure 3 The inertial neural network shown specifically acquires the relative pose information between adjacent frames of point cloud data through a vision-inertial SLAM system and combines it with IMU data to construct the output of the translation vector of continuous frames of point cloud data and its uncertainty.
[0024] Specifically, the supervised loss includes static point data supervised loss, dynamic 3D motion data supervised loss, optical flow information data supervised loss, and feature consistency-based self-supervised loss. These losses are obtained by weighting and summing the static point data supervised loss, dynamic 3D motion data supervised loss, optical flow information data supervised loss, and feature consistency-based self-supervised loss. In this embodiment, the feature consistency-based self-supervised loss is represented as the Chamfer distance information data supervised loss, where Chamfer distance represents the chamfer distance. It is used to evaluate the "nearest neighbor" metric of the similarity between two sets of discrete point clouds. By calculating the sum (or average) of the distances from each point to the nearest point in the other set of points, the overlap and alignment of the two sets of point clouds are quantified from a bidirectional perspective.
[0025] Specifically, the static point supervision term loss of point cloud data is obtained in the following way: The output of the inertial neural network is tightly coupled with the pre-integrated residual of the IMU for optimization, and the optimized rigid body transformation matrix is output. The optimized rigid body transformation matrix is used to map the point cloud data of the previous frame to the next frame, and the static point supervision term loss is calculated.
[0026] Here, IMU data represents inertial measurement unit data; specifically, IMU data can be represented as: , where a represents the acceleration in the x, y, and z directions, and ω represents the angular velocity in the x, y, and z directions.
[0027] The relative pose information between adjacent frames and IMU data are input into the inertial neural network, which outputs an estimated translation vector. and its uncertainty covariance matrix :
[0028] in, , , These are the estimated translation vectors. The uncertainty factor estimated by translation in the three directions of x, y, and z is in the form of three log standard deviations.
[0029] Preferably, the inertial neural network is trained using maximum likelihood loss based on the estimated translation vector and the uncertainty covariance matrix:
[0030] in, For true translation, To estimate the translation vector, N is the number of training samples.
[0031] Specifically, the output of the inertial neural network is tightly coupled with the pre-integrated residual of the IMU for optimization, which is achieved by constructing a sliding window graph optimization framework. In this framework, the nodes of the sliding window graph optimization are sequences of state variables. , , , These represent position, velocity, and attitude, respectively; the two types of factor edges in the sliding window graph optimization framework are IMU pre-integration factor edges and neural network translation supervision edges. The IMU pre-integration factor compares the actual changes (position, velocity, attitude) between the two current states with the results predicted by IMU integration to calculate the error. The IMU pre-integration residual term is:
[0032] In the formula, It is the first Position and velocity at any given moment It is the vector of gravitational acceleration. It is the first The frame rotation matrix, It is the first Frame pose (quaternion). It is the first Frame to IMU pre-integration between frames.
[0033] The neural network translation supervision edge is constructed based on the translation residual term in the following formula:
[0034] In the formula, To estimate the translation vector, the current estimate is , indicating from the first Frame position conversion back to the first The relative translation in the frame coordinate system, therefore this residual term represents the error between the currently estimated relative position change and the translation predicted by the neural network.
[0035] Specifically, the objective function of the sliding window graph optimization framework is the weighted minimization of the joint residuals: ,in, This represents the IMU pre-integration residual term. This represents the relative pose error of the visual system. This represents the relative translation residual term provided by the neural network; minimizing this weighted sum yields the rigid body transformation estimate with drift compensation. It has higher accuracy than any single source and is anti-drift.
[0036] Specifically, the loss function can be expressed as:
[0037] in, This represents a self-supervised term based on feature consistency. To adjust the static point supervision term loss of hyperparameters; static point supervision term loss of point cloud data. ,in To make the current frame point The actual motion of static points is obtained by mapping rigid body transformation to the next frame. In the formula, Represents the identity matrix. Represents the rotation matrix. Represents the translation vector. The calculation process for static point supervision item loss in point cloud data is as follows: Figure 4 As shown.
[0038] The loss of the dynamic 3D motion supervision term for point cloud data is specifically calculated based on the 3D trajectory of each dynamic point in the point cloud data: Loss of the dynamic 3D motion supervision term for point cloud data ,in , Let be the 3D trajectory of the i-th dynamic point in the point cloud data at time t.
[0039] Specifically, optical flow refers to the instantaneous velocity of motion in point cloud data. The optical flow information data supervision term loss is calculated based on the set of all dynamic points in the point cloud data and the optical flow information: Optical Flow Information Data Supervision Term Loss ,in This represents the set of all dynamic points in the point cloud data. Indicates optical flow. This represents the initial scene flow prediction result. The calculation process for the dynamic 3D motion supervision term loss and the optical flow information data supervision term loss is as follows: Figure 5 As shown, the optical flow information data of the input image is obtained using the PWC-Net model, and the optical flow information data is masked using a foreground mask to obtain the processed optical flow information data. This processed optical flow information data is then combined with the 3D trajectory of each dynamic point in the point cloud data obtained by the visual-inertial SLAM system and compared with the initial scene flow prediction results to calculate the optical flow information data supervision term loss.
[0040] In the actual prediction process, only the millimeter-wave radar point cloud data from the previous and current frames needs to be input to directly predict the three-dimensional motion vector of each point. It achieves dependency-free, lightweight, and highly robust scene flow estimation, applicable to various complex or perceptually degraded environments.
[0041] To verify the effectiveness of the method of this invention, a motion trajectory was preset, and the millimeter-wave radar scene flow estimation method proposed in this invention was deployed for evaluation during the movement of the micro UAV. The scene flow estimation results comparison diagram is shown in Figure 6(a). The estimated trajectory of the UAV is highly consistent with the actual trajectory, indicating that the method has good motion estimation accuracy. In addition, the three-dimensional trajectory result comparison diagram in Figure 6(b) shows the three-dimensional positioning error in the X, Y, and Z directions. The overall error fluctuation is small and the stability is strong, indicating that the method can also stably output pose estimation in complex environments such as without GPS and low light.
[0042] In addition, embodiments of the present invention also provide a scene flow estimation system based on millimeter-wave radar, such as Figure 7 As shown, the system includes: The data processing module is used to acquire continuous frame point cloud data from millimeter-wave radar and perform preprocessing. The scene flow estimation module is used to input preprocessed point cloud data into the millimeter-wave radar scene flow estimation model, output the three-dimensional motion vector of each point, and obtain the scene flow estimation result. The millimeter-wave radar scene flow estimation model is established based on a supervision strategy that integrates vision and inertial sensors, and is trained with the goal of minimizing the supervision term loss. The supervision strategy that integrates vision and inertial sensors specifically obtains the relative pose information between adjacent frames of point cloud data, and combines it with IMU data to construct an inertial neural network, obtain the mapping relationship between IMU data and three-dimensional translation, and predict the scene flow based on the mapping relationship.
[0043] The various modules or mechanisms of the system are mainly used to implement the various steps of the above method embodiments, and will not be described in detail here.
[0044] In addition, embodiments of the present invention also provide a computer storage medium storing a computer program executable by a processor, the computer program executing the scene flow estimation method based on millimeter-wave radar described above.
[0045] This invention provides a scene flow estimation method based on millimeter-wave radar. It constructs a millimeter-wave radar scene flow estimation model based on a supervised strategy that integrates visual and inertial sensors. The relative pose provided by visual-inertial SLAM is used as a training label, and an inertial neural network is employed to learn the mapping relationship between IMU sequences and 3D translation, thereby achieving precise guidance for millimeter-wave radar scene flow estimation. Simultaneously, the supervised term loss is calculated, and the model is trained with the goal of minimizing this loss. Static and dynamic supervised signals are fused through a unified loss function, effectively improving the global and local accuracy of scene flow estimation and compensating for the inherent limitations of radar in dynamic detail modeling.
[0046] Furthermore, this invention constructs a sliding window graph optimization framework to tightly couple the output of the inertial neural network with the pre-integrated residual of the IMU, thereby achieving joint optimization of the state. This enables the millimeter-wave radar scene flow estimation model to have adaptive weighting capability and anti-drift performance, enhancing the reliability and stability of the calculation results.
[0047] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.
[0048] The order of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0049] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A scene flow estimation method based on millimeter-wave radar, characterized in that, The method includes: Acquire continuous frame point cloud data from millimeter-wave radar and perform preprocessing; The preprocessed point cloud data is input into the millimeter-wave radar scene flow estimation model, which outputs the three-dimensional motion vector of each point to obtain the scene flow estimation result. The millimeter-wave radar scene flow estimation model is established based on a supervision strategy that integrates vision and inertial sensors, and is trained with the goal of minimizing the supervision term loss. The supervision strategy that integrates vision and inertial sensors specifically obtains the relative pose information between adjacent frames of point cloud data, and combines it with IMU data to construct an inertial neural network to obtain the mapping relationship between IMU data and three-dimensional translation. The scene flow is then predicted based on the mapping relationship.
2. The scene flow estimation method based on millimeter-wave radar according to claim 1, characterized in that, The supervised loss specifically includes static point supervised loss of point cloud data, dynamic 3D motion supervised loss of point cloud data, and self-supervised loss; among which, the self-supervised loss specifically includes optical flow information data supervised loss and chamfer distance information data supervised loss.
3. The scene flow estimation method based on millimeter-wave radar according to claim 1, characterized in that, Acquire continuous frame point cloud data from millimeter-wave radar, where each frame of point cloud data includes three-dimensional spatial points and their velocity and energy; the preprocessing specifically includes using a point cloud feature extraction network to extract local geometric features from each frame of point cloud in the frame sequence, generating point-level feature tensors, and obtaining the initial rigid body transformation matrix and the initial scene flow.
4. The scene flow estimation method based on millimeter-wave radar according to claim 2, characterized in that, Specifically, the inertial neural network acquires the relative pose information between adjacent frames of point cloud data through a vision-inertial SLAM system and constructs it in combination with IMU data, and outputs the translation vector of continuous frames of point cloud data and its uncertainty. The static point supervision term loss of point cloud data specifically optimizes the output of the inertial neural network and the pre-integrated residual of the IMU data by tightly coupling them, outputting an optimized rigid body transformation matrix, and using the optimized rigid body transformation matrix to map the point cloud data of the previous frame to the next frame, so as to calculate the static point supervision term loss of point cloud data.
5. The scene flow estimation method based on millimeter-wave radar according to claim 2, characterized in that, The loss of the dynamic 3D motion supervision item for point cloud data is specifically calculated based on the 3D trajectory of each dynamic point in the point cloud data.
6. The scene flow estimation method based on millimeter-wave radar according to claim 2, characterized in that, The loss of the optical flow information data supervision item is specifically calculated based on the set of all dynamic points in the point cloud data and the optical flow information.
7. The scene flow estimation method based on millimeter-wave radar according to claim 2, characterized in that, The supervised loss is obtained by weighting and summing the supervised loss of static points in point cloud data, the supervised loss of dynamic 3D motion in point cloud data, the supervised loss of optical flow information data, and the supervised loss of chamfer distance information data.
8. The scene flow estimation method based on millimeter-wave radar according to claim 4, characterized in that, The inertial neural network output is tightly coupled with the pre-integrated residual of the IMU data for optimization, specifically by constructing a sliding window graph optimization framework. The nodes of the sliding window graph optimization framework are sequences of state variables containing position, velocity, and attitude. The two types of factor edges in the sliding window graph optimization framework include IMU pre-integrated factor edges and neural network translation supervision edges.
9. A scene flow estimation system based on millimeter-wave radar, characterized in that, The system includes: The data processing module is used to acquire continuous frame point cloud data from millimeter-wave radar and perform preprocessing. The scene flow estimation module is used to input preprocessed point cloud data into the millimeter-wave radar scene flow estimation model, output the three-dimensional motion vector of each point, and obtain the scene flow estimation result. The millimeter-wave radar scene flow estimation model is established based on a supervision strategy that integrates vision and inertial sensors, and is trained with the goal of minimizing the supervision term loss. The supervision strategy that integrates vision and inertial sensors specifically obtains the relative pose information between adjacent frames of point cloud data, and combines it with IMU data to construct an inertial neural network, obtain the mapping relationship between IMU data and three-dimensional translation, and predict the scene flow based on the mapping relationship.
10. A computer storage medium, characterized in that, It contains a computer program that can be executed by a processor, which performs a scene flow estimation method based on millimeter-wave radar according to any one of claims 1-8.