Spatial perception and target detection method and system based on 4D millimeter wave radar
By introducing depth estimation, posture estimation and regression network models into the 4D millimeter wave radar system, combining the data of 4D millimeter wave radar and camera, the problem of unpredictable image information and supervising 4D millimeter wave radar data in the prior art is solved. High-precision object detection under extreme conditions is achieved.
Patent Information
- Application Number
- CN202411134708.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art cannot predict future image information based on detected image information, and cannot supervise the image information detected by 4D millimeter wave radar, resulting in insufficient detection accuracy in extreme weather and low light conditions.
A spatial perception and target detection system based on 4D millimeter wave radar is designed, including depth estimation model, position estimation model and regression network model. Through the fusion of 4D millimeter wave radar, camera and lidar, the depth estimation network and bit are used. The pose estimation network implements self-supervised learning, can predict image information and generate depth maps and pose transformation matrix, and further output common target information through the regression network.
On the basis of low cost and strong anti-interference ability, the detection accuracy under conditions such as rain and snow, dark light, and backlight are improved. The image can be predicted based on image information and the images detected by 4D millimeter wave radar are detected. Effective supervision of information.
Smart Images

Figure CN119068433A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a space perception and target detection method and system based on 4D millimeter-wave radar. Background Art
[0002] In the field of autonomous driving, perception fusion solutions can be divided into multi-camera fusion and lidar and camera fusion according to sensor type; the multi-camera fusion solution has the advantage of low cost. With the development of technology, the application of 4D millimeter-wave radar is becoming more and more extensive. Compared with traditional 3D millimeter-wave radar, 4D millimeter-wave radar has increased the perception capability in the height dimension, which greatly increases the number of points detected by the radar and has better imaging effects, making it possible to replace lidar in terms of technical solutions; on the other hand, due to the strong anti-interference ability of 4D millimeter-wave radar, it has good anti-interference performance against extreme weather.
[0003] For example, Chinese patent CN116797900A discloses a method for data fusion and target detection of a camera and a 4D millimeter-wave radar, which includes the following steps: the camera collects images in front and senses the environment in front, where the sensed targets include people and vehicles; the 4D millimeter-wave radar detects the environment in front, and outputs data in four dimensions: distance, speed, horizontal angle, and height of the target; the camera and the 4D millimeter-wave radar are jointly calibrated, and the image data and the point cloud data collected by the 4D millimeter-wave radar are synchronized in time and space through a time and space synchronization algorithm; after the time and space synchronization, the target in the image is detected through a neural network, and the target information is determined through a fusion decision algorithm.
[0004] However, the technology has the following problems: it is impossible to predict future image information based on the detected image information, and it is impossible to supervise the image information detected by the 4D millimeter wave radar.
[0005] Based on this, the present invention designs a spatial perception and target detection method and system based on 4D millimeter-wave radar to solve the above problems. Summary of the invention
[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides a space perception and target detection method and system based on 4D millimeter wave radar.
[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0008] A spatial perception and target detection system based on 4D millimeter wave radar, including a depth estimation model, a pose estimation model and a regression network model, wherein the depth estimation model is the main part of the network;
[0009] The depth estimation model includes an inference pathway unit and a training pathway unit;
[0010] The reasoning pathway unit includes a 4D millimeter wave radar, an analytical projection processing unit, a camera, and a depth estimation network, and the core module is the depth estimation network;
[0011] 4D millimeter wave radar, used to collect data in four dimensions: distance, speed, horizontal angle, and height of the surrounding environment and transmit the data information to the analytical projection processing unit;
[0012] The parsing and projection processing unit is responsible for parsing the data information collected by the 4D millimeter-wave radar to obtain the 4D millimeter-wave radar point cloud coordinates and Doppler velocity values and projecting them into the camera's image space, and then uploading the parsed data information to the depth estimation network;
[0013] A camera, used to collect image data and transmit the data to the depth estimation network;
[0014] The depth estimation network is used to receive and process the radar data information in the analytical projection processing unit and the image data information collected by the camera to finally obtain the depth value;
[0015] The training pathway unit also includes a laser radar;
[0016] LiDAR, which is used to collect point cloud data information and calculate the loss function as a supervisory signal and then upload the data to the depth estimation network;
[0017] The pose estimation model is an optional part of the network and is only used in the training phase. The pose estimation model is a supplement to the depth estimation model and is used to achieve self-supervision. The core module of the pose estimation model is the pose estimation network.
[0018] The pose estimation model inputs the image into the pose estimation network according to the image data information collected by the camera to obtain the pose transformation matrix; thereby obtaining a depth map, and predicting the current image according to the depth map;
[0019] The regression network model is the task branch part of the network, which is used for outputting space occupation grids and general targets. The core module of the regression network model is the regression network.
[0020] The regression network model converts the depth map obtained by the depth estimation network into a point cloud, and general target information can be regressed from the point cloud through the regression network.
[0021] Furthermore, the 4D millimeter-wave radar obtains data in four dimensions: distance, speed, horizontal angle, and height. The parsing and projection processing unit is responsible for parsing the 4D millimeter-wave radar raw data, obtaining the spatial coordinates and Doppler velocity values of each point in the radar coordinate system, and projecting them into the image space of the camera. The value of each pixel is the depth value and the Doppler velocity value. The projection process requires the intrinsic parameters of the camera and the extrinsic parameters of the 4D millimeter-wave radar to the camera, and then the image is obtained from the camera, and after preprocessing, it is merged with the radar projection image obtained in step one and input into the depth estimation network.
[0022] Furthermore, in the reasoning pathway unit, the output depth map of the network can be obtained directly without the access of the lidar.
[0023] Furthermore, in the training path unit, the point cloud of the lidar needs to be transferred to the image coordinate system through the camera intrinsic parameters and the external parameters of the lidar to the camera. The pixel value of the image is the depth value, and this is used as the supervision signal of the depth estimation network to continuously update the network parameters.
[0024] Furthermore, the pose estimation model outputs the collected image data information as a previous frame image, a current frame image and a next frame image through a camera. The previous frame image and the current frame image or the next frame image and the current frame image are grouped together and input into a pose estimation network to obtain a pose transformation matrix. The transformation matrix and the depth map of the current frame image obtained through a depth network are used to predict the previous frame image or the next frame image to the current frame image in turn, which is called a predicted frame image. Then, the photometric loss function between the predicted frame image and the current frame image is calculated, and the pose estimation network and the depth estimation network parameters are updated by back propagation.
[0025] Furthermore, the regression network model converts the depth map obtained by the depth estimation network into a point cloud based on the internal and external parameters of the camera, and the point cloud is in the vehicle body coordinate system. The point cloud information is then used to generate a spatial occupancy grid, and the general target information can be regressed from the point cloud through the regression network.
[0026] Furthermore, the present invention also proposes a spatial perception and target detection method based on 4D millimeter wave radar, and the specific implementation steps of the reasoning path unit are:
[0027] The specific implementation steps of the reasoning path unit are:
[0028] Step 1: Determine whether it is an online test. If it is an online test, start the system and initialize each sensor. If not, pass in the offline test configuration.
[0029] Step 2: Complete the initialization of the entire system based on the different situations in step 1;
[0030] Step 3: Get a frame of image data from the camera or offline data set and complete the image preprocessing;
[0031] Step 4: Obtain a frame of 4D millimeter-wave radar data from the 4D millimeter-wave radar sensor or offline data set;
[0032] Step 5: The parsing projection processing unit is responsible for parsing the 4D millimeter wave radar data, obtaining the 4D millimeter wave radar point cloud coordinates and Doppler velocity values, and then projecting the point cloud Doppler velocity into the camera's image space based on the camera's internal parameters and the extrinsic parameters of the 4D millimeter wave radar to the camera. Each pixel value is a depth value and a Doppler velocity value.
[0033] Step 6: Combine the image data obtained in step 3 with the radar data obtained in step 5 and input them into the depth estimation network;
[0034] Step 7: Complete the forward reasoning of the depth estimation network and obtain the depth value.
[0035] Furthermore, the specific implementation steps of the training path unit are:
[0036] Step 1: Complete initialization and load the data set;
[0037] Step 2: Get a frame of image data from the training data set and complete the image preprocessing;
[0038] Step 3: Get a frame of 4D millimeter-wave radar data from the training data set. The parsing and projection processing unit is responsible for parsing the 4D millimeter-wave radar data, obtaining the 4D millimeter-wave radar point cloud coordinates and Doppler velocity values, and then projecting the point cloud Doppler velocity into the camera's image space based on the camera's internal parameters and the extrinsic parameters of the 4D millimeter-wave radar to the camera. Each pixel value is a depth value and a Doppler velocity value.
[0039] Step 4: Get a frame of lidar data from the training data set, project the lidar point cloud into the camera's image space based on the camera's intrinsic parameters and the lidar's extrinsic parameters to the camera, and each pixel value is a depth value;
[0040] Step 5: Combine the data obtained in step 2 and step 3, input them into the depth estimation network, and obtain the depth value output by the network through the forward path;
[0041] Step 6: Use the depth value in step 4 as the supervision signal of the depth value in step 5, calculate the loss function and update the network parameters through back propagation.
[0042] Furthermore, the specific implementation steps of the pose estimation model are:
[0043] Step 1: Complete the initialization task and load the data set;
[0044] Step 2: Obtain data information of the previous frame image, the current frame image and the next frame image from the training data set;
[0045] Step 3: The previous frame image and the current frame image are grouped together, and the current frame image and the next frame image are grouped together. The three frames can get two sets of data, which are input into the pose estimation network in sequence;
[0046] Step 4: Execute the forward path of the pose estimation network to obtain the pose transformation matrix;
[0047] Step 5: Use the transformation matrix and the depth map obtained by the current frame image through the deep network to predict the previous frame image or the next frame image to the current frame image in sequence, which is called the predicted frame image;
[0048] Step 6: Calculate the photometric loss function between the predicted frame image and the current frame image, and back-propagate to update the parameters of the pose estimation network and the depth estimation network.
[0049] Furthermore, the specific implementation steps of the regression network model are:
[0050] Step 1: Obtain the network output depth map through the depth estimation network;
[0051] Step 2: Convert the depth map obtained by the depth estimation model into a point cloud through the camera internal and external parameters, and the point cloud is in the vehicle body coordinate system;
[0052] Step 3: Generate a space occupancy grid based on the point cloud information within a certain range of the vehicle body, which can be used for downstream regulatory tasks;
[0053] Step 4: The general target information can be regressed from the point cloud through the regression network module.
[0054] Compared with the prior art, the present invention has the following beneficial effects:
[0055] The present invention proposes a spatial perception and general target detection method based on the fusion of 4D millimeter-wave radar and camera. The image information detected by the 4D millimeter-wave radar can be supervised according to the future image information predicted by the detected image information. The method has the advantages of low cost, strong anti-interference ability in rainy and snowy weather, and low accuracy loss under conditions such as night, dim light, and backlight. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0057] Figure 1 This is a principle block diagram of the space perception and target detection system based on 4D millimeter wave radar of the present invention;
[0058] Figure 2 is a flow chart of the reasoning pathway unit of the present invention;
[0059] Figure 3 is a flow chart of a training pathway unit of the present invention;
[0060] Figure 4 It is a flow chart of the posture estimation network model of the present invention;
[0061] Figure 5 Flow chart of the regression network model of the present invention.
[0062] The numbers in the figure represent:
[0063] 1. Depth estimation network model 2. Pose estimation network model 3. Regression network model. DETAILED DESCRIPTION
[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0065] Embodiment 1
[0066] In some embodiments, please refer to the attached instructions. Figure 1-5 , a spatial perception and target detection system based on 4D millimeter wave radar, including a depth estimation model 1, a pose estimation model 2 and a regression network model 3, wherein the depth estimation model 1 is the main part of the network;
[0067] The depth estimation model 1 includes a reasoning path unit and a training path unit;
[0068] The reasoning pathway unit includes a 4D millimeter wave radar, an analytical projection processing unit, a camera, and a depth estimation network, and the core module is the depth estimation network;
[0069] 4D millimeter wave radar, used to collect data in four dimensions: distance, speed, horizontal angle, and height of the surrounding environment and transmit the data information to the analytical projection processing unit;
[0070] The parsing and projection processing unit is responsible for parsing the data information collected by the 4D millimeter-wave radar to obtain the 4D millimeter-wave radar point cloud coordinates and Doppler velocity values and projecting them into the camera's image space, and then uploading the parsed data information to the depth estimation network;
[0071] A camera, used to collect image data and transmit the data to the depth estimation network;
[0072] The depth estimation network is used to receive and process the radar data information in the analytical projection processing unit and the image data information collected by the camera to finally obtain the depth value;
[0073] The training pathway unit also includes a laser radar;
[0074] LiDAR, which is used to collect point cloud data information and calculate the loss function as a supervisory signal and then upload the data to the depth estimation network;
[0075] The pose estimation model 2 is an optional part of the network and is only used in the training phase. The pose estimation model 2 is a supplement to the depth estimation model 1 and is used to achieve self-supervision. The core module of the pose estimation model 2 is the pose estimation network.
[0076] The pose estimation model 2 inputs the image into the pose estimation network according to the image data information collected by the camera to obtain the pose transformation matrix; thereby obtaining a depth map, and predicting the current image according to the depth map;
[0077] The regression network model 3 is the task branch part of the network, which is used for the output of space occupation grid and general target. The core module of the regression network model 3 is the regression network;
[0078] The regression network model 3 converts the depth map obtained by the depth estimation network into a point cloud, and general target information can be regressed from the point cloud through the regression network.
[0079] Embodiment 2
[0080] In some embodiments, Figure 1-5 As shown, as a preferred embodiment of the present invention, the 4D millimeter wave radar obtains data in four dimensions: distance, speed, horizontal angle, and height. The parsing projection processing unit is responsible for parsing the 4D millimeter wave radar raw data, obtaining the spatial coordinates and Doppler velocity values of each point in the radar coordinate system, and projecting them into the image space of the camera. The value of each pixel is the depth value and the Doppler velocity value. In the projection process, the intrinsic parameters of the camera and the extrinsic parameters of the 4D millimeter wave radar to the camera are needed, and then the image is obtained from the camera, and after preprocessing, it is merged with the radar projection picture obtained in step one and input into the depth estimation network.
[0081] In the reasoning pathway unit, the output depth map of the network can be obtained directly without the access of the lidar.
[0082] In the training path unit, the point cloud data information of the lidar needs to be transferred to the image coordinate system through the camera intrinsic parameters and the lidar to the camera extrinsic parameters. The pixel value of the image is the depth value, and this is used as the supervision signal of the depth estimation network to continuously update the network parameters.
[0083] The pose estimation model 2 outputs the collected image data information as the previous frame image, the current frame image and the next frame image through the camera. The previous frame image and the current frame image or the next frame image and the current frame image are grouped together and input into the pose estimation network to obtain the pose transformation matrix. The transformation matrix and the depth map of the current frame image obtained through the depth network are used to predict the previous frame image or the next frame image to the current frame image in turn, which is called the predicted frame image. Then the photometric loss function between the predicted frame image and the current frame image is calculated, and the pose estimation network and the depth estimation network parameters are updated by back propagation.
[0084] The regression network model 3 converts the depth map obtained by the depth estimation network into a point cloud according to the internal and external parameters of the camera, and the point cloud is in the vehicle body coordinate system. The point cloud information is then used to generate a spatial occupancy grid. The general target information can be regressed from the point cloud through the regression network. Since this model only uses geometric information, a general target is obtained, which is more universal.
[0085] Embodiment 3
[0086] In some embodiments, Figure 1-5 As shown, as a preferred embodiment of the present invention, the present invention also proposes a space perception and target detection method based on 4D millimeter wave radar, and the specific implementation steps of the reasoning path unit are:
[0087] Step 1: Determine whether it is an online test. If it is an online test, start the system and initialize each sensor. If not, pass in the offline test configuration.
[0088] Step 2: Complete the initialization of the entire system based on the different situations in step 1;
[0089] Step 3: Get a frame of image data from the camera or offline data set and complete the image preprocessing;
[0090] Step 4: Obtain a frame of 4D millimeter-wave radar data from the 4D millimeter-wave radar sensor or offline data set;
[0091] Step 5: The parsing projection processing unit is responsible for parsing the 4D millimeter wave radar data, obtaining the 4D millimeter wave radar point cloud coordinates and Doppler velocity values, and then projecting the point cloud Doppler velocity into the camera's image space based on the camera's internal parameters and the extrinsic parameters of the 4D millimeter wave radar to the camera. Each pixel value is a depth value and a Doppler velocity value.
[0092] Step 6: Combine the image data obtained in step 3 with the radar data obtained in step 5 and input them into the depth estimation network;
[0093] Step 7: Complete the forward reasoning of the depth estimation network and obtain the depth value.
[0094] The specific implementation steps of the training path unit of the depth estimation model 1 are:
[0095] Step 1: Complete initialization and load the data set;
[0096] Step 2: Get a frame of image data from the training data set and complete the image preprocessing;
[0097] Step 3: Get a frame of 4D millimeter-wave radar data from the training data set. The parsing and projection processing unit is responsible for parsing the 4D millimeter-wave radar data, obtaining the 4D millimeter-wave radar point cloud coordinates and Doppler velocity values, and then projecting the point cloud Doppler velocity into the camera's image space based on the camera's internal parameters and the extrinsic parameters of the 4D millimeter-wave radar to the camera. Each pixel value is a depth value and a Doppler velocity value.
[0098] Step 4: Get a frame of lidar data from the training data set, project the lidar point cloud into the camera's image space based on the camera's intrinsic parameters and the lidar's extrinsic parameters to the camera, and each pixel value is a depth value;
[0099] Step 5: Combine the data obtained in step 2 and step 3, input them into the depth estimation network, and obtain the depth value output by the network through the forward path;
[0100] Step 6: Use the depth value in step 4 as the supervision signal of the depth value in step 5, calculate the loss function and update the network parameters through back propagation.
[0101] The specific implementation steps of the posture estimation model 2 are:
[0102] Step 1: Complete the initialization task and load the data set;
[0103] Step 2: Obtain data information of the previous frame image, the current frame image and the next frame image from the training data set;
[0104] Step 3: The previous frame image and the current frame image are grouped together, and the current frame image and the next frame image are grouped together. The three frames can get two sets of data, which are input into the pose estimation network in sequence;
[0105] Step 4: Execute the forward path of the pose estimation network to obtain the pose transformation matrix;
[0106] Step 5: Use the transformation matrix and the depth map obtained by the current frame image through the deep network to predict the previous frame image or the next frame image to the current frame image in sequence, which is called the predicted frame image;
[0107] Step 6: Calculate the photometric loss function between the predicted frame image and the current frame image, and back-propagate to update the parameters of the pose estimation network and the depth estimation network.
[0108] The specific implementation steps of the regression network model 3 are:
[0109] Step 1: Obtain the network output depth map through the depth estimation network;
[0110] Step 2: Convert the depth map obtained by the depth estimation model 1 into a point cloud through the camera internal and external parameters, and the point cloud is in the vehicle body coordinate system;
[0111] Step 3: Generate a space occupancy grid based on the point cloud information within a certain range of the vehicle body, which can be used for downstream regulatory tasks;
[0112] Step 4: The general target information can be regressed from the point cloud through the regression network module.
[0113] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A spatial perception and target detection system based on 4D millimeter wave radar, comprising a depth estimation model (1), a pose estimation model (2) and a regression network model (3), characterized in that: The depth estimation model (1) is the main part of the network; The depth estimation model (1) comprises an inference path unit and a training path unit; The reasoning pathway unit includes a 4D millimeter wave radar, an analytical projection processing unit, a camera, and a depth estimation network, and the core module is the depth estimation network; 4D millimeter wave radar, used to collect data in four dimensions: distance, speed, horizontal angle, and height of the surrounding environment and transmit the data information to the analytical projection processing unit; The parsing and projection processing unit is responsible for parsing the data information collected by the 4D millimeter-wave radar to obtain the 4D millimeter-wave radar point cloud coordinates and Doppler velocity values and projecting them into the camera's image space, and then uploading the parsed data information to the depth estimation network; A camera, used to collect image data and transmit the data to the depth estimation network; The depth estimation network is used to receive and process the radar data information in the analytical projection processing unit and the image data information collected by the camera to finally obtain the depth value; The training pathway unit also includes a laser radar; LiDAR, which is used to collect point cloud data information and calculate the loss function as a supervisory signal and then upload the data to the depth estimation network; The pose estimation model (2) is an optional part of the network and is only used in the training phase. The pose estimation model (2) is a supplement to the depth estimation model (1) and is used to achieve self-supervision. The core module of the pose estimation model (2) is the pose estimation network. The pose estimation model (2) inputs the image into the pose estimation network according to the image data information collected by the camera to obtain a pose transformation matrix; thereby obtaining a depth map, and predicting the current image according to the depth map; The regression network model (3) is the task branch part of the network, which is used to output the space occupation grid and the general target. The core module of the regression network model (3) is the regression network; The regression network model (3) converts the depth map obtained by the depth estimation network into a point cloud, and general target information can be regressed from the point cloud through the regression network.
2. The spatial perception and target detection system based on 4D millimeter wave radar according to claim 1 is characterized in that: The 4D millimeter-wave radar obtains data in four dimensions: distance, speed, horizontal angle, and height. The parsing and projection processing unit is responsible for parsing the 4D millimeter-wave radar raw data, obtaining the spatial coordinates and Doppler velocity values of each point in the radar coordinate system, and projecting them into the image space of the camera. The value of each pixel is the depth value and the Doppler velocity value. The projection process requires the use of the camera's intrinsic parameters and the extrinsic parameters of the 4D millimeter-wave radar to the camera, and then the image is obtained from the camera, and after preprocessing, it is merged with the radar projection image obtained in step one and input into the depth estimation network.
3. The spatial perception and target detection system based on 4D millimeter wave radar according to claim 2 is characterized in that: In the reasoning pathway unit, the output depth map of the network can be obtained directly without the access of the lidar.
4. The spatial perception and target detection system based on 4D millimeter wave radar according to claim 3 is characterized in that: In the training path unit, the point cloud of the lidar needs to be transferred to the image coordinate system through the camera intrinsic parameters and the external parameters of the lidar to the camera. The pixel value of the image is the depth value, and this is used as the supervision signal of the depth estimation network to continuously update the network parameters.
5. The spatial perception and target detection system based on 4D millimeter wave radar according to claim 4 is characterized in that: The pose estimation model (2) outputs the collected image data information as a previous frame image, a current frame image and a next frame image through a camera. The previous frame image and the current frame image or the next frame image and the current frame image are grouped together and input into a pose estimation network to obtain a pose transformation matrix. The transformation matrix and the depth map of the current frame image obtained through a depth network are used to predict the previous frame image or the next frame image to the current frame image, which is called a predicted frame image. Then, the photometric loss function between the predicted frame image and the current frame image is calculated, and the pose estimation network and the depth estimation network parameters are updated by back propagation.
6. The spatial perception and target detection system based on 4D millimeter wave radar according to claim 5 is characterized in that: The regression network model (3) converts the depth map obtained by the depth estimation network into a point cloud based on the internal and external parameters of the camera, and the point cloud is in the vehicle body coordinate system. The point cloud information is then used to generate a spatial occupancy grid. The general target information can be regressed from the point cloud through the regression network.
7. A spatial perception and target detection method based on 4D millimeter wave radar according to claim 6, characterized in that: The specific implementation steps of the reasoning path unit are: Step 1: Determine whether it is an online test. If it is an online test, start the system and initialize each sensor. If not, pass in the offline test configuration. Step 2: Complete the initialization of the entire system based on the different situations in step 1; Step 3: Get a frame of image data from the camera or offline data set and complete the image preprocessing; Step 4: Obtain a frame of 4D millimeter-wave radar data from the 4D millimeter-wave radar sensor or offline data set; Step 5: The parsing projection processing unit is responsible for parsing the 4D millimeter wave radar data, obtaining the 4D millimeter wave radar point cloud coordinates and Doppler velocity values, and then projecting the point cloud Doppler velocity into the camera's image space based on the camera's internal parameters and the extrinsic parameters of the 4D millimeter wave radar to the camera. Each pixel value is a depth value and a Doppler velocity value. Step 6: Combine the image data obtained in step 3 with the radar data obtained in step 5 and input them into the depth estimation network; Step 7: Complete the forward reasoning of the depth estimation network and obtain the depth value.
8. The method for spatial perception and target detection based on 4D millimeter wave radar according to claim 7, characterized in that: The specific implementation steps of the training path unit are: Step 1: Complete initialization and load the data set; Step 2: Get a frame of image data from the training data set and complete the image preprocessing; Step 3: Get a frame of 4D millimeter-wave radar data from the training data set. The parsing and projection processing unit is responsible for parsing the 4D millimeter-wave radar data, obtaining the 4D millimeter-wave radar point cloud coordinates and Doppler velocity values, and then projecting the point cloud Doppler velocity into the camera's image space based on the camera's internal parameters and the extrinsic parameters of the 4D millimeter-wave radar to the camera. Each pixel value is a depth value and a Doppler velocity value. Step 4: Get a frame of lidar data from the training data set, project the lidar point cloud into the camera's image space based on the camera's intrinsic parameters and the lidar's extrinsic parameters to the camera, and each pixel value is a depth value; Step 5: Combine the data obtained in step 2 and step 3, input them into the depth estimation network, and obtain the depth value output by the network through the forward path; Step 6: Use the depth value in step 4 as the supervision signal of the depth value in step 5, calculate the loss function and update the network parameters through back propagation.
9. The spatial perception and target detection method based on 4D millimeter wave radar according to claim 8 is characterized in that: The specific implementation steps of the pose estimation model (2) are: Step 1: Complete the initialization task and load the data set; Step 2: Obtain data information of the previous frame image, the current frame image and the next frame image from the training data set; Step 3: The previous frame image and the current frame image are grouped together, and the current frame image and the next frame image are grouped together. The three frames can get two sets of data, which are input into the pose estimation network in sequence; Step 4: Execute the forward path of the pose estimation network to obtain the pose transformation matrix; Step 5: Use the transformation matrix and the depth map obtained by the current frame image through the deep network to predict the previous frame image or the next frame image to the current frame image in sequence, which is called the predicted frame image; Step 6: Calculate the photometric loss function between the predicted frame image and the current frame image, and back-propagate to update the pose estimation network and depth estimation network parameters.
10. The method for spatial perception and target detection based on 4D millimeter wave radar according to claim 9, characterized in that: The specific implementation steps of the regression network model (3) are: Step 1: Obtain the network output depth map through the depth estimation network; Step 2: Convert the depth map obtained by the depth estimation model (1) into a point cloud using the camera internal and external parameters, and the point cloud is in the vehicle body coordinate system; Step 3: Generate a space occupancy grid based on the point cloud information within a certain range of the vehicle body, which can be used for downstream regulatory tasks; Step 4: The general target information can be regressed from the point cloud through the regression network module.
Citation Information
Patent Citations
Camera and 4D millimeter wave radar data fusion and target detection method
CN116797900A
