A Simulation Method for 4D Millimeter-Wave Radar Sensors Based on Ray-tracing Technology

Abstract

The present invention relates to the field of autonomous driving simulation technology, and specifically to a simulation method for a 4D millimeter-wave radar sensor based on Ray-tracing technology. The method includes configuring the configuration parameters of the 4D millimeter-wave radar by means of the ground truth sensor model of the simulation platform, then loading the executable program of the 4D millimeter-wave radar in the simulation platform, and configuring the UDP data publishing address and the beam angle density information; the present invention can construct a 4D millimeter-wave radar model for vehicle autonomous driving, which can be compatible with various existing virtual simulation platforms on the market, and can provide users with a detailed list of target object information and radar point cloud information in JSON format via UDP.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving simulation, and specifically provides a simulation method for a 4D millimeter-wave radar sensor based on Ray-tracing technology. Background Art

[0002] In vehicle autonomous driving, the application of sensors such as cameras, millimeter-wave radars, and lidars is the core content for accurate environmental perception and reconstruction. However, each has its own shortcomings and deficiencies. The 4D millimeter-wave radar uses a large multi-input (MIMO) antenna array for echo positioning. Compared with the traditional millimeter-wave radar that detects three dimensions of distance, azimuth, and speed, it can also obtain information about the target object in the vertical direction and has the potential to replace lidars to a certain extent.

[0003] By improving algorithms and increasing the radar aperture, the 4D millimeter-wave radar can output dense point cloud data similar to lidars, with a resolution of up to within 1°, greatly enhancing the recognition accuracy. At the same time, it has better perception ability in harsh rainy and foggy weather conditions and has high potential in terms of performance, anti-interference ability, and cost reduction. The market scale has a stable growth and the technical prospect is excellent.

[0004] Currently, for the simulation technologies of cameras and lidars in various vehicle autonomous driving simulation platforms on the market, they are already very perfect. However, there are still huge gaps in the 4D millimeter-wave radar part. For example, the Carla simulation platform includes this type of sensor, but the point cloud is only calculated to the surface of the ideal cuboid frame rather than the surface of the actual 3D model, which does not match the real point cloud distribution. Summary of the Invention

[0005] The purpose of the present invention is to provide a simulation method for a 4D millimeter-wave radar sensor based on Ray-tracing technology to solve the problems raised in the above background art.

[0006] To achieve the above purpose, the present invention provides the following technical solution: A simulation method for a 4D millimeter-wave radar sensor based on Ray-tracing technology, the method comprising the following steps:

[0007] Step 1, with the help of the ground truth perception sensor model of the simulation platform, configure the configuration parameters of the 4D millimeter-wave radar. Subsequently, load the executable program of the 4D millimeter-wave radar in the simulation platform, and configure the UDP data publishing address and the harness angle density information;

[0008] Step 2, after deploying the vehicle, sensors, static terrain, and dynamic scene, run the simulation, obtain the static target object information from the static terrain and scene description file, and use the simulation platform API to obtain the vehicle information and dynamic target object information;

[0009] Step 3: Use the platform sensors to sense the targets within the detection range, all of which are part of the static and dynamic targets in Step 2. Search for the 3D model files, parse the data of the outer surface of the models, and establish a Ray-tracing scene.

[0010] Step 4: Imagine starting from the position of the 4D millimeter-wave radar and shooting a number of ray bundles into the Ray-tracing scene within the sensor field of view angle. Use the Ray-tracing technology to calculate the intersection points of the rays and the outer surface of the targets.

[0011] Step 5: According to the position of the vehicle where the sensor is installed and the intersection point information of the rays and the targets, calculate the millimeter-wave radar point cloud and other true value information of the targets. Select the intersection points with the incident angle of the rays less than a certain value with the outer surface as the radar point cloud.

[0012] Step 6: Organize the collected point cloud data at each simulation time point and the true value data of various targets. Configure the target IP address and port number for the executable file of the 4D millimeter-wave radar. Send the collected and sorted information to the specified address in JSON format through the UDP protocol.

[0013] In the above Step 1, according to the simulation platform used, deploy the true value sensing sensor model that meets the requirements on the vehicle so that its installation position, field of view angle, beam shape, detection distance, and detectable target type conform to the target 4D millimeter-wave radar.

[0014] In the above Step 2, the static target information specifically includes: target name, type, 3D model file name and storage path, position and angle in the global coordinate system; the ego-vehicle information includes: the position, orientation, speed, and acceleration of the vehicle; the dynamic target information includes: target name, ID, type, 3D model file name and storage path, position, angle, speed, and acceleration in the global coordinate system.

[0015] In the above Step 3, according to the name of the target 3D model file read, search for the corresponding 3D model file in the file system and read the vertex normal information of the model aggregate, specifically including:

[0016] S1: Use to load the 3D model file;

[0017] S2: Obtain all the triangular vertices and normal information describing the outer contour of the object in the 3D model file;

[0018] S3: Organize the data, including all the triangular vertices, normal information, and target type, ID, position, and direction that make up the outer contour of each target;

[0019] S4: Construct a Ray-tracing scene based on the organized data.

[0020] In step 4, the angular density of several beam bundles emitted into the Ray-tracing scene is selected according to user configuration, the emission quantity is calculated based on the angular density and the field of view angle size, and a random error is added to the actual emission angle of the beam bundles.

[0021] Use Ray-tracing technology to calculate the intersection situation between the beam bundles and the Ray-tracing scene, which specifically includes the following steps:

[0022] S1: Import the above Ray-tracing scene;

[0023] S2: Calculate the ray direction after adding the random error, calculate the intersection situation between each ray and the Ray-tracing scene, and only consider the first intersection;

[0024] S3: Record the azimuth angle, elevation angle, distance, incident angle, and material information of the intersection point.

[0025] In step 5, calculating the millimeter-wave radar point cloud data specifically includes: azimuth angle, elevation angle, distance, reflection intensity, and the target ID corresponding to the ray;

[0026] The calculation of other true value information of the target includes: target ID; type; speed and acceleration corresponding to the sensor coordinate system, vehicle coordinate system, and world coordinate system; visible ratio; information on the duration of entering the sensor's field of view.

[0027] In step 6, the point cloud data and various types of target true value data collected at each simulation time point specifically include: data acquisition time, target ID, target type, target position, speed, and acceleration information in the sensor coordinate system, vehicle coordinate system, and world coordinate system, as well as the coordinates, reflection intensity, distance, azimuth angle, and elevation angle of the radar point cloud in the sensor coordinate system.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] The present invention can construct a 4D millimeter-wave radar model for vehicle autonomous driving, which can be compatible with various existing virtual simulation platforms on the market, and can provide a detailed list of target information and radar point cloud information to users in JSON format via UDP;

[0030] The 4D millimeter-wave radar model simulation method of the present invention can realize the detection of the target object's external contour, form a dense radar point cloud, and make up for the deficiencies of the traditional millimeter-wave radar model in the detection dimension and data volume in the simulation field;

[0031] Compared with traditional technologies, for the calculation of point clouds, the present invention can transfer the computational workload to the GPU for operation to the greatest extent by introducing the Ray-tracing technology and combining it with a suitable computational library and hardware, giving full play to the advantages of GPU parallel computing, and enabling real-time calculation of large-scale and accurate radar point cloud data. Brief Description of the Drawings

[0032] Figure 1 It is a schematic flowchart of the simulation method of the 4D millimeter-wave radar sensor based on the Ray-tracing technology of the present invention.

[0033] Figure 2 It is a schematic diagram of the output structure of the true value data information of the target object of the present invention.

[0034] Figure 3 It is a schematic diagram of the output structure of the point cloud data information of the target object of the present invention. Detailed Embodiments

[0035] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0036] Please refer to Figures 1 to 3 , the present invention provides a technical solution:

[0037] The present invention first constructs a 4D millimeter-wave radar model that can adapt to multiple simulation platforms, uses the Ray-tracing method of GPU parallel computing to dynamically load 3D models, calculates radar point clouds in real time, and can present a list of target object information to users in JSON format through the UDP protocol.

[0038] The core of the present invention lies in using the Ray-tracing technology for the calculation of point clouds, giving full play to the advantages of GPU parallel computing, and realizing real-time calculation of large-scale and accurate radar point cloud data.

[0039] Ray-tracing, that is, ray tracing, obtains a model of the path of the ray by tracking the rays that interact with the optical surface, and is a GPU loading technology in the current technical background.

[0040] As Figure 1 shown, the present invention first uses the true value perception sensor model of the simulation platform to simulate the configuration of the 4D millimeter-wave radar: specifically including:

[0041] 1. The installation position of the sensor in the vehicle coordinate system;

[0042] 2. The installation angles of the sensor in the vehicle coordinate system, including pitch angle, heading angle, and yaw angle;

[0043] 3. The horizontal field of view and vertical field of view of the sensor;

[0044] 4. The shape of the sensor harness;

[0045] 5. The detection range of the sensor;

[0046] 6. The types of targets that the sensor can detect;

[0047] Subsequently, load the executable program of the 4D millimeter-wave radar on the simulation platform, and configure the UDP data publishing address, harness angle density information, the IP address, port, and radar harness density for receiving the target list.

[0048] After deploying the vehicle, sensor, static terrain, and dynamic scenario, run the simulation. The executable program of the 4D millimeter-wave radar first obtains the information of static targets from the static terrain and dynamic scenario description files, such as triangular cones, water horses, traffic lights, signs, etc. The information includes:

[0049] 1. The name, type, 3D model name, and path of the target;

[0050] 2. The coordinate position of the target in the world coordinate system;

[0051] During the operation, the executable program of the 4D millimeter-wave radar continuously obtains the vehicle information and the ground truth data of other targets through the simulation scenario API, specifically including:

[0052] 1. The real-time position of the vehicle in the world coordinate system;

[0053] 2. The real-time attitude of the vehicle in the world coordinate system;

[0054] 3. The real-time speed and acceleration of the vehicle;

[0055] 4. The information of detectable targets within the sensor's field of view, including the following:

[0056] 4.1. ID, name, 3D model file name, type;

[0057] 4.2. The relative position, relative speed, relative acceleration, and angle of the target in the sensor coordinate system;

[0058] 4.3. The relative position, relative speed, relative acceleration, and angle of the target in the vehicle coordinate system;

[0059] 4.4. The absolute position, absolute speed, absolute acceleration, and angle of the target in the world coordinate system.

[0060] The platform sensors are used to sense the targets within the detection range, and all targets should be within the above-mentioned target range. Search for 3D model files according to the file name and path, parse the data of the outer surface of the model, and establish a Ray-tracing scene, specifically including:

[0061] 1. The triangular vertices and normal information that make up the outer surface of each target;

[0062] 1.1. Use to load the 3D model file;

[0063] 1.2. Obtain all the triangular vertices and normal information that describe the outer contour of the object in the 3D model file;

[0064] 1.3. Organize the data, including all the triangular vertices and normal information that make up the outer contour of each target, as well as the target type, ID, position, and orientation;

[0065] 2. Model material;

[0066] 3. Combining the absolute position and angle of each target in front in the world coordinate system, the position and angle of each target in the Ray-tracing scene can be determined.

[0067] The executable program of the 4D millimeter-wave radar, based on the above-built scene, uses Ray-tracing technology to emit a number of ray bundles from the position of the 4D millimeter-wave radar within the sensor field of view angle towards the Ray-tracing scene, and calculates the intersection-related information between the rays and the models in the Ray-tracing scene.

[0068] 1. The angular density of the rays is determined by the model settings, and the number of rays is jointly determined by the angular density and the field of view angle. A tiny random error is added to each ray;

[0069] 2. Only consider the first intersection of the rays with the targets;

[0070] 3. The intersection-related information specifically includes;

[0071] 3.1. Data acquisition time;

[0072] 3.2. The position of the intersection in the sensor coordinate system, including distance, horizontal angle, and vertical angle;

[0073] 3.3. The reflection angle formed by the ray and the triangle on the model surface;

[0074] 3.4. The material of the position where the intersection is located;

[0075] 3.5. Calculate the reflection intensity based on the intersection distance, reflection angle, and material.

[0076] The executable program of the 4D millimeter-wave radar calculates the millimeter-wave radar point cloud and some other true value information of the target object based on information such as the location of the vehicle where the sensor is installed and the intersection point of the ray and the target object; the intersection points with the incident angle of the ray and the outer surface less than a certain value are selected as the radar point cloud.

[0077] 1. Among them, the point cloud information specifically includes:

[0078] 1.1. Data acquisition time;

[0079] 1.2. Target object ID and type where the intersection point is located;

[0080] 1.3. Distance, horizontal angle, and vertical angle of the point cloud in the sensor coordinate system;

[0081] 1.3. Three-dimensional coordinates of the point cloud in the sensor coordinate system;

[0082] 1.4. Point cloud reflection intensity.

[0083] 2. Among them, the other true value information of the target object includes:

[0084] 2.1. Data acquisition time;

[0085] 2.2. Target object ID;

[0086] 2.3. Target object type;

[0087] 2.4. Relative position, relative velocity, relative acceleration, and angle of the target object in the sensor coordinate system;

[0088] 2.5. Relative position, relative velocity, relative acceleration, and angle of the target object in the vehicle coordinate system;

[0089] 2.6. Absolute position, absolute velocity, absolute acceleration, and angle of the target object in the world coordinate system;

[0090] 2.7. Visible ratio;

[0091] 2.8. Information on the duration of entering the sensor's field of view.

[0092] After the position of each frame of the vehicle and the position of the dynamic target objects in the scene change, the positions and states of each target object in the Ray-tracing scene are correspondingly modified, and Ray-tracing calculation is performed again.

[0093] Organize the collected point cloud data and various types of ground truth data for each simulation time point, configure the target IP address and port number for the 4D millimeter-wave radar executable file, and send the collected and sorted information to the specified address in JSON format through the UDP protocol; the collected point cloud data and various types of ground truth data for each simulation time point specifically include: data acquisition time, target ID, target type, sensor coordinate system, vehicle coordinate system, target position, speed, and acceleration information in the world coordinate system, as well as the coordinates, reflection intensity, distance, azimuth angle, and elevation angle of the radar point cloud in the sensor coordinate system.

[0094] The user can stop the simulation at any time or stop according to the simulation end condition set by this test case.

[0095] The 4D radar sensor model of the present invention can support the completion of the above simulation method. The model includes a radar parameter configuration module, a simulation interface module, a data acquisition module, a Ray-tracing module, a data extraction and calculation module, and a simulation control module.

[0096] Radar parameter configuration module: This module is responsible for obtaining and storing various configuration parameters of the 4D millimeter-wave radar sensor to ensure that the radar settings meet the actual application requirements. The installation position and field of view angle parameters of the 4D millimeter-wave radar sensor are the same as those of the ordinary millimeter-wave radar configured for the simulation vehicle and can be obtained by reading the sensor configuration file.

[0097] Simulation interface module: This module ensures the adaptation and compatibility between the 4D millimeter-wave radar model and different simulation platforms, and provides corresponding API interfaces for different virtual simulation platforms. It realizes communication and synchronization with the simulation platform during the simulation. At the same time, it provides an error capture and handling mechanism, which can give corresponding feedback for the user to actively handle when the data transmission or interface call fails.

[0098] Data acquisition module: This module extracts vehicle and target information from the simulation environment. It determines the position and attitude of the vehicle in the environment by obtaining vehicle information, and determines the scene elements participating in Ray-tracing through the target information sensed by the ground truth sensor, providing data for subsequent calculations.

[0099] And it real-time updates the state information of the vehicle and the target during the simulation operation to ensure simulation synchronization.

[0100] Ray-tracing module: It realizes the framework for calculating Ray-tracing radar point clouds. Based on the established model scene, it uses the parallel acceleration ability of the GPU to calculate point clouds in real time, giving full play to the hardware performance.

[0101] Data extraction and calculation module: Responsible for extracting key information from the data acquisition module and the Ray-tracing module and performing calculations. It includes two parts: point cloud data and ground truth data. In addition to the directly available part, it is necessary to calculate the reflection intensity according to the intersection distance, reflection angle, and material, and calculate the actual position of the target object according to the relative position between the host vehicle and the target object, etc.

[0102] Simulation control module: This system has a simulation control function, which can start and terminate the simulation process and allows users to set the end time of the simulation. In addition, the system also supports the pause operation of the simulation, so that users can interrupt the simulation when needed. During the simulation, the system will monitor the working status of the 4D radar, record any abnormal situations, and trigger an alarm mechanism if necessary. At the same time, the system will save the key information of the simulation, including the start and end times of the simulation and the target object data during operation, for subsequent analysis and evaluation. The simulation control module is provided with a start-stop control module to realize the start, pause, and termination operations of the simulation. The simulation control module is provided with a log recording module, which can save the key information including the start time, end time, and target object data during operation in the simulation process.

[0103] The present invention is a simulation method for a physical-level 4D millimeter-wave radar sensor model designed for autonomous driving simulation and sensor simulation. Its greatest feature is that, different from the traditional millimeter-wave radar simulation model that only obtains the characteristics of the nearest point of the target object, this model can obtain a large area of radar point cloud data of the target object, and there is a significant improvement in data integrity and accuracy. At the same time, taking advantage of the parallel computing suitability of Ray-tracing, the performance of the GPU is fully utilized, and it has higher potential in generating and processing dense point clouds.

[0104] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A simulation method for a 4D millimeter-wave radar sensor based on Ray-tracing technology, characterized in that, The method includes the following steps: Step 1, with the help of the ground truth perception sensor model of the simulation platform, configure the configuration parameters of the 4D millimeter-wave radar. Subsequently, load the executable program of the 4D millimeter-wave radar in the simulation platform, and configure the UDP data publishing address and beam angle density information; Step 2, after deploying the vehicle, sensors, static terrain and dynamic scenarios, run the simulation. Obtain the static target information from the static terrain and scenario description files, and use the simulation platform API to obtain the ego-vehicle information and dynamic target information; Step 3, use the platform sensors to sense the targets within the detection range, which are all part of the static and dynamic targets in Step 2. Search for the 3D model files, parse the outer surface data of the models, and establish a Ray-tracing scene; Step 4, hypothetically starting from the position of the 4D millimeter-wave radar, emit a number of beams within the sensor's field of view angle towards the Ray-tracing scene, and use the Ray-tracing technology to calculate the intersection points of the rays and the outer surface of the targets; Step 5, according to the position of the vehicle where the sensor is installed and the intersection points of the rays and the outer surface of the targets, calculate the millimeter-wave radar point cloud and other ground truth information of the targets; select the intersection points with the incident angle of the ray and the outer surface less than a certain value as the radar point cloud; Step 6, organize the point cloud data and various types of target ground truth data collected at each simulation time point, configure the target IP address and port number for the 4D millimeter-wave radar executable file, and send the collected and sorted information to the specified address in JSON format through the UDP protocol; The static target information specifically includes: target name, type, 3D model file name and storage path, position and angle in the global coordinate system; the ego-vehicle information includes: the position, orientation, speed and acceleration of the vehicle; the dynamic target information includes: target name, ID, type, 3D model file name and storage path, position, angle, speed and acceleration in the global coordinate system; The other ground truth information of the targets includes: target ID; type; speed and acceleration corresponding to the sensor coordinate system, vehicle coordinate system and world coordinate system; visible ratio; duration information of entering the sensor's field of view; The point cloud data and various types of target ground truth data collected at each simulation time point specifically include: data acquisition time, target ID, target type, target position, speed, acceleration information in the sensor coordinate system, vehicle coordinate system, and world coordinate system, as well as the coordinates, reflection intensity, distance, azimuth angle and elevation angle of the radar point cloud in the sensor coordinate system.

2. The simulation method of a 4D millimeter-wave radar sensor based on Ray-tracing technology according to claim 1, characterized in that: In Step 1, according to the used simulation platform, deploy the ground truth perception sensor model that meets the requirements on the vehicle so that its installation position, field of view angle, beam shape, detection distance, and detectable target type conform to the target 4D millimeter-wave radar.

3. A simulation method for a 4D millimeter-wave radar sensor based on Ray-tracing technology according to claim 1, characterized in that: In Step 3, according to the read 3D model file name of the target, search for the corresponding 3D model file in the file system and read the vertex normal information of the model aggregate, specifically including: S1: Use to load the 3D model file; S2: Obtain all the triangular vertices and normal information describing the outer contour of the object in the 3D model file; S3: Organize the data, including all the triangular vertices and normal information that make up the outer contour of each target object, as well as the target object type, ID, position, and orientation; S4: Construct a Ray-tracing scene based on the organized data.

4. A simulation method of a 4D millimeter-wave radar sensor based on Ray-tracing technology according to claim 1, characterized in that: In step 4, the angular density of several bundles of rays emitted into the Ray-tracing scene is selected according to user configuration, the emission quantity is calculated based on the angular density and the field of view angle, and a random error is added to the actual emission angle of the ray bundles.

5. A simulation method for a 4D millimeter-wave radar sensor based on Ray-tracing technology according to claim 4, characterized in that: Use Ray-tracing technology to calculate the intersection of the ray bundles with the Ray-tracing scene, specifically including the following steps: S1: Import the above Ray-tracing scene; S2: Calculate the ray direction after adding the random error, calculate the intersection of each ray with the Ray-tracing scene, and only consider the first intersection; S3: Record the azimuth angle, elevation angle, distance, incident angle, and material information of the intersection point.

6. The simulation method of a 4D millimeter-wave radar sensor based on Ray-tracing technology according to claim 1, characterized in that: In step 5, calculating the millimeter-wave radar point cloud data specifically includes: azimuth angle, elevation angle, distance, reflection intensity, and the target object ID corresponding to the ray.

Images