Robot environment perception system based on multi-radar fusion

Abstract

The application discloses a kind of robot environment perception systems based on multi-radar fusion, it is related to robot environment perception field.The system ingeniously fuses the radar sensor data of multiple different types, different deployment locations, realizes the high-precision, high reliability, low power consumption perception to the three-dimensional environment around robot by advanced feature extraction, correlation matching, information fusion and dynamic environment modeling etc.Technologies.System can effectively identify and track static obstacles, dynamic targets, and accurately estimate its position, velocity, attitude and size, provide reliable environmental information for the autonomous navigation, obstacle avoidance, path planning, target interaction etc.of robot.The application greatly improves the environmental perception ability of robot in various complex scenes by fusion strategy, dynamic adaptive algorithm and effective use of redundant sensor information.

Description

Technical Field

[0001] This invention relates to the field of robot environmental perception, and specifically to a robot environmental perception system based on multi-radar fusion. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, robots need to operate autonomously in various complex, dynamic, and unknown environments. Environmental perception is the foundation for robots to achieve core functions such as autonomous navigation, obstacle avoidance, path planning, and target interaction. Currently, robot environmental perception mainly relies on single sensors or simple combinations of multiple sensors, such as LiDAR, cameras, millimeter-wave radar, and ultrasonic sensors.

[0003] Existing sensor fusion technologies have the following drawbacks:

[0004] Simple point cloud stitching: Simply superimposing point cloud data from different sensors can easily lead to point cloud overlap, information redundancy or conflict, and cannot effectively handle the accuracy differences between different sensors.

[0005] Feature-based fusion: Extracting low-level or high-level features from different sensors and fusing them, but the robustness and universality of feature extraction are key challenges.

[0006] End-to-end fusion: Using deep learning to directly fuse raw sensor data has great potential, but it requires a large amount of labeled data, and the generalization ability of the model needs further verification. Summary of the Invention

[0007] Purpose of the invention: To provide a robot environmental perception system based on multi-radar fusion to solve the above-mentioned problems existing in the prior art.

[0008] Technical solution: A robot environmental perception system based on multi-radar fusion, comprising:

[0009] Multiple radar sensor units: Includes at least two different types of radar sensors, such as millimeter-wave radar (MMW), lidar (LiDAR), ultrasonic radar (US), etc., and deploys them on the robot body in different spatial layouts according to the robot's task requirements and application scenarios.

[0010] Raw data preprocessing module: Performs noise reduction, filtering, and distortion correction on the raw point cloud data or echo signals collected by each radar sensor unit;

[0011] Local feature extraction module: Extracts local three-dimensional geometric features, reflection intensity features, echo time-domain features, and frequency-domain features from the preprocessed data of each radar sensor unit;

[0012] Multi-sensor data alignment and calibration module: Establishes spatial geometric relationships (extrinsic parameters) and time synchronization mechanisms between different radar sensors to ensure spatial and temporal consistency of data from different sensors;

[0013] Feature-level or point cloud-level fusion module: used to integrate information from different radar sensors;

[0014] Dynamic Environment Modeling and Tracking Module: Based on the fused environmental information, it accurately models and continuously tracks dynamic targets in the environment;

[0015] Output interface: Provides the fused high-precision and robust environmental perception results (such as obstacle list, environment map, target attributes, etc.) to the robot's upper-level control system (such as navigation, decision making, etc.).

[0016] In a further embodiment, the feature-level or point cloud-level fusion module includes:

[0017] Target association-based fusion: Matching potential targets detected by different radar sensors using geometric, kinematic, or semantic association algorithms, and then fusing the matched target information;

[0018] Voxel / grid-based fusion: Projecting data from different radar sensors onto a unified 3D voxel space or grid map, and generating a unified environmental occupancy grid map or dense point cloud through a voting mechanism or probabilistic fusion.

[0019] End-to-end fusion based on deep learning: Using deep neural networks, raw data or extracted features from different radar sensors are used as input to directly output environmental perception results;

[0020] The dynamic environment modeling and tracking module includes:

[0021] State estimator: Based on the fused environmental information, Kalman filtering and particle filtering algorithms are used to estimate and predict the state (position, velocity, acceleration, attitude, etc.) of the detected dynamic target.

[0022] Multi-target tracker: Manages and maintains the IDs of multiple targets, distinguishes different targets, and continuously tracks the target's motion trajectory;

[0023] Static environment representation: Create a static environment map around the robot, including obstacles and terrain information, and support map updates and maintenance.

[0024] In a further embodiment, the plurality of radar sensor units include at least one millimeter-wave radar for providing long-range, all-weather detection capabilities; and at least one lidar for providing high-precision three-dimensional geometric information; the plurality of radar sensor units include various types of radars with different fields of view, for example, a long-range millimeter-wave radar for long-range detection, multiple short-range millimeter-wave radars for short-range blind zone coverage, and a lidar for high-precision modeling.

[0025] In a further embodiment, the features extracted by the local feature extraction module include: point cloud density, normal vector, curvature, point cloud shape descriptor, echo amplitude, Doppler velocity, and phase information.

[0026] In a further embodiment, the multi-sensor data alignment and calibration module employs an iterative nearest point (ICP) algorithm, a point cloud registration algorithm, or an automatic calibration method based on prior knowledge.

[0027] In a further embodiment, the feature-level fusion is performed by calculating the similarity or consistency of features between different sensors and combining prior information for weighted fusion.

[0028] In a further embodiment, the point cloud-level fusion adopts a probability-based fusion method, which accumulates or averages the occupancy probability information output by each radar sensor in a shared space.

[0029] In a further embodiment, the deep learning-based end-to-end fusion employs a multimodal fusion network, which includes an input layer and a feature fusion layer specifically designed for different radar sensor data.

[0030] In a further embodiment, the dynamic environment modeling and tracking module has the ability to predict target behavior, for example, by learning historical movement trajectories to predict possible future movement directions and intentions.

[0031] In a further embodiment, the dynamic environment modeling and tracking module can adaptively adjust the parameters of the state estimator to cope with the motion characteristics of different targets and sensor noise levels.

[0032] In a further embodiment, the plurality of radar sensor units include at least one coaxial phase-geometry composite radar, which fuses at least two different frequency bands of electromagnetic waves at the physical waveform level through a single transmitting and receiving device.

[0033] The coaxial phase-geometry composite radar described above has a raw data preprocessing module that can decouple and generate an enhanced point cloud from the composite echo signal. In addition to the three-dimensional geometric coordinates, each point in the point cloud is natively bound to attribute information from radar waves of different frequency bands, including: Doppler velocity and material information characterized by radar cross-section.

[0034] In a further embodiment, the dynamic environment modeling and tracking module directly utilizes the velocity attribute inherent in each point in the enhanced point cloud, instantly distinguishing between static and dynamic targets without the need for clustering and trajectory tracking; and uses the material information to assist in the preliminary identification of target categories.

[0035] In a further embodiment, the dynamic adaptive fusion adjusts the weights based on quantifiable sensor data quality metrics, including:

[0036] For lidar: point cloud density, signal-to-noise ratio;

[0037] For millimeter-wave radar: echo signal strength, spectral purity;

[0038] For coaxial phase-geometry composite radar: signal-to-noise ratio of decoupled signals in each frequency band;

[0039] The fusion module dynamically adjusts the contribution weight of each sensor's data in the fusion process based on preset threshold rules or machine learning models.

[0040] Beneficial Effects: This invention relates to a robot environmental perception system based on multi-radar fusion, which has the following beneficial effects:

[0041] 1. Significantly improved perception accuracy: By complementing and fusing information from multiple radars, the insufficient accuracy of a single sensor is effectively compensated for, and the estimation accuracy of attributes such as the position, size, and speed of objects is improved.

[0042] 2. Significantly enhanced robustness: It solves the perception bottleneck of a single sensor in harsh weather, complex lighting, occlusion and other conditions, enabling the robot to maintain reliable perception capabilities in various complex environments.

[0043] 3. Wider environmental coverage: The combination of different types of radar can achieve all-round environmental perception from close range to long range, from details to the whole.

[0044] 4. Enhanced target recognition capability: By integrating information from multiple sensors in different physical domains, richer features can be provided, improving the accuracy and reliability of target recognition, and even enabling preliminary classification of target types.

[0045] 5. Optimization of power consumption and cost: By rationally selecting the type and number of sensors and adopting efficient fusion algorithms, the overall power consumption and cost of the system can be optimized while meeting performance requirements.

[0046] 6. Provide a more reliable foundation for upper-level applications: High-quality environmental perception results directly empower advanced functions of robots such as autonomous navigation, path planning, and decision control, significantly improving the robot's autonomy and intelligence level. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the composition framework of the robot environmental perception system based on multi-radar fusion as described in this invention. Detailed Implementation

[0048] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0049] The robot environmental perception system based on multi-radar fusion involved in this invention mainly includes:

[0050] Multiple radar sensor units: Includes at least two different types of radar sensors, such as millimeter-wave radar (MMW), lidar (LiDAR), ultrasonic radar (US), etc., and deploys them on the robot body in different spatial layouts according to the robot's task requirements and application scenarios.

[0051] Raw data preprocessing module: Performs noise reduction, filtering, and distortion correction on the raw point cloud data or echo signals collected by each radar sensor unit;

[0052] Local feature extraction module: Extracts local three-dimensional geometric features, reflection intensity features, echo time-domain features, and frequency-domain features from the preprocessed data of each radar sensor unit;

[0053] Multi-sensor data alignment and calibration module: Establishes spatial geometric relationships (extrinsic parameters) and time synchronization mechanisms between different radar sensors to ensure spatial and temporal consistency of data from different sensors;

[0054] Feature-level or point cloud-level fusion module: used to integrate information from different radar sensors;

[0055] Dynamic Environment Modeling and Tracking Module: Based on the fused environmental information, it accurately models and continuously tracks dynamic targets in the environment;

[0056] Output interface: Provides the fused high-precision and robust environmental perception results (such as obstacle list, environment map, target attributes, etc.) to the robot's upper-level control system (such as navigation, decision making, etc.).

[0057] The feature-level or point cloud-level fusion module includes:

[0058] Target association-based fusion: Matching potential targets detected by different radar sensors using geometric, kinematic, or semantic association algorithms, and then fusing the matched target information;

[0059] Voxel / grid-based fusion: Projecting data from different radar sensors onto a unified 3D voxel space or grid map, and generating a unified environmental occupancy grid map or dense point cloud through a voting mechanism or probabilistic fusion.

[0060] End-to-end fusion based on deep learning: Using deep neural networks, raw data or extracted features from different radar sensors are used as input to directly output environmental perception results;

[0061] The dynamic environment modeling and tracking module includes:

[0062] State estimator: Based on the fused environmental information, Kalman filtering and particle filtering algorithms are used to estimate and predict the state (position, velocity, acceleration, attitude, etc.) of the detected dynamic target.

[0063] Multi-target tracker: Manages and maintains the IDs of multiple targets, distinguishes different targets, and continuously tracks the target's motion trajectory;

[0064] Static environment representation: Create a static environment map around the robot, including obstacles and terrain information, and support map updates and maintenance.

[0065] The multiple radar sensor units include at least one millimeter-wave radar for providing long-range, all-weather detection capabilities; and at least one lidar for providing high-precision three-dimensional geometric information; the multiple radar sensor units include various types of radars with different field of view, for example, a long-range millimeter-wave radar for long-range detection, multiple short-range millimeter-wave radars for short-range blind zone coverage, and a lidar for high-precision modeling.

[0066] The features extracted by the local feature extraction module include: point cloud density, normal vector, curvature, point cloud shape descriptor, echo amplitude, Doppler velocity, and phase information.

[0067] The multi-sensor data alignment and calibration module employs an iterative nearest point (ICP) algorithm, a point cloud registration algorithm, or an automatic calibration method based on prior knowledge.

[0068] The feature-level fusion is achieved by calculating the similarity or consistency of features between different sensors and then performing weighted fusion in conjunction with prior information.

[0069] The point cloud-level fusion adopts a probability-based fusion method, which accumulates or averages the occupancy probability information output by each radar sensor in a shared space.

[0070] The deep learning-based end-to-end fusion employs a multimodal fusion network, which includes an input layer and a feature fusion layer specifically designed for data from different radar sensors.

[0071] The dynamic environment modeling and tracking module has the ability to predict target behavior, for example, by learning historical movement trajectories to predict possible future movement directions and intentions.

[0072] The dynamic environment modeling and tracking module can adaptively adjust the parameters of the state estimator to cope with the motion characteristics of different targets and sensor noise levels.

[0073] In a further embodiment, the plurality of radar sensor units include at least one coaxial phase-geometry composite radar, which fuses at least two different frequency bands of electromagnetic waves at the physical waveform level through a single transmitting and receiving device.

[0074] The coaxial phase-geometry composite radar described above has a raw data preprocessing module that can decouple and generate an enhanced point cloud from the composite echo signal. In addition to the three-dimensional geometric coordinates, each point in the point cloud is natively bound to attribute information from radar waves of different frequency bands, including: Doppler velocity and material information characterized by radar cross-section.

[0075] In a further embodiment, the dynamic environment modeling and tracking module directly utilizes the velocity attribute inherent in each point in the enhanced point cloud, instantly distinguishing between static and dynamic targets without the need for clustering and trajectory tracking; and uses the material information to assist in the preliminary identification of target categories.

[0076] In a further embodiment, the dynamic adaptive fusion adjusts the weights based on quantifiable sensor data quality metrics, including:

[0077] For lidar: point cloud density, signal-to-noise ratio;

[0078] For millimeter-wave radar: echo signal strength, spectral purity;

[0079] For coaxial phase-geometry composite radar: signal-to-noise ratio of decoupled signals in each frequency band;

[0080] The fusion module dynamically adjusts the contribution weight of each sensor's data in the fusion process based on preset threshold rules or machine learning models.

[0081] Furthermore, this invention discloses a robot environmental perception system based on multi-radar fusion, comprising:

[0082] 1. Multiple radar sensor units:

[0083] The system employs at least two different types of radar sensors, such as millimeter-wave radar (MMW), lidar (LiDAR), and / or ultrasonic radar (US).

[0084] These radar sensors can be deployed in an optimized spatial layout according to the robot's specific application scenario and task requirements. For example, long-range millimeter-wave radar can be installed on the top or front of the robot for long-range environmental monitoring; short-range millimeter-wave radar or ultrasonic radar can be installed around the robot for close-range blind spot coverage and obstacle detection; and lidar can be installed on a stable platform to acquire high-precision 3D environmental models.

[0085] The configuration of different radar sensors can be flexibly selected based on cost, power consumption, and performance requirements.

[0086] 2. Raw data preprocessing module:

[0087] The raw data collected by each radar sensor unit, such as point cloud data from lidar and echo signals (distance, angle, velocity, intensity, etc.) from millimeter-wave radar, needs to be preprocessed.

[0088] Preprocessing operations include, but are not limited to: point cloud denoising (such as removing isolated points and noise points), filtering (such as Gaussian filtering and median filtering), point cloud distortion correction (such as compensating for distortion caused by sensor motion), gain correction of echo signals, and multipath suppression.

[0089] 3. Local Feature Extraction Module:

[0090] For each radar sensor data after preprocessing, its local three-dimensional geometric features, reflection intensity features, echo time domain features, and frequency domain features are extracted.

[0091] Geometric features: For example, the normal vector, curvature, local point density, and point cloud shape descriptors (such as FPFH, SHOT, etc.).

[0092] Reflection intensity characteristics: For example, the echo intensity of lidar and the RCS (radar cross-section) of millimeter-wave radar.

[0093] Echo time-domain / frequency-domain characteristics: For millimeter-wave radar, the amplitude, phase, and Doppler frequency (i.e., velocity information) of the echo can be extracted.

[0094] 4. Multi-sensor data alignment and calibration module:

[0095] In order to effectively fuse data from different radar sensors, it is necessary to establish spatial geometric relationships (extrinsic parameters) and time synchronization mechanisms between them.

[0096] Spatial calibration: Determine the precise pose (translation and rotation) of each radar sensor in the robot coordinate system, as well as the relative poses between sensors. This can be achieved through manual calibration, automatic calibration algorithms (such as ICP algorithms based on point cloud matching, feature matching algorithms), or by combining auxiliary sensors such as IMUs (Inertial Measurement Units).

[0097] Time synchronization: Ensures that data from different radar sensors accurately corresponds to the same point in time. This can be achieved using hardware synchronization (such as the PTP protocol) or software synchronization (such as timestamp-based interpolation correction).

[0098] 5. Feature-level or point cloud-level fusion module:

[0099] This module is the core of the system, responsible for integrating information from different radar sensors. This invention proposes multiple fusion strategies, which can be selected or combined according to actual needs.

[0100] Target-related fusion:

[0101] First, target detection is performed independently for each radar sensor (e.g., based on clustering, DBSCAN, or deep learning-based target detection algorithms).

[0102] Then, target association algorithms (such as the Hungarian algorithm or JPDAF) are used to match potential targets detected from different sensors. The matching criteria can be the target's geometric position, velocity, size, appearance features, etc.

[0103] Once the targets are associated, their information is fused, for example, by weighted averaging or Bayesian updates, to obtain a more accurate estimate of the target state.

[0104] Voxel / raster-based fusion:

[0105] All radar sensor data are projected into a unified three-dimensional voxel space or a two-dimensional / three-dimensional raster map.

[0106] For each voxel or grid, the probability of its occupancy can be calculated based on information from different sensors. For example, if a lidar has a measurement point within the voxel, its occupancy probability increases; if a millimeter-wave radar detects a target near the voxel, it also contributes to its occupancy probability.

[0107] A uniform environment occupancy raster map or dense point cloud is generated by accumulating, averaging, or using more complex probabilistic fusion algorithms (such as Bayesian filtering). This method is particularly suitable for constructing static environment maps.

[0108] End-to-end fusion based on deep learning:

[0109] Deep neural networks (such as PointNet++, VoxelNet, BEV-Net, etc.) can be used to directly process raw data or extracted intermediate features from different radar sensors.

[0110] The design of a network needs to consider how to effectively fuse sensor data from different modalities. For example, a multi-branch network can be used, with each branch processing data from one type of sensor, and then the features are concatenated, added, or fused through an attention mechanism in an intermediate layer.

[0111] End-to-end fusion can directly output higher-level perceptual information such as object detection boxes, semantic segmentation results, and scene understanding.

[0112] 6. Dynamic Environment Modeling and Tracking Module:

[0113] Based on the fused environmental information, dynamic targets in the environment are accurately modeled and continuously tracked.

[0114] State estimator: For each detected target, algorithms such as extended Kalman filter (EKF), unscented Kalman filter (UKF), or particle filter (PF) are used to update and predict the target's motion state (position, velocity, acceleration, attitude, size, etc.) in real time based on the fused measurement information.

[0115] Multi-target tracker: Responsible for managing and maintaining the IDs of all dynamic targets in the scene, ensuring that measurement values ​​are correctly assigned to the tracker when targets appear, disappear, merge, or split, and maintaining a stable trajectory for each target.

[0116] Static environment representation: The fused information can also be used to build and update the robot's static environment map, for example, by adding the outline information of obstacles to the local map or building a global map.

[0117] 7. Output Interface:

[0118] The fused, high-precision, and robust environmental perception results are output to the robot's upper-level control system in a structured data format, for example:

[0119] A list containing all detected obstacles (including their location, size, speed, and type).

[0120] A semantic map or occupancy grid map representing the robot's surrounding environment.

[0121] Detailed attribute information about a specific target (such as pedestrians or vehicles).

[0122] Furthermore, the present invention is described in detail below:

[0123] Innovative fusion strategies for multimodal heterogeneous radars:

[0124] This invention is not simply a matter of splicing data from different radars, but rather fully considers the differences in detection principles and performance indicators (accuracy, range, resolution, penetration, cost, power consumption, and anti-interference capabilities) of different radar sensors (such as millimeter-wave radar, lidar, and ultrasonic radar).

[0125] Through a carefully designed spatial layout and a multi-layered fusion strategy, the advantages of different radars are complemented. For example, millimeter-wave radar is used for long-range, all-weather detection, combined with the high-precision modeling capabilities of lidar and the fine-grained short-range detection of ultrasonic radar. This fusion of heterogeneous radars has greater versatility and robustness than the fusion of single-type radars.

[0126] Creative fusion methods: For example, an "enhanced perception" fusion method can be designed, in which when the performance of lidar degrades due to bad weather (such as heavy fog), the system can prioritize relying on the low-precision but all-weather target information provided by millimeter-wave radar, and combine it with the Doppler information of millimeter-wave radar to help estimate the target velocity, and then try to use probabilistic methods to invert possible lidar point cloud areas, so as to maintain a certain perception capability even in harsh environments.

[0127] Dynamically adaptive fusion and modeling algorithms:

[0128] Traditional fusion algorithms often employ fixed weights or fusion rules. The system proposed in this invention uses a fusion algorithm that dynamically and adaptively adjusts weight allocation and fusion strategies based on the complexity of the current environment, the number and motion state of targets, and the quality of data from each sensor.

[0129] For example, in areas with dense targets, more emphasis can be placed on sensors that can distinguish targets; when the environment changes drastically, the weight of new data can be appropriately increased; when the data quality of a certain sensor deteriorates (such as when lidar is interfered with by rain), its weight in the fusion will be reduced, and other sensors will be relied upon instead.

[0130] Adaptability of dynamic environment modeling: State estimators and multi-target trackers can dynamically adjust filter parameters based on the target's motion characteristics (such as uniform speed, uniform acceleration, maneuvering, etc.) and the uncertainty of sensor measurements. For example, when the target maneuvers, the covariance of acceleration can be increased to enable it to respond more quickly to changes in the target's motion.

[0131] Learning-based sensor redundancy information mining and utilization:

[0132] This invention not only utilizes the explicit information output by radar sensors (such as point cloud coordinates and target detection boxes), but more importantly, it uses deep learning technology to mine the implicit features contained in different radar sensors in different signal domains (such as echo intensity, Doppler spectrum, phase information, polarization information, etc.).

[0133] For example, millimeter-wave radar contains the target's RCS (radar cross-section), Doppler information (velocity), and can even preliminarily determine the type of target (such as vehicles and pedestrians) by analyzing the details of the echo signal.

[0134] By using multimodal deep learning models, these low-level and high-level features can be effectively fused, thereby improving the accuracy of target recognition and attribute estimation (such as size, pose, and type), and discovering information that is difficult to perceive directly by a single sensor.

[0135] Robustness and effective use of redundant information:

[0136] By integrating multiple radar sensors of different types, the system inherently possesses sensor redundancy. When one or more sensors fail, are blocked, or experience performance degradation under specific conditions, other sensors can still provide partial environmental information, ensuring the basic operation of the system.

[0137] This invention can intelligently utilize this redundant information, not only through simple complementarity but also for cross-validation. For example, when two different sensors detect a target with highly consistent attributes, the confidence level in that target can be increased. Conversely, if the detection results are inconsistent, more in-depth analysis or alarms can be triggered.

[0138] This redundancy mechanism makes the system more robust to complex environmental conditions such as severe weather (such as rain, snow, and fog), changes in lighting, and occlusion, significantly improving the robot's ability to work reliably in the real world.

[0139] Furthermore, specific embodiments of the present invention are as follows:

[0140] Example 1: Environmental perception of autonomous vehicles in urban road scenarios;

[0141] We have built a comparative testing platform: A) a traditional lidar + millimeter-wave radar fusion system; B) a system integrating the coaxial composite radar of this invention.

[0142] Test scenario: Dense fog at night, visibility 50 meters, the scenario includes stationary vehicles, moving pedestrians, and metal fences.

[0143] Sensor configuration:

[0144] A long-range millimeter-wave radar (24 GHz) is deployed at the front of the vehicle, with a detection range of 200 meters, used to detect vehicles and obstacles at a distance ahead.

[0145] Four short-range millimeter-wave radars (77 GHz) are deployed around the vehicle, with a detection range of 30 meters, for close-range blind spot coverage and pedestrian detection.

[0146] A 3D LiDAR (e.g., Velodyne HDL-64E) is deployed on the roof of the vehicle to provide a high-precision 360-degree three-dimensional point cloud.

[0147] Preprocessing: Noise reduction, point cloud distortion correction (combined with IMU data), and timestamp synchronization are performed on all sensor data.

[0148] Feature extraction: LiDAR extracts the normals and curvature of point clouds; millimeter-wave radar extracts the range, velocity, and RCS of targets.

[0149] Calibration: Using the ROS (Robot Operating System) calibration toolkit, the relative pose of each sensor and the vehicle body is accurately calibrated by scanning the target points, and time synchronization is performed.

[0150] Fusion:

[0151] Preliminary target detection: LiDAR performs 3D target detection using PointRCNN; millimeter-wave radar detects targets using radar signal processing algorithms.

[0152] Target Association and Fusion: The JPDAF (Joint Probabilistic Data Association Filter) algorithm is used to associate targets detected by different sensors. When the lidar and millimeter-wave radar detect the same target, their states (position, velocity, size) are fused. For example, the high-precision position of the lidar is used to correct the coarse position of the millimeter-wave radar, and the velocity information of the millimeter-wave radar is used to assist the Kalman filter to improve the accuracy of velocity estimation.

[0153] Dynamic environment modeling: For the fused target list, the state is estimated using UKF (Unscented Kalman Filter), and the target IDs are managed using a multi-target tracker.

[0154] Output: The output includes a list of obstacles with precise location, speed, and size in front of, to the sides, and behind the vehicle, as well as a high-precision grid map of the surrounding environment.

[0155] Beneficial effects of the application:

[0156] Robustness in adverse weather conditions: In heavy rain or dense fog, the performance of lidar degrades, and the system will rely more on target information provided by millimeter-wave radar. However, it can still assist the vehicle in judging driving safety through Doppler information from millimeter-wave radar.

[0157] Pedestrian detection robustness: Millimeter-wave radar can penetrate certain obstacles (such as sparse vegetation). In cases where lidar has difficulty detecting pedestrians, millimeter-wave radar may be able to capture their speed information. Combining the two can improve the success rate of pedestrian detection.

[0158] Traffic light and sign recognition assistance: Combining automotive-grade millimeter-wave radar to sense the motion status of traffic lights (such as whether they are flashing) with camera image information, it assists in recognizing traffic signals.

[0159] Example 2: Industrial Robot Warehouse Inspection

[0160] Sensor configuration:

[0161] A 3D LiDAR (such as Slam-3D) is mounted on top of the robot for SLAM (Simultaneous Localization and Mapping) and high-precision environment modeling.

[0162] Two small millimeter-wave radars with wide field of view are installed in front of and behind the robot, respectively, to detect static obstacles such as shelves and boxes, as well as moving forklifts or personnel.

[0163] Four ultrasonic sensors are mounted on the bottom of the robot to detect ground depressions and low obstacles, and to assist in precise close-range positioning.

[0164] Preprocessing and calibration: Similar to Example 1.

[0165] Fusion:

[0166] SLAM mapping: Primarily accomplished by LiDAR, it constructs a high-precision 3D point cloud map.

[0167] Dynamic obstacle detection and tracking: This involves distinguishing moving targets detected by millimeter-wave radar from static environments scanned by lidar. If millimeter-wave radar detects a target, but lidar does not detect an object in the area, the target is considered dynamic.

[0168] Low Obstacles and Ground Detection: Data from ultrasonic sensors is compared with bottom scan data from lidar to identify ground anomalies or low obstacles.

[0169] Grid-based fusion: Data from all sensors is projected onto a unified two-dimensional occupancy grid map. LiDAR and millimeter-wave radar provide occupancy probabilities for the grid, while ultrasonic sensors assist in labeling the attributes of ground areas.

[0170] Output: Real-time updated robot position, high-precision 3D map, and tracking information for dynamic obstacles.

[0171] Beneficial effects of the application:

[0172] Obstacle avoidance in complex shelving environments: LiDAR provides detailed outlines of the shelving, while millimeter-wave radar detects pedestrians or forklifts that may be moving through the gaps in the shelving, ensuring the robot's safe passage.

[0173] Ground anomaly detection: Combining ultrasonic and lidar data, it effectively detects potholes, slopes, and other features on the ground, providing a basis for the robot to plan a smooth path.

[0174] Efficient inspection in unmanned warehouses: The fused perception information enables robots to navigate autonomously in complex warehouse environments, avoid obstacles, and efficiently complete tasks such as inventory and inspection.

[0175] To fundamentally improve fusion performance and simplify subsequent algorithms, this invention proposes an innovative hardware deep fusion scheme as a preferred embodiment. The core of this scheme lies in coaxial phase-geometry composite radar.

[0176] This radar employs a unique coaxial common-path optics and radio frequency design, physically coupling a laser beam (e.g., 905nm or 1550nm) with a millimeter-wave radio frequency signal (e.g., 77GHz) at the transmitting antenna, ensuring they are transmitted along the same optical axis / beam path. The receiving end uses a common-aperture receiver to synchronously receive the composite signal reflected back from the target.

[0177] The raw data preprocessing module includes a dedicated waveform decoupling and attribute extraction algorithm. This algorithm separates the laser time-of-flight (TOF) information from the millimeter-wave Doppler frequency shift and phase change information through time-frequency analysis of the composite echo signal.

[0178] Output: An enhanced point cloud. Each point contains:

[0179] Geometric information (x, y, z): calculated with high precision by laser ToF (accuracy up to ±2cm).

[0180] Velocity information (v): directly derived from millimeter-wave Doppler frequency shift (velocity resolution up to 0.1 m / s), this velocity is a point-level native attribute.

[0181] Material attribute label (o): Preliminary judgment based on the millimeter-wave echo intensity (RCS) corresponding to this point. For example, set the RCS threshold Th metal (e.g., -10 dBsm calibrated experimentally). When the RCS of a point is greater than Th metal, it is marked as "high reflectivity / possibly metallic"; otherwise, it is marked as "low reflectivity / possibly non-metallic".

[0182] Instantaneous identification of dynamic targets: Traditional methods require clustering of point clouds and then calculating target velocity through multi-frame tracking. In this method, any point with a velocity value |v| > 0.2 m / s can be immediately regarded as a dynamic point, achieving separation of static and dynamic targets without tracking.

[0183] Target classification enhancement: Static vehicles and static fences may be geometrically similar, but their point cloud mean RCS values ​​differ significantly (vehicles are typically >-5 dBsm, while fences are typically <-20 dBsm). The system can directly use this attribute to distinguish them.

[0184] Robustness in extreme environments: In dense fog (visibility < 50 meters), the density of laser point clouds may decrease by 70%. In this situation, the system will automatically reduce the fusion weight of geometric information and increase the weight of velocity and existence probability information provided by meter-wavelength lasers. The fusion module adaptively adjusts based on the quantitative indicator of laser point cloud density (points / cubic meter). For example, when the density is below the threshold D thresh, a robust fusion mode is triggered.

[0185] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A robot environmental perception system based on multi-radar fusion, characterized in that: include: Multiple radar sensor units: Includes at least two different types of radar sensors, and deploys them on the robot body in different spatial layouts according to the robot's task requirements and application scenarios; Raw data preprocessing module: Performs noise reduction, filtering, and distortion correction on the raw point cloud data or echo signals collected by each radar sensor unit; Local feature extraction module: Extracts local three-dimensional geometric features, reflection intensity features, and echo time domain features from the preprocessed data of each radar sensor unit; Multi-sensor data alignment and calibration module: Establishes spatial geometric relationships and time synchronization mechanisms between different radar sensors to ensure spatial and temporal consistency of data from different sensors; Feature-level or point cloud-level fusion module: used to integrate information from different radar sensors; Dynamic Environment Modeling and Tracking Module: Based on the fused environmental information, it accurately models and continuously tracks dynamic targets in the environment; Output interface: Provides the fused, high-precision, and robust environmental perception results to the robot's upper-level control system.

2. The robot environment perception system based on multi-radar fusion according to claim 1, characterized in that: The feature-level or point cloud-level fusion module includes: Target association-based fusion: Matching potential targets detected by different radar sensors using geometric, kinematic, or semantic association algorithms, and then fusing the matched target information; Voxel-based fusion: Projecting data from different radar sensors onto a unified 3D voxel space or grid map, and generating a unified environmental occupancy grid map or dense point cloud through a voting mechanism or probabilistic fusion. End-to-end fusion based on deep learning: Using deep neural networks, raw data or extracted features from different radar sensors are used as input to directly output environmental perception results; The dynamic environment modeling and tracking module includes: State estimator: Based on the fused environmental information, Kalman filtering and particle filtering algorithms are used to estimate and predict the state of the detected dynamic target; Multi-target tracker: Manages and maintains the IDs of multiple targets, distinguishes different targets, and continuously tracks the target's motion trajectory; Static environment representation: Create a static environment map around the robot, including obstacles and terrain information, and support map updates and maintenance.

3. The robot environment perception system based on multi-radar fusion according to claim 1, characterized in that: The plurality of radar sensor units include at least one millimeter-wave radar for providing long-range, all-weather detection capabilities; and at least one lidar for providing high-precision three-dimensional geometric information; The multiple radar sensor units include radars of various types and different field of view angles.

4. A robot environment perception system based on multi-radar fusion according to claim 3, characterized in that: The features extracted by the local feature extraction module include: point cloud density, normal vector, curvature, point cloud shape descriptor, echo amplitude, Doppler velocity, and phase information.

5. A robot environment perception system based on multi-radar fusion according to claim 3, characterized in that: The multi-sensor data alignment and calibration module adopts an automatic calibration method based on iterative nearest point algorithm, point cloud registration algorithm, or prior knowledge. The feature-level fusion is achieved by calculating the similarity or consistency of features between different sensors and then performing weighted fusion in conjunction with prior information.

6. The robot environment perception system based on multi-radar fusion according to claim 1, characterized in that: The point cloud-level fusion adopts a probability-based fusion method, which accumulates or averages the occupancy probability information output by each radar sensor in a shared space. The deep learning-based end-to-end fusion employs a multimodal fusion network, which includes an input layer and a feature fusion layer specifically designed for data from different radar sensors.

7. A robot environment perception system based on multi-radar fusion according to claim 1, characterized in that: The dynamic environment modeling and tracking module described above has the ability to predict target behavior; The dynamic environment modeling and tracking module can adaptively adjust the parameters of the state estimator to cope with the motion characteristics of different targets and sensor noise levels.

8. A robot environment perception system based on multi-radar fusion according to claim 1, characterized in that: The plurality of radar sensor units include at least one coaxial phase-geometry composite radar, which fuses at least two different frequency bands of electromagnetic waves at the physical waveform level through a single transmitting and receiving device. The coaxial phase-geometry composite radar described above has a raw data preprocessing module that can decouple and generate an enhanced point cloud from the composite echo signal. In addition to the three-dimensional geometric coordinates, each point in the point cloud is natively bound to attribute information from radar waves of different frequency bands, including: Doppler velocity and material information characterized by radar cross-section.

9. A robot environmental perception system based on multi-radar fusion according to claim 1, characterized in that: The dynamic environment modeling and tracking module directly utilizes the velocity attribute inherent in each point in the enhanced point cloud, instantly distinguishing between static and dynamic targets without the need for clustering and trajectory tracking; and uses the material information to assist in the preliminary identification of target categories.

10. A robot environmental perception system based on multi-radar fusion according to claim 1 or 9, characterized in that: The dynamic adaptive fusion adjusts its weights based on quantifiable sensor data quality metrics, including: For lidar: point cloud density, signal-to-noise ratio; For millimeter-wave radar: echo signal strength, spectral purity; For coaxial phase-geometry composite radar: signal-to-noise ratio of decoupled signals in each frequency band; The fusion module dynamically adjusts the contribution weight of each sensor's data in the fusion process based on preset threshold rules or machine learning models.

Images