Unmanned vehicle obstacle detection method and system based on multi-sensor fusion

Abstract

The application provides an unmanned vehicle obstacle detection method and system based on multi-sensor fusion, the method comprising synchronously collecting millimeter wave radar non-uniform echo, laser radar point cloud and camera image; performing frequency domain processing on the radar echo to generate an initial three-dimensional spectrum; clustering the point cloud to generate an obstacle candidate region and boundary confidence, and converting the obstacle candidate region and boundary confidence into a spectrum region of interest; performing semantic segmentation and inverse perspective transformation on the image to obtain a bird's eye view target class prior; performing bounded nonlinear spectrum peak sharpening and background spectrum intensity suppression on the spectrum region of interest in combination with the geometric size of the candidate region to obtain an enhanced spectrum; extracting a radar spectrum peak quality factor and a radial motion feature and aligning them to a bird's eye view coordinate system; constructing a Bayesian evidence matrix, obtaining an obstacle occupancy state through posterior probability reasoning, and fusing and outputting a motion attribute and a geometric attribute.

Description

Technical Field

[0001] This application belongs to the field of multi-sensor fusion, and in particular relates to an obstacle detection method and system for unmanned vehicles based on multi-sensor fusion. Background Technology

[0002] Vehicle-mounted environmental perception systems typically rely on multiple sensors, including millimeter-wave radar, lidar, and cameras, to acquire road environment information. Cameras provide color, texture, and semantic information, but their perception stability is easily degraded under adverse weather conditions, sudden changes in light intensity, or occlusion. LiDAR can acquire high-precision 3D point cloud data to characterize the geometric contours and spatial boundaries of obstacles, but its penetration capability is limited in rain, snow, and fog, and it lacks direct Doppler velocity measurement capabilities. Millimeter-wave radar has strong penetration and all-weather operation capabilities, and can measure the radial motion information of targets, but its spatial resolution is relatively limited, making it difficult to provide complete target shape features. Therefore, a single sensor cannot meet the requirements of autonomous vehicles for obstacle detection accuracy, stability, and robustness in complex traffic scenarios. Multi-sensor fusion perception has become an important development direction in the field of autonomous vehicle environmental perception.

[0003] However, data acquired by different sensors exhibit significant heterogeneity in data structure, sampling dimensions, spatial coordinates, and feature representation methods. Achieving spatiotemporal alignment, feature interaction, and probabilistic fusion of multi-source data within a unified framework remains a critical technical challenge in the field of autonomous driving perception. Existing fusion solutions often remain at the result-level fusion or shallow feature stitching level, failing to fully leverage the motion perception capabilities of millimeter-wave radar, the spatial boundary representation capabilities of lidar, and the category semantic prior capabilities of cameras. In complex environments, this can easily lead to inaccurate estimation of obstacle geometric attributes, delayed motion state recognition, and false positives or false negatives. Therefore, a novel multi-sensor fusion obstacle detection method is urgently needed to improve the accuracy and consistency of autonomous vehicles' comprehensive perception of obstacle presence, motion state, and geometric attributes. Summary of the Invention

[0004] To address the problems of heterogeneous multi-source sensor data, difficulties in spatiotemporal alignment, and shallow fusion levels in existing technologies, which lead to insufficient accuracy and stability in the joint estimation of obstacle existence, motion, and geometric attributes, this invention provides an obstacle detection method and system for unmanned vehicles based on multi-sensor fusion.

[0005] In a first aspect, the present invention provides an obstacle detection method for unmanned vehicles based on multi-sensor fusion, the method comprising:

[0006] The system acquires non-uniform echo signals from millimeter-wave radar, point clouds from lidar, and images from a camera. It performs a non-uniform Fourier transform on the non-uniform echo signal in a fast time dimension, and performs Fourier transforms in a slow time dimension and an array dimension to generate an initial three-dimensional spectrum. It then performs clustering processing on the lidar point cloud to generate obstacle candidate regions and boundary confidence scores, and converts the obstacle candidate regions into spectral interest regions that match the initial three-dimensional spectrum. Finally, it performs semantic segmentation and inverse perspective transformation on the camera images to obtain the prior probability of the target category from a bird's-eye view. Within the region of interest of the initial three-dimensional spectrum, a nonlinear function based on the geometric dimensions of the obstacle candidate region is applied to sharpen the spectral peaks, and the background spectral intensity estimation basis is subtracted to obtain the enhanced three-dimensional spectrum; based on the preset scattering model, the geometric dimensions of the obstacle candidate region, and the enhanced three-dimensional spectrum, the radar spectral peak quality factor and motion features are extracted and aligned to the Cartesian coordinate system of the bird's-eye view. A Bayesian evidence matrix is ​​constructed in each grid of the Cartesian coordinate system of the bird's-eye view. The elements of the Bayesian evidence matrix include the spectral peak quality factor, the motion feature, the boundary confidence, and the prior probability of the target category. Posterior probability maximization inference is performed on the Bayesian evidence matrix to obtain the obstacle occupancy state, and the motion state and geometric attributes of the obstacle are estimated by fusing them.

[0007] On the other hand, the present invention also provides an obstacle detection system for unmanned vehicles based on multi-sensor fusion, the system comprising the following modules: The acquisition module is used to acquire non-uniform echo signals from millimeter-wave radar, lidar point clouds, and camera images; perform non-uniform Fourier transform on the non-uniform echo signals in the fast time dimension, and perform Fourier transform on the slow time dimension and array dimension to generate an initial three-dimensional spectrum; perform clustering processing on the lidar point clouds to generate obstacle candidate regions and boundary confidence scores, and convert the obstacle candidate regions into spectral interest regions that match the initial three-dimensional spectrum; perform semantic segmentation and inverse perspective transformation on the camera images to obtain the prior probability of target categories under a bird's-eye view. The calculation module is used to sharpen the spectral peaks within the region of interest of the initial three-dimensional spectrum by applying a nonlinear function based on the geometric dimensions of the obstacle candidate region, and subtracting the background spectral intensity estimation basis to obtain the enhanced three-dimensional spectrum; based on the preset scattering model, the geometric dimensions of the obstacle candidate region, and the enhanced three-dimensional spectrum, the radar spectral peak quality factor and motion features are extracted and aligned to the Cartesian coordinate system of the bird's-eye view. An estimation module is used to construct a Bayesian evidence matrix in each grid of the Cartesian coordinate system of the bird's-eye view. The elements of the Bayesian evidence matrix include the spectral peak quality factor, the motion feature, the boundary confidence, and the prior probability of the target category. Posterior probability maximization inference is performed on the Bayesian evidence matrix to obtain the obstacle occupancy state, and the motion state and geometric attributes of the estimated obstacle are fused together.

[0008] This invention processes the non-uniform echo signal from millimeter-wave radar using non-uniform Fourier transform to reduce spectral leakage caused by irregular sampling. It utilizes candidate regions generated from lidar point clouds to transform and obtain spectral regions of interest, guiding radar spectral enhancement processing. Furthermore, it combines the geometric dimensions of the candidate regions with bounded nonlinear peak sharpening and background spectral intensity suppression, which helps improve the discriminability of radar features in the target area. Further, it aligns the radar spectral peak quality factor, radial velocity motion characteristics, lidar boundary confidence, and visual category prior probability to the bird's-eye view coordinate system to construct a Bayesian evidence matrix. This matrix is ​​then combined with Markov random fields and posterior probability inference to obtain the obstacle occupancy state. Finally, it fuses and outputs motion and geometric attributes, thereby improving the stability and consistency of obstacle detection for autonomous vehicles. Attached Figure Description

[0009] Figure 1 The flowchart shows an obstacle detection method for autonomous vehicles based on multi-sensor fusion. Figure 2 This is a schematic diagram comparing the range spectrum before and after processing of non-uniform radar echo signals. Figure 3 This is a comparison chart of the velocity estimation errors in three sets of experiments. Detailed Implementation

[0010] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0011] It should be understood that the terms “comprising” and “having”, and any variations thereof, in the embodiments of this specification are intended to cover but not exclude inclusion. For example, a product or device that includes a series of components is not necessarily limited to those components that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0012] Example 1 In Embodiment 1 of the present invention, an obstacle detection method for unmanned vehicles based on multi-sensor fusion is provided, such as... Figure 1 As shown, it includes: S1 acquires the non-uniform echo signal from the millimeter-wave radar, the point cloud from the lidar, and the image from the camera.

[0013] By using the message subscription mechanism in the robot's operating system, relevant topic callback functions are invoked to synchronously acquire data from multiple sensor sources.

[0014] Using the driver interface built into the radar module, the non-uniformly sampled intermediate frequency echo signal data stream after analog-to-digital conversion is acquired.

[0015] The PCLPointCloud2 data structure in the point cloud library is used to receive the point cloud dataset generated by 3D LiDAR scanning. This dataset contains the 3D spatial coordinates and reflection intensity information of each scan point.

[0016] The system reads real-time RGB 3D array images captured by the vehicle-mounted camera and calculates the time difference between data from different sensors using a timestamp alignment algorithm. If the time difference is less than a set threshold of 10ms, synchronization is considered successful.

[0017] In some embodiments, acquiring the non-uniform echo signal from the millimeter-wave radar, the point cloud from the lidar, and the camera image includes: The vehicle-mounted synchronization trigger controller sends global timestamp synchronization pulses to the millimeter-wave radar, lidar, and camera at a fixed frequency. Record the non-uniform echo signal with time-stamped frequency hopping interval output from the intermediate frequency receiving link of the millimeter-wave radar; Receive the lidar point cloud in a three-dimensional floating-point coordinate system calculated by the lidar; It acquires a time-synchronized, distortion-corrected color pixel matrix image output from the camera.

[0018] The vehicle-mounted synchronous trigger controller uses a field-programmable gate array or microcontroller unit as its hardware foundation. It uses a hardware timer to simultaneously send global timestamp synchronization pulses with a pulse width of 1μs to 5μs and NMEA time messages to millimeter-wave radar, lidar and camera at a fixed frequency of, for example, 10Hz.

[0019] Upon receiving the synchronization pulse, the intermediate frequency (IF) receiving link of the millimeter-wave radar will output a non-uniform echo signal. Due to the frequency hopping modulation strategy, this signal exhibits a non-uniform offset in the sampling interval in the fast time dimension. The frequency hopping interval time offset typically fluctuates between 0.1 μs and 2 μs. At this time, the timestamp information corresponding to each sampling point will be recorded.

[0020] Simultaneously, the LiDAR will calculate the 3D point cloud data for each frame based on the received synchronization time, and output its horizontal, vertical, and 3D coordinates, reflectivity intensity, and relative time offset in the sensor's own Cartesian coordinate system in single-precision floating-point format. Furthermore, the RGB camera completes synchronized exposure under the drive of a trigger signal, acquiring a time-synchronized raw image, and calls upon the camera intrinsic distortion matrix containing radial and tangential distortion parameters, pre-acquired during the calibration phase.

[0021] The original image is subjected to pixel-by-pixel distortion correction using bilinear interpolation, and the resulting color pixel matrix image is aligned to the same global time base.

[0022] S2, perform a non-uniform Fourier transform on the non-uniform echo signal in the fast time dimension, and perform Fourier transforms in the slow time dimension and the array dimension to generate an initial three-dimensional spectrum.

[0023] Since the sampling points of the non-uniform echo signal are not equidistant in the fast time dimension, a non-uniform fast Fourier transform algorithm is used to resample, interpolate, and frequency-domain map the echo signal in each cycle, converting the time-domain signal into a distance-domain signal to obtain the distance resolution.

[0024] In the slow time dimension, signals from the same distance unit are grouped into a sequence, and a one-dimensional fast Fourier transform algorithm is executed to extract Doppler frequency shift information and obtain the target's velocity information.

[0025] In the array dimension, a one-dimensional fast Fourier transform algorithm is performed on the signals from the same distance and velocity elements of multiple receiving antenna channels to obtain azimuth information. This generates an initial three-dimensional spectrum matrix containing distance, Doppler velocity, and azimuth dimensions.

[0026] In some embodiments, performing a non-uniform Fourier transform on the non-uniform echo signal in the fast time dimension and performing Fourier transforms in the slow time dimension and the array dimension to generate an initial three-dimensional spectrum includes: The non-uniform fast Fourier transform algorithm is used to perform distance dimension resampling and frequency domain mapping on the fast time sampling points of the non-uniform echo signal with time offset, and the distance dimension spectrum is extracted. Perform a discrete Fourier transform on the echo phase difference of the same distance cell in the slow time dimension to obtain the Doppler velocity spectrum; A spatial discrete Fourier transform is performed on the spatial phase difference of the multi-channel receiving antenna in the array dimension to calculate the azimuth angle of the target and synthesize an initial three-dimensional spectrum containing range, Doppler and angle information.

[0027] For non-uniform fast-time sampling points with time offsets, a non-uniform Fast Fourier Transform (FFT) algorithm is applied. Using an equal interpolation kernel (Kaiser-Bessel kernel), the non-uniform original echo signal is projected and oversampled onto a uniformly distributed high-density one-dimensional time grid. The oversampling rate is preferably configured to 1.5 to 2.0 to reduce spectral leakage caused by non-uniform sampling. A comparison of the range spectrum before and after non-uniform radar echo signal processing is provided. Figure 2 As shown.

[0028] After completing the frequency domain mapping, the range dimension spectrum with, for example, 512 or 1024 range gates is extracted by standard 1D-FFT operation. At this time, the preferred range of system range resolution is configured between 0.1 meters and 0.3 meters.

[0029] In the slow time dimension, for continuously transmitted frequency-modulated continuous wave pulses, such as 128 to 256, the phase change history of the echo signal within each specific range cell is extracted. After suppressing sidelobes by applying a Hanning window, a discrete Fourier transform is performed to obtain the Doppler velocity spectrum, with a velocity resolution set between 0.1 m / s and 0.5 m / s.

[0030] After obtaining the two-dimensional spectrum, the radar's multi-channel receiving antenna array is subjected to a discrete Fourier transform, for example, with 64 zero-padding, by utilizing the echo spatial phase difference between different antenna elements in the array dimension.

[0031] By solving azimuth information with an angular resolution between 1° and 3°, a characteristic three-dimensional spectrum matrix data block integrating range, Doppler velocity, and azimuth angle is synthesized.

[0032] Meanwhile, the system caches the initial three-dimensional spectrum without nonlinear gain modulation and performs linear power calibration on it based on the radar system calibration parameters to generate a linear calibration power spectrum. The linear calibration power spectrum is used for the conversion of the equivalent radar cross section measurement value, and the enhanced three-dimensional spectrum is mainly used for the determination of candidate spectral peak positions and convexity calculation.

[0033] S3, perform clustering processing on the lidar point cloud to generate obstacle candidate regions and boundary confidence, and convert the obstacle candidate regions into spectral interest regions that match the initial three-dimensional spectrum.

[0034] For the input LiDAR point cloud, the VoxelGrid filter function from the point cloud library is called to perform voxel downsampling and remove redundant points. Then, the Random Sample Consensus Algorithm (RANSAC) is used to remove ground point clouds.

[0035] The density-based, noisy application spatial clustering algorithm DBSCAN is invoked to perform density clustering on the non-ground point cloud, generating multiple independent obstacle geometry clusters.

[0036] Principal component analysis (PCA) is used to calculate the length, width, and orientation angle of each geometric cluster. Boundary confidence values ​​are then calculated based on the density gradient of the point cloud at the boundaries.

[0037] The coordinates of the 3D bounding boxes of each obstacle geometric cluster are extracted, and the horizontal projection of the 3D bounding boxes is mapped onto a 2D bird's-eye view grid plane to generate corresponding obstacle candidate regions. Based on the extrinsic parameter calibration relationship between the radar, lidar, and vehicle unified coordinate system, the obstacle candidate regions are transformed into the radar range-angle coordinate domain to form a 2D spectral region of interest; then, it is extended along the Doppler velocity dimension of the initial 3D spectrum to generate a 3D spectral region of interest that matches the coordinate dimension of the initial 3D spectrum.

[0038] In some embodiments, the step of clustering the lidar point cloud to generate obstacle candidate regions and boundary confidence scores, and converting the obstacle candidate regions into spectral interest regions that match the initial three-dimensional spectrum, includes: The lidar point cloud is divided into a three-dimensional voxel grid, and discrete noise points with a point cloud density lower than a preset threshold are filtered out. The connected elements are aggregated into independent obstacle geometry clusters using a density clustering algorithm, and the three-dimensional bounding box size of the obstacle geometry clusters is calculated. Calculate the variance of the distribution density of the point cloud at the edge of the obstacle geometric cluster, and generate a boundary confidence level between 0 and 1 based on the variance through negative correlation mapping; Extract the two-dimensional orthogonal projection of the three-dimensional bounding box onto the horizontal plane as a spatial region of interest, and transform and align the coordinate system to the unified bird's-eye view coordinate system of the vehicle; Based on the extrinsic parameter calibration relationship between radar, lidar and vehicle unified coordinate system, the spatial region of interest is converted into a two-dimensional spectral region of interest in the radar range-angle coordinate domain, and the two-dimensional spectral region of interest is extended along the Doppler dimension into a three-dimensional spectral region of interest.

[0039] The input 3D floating-point LiDAR point cloud space is divided into a uniform 3D voxel grid with a fixed side length of 0.1 meters to 0.2 meters. Voxel filtering is performed by counting the number of points in each voxel, removing voxels with fewer than 3 to 5 points in their internal cloud, and filtering out sparse noise points caused by rain, fog, or sensor thermal noise.

[0040] Using the density-based DBSCAN clustering algorithm, with a connected neighborhood search radius set to 0.3 m to 0.5 m, connected voxels that meet the density condition are clustered into obstacle geometric clusters. Principal component analysis is then used to obtain the 3D directed bounding box of each obstacle geometric cluster, outputting its length, width, and height physical dimensions.

[0041] Using the outer surface of the three-dimensional directed bounding box as a boundary reference, a boundary neighborhood buffer region is constructed within a 0.2-meter range both inside and outside the outer surface. Within this buffer region, the number of point clouds for each voxel is counted using a 0.1-meter voxel grid, and the shortest distance from each point cloud to the outer surface is calculated. The number of point clouds for each voxel is normalized and its variance is calculated to obtain a density discrete term. The shortest distance from each point cloud to the outer surface is normalized and its variance is calculated to obtain a distance discrete term. The mean absolute value of the difference in the number of point clouds between adjacent voxels is calculated and normalized to obtain a density variation term. The density discrete term, distance discrete term, and density variation term are weighted and summed according to preset weights to obtain a distribution stability index. This is then further analyzed using... Calculate the boundary confidence, where, For boundary confidence, λ is the distribution stability index, and λ is the scale parameter, preferably 1.0.

[0042] The three-dimensional bounding box is projected onto a horizontal plane to generate a two-dimensional orthogonal polygon. Using a pre-calibrated extrinsic transformation matrix from the LiDAR to the vehicle unified coordinate system, the two-dimensional orthogonal polygon is transformed into the vehicle unified bird's-eye view Cartesian coordinate system, serving as an obstacle candidate region. For the boundary points or covering grid points of the obstacle candidate region, based on the extrinsic calibration relationship between the radar, LiDAR, and vehicle unified coordinate system, they are transformed into the radar coordinate system. The corresponding range index and azimuth index are calculated to form a two-dimensional spectral region of interest within the radar range-angle coordinate domain. The Doppler dimension extension range is determined based on the measurable radial relative velocity range of the millimeter-wave radar. The two-dimensional spectral region of interest is then expanded unit by unit according to the width of the Doppler velocity unit of the initial three-dimensional spectrum, generating a three-dimensional spectral region of interest that matches the range dimension, angle dimension, and Doppler dimension of the initial three-dimensional spectrum.

[0043] S4, perform semantic segmentation and inverse perspective transformation on the camera image to obtain the prior probability of the target category under the bird's-eye view.

[0044] The acquired real-time images are input into the deployed DeepLabV3Plus semantic segmentation network model.

[0045] After feature extraction from the backbone network and multi-scale feature fusion from the hollow spatial pyramid pooling layer, the output layer of the network model outputs a pixel-wise class probability map.

[0046] Using the pre-calibrated intrinsic and extrinsic translation and rotation matrices of the camera, the homography transformation matrix from the image plane to the bird's-eye view plane is calculated, and a perspective transformation mapping operation is performed on the probability map of this category.

[0047] Normalize the probability values ​​of various targets after mapping to the bird's-eye view plane, and output the prior probability matrix of each grid belonging to the target category of vehicle, pedestrian or background in the bird's-eye view coordinate system.

[0048] In some embodiments, performing semantic segmentation and inverse perspective transformation on the camera image to obtain the prior probability of the target category in the bird's-eye view includes: The camera image is input into a pre-trained fully convolutional neural network, which outputs a pixel-level multi-class semantic segmentation mask and a normalized probability distribution for each class. An inverse perspective mapping matrix is ​​constructed using a pre-calibrated camera intrinsic parameter matrix and an extrinsic parameter matrix relative to a unified bird's-eye view coordinate system. The inverse perspective mapping matrix is ​​used to reproject the target bottom pixels in the multi-category semantic segmentation mask and probability distribution from the two-dimensional image perspective space to the two-dimensional planar grid of the unified bird's-eye view. Bilinear interpolation is performed on the reprojected planar grid to generate the prior probability of the target category for each grid cell in the bird's-eye view.

[0049] The image processing pipeline scales and normalizes time-synchronized and distortion-corrected camera images before feeding them into a pre-trained, converged fully convolutional neural network (LCN) on an autonomous driving dataset. This LCN includes a backbone feature extraction module, a dilated spatial pyramid pooling module, and a decoder module.

[0050] The input to the fully convolutional neural network is a preprocessed image tensor with dimensions W (width), H (height), and C (number of channels). The backbone feature extraction module employs a deep residual network structure to perform multi-scale feature extraction on the input image, outputting a high-dimensional semantic feature map. The dilated spatial pyramid pooling module performs parallel sampling of the high-dimensional semantic feature map through dilated convolutions with different sampling rates. Dilated convolutions expand the receptive field and extract multi-scale contextual information by introducing a dilation rate r between the sampling points of the convolution kernel.

[0051] The decoder module concatenates multi-scale contextual information with shallow features from the backbone feature extraction module and restores the original image resolution through bilinear interpolation upsampling. The output module of the fully convolutional neural network calculates the probability of each category through a Softmax activation function layer.

[0052] After forward inference, the network outputs a pixel-level multi-class semantic segmentation mask with the same resolution as the original image, as well as a multi-dimensional normalized probability distribution tensor for each pixel belonging to the categories of pedestrian, passenger car, truck or background, ensuring that the sum of the probabilities of each category of a single pixel is 1.

[0053] The 3×3 camera intrinsic parameter matrix obtained by the system during offline factory calibration includes focal length and optical center parameters, as well as a 4×4 rigid body transformation extrinsic parameter matrix that reflects the physical relationship between the camera optical center and the vehicle coordinate system.

[0054] Using the principle of homography and the assumption of a zero-height horizontal road surface, a 3×3 inverse perspective mapping matrix is ​​calculated to transform the perspective plane of a two-dimensional image into a vehicle top-view bird's-eye view plane. Based on the inverse perspective mapping matrix, the image algorithm first performs connected component analysis on the foreground target region in the semantic mask image, extracting the lowest contour points of each connected component as pixels to be reprojected. When the bottom contour of the target is occluded or incomplete, the position of the pixels to be reprojected is constrained using the lower boundary of the target's bounding box, the tracking results of adjacent frames, or the candidate region of the LiDAR, in order to reduce the reprojection deviation of the bird's-eye view.

[0055] These pixels to be reprojected, along with their multi-class probability distribution tensors, are transformed using an inverse perspective mapping matrix and reprojected onto a unified bird's-eye view with a size set to 100 meters in front and 30 meters to the left and right, and a resolution of 0.1m × 0.1m. Figure 2 Within the 3D grid.

[0056] To address the raster jaggedness and pixel gaps resulting from discrete perspective reprojection, a bilinear interpolation filtering algorithm is applied to the grid plane. Based on the probability distribution values ​​of the four adjacent projected grid nodes and their Euclidean distance weights with the blank grid, bilinear interpolation is used to complete the blank grid that was not projected, thereby generating a smooth and continuous bird's-eye view raster-level target category prior probability layer.

[0057] S5, within the region of interest of the initial three-dimensional spectrum, a nonlinear function based on the geometric dimensions of the obstacle candidate region is applied to sharpen the spectral peaks, and the background spectral intensity estimation basis is subtracted to obtain the enhanced three-dimensional spectrum.

[0058] A three-dimensional gain tensor is constructed based on the spectral region of interest, and the initial three-dimensional spectrum is converted into normalized spectral intensity. For spectral units falling within the spectral region of interest, a bounded nonlinear gain is set according to the geometric dimensions of the obstacle candidate region; for spectral units not falling within the spectral region of interest, a constant gain is set, thereby modulating the spectral intensity and sharpening the spectral peaks of the initial three-dimensional spectrum. Subsequently, a preset background spectral intensity estimation algorithm is used to estimate the background spectral intensity basis, and the background spectral intensity basis is subtracted from the sharpened spectral intensity matrix. Spectral intensities below zero are truncated to obtain the enhanced three-dimensional spectral matrix. The preset background spectral intensity estimation algorithm includes a constant false alarm rate estimation method based on a local sliding window or a morphological opening operation method based on three-dimensional structuring elements.

[0059] In some embodiments, the step of applying a nonlinear function based on the geometric dimensions of the obstacle candidate region to sharpen spectral peaks within the spectral region of interest of the initial three-dimensional spectrum, and subtracting the background spectral intensity estimation basis to obtain the enhanced three-dimensional spectrum, includes: Based on the geometric dimensions of the obstacle candidate region and the preset reference dimensions, the dimensionless characterization value of the obstacle candidate region is calculated, and the gain coefficient of the nonlinear power function is set based on the dimensionless characterization value. Nonlinear amplification is performed on the intensity of the three-dimensional spectrum falling within the said region of interest. For spectral cells that do not fall within any of the aforementioned regions of interest, linear scaling is performed using a constant gain coefficient. The morphological opening operation of the three-dimensional structuring element is applied to the processed three-dimensional spectral intensity data to estimate the slow-varying clutter substrate as the background spectral intensity estimation substrate. The background spectral intensity estimation basis is subtracted from the sharpened and amplified spectral intensity matrix on a per-spectral-unit basis to reduce clutter interference and output an enhanced three-dimensional spectrum.

[0060] The two-dimensional projected length and width of each obstacle candidate region are obtained. The ratio of the candidate region area to a preset reference area is calculated to obtain a dimensionless size characterization value. Based on this dimensionless size characterization value, the gain coefficient of the nonlinear power function model is set. The mapping relationship between the dimensionless size characterization value and the gain coefficient is determined through historical calibration samples, experimental lookup tables, or a parameter mapping model obtained through training, and is limited to a preset range by a truncation function, for example, an upper limit threshold of 3.0. The exponent parameter of the power function is preferably set in the range of 1.2 to 1.5. Before nonlinear modulation, the spectral amplitude is converted into a nonnegative spectral intensity and locally normalized. The nonlinear modulation function can be expressed as a bounded gain mapping of the normalized spectral intensity, so that spectral peaks higher than the local background receive higher gains, while weak spectral components close to the background receive lower gains.

[0061] By using three-dimensional index matching, radar spectrum voxel units falling within the three-dimensional spectral region of interest (ROI) range obtained by transforming and expanding the candidate region of lidar are sharpened and amplified using the aforementioned nonlinear power function. For background spectrum voxels outside all ROI ranges, a constant gain coefficient and exponent equal to 1 are uniformly used to perform a linear scaling operation on the original value, without introducing additional gain.

[0062] After completing the initial modulation of the spectrum array, a three-dimensional opening operation based on grayscale morphology is applied to the three-dimensional data domain of the range-Doppler angle. Specifically, a three-dimensional ellipsoidal structural element with a preferred size of 5×5×3 is constructed, and three-dimensional erosion and dilation operations are continuously performed on the three-dimensional spectrum to smooth out small, sharp spectral peaks while preserving a large-area, slowly varying distributed ground clutter background substrate. This is used to estimate the low-frequency spatial background spectral intensity substrate, which is used to enhance background subtraction in the spectrum and is not used as a basis for radar received power or radar cross section conversion.

[0063] By subtracting the extracted background spectral intensity basis from the sharpened and amplified full 3D spectral intensity matrix at the voxel level, and then truncating negative results to zero using the ReLU function, this operation suppresses randomly distributed noise and fixed environmental clutter, outputting an enhanced 3D spectral data block.

[0064] S6. Based on the preset scattering model, the geometric dimensions of the obstacle candidate region, and the enhanced three-dimensional spectrum, extract the radar spectral peak quality factor and motion features, and align them to the Cartesian coordinate system of the bird's-eye view.

[0065] Based on the geometric dimensions, geometric orientation angle, and prior probability of the target category of the candidate obstacle region, a preset scattering model or equivalent scattering parameter table is invoked to calculate the expected value of the radar cross section. The candidate spectral peak positions are determined using the enhanced three-dimensional spectrum, and the normalized convexity is calculated. The spectral peak power characterization value corresponding to the candidate spectral peak position is read from the linear calibration power spectrum without nonlinear gain modulation, and combined with radar system calibration parameters and range compensation factors, the spectral peak power characterization value is converted into an equivalent radar cross section measurement value. A physical consistency metric is generated based on the matching relationship between the equivalent radar cross section measurement value and the expected radar cross section value, and fused with the normalized convexity to generate a spectral peak quality factor. The Doppler frequency shift corresponding to the candidate spectral peak position is extracted, converted into a radial velocity component, and aligned to the Cartesian coordinate system of the bird's-eye view based on the external parameter transformation relationship of radar range, azimuth angle, and radar-to-vehicle coordinate system, forming radar observation features.

[0066] In some embodiments, the step of extracting radar spectral peak quality factors and motion features based on a preset scattering model, the geometric dimensions of the obstacle candidate region, and the enhanced three-dimensional spectrum, and aligning them to the Cartesian coordinate system of the bird's-eye view, includes: Substitute the geometric dimensions corresponding to the candidate obstacle region into the far-field electromagnetic scattering equation corresponding to the candidate target category determined based on the prior probability of the target category, and calculate the theoretical expected value of the radar cross section. Local maxima are searched within the enhanced three-dimensional spectrum. These local maxima are used as candidate spectral peak positions. The corresponding Doppler velocities are extracted, and the ratio of the spectral peak to the mean of the local neighborhood is calculated and then normalized to obtain the normalized convexity. In the linear calibration power spectrum without nonlinear gain modulation, the peak power characterization value corresponding to the candidate peak position is read, and based on the radar system calibration parameters and range compensation factor, the peak power characterization value is converted into the equivalent radar cross section measurement value. A physical consistency metric is generated based on the matching relationship between the measured equivalent radar cross section and the expected radar cross section. The physical consistency metric and the normalized convexity are weighted and summed to generate the spectral peak quality factor. The extracted Doppler velocity is used as the motion feature. The spectral peak quality factor and the motion feature are then aligned to the Cartesian coordinate system of the bird's-eye view.

[0067] The geometric dimensions corresponding to the obstacle candidate region are substituted into the far-field electromagnetic scattering equation corresponding to the candidate target category determined based on the prior probability of the target category to calculate the theoretical expected value of the radar cross section. Specifically, this includes: selecting the foreground category with the highest probability value as the candidate target category according to the prior probability of the target category; obtaining the length, width, height, and geometric orientation angle of the obstacle candidate region, and looking up the corresponding scattering correction coefficient from a pre-established equivalent scattering parameter table according to the millimeter-wave radar operating wavelength, target category, and the geometric orientation angle; converting the length, width, and height into the corresponding equivalent scattering area, and correcting the equivalent scattering area using the scattering correction coefficient to obtain the expected value of the radar cross section.

[0068] A three-dimensional local maximum detection (LoM) or three-dimensional constant false alarm rate (CFAR) algorithm is employed to traverse and search for local maxima points in the enhanced three-dimensional spectral data block whose amplitudes exceed those within a 3×3×3 sliding window neighborhood. These LoM points are then used as candidate spectral peak positions, and their corresponding Doppler velocity scalars are extracted. For each extracted maximum point, the ratio of its amplitude to the mean of non-peak values ​​within a surrounding 5×5×5 region is calculated as a convexity index representing the sharpness of the signal. This convexity index is then normalized to obtain the normalized convexity.

[0069] After determining the candidate spectral peak position, the peak power characterization value corresponding to that position is read from the linear calibration power spectrum without nonlinear gain modulation. Based on the radar equations, and considering the radar system calibration parameters, transmit power, antenna gain, operating wavelength, and target radial range compensation term, the peak power characterization value is converted into an equivalent radar cross section (RCS) measurement. Subsequently, the equivalent RCS measurement value and the expected RCS value are uniformly converted to the logarithmic domain, and the difference between them is calculated. This difference is then input into an exponential decay function to obtain a physical consistency metric with values ​​ranging from 0 to 1. The exponential decay function includes a scale parameter and a lower cutoff value to reduce the impact of extremely small or anomalous measurements on the physical consistency metric.

[0070] By setting fixed weighting coefficients such as 0.6 and 0.4, the physical consistency metric and the normalized convexity index are linearly weighted and fused to generate a spectral peak quality factor with a value in the range of 0 to 1, which is used to characterize the comprehensive matching degree between the radar spectral peak and the physical scattering characteristics and local sharpness of the candidate target.

[0071] The quality factors of each spectral peak and the Doppler radial velocity, which are motion characteristics, are used in conjunction with the radar's range and azimuth parameters, along with the vehicle's external parameter attitude transformation matrix (i.e., rotation and translation mapping), and aligned to a 0.1m × 0.1m resolution bird's-eye view of the vehicle center through a polar coordinate to Cartesian coordinate system transformation. Figure 2 In the corresponding raster node of the Vicarette coordinate system.

[0072] S7. Construct a Bayesian evidence matrix in each grid of the Cartesian coordinate system of the bird's-eye view. The elements of the Bayesian evidence matrix include the spectral peak quality factor, the motion feature, the boundary confidence, and the prior probability of the target category.

[0073] Initialize a multi-channel two-dimensional grid array as a Bayesian evidence matrix, whose spatial resolution and size are consistent with the environmental raster map.

[0074] At each discrete grid cell position in the matrix, multiple sensor observation features aligned to that spatial position are assigned values: the first channel is filled with a normalized spectral peak quality factor, representing the radar target detection confidence; the second channel is filled with radial velocity, representing the line-of-sight motion attribute; the third channel is filled with boundary confidence calculated based on the lidar density gradient, representing the shape and contour clarity; the fourth to Mth channels are filled with the target category prior probability distribution calculated by a deep learning network, used to represent visual semantic information; where M is determined according to the preset number of target categories.

[0075] For idle grid cells with missing features, assign zero values ​​or default background probability values ​​corresponding to the features to obtain a feature tensor matrix that incorporates evidence from multiple sources.

[0076] S8, perform posterior probability maximization inference on the Bayesian evidence matrix to obtain the obstacle occupancy state, and fuse the estimated motion state and geometric properties of the obstacle.

[0077] A grid-level probabilistic graphical model is constructed based on Markov random fields. Each grid in the Bayesian evidence matrix is ​​used as a graph node. A negative logarithmic posterior energy function is constructed based on sensor observation characteristics, prior probability of grid occupancy, and spatial smoothing constraints between adjacent grids. The negative logarithmic posterior energy function is globally minimized using a graph cut optimization algorithm to obtain the occupancy status label of each grid. For continuous grid regions determined to be occupied, the radial velocity, boundary confidence, and candidate region geometry within the region are fused to output the obstacle's motion state, length, width, and heading angle parameters. The heading angle parameters are obtained from the geometric cluster orientation angle obtained during the lidar point cloud clustering stage, the candidate region boundary fitting results, or the trajectory direction estimation of consecutive frames.

[0078] In some embodiments, performing posterior probability maximization inference on the Bayesian evidence matrix to obtain the obstacle occupancy state and fusing the estimated obstacle motion state and geometric properties includes: Establish spatial smoothing constraints between adjacent grid states using Markov random fields; Based on the spectral peak quality factor, the boundary confidence, and the prior probability of grid occupancy determined by the prior probability of the target category, a Bayesian joint posterior probability equation is constructed. The occupancy status identifier of each grid cell is obtained by minimizing the negative log-posterior energy function corresponding to the Bayesian joint posterior probability equation. When the occupancy status indicator indicates occupancy, the motion state and multi-dimensional attribute state of the obstacle are output by combining the vehicle speed information, the motion characteristics, and the geometric dimensions of the obstacle candidate region.

[0079] A pairwise Markov random field is set on a Cartesian 2D grid layer of a bird's-eye view with a resolution of 0.1m × 0.1m. For adjacent grids in the 8-neighborhood system, a Potts energy model is set as a spatial smoothing constraint to penalize state transitions. When adjacent grids occupy inconsistent states, positive energy penalties with coefficients such as 0.5 to 1.2 are used to help maintain the connectivity and smoothness of the internal states of the same obstacle's physical extension.

[0080] Within this framework, the radar spectral peak quality factor and lidar boundary confidence, which are aligned to the corresponding grid and range from 0 to 1, are extracted and used as millimeter-wave radar observation features and lidar observation features, respectively. These are then input into a preset conditional likelihood model to generate corresponding conditional likelihood values. Simultaneously, the foreground occupancy prior probability is calculated based on the normalized target category prior probability generated by the image module. For example, the occupancy prior probability is calculated by subtracting the background category probability from 1.

[0081] In one specific implementation, the preset conditional likelihood model is a Gaussian conditional likelihood model. During the offline calibration phase, the mean and variance of the spectral peak quality factor and boundary confidence are calculated for both occupied and unoccupied states to obtain the occupied state parameter set and the unoccupied state parameter set. During the online inference phase, the spectral peak quality factor and boundary confidence of the current grid are substituted into the Gaussian probability density functions corresponding to the occupied and unoccupied state parameter sets, respectively, to obtain the conditional likelihood values ​​for the millimeter-wave radar and the lidar.

[0082] Using Bayes' theorem, the negative logarithmic expressions corresponding to each conditional likelihood value are treated as observed data terms and weighted together with a smoothing penalty term derived from Markov random fields to construct a negative logarithmic posterior energy function representing the global grid state network. Then, the maximum flow minimum cut graph cut optimization algorithm is invoked to transform probabilistic inference into edge cutting solutions for the flow network.

[0083] Through iterative solutions, the algorithm searches for the set of cut edges that minimize or reduce the total energy equation, effectively maximizing the joint posterior probability. It outputs a binary state label for each grid cell, where 0 represents background and 1 represents occupied state. In this implementation, graph cut optimization is primarily used for occupancy state inference. For continuous local grid regions determined to be stably occupied, attribute estimation is performed by fusing motion features, boundary confidence, and candidate region geometry within the corresponding grid cells. For continuous local grid regions determined to be stably occupied, the vehicle's 3D velocity and heading angle parameters from the inertial navigation system are acquired in real-time.

[0084] By subtracting or compensating for the vehicle's own motion vector components through vector geometric projection and superimposing the radar Doppler relative motion characteristics within the grid, the absolute radial velocity component of the obstacle along the radar line of sight can be calculated.

[0085] Based on the spatial index, the geometric dimensions or bounding box physical dimensions of the obstacle candidate region extracted in advance by the LiDAR are retrieved in reverse and fused. The output is a multi-dimensional attribute state environment perception result containing the target outline, category classification, absolute radial velocity component along the radar line of sight, and three-dimensional geometric dimensions.

[0086] The ablation experiment used real-world road data collected by an autonomous driving test vehicle equipped with millimeter-wave radar, lidar, and cameras to construct a multimodal sensor test set encompassing urban congestion and rainy / foggy weather. Three comparison conditions were set up: the baseline condition used a traditional perception algorithm employing uniform sampling Fast Fourier Transform (FST) and conventional target-level decision fusion; ablation condition one, building upon the baseline, utilized non-uniform FFT and a spatial region of interest nonlinear spectral peak sharpening module; and ablation condition two, the complete solution, added a Markov random field and Bayesian joint posterior probability global optimization fusion module to condition one.

[0087] The average obstacle detection accuracy under baseline conditions was 75.2%, the average radial velocity estimation error was 0.45 m / s, and the false alarm rate was 8.5%. Using the radar signal optimization module, ablation condition one improved the average accuracy to 82.4%, reduced the radial velocity estimation error to 0.21 m / s, and lowered the false alarm rate to 4.2%. Ablation condition two, employing the complete solution, achieved superior performance, with an average accuracy of 88.6%, a velocity estimation error reduced to 0.18 m / s, a false alarm rate reduced to 1.8%, and a comprehensive success rate of 95% for multi-dimensional attribute estimation in the scene. The comparison results of the average radial velocity estimation error of the three experimental schemes are as follows: Figure 3 As shown, the velocity estimation accuracy increases with the progressive stacking of non-uniform Fourier transform, spectral peak sharpening, and Bayesian global fusion modules.

[0088] Ablation condition one improves velocity error and false alarm rate compared to the baseline because the non-uniform fast Fourier transform solves the spectral leakage caused by timestamp frequency hopping. Simultaneously, peak sharpening and background subtraction based on the lidar region of interest enhance the signal-to-noise ratio of the real target and suppress environmental clutter. The overall performance improvement of ablation condition two is mainly due to the fusion of visual semantic prior probabilities, lidar quality factors, and lidar boundary confidence under a bird's-eye view. Furthermore, the spatial smoothing constraint term provided by the Markov random field, combined with a graph cut optimization algorithm, filters out discrete noise and maintains the connectivity of the target's physical boundaries, thereby improving the stability of obstacle joint state estimation in complex environments.

[0089] Example 2 Embodiment 2 of the present invention proposes an obstacle detection system for unmanned vehicles based on multi-sensor fusion, comprising the following modules: The acquisition module is used to acquire non-uniform echo signals from millimeter-wave radar, lidar point clouds, and camera images; perform non-uniform Fourier transform on the non-uniform echo signals in the fast time dimension, and perform Fourier transform on the slow time dimension and array dimension to generate an initial three-dimensional spectrum; perform clustering processing on the lidar point clouds to generate obstacle candidate regions and boundary confidence scores, and convert the obstacle candidate regions into spectral interest regions that match the initial three-dimensional spectrum; perform semantic segmentation and inverse perspective transformation on the camera images to obtain the prior probability of target categories under a bird's-eye view. The calculation module is used to sharpen the spectral peaks within the region of interest of the initial three-dimensional spectrum by applying a nonlinear function based on the geometric dimensions of the obstacle candidate region, and subtracting the background spectral intensity estimation basis to obtain the enhanced three-dimensional spectrum; based on the preset scattering model, the geometric dimensions of the obstacle candidate region, and the enhanced three-dimensional spectrum, the radar spectral peak quality factor and motion features are extracted and aligned to the Cartesian coordinate system of the bird's-eye view. An estimation module is used to construct a Bayesian evidence matrix in each grid of the Cartesian coordinate system of the bird's-eye view. The elements of the Bayesian evidence matrix include the spectral peak quality factor, the motion feature, the boundary confidence, and the prior probability of the target category. Posterior probability maximization inference is performed on the Bayesian evidence matrix to obtain the obstacle occupancy state, and the motion state and geometric attributes of the estimated obstacle are fused together.

[0090] In some embodiments, acquiring the non-uniform echo signal from the millimeter-wave radar, the point cloud from the lidar, and the camera image includes: The vehicle-mounted synchronization trigger controller sends global timestamp synchronization pulses to the millimeter-wave radar, lidar, and camera at a fixed frequency. Record the non-uniform echo signal with time-stamped frequency hopping interval output from the intermediate frequency receiving link of the millimeter-wave radar; Receive the lidar point cloud in a three-dimensional floating-point coordinate system calculated by the lidar; It acquires a time-synchronized, distortion-corrected color pixel matrix image output from the camera.

[0091] In some embodiments, performing a non-uniform Fourier transform on the non-uniform echo signal in the fast time dimension and performing Fourier transforms in the slow time dimension and the array dimension to generate an initial three-dimensional spectrum includes: The non-uniform fast Fourier transform algorithm is used to perform distance dimension resampling and frequency domain mapping on the fast time sampling points of the non-uniform echo signal with time offset, and the distance dimension spectrum is extracted. Perform a discrete Fourier transform on the echo phase difference of the same distance cell in the slow time dimension to obtain the Doppler velocity spectrum; A spatial discrete Fourier transform is performed on the spatial phase difference of the multi-channel receiving antenna in the array dimension to calculate the azimuth angle of the target and synthesize an initial three-dimensional spectrum containing range, Doppler and angle information.

[0092] In some embodiments, the step of clustering the lidar point cloud to generate obstacle candidate regions and boundary confidence scores, and converting the obstacle candidate regions into spectral interest regions that match the initial three-dimensional spectrum, includes: The lidar point cloud is divided into a three-dimensional voxel grid, and discrete noise points with a point cloud density lower than a preset threshold are filtered out. The connected elements are aggregated into independent obstacle geometry clusters using a density clustering algorithm, and the three-dimensional bounding box size of the obstacle geometry clusters is calculated. Calculate the variance of the distribution density of the point cloud at the edge of the obstacle geometric cluster, and generate a boundary confidence level between 0 and 1 based on the variance through negative correlation mapping; Extract the two-dimensional orthogonal projection of the three-dimensional bounding box onto the horizontal plane as a spatial region of interest, and transform and align the coordinate system to the unified bird's-eye view coordinate system of the vehicle; Based on the extrinsic parameter calibration relationship between radar, lidar and vehicle unified coordinate system, the spatial region of interest is converted into a two-dimensional spectral region of interest in the radar range-angle coordinate domain, and the two-dimensional spectral region of interest is extended along the Doppler dimension into a three-dimensional spectral region of interest.

[0093] In some embodiments, the step of applying a nonlinear function based on the geometric dimensions of the obstacle candidate region to sharpen spectral peaks within the spectral region of interest of the initial three-dimensional spectrum, and subtracting the background spectral intensity estimation basis to obtain the enhanced three-dimensional spectrum, includes: Based on the geometric dimensions of the obstacle candidate region and the preset reference dimensions, the dimensionless characterization value of the obstacle candidate region is calculated, and the gain coefficient of the nonlinear power function is set based on the dimensionless characterization value. Nonlinear amplification is performed on the intensity of the three-dimensional spectrum falling within the said region of interest. For spectral cells that do not fall within any of the aforementioned regions of interest, linear scaling is performed using a constant gain coefficient. The morphological opening operation of the three-dimensional structuring element is applied to the processed three-dimensional spectral intensity data to estimate the slow-varying clutter substrate as the background spectral intensity estimation substrate. The background spectral intensity estimation basis is subtracted from the sharpened and amplified spectral intensity matrix on a per-spectral-unit basis to reduce clutter interference and output an enhanced three-dimensional spectrum.

[0094] In some embodiments, the step of extracting radar spectral peak quality factors and motion features based on a preset scattering model, the geometric dimensions of the obstacle candidate region, and the enhanced three-dimensional spectrum, and aligning them to the Cartesian coordinate system of the bird's-eye view, includes: Substitute the geometric dimensions corresponding to the candidate obstacle region into the far-field electromagnetic scattering equation corresponding to the candidate target category determined based on the prior probability of the target category, and calculate the theoretical expected value of the radar cross section. Local maxima are searched within the enhanced three-dimensional spectrum. These local maxima are used as candidate spectral peak positions. The corresponding Doppler velocities are extracted, and the ratio of the spectral peak to the mean of the local neighborhood is calculated and then normalized to obtain the normalized convexity. In the linear calibration power spectrum without nonlinear gain modulation, the peak power characterization value corresponding to the candidate peak position is read, and based on the radar system calibration parameters and range compensation factor, the peak power characterization value is converted into the equivalent radar cross section measurement value. A physical consistency metric is generated based on the matching relationship between the measured equivalent radar cross section and the expected radar cross section. The physical consistency metric and the normalized convexity are weighted and summed to generate the spectral peak quality factor. The extracted Doppler velocity is used as the motion feature. The spectral peak quality factor and the motion feature are then aligned to the Cartesian coordinate system of the bird's-eye view.

[0095] In some embodiments, performing semantic segmentation and inverse perspective transformation on the camera image to obtain the prior probability of the target category in the bird's-eye view includes: The camera image is input into a pre-trained fully convolutional neural network, which outputs a pixel-level multi-class semantic segmentation mask and a normalized probability distribution for each class. An inverse perspective mapping matrix is ​​constructed using a pre-calibrated camera intrinsic parameter matrix and an extrinsic parameter matrix relative to a unified bird's-eye view coordinate system. The inverse perspective mapping matrix is ​​used to reproject the target bottom pixels in the multi-category semantic segmentation mask and probability distribution from the two-dimensional image perspective space to the two-dimensional planar grid of the unified bird's-eye view. Bilinear interpolation is performed on the reprojected planar grid to generate the prior probability of the target category for each grid cell in the bird's-eye view.

[0096] In some embodiments, performing posterior probability maximization inference on the Bayesian evidence matrix to obtain the obstacle occupancy state and fusing the estimated obstacle motion state and geometric properties includes: Establish spatial smoothing constraints between adjacent grid states using Markov random fields; Based on the spectral peak quality factor, the boundary confidence, and the prior probability of grid occupancy determined by the prior probability of the target category, a Bayesian joint posterior probability equation is constructed. The occupancy status identifier of each grid cell is obtained by minimizing the negative log-posterior energy function corresponding to the Bayesian joint posterior probability equation. When the occupancy status indicator indicates occupancy, the motion state and multi-dimensional attribute state of the obstacle are output by combining the vehicle speed information, the motion characteristics, and the geometric dimensions of the obstacle candidate region.

[0097] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and to aid in understanding a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art, upon reading this specification, may readily identify some of the devices as separate embodiments. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. And the content of each secondary embodiment is valid even if it contains fewer than all the features of a single foregoing disclosed embodiment.

Claims

1. An obstacle detection method for unmanned vehicles based on multi-sensor fusion, characterized in that, include: Acquire non-uniform echo signals from millimeter-wave radar, point clouds from lidar, and images from cameras; The non-uniform echo signal is subjected to a non-uniform Fourier transform in the fast time dimension, and Fourier transforms are performed in the slow time dimension and the array dimension to generate an initial three-dimensional spectrum; the lidar point cloud is clustered to generate obstacle candidate regions and boundary confidence, and the obstacle candidate regions are converted into spectral interest regions that match the initial three-dimensional spectrum; semantic segmentation and inverse perspective transformation are performed on the camera image to obtain the prior probability of the target category under the bird's-eye view; Within the region of interest of the initial three-dimensional spectrum, a nonlinear function based on the geometric dimensions of the obstacle candidate region is applied for peak sharpening, and the background spectral intensity estimation basis is subtracted to obtain the enhanced three-dimensional spectrum. Based on a preset scattering model, the geometric dimensions of the obstacle candidate region, and the enhanced three-dimensional spectrum, the radar peak quality factor and motion features are extracted and aligned to the Cartesian coordinate system of the bird's-eye view, including: Substitute the geometric dimensions corresponding to the candidate obstacle region into the far-field electromagnetic scattering equation corresponding to the candidate target category determined based on the prior probability of the target category, and calculate the theoretical expected value of the radar cross section. Local maxima are searched within the enhanced three-dimensional spectrum. These local maxima are used as candidate spectral peak positions. The corresponding Doppler velocities are extracted, and the ratio of the spectral peak to the mean of the local neighborhood is calculated and then normalized to obtain the normalized convexity. In the linear calibration power spectrum without nonlinear gain modulation, the peak power characterization value corresponding to the candidate peak position is read, and based on the radar system calibration parameters and range compensation factor, the peak power characterization value is converted into the equivalent radar cross section measurement value. A physical consistency metric is generated based on the matching relationship between the measured equivalent radar cross section and the expected radar cross section. The physical consistency metric and the normalized convexity are weighted and summed to generate the spectral peak quality factor. The extracted Doppler velocity is used as the motion feature. The spectral peak quality factor and the motion feature are then aligned to the Cartesian coordinate system of the bird's-eye view. A Bayesian evidence matrix is ​​constructed in each grid of the Cartesian coordinate system of the bird's-eye view. The elements of the Bayesian evidence matrix include the spectral peak quality factor, the motion feature, the boundary confidence, and the prior probability of the target category. Posterior probability maximization inference is performed on the Bayesian evidence matrix to obtain the obstacle occupancy state, and the motion state and geometric attributes of the obstacle are estimated by fusing them.

2. The method according to claim 1, characterized in that, The acquisition of non-uniform echo signals from millimeter-wave radar, point clouds from lidar, and camera images includes: The vehicle-mounted synchronization trigger controller sends global timestamp synchronization pulses to the millimeter-wave radar, lidar, and camera at a fixed frequency. Record the non-uniform echo signal with time-stamped frequency hopping interval output from the intermediate frequency receiving link of the millimeter-wave radar; Receive the lidar point cloud in a three-dimensional floating-point coordinate system calculated by the lidar; It acquires a time-synchronized, distortion-corrected color pixel matrix image output from the camera.

3. The method according to claim 1, characterized in that, The step of performing a non-uniform Fourier transform on the non-uniform echo signal in the fast time dimension, and performing Fourier transforms in the slow time dimension and the array dimension to generate an initial three-dimensional spectrum includes: The non-uniform fast Fourier transform algorithm is used to perform distance dimension resampling and frequency domain mapping on the fast time sampling points of the non-uniform echo signal with time offset, and the distance dimension spectrum is extracted. Perform a discrete Fourier transform on the echo phase difference of the same distance cell in the slow time dimension to obtain the Doppler velocity spectrum; A spatial discrete Fourier transform is performed on the spatial phase difference of the multi-channel receiving antenna in the array dimension to calculate the azimuth angle of the target and synthesize an initial three-dimensional spectrum containing range, Doppler and angle information.

4. The method according to claim 2 or 3, characterized in that, The step of clustering the lidar point cloud to generate obstacle candidate regions and boundary confidence scores, and converting the obstacle candidate regions into spectral interest regions that match the initial three-dimensional spectrum, includes: The lidar point cloud is divided into a three-dimensional voxel grid, and discrete noise points with a point cloud density lower than a preset threshold are filtered out. The connected elements are aggregated into independent obstacle geometry clusters using a density clustering algorithm, and the three-dimensional bounding box size of the obstacle geometry clusters is calculated. Calculate the variance of the distribution density of the point cloud at the edge of the obstacle geometric cluster, and generate a boundary confidence level between 0 and 1 based on the variance through negative correlation mapping; Extract the two-dimensional orthogonal projection of the three-dimensional bounding box onto the horizontal plane as a spatial region of interest, and transform and align the coordinate system to the unified bird's-eye view coordinate system of the vehicle; Based on the extrinsic parameter calibration relationship between radar, lidar and vehicle unified coordinate system, the spatial region of interest is converted into a two-dimensional spectral region of interest in the radar range-angle coordinate domain, and the two-dimensional spectral region of interest is extended along the Doppler dimension into a three-dimensional spectral region of interest.

5. The method according to claim 1, characterized in that, Within the region of interest of the initial three-dimensional spectrum, a nonlinear function based on the geometric dimensions of the obstacle candidate region is applied for peak sharpening, and the enhanced three-dimensional spectrum is obtained by subtracting the background spectral intensity estimation basis, including: Based on the geometric dimensions of the obstacle candidate region and the preset reference dimensions, the dimensionless characterization value of the obstacle candidate region is calculated, and the gain coefficient of the nonlinear power function is set based on the dimensionless characterization value. Nonlinear amplification is performed on the intensity of the three-dimensional spectrum falling within the said region of interest. For spectral cells that do not fall within any of the aforementioned regions of interest, linear scaling is performed using a constant gain coefficient. The morphological opening operation of the three-dimensional structuring element is applied to the processed three-dimensional spectral intensity data to estimate the slow-varying clutter substrate as the background spectral intensity estimation substrate. The background spectral intensity estimation basis is subtracted from the sharpened and amplified spectral intensity matrix on a per-spectral-unit basis to reduce clutter interference and output an enhanced three-dimensional spectrum.

6. The method according to claim 1, characterized in that, The step of performing semantic segmentation and inverse perspective transformation on the camera image to obtain the prior probability of the target category in the bird's-eye view includes: The camera image is input into a pre-trained fully convolutional neural network, which outputs a pixel-level multi-class semantic segmentation mask and a normalized probability distribution for each class. An inverse perspective mapping matrix is ​​constructed using a pre-calibrated camera intrinsic parameter matrix and an extrinsic parameter matrix relative to a unified bird's-eye view coordinate system. The inverse perspective mapping matrix is ​​used to reproject the target bottom pixels in the multi-category semantic segmentation mask and probability distribution from the two-dimensional image perspective space to the two-dimensional planar grid of the unified bird's-eye view. Bilinear interpolation is performed on the reprojected planar grid to generate the prior probability of the target category for each grid cell in the bird's-eye view.

7. The method according to claim 1, characterized in that, The step of performing posterior probability maximization inference on the Bayesian evidence matrix to obtain the obstacle occupancy state and fusing the estimated obstacle motion state and geometric attributes includes: Establish spatial smoothing constraints between adjacent grid states using Markov random fields; Based on the spectral peak quality factor, the boundary confidence, and the prior probability of grid occupancy determined by the prior probability of the target category, a Bayesian joint posterior probability equation is constructed. The occupancy status identifier of each grid cell is obtained by minimizing the negative log-posterior energy function corresponding to the Bayesian joint posterior probability equation. When the occupancy status indicator indicates occupancy, the motion state and multi-dimensional attribute state of the obstacle are output by combining the vehicle speed information, the motion characteristics, and the geometric dimensions of the obstacle candidate region.

8. A system for implementing the obstacle detection method for unmanned vehicles based on multi-sensor fusion as described in any one of claims 1 to 7, characterized in that, include: The acquisition module is used to acquire non-uniform echo signals from millimeter-wave radar, point clouds from lidar, and images from cameras. The non-uniform echo signal is subjected to a non-uniform Fourier transform in the fast time dimension, and Fourier transforms are performed in the slow time dimension and the array dimension to generate an initial three-dimensional spectrum; the lidar point cloud is clustered to generate obstacle candidate regions and boundary confidence, and the obstacle candidate regions are converted into spectral interest regions that match the initial three-dimensional spectrum; semantic segmentation and inverse perspective transformation are performed on the camera image to obtain the prior probability of the target category under the bird's-eye view; The calculation module is used to sharpen the spectral peaks within the region of interest of the initial three-dimensional spectrum by applying a nonlinear function based on the geometric dimensions of the obstacle candidate region, and subtracting the background spectral intensity estimation basis to obtain the enhanced three-dimensional spectrum; based on the preset scattering model, the geometric dimensions of the obstacle candidate region, and the enhanced three-dimensional spectrum, the radar spectral peak quality factor and motion features are extracted and aligned to the Cartesian coordinate system of the bird's-eye view. An estimation module is used to construct a Bayesian evidence matrix in each grid of the Cartesian coordinate system of the bird's-eye view. The elements of the Bayesian evidence matrix include the spectral peak quality factor, the motion feature, the boundary confidence, and the prior probability of the target category. Posterior probability maximization inference is performed on the Bayesian evidence matrix to obtain the obstacle occupancy state, and the motion state and geometric attributes of the estimated obstacle are fused together.

9. The system according to claim 8, characterized in that, The acquisition of non-uniform echo signals from millimeter-wave radar, point clouds from lidar, and camera images includes: The vehicle-mounted synchronization trigger controller sends global timestamp synchronization pulses to the millimeter-wave radar, lidar, and camera at a fixed frequency. Record the non-uniform echo signal with time-stamped frequency hopping interval output from the intermediate frequency receiving link of the millimeter-wave radar; Receive the lidar point cloud in a three-dimensional floating-point coordinate system calculated by the lidar; It acquires a time-synchronized, distortion-corrected color pixel matrix image output from the camera.

Images