Target detection method and device based on Leiyu fusion perception and storage medium

Abstract

The invention discloses a target detection method and device based on Leiyu fusion perception and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: inputting radar original signal data into a first target detection model for processing, and outputting first target detection information; inputting the visual image data into a second target detection model for processing, and outputting second target detection information; converting the first target detection information and the second target detection information to a unified reference coordinate system based on calibration parameters between the millimeter wave radar and the visual sensor; constructing a fusion feature vector according to the first target detection information and the second target detection information under the reference coordinate system; and inputting the fusion feature vector into a fusion decision network for decision-level fusion, and outputting a fused target detection result. According to the method, the mismatching rate in a complex scene is effectively reduced, and the accuracy and stability of a fusion result are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a target detection method, device and storage medium based on radar-visual fusion perception. Background Technology

[0002] With the rapid development of autonomous driving, intelligent transportation, and environmental perception technologies, multi-sensor fusion systems have become a key means to achieve high-precision and robust environmental perception. Currently, commonly used environmental perception sensors mainly include LiDAR, cameras, and millimeter-wave radar. While LiDAR possesses high spatial resolution and 3D modeling capabilities, it suffers from high cost, large size, and significant performance degradation in rain, fog, and strong light environments. Monocular cameras can provide rich semantic information but cannot directly acquire target distance and are sensitive to changes in lighting. In contrast, millimeter-wave radar and binocular cameras offer unique advantages in performance, cost, and functional complementarity: millimeter-wave radar can operate stably in complex weather and low-visibility conditions, directly acquiring target distance, speed, and orientation information; binocular cameras provide detailed visual semantics and spatial structure information. The effective combination of these two technologies can compensate for the limitations of a single sensor and achieve a good balance between perception accuracy, robustness, and real-time performance, thus being considered an ideal combination for multi-sensor fusion systems.

[0003] In multi-sensor fusion methods, common fusion levels include data-level, feature-level, and decision-level fusion. Data-level fusion directly aligns the original data at the signal or pixel level, theoretically preserving information to the greatest extent. However, due to significant differences between radar and camera data in terms of dimensionality, sampling rate, and noise characteristics, achieving accurate synchronization and registration is costly and computationally complex, making it difficult to meet real-time requirements. Feature-level fusion extracts sensor features separately and fuses them in an intermediate layer. While this alleviates the registration problem to some extent, the inconsistent feature spaces between radar and vision often rely on manually set weighting rules, lacking adaptability and generalization ability. In contrast, decision-level fusion performs matching and information fusion at the result level after radar and camera have completed target detection separately. It has advantages such as clear structure, low coupling, and easy expansion. Decision-level fusion not only fully utilizes the independent detection capabilities of each sensor but also integrates spatial and semantic information at the target level, making the fusion process more flexible and robust. Especially when the fusion strategy is implemented by a deep learning model, decision-level fusion can adaptively model multimodal feature relationships, thus maintaining high accuracy and robustness in complex environments.

[0004] However, existing decision-level fusion methods for radar and vision still have significant limitations. In radar target detection, most systems use traditional constant false alarm rate (CFAR) detection methods, generating point clouds by sequentially performing fixed processing steps on raw millimeter-wave radar data, including range fast Fourier transform, Doppler fast Fourier transform, CFR detection, and direction-of-arrival estimation. These methods are prone to missed detections and false alarms in low signal-to-noise ratio, strong clutter, and multipath reflection environments. Furthermore, because the detection threshold depends on manual settings, they lack adaptability and struggle to meet the real-time intelligent detection needs of complex dynamic scenes. Regarding fusion strategies, existing methods typically project the radar point cloud onto an image coordinate system and match and fuse the radar and camera detection results based on manually set spatial distance thresholds, category consistency, or confidence weighting rules. For example, patent document CN116699602A describes a scheme that uses traditional CFR detection and frequency domain analysis at the radar end to obtain the target position, then combines it with camera detection results, achieving fusion through simple distance thresholds and coordinate transformations. Although this method achieves information complementarity to a certain extent, its matching process still relies heavily on manual rules and lacks intelligent adaptability. In complex scenarios or when multiple targets overlap, it is prone to mismatch and missed match, resulting in insufficient stability of the fusion results.

[0005] Furthermore, although deep learning-based radar detection methods have gradually developed in recent years, most methods still require fixed signal processing on the raw analog-to-digital conversion data of the radar before input, such as generating range Doppler maps using fast Fourier transform. This kind of preprocessing not only introduces computational redundancy and information loss, but also limits the model's ability to learn the intrinsic feature correlation between radar signals and target attributes end-to-end, thus restricting the upper limit of model performance and generalization ability.

[0006] In summary, existing decision-level fusion methods based on traditional constant false alarm rate (CFAR) detection and manual matching strategies have not fully leveraged the potential of deep learning in cross-modal feature learning and adaptive fusion. Therefore, there is an urgent need for a millimeter-wave radar and binocular camera fusion perception scheme capable of end-to-end learning and adaptive fusion, in order to comprehensively improve the system's detection accuracy, robustness, and intelligence in complex environments. Summary of the Invention

[0007] To address the aforementioned deficiencies in existing technologies, the present invention aims to provide a target detection method, device, and storage medium based on radar-visual fusion perception, thereby solving at least one of the problems of existing decision-level fusion methods, such as reliance on manual rules, poor adaptability of radar detection algorithms, strong preprocessing dependence, and difficulty in achieving end-to-end optimization of the system.

[0008] This invention solves the above-mentioned technical problems through the following technical solution: a target detection method based on radar-visual fusion perception, comprising:

[0009] Simultaneously trigger millimeter-wave radar and vision sensor to acquire time-synchronized raw signal data from millimeter-wave radar and image data from vision sensor;

[0010] The original signal data is input into a pre-trained first target detection model for processing, and the first target detection information in the radar coordinate system is output; wherein, the first target detection model includes a signal feature extraction module, which is used to perform adaptive frequency domain feature extraction on the original signal data;

[0011] The image data is input into a pre-trained second target detection model for processing, and the second target detection information in the visual coordinate system is output.

[0012] Based on the calibration parameters between the millimeter-wave radar and the vision sensor, the first target detection information and the second target detection information are converted to a unified reference coordinate system;

[0013] Construct a fused feature vector based on the first and second target detection information in the reference coordinate system;

[0014] The fused feature vector is input into the fusion decision network for decision-level fusion, and the fused target detection result is output.

[0015] This invention employs a fusion decision network to perform decision-level fusion of constructed fusion feature vectors, replacing the traditional matching method that relies on manually setting fixed rules such as spatial distance thresholds and confidence weights. The fusion decision network can automatically learn the complex nonlinear correspondence between radar targets and visual targets from training data, thereby dynamically adapting to changes in different target types, spatial distributions, and environmental contexts. Therefore, in complex scenarios with dense, overlapping, or mutually occluded targets, the method of this invention can significantly reduce the false matching rate and the missed matching rate, greatly improving the stability, accuracy, and scene adaptability of the fusion results.

[0016] This invention employs a first target detection model, including a signal feature extraction module, to directly process the raw signal data of millimeter-wave radar, abandoning the traditional constant false alarm rate (CFAR) detection algorithm and its dependent fixed signal processing chain. The signal feature extraction module adaptively extracts deep frequency domain features related to the target from the raw signal without relying on manually preset detection thresholds. Therefore, this invention effectively overcomes the limitations of traditional CFAR detection, which is prone to missed detections and false alarms in harsh environments such as low signal-to-noise ratio, strong clutter, and multipath reflection. It significantly improves the detection capability for weak targets while suppressing false alarms, achieving more robust and accurate radar target perception.

[0017] This invention directly performs adaptive frequency domain feature extraction on the raw signal data through a signal feature extraction module, eliminating the need for fixed signal preprocessing steps such as generating range-Doppler maps using Fast Fourier Transform. This not only reduces computational redundancy and potential information loss caused by fixed preprocessing, but more importantly, it achieves end-to-end feature learning from the raw radar signal to target attributes. This approach enables the model to autonomously discover the intrinsic mapping relationship between the signal and the target, thereby fully releasing the model's performance potential, improving detection accuracy, and enhancing the model's generalization ability under different scenarios and configurations.

[0018] Furthermore, the signal feature extraction module includes a range-dimensional fast Fourier transform sub-network, a velocity-dimensional fast Fourier transform sub-network, and an azimuth-dimensional fast Fourier transform sub-network, which are cascaded in sequence.

[0019] The distance-dimensional fast Fourier transform subnetwork is used to extract frequency domain features along the fast time sampling dimension of the original signal data to obtain distance features;

[0020] The velocity-dimensional Fast Fourier Transform subnetwork is used to extract frequency domain features along the slow time frame sequence dimension of the distance feature to obtain the velocity feature;

[0021] The azimuth-dimensional fast Fourier transform subnetwork is used to extract frequency domain features along the receiving antenna channel dimension in the velocity features to obtain azimuth features.

[0022] In this invention, the signal feature extraction module employs cascaded Fast Fourier Transform (FFT) subnetworks for range, velocity, and azimuth dimensions. Feature transformations are performed sequentially along the fast-time sampling dimension, slow-time frame sequence dimension, and receiver antenna channel dimension of the original signal data. Its structure directly corresponds to the three-dimensional FFT process of range, velocity, and azimuth in traditional radar signal processing. However, unlike the fixed and unadjustable transform kernel in traditional methods, this invention implements the transformation through trainable linear layers. The weights approximate a Discrete Fourier Transform during initialization and are adaptively adjusted during training. This enables the network to learn end-to-end the nonlinear mapping from the original signal to the optimal frequency domain features for the task, thereby significantly improving feature representation capability and target detection accuracy.

[0023] By processing the three physical dimensions of distance, velocity, and angle sequentially, this module can systematically separate and fuse the multi-dimensional attribute features of the target. This structured feature extraction method not only preserves the complete physical information of the radar signal, but also strengthens the correlation and differentiation capabilities between features of different dimensions through the nonlinear transformation of the neural network. This effectively improves the model's ability to identify real targets in clutter, noise, and multi-target interference environments, and enhances the system's robustness in complex dynamic scenarios.

[0024] Traditional radar detection methods require preprocessing of the raw ADC data using fixed procedures such as Fast Fourier Transform (FFT). This process can lead to information loss (e.g., sidelobe leakage, resolution limitations) and cannot be optimized for specific tasks. This invention embeds the frequency domain transformation process into a trainable network module, avoiding computational redundancy and inherent errors associated with independent preprocessing steps. This enables seamless and adaptive extraction from the raw signal to high-level semantic features. This not only improves the overall efficiency of signal processing and feature extraction but also enhances the model's generalization performance across different scenarios and configurations.

[0025] The modular design of the three cascaded sub-networks clearly corresponds to the physical dimensions of radar signal processing, facilitating individual adjustment and optimization. The extracted azimuth features (including real and imaginary parts) can be directly input into the subsequent target detection backbone network through operations such as splicing, realizing a smooth integration from signal processing to target detection, and providing effective key technical support for building a high-performance, end-to-end radar perception system.

[0026] Furthermore, the distance-dimensional Fast Fourier Transform (FST) subnetwork, the velocity-dimensional FFT subnetwork, and the azimuth-dimensional FFT subnetwork all include:

[0027] The first linear layer is used to perform a linear transformation on the real part of the input signal;

[0028] The second linear layer is used to perform linear transformation on the imaginary part of the input signal;

[0029] The complex number calculation unit is used to calculate the real part and imaginary part features after frequency domain transformation based on the outputs of the first linear layer and the second linear layer.

[0030] The weight matrices of the first linear layer and the second linear layer are approximated by the cosine and sine components of the discrete Fourier transform, respectively, during initialization.

[0031] By embedding a learnable complex signal processing mechanism into a neural network, a physically interpretable and adaptive frequency domain transformation of the raw radar signal is achieved. By setting independent linear layers for the real and imaginary parts of the signal and initializing the weights with cosine and sine components of the discrete Fourier transform, the network possesses the ability to approximate the standard Fourier transform from the early stages of training. During training, the weights can be further adaptively adjusted, thereby learning a frequency domain feature mapping that is superior to a fixed transform and highly relevant to the detection task.

[0032] By explicitly processing the real and imaginary features through complex number computation units and synthesizing the final output according to complex number arithmetic rules, this structure fully preserves the phase and amplitude information of the radar signal. This provides the network with richer input features, helping to more accurately analyze the target's range, velocity, and angle information.

[0033] By modeling the complex Fourier transform process as a completely differentiable neural network, it can be jointly trained and optimized end-to-end with the subsequent target detection backbone network. This eliminates the gap between the fixed transformation module and the subsequent learning module in the traditional process, and allows gradients to propagate back through the entire network. This enables joint optimization of feature extraction and transformation processes for the final detection target, thereby improving the overall system performance.

[0034] Furthermore, the specific calculation formulas for the real and imaginary features after frequency domain transformation are as follows:

[0035] ;

[0036] ;

[0037] in, Indicates the real part feature; Indicates the characteristics of the imaginary part; This represents the weight matrix of the first linear layer; This represents the weight matrix of the second linear layer; Represents the real part of the input signal; This represents the imaginary part of the input signal.

[0038] Furthermore, the signal feature extraction module also includes a splicing module, which is used to splice the real and imaginary features of the azimuth features output by the azimuth-dimensional fast Fourier transform sub-network in the channel dimension to form a composite feature map.

[0039] In this embodiment, the transformed real and imaginary feature maps are concatenated along the channel dimension. This operation fully preserves all the information of the radar signal in the frequency domain (the real and imaginary parts together constitute a complex representation). Compared to using only the amplitude spectrum or individual real / imaginary parts, this composite feature map provides the subsequent target detection backbone network with richer features that simultaneously include the target energy distribution and phase structure, helping to improve the accuracy and robustness of detection.

[0040] Current mainstream object detection backbone networks (such as YOLOv8) typically process real tensors. The concatenation operation cleverly transforms complex features (two real tensors) into a single real tensor with double the number of channels, allowing direct input without altering the backbone network structure. This achieves a seamless and lossless connection between the complex signal processing domain and real-valued deep learning networks, ensuring efficient information transfer and system simplicity.

[0041] Furthermore, the first target detection model also includes a target detection backbone network, which is used to perform target detection on the composite feature map output by the signal feature extraction module, so as to output the target coordinates and confidence information in the radar coordinate system.

[0042] By connecting the signal feature extraction module in series with the target detection backbone network, this invention forms a complete and coherent processing chain from the input of raw radar ADC data to the final output of target position and confidence level. This achieves true end-to-end radar target detection, enabling joint optimization of all parameters from low-level signal processing to high-level target recognition, thereby maximizing the performance potential of deep learning models.

[0043] Object detection backbone networks (such as YOLOv8) are mature visual architectures pre-trained and validated on a large amount of image data, possessing powerful feature extraction and spatial localization capabilities. This invention innovatively applies this architecture to radar feature maps after frequency domain feature extraction and stitching, essentially "translating" radar information into a format that visual networks can effectively process. This leverages the technical advantages of advanced models in the vision field, ensuring a high-performance detection benchmark, while significantly reducing the development costs and risks of designing complex detection networks specifically for radar signals.

[0044] Furthermore, the first target detection information includes the radar-detected target position and confidence level, and the second target detection information includes the visually detected target position, confidence level, and category;

[0045] Based on the first and second target detection information in the reference coordinate system, a fused feature vector is constructed, including:

[0046] The spatial overlap is calculated based on the radar-detected target position and the visually detected target position in the reference coordinate system; wherein, the spatial overlap is the crossover ratio between the projection areas of the radar-detected target and the visually detected target on the horizontal two-dimensional plane of the reference coordinate system.

[0047] Based on the radar-detected target position in the reference coordinate system, calculate the normalized distance of the radar-detected target relative to the origin of the reference coordinate system;

[0048] Based on the position of the visually detected target in the reference coordinate system, calculate the normalized distance of the visually detected target relative to the origin of the reference coordinate system;

[0049] A fused feature vector is constructed based on the spatial overlap, the confidence level of the radar-detected target, the confidence level of the visually detected target, the normalized distance of the radar-detected target relative to the origin of the reference coordinate system, and the normalized distance of the visually detected target relative to the origin of the reference coordinate system.

[0050] This invention systematically integrates heterogeneous information from radar and vision sensors through a meticulously designed five-dimensional feature vector: spatial consistency (measured by cross-union ratio), individual reliability (expressed by confidence level), and target spatial distribution (described by normalized distance). This provides a comprehensive and structured input for the subsequent fusion decision network, encompassing both objective geometric relationships and the subjective judgments of each sensor, thus laying a solid foundation for achieving intelligent and adaptive fusion.

[0051] By calculating the intersection-union ratio and normalized distance, the method of this invention transforms the raw position information of different sensors at different locations and scales into a standardized metric decoupled from absolute coordinates and with a unified scale. This effectively eliminates the interference caused by differences in the dimensions and numerical ranges of the raw data on model training, enabling the neural network to learn effective fusion rules more quickly and stably, thereby improving training efficiency and model robustness.

[0052] The constructed fusion feature vectors have clear physical meanings: the intersection-union ratio directly reflects whether the targets are "in the same place," the confidence score reflects how confident each sensor is, and the normalized distance reflects how far away the targets are. This design makes the model's learning process highly consistent with the logic of human fusion judgments, which not only enhances the model's interpretability but also enables it to better understand and adapt to complex scenarios with different spatial layouts and varying sensor reliability, thus making more reasonable fusion decisions.

[0053] Furthermore, the normalized distance is obtained by dividing the Euclidean distance from the two-dimensional coordinates of the target in the reference coordinate system to the origin by a preset maximum effective sensing distance; wherein, the maximum effective sensing distance is set according to the detection performance of the millimeter-wave radar and the visual sensor and the target application scenario.

[0054] By dividing the absolute Euclidean distance of the target by a preset maximum effective sensing distance, the method of this invention normalizes the distance information of all targets to a relatively uniform numerical range (usually [0,1]). This effectively solves the problem that the large dimensions and wide range of the original distance values ​​lead to unstable neural network training and slow convergence. At the same time, this parameter is injected into the model as prior knowledge, guiding the network to focus on targets within the effective detection range, thereby improving the model's adaptability and generalization ability under different distance distribution scenarios.

[0055] The maximum effective sensing distance can be set according to sensor performance and application scenario, which makes the method of this invention highly flexible and practical. For radars or cameras with different detection ranges, or in different applications such as urban scenarios (focusing on close range) and high-speed scenarios (focusing on long range), this parameter can be adjusted to optimize the sensing focus of the model, ensuring that the method maintains optimal performance under different hardware configurations and task requirements.

[0056] The calculation method of normalized distance implicitly assumes a reasonable physical principle: targets closer to the origin are generally perceived as more reliable and their information more important; while targets approaching or exceeding the maximum effective perception distance have decreased perception reliability and should therefore be less important. Through normalization, the model can naturally learn this relationship between distance and reliability, thus more reasonably weighing the evidence of near and far targets during fusion decision-making. This avoids assigning inappropriate weights to unreliable distant targets at the edges, thereby improving the overall reliability of the fusion results.

[0057] Furthermore, the fusion decision network includes a feature encoder and at least one task prediction head connected to the output of the feature encoder;

[0058] The feature encoder is used to perform nonlinear transformation and high-order feature extraction on the input fused feature vector to obtain encoded features;

[0059] The task prediction head is used to perform at least one of the following tasks based on the features output by the feature encoder: target matching prediction, location regression prediction, and target classification prediction.

[0060] This invention abandons traditional matching and decision-making methods based on fixed thresholds or rules, and instead employs an end-to-end learning approach using a neural network consisting of a "feature encoder + task prediction head". The feature encoder automatically learns the complex relationships between heterogeneous features, while the task prediction head directly outputs high-level decisions such as matching, localization, and classification. This allows the fusion process to dynamically adjust based on input data, making more accurate and robust intelligent decisions than fixed rules in complex scenarios such as dense targets, occlusion, or conflicting sensor confidence levels.

[0061] The fusion decision network employs a modular architecture with a shared feature encoder followed by multiple independent task prediction heads. This design allows multiple related tasks (matching, location regression, classification) to be completed in parallel within a single forward propagation, sharing the computation of underlying feature extraction and significantly improving inference efficiency. Simultaneously, multiple tasks are jointly trained through the shared encoder, and the supervision signals from different tasks can mutually reinforce each other, guiding the feature encoder to learn a more discriminative and generalizable fusion feature representation, thereby optimizing the overall perceptual accuracy of the system.

[0062] The three outputs of the fusion decision network (matching probability, position offset, and category distribution) constitute a complete and actionable description of the environmental target: the matching probability solves the data association problem of "whether the radar and camera see the same object"; the position offset provides more accurate positioning optimization based on the original sensor observations; and the category distribution provides semantic information about the target. These structured outputs can be directly used in the planning and control modules of autonomous driving, realizing an end-to-end perception closed loop from raw sensor signals to high-level application commands.

[0063] Furthermore, the feature encoder includes a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence; an activation function layer and a dropout regularization layer are respectively provided after the first fully connected layer and the second fully connected layer;

[0064] The first fully connected layer is used to transform the input fused feature vector from a first dimension to a second dimension;

[0065] The second fully connected layer is used to transform the features of the second dimension to the third dimension;

[0066] The third fully connected layer is used to transform the features of the third dimension to the fourth dimension and output them.

[0067] By employing a stacked structure of multiple fully connected layers, the feature encoder of this invention can perform multiple complex nonlinear transformations on the input fused feature vector. This enables it to learn and extract high-order abstract features far exceeding those of the original input (such as simple intersection-over-union ratio, confidence, and distance), thereby more deeply uncovering the implicit and complex matching and correlation patterns between radar and visual targets, and providing strong feature support for subsequent accurate decision-making.

[0068] Fully connected layers are the most fundamental and mature components in neural networks, boasting high efficiency in both forward and backward propagation, and have been optimized to the extreme in modern deep learning frameworks. Using this standardized stacked structure to construct the feature encoder gives the network advantages such as stable training, good convergence, and controllable computational overhead. This ensures that the fusion decision network, while pursuing high performance, can meet the stringent requirements of computational efficiency and stability for real-time systems such as autonomous driving.

[0069] Setting activation function layers (such as ReLU) between adjacent fully connected layers is key to injecting nonlinear capabilities into the network. This enables the feature encoder to fit the complex nonlinear mapping between input features and high-level fusion concepts, thereby significantly enhancing the model's ability to distinguish difficult examples (such as different targets that are spatially close) and improving the model's generalization performance in unseen new scenes and target distributions, avoiding the limitations of simple linear decision-making.

[0070] Furthermore, the task prediction head includes a matching head, which comprises a fourth fully connected layer, a fifth fully connected layer, and a sigmoid activation function layer connected in sequence, for outputting the matching probability between the radar-detected target and the visually detected target.

[0071] The matching head uses a structure of a fully connected layer and a sigmoid activation function layer to map the encoded abstract features into a scalar value between 0 and 1. This value directly and clearly represents the probability that "the radar and visually detected targets belong to the same object," providing a continuous, differentiable, and statistically significant confidence input for subsequent target tracking, trajectory fusion, and other modules, replacing the traditional hard matching rule of either 0 or 1.

[0072] The matching head consists of only a fully connected layer and an activation function, resulting in an exceptionally simple structure with minimal parameters and computational cost. This ensures that after complex feature encoding, the final matching judgment can be completed with extremely low latency, making it highly suitable for real-time applications such as autonomous driving and guaranteeing the efficient operation of the entire fusion system.

[0073] Furthermore, the task prediction head includes a regression head, which includes a sixth fully connected layer and a seventh fully connected layer connected in sequence, used to output the position offset of the fused target detection center relative to the radar detection center.

[0074] The regression head learns and outputs a two-dimensional position offset directly through a simple fully connected layer. This offset is designed to finely correct the original position detected by radar, in order to fuse supplementary spatial information provided by sensors such as vision. This allows the final fused target position to surpass the positioning accuracy limit of any single sensor, especially in the lateral position of the target or the positioning of stationary / low-speed targets, significantly improving the overall positioning accuracy of the perception system.

[0075] The regression head consists of only fully connected layers, making it one of the lightest components in a neural network. This minimalist design means that the added parameters and computational cost are negligible while introducing key localization optimization features. This perfectly meets the stringent efficiency requirements of embedded platforms or real-time systems, ensuring that improvements in localization performance do not come at the expense of system speed.

[0076] Due to factors such as installation location, time synchronization, and calibration errors, radar and vision sensors may exhibit inherent systematic biases in their observed positions, even for the same target. The regression head can automatically learn these cross-modal systematic bias patterns from large amounts of data and provide adaptive compensation. This reduces the absolute dependence on sensor calibration accuracy to some extent, enhancing the system's fault tolerance and robustness in practical deployments.

[0077] Furthermore, the task prediction head includes a classification head, which comprises an eighth fully connected layer, a ninth fully connected layer, and a normalized exponential activation function layer connected in sequence, for outputting the confidence distribution of the fused target category.

[0078] The classification head receives fused information encoded by both radar and visual features and outputs a confidence distribution for all possible categories. This allows the final target category determination to integrate the rich semantics of vision with the stable presence evidence from radar. Especially when visual perception is limited (e.g., in low light or with partial occlusion), the auxiliary information provided by radar can effectively support or correct classification decisions, thereby improving the overall accuracy of target recognition and its reliability under adverse conditions.

[0079] The classification head, using a structure of fully connected layers and a normalized exponential (Softmax) activation function layer, outputs a standard probability distribution where the sum of the probabilities of all classes is 1. This output not only provides the most probable class but also quantifies the model's confidence in that judgment (the distribution of probabilities for each class), offering crucial uncertainty information to downstream planning and control modules and supporting risk-aware decision-making. Furthermore, the standardized probabilities facilitate weighted fusion or cascaded judgments with the outputs of other classifiers in the system (such as pure visual classifiers).

[0080] Based on the same concept, the present invention also provides an electronic device, including a memory, a processor, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the target detection method based on radar-visual fusion perception as described above.

[0081] Based on the same concept, the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implements the target detection method based on radar-visual fusion perception as described above.

[0082] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0083] This invention achieves adaptive decision-level fusion through a fusion decision network, replacing the traditional matching method that relies on manual rules. This effectively reduces the false matching rate in complex scenarios and improves the accuracy and stability of the fusion results. It employs a target detection model with a signal feature extraction module to directly process the raw signal data, overcoming the problems of high false alarm rates and missed detections in environments with low signal-to-noise ratios and strong clutter, as seen in traditional constant false alarm rate (CFAR) detection algorithms. This results in more robust and accurate radar target detection. Furthermore, it eliminates the need for fixed signal preprocessing steps, directly and adaptively extracting frequency domain features from the raw signal, avoiding information loss and computational redundancy, and improving the model's detection accuracy and generalization ability.

[0084] This invention integrates radar detection, visual detection, coordinate unification, and decision fusion into a complete deep learning-driven process, enabling collaborative training and end-to-end optimization across all stages. This overcomes the performance bottlenecks and error accumulation problems caused by traditional modular independent designs, thereby further improving the overall accuracy, real-time performance, and environmental robustness of target detection at the system level, better meeting the needs for reliable environmental perception in highly dynamic and complex scenarios such as autonomous driving and intelligent transportation. Attached Figure Description

[0085] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is a flowchart of the target detection method based on radar-visual fusion perception in an embodiment of the present invention;

[0087] Figure 2 This is a diagram of the architecture of the first target detection model in an embodiment of the present invention;

[0088] Figure 3 This is a diagram of the Fast Fourier Transform subnetwork architecture in an embodiment of the present invention;

[0089] Figure 4 This is a diagram of the fusion decision network architecture in an embodiment of the present invention. Detailed Implementation

[0090] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0092] Example 1

[0093] like Figure 1 As shown, the target detection method based on radar-visual fusion perception provided in this embodiment of the invention includes the following steps:

[0094] Step S1: Synchronously trigger the millimeter-wave radar and the vision sensor to acquire time-synchronized raw signal data from the millimeter-wave radar and image data from the vision sensor.

[0095] To achieve precise time synchronization between millimeter-wave radar and visual sensors (such as binocular cameras), step S1 aims to solve the time alignment problem, the core of which is to ensure that data frames from different sensors correspond to the same physical moment. Specifically, one or more of the following schemes can be combined:

[0096] The first approach involves deploying and running a precise time protocol or network time protocol service on the data processing unit (such as an industrial computer or embedded platform) that connects the millimeter-wave radar and vision sensors. This ensures that the system clock is synchronized with a high-precision reference clock (such as a GPS clock source). The aim of this approach is to control the local system clock deviations relied upon by different sensor data acquisitions to within milliseconds, providing a unified time reference for data correlation.

[0097] The second approach involves the central processing unit sending a start acquisition command to the millimeter-wave radar and the vision sensor almost simultaneously via software instructions. Due to the non-deterministic delays in operating system scheduling, bus transmission, and sensor response, the synchronization accuracy of this method is typically lower than that of the hardware-triggered approach, but it is relatively simpler to implement.

[0098] Regardless of the synchronization mechanism used, after each frame of sensor data (raw radar signal data packet, image data packet) is acquired, the central processing unit immediately assigns it a high-precision system timestamp. In subsequent data processing pipelines, the radar frame and image frame with the closest timestamps are matched to achieve time pairing of the sensor data streams. This method effectively compensates for the minor asynchronicities introduced by internal sensor processing and data readout.

[0099] Step S2: Input the raw signal data collected in step S1 into the pre-trained first target detection model for processing, and output the first target detection information in the radar coordinate system.

[0100] Step S2 aims to utilize a pre-trained first target detection model to perform end-to-end processing on the time-synchronized raw radar signal data acquired in step S1, directly outputting the first target detection information in the radar coordinate system. Its implementation includes two stages: model pre-training and online inference.

[0101] The pre-training of the first target detection model was done offline, aiming to allow the network to learn the mapping relationship from raw radar signals to target location and category. The specific steps are as follows:

[0102] Step A1: Construction of the training dataset.

[0103] For the radar portion, each training sample is a segment of preprocessed raw radar ADC data. Its corresponding labels are obtained as follows: traditional signal processing procedures (3D FFT, CFAR, etc.) are performed on this segment of raw ADC data to generate a range-azimuth intensity map; subsequently, the target bounding box and its category are labeled on the range-azimuth intensity map. This constructs a dataset that uses preprocessed raw radar ADC data as input and target bounding boxes in the range-azimuth image coordinate system as supervised labels. These labels can be obtained through one or more of the following methods:

[0104] Manual annotation: The annotator directly outlines the target on the distance-azimuth intensity map and labels it with the category.

[0105] Semi-automatic annotation: Utilizing spatiotemporally synchronized and already annotated visual image data (such as target bounding boxes in images), combined with precise calibration extrinsic parameters between radar and camera, the visually detected target is projected onto the radar's range-azimuth image coordinate system. After filtering and verification, labels are generated for the radar data. This method can efficiently obtain large amounts of radar annotation data.

[0106] Automatic simulation generation: Simultaneously generate raw radar ADC data and corresponding precise target positions and category labels in a simulation environment.

[0107] Step A2: Model Architecture and Initialization.

[0108] Raw radar ADC data typically contains three main dimensions: fast time sampling dimension, slow time frame sequence dimension, and receive antenna channel dimension. In traditional radar signal processing, fast Fourier transforms are usually performed along these three dimensions to extract the target's frequency domain features in terms of range, Doppler velocity, and azimuth. Peak analysis is then used to achieve spatial positioning and motion parameter analysis of the target.

[0109] To introduce the aforementioned signal processing mechanism into a deep learning network model and realize an end-to-end millimeter-wave radar target detection network, this invention designs a signal feature extraction module consisting of three layers of Fast Fourier Transform (FFT) sub-networks, corresponding to the three dimensions of range FFT, Doppler FFT, and azimuth FFT, respectively. The structure and computation process of each FFT sub-network are basically the same, only the data dimensions they operate on are different.

[0110] Specifically, such as Figure 2 As shown, the first target detection model includes a signal feature extraction module, a splicing module, and a target detection backbone network (e.g., the YOLOv8 model). The signal feature extraction module includes a range-dimensional fast Fourier transform sub-network, a velocity-dimensional fast Fourier transform sub-network, and an orientation-dimensional fast Fourier transform sub-network, which are cascaded in sequence.

[0111] The range-dimensional Fast Fourier Transform (FFT) subnetwork is used to extract frequency domain features along the fast time sampling dimension of the original radar ADC data to obtain range features; the velocity-dimensional FFT subnetwork is used to extract frequency domain features along the slow time frame sequence dimension of the range features to obtain velocity features; and the azimuth-dimensional FFT subnetwork is used to extract frequency domain features along the receive antenna channel dimension of the velocity features to obtain azimuth features.

[0112] like Figure 3 As shown, the distance-dimensional Fast Fourier Transform (FFT) subnetwork, the velocity-dimensional FFT subnetwork, and the azimuth-dimensional FFT subnetwork all include a first linear layer, a second linear layer, and a complex computation unit. In each subnetwork layer, the input complex signal is first separated into its real and imaginary parts, which are then fed into two independent linear layers for feature transformation. The weight matrices of these two linear layers are initialized using an approximation of the Discrete Fourier Transform (DFT) form, specifically as follows:

[0113] (1)

[0114] (2)

[0115] in, This represents the weight matrix of the first linear layer (or the real part linear layer). Represents the weight matrix The weight value at position (i,j); This represents the weight matrix of the second linear layer (or the imaginary linear layer). Represents the weight matrix The weight value at position (i,j); i represents the sampling point index of the input signal in the current processing dimension, j represents the component index of the frequency domain feature after transformation; K represents the total number of sampling points in the current processing dimension.

[0116] In the distance dimension fast Fourier transform subnetwork, K represents the total number of sampling points in the fast time dimension, i represents the sampling point index in the fast time dimension, and j represents the component index of the frequency domain feature in the distance dimension, i.e. a specific distance unit.

[0117] In the velocity dimension fast Fourier transform subnetwork, K represents the frame sequence length (number of pulses) in the slow time dimension, i represents the pulse index in the slow time dimension, and j represents the component index of the frequency domain feature in the velocity dimension, i.e. a specific velocity unit.

[0118] In the azimuth-dimensional fast Fourier transform subnetwork, K represents the total number of channels in the receiving antenna dimension, i represents the physical location index of the receiving antenna channel, and j represents the component index of the azimuth-dimensional frequency domain feature, i.e., a specific angular unit (or beam pointing).

[0119] With this initialization method, the linear layer can simulate the frequency domain mapping process of the Fourier transform in the initial state, and then further adaptively fine-tune the weights through backpropagation during the training process to achieve optimal radar signal frequency domain feature extraction.

[0120] Complex number computation unit: After linear layer mapping of the real and imaginary parts, complex linear transformation calculations are performed to obtain the real and imaginary features of the Fourier transform result. The specific formula is as follows:

[0121] (3)

[0122] (4)

[0123] in, Indicates the real part feature; Indicates the characteristics of the imaginary part; Represents the real part of the input signal; This represents the imaginary part of the input signal.

[0124] Through this structural design, the network can learn the Fourier transform features that best meet the task requirements in an end-to-end manner, realizing the adaptive extraction of frequency domain features from the radar's raw ADC data without the need for traditional manual preprocessing steps. This allows the network to automatically learn the mapping relationship between signals and target features during training, achieving true data-driven target detection.

[0125] The real and imaginary features output from the azimuth-dimensional Fast Fourier Transform sub-network are concatenated along the channel dimension by a concatenation module to form a composite feature map. This composite feature map is then input into the target detection backbone network for target classification and regression, outputting the target prediction results: target location and confidence level. The specific concatenation formula is as follows:

[0126] (5)

[0127] in, H represents the height of the composite feature map, W represents the width of the composite feature map, and C1 and C2 represent the number of channels of the real and imaginary features of the range and azimuth (the real and imaginary features output by the azimuth dimension fast Fourier transform subnetwork), respectively. This represents the real part feature of the output of the azimuth-dimensional Fast Fourier Transform subnetwork; The imaginary part of the output of the azimuth-dimension fast Fourier transform subnetwork is represented. This indicates a channel-dimensional stacking operation.

[0128] Step A3: Model training.

[0129] The training dataset constructed in step A1 is input into the first object detection model, and optimized using stochastic gradient descent or its variants. The loss function typically includes classification loss (such as cross-entropy loss) and regression loss (such as mean squared error loss or IoU loss) to simultaneously optimize object recognition and localization accuracy.

[0130] During training, the linear layer weights of the signal feature extraction module will be adaptively adjusted from the FFT initialization state through backpropagation to learn the optimal frequency domain feature transformation for the target detection task, rather than a fixed Fourier transform.

[0131] During the online inference phase, a pre-trained first-object detection model is used for real-time detection.

[0132] Step B1, Data Preprocessing: The raw radar signal data (i.e., raw radar ADC data) obtained in step S1 will be preprocessed in the same way as in the training phase (such as formatting and normalization) to form a three-dimensional tensor that meets the model input requirements.

[0133] Step B2: Forward inference and post-processing of the model.

[0134] The preprocessed raw radar ADC data is input into the pre-trained first target detection model, and its forward inference process includes:

[0135] The preprocessed raw radar ADC data is sequentially passed through the three-layer sub-network of the signal feature extraction module to extract range-velocity-azimuth frequency domain features and generate a composite feature map. The composite feature map is processed by the target detection backbone network to output preliminary detection results, which include multiple candidate target bounding boxes, confidence scores for each category, and presence confidence scores in the range-azimuth image coordinate system.

[0136] The post-processing includes confidence filtering and non-maximum suppression, and coordinate transformation. Specifically, confidence filtering and non-maximum suppression: based on a set existence confidence threshold, low-confidence target bounding boxes are filtered out from multiple candidate target bounding boxes, and non-maximum suppression is performed on the remaining target bounding boxes to obtain the final target set.

[0137] Coordinate transformation: The pixel coordinates of each target in the range-azimuth image coordinate system are converted into physical coordinates in the radar coordinate system according to the range resolution and azimuth resolution of the radar system. The first target detection information finally output by step S2 is one or more targets described in the radar coordinate system. The information of each target includes at least the target position (represented in polar coordinates) and the target confidence (i.e., the probability value that the target is a real target).

[0138] Step S2 enables end-to-end adaptive sensing from raw radar signal data to target-level semantic information, providing precise radar sensing input for subsequent coordinate unification and decision fusion.

[0139] Step S3: Input the image data acquired in step S1 into the pre-trained second target detection model for processing, and output the second target detection information in the visual coordinate system.

[0140] Step S3 aims to utilize a pre-trained second object detection model to process the time-synchronized visual sensor (taking a stereo camera as an example) image data acquired in step S1, and output second object detection information in the visual coordinate system. Its implementation includes two stages: model pre-training and online inference.

[0141] The pre-training phase aims to train a neural network model capable of identifying target categories and locations in images, i.e., a second object detection model. Each training sample in the training dataset is an RGB image, with corresponding labels representing the bounding box and category of the target within the image. The second object detection model can employ a mature convolutional neural network architecture (such as the YOLO series). The training dataset is input into the second object detection model, and an optimization algorithm (such as stochastic gradient descent) minimizes the loss function between the predicted bounding box, category, and ground truth label, allowing the model to learn the mapping relationship from image pixels to target semantics. After training, a fixed set of model weight parameters is obtained and saved as the pre-trained second object detection model.

[0142] In the online inference phase, a pre-trained second target detection model is used to perform forward computation on the visual sensor image data synchronously acquired in real time during step S1, and the three-dimensional information of the target is recovered by combining it with a geometric model. The specific process includes:

[0143] Step S3.1: Input image preprocessing, specifically including distortion correction and stereo correction.

[0144] Distortion correction: The visual sensor image data obtained in step S1 (taking a binocular camera as an example, the image data includes the left eye image and the right eye image) is corrected using pre-calibrated camera distortion coefficients to eliminate lens distortion.

[0145] Stereo correction: Using pre-calibrated binocular camera extrinsics, stereo correction is performed on the distortion-corrected left and right eye images to align the corresponding epipolar lines horizontally, preparing for subsequent stereo matching.

[0146] Step S3.2: Forward inference of the model.

[0147] The preprocessed left-eye corrected image (or right-eye corrected image) is input into the pre-trained second object detection model. The model performs forward propagation and outputs preliminary detection results for the left-eye corrected image, including the object bounding box (in the image pixel coordinate system), object category, and object confidence (including category confidence and presence confidence).

[0148] Step S3.3: 3D information restoration and post-processing.

[0149] This step does not involve model learning; it follows a fixed computational process based on multi-view geometry and camera models, and is considered existing technology. On the stereo-corrected image pair, for each detected target bounding box in the left-eye corrected image, its disparity is calculated using a stereo matching algorithm. Based on the baseline and focal length of the binocular cameras, the disparity is converted into depth values. The statistical measure (such as the average value) of the effective depth values ​​within the target bounding box is taken as the depth estimate for that target.

[0150] Based on the target's pixel coordinates in the image pixel coordinate system, the camera intrinsic parameter matrix, and the depth, the three-dimensional coordinates and azimuth of the target in the camera coordinate system are calculated by back-projection using a pinhole camera model.

[0151] The final output of step S3 is the second target detection information, which is one or more targets described in the camera coordinate system. The information of each target includes at least the target position (three-dimensional coordinates), target category and target confidence.

[0152] Step S4: Based on the calibration parameters between the millimeter-wave radar and the vision sensor, convert the first target detection information and the second target detection information to a unified reference coordinate system.

[0153] Step S4 aims to unify the perception results obtained from single-modal detection, which are located in different sensor coordinate systems, into a common reference coordinate system (such as the vehicle coordinate system or the world coordinate system) through rigid body transformation, so as to provide spatially aligned input for subsequent fusion decisions.

[0154] Before system deployment, joint calibration of the millimeter-wave radar and vision sensors (such as binocular cameras) must be completed. Through calibration, the following key parameters are acquired and stored:

[0155] The extrinsic parameter matrix contains a rotation matrix and a translation vector. The extrinsic parameter matrix defines the rigid body transformation relationship from the visual sensor coordinate system to the millimeter-wave radar coordinate system (or vice versa, or both to a third reference coordinate system).

[0156] Timestamp alignment: Ensure that the first target detection information and the second target detection information correspond to the same moment in time.

[0157] First target detection information: located in the radar coordinate system, expressed in polar coordinates, including the target's slant range, azimuth and elevation angles, or two-dimensional coordinates that have been converted to Cartesian coordinates.

[0158] Second target detection information: Located in the camera coordinate system, represented in Cartesian coordinates, containing the target's three-dimensional coordinates.

[0159] Depending on the reference frame defined during calibration, there are three main transformation paths:

[0160] The first approach is to unify to the radar coordinate system:

[0161] This approach uses the millimeter-wave radar coordinate system as a unified reference coordinate system. Using the rigid body transformation extrinsic parameter matrix obtained through joint calibration, which transforms the camera coordinate system to the radar coordinate system, the second target detection information in the camera coordinate system is converted to the radar coordinate system.

[0162] The second method is to unify to the camera coordinate system:

[0163] This path uses the visual sensor (camera) coordinate system as a unified reference coordinate system. Using the inverse transformation of the aforementioned extrinsic matrix (a rigid body transformation extrinsic matrix from the camera coordinate system to the radar coordinate system), the first target detection information in the radar coordinate system is transformed to the camera coordinate system.

[0164] The third approach is to unify to an independent global coordinate system:

[0165] This approach uses a third-party coordinate system (such as the vehicle coordinate system or the world coordinate system) independent of the radar and camera as a unified reference coordinate system. This approach requires pre-calibrating the transformation relationships of the radar and camera relative to this global coordinate system to obtain transformation parameters. Subsequently, the detection information of the first and second targets is transformed to this global coordinate system respectively.

[0166] Step S4 outputs a list of target information that has been transformed to a unified reference coordinate system. Each target in the list includes its transformed coordinates, original confidence score, and category information (provided by visual inspection). This spatially aligned multimodal target data will be directly used as input to the subsequent fusion decision network.

[0167] Step S5: Construct a fused feature vector based on the first target detection information and the second target detection information in the reference coordinate system.

[0168] Step S5 aims to construct a structured fusion feature vector from the radar and visual target detection information, which has been transformed to a unified reference coordinate system, for subsequent processing by the fusion decision network. The specific implementation process is as follows:

[0169] Assuming that after step S4, coordinate system one has been obtained, under the same reference coordinate system (such as the vehicle body coordinate system):

[0170] First target detection information set: contains M radar-detected targets. The information for each target m is […]. ],in( ) represents the two-dimensional plane coordinates of the target m (usually ignoring height or taking the ground projection). Let m be the detection confidence level for target m.

[0171] The second object detection information set contains N visually detected objects. The information for each object n is […]. ],in( Let be the two-dimensional plane coordinates of the target n. Let n be the detection confidence level for target n. The category label for target n.

[0172] To explore all possible cross-modal associations, candidate matching pairs are constructed for all radar-detected targets and visually detected targets in the current frame. Specifically, for each combination of radar-detected target m and each visually detected target n, a candidate matching pair (m,n) is constructed. Theoretically, this will generate M×N candidate matching pairs.

[0173] For each candidate matching pair (m,n), a five-dimensional fusion feature vector is calculated and constructed, with the dimensions and meanings as follows:

[0174] Spatial overlap The calculation of the spatial overlap between radar-detected target m and visually detected target n on a two-dimensional plane in a unified reference coordinate system is achieved by calculating the distance intersection-union ratio between the two targets.

[0175] First, define a circular region with a fixed radius r centered on its own coordinates for each target, or calculate the projection of its rectangular region under the bird's-eye view (BEV) based on the size of the target's bounding box; then, calculate the intersection-union ratio between the two projected regions as the spatial overlap.

[0176] Confidence level of radar target detection : Directly use the confidence score of target m output by the first target detection model.

[0177] Confidence of visual target detection : The confidence level of target n directly used from the output of the second target detection model.

[0178] Radar Target Normalized Range Calculate the Euclidean distance from the radar-detected target m to the origin O of the reference coordinate system, and then normalize it. The specific formula is as follows:

[0179] (6)

[0180] in, This indicates the maximum effective sensing distance, which is set based on the detection performance of the millimeter-wave radar and visual sensor, as well as the target application scenario.

[0181] Normalized distance of visually detected target Calculate the Euclidean distance from the visually detected target n to the origin O of the reference coordinate system, and then normalize it. The specific formula is as follows:

[0182] (7)

[0183] The output of step S5 is a set of fused feature vectors { }, each vector This corresponds to a candidate radar-visual target association hypothesis. This set of fused feature vectors fully encodes information about the candidate matching pair in terms of spatial consistency, modal reliability, and absolute position. It will serve as the direct input to the fusion decision network to determine whether the association is valid and how to perform information fusion.

[0184] Step S6: Input the fused feature vector into the fusion decision network for decision-level fusion, and output the fused target detection result.

[0185] Step S6 aims to utilize a pre-trained fusion decision network to process the fusion feature vector constructed in step S5, achieving intelligent matching, localization optimization, and category confirmation of radar and visually detected targets, ultimately outputting the fused target detection result. Its implementation consists of two stages: model pre-training and online inference.

[0186] The pre-training (offline) of the fusion decision network aims to train the network to learn the mapping relationship from the fusion feature vector to the final fusion decision.

[0187] Training data preparation: Use a dataset containing accurate spatiotemporal alignment and unique target ID annotations. For each frame of training data, perform the same process as step S5: Based on the radar and visual detection results of that frame, construct all possible radar-visual target candidate matching pairs, calculate a five-dimensional fusion feature vector for each candidate matching pair, and form the input feature set for that frame.

[0188] Based on the unique target ID provided in the annotation, generate supervision labels for each candidate matching pair:

[0189] Matching label: A binary label (1 / 0) that indicates whether the matching pair corresponds to the same real object.

[0190] Regression label (positive samples only): The offset (dx, dy) of the fused target center relative to the radar detection center, specifically formulated as follows:

[0191] (8)

[0192] Category label (positive samples only): The category of the real object.

[0193] Construct a fusion decision-making network. For example... Figure 4 As shown, the fusion decision network adopts the classic architecture of "shared encoder-multi-task head", consisting of a shared feature encoder and three independent task prediction heads (matching head, regression head, and classification head) connected in parallel. This design allows the network to learn a general representation of cross-modal features and perform specialized predictions for different tasks.

[0194] The feature encoder is a multilayer perceptron with three fully connected layers, responsible for mapping low-dimensional fused feature vectors to high-dimensional abstract feature representations. Its specific structure and data flow are as follows:

[0195] Input: A five-dimensional fused feature vector .

[0196] The first fully connected layer (FC1) performs a linear transformation on the fused feature vector, increasing the input dimension from 5 to 32; then, non-linearity is introduced through the ReLU activation function; to enhance the robustness of the model and prevent overfitting, a Dropout operation is applied to the features after introducing non-linearity, randomly discarding some neuron outputs during training.

[0197] The second fully connected layer (FC2) performs a linear transformation on the 32-dimensional features after Dropout from the first fully connected layer, increasing the feature dimension from 32 to 64; then ReLU activation and Dropout are performed again.

[0198] The third fully connected layer (FC3) performs a linear transformation on the 64-dimensional features from the Dropout layer of the second fully connected layer, maintaining the feature dimension at 64; then ReLU activation is performed to form the final encoded features. This encoded feature integrates spatial, confidence, and distance information from the original five-dimensional fused feature vector, and has stronger expressive power and non-linear correlation.

[0199] Three task prediction heads receive the same 64-dimensional encoded features from the feature encoder in parallel and perform specialized processing on each.

[0200] The matching head is used to determine whether the radar and visual detection targets belong to the same real object. Its specific structure includes:

[0201] The fourth fully connected layer (FC4) linearly transforms the 64-dimensional encoded features to 32 dimensions, followed by ReLU activation.

[0202] The fifth fully connected layer (FC5) linearly transforms the 32-dimensional features to 1 dimension.

[0203] Sigmoid activation function layer: Compresses the 1-dimensional output to the (0,1) interval to obtain the matching probability p_match. The closer the p_match value is to 1, the higher the probability of matching.

[0204] The regression head is used to predict the two-dimensional offset of the precise center of the fused target relative to the original radar detection center. Its specific structure includes:

[0205] The sixth fully connected layer (FC6) linearly transforms the 64-dimensional encoded features to 32 dimensions, followed by ReLU activation.

[0206] The seventh fully connected layer (FC7) linearly transforms the 32-dimensional features to 2-dimensionality, resulting in a two-dimensional vector [dx, dy], representing the offsets on the X and Y axes of the reference coordinate system, respectively. This layer does not use a non-linear activation function to maintain the continuity of the output, making it suitable for regression tasks.

[0207] The classification head is used to predict the semantic category of the fused target, and its specific structure includes:

[0208] The eighth fully connected layer (FC8) linearly transforms the 64-dimensional encoded features to 32 dimensions, followed by ReLU activation.

[0209] The ninth fully connected layer (FC9) linearly transforms the 32-dimensional features to the num_classes dimension, where num_classes is the total number of target categories preset by the system.

[0210] Softmax activation function layer: Normalizes the output of the num_classes dimension into a probability distribution such that the sum of the probabilities of all classes is 1.

[0211] The architecture of the fusion decision network reduces model complexity through parameter sharing and improves the performance and generalization ability of each task through multi-task joint learning, which is the core of achieving efficient and intelligent decision-level fusion.

[0212] Training the fusion decision network: Input the generated set of input features into the fusion decision network, minimize the multi-task loss function (usually a weighted sum of the binary cross-entropy loss of the matching task, the smooth L1 loss of the regression task, and the cross-entropy loss of the classification task) through the optimization algorithm, and iteratively update the network weights until the fusion decision network converges.

[0213] During the online inference phase, a pre-trained fusion decision network is used to perform forward computation and decision-making on the fusion feature vectors of the real-time scene. Specifically, the set of fusion feature vectors constructed in real-time in step S5 is input into the pre-trained fusion decision network; each fusion feature vector undergoes a nonlinear transformation via a feature encoder to obtain high-dimensional encoded features; the high-dimensional encoded features are then sent to three parallel task prediction heads:

[0214] The matching head outputs the matching probability p_match∈[0,1] for the corresponding candidate matching pair; the regression head outputs the predicted position offset (dx,dy); and the classification head outputs the class probability distribution p_class.

[0215] Matching decision: For each candidate matching pair, if its matching probability is higher than a preset threshold (such as 0.5), it is determined to be a valid matching pair.

[0216] Target information fusion: For each valid matching pair (m,n), its position is determined using radar detection (m,n). The offset predicted by the fusion decision network is corrected, and the corrected position is ( +dx, +dy); its category adopts the category with the highest probability in the category probability distribution, or directly adopts the visual detection category; its confidence is a combination of matching probability, , The generated fusion confidence score.

[0217] Finally, all fused targets, along with unmatched high-confidence single-sensor detected targets (which can be retained according to application requirements), are aggregated to form the final target list. The final target list after fusion decision is output, and each target in the list includes its precise location in a unified reference coordinate system, semantic category, confidence level, and source information (e.g., "Ray-Vision Fusion").

[0218] Example 2

[0219] This invention also provides an electronic device, which includes: a memory, a processor, and a computer program or instructions stored in the memory. The processor executes the computer program or instructions to implement the target detection method based on radar-visual fusion perception in this invention.

[0220] Although not shown, the electronic device includes a processor that can perform various appropriate operations and processes based on programs and / or data stored in read-only memory (ROM) or loaded from a storage portion into random access memory (RAM). The processor can be a multi-core processor or may contain multiple processors. In some embodiments, the processor may include a general-purpose main processor and one or more specialized coprocessors, such as a central processing unit, graphics processing unit (GPU), neural network processor (NPU), digital signal processor (DSP), etc. Various programs and data required for device operation are also stored in RAM. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0221] The processor and memory described above are used together to execute programs / instructions stored in the memory. When the program / instructions are executed by the computer, they can implement the methods, steps, or functions described in the above embodiments.

[0222] Although not shown, embodiments of the present invention also provide a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implements the target detection method based on radar-visual fusion perception in embodiments of the present invention.

[0223] Readable storage media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0224] The above description only discloses specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or modifications that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A target detection method based on radar and vision fusion perception, characterized in that, The detection method comprises: Synchronously triggering a millimeter wave radar and a visual sensor to collect time-synchronized millimeter wave radar original signal data and visual sensor image data; Inputting the original signal data into a pre-trained first target detection model for processing to output first target detection information in a radar coordinate system; wherein the first target detection model comprises a signal feature extraction module for adaptively extracting frequency domain features from the original signal data; Inputting the image data into a pre-trained second target detection model for processing to output second target detection information in a visual coordinate system; Based on calibration parameters between the millimeter wave radar and the visual sensor, converting the first target detection information and the second target detection information to a unified reference coordinate system; According to the first target detection information and the second target detection information in the reference coordinate system, constructing a fusion feature vector; Inputting the fusion feature vector into a fusion decision network for decision-level fusion to output a fused target detection result. 2.The target detection method based on radar and visual fusion perception according to claim 1, characterized in that, The signal feature extraction module comprises a distance dimension fast Fourier transform subnetwork, a velocity dimension fast Fourier transform subnetwork and an azimuth dimension fast Fourier transform subnetwork which are sequentially cascaded; The distance dimension fast Fourier transform subnetwork is configured to extract frequency domain features along a fast time sampling dimension of the original signal data to obtain distance features; The velocity dimension fast Fourier transform subnetwork is configured to extract frequency domain features along a slow time frame sequence dimension in the distance features to obtain velocity features; The azimuth dimension fast Fourier transform subnetwork is configured to extract frequency domain features along a receiving antenna channel dimension in the velocity features to obtain azimuth features. 3.The target detection method based on the radar and visual fusion perception according to claim 2, characterized in that, The distance dimension fast Fourier transform subnetwork, the velocity dimension fast Fourier transform subnetwork and the azimuth dimension fast Fourier transform subnetwork each comprise: A first linear layer configured to perform linear transformation on a real part of an input signal; A second linear layer configured to perform linear transformation on an imaginary part of the input signal; A complex number calculation unit configured to calculate real part features and imaginary part features after frequency domain transformation based on outputs of the first linear layer and the second linear layer; Wherein, weight matrices of the first linear layer and the second linear layer are approximated by cosine components and sine components of a discrete Fourier transform during initialization. 4.The target detection method based on the radar and visual fusion perception according to claim 3, characterized in that, A specific calculation formula of the real part features and the imaginary part features after frequency domain transformation is as follows: ； ； wherein represents a real part characteristic; represents an imaginary part characteristic; represents a weight matrix of the first linear layer; represents a weight matrix of the second linear layer; represents a real part of the input signal; represents an imaginary part of the input signal.

5. The method according to any one of claims 2-4, wherein, The signal feature extraction module further comprises a splicing module configured to splice real part features and imaginary part features of the azimuth features output by the azimuth dimension fast Fourier transform subnetwork in a channel dimension to form a composite feature map. 6.The target detection method based on the radar and visual fusion perception according to claim 1, characterized in that, The first target detection model further comprises a target detection backbone network configured to perform target detection on the composite feature map output by the signal feature extraction module to output target coordinates and confidence information in the radar coordinate system. 7.The target detection method based on the radar and visual fusion perception according to claim 1, characterized in that, The first target detection information comprises radar detected target positions and confidence, and the second target detection information comprises visual detected target positions, confidence and categories. According to the first target detection information and the second target detection information under the reference coordinate system, a fusion feature vector is constructed, comprising: According to the radar detection target position and the visual detection target position under the reference coordinate system, a spatial overlap degree is calculated; wherein the spatial overlap degree is the intersection-over-union between the projection areas of the radar detection target and the visual detection target on the horizontal two-dimensional plane of the reference coordinate system; According to the radar detection target position under the reference coordinate system, a normalized distance of the radar detection target relative to the origin of the reference coordinate system is calculated; According to the visual detection target position under the reference coordinate system, a normalized distance of the visual detection target relative to the origin of the reference coordinate system is calculated; According to the spatial overlap degree, the confidence of the radar detection target, the confidence of the visual detection target, the normalized distance of the radar detection target relative to the origin of the reference coordinate system, and the normalized distance of the visual detection target relative to the origin of the reference coordinate system, a fusion feature vector is constructed. 8.The target detection method based on the radar and visual fusion perception according to claim 7, characterized in that, The normalized distance is obtained by dividing the Euclidean distance of the target two-dimensional coordinate under the reference coordinate system to the origin by a preset maximum effective sensing distance; wherein the maximum effective sensing distance is set according to the detection performance of the millimeter wave radar and the visual sensor and the target application scenario. 9.The target detection method based on the radar and visual fusion perception according to claim 1, characterized in that, The fusion decision network comprises a feature encoder and at least one task prediction head connected to the output of the feature encoder; The feature encoder is used for nonlinear transformation and high-order feature extraction of the input fusion feature vector to obtain encoded features; The task prediction head is used for performing at least one of target matching prediction, position regression prediction and target classification prediction based on the features output by the feature encoder.

10. The target detection method based on the radar and visual fusion perception according to claim 9, characterized in that, The feature encoder comprises a first fully connected layer, a second fully connected layer and a third fully connected layer connected in sequence; an activation function layer and a dropout regularization layer are respectively arranged after the first fully connected layer and the second fully connected layer; The first fully connected layer is used for transforming the input fusion feature vector from a first dimension to a second dimension; The second fully connected layer is used for transforming the features of the second dimension to a third dimension; The third fully connected layer is used for transforming the features of the third dimension to a fourth dimension and outputting. 11.The target detection method based on the radar and visual fusion perception according to claim 9, characterized in that, The task prediction head comprises a matching head, which comprises a fourth fully connected layer, a fifth fully connected layer and an S-type activation function layer connected in sequence, and is used for outputting the matching probability between the radar detection target and the visual detection target. 12.The target detection method based on the radar and visual fusion perception according to claim 9, characterized in that, The task prediction head comprises a regression head, which comprises a sixth fully connected layer and a seventh fully connected layer connected in sequence, and is used for outputting the position offset of the fused target detection center relative to the radar detection center. 13.The target detection method based on the radar and visual fusion perception according to claim 9, characterized in that, The task prediction head comprises a classification head, which comprises an eighth fully connected layer, a ninth fully connected layer and a normalized exponential activation function layer connected in sequence, and is used for outputting the confidence distribution of the fused target category.

14. An electronic device comprising a memory, a processor, and a computer program or instructions stored on the memory, wherein the computer program or instructions, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-13. The processor executes the computer program or instructions to implement the target detection method based on radar-visual fusion perception according to any one of claims 1-13.

15. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions, when executed by the processor, implement the target detection method based on the radar and vision fusion perception according to any one of claims 1-13.

Images