Target detection method and device based on radar and visual fusion perception, and storage medium

Abstract

The application discloses a target detection method and device based on radar and vision fusion perception and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: inputting radar original signal data into a first target detection model for processing, and outputting first target detection information; inputting visual image data into a second target detection model for processing, and outputting second target detection information; converting the first target detection information and the second target detection information to a unified reference coordinate system based on calibration parameters between a millimeter wave radar and a vision sensor; constructing a fusion feature vector according to the first target detection information and the second target detection information in the reference coordinate system; and inputting the fusion feature vector into a fusion decision network for decision-level fusion, and outputting a fused target detection result. The application effectively reduces the false matching rate in a complex scene, and improves the accuracy and stability of the fusion result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a target detection method and device based on radar-vision fusion perception and a storage medium. BACKGROUND

[0002] With the rapid development of automatic driving, intelligent transportation and environmental perception technology, a multi-sensor fusion system has become a key means to realize high-precision and high-robustness environmental perception. Currently commonly used environmental perception sensors mainly include laser radars, cameras and millimeter wave radars, etc. Although the laser radar has high spatial resolution and three-dimensional modeling capability, it has problems such as high cost, large volume and significant performance decline in rain, fog and strong light environments; the monocular camera can provide rich semantic information, but cannot directly obtain target distance and is sensitive to light changes. In comparison, the millimeter wave radar and the binocular camera have unique advantages in performance, cost and functional complementation: the millimeter wave radar can work stably in complex weather and low visibility conditions and directly obtain distance, speed and direction information of the target; the binocular camera can provide fine visual semantics and spatial structure information. The effective combination of the two can not only make up for the limitations of a single sensor, but also achieve a good balance between perception accuracy, robustness and real-time performance, and thus is regarded as an ideal combination of a multi-sensor fusion system.

[0003] In the multi-sensor fusion method, common fusion levels include data-level, feature-level and decision-level fusion. Data-level fusion directly aligns the original data at the signal or pixel level, which can theoretically retain the most information, but due to the significant differences in dimension, sampling rate and noise characteristics between radar and camera data, the cost of accurate synchronization and registration is high and the computational complexity is large, which is difficult to meet the real-time requirement. Feature-level fusion extracts sensor features respectively and fuses them at the intermediate layer, which alleviates the registration problem to some extent, but because the radar and visual feature spaces are inconsistent, the fusion effect often depends on the artificially set weighting rules, lacking adaptability and generalization ability. In comparison, decision-level fusion matches and fuses information at the result level after the radar and camera complete target detection respectively, which has the advantages of clear structure, low coupling degree and easy expansion. Decision-level fusion not only makes full use of the independent detection capability of each sensor, but also integrates spatial and semantic information at the target level, making the fusion process more flexible and robust. Especially when the fusion strategy is realized by a deep learning model, decision-level fusion can adaptively model the relationship between multi-modal features, thereby maintaining high accuracy and robustness in complex environments.

[0004] However, the existing radar and visual decision-level fusion methods still have obvious limitations. In terms of radar target detection, most systems follow the traditional constant false alarm rate detection method, which generates point clouds by sequentially performing distance fast Fourier transform, Doppler fast Fourier transform, constant false alarm rate detection, and direction of arrival estimation on the raw data of the millimeter wave radar. Such methods are prone to miss detection and false alarm in low signal-to-noise ratio, strong clutter, and multipath reflection environments. Moreover, since the detection threshold is dependent on manual setting, the methods lack adaptability and are difficult to meet the real-time intelligent detection needs of complex dynamic scenes. In terms of fusion strategy, existing methods usually project radar point clouds into the image coordinate system and match and fuse radar and camera detection results based on manually set spatial distance thresholds, category consistency, or confidence weighting rules. For example, the patent document with publication number CN116699602A obtains target positions by using traditional constant false alarm rate detection and frequency domain analysis on the radar side, and then combines the camera detection results to complete the fusion with the help of simple distance threshold and coordinate transformation. Although this method achieves information complementation to some extent, its matching process still highly depends on manual rules and lacks intelligent adaptability, which can lead to mismatching and missing matching in complex scenes or multi-target overlapping situations, resulting in insufficient stability of the fusion results.

[0005] In addition, although radar detection methods based on deep learning have gradually developed in recent years, most of them still need to perform fixed signal processing on the radar raw analog-to-digital conversion data before input, such as generating a range-Doppler map through fast Fourier transform. Such preprocessing not only brings about computational redundancy and information loss, but also limits the ability of the model to learn the inherent feature correlation between the radar signal and the target attribute in an end-to-end manner, thereby restricting the upper limit of the model performance and the generalization ability.

[0006] In summary, the existing decision-level fusion methods based on traditional constant false alarm rate detection and manual matching strategy have not fully tapped the potential of deep learning in cross-modal feature learning and adaptive fusion. Therefore, there is an urgent need for a millimeter wave radar and binocular camera fusion perception scheme that can achieve end-to-end learning and has adaptive fusion capability to comprehensively improve the detection accuracy, robustness, and intelligence level of the system in complex environments. SUMMARY

[0007] In view of the above-mentioned defects in the prior art, the purpose of the present application is to provide a target detection method, device, and storage medium based on radar and visual fusion perception, to solve at least one of the problems of the existing decision-level fusion method relying on manual rules, the poor adaptability of the radar detection algorithm, the dependence on preprocessing, and the difficulty for the system to achieve end-to-end optimization.

[0008] The present application solves the above technical problems by the following technical solutions: a target detection method based on radar and visual fusion perception, comprising:

[0009] The millimeter wave radar and the visual sensor are triggered synchronously to collect time-synchronized millimeter wave radar raw signal data and visual sensor image data;

[0010] The raw signal data is input into a pre-trained first target detection model for processing to output first target detection information in a radar coordinate system; wherein the first target detection model comprises a signal feature extraction module for adaptively extracting frequency domain features of the raw signal data;

[0011] The image data is input into a pre-trained second target detection model for processing to output second target detection information in a visual coordinate system;

[0012] Based on the calibration parameters between the millimeter wave radar and the visual sensor, the first target detection information and the second target detection information are converted to a unified reference coordinate system;

[0013] The first target detection information and the second target detection information in the reference coordinate system are used to construct a fusion feature vector;

[0014] The fusion feature vector is input into a fusion decision network for decision-level fusion to output a fused target detection result.

[0015] The fusion decision network automatically learns the complex nonlinear correspondence between radar targets and visual targets from training data, thereby dynamically adapting to changes in different target types, spatial distribution and environmental context.

[0016] The first target detection model comprising a signal feature extraction module directly processes the raw signal data of the millimeter wave radar, abandoning the traditional constant false alarm rate detection algorithm and the fixed signal processing chain it relies on. The signal feature extraction module can adaptively extract deep frequency domain features related to the target from the raw signal without relying on artificially pre-set detection thresholds. Therefore, the method effectively overcomes the limitations of traditional constant false alarm rate detection in low signal-to-noise ratio, strong clutter, multipath reflection and other harsh environments, significantly improves the detection ability of weak targets, suppresses false alarms, and realizes more robust and accurate radar target perception.

[0017] The signal feature extraction module directly performs adaptive frequency domain feature extraction on the original signal data, without performing fixed signal preprocessing steps such as fast Fourier transform to generate a range-Doppler map in advance. This not only reduces the calculation redundancy and potential information loss caused by fixed preprocessing, but more importantly, realizes end-to-end feature learning from the original radar signal to the target attribute. This way enables the model to autonomously mine the inherent mapping relationship between the signal and the target, thereby fully releasing the performance potential of the model, improving the detection accuracy, and enhancing the generalization ability of the model in different scenarios and configurations.

[0018] Further, the signal feature extraction module comprises a range dimension fast Fourier transform subnetwork, a velocity dimension fast Fourier transform subnetwork and an azimuth dimension fast Fourier transform subnetwork which are sequentially cascaded;

[0019] The range dimension fast Fourier transform subnetwork is configured to perform frequency domain feature extraction along the fast time sampling dimension of the original signal data to obtain range features.

[0020] The velocity dimension fast Fourier transform subnetwork is configured to perform frequency domain feature extraction along the slow time frame sequence dimension in the range features to obtain velocity features.

[0021] The azimuth dimension fast Fourier transform subnetwork is configured to perform frequency domain feature extraction along the receiving antenna channel dimension in the velocity features to obtain azimuth features.

[0022] In the present application, the signal feature extraction module adopts a range dimension, a velocity dimension and an azimuth dimension fast Fourier transform subnetwork which are sequentially cascaded, and sequentially performs feature transformation along the fast time sampling dimension, the slow time frame sequence dimension and the receiving antenna channel dimension of the original signal data. The structure directly corresponds to the three-dimensional fast Fourier transform process of range, velocity and azimuth in traditional radar signal processing. However, unlike the fixed and non-adjustable transform kernel in the traditional method, the present application realizes transformation through a trainable linear layer, the weight of which is approximately a discrete Fourier transform in the initialization stage and is adaptively adjusted in the training process. This enables the network to learn the nonlinear mapping from the original signal to the task-optimal frequency domain features end-to-end, thereby significantly improving the feature expression ability and target detection accuracy.

[0023] By sequentially processing the range, velocity and angle three physical dimensions, this module can systematically separate and fuse the multi-dimensional attribute features of the target. This structured feature extraction method not only preserves the complete physical information of the radar signal, but also strengthens the correlation and differentiation ability between different dimensional features through the nonlinear transformation of the neural network, thereby effectively improving the recognition ability of the model to the real target in the clutter, noise and multi-target interference environment, and enhancing the robustness of the system in complex dynamic scenarios.

[0024] The traditional radar detection method needs to perform fixed pre-processing such as fast Fourier transform on original ADC data, which may cause information loss (such as sidelobe leakage, limited resolution) and cannot be optimized according to the task. The frequency domain transformation process is embedded in the trainable network module, which avoids the calculation redundancy and inherent error brought by the independent preprocessing link, and realizes seamless and adaptive extraction from original signal to high-level semantic features. This not only improves the overall efficiency of signal processing and feature extraction, but also enhances the generalization performance of the model under different scenarios and configurations.

[0025] The modular design of the three sub-networks corresponds to the physical dimensions of radar signal processing, facilitating individual adjustment and optimization. The extracted azimuth features (including real and imaginary parts) can be directly input into the subsequent target detection backbone network through splicing and other operations, realizing smooth integration from signal processing to target detection, and providing effective key technology support for building high-performance, end-to-end radar perception systems.

[0026] Further, the range dimension fast Fourier transform sub-network, the velocity dimension fast Fourier transform sub-network and the azimuth dimension fast Fourier transform sub-network each comprises:

[0027] A first linear layer for performing linear transformation on the real part of the input signal;

[0028] A second linear layer for performing linear transformation on the imaginary part of the input signal;

[0029] A complex number calculation unit for calculating the real part feature and the imaginary part feature after frequency domain transformation according to the outputs of the first linear layer and the second linear layer;

[0030] Wherein, the weight matrix of the first linear layer and the weight matrix of the second linear layer are approximated by cosine components and sine components of discrete Fourier transform during initialization.

[0031] Embedding a learnable complex signal processing mechanism in the neural network realizes a physically interpretable and adaptive frequency domain transformation of the radar raw signal. By setting independent linear layers for the real part and the imaginary part of the signal respectively, and using the cosine and sine components of the discrete Fourier transform for weight initialization, the network has the ability to approximate the standard Fourier transform at the beginning of training. During the training process, the weights can be further adapted to learn more optimal frequency domain feature mapping related to the detection task.

[0032] The real part and the imaginary part feature are explicitly processed by the complex number calculation unit, and the final output is synthesized according to the complex operation rule. This structure completely preserves the phase and amplitude information of the radar signal. This provides the network with more rich input features, which helps to more accurately analyze the distance, speed and angle information of the target.

[0033] The complex Fourier transform process is modeled as a differentiable neural network, which enables joint training and end-to-end optimization with the subsequent target detection backbone network, eliminating the gap between the fixed transform module and the subsequent learning module in the traditional process, allowing gradients to be backpropagated throughout the network, thereby optimizing feature extraction and transformation processes for the final detection target, improving overall system performance.

[0034] Further, the specific calculation formula of the real part feature and the imaginary part feature after the frequency domain transformation is:

[0035] ;

[0036] ;

[0037] wherein, represents the real part feature; represents the imaginary part feature; represents the weight matrix of the first linear layer; represents the weight matrix of the second linear layer; represents the real part of the input signal; represents the imaginary part of the input signal.

[0038] Further, the signal feature extraction module further comprises a splicing module, which is used to splice the real part feature and the imaginary part feature of the azimuth feature output by the azimuth dimension fast Fourier transform sub-network in the channel dimension to form a composite feature map.

[0039] In this embodiment, the real part feature map and the imaginary part feature map obtained after transformation are spliced in the channel dimension, which completely preserves all the information of the radar signal in the frequency domain (the real part and the imaginary part together constitute a complex representation). Compared with using only the amplitude spectrum or the real part / imaginary part alone, this composite feature map provides the subsequent target detection backbone network with more rich features that contain both target energy distribution and phase structure, which helps to improve the detection accuracy and robustness.

[0040] The current mainstream target detection backbone network (such as YOLOv8) usually processes real number tensors. The splicing operation skillfully converts the complex number feature (two real number tensors) into a real number tensor with double the number of channels, so that it can be directly input without changing the backbone network structure, realizing seamless and lossless connection between the complex signal processing domain and the real deep learning network, ensuring the efficiency of information transmission and the simplicity of the system.

[0041] Further, the first target detection model further comprises a target detection backbone network, which is used to perform target detection on the composite feature map output by the signal feature extraction module to output target coordinates and confidence information in the radar coordinate system.

[0042] By connecting the signal feature extraction module with the target detection backbone network, the application forms a complete and coherent processing chain from the input of radar raw ADC data to the output of the final target position and confidence. This realizes the real end-to-end radar target detection, so that all parameters from low-level signal processing to high-level target recognition can be jointly optimized, thereby maximizing the performance potential of deep learning models.

[0043] The target detection backbone network (such as YOLOv8) is a mature visual architecture pre-trained and verified by a large amount of image data, which has strong feature extraction and spatial positioning capabilities. The application innovatively applies this architecture to the radar feature map after frequency domain feature extraction and splicing, which is essentially to "translate" the radar information into a format that can be effectively processed by visual networks. This not only takes advantage of the technical advantages of advanced models in the visual field, ensuring high-performance detection benchmarks, but also significantly reduces the development cost and risk of designing complex detection networks specifically for radar signals.

[0044] Further, the first target detection information includes radar detection target position and confidence, and the second target detection information includes visual detection target position and confidence and category;

[0045] According to the first target detection information and the second target detection information under the reference coordinate system, a fusion feature vector is constructed, comprising:

[0046] According to the radar detection target position and the visual detection target position under the reference coordinate system, a spatial overlap degree is calculated; wherein the spatial overlap degree is the intersection over union between the projection areas of the radar detection target and the visual detection target in the horizontal two-dimensional plane of the reference coordinate system;

[0047] According to the radar detection target position under the reference coordinate system, a normalized distance of the radar detection target relative to the origin of the reference coordinate system is calculated;

[0048] According to the visual detection target position under the reference coordinate system, a normalized distance of the visual detection target relative to the origin of the reference coordinate system is calculated;

[0049] According to the spatial overlap degree, the confidence of the radar detection target, the confidence of the visual detection target, the normalized distance of the radar detection target relative to the origin of the reference coordinate system, and the normalized distance of the visual detection target relative to the origin of the reference coordinate system, a fusion feature vector is constructed.

[0050] The present application systematically integrates heterogeneous information from two kinds of sensors, radar and vision, through the carefully designed five-dimensional feature vector: spatial consistency (measured by intersection over union), respective reliability (expressed by confidence), target spatial distribution (described by normalized distance). This provides a comprehensive and structured input for the subsequent fusion decision network, which contains both objective geometric relationships and subjective judgments of each sensor, and is a solid foundation for intelligent and adaptive fusion.

[0051] By calculating the intersection over union and the normalized distance, the present application method converts the original position information of different sensors at different positions and different scales into standardized measurements that are decoupled from absolute coordinates and have uniform scales. This effectively eliminates the interference caused by the differences in dimensions and numerical ranges of the original data on model training, enabling the neural network to learn effective fusion rules more quickly and stably, and improving training efficiency and model robustness.

[0052] The constructed fusion feature vector has clear physical meaning: the intersection over union directly reflects whether the target is "in the same place", the confidence reflects how "sure" each sensor is, and the normalized distance reflects how "far" the target is. This design makes the learning process of the model highly consistent with the logic of human fusion judgment, not only enhancing the interpretability of the model, but also enabling it to better understand and adapt to complex scenarios with different spatial layouts and different sensor reliability changes, thereby making more reasonable fusion decisions.

[0053] Further, the normalized distance is obtained by dividing the Euclidean distance from the two-dimensional coordinates of the target in the reference coordinate system to the origin by a preset maximum effective perception distance; wherein the maximum effective perception distance is set according to the detection performance of the millimeter wave radar and the vision sensor and the target application scenario.

[0054] By dividing the absolute Euclidean distance of the target by a preset maximum effective perception distance, the present application method normalizes the distance information of all targets to a relatively uniform numerical range (usually [0, 1]). This effectively solves the problem of unstable neural network training and slow convergence caused by the large dimension and wide span of the original distance value. At the same time, this parameter is injected into the model as prior knowledge, guiding the network to focus on targets within the effective detection range, and improving the adaptability and generalization ability of the model in different distance distribution scenarios.

[0055] The maximum effective perception distance can be set according to the sensor performance and application scenario, which makes the present application method highly flexible and practical. For radars or cameras with different detection ranges, or in different applications such as urban scenarios (focusing on close distances) and high-speed scenarios (focusing on long distances), this parameter can be adjusted to optimize the perception focus of the model, ensuring that the method maintains optimal performance under different hardware configurations and task requirements.

[0056] The calculation of the normalized distance implicitly assumes a reasonable physical hypothesis: the closer to the origin, the more reliable the perception of the target and the more important the information; and the closer to or beyond the maximum effective perception distance, the less reliable the perception of the target and the less important the information. Through normalization, the model can naturally learn the correlation between distance and reliability, so as to more reasonably weigh the evidence of near and far targets when making a fusion decision, avoid giving inappropriate weight to unreliable long-distance targets at the edge, and improve the overall reliability of the fusion result.

[0057] Further, the fusion decision network comprises a feature encoder and at least one task prediction head connected to the output of the feature encoder.

[0058] The feature encoder is configured to perform nonlinear transformation and high-order feature extraction on the input fusion feature vector to obtain encoded features.

[0059] The task prediction head is configured to perform at least one of target matching prediction, position regression prediction, and target classification prediction based on the features output by the feature encoder.

[0060] The present application discards the traditional matching and decision-making method based on fixed threshold or rules, and instead uses a neural network composed of a "feature encoder + task prediction head" for end-to-end learning. The feature encoder automatically learns the complex correlation between heterogeneous features, and the task prediction head directly outputs high-level decisions such as matching, positioning, and classification. This enables the fusion process to dynamically adjust according to the input data, making more accurate and robust intelligent decisions than fixed rules in complex scenarios such as target density, occlusion, or sensor confidence conflict.

[0061] The fusion decision network adopts a modular architecture of a shared feature encoder followed by multiple independent task prediction heads. This design allows multiple related tasks (matching, position regression, classification) to be completed in parallel in one forward propagation, sharing the computation of bottom-level feature extraction, significantly improving inference efficiency. At the same time, multiple tasks are jointly trained through a shared encoder, and the supervision signals of different tasks can promote each other, guiding the feature encoder to learn more discriminative and generalizable fusion feature representations, thereby optimizing the overall perception accuracy of the system.

[0062] The three results (matching probability, position offset, and class distribution) output by the fusion decision network constitute a complete and operational description of the environment target: the matching probability solves the data association problem of "whether the radar and camera see the same object"; the position offset provides more accurate positioning optimization based on the original sensor observation; and the class distribution provides semantic information of the target. These structured outputs can be directly used in the planning and control modules of autonomous driving, realizing an end-to-end perception closed loop from raw sensor signals to high-level application instructions.

[0063] Further, the feature encoder comprises a first fully connected layer, a second fully connected layer and a third fully connected layer connected in sequence; an activation function layer and a dropout regularization layer are arranged after the first fully connected layer and the second fully connected layer respectively;

[0064] The first fully connected layer is used for transforming the input fusion feature vector from a first dimension to a second dimension;

[0065] The second fully connected layer is used for transforming the feature of the second dimension to a third dimension;

[0066] The third fully connected layer is used for transforming the feature of the third dimension to a fourth dimension and outputting.

[0067] By adopting the stacked structure of multiple fully connected layers, the feature encoder of the present application can perform multiple and complex nonlinear transformations on the input fusion feature vector. This enables it to learn and extract high-order abstract features far beyond the original input (such as simple intersection over union, confidence, distance), thereby more deeply mining the implicit and complex matching and association rules between radar and visual targets, and providing strong feature support for subsequent accurate decision-making.

[0068] The fully connected layer is the most basic and mature component in neural networks, with high forward calculation and back propagation efficiency, and has been optimized to the extreme in modern deep learning frameworks. Adopting this standardized layering structure to build the feature encoder makes the network have the advantages of stable training, good convergence, and controllable computational overhead. This ensures that the fusion decision network can meet the stringent requirements of real-time systems such as autonomous driving for computational efficiency and stability while pursuing high performance.

[0069] Arranging an activation function layer (such as ReLU) between adjacent fully connected layers is the key to injecting nonlinearity into the network. This enables the feature encoder to fit the complex nonlinear mapping relationship between the input features and the high-level fusion concepts, thereby significantly enhancing the model's ability to distinguish difficult cases (such as different targets with close spatial positions), and improving the model's generalization performance in new scenarios and new target distributions that have not been seen before, avoiding the limitation of simple linear decision-making.

[0070] Further, the task prediction head comprises a matching head, the matching head comprising a fourth fully connected layer, a fifth fully connected layer and a sigmoid activation function layer connected in sequence, for outputting the matching probability between the radar detection target and the visual detection target.

[0071] The matching head maps the encoded abstract features to a scalar value between 0 and 1 through a full connection layer + S-shaped (sigmoid) activation function layer. The value directly and clearly represents the probability that the radar and visual detection targets belong to the same object, providing a continuous, differentiable, and statistically meaningful confidence input for subsequent target tracking, trajectory fusion, and other modules, replacing the traditional hard matching rule of 0 or 1.

[0072] The matching head is composed of only a full connection layer and an activation function, with an extremely simple structure and minimal parameter and computation amount. This ensures that after completing complex feature encoding, the final matching judgment link can be completed with extremely low delay, which is very suitable for high real-time application scenarios such as autonomous driving, ensuring efficient operation of the entire fusion system.

[0073] Further, the task prediction head includes a regression head, the regression head includes a sixth full connection layer and a seventh full connection layer connected in sequence, and is used for outputting a position offset of a fused target detection center relative to a radar detection center.

[0074] The regression head directly learns and outputs a two-dimensional position offset through a simple full connection layer. This offset aims to finely correct the original position of the radar detection to fuse the supplementary spatial information provided by the vision sensor. This enables the final fused target position to exceed the positioning accuracy limit of any single sensor, especially in target lateral position or static / low-speed target positioning, which can significantly improve the positioning accuracy of the overall perception system.

[0075] The regression head is composed of only a full connection layer, which is one of the lightest components in a neural network. This minimalist design means that while introducing key positioning optimization functions, the added parameter and computation cost is almost negligible. This perfectly meets the stringent requirements of embedded platforms or real-time systems for algorithm efficiency, ensuring that the improvement in positioning performance does not come at the expense of system speed.

[0076] Due to factors such as installation position, time synchronization, and calibration error between radar and vision sensors, there may be inherent systematic deviations in the observed positions of the same target. The regression head can automatically learn this cross-modal systematic deviation pattern from a large amount of data and provide adaptive compensation. This reduces the absolute dependence on sensor calibration accuracy to some extent, enhancing the fault tolerance and robustness of the system in actual deployment.

[0077] Further, the task prediction head includes a classification head, the classification head includes an eighth full connection layer, a ninth full connection layer, and a normalized exponential activation function layer connected in sequence, and is used for outputting a confidence distribution of a fused target class.

[0078] The classification head receives the fusion information jointly encoded by radar and visual features, and outputs a confidence distribution of all possible categories. This enables the final target category judgment to integrate the rich semantics of visual perception and the stable existence evidence of radar. Especially when visual perception is limited (such as dim light, partial occlusion), the auxiliary information provided by radar can effectively support or correct the classification decision, thereby improving the accuracy of overall target recognition and reliability in harsh conditions.

[0079] Through the structure of a fully connected layer + a Softmax activation function layer, the classification head outputs a standard probability distribution in which the sum of all category probabilities is 1. This output form not only gives the most likely category, but also quantifies the confidence degree of the model for the judgment (the distribution of the probabilities of each category), providing important uncertainty information for the downstream planning control module to support risk-aware decision-making. At the same time, the standardized probability is also convenient for weighted fusion or cascaded judgment with the outputs of other classifiers (such as pure visual classifiers) in the system.

[0080] Based on the same concept, the present application also provides an electronic device comprising a memory, a processor, and a computer program or instructions stored on the memory, wherein the processor executes the computer program or instructions to implement the target detection method based on radar-visual fusion perception as described above.

[0081] Based on the same concept, the present application also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions are executed by a processor to implement the target detection method based on radar-visual fusion perception as described above.

[0082] Compared with the prior art, the present application has the following beneficial effects:

[0083] The present application realizes adaptive decision-level fusion through a fusion decision network, replacing the traditional matching method relying on artificial rules, effectively reducing the false matching rate in complex scenes, and improving the accuracy and stability of the fusion result. The target detection model with a signal feature extraction module directly processes the original signal data, overcoming the problem of high false alarm rate and missed detection in low signal-to-noise ratio and strong clutter environments in traditional constant false alarm rate detection algorithms, and realizing more robust and accurate radar target detection. Without relying on fixed signal preprocessing steps, the frequency domain features are adaptively extracted from the original signal, avoiding information loss and computational redundancy, and improving the detection accuracy and generalization ability of the model.

[0084] The radar detection, visual detection, coordinate unification and decision fusion are integrated into a complete deep learning driven process, so that each link can be collaboratively trained and end-to-end optimized. This overcomes the performance bottleneck and error accumulation problem caused by traditional independent design of sub-modules, thereby further improving the overall precision, real-time performance and environmental robustness of target detection at the system level, and better meeting the demand for reliable environmental perception in high dynamic and complex scenarios such as automatic driving and intelligent transportation. BRIEF DESCRIPTION OF DRAWINGS

[0085] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the following embodiment description will be briefly introduced. Obviously, the drawings in the following description are only one embodiment of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0086] Figure 1 is a target detection method flowchart based on radar and visual fusion perception in the embodiment of the present application;

[0087] Figure 2 is a first target detection model architecture diagram in the embodiment of the present application;

[0088] Figure 3 is a fast Fourier transform subnetwork architecture diagram in the embodiment of the present application;

[0089] Figure 4 is a fusion decision network architecture diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0090] The technical solutions in the present application will be described clearly and completely in the following with reference to the drawings in the embodiment of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0091] The technical solutions of the present application will be described in detail in the following with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0092] Embodiment one

[0093] As shown in Figure 1 The target detection method based on radar and visual fusion perception provided by the embodiment of the present application includes the following steps:

[0094] Step S1: Synchronize the millimeter wave radar and the vision sensor to collect time-synchronized millimeter wave radar raw signal data and vision sensor image data.

[0095] To achieve accurate time synchronization of millimeter wave radar and vision sensor (such as binocular camera) data, step S1 aims to solve the time alignment problem, the core of which is to ensure that the data frames from different sensors correspond to the same physical time. Specific implementation can use one or more of the following combinations of solutions:

[0096] The first solution: On the data processing unit (such as industrial computer, embedded platform) connected to the millimeter wave radar and the vision sensor, deploy and run the Precision Time Protocol or Network Time Protocol service to keep the system clock synchronized with the high-precision reference clock (such as GPS clock source). This solution aims to control the local system clock deviation relied on by different sensor data collection within milliseconds, providing a unified time reference for data association.

[0097] The second solution: The central processing unit sends a start collection command to the millimeter wave radar and the vision sensor almost simultaneously through software instructions. Due to the non-deterministic delay in operating system scheduling, bus transmission, and sensor response, the synchronization accuracy of this method is usually lower than that of the hardware trigger solution, but it is relatively simple to implement.

[0098] Regardless of the synchronization mechanism described above, after each frame of sensor data (radar raw signal data packet, image data packet) is collected, a high-precision system timestamp is immediately labeled by the central processing unit. In the subsequent data processing pipeline, the radar frame and the image frame closest to the timestamp are matched to achieve time pairing of the sensor data stream. This method can effectively compensate for the slight asynchrony introduced by internal sensor processing, data reading, etc.

[0099] Step S2: Input the raw signal data collected in step S1 into the pre-trained first target detection model for processing, and output the first target detection information in the radar coordinate system.

[0100] Step S2 aims to use the pre-trained first target detection model to perform end-to-end processing on the time-synchronized radar raw signal data obtained in step S1, and directly output the first target detection information in the radar coordinate system. Its implementation includes two stages of model pre-training and online inference.

[0101] The pre-training of the first target detection model is completed offline, aiming to let the network learn the mapping relationship from the original radar signal to the target position and category. The specific steps are as follows:

[0102] Step A1, training data set construction.

[0103] For the radar part, each training sample is a piece of pre-processed radar raw ADC data. Its corresponding label is obtained by performing traditional signal processing procedures (3D FFT, CFAR, etc.) on the piece of raw ADC data to generate a range-azimuth intensity map, and then labeling the target bounding box and its class on the range-azimuth intensity map. In this way, a dataset is constructed with pre-processed radar raw ADC data as input and target boxes in the range-azimuth image coordinate system as supervised labels. These labels can be obtained in one or more of the following ways:

[0104] Manual labeling: the labeler directly frames the target and labels the class on the range-azimuth intensity map.

[0105] Semi-automatic labeling: use spatiotemporally synchronized and labeled visual image data (such as target bounding boxes in the image), combined with the accurate calibration parameters between radar and camera, to project the visual detection target to the range-azimuth image coordinate system of the radar, and after filtering and verification, generate labels for radar data. This method can efficiently obtain a large amount of labeled radar data.

[0106] Automatic simulation generation: generate radar raw ADC data and corresponding accurate target position and class labels simultaneously in a simulation environment.

[0107] Step A2, model architecture and initialization.

[0108] For radar raw ADC data, it usually contains three main dimensions: fast time sampling dimension, slow time frame sequence dimension, and receiving antenna channel dimension. In traditional radar signal processing, fast Fourier transform is usually performed along these three dimensions to extract the frequency domain features of the target in range, Doppler velocity, and azimuth angle, and to realize spatial positioning and motion parameter analysis of the target through peak value analysis.

[0109] To introduce the above signal processing mechanism into the deep learning network model and realize an end-to-end millimeter wave radar target detection network, the present application designs a signal feature extraction module composed of three layers of fast Fourier transform (FFT) subnetworks, corresponding to the range FFT, Doppler FFT, and azimuth FFT three dimensions. The structure and calculation process of each layer of FFT subnetwork are basically the same, only the data dimension they act on is different.

[0110] Specifically, as shown in Figure 2 the first target detection model includes a signal feature extraction module, a splicing module, and a target detection backbone network (such as a YOLOv8 model), and the signal feature extraction module includes a range dimension fast Fourier transform subnetwork, a velocity dimension fast Fourier transform subnetwork, and an azimuth dimension fast Fourier transform subnetwork cascaded in sequence.

[0111] The distance dimension fast Fourier transform subnetwork is used for frequency domain feature extraction along a fast time sampling dimension of radar original ADC data to obtain a distance feature; the velocity dimension fast Fourier transform subnetwork is used for frequency domain feature extraction along a slow time frame sequence dimension in the distance feature to obtain a velocity feature; and the azimuth dimension fast Fourier transform subnetwork is used for frequency domain feature extraction along a receiving antenna channel dimension in the velocity feature to obtain an azimuth feature.

[0112] As shown in Figure 3 The distance dimension fast Fourier transform subnetwork, the velocity dimension fast Fourier transform subnetwork and the azimuth dimension fast Fourier transform subnetwork each include a first linear layer, a second linear layer and a complex number calculation unit. In each layer subnetwork, an input complex number signal is first separated into a real part and an imaginary part, and is respectively sent into two independent linear layers for feature transformation. The weight matrix of the two linear layers adopts an approximate form of a discrete Fourier transform at initialization, and a specific formula is as follows:

[0113] (1)

[0114] (2)

[0115] wherein W1 represents a weight matrix of the first linear layer (or a real part linear layer), W1(i,j) represents a weight value of the weight matrix W1 at a position (i,j); W2 represents a weight matrix of the second linear layer (or an imaginary part linear layer), W2(i,j) represents a weight value of the weight matrix W2 at a position (i,j); W1 represents a weight matrix of the first linear layer (or a real part linear layer), W1(i,j) represents a weight value of the weight matrix W1 at a position (i,j); W2 represents a weight matrix of the second linear layer (or an imaginary part linear layer), W2(i,j) represents a weight value of the weight matrix W2 at a position (i,j); i represents a sampling point index of an input signal in a current processing dimension, j represents a component index of a frequency domain feature after transformation, and K represents a total sampling point number of the current processing dimension.

[0116] In the distance dimension fast Fourier transform subnetwork, K represents a total sampling point number of a fast time dimension, i represents a sampling point index of the fast time dimension, and j represents a component index of a distance dimension frequency domain feature, that is, a specific distance unit;

[0117] In the velocity dimension fast Fourier transform subnetwork, K represents a frame sequence length (pulse number) of a slow time dimension, i represents a pulse index of the slow time dimension, and j represents a component index of a velocity dimension frequency domain feature, that is, a specific velocity unit;

[0118] In the azimuth dimension fast Fourier transform subnetwork, K represents a total number of receiving antenna channels, i represents a physical position index of a receiving antenna channel, and j represents a component index of an azimuth dimension frequency domain feature, that is, a specific angle unit (or beam pointing direction).

[0119] Through this initialization mode, the linear layer can simulate the frequency domain mapping process of the Fourier transform in the initial state, so as to further adaptively fine-tune the weight through back propagation in the training process, and realize the optimal radar signal frequency domain feature extraction.

[0120] The complex calculation unit: after the real part and the imaginary part linear layer mapping, respectively, the complex linear transformation calculation is performed to obtain the real part feature and the imaginary part feature of the Fourier transform result, and the specific formula is:

[0121] (3)

[0122] (4)

[0123] wherein, represents the real part feature; represents the imaginary part feature; represents the real part of the input signal; represents the imaginary part of the input signal.

[0124] Through this structure design, the network can learn the Fourier transform feature most suitable for the task requirement in an end-to-end manner, realize the adaptive extraction of the radar original ADC data to the frequency domain feature, and does not need the traditional artificial preprocessing step, so that the network can automatically learn the mapping relationship from the signal to the target feature in the training process, and realize the truly data-driven target detection.

[0125] The real part feature and the imaginary part feature output by the azimuth dimension fast Fourier transform subnetwork are spliced in the channel dimension through the splicing module to form a composite feature map, and the composite feature map is then input into the target detection backbone network to perform target classification and regression, and output the prediction results of the target: target position and confidence. The specific splicing formula is:

[0126] (5)

[0127] wherein, represents the composite feature map, H represents the height of the composite feature map, W represents the width of the composite feature map, C1 and C2 respectively represent the channel number of the distance azimuth real part feature and the distance azimuth imaginary part feature (the real part feature and the imaginary part feature output by the azimuth dimension fast Fourier transform subnetwork); represents the real part feature output by the azimuth dimension fast Fourier transform subnetwork; represents the imaginary part feature output by the azimuth dimension fast Fourier transform subnetwork; represents the channel dimension stacking operation.

[0128] Step A3, model training.

[0129] The training data set constructed in step A1 is input into the first target detection model, which is optimized using stochastic gradient descent or its variants. The loss function usually includes a classification loss (such as cross-entropy loss) and a regression loss (such as mean square error loss or IoU loss) to optimize both target recognition and positioning accuracy.

[0130] During training, the linear layer weights of the signal feature extraction module will be adaptively adjusted from the FFT initialization state through backpropagation to learn the optimal frequency domain feature transformation for the target detection task, rather than a fixed Fourier transform.

[0131] In the online inference phase, the pre-trained first target detection model is used for real-time detection:

[0132] Step B1, data preprocessing: the radar raw signal data (i.e. radar raw ADC data) obtained from step S1 is preprocessed (such as formatting and normalization) in the same way as in the training phase to form a three-dimensional tensor that meets the model input requirements.

[0133] Step B2, model forward inference and post-processing.

[0134] The preprocessed radar raw ADC data is input into the pre-trained first target detection model, and the forward inference process includes:

[0135] The preprocessed radar raw ADC data is sequentially passed through the three-layer subnetwork of the signal feature extraction module for distance-velocity-azimuth frequency domain feature extraction and generates a composite feature map; the composite feature map is processed by the target detection backbone network to output preliminary detection results, which include multiple candidate target bounding boxes, class confidence, and existence confidence in the distance-azimuth image coordinate system.

[0136] The post-processing process includes confidence filtering and non-maximum suppression, as well as coordinate conversion. Confidence filtering and non-maximum suppression: according to the set existence confidence threshold, low-confidence target bounding boxes are filtered out from the multiple candidate target bounding boxes, and non-maximum suppression is performed on the remaining target bounding boxes to obtain the final target set.

[0137] Coordinate conversion: the pixel coordinates of each target in the distance-azimuth image coordinate system are converted into physical coordinates in the radar coordinate system according to the distance resolution and azimuth resolution of the radar system. The first target detection information output by step S2 is one or more targets described in the radar coordinate system, and the information of each target at least includes target position (represented in polar coordinates) and target confidence (i.e. the probability value that the target is a real target).

[0138] Through step S2, end-to-end and adaptive perception from radar raw signal data to target-level semantic information is realized, thereby providing accurate radar perception input for subsequent coordinate unification and decision fusion.

[0139] Step S3: inputting the image data collected in step S1 into a pre-trained second target detection model for processing, and outputting second target detection information in a visual coordinate system.

[0140] Step S3 aims to utilize a pre-trained second target detection model to process the time-synchronized visual sensor (for example, a binocular camera) image data obtained in step S1, and output second target detection information in a visual coordinate system. The implementation includes two stages of model pre-training and online inference.

[0141] The model pre-training stage aims to train a neural network model capable of identifying target categories and positions from images, i.e., a second target detection model. Each training sample of the training data set is an RGB image, and the corresponding labeled label is the target bounding box and its category in the image. The second target detection model can adopt a mature convolutional neural network architecture (such as the YOLO series). The training data set is input into the second target detection model, and the loss function between the predicted bounding box and the category and the real label is minimized through an optimization algorithm (such as stochastic gradient descent) to make the model learn the mapping relationship from image pixels to target semantics. After training, a set of fixed model weight parameters is obtained, which is saved as a pre-trained second target detection model.

[0142] In the online inference stage, the pre-trained second target detection model is used to perform forward calculation on the visual sensor image data collected in step S1 in real time, and the three-dimensional information of the target is recovered in combination with the geometric model. The specific process includes:

[0143] Step S3.1: input image preprocessing, specifically including distortion correction and stereo correction.

[0144] Distortion correction: the visual sensor image data (for example, binocular camera, the image data includes left eye image and right eye image) obtained in step S1 is corrected using the pre-calibrated camera distortion coefficient to eliminate lens distortion.

[0145] Stereo correction: the pre-calibrated binocular camera external parameters are used to perform stereo correction on the left and right eye images that have been corrected for distortion, so that the corresponding epipolar lines are horizontally aligned, thereby preparing for subsequent stereo matching.

[0146] Step S3.2: model forward inference.

[0147] The pre-processed left eye corrected image (or right eye corrected image) is input into the pre-trained second target detection model, the model performs forward propagation, and the preliminary detection result for the left eye corrected image is output, including target bounding box (in image pixel coordinate system), target category and target confidence (including category confidence and existence confidence).

[0148] Step S3.3: three-dimensional information recovery and post-processing.

[0149] This link does not involve model learning, and is a fixed calculation process based on multi-view geometry and camera model, which belongs to the prior art. On the stereo corrected image pair, for each detected target bounding box in the left eye corrected image, the disparity is calculated by a stereo matching algorithm. According to the baseline and focal length of the binocular camera, the disparity is converted into a depth value. The statistical quantity (such as the average value) of the effective depth values within the target bounding box is taken as the depth estimation of the target.

[0150] According to the pixel coordinates of the target in the image pixel coordinate system, the camera intrinsic matrix and the depth, the three-dimensional coordinates and the azimuth angle of the target in the camera coordinate system are calculated by back projection using the pinhole camera model.

[0151] The final output of step S3 is the second target detection information, which is one or more targets described in the camera coordinate system, and the information of each target at least includes target position (three-dimensional coordinates), target category and target confidence.

[0152] Step S4: based on the calibration parameters between the millimeter wave radar and the vision sensor, the first target detection information and the second target detection information are converted to a unified reference coordinate system.

[0153] Step S4 aims to unify the perception results obtained by single modal detection, which are located in different sensor coordinate systems, to a common reference coordinate system (such as vehicle body coordinate system or world coordinate system) through rigid transformation, providing spatially aligned input for subsequent fusion decision.

[0154] Before system deployment, joint calibration of millimeter wave radar and vision sensor (such as binocular camera) needs to be completed in advance. Through calibration, the following key parameters are obtained and stored:

[0155] Extrinsic matrix: contains a rotation matrix and a translation vector. The extrinsic matrix defines the rigid transformation relationship from the vision sensor coordinate system to the millimeter wave radar coordinate system (or vice versa, or to a third reference coordinate system).

[0156] Timestamp alignment relationship: ensures that the first target detection information and the second target detection information correspond to the same time.

[0157] First target detection information: Located in radar coordinate system, expressed in polar coordinate form, containing the slant range, azimuth angle and pitch angle of the target, or two-dimensional coordinates converted from Cartesian coordinates.

[0158] Second target detection information: Located in camera coordinate system, expressed in Cartesian coordinate form, containing the three-dimensional coordinates of the target.

[0159] According to the reference system defined during calibration, there are three mainstream conversion paths:

[0160] The first one, unified to radar coordinate system:

[0161] This path takes the millimeter wave radar coordinate system as the unified reference coordinate system. The second target detection information in the camera coordinate system is converted to the radar coordinate system by using the rigid body transformation extrinsic parameter matrix from the camera coordinate system to the radar coordinate system obtained by joint calibration.

[0162] The second one, unified to camera coordinate system:

[0163] This path takes the visual sensor (camera) coordinate system as the unified reference coordinate system. The first target detection information in the radar coordinate system is converted to the camera coordinate system by using the inverse transformation of the above extrinsic parameter matrix (rigid body transformation extrinsic parameter matrix from camera coordinate system to radar coordinate system).

[0164] The third one, unified to an independent global coordinate system:

[0165] This path takes a third-party coordinate system independent of radar and camera (such as vehicle body coordinate system, world coordinate system) as the unified reference coordinate system. This path requires pre-calibration of the transformation relationship of radar and camera relative to the global coordinate system to obtain the transformation parameters. Then, the first and second target detection information are converted to this global coordinate system.

[0166] Step S4 outputs the target information list converted to the unified reference coordinate system. Each target in the list contains its converted coordinates, original confidence and category information (provided by visual detection). These spatially aligned multi-modal target data will be directly used as input for the subsequent fusion decision network.

[0167] Step S5: Construct a fusion feature vector according to the first target detection information and the second target detection information in the reference coordinate system.

[0168] Step S5 aims to construct radar and visual target detection information converted to the unified reference coordinate system into a structured fusion feature vector that can be processed by the subsequent fusion decision network. The specific implementation process is as follows:

[0169] Assuming that after the coordinate system in step S4, the following has been obtained in the same reference coordinate system (such as vehicle body coordinate system):

[0170] First target detection information set: contains M radar detection targets. The information of each target m is ] where is the two-dimensional planar coordinate of target m (usually ignoring the height or taking the ground projection), is the detection confidence of target m.

[0171] Second target detection information set: contains N visual detection targets. The information of each target n is ] where is the two-dimensional planar coordinate of target n, is the detection confidence of target n, is the class label of target n.

[0172] To explore all possible cross-modal associations, candidate matching pairs are constructed for all radar detection targets and visual detection targets of the current frame. Specifically, for each combination of a radar detection target m and a visual detection target n, a candidate matching pair (m, n) is constructed. In theory, this will produce M x N candidate matching pairs.

[0173] For each candidate matching pair (m, n), a five-dimensional fusion feature vector is calculated and constructed, with dimensions and meanings as follows:

[0174] Spatial overlap degree : Calculate the spatial overlap degree of radar detection target m and visual detection target n in the unified reference coordinate system on the two-dimensional plane, which is realized by calculating the distance intersection ratio of the two.

[0175] First, set a circular region with a fixed radius r centered on each target, or calculate the projection of its rectangular region in the bird's eye view (BEV) according to the size of the target bounding box; then, calculate the intersection ratio between the two projection regions as the spatial overlap degree.

[0176] Confidence of radar detection target : Directly use the confidence of target m output by the first target detection model.

[0177] Confidence of visual detection target : Directly use the confidence of target n output by the second target detection model.

[0178] Normalized distance of radar detection target : Calculate the Euclidean distance of radar detection target m to the origin O of the reference coordinate system and perform normalization processing, the specific formula is:

[0179] (6)

[0180] wherein, represents the set maximum effective perception distance, which is set according to the detection performance of the millimeter wave radar and the vision sensor and the target application scenario.

[0181] Vision detection target normalized distance : Calculate the Euclidean distance of the vision detection target n to the origin O of the reference coordinate system and perform normalization processing, and the specific formula is:

[0182] (7)

[0183] The output of step S5 is a set of fusion feature vectors , each vector corresponds to a candidate radar-vision target association hypothesis. This set of fusion feature vectors completely encodes the information of the candidate matching pair in terms of spatial consistency, modal reliability and absolute position, which will be directly input into the fusion decision network to determine whether the association is established and how to perform information fusion.

[0184] Step S6: input the fusion feature vector into the fusion decision network for decision-level fusion, and output the fused target detection result.

[0185] Step S6 aims to use the pre-trained fusion decision network to process the fusion feature vector constructed in step S5, realize intelligent matching, positioning optimization and class confirmation of radar and vision detection targets, and finally output the fused target detection result. Its implementation is divided into two stages of model pre-training and online inference.

[0186] The pre-training (offline) of the fusion decision network aims to train the network to learn the mapping relationship from the fusion feature vector to the final fusion decision.

[0187] Training data preparation: use a dataset containing accurate spatio-temporal alignment and target unique ID annotation. For each frame of training data, perform the same process as step S5: based on the radar and vision detection results of the frame, construct all possible radar-vision target candidate matching pairs, calculate a five-dimensional fusion feature vector for each candidate matching pair, and form the input feature set of the frame.

[0188] According to the target unique ID provided by the annotation, a supervised label is generated for each candidate matching pair:

[0189] Matching label: binary label (1 / 0), indicating whether the matching pair corresponds to the same real object.

[0190] Regression label (only positive samples): the offset amount (dx, dy) of the fusion target center relative to the radar detection center, and the specific formula is:

[0191] (8)

[0192] Class label (positive samples only): the class of the real object.

[0193] The fusion decision network is constructed. As shown in Figure 4 , the fusion decision network adopts the classic architecture of “shared encoding-multi-task head”, which is composed of a shared feature encoder and three independent task prediction heads (matching head, regression head, classification head) in parallel. This design allows the network to learn a general representation of cross-modal features and specialized predictions for different tasks.

[0194] The feature encoder is a multi-layer perceptron with three fully connected layers, responsible for mapping the low-dimensional fusion feature vector to a high-dimensional abstract feature representation. Its specific structure and data flow are as follows:

[0195] Input: a five-dimensional fusion feature vector .

[0196] First fully connected layer (FC1): Perform linear transformation on the fusion feature vector, increase the input dimension from 5 to 32; then introduce nonlinearity through the ReLU activation function; to enhance the robustness of the model and prevent overfitting, apply the Dropout operation to the feature after introducing nonlinearity, randomly discard part of the neuron output during training.

[0197] Second fully connected layer (FC2): Perform linear transformation on the 32-dimensional feature after Dropout from the first fully connected layer, increase the feature dimension from 32 to 64; then also perform ReLU activation and Dropout.

[0198] Third fully connected layer (FC3): Perform linear transformation on the 64-dimensional feature after Dropout from the second fully connected layer, keep the feature dimension at 64; then perform ReLU activation to form the final encoding feature. This encoding feature combines the spatial, confidence, and distance information in the original five-dimensional fusion feature vector, and has stronger expression ability and non-linear correlation.

[0199] The three task prediction heads receive the same 64-dimensional encoding feature from the feature encoder in parallel and perform specialized processing respectively.

[0200] The matching head is used to determine whether the radar and visual detection targets belong to the same real object, and its specific structure includes:

[0201] Fourth fully connected layer (FC4): Linearly transform the 64-dimensional encoding feature to 32-dimensional, followed by ReLU activation.

[0202] Fifth fully connected layer (FC5): Linearly transform the 32-dimensional feature to 1-dimensional.

[0203] Sigmoid activation layer: compresses the 1-dimensional output to the interval (0, 1) to obtain the matching probability p_match. The closer the value of p_match is to 1, the higher the matching probability.

[0204] The regression head is used to predict the two-dimensional offset of the precise center of the fused target relative to the original radar detection center. Its specific structure includes:

[0205] The sixth fully connected layer (FC6): linearly transforms the 64-dimensional encoded features to 32 dimensions, followed by a ReLU activation.

[0206] The seventh fully connected layer (FC7): linearly transforms the 32-dimensional features to 2 dimensions to obtain a two-dimensional vector [dx, dy], representing the offset in the X-axis and Y-axis of the reference coordinate system, respectively. This layer does not use a nonlinear activation function to maintain the continuity of the output, which is suitable for regression tasks.

[0207] The classification head is used to predict the semantic category of the fused target. Its specific structure includes:

[0208] The eighth fully connected layer (FC8): linearly transforms the 64-dimensional encoded features to 32 dimensions, followed by a ReLU activation.

[0209] The ninth fully connected layer (FC9): linearly transforms the 32-dimensional features to num_classes dimensions, where num_classes is the total number of target categories preset by the system.

[0210] Softmax activation layer: normalizes the num_classes-dimensional output to a probability distribution, so that the sum of all class probabilities is 1.

[0211] The architecture of the fusion decision network reduces model complexity through parameter sharing and improves the performance and generalization ability of each task through multi-task joint learning, which is the core of achieving efficient and intelligent decision-level fusion.

[0212] Fusion decision network training: input the generated input feature set into the fusion decision network, minimize the multi-task loss function (usually the weighted sum of the binary cross-entropy loss of the matching task, the smooth L1 loss of the regression task, and the cross-entropy loss of the classification task) through optimization algorithms, and iteratively update the network weights until the fusion decision network converges.

[0213] In the online inference stage, the pre-trained fusion decision network is used to perform forward calculation and decision on the fusion feature vectors of real-time scenes. Specifically, the fusion feature vector set constructed in real time in step S5 is input into the pre-trained fusion decision network; each fusion feature vector is subjected to nonlinear transformation via a feature encoder to obtain a high-dimensional encoded feature; the high-dimensional encoded feature is sent to three parallel task prediction heads respectively:

[0214] The matching head outputs the matching probability p_match of the corresponding candidate matching pair, wherein p_match belongs to [0, 1]; the regression head outputs the predicted position offset (dx, dy); and the classification head outputs the category probability distribution p_class.

[0215] Matching decision: for each candidate matching pair, if the matching probability is higher than a preset threshold (such as 0.5), the candidate matching pair is determined as an effective matching pair.

[0216] Target information fusion: for each effective matching pair (m, n), the position thereof is corrected by using the radar detection position (x_r, y_r) and the offset predicted by the fusion decision network, and the corrected position (x_f, y_f) is obtained; the category thereof is the category with the highest probability in the category probability distribution, or directly uses the visual detection category; and the confidence thereof is the fusion confidence generated by comprehensively matching the matching probability, the radar detection confidence, and the visual detection confidence.

[0217] Finally, all the fused targets and the high-confidence single-sensor detection targets that are not matched (which can be retained according to application requirements) are aggregated to form a final target list. The final target list after fusion decision is output, and each target in the list includes the accurate position, semantic category, confidence, and source information (such as “radar-visual fusion”) of the target in the unified reference coordinate system.

[0218] Embodiment Two

[0219] The embodiment of the present application also provides an electronic device, which comprises a memory, a processor, and a computer program or instruction stored in the memory, and the processor executes the computer program or instruction to implement the target detection method based on radar-visual fusion perception in the embodiment of the present application.

[0220] ​​​​​Although not shown, the electronic device includes a processor that can perform various appropriate operations and processes according to programs and / or data stored in a read-only memory (ROM) or programs and / or data loaded from a storage section into a random access memory (RAM). The processor can be one multi-core processor or can include a plurality of processors. In some embodiments, the processor can include a general-purpose main processor and one or more special-purpose co-processors, such as a central processing unit, a graphics processing unit (GPU), a neural processing unit (NPU), a digital signal processor (DSP), and the like. In the RAM, various programs and data required for device operations are also stored. The processor, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0221] The above processor and memory are used together to execute programs / instructions stored in the memory, which, when executed by a computer, can implement the methods, steps, or functions described in the above embodiments.

[0222] Although not shown, the embodiments of the present application also provide a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implements the target detection method based on radar and vision fusion perception in the embodiments of the present application.

[0223] The readable storage medium includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0224] The above only discloses specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or modifications within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A target detection method based on radar and vision fusion perception, characterized in that, The detection method comprises: Synchronously triggering a millimeter wave radar and a visual sensor to collect time-synchronized millimeter wave radar original signal data and visual sensor image data; Inputting the original signal data into a pre-trained first target detection model for processing to output first target detection information in a radar coordinate system; wherein the first target detection model comprises a signal feature extraction module for adaptively extracting frequency domain features from the original signal data; Inputting the image data into a pre-trained second target detection model for processing to output second target detection information in a visual coordinate system; Based on calibration parameters between the millimeter wave radar and the visual sensor, converting the first target detection information and the second target detection information to a unified reference coordinate system; According to the first target detection information and the second target detection information in the reference coordinate system, constructing a fusion feature vector; wherein for a candidate matching pair consisting of one radar detection target and one visual detection target in the same frame, the fusion feature vector comprises: spatial overlap of the radar detection target and the visual detection target in the reference coordinate system, confidence of the radar detection target, confidence of the visual detection target, normalized distance of the radar detection target relative to the origin of the reference coordinate system, and normalized distance of the visual detection target relative to the origin of the reference coordinate system; Inputting the fusion feature vector into a fusion decision network for decision-level fusion to output a fused target detection result; wherein the fusion decision network comprises a shared feature encoder and three task prediction heads connected to the output of the feature encoder; the feature encoder is used for nonlinear transformation and high-order feature extraction of the input fusion feature vector; the three task prediction heads respectively comprise: A matching head for predicting the matching probability between the radar detection target and the visual detection target based on the features output by the feature encoder; A regression head for predicting the position offset of the fused target detection center relative to the radar detection center based on the features output by the feature encoder; A classification head for predicting the confidence distribution of the fused target class based on the features output by the feature encoder. 2.The target detection method based on radar and visual fusion perception according to claim 1, characterized in that, The signal feature extraction module comprises a distance dimension fast Fourier transform subnetwork, a velocity dimension fast Fourier transform subnetwork and an azimuth dimension fast Fourier transform subnetwork which are sequentially connected; The distance dimension fast Fourier transform subnetwork is used for frequency domain feature extraction along the fast time sampling dimension of the original signal data to obtain distance features; The velocity dimension fast Fourier transform subnetwork is used for frequency domain feature extraction along the slow time frame sequence dimension in the distance features to obtain velocity features; The azimuth dimension fast Fourier transform subnetwork is used for frequency domain feature extraction along the receiving antenna channel dimension in the velocity features to obtain azimuth features. 3.The target detection method based on the radar and visual fusion perception according to claim 2, characterized in that, The distance dimension fast Fourier transform subnetwork, the velocity dimension fast Fourier transform subnetwork and the azimuth dimension fast Fourier transform subnetwork all comprise: A first linear layer for linear transformation of the real part of the input signal; a second linear layer configured to perform a linear transformation on an imaginary part of the input signal; a complex number calculation unit configured to calculate real part features and imaginary part features after a frequency domain transformation according to outputs of the first linear layer and the second linear layer. The weight matrix of the first linear layer and the weight matrix of the second linear layer are respectively approximated by cosine components and sine components of a discrete Fourier transform during initialization. 4.The target detection method based on the radar and visual fusion perception according to claim 3, characterized in that, The specific calculation formula of the real part features and the imaginary part features after the frequency domain transformation is as follows: ； ； wherein represents a real part characteristic; represents an imaginary part characteristic; represents a weight matrix of the first linear layer; represents a weight matrix of the second linear layer; represents a real part of the input signal; represents an imaginary part of the input signal.

5. The method of claim 2-4, wherein, The signal feature extraction module further includes a splicing module configured to splice real part features and imaginary part features of azimuth features output by the azimuth dimension fast Fourier transform subnetwork in a channel dimension to form a composite feature map. 6.The target detection method based on the radar and visual fusion perception according to claim 1, characterized in that, The first target detection model further includes a target detection backbone network configured to perform target detection on the composite feature map output by the signal feature extraction module to output target coordinates and confidence information in a radar coordinate system. 7.The target detection method based on the radar and visual fusion perception according to claim 1, characterized in that, The normalized distance is obtained by dividing a Euclidean distance from a two-dimensional coordinate of the target in a reference coordinate system to an origin by a preset maximum effective sensing distance; wherein the maximum effective sensing distance is set according to detection performance of the millimeter wave radar and the vision sensor and a target application scenario. 8.The target detection method based on the radar and visual fusion perception according to claim 1, characterized in that, The feature encoder includes a first fully connected layer, a second fully connected layer and a third fully connected layer connected in sequence; an activation function layer and a dropout regularization layer are respectively arranged after the first fully connected layer and the second fully connected layer; The first fully connected layer is configured to transform the input fusion feature vector from a first dimension to a second dimension; The second fully connected layer is configured to transform the features in the second dimension to a third dimension; The third fully connected layer is configured to transform the features in the third dimension to a fourth dimension and output. 9.The target detection method based on the radar and visual fusion perception according to claim 1, characterized in that, The matching head includes a fourth fully connected layer, a fifth fully connected layer and an S-type activation function layer connected in sequence. 10.The target detection method based on the radar and visual fusion perception according to claim 1, characterized in that, The regression head includes a sixth fully connected layer and a seventh fully connected layer connected in sequence. 11.The target detection method based on the radar and visual fusion perception according to claim 1, characterized in that, The classification head includes an eighth fully connected layer, a ninth fully connected layer and a normalized exponential activation function layer connected in sequence.

12. An electronic device comprising a memory, a processor, and a computer program or instructions stored on the memory, wherein the computer program or instructions, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-11. The processor executes the computer program or instructions to implement the target detection method based on radar-vision fusion perception according to any one of claims 1-11.

13. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the target detection method based on radar-vision fusion perception according to any one of claims 1-11.

Images