Millimeter wave radar and camera fused target detection method and system

Abstract

The invention belongs to the technical field of intelligent driving and vehicle perception. The invention provides a millimeter-wave radar and camera fused target detection method and system. The method comprises the following steps: obtaining a millimeter-wave radar mask pattern according to obtained millimeter-wave radar data; respectively inputting the millimeter wave radar mask image and the RGB image into two channels of a backbone network of the YOLO network model, embedding a frequency domain self-attention mechanism based on Fourier transform behind convolutional layers at the same positions of the two channels, and finally obtaining a target category and a two-dimensional bounding box; and determining the actual size and rotation angle of the target according to the two-dimensional bounding box, and determining the three-dimensional coordinate and attitude of the target according to the candidate depth of the millimeter wave radar point set target falling into the two-dimensional bounding box. According to the method, the detection precision and robustness of the model in complex environments such as low illumination, shielding, rain and fog are effectively improved while the detection speed is kept.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving and vehicle perception technology, specifically to a target detection method and system that integrates millimeter-wave radar and camera. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In commercial vehicle autonomous driving and its Advanced Driving Assistance Systems (ADAS), target detection is a crucial component of environmental perception, and its accuracy and robustness directly impact vehicle safety decisions and control execution. Currently, camera-based visual detection algorithms (such as YOLO and Faster R-CNN) have achieved high accuracy in bright, well-lit scenarios. However, in low-light conditions such as nighttime, tunnels, rain, and fog, or in complex weather conditions, visual sensors are susceptible to insufficient light, reflection interference, and noise pollution, leading to unstable detection results and increased false positive and false negative rates. Millimeter-wave radar offers the advantage of all-weather operation, accurately sensing target distance and speed information, and maintaining high reliability in low-light or obstructed environments. However, radar data has low spatial resolution and incomplete target contour information, making it difficult to independently achieve accurate target recognition and classification.

[0004] Most existing multi-sensor fusion methods rely on complex geometric calibration and point cloud projection processes, or use post-fusion strategies to overlay result-level information. These methods often suffer from problems such as large spatiotemporal synchronization errors, insufficient feature interaction, complex model structures, and poor real-time performance, making it difficult to meet the dual requirements of commercial vehicles for high robustness and high efficiency. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a target detection method and system that integrates millimeter-wave radar and camera. By designing a millimeter-wave mask generation strategy and introducing a frequency-domain self-attention feature enhancement module, it achieves efficient fusion of cross-modal features and fine-grained feature enhancement. While maintaining detection speed, it effectively improves the detection accuracy and robustness of the model in complex environments such as low light, occlusion, rain, and fog.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a target detection method that integrates millimeter-wave radar and camera.

[0007] A target detection method that fuses millimeter-wave radar and camera, comprising the following steps: A millimeter-wave radar mask is obtained based on the acquired millimeter-wave radar data; The millimeter-wave radar mask is input into the first channel of the YOLO network model backbone, and the RGB image acquired by the camera is input into the second channel of the YOLO network model backbone. The convolutional layers at the same positions of the first and second channels are followed by a frequency domain self-attention mechanism based on Fourier transform. The enhanced RGB image features output from the first channel and the enhanced millimeter-wave radar mask features from the second channel are fused element-wise at multiple scales to obtain the target category and two-dimensional bounding box. The actual size and rotation angle of the target are determined based on the two-dimensional bounding box. The three-dimensional coordinates and attitude of the target are determined based on the candidate depth of the millimeter-wave radar point set that falls within the two-dimensional bounding box.

[0008] In one implementation of the first aspect of the present invention, obtaining a millimeter-wave radar mask based on acquired millimeter-wave radar data includes: For millimeter-wave radar points, first perform rotation and translation operations to obtain the points in the lidar coordinate system and expand them into homogeneous coordinates; By using the camera's extrinsic and intrinsic parameter matrices, homogeneous coordinate points are projected onto the image, and a coordinate transformation from LiDAR to the camera is performed. The transformed coordinates are then projected onto the pixel coordinate system, and the pixel coordinates are calculated. Based on the target speed, the targets are divided into three categories: pedestrians, cyclists, and vehicles; For the projected pixel coordinates, draw a rectangle on the image as a mask; Adjust the size of the mask according to the distance from the vehicle; Adjust the aspect ratio of the mask based on the target's direction of motion; The color of the mask is weighted by the scattering cross-section, with pedestrians, cyclists, and vehicles each corresponding to a different base color. The mask color is then adjusted according to the relative size of the scattering cross-section to obtain the final millimeter-wave radar mask image.

[0009] In one implementation of the first aspect of the present invention, the frequency domain self-attention mechanism based on Fourier transform includes: The query vector, key vector, and value vector of the feature map output by the convolutional layer are extracted through a convolutional module. The query vector and key vector are subjected to Fast Fourier Transform and correlation modeling is performed in the frequency domain to obtain the frequency domain self-attention weights. The attention weights are multiplied by the value vector to obtain the enhanced features. The enhanced features are processed by the convolutional layer and then residually connected with the original features to obtain the output of the frequency domain self-attention mechanism.

[0010] In one implementation of the first aspect of the present invention, the first channel and the second channel have the same structure, each including a first convolutional layer, a second convolutional layer, a first frequency domain self-attention mechanism module, a first C2f module, a third convolutional layer, a second frequency domain self-attention mechanism module, a second C2f module, a fourth convolutional layer, a third frequency domain self-attention mechanism module, a third C2f module, a fifth convolutional layer, a fourth C2f module, and an SPPF module connected in sequence.

[0011] In one implementation of the first aspect of the present invention, determining the three-dimensional coordinates and pose of the target based on the candidate depth includes: Based on the candidate depth and the known true depth, construct linear constraint equations based on perspective projection; The system enumerates various combinations of angles and 2D boundary coordinates, uses the least squares method to solve for the center position T, and calculates the projection error for each combination. At the same time, a radar depth constraint term is added to the projection error to construct an error objective function. The combination with the smallest error objective function is selected as the optimal result, thus obtaining the target's three-dimensional coordinates and attitude.

[0012] In one implementation of the first aspect of the present invention, the two-dimensional bounding box includes: the coordinates of the center point of the two-dimensional bounding box, the width and height of the two-dimensional bounding box, the confidence score of the existence of the target, and the category probability distribution.

[0013] Secondly, the present invention provides a target detection system that integrates millimeter-wave radar and camera.

[0014] A target detection system that integrates millimeter-wave radar and camera includes: The millimeter-wave radar mask unit is configured to obtain a millimeter-wave radar mask image based on the acquired millimeter-wave radar data. The dual-channel enhanced feature extraction unit is configured to: input the millimeter-wave radar mask image into the first channel of the YOLO network model backbone network, input the RGB image acquired by the camera into the second channel of the YOLO network model backbone network, and embed a frequency domain self-attention mechanism based on Fourier transform after the convolutional layers at the same positions of the first and second channels. The two-dimensional target detection unit is configured to perform multi-scale element-wise fusion of the enhanced RGB image features output from the first channel and the enhanced millimeter-wave radar mask features output from the second channel to obtain the target category and two-dimensional bounding box. The three-dimensional target detection unit is configured to: determine the actual size and rotation angle of the target based on the two-dimensional bounding box, determine the three-dimensional coordinates and attitude of the target based on the candidate depth of the millimeter-wave radar point set falling within the two-dimensional bounding box, and determine the candidate depth of the target.

[0015] Thirdly, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the millimeter-wave radar and camera fusion target detection method of the first aspect of the present invention.

[0016] Fourthly, the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and execute the millimeter-wave radar and camera fusion target detection method of the first aspect of the present invention.

[0017] Fifthly, the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the target detection method of millimeter-wave radar and camera fusion according to the first aspect of the present invention.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This invention innovatively fuses millimeter-wave radar data with camera images, not only using the spatial structure and semantic information of the image model to compensate for the sparsity and geometric deficiencies of the radar point cloud, but also enhancing the perception of moving targets with the help of radar velocity information, thereby improving the robustness and accuracy of detection. Through this complementary fusion, the advantages of the two types of sensors can be better utilized to form a detection method adapted to low-light environments.

[0019] This invention innovatively improves the YOLO backbone network by designing a frequency domain self-attention mechanism based on Fourier transform (FFT Self-attention, FFTSA) to enhance the model's response to fine-grained edges and weakly textured regions. This module can globally capture structural features in the frequency domain, improving the network's robustness in scenarios with blurred target boundaries, partial occlusion, and low lighting.

[0020] This invention designs a dual enhancement attention (DEA) module, which further promotes the collaborative representation and feature fusion between RGB images and millimeter-wave radar masks. This module achieves cross-branch information guidance during the feature interaction stage, preserving both local spatial relationships and global semantic context, effectively improving the model's ability to fuse multimodal features.

[0021] The features extracted from the two branches of the YOLO backbone network in this invention are fused at multiple scales in the YOLO Neck structure, and the detection head outputs two-dimensional bounding boxes of vehicles, pedestrians and obstacles, realizing accurate detection and localization in low-light scenes.

[0022] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 A schematic flowchart of a target detection method that fuses millimeter-wave radar and camera, provided as an exemplary embodiment of the present invention; Figure 2 An image schematic diagram provided for an exemplary embodiment of the present invention, wherein, Figure 2 (A) in the image is an RGB image. Figure 2 (B) in the image is a millimeter-wave radar mask diagram; Figure 3 An overall framework diagram of a 2D detection method provided in an exemplary embodiment of the present invention; Figure 4 A schematic diagram of an FFT-based self-attention feature extraction module provided as an exemplary embodiment of the present invention; Figure 5 A 2D detection result visualization diagram is provided as an exemplary embodiment of the present invention, wherein, Figure 5 (A) in the diagram is a visualization of the 2D detection results of the RGB image. Figure 5 (B) in the diagram is a visualization of the 2D detection results of the millimeter-wave mask; Figure 6 A 3D output result visualization diagram provided for an exemplary embodiment of the present invention; Figure 7 A schematic diagram of a target detection system that fuses millimeter-wave radar and camera, provided as an exemplary embodiment of the present invention; Figure 8 A schematic diagram of a computer device provided for an exemplary embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] This implementation proposes a target detection method that fuses millimeter-wave radar and camera. By designing a millimeter-wave mask generation strategy and introducing a frequency-domain self-attention feature enhancement module (FFTSA), it achieves efficient fusion of cross-modal features and fine-grained feature enhancement. This scheme effectively improves the detection accuracy and robustness of the model in complex environments such as low light, occlusion, rain, and fog while maintaining detection speed, providing an efficient and deployable multimodal fusion solution for commercial vehicle environmental perception. Specifically, such as... Figure 1 As shown, the process includes the following: S101: Millimeter-wave radar mask fabrication.

[0028] In low-light environments, a single visual sensor cannot guarantee the stability and reliability of target detection, thus requiring the introduction of millimeter-wave radar data as a supplement. However, unlike lidar, millimeter-wave radar point clouds lack rich geometric structure information, characterized by sparse point clouds and high noise levels, making direct application to spatial target modeling difficult. Nevertheless, millimeter-wave radar possesses inherent advantages: strong robustness to lighting and weather conditions, and the ability to directly acquire radial velocity information of targets, providing powerful support for target recognition and tracking in dynamic scenarios.

[0029] Based on this characteristic, this invention fuses millimeter-wave radar data with camera images. On the one hand, it uses the spatial structure and semantic information of the image model to compensate for the sparsity and geometric deficiencies of the radar point cloud; on the other hand, it uses the speed information of the radar to enhance the perception of moving targets, thereby improving the robustness and accuracy of detection. Through this complementary fusion, the advantages of both types of sensors can be better utilized to form a detection method adapted to low-light environments.

[0030] The process of projecting millimeter-wave radar point clouds and converting them into millimeter-wave radar mask images has the following advantages: (1) After modal alignment and masking, radar information is mapped onto the same two-dimensional plane as the camera image, so that the two types of data are in the same spatial dimension. In this way, during fusion, there is no need for complex point cloud-pixel matching, but directly superimposed and aligned in the image domain. (2) Feature simplification: The original millimeter wave point cloud is sparse and noisy, and it is difficult to process directly. By converting it into a mask, the point cloud information can be expressed in the form of region labeling, weakening the interference of noise points on the model, and making it easier to combine with image features. (3) Semantic enhancement: The mask can not only represent the location of radar points, but also further encode attributes such as speed and distance (such as different channels or different colors), thereby embedding the advantageous information of millimeter wave radar into the semantic features of the image and improving the model's ability to recognize dynamic targets.

[0031] The present invention aims to fuse target information (point cloud / scattering points) detected by millimeter-wave radar with camera images to generate a millimeter-wave radar mask on the image plane, so as to facilitate subsequent target detection, recognition and multimodal fusion.

[0032] The data used in this implementation includes millimeter-wave radar data and camera data. The millimeter-wave radar data (mmdata) is converted into txt fields, specifically including: (1) Target_DistLong: longitudinal distance (2) Target_DistLat: Horizontal distance (3) Target_VrelLong: longitudinal relative velocity (4) Target_VrelLat: Lateral relative velocity (5) Target_RCS: Scattering cross-section (reflection intensity) (6) The remaining IDs and timestamps are used for matching.

[0033] Camera data (image & calibration), specifically, includes: left eye image (1280×720); intrinsic parameter matrix P2; extrinsic parameter matrix Trvelo→cam.

[0034] To accurately map millimeter-wave radar points onto the camera pixel plane, the joint calibration results of the millimeter-wave radar and camera are first used to obtain the rigid transformation (joint extrinsic parameters) from the millimeter-wave radar to the camera and the camera intrinsic parameters. Let a point in the millimeter-wave radar coordinate system be denoted as... Expand the point to homogeneous coordinates Let the extrinsic parameters of the millimeter-wave radar to the camera be (4×4 homogeneous transformation matrix). )for: (1); in, For rotation matrix, This is the translation vector (unit: meters). Represents the set of real numbers. Represents the transpose of the three-dimensional zero vector, and the camera intrinsic parameter matrix. for: (2); in, and This is the focal length in pixels. and Principal point coordinates This is the pixel-to-pixel inter-axis term (usually 0).

[0035] Transform millimeter-wave radar points to the camera coordinate system: (3); in, The X-axis coordinates of a point in the camera coordinate system; The Y-axis coordinate of a point in the camera coordinate system; Represents the Z-axis coordinate of a point in the camera coordinate system; The 4×1 homogeneous coordinates of the millimeter-wave radar points after transformation to the camera coordinate system; The homogeneous coordinates of a point in the millimeter-wave radar coordinate system.

[0036] Perform perspective projection in the camera coordinate system (using camera intrinsics) to obtain pixel homogeneous coordinates. : (4); in, The X-axis component representing the homogeneous pixel coordinates; The Y-axis component representing the homogeneous pixel coordinates; The scaling factor representing homogeneous coordinates; This represents the camera intrinsic parameter matrix.

[0037] Normalization yields pixel coordinates: (5); (6); in, Represents the normalized pixel X coordinate; Represents the normalized pixel Y coordinate; This represents the focal length of the camera along the X-axis. This represents the focal length of the camera along the Y-axis.

[0038] Only points that satisfy the following conditions are retained as valid mappings: (1) Depth is positive: (The point is located in front of the camera); (2) Pixels within the image range: ,in, and These are the image width and height (in pixels), respectively.

[0039] Preliminary classification based on target speed (provided by radar) (classification criteria are empirical values): Pedestrian: (7); in, The velocity in the X direction represents the pedestrian Pedestrian's velocity. This represents the speed of pedestrian Pedestrian in the Y direction.

[0040] Cyclist: (8); in, This represents the speed of the cyclist (Cyclist) in the X direction. This represents the speed of the cyclist (Cyclist) in the Y direction.

[0041] Vehicle: (9); in, Represents the vehicle's velocity in the X direction. This represents the speed of the vehicle in the Y direction.

[0042] For the projected points, draw a rectangular frame on the image to form a mask.

[0043] Based on the distance from the vehicle Adjust the size of the rectangle: (10); in, The base size is 30 pixels. It is a non-zero local minimum value.

[0044] Combined with the direction of movement Adjust the aspect ratio of the rectangle: (11); (12); in, Represents velocity in the Y direction. Represents velocity in the X direction.

[0045] Using scattering cross-section Weighting the mask colors: (13); in, This is an empirical constant, applicable to different target categories. The values ​​are different (pedestrians 50, cyclists 100, vehicles 150). It is the incremental scaling factor of the mask color, through The value of determines the scattering cross-section. Maximum range of weighted increments for color Draw a rectangular bounding box on the image plane and overlay it as a single-channel grayscale mask, where: The grayscale range for pedestrians is [50–250]; for cyclists it is [100–250]; and for vehicles it is [150–250]. Each frame of the RGB image is associated with a corresponding mask. Figure 1 One pair of storage, both are completely identical in spatial resolution and pixel size, and the mask image is saved in PNG format, such as... Figure 2 As shown, the mask image is used to represent the projection result of millimeter-wave radar on the image plane and can be fused with RGB images.

[0046] S102: 2D target detection.

[0047] The entire detection network is designed around a dual-channel YOLO architecture, relying on RGB images acquired by a camera and mask images generated by millimeter-wave radar as inputs. For ease of explanation, the network model of this implementation will be abbreviated as YOLO_RC. The inputs of the two channels in this implementation are... and .

[0048] YOLO_RC aims to address the issue of decreased detection accuracy and robustness in commercial vehicles under low-light conditions such as nighttime, tunnels, rain, and fog, caused by insufficient lighting and severe noise interference from single vision sensors. YOLO_RC is built upon a fusion solution of millimeter-wave radar and camera, avoiding reliance on a single modality and enhancing detection robustness and deployment practicality in complex traffic scenarios. The overall architecture is as follows: Figure 3 As shown, the backbone includes two channels. The first and second channels have the same structure, both including a first convolutional layer (Conv), a second convolutional layer (Conv), a first frequency domain self-attention mechanism module, a first C2f module, a third convolutional layer (Conv), a second frequency domain self-attention mechanism module (FFTSA), a second C2f module, a fourth convolutional layer (Conv), a third frequency domain self-attention mechanism module (FFTSA), a third C2f module, a fifth convolutional layer (Conv), a fourth C2f module, and an SPPF module, all connected sequentially. The millimeter-wave radar masking strategy proposed in this implementation projects sparse millimeter-wave point clouds onto the image plane and transforms them into a mask with semantic priors. This mask not only contains the spatial distribution information of radar targets but also embeds dynamic attributes such as velocity as auxiliary channel inputs to enhance the network's ability to focus on potential target regions. In the backbone network of YOLO_RC, this invention designs a frequency domain self-attention mechanism based on Fourier transform (FFT Self-attention, abbreviated as FFTSA module in the figure) to enhance the model's response to fine-grained edges and weakly textured regions. This module can globally capture structural features in the frequency domain, improving the network's robustness in scenarios with blurred target boundaries, partial occlusion, and low illumination.

[0049] Furthermore, to further promote the collaborative representation and feature fusion between RGB images and radar masks, a dual enhancement attention module (DEA module) was designed. This module guides information across branches during the feature interaction stage, preserving local spatial relationships while taking into account global semantic context, effectively improving the model's ability to fuse multimodal features. Finally, the features extracted from the two branches are fused at multiple scales in the Neck structure of YOLO_RC, and the detection head outputs two-dimensional bounding boxes for vehicles, pedestrians, and obstacles, achieving accurate detection and localization in low-light scenes.

[0050] More specifically, Figure 2 In the process, the feature maps of the two branches interact across modalities through multiple "DEA" modules. The Neck structure adopts a feature pyramid structure and fuses features of different scales through the operation of "upsampling → CONCAT → C2f" to generate feature maps of different scales. The "Conv" and "C2f" modules are used multiple times for feature processing during the process. The feature maps of the three scales are then processed by the "C2f" module and then enter the "Decect" module, finally outputting the detection results of the three scales.

[0051] In the complex traffic environments of autonomous driving, detected targets often exhibit multi-scale textures, complex boundary structures, and occlusion and overlap between vehicles and pedestrians. Traditional convolutional operations, due to their limited receptive field, struggle to capture long-distance dependencies, thus limiting the model's detection performance in low-light, crowded, or noisy scenarios. Therefore, this invention introduces a Fast Fourier Transform-based Self-Attention (FFTSA) mechanism into the backbone structure of the detection network, such as... Figure 4As shown, this mechanism enhances the network's ability to model boundary details, global structure, and long-range dependencies. It efficiently models the correlation between channels in the frequency domain with low computational and memory overhead, making it suitable for the real-time and lightweight deployment requirements in autonomous driving scenarios.

[0052] Figure 4 In this process, a color RGB image is used as input. The first step is to pass it through a convolutional module to extract key features from the image, such as the edges and contours of the target. After feature extraction, a normalization is performed to make the data more stable. Then, these features are divided into three paths, resulting in three sets of data: Q, K, and V. Q is like a "query" used to find important features; K is a "matcher" that works with Q to determine the correlation between features; and V is the "feature source," which is the core feature to be enhanced later.

[0053] Then, Q and K need to be subjected to Fast Fourier Transform (FFT) to transform them from the original image space to another dimension (frequency domain). In the frequency domain, the correlation between Q and K is calculated to obtain the "attention weights". This distinguishes which features are important and should be focused on, and which are not important and can be weakened. Then, an Inverse Fourier Transform (IFFT) is performed to transform the result back to the original space. After normalization, it is multiplied with V and the previously calculated weights are used to adjust the features of V, making the important features stronger.

[0054] After adjustment, a small 1×1 convolution is used to integrate these enhanced features. Finally, these enhanced features are added together with the original RGB features extracted by the initial convolution (this step is called residual connection) to avoid losing useful information during enhancement. Finally, the enhanced RGB features are output.

[0055] The radar branch below is the same as the RGB branch: the input is the data collected by the radar (which can capture the target distance and speed signals). It also first extracts features through convolution, then normalizes and splits into three paths: Q / K / V. Then, FFT is performed on Q and K, the correlation weights are calculated, and the inverse transformation is performed back. It is then combined with V to adjust the features, and then integrated with 1×1 convolution. Finally, it is added to the original radar features to output the enhanced radar features.

[0056] In this implementation, the FTSA module is applied to the backbone networks of the RGB main branch and the millimeter-wave mask (Radar Mask, M) auxiliary branch, respectively, and is connected after the convolutional feature map of each branch to improve the multimodal feature representation capability. Let the input feature map of the RGB or M branch at a certain layer be... The two branches are the RGB branch and the mask branch.

[0057] First, the query vector Q, key vector K, and value vector V features of the feature map are extracted using a convolutional module: (14); Subsequently, and Perform a Fast Fourier Transform (FFT) and model the correlation in the frequency domain. The formula for calculating the frequency domain self-attention weights is as follows: (15); in, This indicates the Fast Fourier Transform operation. Indicates inverse transform, symbol This represents the conjugate transpose in the complex domain. Due to the conjugate symmetry of the FFT transform, this method only requires storing half the data in the frequency domain, thus significantly reducing memory overhead. Next, attention weights... AND value vector Multiply to obtain the enhanced features: (16); in, The representation layer normalization operation is used to stabilize the training process and normalize the attention response. Finally, the enhanced features are processed through... The convolutional layers are integrated and residually connected with the original features to obtain the final output: (17); in, This represents a 1×1 convolution operation.

[0058] This module can be flexibly embedded after the convolutional layers of the YOLO backbone network, operating on both the RGB and radar mask branches. Compared to the traditional Transformer multi-head attention mechanism, FFTSA fully leverages the efficiency of frequency domain modeling, improving the model's ability to model complex structures while maintaining low computational overhead. By introducing FFTSA, the detection network can more effectively capture the boundary information, multi-scale structural features, and long-range dependencies of traffic targets, thus significantly enhancing perception robustness in low-light, occlusion, and noise interference environments.

[0059] Finally, the two enhanced branch features are fused element-wise (Dual Enhancement Attention): (18); in the formula The enhanced features corresponding to the RGB channels, These correspond to the enhanced features in the radar channel. Both are enhanced features obtained after processing at each layer in both the visual and radar modalities; subsequent element-wise operations (⊕) yield the fused features. Specifically, the three enhanced features from each of the two channels, with scales of 80×80×256, 40×40×512, and 20×20×1024 respectively, come from the second C2f layer, the third C2f layer, and the third SPPF layer in the Backbone; finally, they are input into the YOLO_RC detection head to complete the detection of the target category and bounding box.

[0060] In the object detection phase, the output of the YOLO_RC network consists of several two-dimensional bounding boxes. Composition, such as Figure 5 As shown, each two-dimensional bounding box contains the following information: (19); in, Indicates the coordinates of the center point of the prediction box. This indicates the width and height of the prediction box. The confidence score indicates the existence of the target. For class probability distribution, The total number of categories, For the first The YOLO_RC network generates two-dimensional bounding boxes. During inference, it removes redundant boxes using non-maximum suppression (NMS) and finally outputs the class label and confidence score for each detected object.

[0061] S103: 3D result output.

[0062] For each detected two-dimensional bounding box The target region is extracted from the RGB image and input into a length, width, and height feature regression network (such as the ResNet18 branch) to statistically analyze attributes related to two-dimensional pose, including observation angle, size bias, and confidence level. (20); in, The observation angle of the target in the image plane (can be represented by single regression or bin+res). Represents relative statistical prior size The residual, Represents the confidence level of the observation angle, and I represents the two-dimensional bounding box. The image is captured from within.

[0063] In the observation angle regression task, "bin+res" is a binning + residual angle representation and prediction method. The core is to split the "direct regression angle value" into two steps to improve the prediction accuracy.

[0064] Bin (interval binning): Divide the range of observed angles (e.g., 0°~360° in the image plane, or 0°~180° in the direction of the target) into several consecutive "angle intervals (bins)". For example, if 0°~180° is divided into 4 bins, then each bin corresponds to a 45° interval (bin1: 0°~45°, bin2: 45°~90°, bin3: 90°~135°, bin4: 135°~180°). This step is achieved through a classification task: the network predicts which bin the current target angle "belongs to" (outputs the probability of each bin, and selects the bin with the highest probability as the interval to which it belongs).

[0065] res (residual): After determining the bin to which the angle belongs, further regress the "offset of the angle within that bin (residual)"; for example: if the predicted angle belongs to bin2 (45°~90°) and the regression results in a residual of +5°, then the final observed angle is 45°+5°=50°.

[0066] Actual size of the target for: (twenty one); in, The statistical prior size representing the target.

[0067] Using camera intrinsic parameters , Represents the pixel coordinates of the camera's principal point along the X-axis of the image. This represents the pixel coordinates of the camera's principal point along the Y-axis of the image. It is based on the coordinate center point of the 2D bounding box. Calculate its horizontal ray in the camera coordinate system: (twenty two); The angle of rotation of the object around the Y-axis is: (twenty three); in, This represents the rotation angle of the target around the Y-axis of the camera coordinate system.

[0068] To compensate for the inherent inertia in depth estimation using monocular methods, millimeter-wave radar point cloud mask data synchronized with the image is introduced for each two-dimensional bounding box. Select the radar point set that falls within the frame. , middle, It is the core parameter describing the spatial extent of a two-dimensional bounding box. The reference coordinates for the bounding box, such as the top left corner / center; , These four parameters (width, height, and width of the bounding box, respectively) collectively define a rectangular region in the image (i.e., the actual area of ​​the two-dimensional bounding box); and (Confidence level) (Category) is an attribute parameter of the bounding box and does not participate in the definition of its spatial extent; , These are radar points. The pixel coordinates projected onto the RGB image plane (i.e., the two-dimensional position of the radar point in the image); therefore, " The actual meaning of "" is: radar point of Inside the defined rectangular region (two-dimensional bounding box).

[0069] And calculate its mean or median depth as the candidate depth for the target: (twenty four); in, For radar point set single radar point The corresponding depth component is the distance dimension information of the radar point relative to the sensing sensor in three-dimensional space.

[0070] Candidate depths can be used as initial values ​​or regularization terms in geometric constraint equations to guide the 3D center coordinates. Optimization.

[0071] Given the true depth and candidate depth, we first construct linear constraint equations based on perspective projection: (25); in, and Derived from angle projection constraints, The coefficient matrix representing the linear constraint equations. This represents the constant term vector of the linear constraint equation. By enumerating the 64 possible combinations of angles and 2D boundary coordinate edges, the 2D boundary coordinate edges refer to the four geometric edges of the two-dimensional bounding box (rectangular structure), namely the pixel coordinate range corresponding to the left boundary, right boundary, top boundary, and bottom boundary. After the 8 corner points are mapped to the two-dimensional plane, they form 4 edges (that is, the 2D box), which have a total of 64 possibilities.

[0072] The center position is determined using the least squares method. , These are the three-dimensional center coordinates of the target to be solved, i.e., the position parameters of the geometric center of the target in three-dimensional space. The projection error for a certain combination is also calculated. : (26); in, These are the coordinates of the target's three-dimensional feature points (such as the target's vertices, contour feature points, etc.), which are discrete points on the predefined three-dimensional model of the target, used to establish a mapping relationship between "three-dimensional points → two-dimensional image pixels" through perspective projection. The target's three-dimensional feature points In an RGB image, the corresponding two-dimensional observation coordinates (pixel coordinates) are indicated by the superscript "2", signifying that the coordinates are parameters in two-dimensional space, corresponding to the [number of pixels]. The actual pixel position of a 3D feature point after it is projected onto the image plane; It is around The rotation matrix of the axis, whose parameters are determined by the observation angle of the target. Determine (the orientation and rotation of the target in the corresponding image plane), used to describe the target's three-dimensional feature points. The attitude rotation relationship relative to the sensor coordinate system; It is the camera intrinsic parameter matrix, which is the inherent parameter matrix of the camera (including parameters such as focal length and principal point pixel coordinates). It is the core matrix in perspective projection transformation that converts points in the camera coordinate system to points in the image pixel coordinate system.

[0073] Simultaneously, a radar depth constraint term is added, forming an optimization equation: (27); in, These are the weighting coefficients of the radar depth constraint term, used to balance the projection error term. With radar depth constraint In total optimization error The contribution percentage in; adjustment The value can be adapted to the reliability of radar depth information in different scenarios; The Z-axis component represents the three-dimensional center coordinate T.

[0074] Finally, the combination with the smallest error is selected as the optimal result, yielding the target's three-dimensional coordinates and pose, such as... Figure 6 As shown, the final output is the 2D detection result of the target. In addition to the target rotation angle, the actual size of the target, and the target's three-dimensional coordinates and pose.

[0075] Figure 7 A target detection system integrating millimeter-wave radar and camera is shown, comprising: The millimeter-wave radar mask unit 701 is configured to obtain a millimeter-wave radar mask image based on the acquired millimeter-wave radar data. The dual-channel enhanced feature extraction unit 702 is configured to: input the millimeter-wave radar mask image into the first channel of the YOLO network model backbone network, input the RGB image acquired by the camera into the second channel of the YOLO network model backbone network, and embed a frequency domain self-attention mechanism based on Fourier transform after the convolutional layers at the same positions of the first and second channels. The two-dimensional target detection unit 703 is configured to perform multi-scale element-wise fusion of the enhanced RGB image features output from the first channel and the enhanced millimeter-wave radar mask features from the second channel to obtain the target category and two-dimensional bounding box. The three-dimensional target detection unit 704 is configured to: determine the actual size and rotation angle of the target based on the two-dimensional bounding box, determine the three-dimensional coordinates and attitude of the target based on the candidate depth of the millimeter-wave radar point set falling within the two-dimensional bounding box.

[0076] It is understood that the aforementioned units can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of the present invention. The aforementioned units are based on logical functional division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, the system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0077] According to another embodiment of the present invention, the system of this embodiment can be constructed by running a computer program (including program code) capable of performing the steps involved in the corresponding method of the present invention on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the aforementioned computing device through the computer-readable recording medium, and run therein.

[0078] Figure 8 A computer device is shown, which includes a processor 801, a communication interface 802, and a computer-readable storage medium 803. The processor 801, communication interface 802, and computer-readable storage medium 803 can be connected via a bus or other means.

[0079] The communication interface 802 is used to receive and send data. The computer-readable storage medium 803 can be stored in the memory of the electronic device. The computer-readable storage medium 803 is used to store computer programs, which include program instructions. The processor 801 is used to execute the program instructions stored in the computer-readable storage medium 803.

[0080] The processor 801 is the computing and control core of electronic devices. It is suitable for implementing one or more instructions, specifically for loading and executing one or more instructions to achieve corresponding methods or functions.

[0081] Processor 801 is configured to perform the following procedure: A millimeter-wave radar mask is obtained based on the acquired millimeter-wave radar data; The millimeter-wave radar mask is input into the first channel of the YOLO network model backbone, and the RGB image acquired by the camera is input into the second channel of the YOLO network model backbone. The convolutional layers at the same positions of the first and second channels are followed by a frequency domain self-attention mechanism based on Fourier transform. The enhanced RGB image features output from the first channel and the enhanced millimeter-wave radar mask features from the second channel are fused element-wise at multiple scales to obtain the target category and two-dimensional bounding box. The actual size and rotation angle of the target are determined based on the two-dimensional bounding box. The three-dimensional coordinates and attitude of the target are determined based on the candidate depth of the millimeter-wave radar point set that falls within the two-dimensional bounding box.

[0082] This invention also provides a computer-readable storage medium, which is a memory device in an electronic device for storing programs and data. It is understood that the computer-readable storage medium here may include both built-in storage media in the electronic device and extended storage media supported by the electronic device. The computer-readable storage medium provides storage space for storing the processing system of the electronic device.

[0083] Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory; alternatively, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0084] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to perform the following process: A millimeter-wave radar mask is obtained based on the acquired millimeter-wave radar data; The millimeter-wave radar mask is input into the first channel of the YOLO network model backbone, and the RGB image acquired by the camera is input into the second channel of the YOLO network model backbone. The convolutional layers at the same positions of the first and second channels are followed by a frequency domain self-attention mechanism based on Fourier transform. The enhanced RGB image features output from the first channel and the enhanced millimeter-wave radar mask features from the second channel are fused element-wise at multiple scales to obtain the target category and two-dimensional bounding box. The actual size and rotation angle of the target are determined based on the two-dimensional bounding box. The three-dimensional coordinates and attitude of the target are determined based on the candidate depth of the millimeter-wave radar point set that falls within the two-dimensional bounding box.

[0085] The present invention also provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the following process: A millimeter-wave radar mask is obtained based on the acquired millimeter-wave radar data; The millimeter-wave radar mask is input into the first channel of the YOLO network model backbone, and the RGB image acquired by the camera is input into the second channel of the YOLO network model backbone. The convolutional layers at the same positions of the first and second channels are followed by a frequency domain self-attention mechanism based on Fourier transform. The enhanced RGB image features output from the first channel and the enhanced millimeter-wave radar mask features from the second channel are fused element-wise at multiple scales to obtain the target category and two-dimensional bounding box. The actual size and rotation angle of the target are determined based on the two-dimensional bounding box. The three-dimensional coordinates and attitude of the target are determined based on the candidate depth of the millimeter-wave radar point set that falls within the two-dimensional bounding box.

[0086] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0087] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, digital cable) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for target detection by fusing millimeter wave radar and camera, characterized in that, The method comprises the following steps: obtaining a millimeter wave radar mask image according to the obtained millimeter wave radar data; inputting the millimeter wave radar mask image into a first channel of a YOLO network model backbone network and inputting an RGB image collected by a camera into a second channel of the YOLO network model backbone network, and embedding a frequency domain self-attention mechanism based on Fourier transform after the convolution layers at the same positions of the first channel and the second channel; performing multi-scale element-by-element fusion on the enhanced RGB image features output by the first channel and the enhanced millimeter wave radar mask image features output by the second channel to obtain a target class and a two-dimensional bounding box; determining the actual size and rotation angle of the target according to the two-dimensional bounding box, determining the three-dimensional coordinates and posture of the target according to the candidate depth of the millimeter wave radar point set falling within the two-dimensional bounding box.

2. The millimeter wave radar and camera fusion target detection method according to claim 1, wherein the millimeter wave radar mask image is obtained according to the obtained millimeter wave radar data, and the method comprises the following steps: performing rotation and translation operations on the millimeter wave radar points to obtain points in a laser radar coordinate system and expand them into homogeneous coordinates; projecting the homogeneous coordinate points onto an image through camera external and internal parameter matrices, performing coordinate transformation of the laser radar to the camera, projecting the transformed coordinates onto a pixel coordinate system, and calculating pixel coordinates; dividing the target into three categories of pedestrians, cyclists and vehicles according to the target speed; drawing a rectangular frame on the image as a mask according to the projected pixel coordinates; adjusting the size of the mask according to the distance from the vehicle; correcting the aspect ratio of the mask in combination with the motion direction of the target; using the scattering cross-sectional area to weight the color of the mask, and using different reference colors for pedestrians, cyclists and vehicles, and then adjusting the mask color according to the relative size of the scattering cross-sectional area to obtain the final millimeter wave radar mask image.

3. The millimeter wave radar and camera fusion target detection method according to claim 1, wherein the frequency domain self-attention mechanism based on Fourier transform comprises the following steps: extracting query vectors, key vectors and value vectors of a feature map output by a convolution layer through a convolution module; performing fast Fourier transform on the query vectors and the key vectors, and modeling correlation in the frequency domain to obtain frequency domain self-attention weights; multiplying the attention weights with the value vectors to obtain enhanced features; and performing residual connection between the enhanced features processed by a convolution layer and the original features to obtain the output of the frequency domain self-attention mechanism.

4. The millimeter wave radar and camera fusion target detection method according to claim 1, wherein the first channel and the second channel have the same structure and both comprise a first convolution layer, a second convolution layer, a first frequency domain self-attention mechanism module, a first C2f module, a third convolution layer, a second frequency domain self-attention mechanism module, a second C2f module, a fourth convolution layer, a third frequency domain self-attention mechanism module, a third C2f module, a fifth convolution layer, a fourth C2f module and an SPPF module connected in sequence.

5. The millimeter wave radar and camera fusion target detection method according to any one of claims 1-4, wherein ​ ​ ​ According to the candidate depth, the three-dimensional coordinates and the pose of the target are determined, comprising: According to the candidate depth and the known real depth, a linear constraint equation based on perspective projection is constructed; Enumerate various combinations of angles and 2D boundary coordinate edges, solve the center position T by least squares method, calculate the projection error of each combination, and add a radar depth constraint term to the projection error to construct an error objective function, select the combination with the minimum error objective function as the optimal result, and obtain the three-dimensional coordinates and the pose of the target.

6. The millimeter wave radar and camera fusion target detection method according to any one of claims 1-4, wherein, The two-dimensional bounding box comprises: the center point coordinates of the two-dimensional bounding box, the width and height of the two-dimensional bounding box, the confidence score of the existence of the target, and the class probability distribution. 7.A target detection system of millimeter wave radar and camera fusion, characterized in that, Comprising: The millimeter wave radar mask unit is configured to obtain a millimeter wave radar mask image according to the obtained millimeter wave radar data; The dual-channel enhanced feature extraction unit is configured to input the millimeter wave radar mask image into the first channel of the YOLO network model backbone network, and input the RGB image collected through the camera into the second channel of the YOLO network model backbone network, and the convolution layers at the same positions of the first channel and the second channel are embedded with a frequency domain self-attention mechanism based on Fourier transform; The two-dimensional target detection unit is configured to perform multi-scale element-by-element fusion on the enhanced RGB image features output by the first channel and the enhanced millimeter wave radar mask image features output by the second channel to obtain the target class and the two-dimensional bounding box; The three-dimensional target detection unit is configured to determine the actual size and rotation angle of the target according to the two-dimensional bounding box, determine the candidate depth of the target according to the millimeter wave radar point set falling within the two-dimensional bounding box, and determine the three-dimensional coordinates and the pose of the target according to the candidate depth.

8. A computer device, comprising: Comprising: A processor and a computer readable storage medium; A processor adapted to execute a computer program; A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor, and the millimeter wave radar and camera fusion target detection method according to any one of claims 1-6 is realized.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program is adapted to be loaded and executed by the processor, and the millimeter wave radar and camera fusion target detection method according to any one of claims 1-6 is realized.

10. A computer program product, characterised in that, The computer program product comprises a computer program, the computer program is executed by the processor, and the millimeter wave radar and camera fusion target detection method according to any one of claims 1-6 is realized.

Images