Radar-camera 3D target detection method and device based on semantic perception enhancement

Abstract

The invention provides a radar-camera 3D target detection method and device based on semantic perception enhancement, and the method comprises the steps: carrying out the feature extraction of multi-view camera image data, and projecting the extracted image features to a bird's-eye view space to generate camera BEV features; converting the millimeter wave radar point cloud data into radar BEV features; performing serialization window attention operation on the radar BEV features to obtain enhanced radar BEV features; predicting a foreground confidence coefficient for a non-empty position in the enhanced radar BEV feature, calculating a semantic radar scattering sectional area, and taking the area as an execution feature diffusion range to obtain a dense radar BEV feature; the camera BEV features and the dense radar BEV features are fused to obtain fused BEV features; and performing target detection based on the fused BEV features, and outputting a 3D detection result. Based on the method, the invention further provides corresponding equipment. According to the invention, synchronous improvement of target detection precision, robustness and practical efficiency is realized.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a radar-camera 3D target detection method and device based on semantic perception enhancement. Background Technology

[0002] 3D object detection is a key technology in environmental perception for autonomous driving and is of great significance to vehicle safety. Currently, multi-view cameras are widely used due to their low cost and rich semantic information, but pure camera systems have significant shortcomings in depth estimation accuracy and robustness under adverse weather conditions. To address these issues, millimeter-wave radar is integrated with multi-view cameras to create a more comprehensive and robust multimodal 3D detection solution.

[0003] However, millimeter-wave radar point clouds are characterized by extreme sparsity and non-uniform distribution, which limits the performance of radar-camera fusion detection. Specifically, radar echoes within a single object exhibit a highly fragmented distribution, making it difficult for the model to establish a coherent object-level feature representation; existing methods, when using radar cross-section to spatially expand radar features, cannot distinguish between foreground objects and background structures, leading to serious semantic ambiguity and reduced detection performance.

[0004] Existing technologies, such as RCBEVDet and RCTrans, attempt to capture long-range dependencies through self-attention mechanisms, but their secondary computational complexity makes them impractical for high-resolution bird's-eye view representations. Therefore, there is an urgent need for a radar-camera fusion detection method that can efficiently correlate fragmented radar point clouds and enhance semantically perceptual features. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a radar-camera 3D target detection method and device based on semantically perceptual enhancement. It can effectively correlate spatially disconnected radar point clusters and selectively perform semantically perceptual feature enhancement on the foreground, achieving efficient and robust multimodal 3D target detection.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention proposes a radar-camera 3D target detection method based on semantic awareness enhancement, comprising the following steps: Feature extraction is performed on the acquired multi-view camera image data, and the extracted image features are projected onto the bird's-eye view space to generate camera BEV features; The acquired millimeter-wave radar point cloud data is converted into radar BEV features, and a binary feature indication map is generated to indicate the location of effective features. The radar BEV features are obtained by performing sequential window attention operations along different spatial axes to associate spatially disconnected radar point clusters. To enhance the confidence of non-empty BEV position prediction in radar BEV features, and to calculate semantic radar cross section by combining radar cross section and depth information, adaptive feature diffusion is performed using semantic radar cross section as the adaptive diffusion range to obtain dense radar BEV features. The camera BEV features are fused with the dense radar BEV features to obtain fused BEV features; based on the fused BEV features, the target object category, 3D bounding box parameters and confidence score are predicted, and the final 3D detection result is output.

[0007] Secondly, the present invention also proposes a radar-camera 3D target detection method and device based on semantic perception enhancement, comprising at least one processor and a memory, wherein the memory stores a computer program, characterized in that the computer program implements the radar-camera 3D target detection method based on semantic perception enhancement when executed by the at least one processor.

[0008] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects: This invention proposes a radar-camera 3D target detection method and device based on semantic perception enhancement. The method includes the following steps: extracting features from acquired multi-view camera image data and projecting the extracted image features onto a bird's-eye view space to generate camera BEV features; converting acquired millimeter-wave radar point cloud data into radar BEV features and generating a binary feature indication map to indicate the location of effective features; associating spatially disconnected radar point clusters with the radar BEV features through sequential window attention operations along different spatial axes to obtain enhanced radar BEV features; predicting foreground confidence for non-empty BEV locations in the enhanced radar BEV features, and calculating the semantic radar cross section (RCS) by combining radar cross section and depth information, performing adaptive feature diffusion using the semantic RCS as the adaptive diffusion range to obtain dense radar BEV features; fusing the camera BEV features with the dense radar BEV features to obtain fused BEV features; predicting the target object category, 3D bounding box parameters, and confidence score based on the fused BEV features, and outputting the final 3D detection result. A corresponding device is also proposed based on this method. This invention effectively solves the key challenges in radar-camera fusion through feature association and semantic enhancement mechanisms, achieving simultaneous improvement in detection accuracy, robustness, and practical efficiency, and providing reliable technical support for high-performance 3D environmental perception in fields such as autonomous driving and intelligent robots.

[0009] This invention designs axial decomposition perception attention, performing sequential window attention calculations along the x and y axes of the bird's-eye view space. This enables the modeling of long-range dependencies with low computational complexity, effectively associating spatially disconnected but belonging-to-the-same-object radar point clusters at the feature level, thereby forming a more complete and consistent target-level feature representation and providing high-quality input for subsequent detection.

[0010] This invention introduces semantic-aware radar enhancement. First, a semantic scoring network predicts the confidence level that a non-empty radar feature location belongs to a foreground target. Then, this confidence level, the original radar cross section (RCS), and target depth information are fused to generate a semantically aware radar cross section (SRCS). Adaptive feature diffusion guided by the SRCS enables features to be reasonably diffused primarily within the foreground target region, greatly reducing the introduction of background noise and significantly improving the semantic purity and target recognition accuracy of the features. Attached Figure Description

[0011] Figure 1 This is a flowchart of the radar-camera 3D target detection method based on semantic awareness enhancement proposed in Embodiment 1 of the present invention; Figure 2 This is an architecture diagram of the radar-camera 3D target detection method based on semantic awareness enhancement proposed in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the radar-camera 3D target detection device based on semantic perception enhancement proposed in Embodiment 2 of the present invention. Detailed Implementation

[0012] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0013] Example 1 The radar-camera 3D target detection method based on semantic perception enhancement proposed in Embodiment 1 of this invention is used to solve the problems of sparse radar point clouds, fragmented distribution, and semantic confusion between foreground and background in the existing radar-camera fusion 3D target detection technology.

[0014] This invention can be applied to fields such as autonomous driving and intelligent robotics. The input to the radar-camera fusion 3D target detection task includes millimeter-wave radar point cloud data and multi-view image data, while the output is the three positions, size, orientation, and category of the object of interest.

[0015] Figure 1 This is a flowchart of the radar-camera 3D target detection method based on semantic awareness enhancement proposed in Embodiment 1 of the present invention; Figure 2 This is an architecture diagram of the radar-camera 3D target detection method based on semantic awareness enhancement proposed in Embodiment 1 of the present invention; combined with Figure 1 and Figure 2 The process of implementing this invention will be explained together.

[0016] In step S1, multi-view camera image data and millimeter-wave radar point cloud data are acquired; This invention deploys a multi-view camera system on an autonomous vehicle or mobile platform, typically including multiple cameras such as front, rear, left, and right cameras, forming a 360-degree surround view coverage. The intrinsic and extrinsic parameters of each camera are pre-calibrated. Simultaneously, it is equipped with at least one millimeter-wave radar sensor, usually located at the front or around the vehicle, whose scanning range covers key areas around the vehicle. Furthermore, the cameras and radar are synchronized in time, using hardware triggering or timestamp-based software synchronization mechanisms to ensure that the collected data is aligned in time.

[0017] The camera system acquires multi-view image data at a fixed frame rate (e.g., 10-30 FPS). Each frame contains RGB three-channel information, and the image resolution is usually 1600×900 or higher.

[0018] Millimeter-wave radar simultaneously acquires point cloud data. Each frame of data contains information about multiple radar points. The attributes of each point include at least: three-dimensional coordinates, radial velocity, radar cross section (RCS), timestamp, and other optional attributes.

[0019] The collected multi-view camera image data and millimeter-wave radar point cloud data are then subjected to spatiotemporal alignment and coordinate unification, as well as the removal of abnormal data.

[0020] In step S2, feature extraction is performed on the acquired multi-view camera image data, and the extracted image features are projected onto the bird's-eye view space to generate camera BEV features; For each viewpoint of camera image data, a camera encoder is used to extract multi-scale image features; the image size for each viewpoint is [size missing]. ,in The height of the image representing each viewpoint; The width of the image representing each viewpoint.

[0021] Then, through viewpoint transformation, the multi-scale image features are projected onto a unified bird's-eye view space to generate camera BEV features; the dimensions of the camera BEV features are... ;in, Indicates the height of the BEV space; Indicates the width of the BEV space; The number of channels representing camera features.

[0022] In step S3, the acquired millimeter-wave radar point cloud data is converted into radar BEV features, and a binary feature indication map is generated to indicate the location of effective features. The shape and size of millimeter-wave radar point cloud data are ; Represents the number of radar points; The attribute dimensions representing each radar point (including three-dimensional coordinates, velocity, radar cross section (RCS), etc.). The radar point cloud is converted into a BEV feature representation using the Pillar coding method. Specifically, the BEV space is divided into... For each grid cell, a multilayer perceptron is used to learn point-level features for radar points within a non-empty grid cell. Then, max pooling is used to aggregate the features of all points within that grid cell to obtain the radar BEV features. Size is ; The number of channels representing radar characteristics; Simultaneously retain a binary feature indicator map. Size is ;in, Indicates position It contains valid radar signatures.

[0023] In step S4, the radar BEV features are correlated with spatially disconnected radar point clusters through sequential window attention operations along different spatial axes to obtain enhanced radar BEV features.

[0024] By sequentially performing window attention operations along different spatial axes, spatially disjoint radar point clusters are efficiently correlated. This involves three steps: window construction, window attention, and window reassembly. Window construction: Given radar BEV features First, the BEV space is divided into multiple vertical windows along the x-axis, with the number of windows being... ,in This is the window width. Each window spans the full BEV height. and fixed width To adapt to the sparsity of radar point clouds, a feature indication map-based approach is used. From each window, at most K valid positions are sampled to obtain the window feature sequence. Size is If there are fewer than K valid positions, the sequence is zero-padded.

[0025] Window attention: window feature sequence Processing is achieved through a multi-head linear attention mechanism, where ReLU activation functions are applied to both the query and the key to achieve linear complexity. Learnable positional encoding is introduced to capture absolute and relative spatial relationships. The positional encoding consists of two parts: absolute positional encoding... Represents the absolute coordinates normalized to [0,1] in the global BEV space, and is used for relative position encoding. This represents the relative offset from the center of the local window. The two position codes are concatenated and then processed through a multilayer perceptron and layer normalization to obtain the final position code. Window attention output enhances window features.

[0026] Window Reconstruction: After window attention processing, the enhanced window features are scattered back to their original spatial locations, reconstructing the BEV feature representation. Since the windows are non-overlapping, each location only receives features from a single window, making the reconstruction process efficient and yielding enhanced x-axis features.

[0027] For y-axis processing, a similar axial window attention process is used to divide the BEV space into multiple horizontal windows along the y-axis direction. That is, the radar BEV features enhanced along the x-axis are divided into multiple horizontal windows along the y-axis direction. The effective feature positions within each horizontal window are sampled to form a window feature sequence, and multi-head linear attention calculation is performed on the window feature sequence to associate spatially disconnected radar point clusters within the same horizontal window.

[0028] By stacking multiple decomposition axis perception attention mechanisms, the receptive field is gradually expanded, ultimately resulting in enhanced radar BEV features. .

[0029] In step S5, the foreground confidence of the non-empty BEV position in the enhanced radar BEV features is predicted, and the semantic radar cross section is calculated by combining the radar cross section and depth information. The semantic radar cross section is used as the adaptive diffusion range to perform adaptive feature diffusion, resulting in dense radar BEV features. By jointly utilizing physical signal properties and semantic priors, sparse radar BEV features are selectively transformed into dense region-level geometric representations.

[0030] For the enhanced radar BEV features, a semantic scoring network is used for each BEV location containing radar features. Forecast confidence score ,in ; The semantic scoring network is implemented using a multilayer perceptron, and the output is processed through a sigmoid activation function to obtain a confidence score. This network is trained using an auxiliary foreground segmentation loss, with the loss function being a binary classification cross-entropy loss. For positions located within the 3D bounding box Its soft label Generate using the following formula based on centrality: ; in, Indicates position Soft tags; Indicates position Distance to the front surface of the bounding box; Indicates position Distance to the back surface of the bounding box; Indicates position Distance to the left surface of the bounding box; Indicates position Distance to the right surface of the bounding box; Indicates position The distance to the top surface of the bounding box; Indicates position The distance to the bottom surface of the bounding box; for positions outside the 3D bounding box, the soft label is set to 0.

[0031] To accurately estimate the spatial extent to which each sparse radar feature should diffuse, the semantic radar cross section (SRCS) is introduced as an adaptive diffusion range. semantic radar cross section It is calculated by integrating physical RCS attributes, foreground confidence, and depth information.

[0032] Based on the radar cross section at this location The foreground confidence score and the radar point depth corresponding to that location. Calculate the semantic radar cross section at this location. The specific formula is as follows: ; in, This indicates that it is a hyperparameter that controls the degree of emphasis on semantic scores; This represents the maximum detection distance. The depth normalization term takes into account the sparser distribution of points at greater distances, enabling a larger diffusion range for distant objects.

[0033] Based on SRCS, feature diffusion is performed to expand finite radar features into a spatially continuous dense representation in the BEV space.

[0034] For each BEV location containing valid radar signatures Diffusion of its characteristics to the diffusion radius Adjacent positions within; for satisfying BEV position , from diffusion characteristics contribution The calculation is as follows: ; in, Indicates position Radar BEV characteristics; For a BEV location that receives feature contributions from multiple radar points, sum all the received spread feature contributions to obtain the final feature of that location. A Gaussian kernel weighted by SRCS ensures that features from semantically relevant foreground locations diffuse within an appropriate spatial range, ultimately yielding dense radar BEV features. .

[0035] In step S6, the camera BEV features are fused with the dense radar BEV features to obtain fused BEV features; The camera BEV features and the dense radar BEV features are concatenated along the channel dimension to obtain the concatenated multimodal features. A convolution operation is then performed on the concatenated multimodal features to achieve interaction and fusion between features, ultimately resulting in the fused BEV features.

[0036] In step S7, the target object's category, 3D bounding box parameters, and confidence score are predicted based on the fused BEV features, and the final 3D detection result is output.

[0037] This invention uses a convolutional neural network to predict the class probability, 3D bounding box parameters, and confidence score for each BEV location, and obtains the final 3D detection result through non-maximum suppression post-processing.

[0038] The fused BEV features are input into a 3D detection head based on a convolutional neural network; the detection head outputs a set of predicted values ​​at each spatial location in the BEV space, specifically including: class probability distribution, 3D bounding box parameters, and confidence score.

[0039] Post-processing is performed on all predictions output by the detection head, specifically including: filtering out predictions with confidence levels below a preset confidence threshold; for the remaining prediction boxes, based on the overlap between boxes calculated using the 3D intersection-union ratio, suppressing prediction boxes belonging to the same category and whose overlap exceeds a set threshold, retaining only the prediction boxes with the highest confidence; and outputting the remaining prediction boxes after filtering and suppression, along with their corresponding categories, bounding box parameters, and confidence levels, as the final 3D detection result.

[0040] The total loss function of this invention is expressed as: ; in, This includes classification loss, bounding box regression loss, and orientation loss; This represents the auxiliary segmentation loss of the semantic scoring network.

[0041] The radar-camera 3D target detection method based on semantic perception enhancement proposed in Embodiment 1 of this invention effectively solves the key problems in radar-camera fusion through feature association and semantic enhancement mechanisms, and achieves simultaneous improvement in detection accuracy, robustness and practical efficiency, providing reliable technical support for high-performance 3D environmental perception in fields such as autonomous driving and intelligent robots.

[0042] Example 2 The present invention also proposes a device, Figure 3 This is a schematic diagram of the radar-camera 3D target detection device based on semantic perception enhancement proposed in Embodiment 2 of the present invention.

[0043] At the hardware level, electronic device 300 includes a processor 310, and optionally, an internal bus 320, a network interface 330, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or it may also include non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its functions. The processor 310, network interface 330, and memory can be interconnected via an internal bus 320. This internal bus 320 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus. The memory is used to store programs. Specifically, the program can include program code, which includes computer operation instructions. The memory can include main memory 340 and non-volatile memory 350, and provides instructions and data to the processor 310. Processor 310 reads the corresponding computer program from non-volatile memory 350 into memory 340 and then runs it, forming a device for locating the target user at the logical level. Processor 310 executes the program stored in memory and specifically performs the following: In step S1, multi-view camera image data and millimeter-wave radar point cloud data are acquired; In step S2, feature extraction is performed on the acquired multi-view camera image data, and the extracted image features are projected onto the bird's-eye view space to generate camera BEV features; In step S3, the acquired millimeter-wave radar point cloud data is converted into radar BEV features, and a binary feature indication map is generated to indicate the location of effective features. In step S4, the radar BEV features are correlated with spatially disconnected radar point clusters through sequential window attention operations along different spatial axes to obtain enhanced radar BEV features. In step S5, the foreground confidence of the non-empty BEV position in the enhanced radar BEV features is predicted, and the semantic radar cross section is calculated by combining the radar cross section and depth information. The semantic radar cross section is used as the adaptive diffusion range to perform adaptive feature diffusion, resulting in dense radar BEV features. In step S6, the camera BEV features are fused with the dense radar BEV features to obtain fused BEV features; In step S7, the target object's category, 3D bounding box parameters, and confidence score are predicted based on the fused BEV features, and the final 3D detection result is output.

[0044] Figure 1It can be applied to processor 810, or implemented by processor 810. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the processor or by instructions in the form of software. The processor mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0045] Example 3 The present invention also proposes a readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the following method steps: In step S1, multi-view camera image data and millimeter-wave radar point cloud data are acquired; In step S2, feature extraction is performed on the acquired multi-view camera image data, and the extracted image features are projected onto the bird's-eye view space to generate camera BEV features; In step S3, the acquired millimeter-wave radar point cloud data is converted into radar BEV features, and a binary feature indication map is generated to indicate the location of effective features. In step S4, the radar BEV features are correlated with spatially disconnected radar point clusters through sequential window attention operations along different spatial axes to obtain enhanced radar BEV features. In step S5, the foreground confidence of the non-empty BEV position in the enhanced radar BEV features is predicted, and the semantic radar cross section is calculated by combining the radar cross section and depth information. The semantic radar cross section is used as the adaptive diffusion range to perform adaptive feature diffusion, resulting in dense radar BEV features. In step S6, the camera BEV features are fused with the dense radar BEV features to obtain fused BEV features; In step S7, the target object's category, 3D bounding box parameters, and confidence score are predicted based on the fused BEV features, and the final 3D detection result is output.

[0046] Embodiment 3 of this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory that stores a computer program, which can be executed by a processor to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0047] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0048] The descriptions of relevant parts in the semantically enhanced radar-camera 3D target detection device provided in Embodiment 2 of this application and the semantically enhanced radar-camera 3D target detection storage medium in Embodiment 3 of this application can be found in the detailed descriptions of the corresponding parts in the semantically enhanced radar-camera 3D target detection method provided in Embodiment 1 of this application, and will not be repeated here.

[0049] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0050] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A radar-camera 3D target detection method based on semantic awareness enhancement, characterized in that, Includes the following steps: Feature extraction is performed on the acquired multi-view camera image data, and the extracted image features are projected onto the bird's-eye view space to generate camera BEV features; The acquired millimeter-wave radar point cloud data is converted into radar BEV features, and a binary feature indication map is generated to indicate the location of effective features. The radar BEV features are obtained by performing sequential window attention operations along different spatial axes to associate spatially disconnected radar point clusters. To enhance the confidence of non-empty BEV position prediction in radar BEV features, and to calculate semantic radar cross section by combining radar cross section and depth information, adaptive feature diffusion is performed using semantic radar cross section as the adaptive diffusion range to obtain dense radar BEV features. The camera BEV features are fused with the dense radar BEV features to obtain fused BEV features; based on the fused BEV features, the target object category, 3D bounding box parameters and confidence score are predicted, and the final 3D detection result is output.

2. The method according to claim 1, characterized in that, Feature extraction is performed on the acquired multi-view camera image data, and the extracted image features are projected onto the bird's-eye view space to generate camera BEV features; Specifically: For camera image data from each viewpoint, multi-scale image features are extracted using a camera encoder; Then, through viewpoint transformation, the multi-scale image features are projected onto a unified bird's-eye view space to generate camera BEV features; the dimensions of the camera BEV features are... ;in, Indicates the height of the BEV space; Indicates the width of the BEV space; The number of channels representing camera features.

3. The method according to claim 1, characterized in that, The acquired millimeter-wave radar point cloud data is converted into radar BEV features, and a binary feature indication map is generated to indicate the location of effective features; specifically, the millimeter-wave radar point cloud data is converted into radar BEV features using the histogram coding method. The shape and size of the millimeter-wave radar point cloud data are as follows: ; Represents the number of radar points; The attribute dimensions representing each radar point; Divide the BEV space into For each radar point within a grid, a multilayer perceptron is used to learn point-level features, and then max pooling is used to aggregate the features of all points within that grid to obtain the radar BEV features. Size is ; The number of channels representing radar characteristics; Simultaneously retain a binary feature indicator map. Size is ;in, Indicates position It contains valid radar signatures.

4. The method according to claim 1, characterized in that, The radar BEV features are subjected to sequential window attention operations along different spatial axes to associate spatially disjoint radar point clusters, resulting in enhanced radar BEV features; specifically: The radar BEV feature is divided into multiple vertical windows along the x-axis. The effective feature positions within each window are sampled to form a window feature sequence. Multi-head linear attention is then performed on the window feature sequence to associate spatially disconnected radar point clusters within the same vertical window. The enhanced window features are then reassembled back to their original spatial positions to obtain the radar BEV feature enhanced along the x-axis. The enhanced radar BEV features along the x-axis are divided into multiple horizontal windows along the y-axis. The effective feature positions within each horizontal window are sampled to form a window feature sequence. Multi-head linear attention is then performed on the window feature sequence to associate spatially disconnected radar point clusters within the same horizontal window. The enhanced window features are then reassembled back to their original spatial positions to obtain the enhanced radar BEV features.

5. The method according to claim 1, characterized in that, The process of performing multi-head linear attention computation on the window feature sequence also includes: A learnable positional encoding is introduced into the window feature sequence. The positional encoding consists of two parts: an absolute positional encoding and a relative positional encoding. The absolute positional encoding represents the normalized absolute coordinates in the global bird's-eye view space, and the relative positional encoding represents the relative offset relative to the local window center. The two positional encodings are concatenated and processed by a multilayer perceptron and layer normalization to obtain the final positional encoding used for attention calculation.

6. The method according to claim 1, characterized in that, To enhance the foreground confidence of non-empty BEV position prediction in radar BEV features, semantic radar cross section (RCS) is calculated by combining radar cross section and depth information; specifically: For the enhanced radar BEV features, a semantic scoring network is used for each BEV location containing radar features. Forecast confidence score ,in ; Based on the radar cross section at this location The foreground confidence score and the radar point depth corresponding to that location. Calculate the semantic radar cross section at this location. The specific formula is as follows: ； in, This indicates that it is a hyperparameter that controls the degree of emphasis on semantic scores; This indicates the maximum detection distance.

7. The method according to claim 6, characterized in that, The process of predicting a foreground confidence score for each BEV location containing radar features is as follows: The semantic scoring network is supervised during training using an auxiliary foreground segmentation loss, which is a binary classification cross-entropy loss; for locations within the 3D bounding box... Its soft label Generate using the following formula based on centrality: ； in, Indicates position Soft tags; Indicates position Distance to the front surface of the bounding box; Indicates position Distance to the back surface of the bounding box; Indicates position Distance to the left surface of the bounding box; Indicates position Distance to the right surface of the bounding box; Indicates position The distance to the top surface of the bounding box; Indicates position Distance to the bottom surface of the bounding box.

8. The method according to claim 6, characterized in that, Adaptive feature diffusion is performed using the semantic radar cross-section as the adaptive diffusion range to obtain dense radar BEV features, specifically: For each BEV location containing valid radar signatures to diffuse its characteristics To the diffusion radius Adjacent positions within; For satisfying BEV position , from diffusion characteristics contribution The calculation is as follows: ； in, Indicates position Radar BEV characteristics; For a BEV location that receives feature contributions from multiple radar points, sum all the received spread feature contributions to obtain the final feature of that location. By performing the above diffusion and summation operations on all radar points, a dense radar BEV feature is finally obtained. .

9. The method according to claim 1, characterized in that, Based on the fused BEV features, the target object's category, 3D bounding box parameters, and confidence score are predicted, and the final 3D detection result is output. Specifically, a convolutional neural network is used to predict the category probability, 3D bounding box parameters, and confidence score for each BEV location, and the final 3D detection result is obtained through non-maximum suppression post-processing.

10. A radar-camera 3D target detection method and device based on semantic perception enhancement, comprising at least one processor and a memory, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the at least one processor, it implements the radar-camera 3D target detection method based on semantic awareness enhancement as described in any one of claims 1 to 9.

Images