A confidence-guided based target detection method, device and medium

Abstract

The application discloses a target detection method and device based on confidence guidance and a medium, relates to the technical field of target detection, and has the technical scheme as follows: radar point cloud data and a visible light image are acquired; a semantic feature vector and a physical attribute feature vector of the radar point cloud data are extracted, the semantic feature vector and the physical attribute feature vector are spliced, and a point cloud feature vector is obtained; a basic confidence and a multi-path factor of the point cloud feature vector are calculated, a point-level confidence of each point cloud in the radar point cloud data is synthesized according to the basic confidence and the multi-path factor, and a confidence heat map is generated according to the point-level confidence of each point cloud; an image feature vector of the visible light image is extracted by a network; the image feature vector and the point cloud feature vector are dynamically weighted and fused according to the confidence heat map, and a fusion feature vector is obtained; and target detection is performed on the fusion feature vector, and a target detection result is obtained.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and more specifically, to a confidence-guided target detection method, device, and medium. Background Technology

[0002] The most common sensor in autonomous vehicles is the camera, which provides high-resolution visual signals. Unfortunately, traditional vision-based object detection often suffers severe performance degradation in foggy and low-light conditions, leading to missed detections or low accuracy. Compared to cameras, millimeter-wave radar (MWR) can easily penetrate snow, smoke, and dust, and can adapt to different environmental conditions, operating in all weather. However, MWR also has a significant drawback: its detection density is insufficient to represent the physical characteristics of a target. Intuitively, fusing MWR and camera data can complement each other, providing richer and more comprehensive information, thereby improving the vehicle's perception system's detection capabilities under various environmental and climatic conditions.

[0003] Existing radar and camera fusion target detection methods suffer from the following problems: First, radar data is inherently noisy due to multipath reflections, high ambiguity, and other factors, making it difficult for traditional methods to distinguish between valid targets and false signals. Second, mainstream fusion frameworks (such as BEVFusion) employ static weight allocation mechanisms, which cannot respond to dynamic fluctuations in sensor quality. Fixed-weight fusion cannot adapt to dynamic scene changes (such as strong light, rain, and fog) and ignores the spatial differences in sensor reliability. Summary of the Invention

[0004] The purpose of this invention is to provide a confidence-guided target detection method, device, and medium, which solves the problems of existing radar and camera fusion detection methods being unable to distinguish between valid targets and false targets and unable to adapt to changes in multiple scenarios.

[0005] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0006] A first aspect of the present invention provides a confidence-guided target detection method, the method comprising:

[0007] Acquire radar point cloud data and visible light images;

[0008] Extract the semantic feature vector and physical attribute feature vector from the radar point cloud data, and concatenate the semantic feature vector and physical attribute feature vector to obtain the point cloud feature vector.

[0009] The base confidence and multipath factor of the point cloud feature vector are calculated, and the point-level confidence of each point cloud in the radar point cloud data is synthesized based on the base confidence and multipath factor. A confidence heatmap is generated based on the point-level confidence of each point cloud.

[0010] The network extracts image feature vectors from visible light images;

[0011] The image feature vector and the point cloud feature vector are dynamically weighted and fused based on the confidence heatmap to obtain the fused feature vector.

[0012] The fused feature vector is used for target detection to obtain the target detection results.

[0013] In one implementation, the PointNet network is used to extract semantic feature vectors from radar point cloud data;

[0014] The physical property feature vector includes radar cross section, velocity component, and altitude.

[0015] In one implementation, a fully connected layer and a ReLU activation function are used to concatenate the semantic feature vector and the physical attribute feature vector to obtain the point cloud feature vector.

[0016] In one implementation, a fully connected layer and a sigmoid function are used to calculate the base confidence of the point cloud feature vectors.

[0017] In one implementation, a two-layer fully connected network is used to calculate the multipath factor of the point cloud feature vector; wherein, the first fully connected network includes a fully connected layer with a weight matrix of 64*64 and a ReLU activation function, and the second fully connected network includes a fully connected layer with a weight vector of 64 and a Sigmoid function.

[0018] In one implementation scheme, the formula for calculating the point-level confidence of each point cloud in the synthesized radar point cloud data based on the base confidence and multipath factor is as follows: ,in, Let i be the point-level confidence level of the i-th point cloud. Based on confidence level, As a multi-path factor, This is the inhibition coefficient.

[0019] In one implementation scheme, the formula for generating a confidence heatmap based on the point-level confidence score of each point cloud is as follows: ,in, The standard deviation of the Gaussian kernel. For the confidence heatmap, (x, y) are the grid coordinates of the bird's-eye view. Let i be the point-level confidence level of the i-th point cloud. Let be the two-dimensional coordinates of the i-th point in the radar point cloud on the bird's-eye view plane.

[0020] In one implementation, before dynamically weighting and fusing the image feature vector and the point cloud feature vector according to the confidence heatmap to obtain the fused feature vector, the method further includes: aligning the dimension of the image feature vector with the confidence heatmap using a deformable cross-attention mechanism.

[0021] A second aspect of the present invention provides an electronic device, including a memory and a processor;

[0022] A memory for storing computer programs, the computer programs including program instructions;

[0023] A processor for executing the program instructions to cause the electronic device to perform steps of a confidence-guided target detection method as provided in the first aspect.

[0024] A third aspect of the present invention provides a computer-readable storage medium comprising a computer program that, when executed by one or more processors, implements a confidence-guided target detection method as provided in the first aspect.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] In the confidence-guided target detection method provided by this invention, physical attributes (such as radar cross section RCS, velocity components, and height information) and semantic features in radar point clouds are jointly modeled. A dual-branch structure (basic confidence branch + multipath detection branch) is used to evaluate the confidence of each point. This mechanism can effectively identify noise points caused by multipath reflections, improving the accuracy of target detection. Secondly, when constructing a confidence heatmap based on point-level confidence, this invention designs a Gaussian kernel function that varies with point cloud height to dynamically adjust the scattering radius, enabling the system to more reasonably perceive the spatial importance of high and low targets (such as pedestrians and streetlights), thereby improving the heatmap quality and fusion accuracy. Finally, using the confidence heatmap as a weight guide, a deformable cross-attention mechanism is used to achieve spatial alignment between camera image features (projected onto the BEV grid) and radar features. The offset is modulated by the confidence heatmap, improving the fusion effect in high-confidence areas. Furthermore, a channel-level dynamic weighting strategy is adopted to adjust the feature fusion ratio of different modalities in real time, thus solving the problems of existing technologies being unable to distinguish between valid and false targets and unable to adapt to changes in multiple scenarios. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0028] Figure 1This is a flowchart illustrating a confidence-guided target detection method provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0030] It should be noted that the terms "comprising" or "may include" used in the various embodiments of this application indicate the presence of the claimed function, operation, or element, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in the various embodiments of this application, the terms "comprising," "having," and their cognates are intended only to indicate a specific feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or adding one or more combinations of the foregoing.

[0031] It should be understood that terms such as "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0032] Figure 1 This is a flowchart illustrating a confidence-guided target detection method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0033] S101 acquires radar point cloud data and visible light images.

[0034] In this embodiment, point cloud data can be acquired by millimeter-wave radar, and visible light images can be acquired by a camera component, which can be a CMOS image sensor.

[0035] S102, extract the semantic feature vector and physical attribute feature vector of the radar point cloud data, and concatenate the semantic feature vector and physical attribute feature vector to obtain the point cloud feature vector.

[0036] In this embodiment, the PointNet network is used to extract semantic feature vectors from radar point cloud data. Physical property feature vector This includes radar cross section, velocity components, and altitude.

[0037] It should be noted that semantic feature vectors and physical attribute feature vectors are extracted for each point cloud in the radar point cloud data.

[0038] The semantic and physical attribute features of each point are concatenated into a 68-dimensional vector. Then, a fully connected layer (weight matrix) is used. The dimensionality is reduced, and then passed through the ReLU activation function to obtain a 64-dimensional fused feature vector. The two types of features are then concatenated and subjected to nonlinear transformation in the fusion layer to obtain the point cloud feature vector. , This represents the feature vector of the point cloud, and ReLU is the ReLU activation function.

[0039] S103, calculate the basic confidence level and multipath factor of the point cloud feature vector, and synthesize the point-level confidence level of each point cloud in the radar point cloud data based on the basic confidence level and multipath factor, and generate a confidence heatmap based on the point-level confidence level of each point cloud.

[0040] In this embodiment, regarding the basic confidence level, a fully connected layer and a sigmoid function are used to calculate the basic confidence level of the point cloud feature vector, i.e. ;in, This is the Sigmoid function, with output values ​​in the interval [0,1]. T represents the matrix transpose. This is a fully connected layer with a weight vector of 64.

[0041] Regarding the multipath factor, this embodiment employs a two-layer fully connected network to calculate the multipath factor of the point cloud feature vector. The first fully connected layer includes a 64*64 weight matrix and a ReLU activation function, while the second fully connected layer includes a 64-weight matrix and a Sigmoid function. The multipath factor is calculated as follows:

[0042] ;in, This is a fully connected layer with a weight matrix of 64*64. This is a fully connected layer with a weight vector of 64. This represents the noise probability caused by the multipath effect at that point, i.e., the multipath factor.

[0043] The formula for calculating the point-level confidence of each point in the synthesized radar point cloud data based on the base confidence and multipath factor is as follows: ,in, Let i be the point-level confidence level of the i-th point cloud. Based on confidence level, As a multi-path factor, The suppression coefficient, .

[0044] Before calculating the confidence heatmap, scattering is performed using a height-adaptive Gaussian kernel, and the dynamic Gaussian kernel standard deviation is calculated for each point: ,in, It is an empirical coefficient. Let be the point cloud height of the i-th point cloud. is the Gaussian kernel standard deviation.

[0045] The BEV plane is divided into a grid, and the value of each BEV grid is calculated using a maximum response strategy. Specifically, the formula for generating a confidence heatmap based on the point-level confidence of each point cloud is: ,in, The standard deviation of the Gaussian kernel. For the confidence heatmap, (x, y) are the grid coordinates of the bird's-eye view. Let i be the point-level confidence level of the i-th point cloud. Let be the two-dimensional coordinates of the i-th point in the radar point cloud on the bird's-eye view plane.

[0046] The value range [0, 1] reflects the confidence level that a valid target exists at this BEV grid location. This confidence heatmap will be used to guide subsequent fusion and computational allocation.

[0047] S104, the extraction network extracts the image feature vector of the visible light image.

[0048] In this embodiment, the image feature vector of the visible light image is extracted using the ResNet-50 network. The ResNet-50 network is a conventional image feature extraction network, and this embodiment will not provide further explanation of it.

[0049] S105, dynamically weight and fuse the image feature vector and the point cloud feature vector according to the confidence heatmap to obtain the fused feature vector.

[0050] In this embodiment, since the confidence heatmap and image feature vector have different dimensions, direct weighted fusion processing is not possible. This embodiment employs the LSS (Lift, Splat, Shoot) method or its variants to project the camera feature maps of each viewpoint onto a unified BEV space. A deformable cross-attention mechanism is used to dynamically align the features of the two modalities. Radar-to-camera alignment uses radar features as the query and camera features as the key-value pair. Camera-to-radar alignment uses camera features as the query and radar features as the key-value pair. ,in, , For query vector, , The coordinates are the reference point (BEV grid center).

[0051] Introducing confidence heatmaps Guide offset prediction and improve calibration accuracy in high-confidence regions: ,in, Confidence information is injected into the query vector, causing the offset to favor high-confidence target regions. The offset is dynamically generated by a multilayer perceptron (MLP) without the need for a pre-defined fixed convolution kernel.

[0052] The alignment process is achieved through a deformable cross-attention mechanism: Where M is the number of attention heads and K is the number of sampling points; The spatial offset modulated by the confidence heatmap; Attention weights are calculated through interaction between queries and key-value pairs. ,in, It is a key-value vector. Feature dimension scaling factor (default) ).

[0053] The aligned features are dynamically weighted and fused at the channel level of the heatmap weights: ,in, This is the feature vector obtained after radar features are aligned using deformable cross-attention. The feature vector is the camera features aligned by deformable cross attention.

[0054] S106, Perform target detection on the fused feature vector to obtain the target detection result.

[0055] In this embodiment, to address the bottleneck of computational efficiency, the present invention divides the detection area into three processing strategies based on the confidence level of each BEV unit in the heatmap: (1) a complete high-precision detection head is used in high-confidence areas; (2) a lightweight detection head is used in medium-confidence areas; and (3) detection is skipped in low-confidence areas and compensated by interpolation. This strategy significantly improves inference efficiency and reduces system power consumption.

[0056] For example, based on confidence heatmaps Select processing strategy:

[0057] like Then, the 256-channel detection head will be used for complete calculation.

[0058] like A 128-channel lightweight detection head is then used.

[0059] like If the calculation is skipped, the grid cell that skips the calculation will output an empty detection result, which will be compensated by the detection head fusion module through interpolation.

[0060] Target detection uses the CenterPoint detector head to output bounding box parameters.

[0061] In summary, the confidence-guided target detection method provided by this invention jointly models the physical attribute features (such as radar cross section RCS, velocity components, and height information) and semantic features in radar point clouds. A dual-branch structure (basic confidence branch + multipath detection branch) is used to evaluate the confidence of each point. This mechanism effectively identifies noise points caused by multipath reflections, improving the accuracy of target detection. Secondly, when constructing a confidence heatmap based on point-level confidence, this invention designs a Gaussian kernel function that varies with point cloud height to dynamically adjust the scattering radius. This allows the system to more reasonably perceive the spatial importance of high and low targets (such as pedestrians and streetlights), thereby improving the heatmap quality and fusion accuracy. Finally, using confidence heatmaps as weight guides, spatial alignment of camera image features (projected onto BEV grids) and radar point cloud features is achieved through a deformable cross-attention mechanism. The offset is modulated by the confidence heatmap, which improves the fusion effect in high-confidence areas. Furthermore, a channel-level dynamic weighting strategy is adopted to adjust the feature fusion ratio of different modalities in real time, thereby solving the problems of existing technologies being unable to distinguish between valid targets and false targets and being unable to adapt to changes in multiple scenarios.

[0062] Therefore, this invention, by combining physical attributes and semantic features through a dual-branch confidence model, is expected to effectively distinguish between real targets and multipath noise, thereby improving the reliability of radar data. Furthermore, based on a highly adaptive heatmap, the dynamic modulation of multimodal feature fusion weights theoretically adapts to sensor quality fluctuations in complex environments such as rain, fog, and strong light. Simultaneously, the confidence-driven computational resource allocation mechanism is expected to significantly reduce computational overhead in low-confidence regions, improving system real-time performance while maintaining detection accuracy. In addition, the modular design allows the solution to be extended to other sensor combinations (such as LiDAR-infrared sensors) and resource-constrained detection platforms.

[0063] The target detection method provided by this invention is applied to a real-world scenario, specifically a typical urban tunnel entrance, where the environment is characterized by alternating bright and dark light interference and multipath reflection from the tunnel sidewalls. Strong light outside the tunnel causes camera overexposure, while shadows inside the tunnel and the metal sidewalls trigger radar multipath reflection. Targets include vehicles ahead (real targets), ghosting shadows on the sidewalls (noise), and low-lying pedestrians.

[0064] First, the radar captures 200 point cloud data points, which are then input into a dual-branch confidence model: the semantic branch extracts features using an improved PointNet++, with the points representing vehicles ahead exhibiting high response values. Dimension 64), the side wall reflection point has abnormal physical properties (RCS value < 0.5, vertical velocity). Mutations are marked by physical branches; multipath detection branches identify sidewall reflection points as noise ( ), its confidence level was suppressed to below 0.1; real vehicle points ( After multi-path factor correction, a high confidence level of 0.94 was retained; pedestrian points are significant due to their height characteristics. The confidence level was 0.85.

[0065] A bird's-eye view heatmap is constructed based on point-level confidence: the height-adaptive Gaussian kernel is used for vehicle points. ) allocate a larger scattering radius ( ), pedestrian point ( ) adopts a compact core ( This ensures that low-lying targets are not overlooked; the confidence heatmap clearly marks the center of the lane as the high confidence zone. The sidewall region is suppressed due to multipath noise. ), pedestrian location generates medium confidence region ( The camera image is overexposed in bright areas and sharp in shadow areas, and the deformable cross-attention dynamic alignment is guided by a confidence heatmap:

[0066] Strong light area ( Offset Modulated by heatmap, radar features are preferentially aligned, and the fusion weights are biased towards radar ( );

[0067] Shadow area ( ): Increase the camera weight to 0.75 to compensate for the sparsity of radar point clouds;

[0068] Pedestrian area ( ): Balanced fusion of dual-modal features to avoid missed detections.

[0069] BEV mesh processing strategy based on confidence heatmap: high confidence zone in the center of the lane ( The 256-channel full-precision detection head is activated to output the precise vehicle position and orientation; the confidence zone in the shoulder ( It adopts a 128-channel lightweight detection head, saving 50% of computing power while still detecting pedestrians; the low confidence area on the side wall ( The calculation is skipped and compensated by interpolation from adjacent grids to avoid false detections.

[0070] This invention also provides an electronic device. The electronic device includes a processor, a memory, a communication interface, and at least one communication bus for connecting the processor, the memory, and the communication interface. The memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (PROM), or portable read-only memory (CD-ROM), and is used for related instructions and data.

[0071] The communication interface is used to receive and send data. The processor can be one or more CPUs; if it is a single-core CPU, it can be a multi-core CPU. The processor in the electronic device reads one or more programs stored in memory and performs the following operations: acquires radar point cloud data and visible light images; extracts semantic feature vectors and physical attribute feature vectors from the radar point cloud data, concatenates them to obtain point cloud feature vectors; calculates the base confidence and multipath factor of the point cloud feature vectors, and synthesizes the point-level confidence of each point cloud in the radar point cloud data based on the base confidence and multipath factor, generating a confidence heatmap based on the point-level confidence of each point cloud; extracts image feature vectors from the visible light images using a network; dynamically weights and fuses the image feature vectors and point cloud feature vectors based on the confidence heatmap to obtain a fused feature vector; and performs target detection on the fused feature vector to obtain the target detection result.

[0072] It should be noted that the specific implementation of each operation can be described above. Figure 1 The corresponding description of the method embodiments shown indicates that the electronic device can be used to execute a confidence-guided target detection method based on the above method embodiments of this application, which will not be described in detail here.

[0073] This invention also provides a computer-readable storage medium, which is a memory device in a computer device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of a terminal. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the confidence-guided target detection method in the above embodiments. Those skilled in the art should understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This invention also provides a computer program product containing program instructions. The computer program product may be software or program products containing program instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one electronic device, it causes the at least one electronic device to perform a confidence-guided target detection method.

[0075] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A target detection method based on confidence level guidance, characterized in that the method... include: Acquire radar point cloud data and visible light images; Extract the semantic feature vector and physical attribute feature vector from the radar point cloud data, and concatenate the semantic feature vector and physical attribute feature vector to obtain the point cloud feature vector. The basic confidence score and multipath factor of the point cloud feature vector are calculated, and the point-level confidence score of each point cloud in the radar point cloud data is synthesized based on the basic confidence score and multipath factor. A confidence heatmap is generated based on the point-level confidence score of each point cloud. Specifically, a fully connected layer and a sigmoid function are used to calculate the basic confidence score of the point cloud feature vector. A two-layer fully connected network is used to calculate the multipath factor of the point cloud feature vector. The first fully connected network layer includes a fully connected layer with a weight matrix of 64*64 and a ReLU activation function, and the second fully connected network layer includes a fully connected layer with a weight vector of 64 and a sigmoid function. The formula for calculating the point-level confidence of each point in the synthesized radar point cloud data based on the base confidence and multipath factor is as follows: ,in, Let i be the point-level confidence level of the i-th point cloud. Based on confidence level, As a multi-path factor, The inhibition coefficient; The formula for generating a confidence heatmap based on the point-level confidence score of each point cloud is as follows: ,in, The standard deviation of the Gaussian kernel. For the confidence heatmap, (x, y) are the grid coordinates of the bird's-eye view. Let i be the point-level confidence level of the i-th point cloud. Let be the two-dimensional coordinates of the i-th point in the radar point cloud on the bird's-eye view plane; The network extracts image feature vectors from visible light images; The image feature vector and the point cloud feature vector are dynamically weighted and fused based on the confidence heatmap to obtain the fused feature vector. The fused feature vector is used for target detection to obtain the target detection results.

2. The target detection method based on confidence level guidance according to claim 1, characterized in that, The PointNet network is used to extract semantic feature vectors from radar point cloud data; The physical property feature vector includes radar cross section, velocity component, and altitude.

3. The target detection method based on confidence level guidance according to claim 1, characterized in that, A fully connected layer and a ReLU activation function are used to concatenate the semantic feature vector and the physical attribute feature vector to obtain the point cloud feature vector.

4. The target detection method based on confidence level guidance according to claim 1, characterized in that, Before dynamically weighting and fusing the image feature vector and the point cloud feature vector based on the confidence heatmap to obtain the fused feature vector, the method further includes: aligning the dimensions of the image feature vector with the confidence heatmap using a deformable cross-attention mechanism.

5. An electronic device, characterized in that, Including memory and processor; A memory for storing computer programs, the computer programs including program instructions; A processor for executing the program instructions to cause the electronic device to perform the steps of a confidence-guided target detection method as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed by one or more processors, implements a confidence-guided target detection method as described in any one of claims 1 to 4.

Images