A Degradation Probability-Guided Multi-View 3D Target Detection Method

Abstract

This invention discloses a degradation-aware guided multi-view 3D target detection method. It acquires multi-view image sequences of the current frame and historical frames captured by a camera, extracts multi-scale image features and pixel-level depth. A degradation probability map is generated based on the pixel differences between the reconstructed image obtained by upsampling the multi-scale image features and the input image. Spatiotemporal feature compensation is performed on the multi-scale image features to obtain degradation enhancement features. High-confidence degradation centers are extracted from the degradation probability map, and an adaptive 3D query is generated by combining pixel-level depth and camera intrinsic and extrinsic parameters. This adaptive query is then combined with a learnable 3D query to form a bi-branch 3D query. The bi-branch 3D query is interactively fused, and the fused 3D query interacts with the current frame image features and degradation enhancement features to obtain fused features. The 3D target detection result is then output through a detection head. This invention can enhance the representation of target structure and foreground information in complex scenes, improving the robustness and localization stability of 3D target detection.

Description

Technical Field

[0001] This invention relates to the technical fields of computer vision and autonomous driving perception, and in particular to a degradation probability-guided multi-view 3D target detection method. Background Technology

[0002] Multi-view 3D target detection is a key technology in autonomous driving and robotic environmental perception. Camera-based 3D detection schemes are characterized by low hardware cost, convenient deployment, and rich semantic information. In real road environments, images in multi-view image sequences may be affected by low illumination, inclement weather, motion blur, or camera imaging anomalies, and image frames may also be missing. The aforementioned image quality degradation weakens texture, edge, and structural cues in the image, reducing the distinguishability of distant targets, occluded targets, and targets with impaired visual cues; missing image frames also reduce the continuity of temporal information, leading to missed target detection, increased 3D localization errors, and noise propagation during cross-view feature aggregation.

[0003] Existing multi-view 3D detection methods typically rely on the assumption that key image region features have good discriminative power, or improve 3D target detection performance by interacting and aggregating multi-view image features through 2D priors, semantic priors, and attention mechanisms. However, in complex degradation scenarios, the degree of degradation and feature reliability vary significantly across different spatial locations of the image. Existing methods often employ global or weakly spatially targeted enhancement strategies, making it difficult to accurately assess local quality and adaptively compensate for severely damaged areas. Furthermore, during cross-view feature fusion, directly aggregating features from degraded and normal regions may incorrectly assign higher weights to noise-contaminated views or regions, causing degradation noise to propagate along the query and sampling paths, leading to decreased stability in target classification and 3D localization. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a degradation probability-guided multi-view 3D target detection method. By explicitly modeling the degree of local degradation of the image and performing spatiotemporal feature compensation on the degradation region, the robustness and localization stability of the model in 3D target detection in complex open environments are improved.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a degradation probability-guided multi-view three-dimensional target detection method, comprising the following steps:

[0006] 1) Obtain a multi-view image sequence captured by the camera, extract multi-scale image features of the multi-view image sequence through a backbone network and a feature pyramid network, and obtain pixel-level depth through a depth estimation network; wherein, the multi-view image sequence includes multi-view images of the current frame and multi-view images of historical frames;

[0007] 2) Input the multi-scale image features into the degradation perception module to obtain a degradation probability map, which is used to characterize the degree of degradation of the corresponding image in spatial location; wherein, the degradation perception module generates a reconstructed image by upsampling the multi-scale image features, and generates the degradation probability map based on the pixel difference between the reconstructed image and the corresponding image in the multi-view image sequence;

[0008] 3) Based on the degradation probability map, perform spatiotemporal feature compensation on the multi-scale image features corresponding to the multi-view images of the current frame and the multi-scale image features corresponding to the multi-view images of the historical frames in the multi-view image sequence to enhance the spatiotemporal feature expression of the degradation region and obtain degradation enhancement features.

[0009] 4) Extract the high-confidence degradation center from the degradation probability map, and combine it with the pixel-level depth and the intrinsic and extrinsic parameters of the camera to convert the high-confidence degradation center into a 3D anchor center; generate an adaptive 3D query based on the 3D anchor center; combine the adaptive 3D query with a preset learnable 3D query to obtain a dual-branch 3D query;

[0010] 5) The dual-branch 3D query is interactively fused using a self-attention mechanism to obtain a fused 3D query; the fused 3D query is then interacted with the multi-scale image features corresponding to the multi-view image of the current frame and the degradation enhancement features to obtain the original feature context and the degradation enhancement feature context; the original feature context and the degradation enhancement feature context are mixed in channel dimension and point dimension to obtain fused features; the fused features are then input into the detection head to output the 3D target detection result.

[0011] Further, in step 1), feature extraction is performed on the multi-view images of the current frame and the multi-view images of the historical frames in the multi-view image sequence. Image features of the multi-view image sequence are extracted through a backbone network, and the image features are input into a feature pyramid network for multi-scale feature fusion to obtain multi-scale image features with different spatial resolutions. At the same time, the multi-view image sequence is input into a depth estimation network to predict pixel-level depth.

[0012] Further, in step 2), the multi-scale image features are input into the degradation perception module to obtain a degradation probability map. Specifically, the following steps are performed: an image feature at a preset scale level is selected from the multi-scale image features; the image feature at the preset scale level is upsampled level by level using two learnable upsampling operators, each of which includes a transposed convolution with a stride of 2 and a ReLU activation function; the upsampled image feature at the preset scale level is then convolved and activated sequentially with a Sigmoid function to obtain a reconstructed image with the same spatial size as the input image feature at the preset scale level; and the degradation probability map is generated based on the pixel differences between the reconstructed image and the corresponding input image.

[0013] Furthermore, step 3) includes the following steps:

[0014] For the pixel coordinates of any viewpoint in the multi-view image of the current frame According to the pixel-level depth corresponding to the pixel coordinates And the corresponding camera's intrinsic parameter matrix I, the pixel coordinates The back projection is a 3D point P in the camera coordinate system, expressed by the following formula:

[0015] ;

[0016] In the formula, Represents pixel coordinates x-coordinate Represents pixel coordinates The ordinate, Represents pixel coordinates pixel-level depth, This represents the inverse of the intrinsic parameter matrix of the corresponding camera, and T represents the matrix transpose;

[0017] For each scale level , will the current frame's first Image features at each scale level are denoted as , will the first in the previous frame Image features at each scale level are denoted as ;in, Indicates the scale-level index. Indicates the current frame time. The previous frame time is represented; N three-dimensional sampling offsets for each three-dimensional point P are obtained through linear network regression, and N sampling points are determined based on the N three-dimensional sampling offsets, where N is a preset number of sampling points; the N sampling points are projected onto the corresponding image planes in the current frame and the previous frame, respectively, and mapped to the . The feature map coordinates corresponding to each scale level are obtained by bilinear interpolation for the current frame. The Middle Sampling characteristics of each sampling point And the previous frame The Middle Sampling characteristics of each sampling point ;

[0018] According to the current frame, the first Image features at each scale level Attention weights for each sampling point are predicted using a linear layer and a softmax activation function. For the current frame The Middle Sampling characteristics of each sampling point And the previous frame The Middle Sampling characteristics of each sampling point Perform weighted aggregation to obtain the first... Aggregated features at each scale level The formula is expressed as follows:

[0019] ;

[0020] In the formula, MLP represents a multilayer perceptron used to fuse and stitch together sampled features. This represents the summation operation. Indicates the sampling point number; Concat indicates the concatenation operation.

[0021] The degradation probability map M is downsampled to the level of the current frame. Image features at each scale level and the aggregation feature With the same spatial resolution, a downsampling degradation probability map is obtained. ; using the downsampling degradation probability map As a soft attention gating, the aggregated features Modulate; combine the modulated aggregated features with the image features Add the residuals to obtain the first... Degradation enhancement features at each scale level The formula is expressed as follows:

[0022] ;

[0023] In the formula, ⊙ represents element-wise multiplication.

[0024] Further, in step 4), the degradation probability map is filtered using a preset threshold, and pixel positions with degradation probabilities not less than the preset threshold are determined as high-confidence degradation centers; for each high-confidence degradation center, its two-dimensional coordinates are obtained. and the corresponding pixel-level depth Based on the intrinsic and extrinsic parameters of the corresponding camera, the data is upscaled to three-dimensional space to obtain the center of the three-dimensional anchor point. The formula is expressed as follows:

[0025] ;

[0026] In the formula, This represents the inverse of the camera's intrinsic parameter matrix. The inverse matrix of the camera's extrinsic parameters is... The two-dimensional coordinates representing the high-confidence degradation center. This represents the x-coordinate of the two-dimensional coordinate system. This represents the ordinate of the two-dimensional coordinate system. This represents the pixel-level depth corresponding to the two-dimensional coordinates, and T represents the matrix transpose. This indicates taking the first to third components of the vector within the parentheses;

[0027] The degradation enhancement features are sampled at the high-confidence degradation center, and the obtained sampled features are input into a multilayer perceptron to regress the target size, orientation, and velocity. The three-dimensional spatial anchor center, as well as the regressed target size, orientation, and velocity, are used as query parameters to form an adaptive three-dimensional query. Then, the adaptive three-dimensional query is combined with a preset learnable three-dimensional query to form a two-branch three-dimensional query.

[0028] Furthermore, step 5) includes the following steps:

[0029] The preset learnable 3D query and the adaptive 3D query are interactively fused using a self-attention mechanism to obtain a fused 3D query. Based on the fused 3D query Each query in the process regresses multiple adaptive sampling offsets; based on these adaptive sampling offsets, for each scale level... From the current frame, respectively Image features at each scale level and the first Degradation enhancement features at each scale level The local context features are sampled to obtain the first... Local context feature tensors of original feature branches at each scale level and degenerate enhancement feature branch local context feature tensor ; convert the original feature branches of each scale level into local context feature tensors Add the corresponding elements together to obtain the original feature context. ; the local context feature tensor of the degradation enhancement feature branch at each scale level Adding the corresponding elements together yields the degenerate-enhanced feature context. ; the original feature context and the context of the degradation enhancement features By concatenating the components, we obtain a unified feature tensor F, expressed by the following formula:

[0030] ;

[0031] In the formula, Concat represents the concatenation operation;

[0032] Based on the fused 3D query A semantic weight matrix is ​​generated through a linear layer. and using the semantic weight matrix The channel importance of the unified feature tensor F is recalibrated, and then channel dimension mixing is performed to obtain semantic denoising features. The formula is expressed as follows:

[0033] ;

[0034] ;

[0035] In the formula, Linear represents a linear layer, LN represents layer normalization, and · represents matrix multiplication;

[0036] Based on the fused 3D query Generate geometric weight matrix and using the geometric weight matrix The local geometric relationships of the sampling points are aggregated to achieve point-dimensional blending, resulting in point-position blended features. The formula is expressed as follows:

[0037] ;

[0038] ;

[0039] In the formula, T represents the matrix transpose;

[0040] Then, the point-to-point blending features are processed through a linear layer. Aggregation is performed to obtain fusion features. Finally, the fusion features will be... Input the detection head, and the three-dimensional target detection results will be output.

[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0042] 1. This invention reconstructs multi-scale image features using a degradation perception module to obtain a reconstructed image. A degradation probability map is generated based on the pixel differences between the reconstructed image and the corresponding images in the multi-view image sequence. This allows image quality degradation caused by low light, fog, motion blur, or camera anomalies to be explicitly characterized in terms of spatial location and degree of degradation. Targeted spatiotemporal feature compensation is then applied to degraded regions based on the degradation probability map. This reduces interference with the original features of non-degraded regions, preserving the original texture, structure, and geometric information of clear regions while supplementing information in degraded regions, thereby improving the targeting and reliability of feature compensation.

[0043] 2. This invention performs spatiotemporal feature compensation on multi-scale image features corresponding to the current frame and historical frames in a multi-view image sequence based on the degradation probability map. Effective contextual information from different times is introduced into the degradation region to obtain degradation enhancement features, thus supplementing and restoring the foreground semantics, texture details, and edge structures weakened by image degradation. Furthermore, this invention generates a 3D anchor point center and an adaptive 3D query based on a high-confidence degradation center, pixel-level depth, and the camera's intrinsic and extrinsic parameters. The adaptive 3D query is combined with a preset learnable 3D query to form a bi-branch 3D query. This allows the 3D query to consider both the prior global target distribution and the local spatial information of the degradation region, which is beneficial for improving the localization and recognition capabilities of distant targets, occluded targets, and targets with damaged textures and edge cues, and reducing missed detections in image degradation environments.

[0044] 3. This invention uses a self-attention mechanism to interactively fuse bi-branch 3D queries, and then uses the fused 3D query to interact with the multi-scale image features and degradation enhancement features corresponding to the current frame to obtain the original feature context and the degradation enhancement feature context. Based on this, the channel importance of the unified feature tensor is recalibrated through channel-dimensional mixing to form semantic denoising features. Furthermore, local geometric relationships between different sampling points are aggregated through point-dimensional mixing. This comprehensively utilizes the original reliable information and the compensated enhancement information, suppressing the propagation of degradation noise during multi-view feature aggregation, maintaining the consistency of global semantic information and local geometric information, thereby improving the expressive power of the fused features and the stability, robustness, and adaptability of 3D object detection in complex degradation scenes. Attached Figure Description

[0045] Figure 1 This is an overall framework diagram of the method of the present invention.

[0046] Figure 2 A schematic diagram of the structure for generating a degradation probability map for the degradation perception module; in the diagram, Sigmoid represents the Sigmoid activation function, ReLU represents the ReLU activation function, and the "-" circle represents pixel-by-pixel subtraction.

[0047] Figure 3 This is a schematic diagram of the spatiotemporal feature compensation process for degradation probability perception; in the diagram, ⊙ represents element-wise multiplication; and "+" circles represent element-wise addition.

[0048] Figure 4 This is a flowchart illustrating the process of obtaining fused features using the fused 3D query. Detailed Implementation

[0049] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0050] like Figure 1 As shown, this embodiment discloses a degradation probability-guided multi-view 3D target detection method. This method uses multi-view image sequences of the current frame and historical frames acquired by a camera as input. Multi-scale image features are extracted through a backbone network and a feature pyramid network, and pixel-level depths corresponding to each pixel position in each view image are predicted through a depth estimation network. Subsequently, the multi-scale image features are input into a degradation perception module to generate a degradation probability map characterizing the degree of degradation in the image's spatial location. Based on the degradation probability map, spatiotemporal feature compensation is performed on the current frame features and historical frame features to obtain degradation enhancement features. Further, an adaptive 3D query is generated based on the degradation probability map, pixel-level depth, and camera intrinsic and extrinsic parameters, and combined with a preset learnable 3D query to form a bi-branch 3D query. After interactive fusion of the bi-branch 3D query through a self-attention mechanism, the fused 3D query interacts with the multi-scale image features and degradation enhancement features of the current frame, respectively. Then, channel dimension mixing and point dimension mixing are performed to obtain fused features. Finally, the fused features are input into a classification and regression detection head to output 3D target detection results.

[0051] In this embodiment, the current frame is recorded as time. The historical frame is selected as the previous frame, i.e., the time frame. The system acquires images of the current and previous frames from various camera perspectives and inputs them into the backbone network and feature pyramid network, respectively, to obtain multi-scale image features with different spatial resolutions. Simultaneously, the multi-view image sequence is input into a depth estimation network to obtain pixel-level depth for each pixel location. These multi-scale image features provide semantic and spatial structure information at different resolutions, while the pixel-level depth is used for subsequent adaptive 3D query generation and spatiotemporal feature compensation.

[0052] like Figure 2 As shown, this embodiment generates a degradation probability map through a degradation perception module. Taking any viewpoint image in a multi-view image sequence as an example, image features with a spatial resolution of one-quarter of the original image are selected from the multi-scale image features corresponding to that viewpoint image, denoted as . If the spatial resolution of the corresponding original image is , Represents high, The width represents the image feature; The spatial resolution is:

[0053] ;

[0054] The image features Used to reconstruct the corresponding input image, characterizing the degree of feature degradation at different spatial locations of the image. The image features... The input degradation sensing module consists of two levels of learnable upsampling operators. Each level of learnable upsampling operator includes a step size of... The transposed convolution and ReLU activation function. Let the initial input features... for:

[0055] ;

[0056] No. The upsampling output is denoted as ,in, The stepwise upsampling process can be represented as:

[0057] ;

[0058] In the formula, Indicates the first The output features of the learnable upsampling operator at each level Indicates the first The output features of the learnable upsampling operator. Indicates the first The step size is transposed convolution, and They represent the first The first and second convolutional layers in the learnable upsampling operator; ReLU represents the ReLU activation function. The transposed convolution doubles the spatial resolution of the feature map, and two ordinary convolutions and ReLU activation are used to further extract the upsampled image features.

[0059] After the first level of upsampling, the height and width of the feature map are both increased from one-quarter to one-half of the corresponding size of the original image. After the second level of upsampling, the height and width of the feature map are increased to have the same spatial resolution as the corresponding original image. Subsequently, convolution and sigmoid activation are applied to the output of the second level of upsampling to obtain the reconstructed image. The formula is expressed as follows:

[0060] ;

[0061] In the formula, This represents the convolution operation. This represents the Sigmoid activation function.

[0062] Reconstruct the image With the corresponding input image Pixel-level differences are calculated, and mean aggregation is performed along the channel dimension to obtain a degradation probability map. :

[0063] ;

[0064] In the formula, This indicates mean aggregation along the image channel dimension. Degradation probability map. The values ​​at each location are used to characterize the degree of degradation at the corresponding spatial location: the larger the difference value, the more difficult it is for the texture, edge, or structural information at that location to be recovered from the image features, and the higher the degree of degradation at that location; the smaller the difference value, the more complete the image information at that location is.

[0065] like Figure 3 As shown, after obtaining the degradation probability map Then, spatiotemporal feature compensation is performed on the multi-scale image features corresponding to the multi-view images of the current frame and the previous frame based on the degradation probability map. For the pixel coordinates of any view image in the current frame... According to the pixel-level depth corresponding to the pixel coordinates And the corresponding camera's intrinsic parameter matrix I, the pixel coordinates The back projection is a 3D point P in the camera coordinate system, expressed by the following formula:

[0066] ;

[0067] In the formula, Represents pixel coordinates x-coordinate Represents pixel coordinates The ordinate, Represents pixel coordinates pixel-level depth, This represents the inverse of the intrinsic parameter matrix of the corresponding camera, and T represents the matrix transpose;

[0068] For each scale level Using a linear network to analyze three-dimensional points return Each 3D sampling offset is used to determine the corresponding 3D sampling point, and each 3D sampling point is projected onto the image plane corresponding to the current frame and the previous frame, respectively, and then mapped to the 3D sampling offset. The feature map coordinates at each scale level are obtained. The current frame's first (i) feature map coordinates are then obtained through bilinear interpolation. Sampling characteristics of each sampling point And the previous frame Sampling characteristics of each sampling point .

[0069] According to the current frame number Image features at each scale level Predicting the first step using a linear layer and the Softmax activation function. Attention weights at each sampling point of each scale level The formula is expressed as follows:

[0070] ;

[0071] In the formula, the first The attention weights corresponding to each sampling point are denoted as follows: The current frame sampling features and the previous frame sampling features at the same sampling point are concatenated and then weighted and aggregated according to attention weights to obtain the first... Aggregated features at each scale level The formula is expressed as follows:

[0072] ;

[0073] In the formula, N is the preset number of sampling points. This represents a multilayer perceptron used to fuse spliced ​​sampled features. This indicates a splicing operation. The aggregated features simultaneously contain local visual information from the current frame and compensation information from historical frames corresponding to that spatial location, thereby restoring weakened foreground semantics and structural cues to the degraded region.

[0074] To avoid causing unnecessary feature perturbation to clear regions, the degradation probability map is... Downsampled to the current frame Image features at each scale level and aggregated features have the same spatial resolution, resulting in a downsampling degradation probability map. The downsampling degradation probability map As a soft attention gating, the aggregated features Modulate; combine the modulated aggregated features with the image features Add the residuals to obtain the first... Degradation enhancement features at each scale level The formula is expressed as follows:

[0075] ;

[0076] In the formula, ⊙ represents element-wise multiplication. When the degradation probability of a certain region is low, the gate value is small, and the injection of aggregated features is suppressed to maintain the original clear texture and structure; when the degradation probability of a certain region is high, the gate value is large, and spatiotemporal aggregated features are selectively introduced to compensate for the feature attenuation caused by degradation. Through this residual gating method, this embodiment can reduce the disturbance to normal regions while restoring the information of degraded regions.

[0077] After completing the generation of degradation enhancement features, based on the degradation probability map An adaptive 3D query is generated and combined with a learnable 3D query to form a bi-branch 3D query. Specifically, a threshold of 0.1 is used to adjust the degradation probability map. The pixel locations with a degradation probability not less than a threshold of 0.1 were selected as high-confidence degradation centers; for each high-confidence degradation center, its two-dimensional coordinates were obtained. and the corresponding pixel-level depth Based on the intrinsic and extrinsic parameters of the corresponding camera, the data is upscaled to three-dimensional space to obtain the center of the three-dimensional anchor point. The formula is expressed as follows:

[0078] ;

[0079] In the formula, This represents the inverse of the camera's intrinsic parameter matrix. The inverse matrix of the camera's extrinsic parameters is... The two-dimensional coordinates representing the high-confidence degradation center. This represents the x-coordinate of the two-dimensional coordinate system. This represents the ordinate of the two-dimensional coordinate system. This represents the pixel-level depth corresponding to the two-dimensional coordinates, and T represents the matrix transpose. This indicates taking the first to third components of the vector within the parentheses;

[0080] The degradation enhancement features are sampled at the high-confidence degradation center. The sampled features are input into a multilayer perceptron to regress the target size, orientation, and velocity. The 3D spatial anchor center, along with the regressed target size, orientation, and velocity, are used as query parameters to form an adaptive 3D query. This adaptive 3D query is then combined with a preset learnable 3D query to form a bi-branch 3D query. The learnable 3D query expresses the target distribution and geometric patterns in the dataset, while the adaptive 3D query provides local, degradation-aware query initialization for potential foreground targets within the degradation region. The bi-branch 3D queries are then interactively fused using a self-attention mechanism to obtain the fused 3D query. .

[0081] like Figure 4As shown, for each query in the fused 3D query, regression... An adaptive sampling offset is used, and local context features are sampled from the multi-scale image features and degradation enhancement features of the current frame, respectively, based on the sampling offset. For the first... At each scale level, the local context features of the original feature branches are obtained. Degradation enhancement feature branch local context features The context features at each scale level are summed element-wise to obtain the original feature context. and degradation enhancement feature context The formula is expressed as follows:

[0082] ;

[0083] ;

[0084] In the formula, L represents the total number of scale levels of the multi-scale image features. The original feature context and the degraded / enhanced feature context are concatenated to obtain a unified feature tensor. The formula is expressed as follows:

[0085] ;

[0086] In this embodiment, the number of three-dimensional queries is set to be... The number of feature channels is Then the dimension of the unified feature tensor F satisfies the following formula:

[0087] ;

[0088] In the formula, Representing the real number field, a semantic weight matrix is ​​generated through a linear layer based on the fused 3D query. Furthermore, the importance of the unified feature tensor is recalibrated along the channel dimension, and then channel-dimensional mixing is performed to obtain semantic denoising features, expressed by the following formula:

[0089] ;

[0090] ;

[0091] In the formula, Linear represents a linear layer. The representation layer normalization operation. The channel dimension blending can enhance semantic channels related to the foreground target and suppress unreliable channel responses caused by degradation noise.

[0092] After obtaining semantic denoising features, a geometric weight matrix is ​​generated based on the fused 3D query. Furthermore, local geometric relationships are aggregated along the sampling point dimension to obtain point-based hybrid features. The formula is expressed as follows:

[0093] ;

[0094] ;

[0095] In the formula, T represents the matrix transpose. By blending point dimensions, local geometric cues from different sampling points, perspectives, and feature branches can be integrated, reducing the impact of a single degenerate perspective or anomalous sampling points on the detection results. Subsequently, a linear layer aggregates the point-level blended features to obtain the fused features. :

[0096] ;

[0097] The fused features are input into the classification and regression detection head, and the output is a three-dimensional target detection result; wherein, the three-dimensional target detection result includes target category, three-dimensional center position, target size, orientation and velocity.

[0098] To verify the detection accuracy and robustness to degradation scenes of the degradation probability-guided multi-view 3D object detection method described in this embodiment, experiments were conducted on the nuScenes dataset and the nuScenes-C dataset, respectively. nuScenes is a large-scale dataset for 3D object detection in autonomous driving, containing 1000 driving scenes, each lasting approximately 20 seconds, and providing 3D object annotations at a frequency of 2Hz. Each scene includes images acquired by six surround-view cameras, covering the area around the vehicle. This embodiment uses the nuScenes standard evaluation protocol to detect ten types of 3D targets, including vehicles, pedestrians, and bicycles. nuScenes-C is a robustness test dataset for degraded scenes built upon nuScenes, including eight image degradation types: brightness variation, low light, fog, snow, motion blur, color quantization, camera malfunction, and frame loss. Each degradation type has three degradation intensity levels. The nuScenes-C dataset can be used to evaluate the detection stability of 3D target detection methods under different image degradation conditions.

[0099] This embodiment uses the mean accuracy (mAP), nuScenes detection score (NDS), and mean resilience (mRR) as the main evaluation indicators. The mean accuracy (mAP) measures the overall detection accuracy of different categories of 3D targets. For each target category and for each threshold with center distances of 0.5m, 1m, 2m, and 4m, the corresponding mean accuracy is calculated based on the precision-recall curve of the detection results. Then, the arithmetic mean of the mean accuracy obtained for all target categories and all center distance thresholds is calculated to obtain the mean accuracy (mAP). The larger the mAP, the higher the accuracy of 3D target classification and localization.

[0100] The nuScenes Detection Score (NDS) is a comprehensive evaluation index for 3D target detection performance. In addition to considering mAP, it also considers the following five true positive error indices: mean position error (mATE), mean size error (mASE), mean orientation error (mAOE), mean velocity error (mAVE), and mean attribute error (mAAE).

[0101] Among them, mATE measures the average translational distance between the predicted target center and the actual target center; mASE measures the average difference between the predicted target size and the actual target size; mAOE measures the average angular difference between the predicted orientation and the actual orientation; mAVE measures the average difference between the predicted velocity and the actual velocity; and mAAE measures the average error in target attribute prediction. For all five error metrics, a smaller value indicates a more accurate detection result. The NDS calculation formula is expressed as:

[0102] ;

[0103] In the formula, This represents the set of the five error indicators; This indicates that the smaller value between 1 and the current error index value e is selected. The larger the NDS, the better the overall performance of the model in terms of target classification, 3D position, size, orientation, velocity, and attribute prediction.

[0104] The mean resilience ratio (mRR) measures the detection performance retained by the model under degraded conditions relative to clear image conditions. For each degradation type, the arithmetic mean of the NDS corresponding to the three degradation levels of the model under that degradation type is taken, and the ratio of this average value to the NDS obtained by the model on clear image data is determined as the resilience ratio (RR) for that degradation type. The arithmetic mean of the resilience ratios for all degradation types is taken to obtain the mean resilience ratio (mRR). The larger the mRR, the higher the detection performance retained by the model under image degradation conditions. In a set of experiments in this embodiment, ResNet-50 or ResNet-101 is used as the backbone network, and class-balanced group sampling and test-phase data augmentation are not used during the experiments. First, on the nuScenes validation set, the method described in this embodiment is compared with multi-view 3D object detection methods such as BEVDet4D, SOLOFusion, Sparse4Dv2, SparseBEV, and StreamPETR. The experimental results are shown in Table 1.

[0105] Table 1. 3D object detection results on the nuScenes validation set.

[0106]

[0107] As shown in Table 1, when using the ResNet-50 backbone network and When considering the input image resolution, this embodiment achieves 45.6% mAP and 56.1% NDS, the highest results among the listed methods. Compared to SparseBEV, this embodiment improves mAP and NDS by 0.8 percentage points and 0.3 percentage points, respectively; compared to StreamPETR, it improves by 0.6 percentage points and 1.1 percentage points, respectively.

[0108] Using ResNet-101 backbone network and At the given input image resolution, this embodiment achieves a mAP of 51.0% and an NDS of 59.6%. Compared to SparseBEV, mAP and NDS are improved by 0.9 and 0.4 percentage points, respectively; compared to the suboptimal results of the listed methods, they are improved by 0.5 and 0.2 percentage points, respectively. These results demonstrate that under the experimental conditions, this embodiment can improve the overall performance of multi-view 3D target detection under different backbone networks and input image resolutions.

[0109] Furthermore, this embodiment is compared with DETR3D, PETR, SOLOFusion, and SparseBEV on the nuScenes-C dataset to evaluate the detection robustness of different methods under image degradation conditions. The average robustness ratio (mRR) of different methods is shown in Table 2, where a higher mRR indicates that the model retains higher detection performance under degradation conditions.

[0110] Table 2 Comparison results of mRR in nuScenes-C degradation scenarios

[0111]

[0112] As shown in Table 2, this embodiment achieves an average toughness ratio (mRR) of 76.4%. Compared to SparseBEV, the average toughness ratio of this embodiment is improved by 0.3 percentage points, and the toughness ratio (RR) under camera failure, low light, and fog conditions is improved by 0.4, 3.2, and 0.9 percentage points, respectively. Notably, this embodiment achieves a toughness ratio of 76.5% under low light conditions, which is higher than the listed comparison methods, indicating that this embodiment can retain more effective detection information even when low light causes attenuation of image texture and foreground semantic information.

[0113] Based on the experimental results in Tables 1 and 2, under the experimental conditions, this embodiment can not only improve mAP and NDS in normal scenarios, but also reduce disturbance errors under camera failure, low light and fog conditions, and improve detection toughness under degradation conditions.

[0114] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A degradation probability-guided multi-view 3D target detection method, characterized in that, Includes the following steps: 1) Obtain a multi-view image sequence captured by the camera, extract multi-scale image features of the multi-view image sequence through a backbone network and a feature pyramid network, and obtain pixel-level depth through a depth estimation network; wherein, the multi-view image sequence includes multi-view images of the current frame and multi-view images of historical frames; 2) Input the multi-scale image features into the degradation perception module to obtain a degradation probability map, which is used to characterize the degree of degradation of the corresponding image in spatial location; wherein, the degradation perception module generates a reconstructed image by upsampling the multi-scale image features, and generates the degradation probability map based on the pixel difference between the reconstructed image and the corresponding image in the multi-view image sequence; 3) Based on the degradation probability map, perform spatiotemporal feature compensation on the multi-scale image features corresponding to the multi-view images of the current frame and the multi-scale image features corresponding to the multi-view images of the historical frames in the multi-view image sequence to enhance the spatiotemporal feature expression of the degradation region and obtain degradation enhancement features. 4) Extract the high-confidence degradation center from the degradation probability map, and combine it with the pixel-level depth and the intrinsic and extrinsic parameters of the camera to convert the high-confidence degradation center into a 3D anchor center; generate an adaptive 3D query based on the 3D anchor center; combine the adaptive 3D query with a preset learnable 3D query to obtain a dual-branch 3D query; 5) The dual-branch 3D query is interactively fused using a self-attention mechanism to obtain a fused 3D query; the fused 3D query is then interacted with the multi-scale image features corresponding to the multi-view image of the current frame and the degradation enhancement features to obtain the original feature context and the degradation enhancement feature context; the original feature context and the degradation enhancement feature context are mixed in channel dimension and point dimension to obtain fused features; the fused features are then input into the detection head to output the 3D target detection result.

2. The degradation probability-guided multi-view three-dimensional target detection method according to claim 1, characterized in that, In step 1), feature extraction is performed on the multi-view images of the current frame and the multi-view images of the historical frames in the multi-view image sequence. Image features of the multi-view image sequence are extracted through a backbone network, and the image features are input into a feature pyramid network for multi-scale feature fusion to obtain multi-scale image features with different spatial resolutions. At the same time, the multi-view image sequence is input into a depth estimation network to predict pixel-level depth.

3. The degradation probability-guided multi-view three-dimensional target detection method according to claim 1, characterized in that, In step 2), the multi-scale image features are input into the degradation perception module to obtain a degradation probability map. Specifically, the following steps are performed: an image feature at a preset scale level is selected from the multi-scale image features; the image feature at the preset scale level is upsampled step by step using two levels of learnable upsampling operators, each level of learnable upsampling operator including a transposed convolution with a stride of 2 and a ReLU activation function; the upsampled image feature at the preset scale level is then convolved and activated sequentially with a Sigmoid function to obtain a reconstructed image with the same spatial size as the input image feature at the preset scale level; the degradation probability map is generated based on the pixel differences between the reconstructed image and the corresponding input image.

4. The degradation probability-guided multi-view three-dimensional target detection method according to claim 1, characterized in that, Step 3) includes the following steps: For the pixel coordinates of any viewpoint in the multi-view image of the current frame According to the pixel-level depth corresponding to the pixel coordinates And the corresponding camera's intrinsic parameter matrix I, the pixel coordinates The back projection is a 3D point P in the camera coordinate system, expressed by the following formula: ； In the formula, Represents pixel coordinates x-coordinate Represents pixel coordinates The ordinate, Represents pixel coordinates pixel-level depth, This represents the inverse of the intrinsic parameter matrix of the corresponding camera, and T represents the matrix transpose; For each scale level , will the current frame's first Image features at each scale level are denoted as , will the first in the previous frame Image features at each scale level are denoted as ;in, Indicates the scale-level index. Indicates the current frame time. The previous frame time is represented; N three-dimensional sampling offsets for each three-dimensional point P are obtained through linear network regression, and N sampling points are determined based on the N three-dimensional sampling offsets, where N is a preset number of sampling points; the N sampling points are projected onto the corresponding image planes in the current frame and the previous frame, respectively, and mapped to the . The feature map coordinates corresponding to each scale level are obtained by bilinear interpolation for the current frame. The Middle Sampling characteristics of each sampling point And the previous frame The Middle Sampling characteristics of each sampling point ; According to the current frame, the first Image features at each scale level Attention weights for each sampling point are predicted using a linear layer and a softmax activation function. For the current frame The Middle Sampling characteristics of each sampling point And the previous frame The Middle Sampling characteristics of each sampling point Perform weighted aggregation to obtain the first... Aggregated features at each scale level The formula is expressed as follows: ； In the formula, MLP represents a multilayer perceptron used to fuse and stitch together sampled features. This represents the summation operation. Indicates the sampling point number; Concat indicates the concatenation operation. The degradation probability map M is downsampled to the level of the current frame. Image features at each scale level and the aggregation feature With the same spatial resolution, a downsampling degradation probability map is obtained. ; using the downsampling degradation probability map As a soft attention gating, the aggregated features Modulate; combine the modulated aggregated features with the image features Add the residuals to obtain the first... Degradation enhancement features at each scale level The formula is expressed as follows: ； In the formula, ⊙ represents element-wise multiplication.

5. The degradation probability-guided multi-view three-dimensional target detection method according to claim 1, characterized in that, In step 4), the degradation probability map is filtered using a preset threshold, and pixel locations with degradation probabilities not less than the preset threshold are determined as high-confidence degradation centers; for each high-confidence degradation center, its two-dimensional coordinates are obtained. and the corresponding pixel-level depth Based on the intrinsic and extrinsic parameters of the corresponding camera, the data is upscaled to three-dimensional space to obtain the center of the three-dimensional anchor point. The formula is expressed as follows: ； In the formula, This represents the inverse of the camera's intrinsic parameter matrix. The inverse matrix of the camera's extrinsic parameters is... The two-dimensional coordinates representing the high-confidence degradation center. This represents the x-coordinate of the two-dimensional coordinate system. This represents the ordinate of the two-dimensional coordinate system. This represents the pixel-level depth corresponding to the two-dimensional coordinates, and T represents the matrix transpose. This indicates taking the first to third components of the vector within the parentheses; The degradation enhancement features are sampled at the high-confidence degradation center, and the obtained sampled features are input into a multilayer perceptron to regress the target size, orientation, and velocity. The three-dimensional spatial anchor center, as well as the regressed target size, orientation, and velocity, are used as query parameters to form an adaptive three-dimensional query. Then, the adaptive three-dimensional query is combined with a preset learnable three-dimensional query to form a two-branch three-dimensional query.

6. The degradation probability-guided multi-view three-dimensional target detection method according to claim 1, characterized in that, Step 5) includes the following steps: The preset learnable 3D query and the adaptive 3D query are interactively fused using a self-attention mechanism to obtain a fused 3D query. Based on the fused 3D query Each query in the process regresses multiple adaptive sampling offsets; based on these adaptive sampling offsets, for each scale level... From the current frame, respectively Image features at each scale level and the first Degradation enhancement features at each scale level The local context features are sampled to obtain the first... Local context feature tensors of original feature branches at each scale level and degenerate enhancement feature branch local context feature tensor ; The original feature branches of each scale level are local context feature tensors. Add the corresponding elements together to obtain the original feature context. ; The local context feature tensor of the degradation enhancement feature branch at each scale level Adding the corresponding elements together yields the degenerate-enhanced feature context. ; the original feature context and the context of the degradation enhancement features By concatenating the components, we obtain a unified feature tensor F, expressed by the following formula: ； In the formula, Concat represents the concatenation operation; Based on the fused 3D query A semantic weight matrix is ​​generated through a linear layer. and using the semantic weight matrix The channel importance of the unified feature tensor F is recalibrated, and then channel dimension mixing is performed to obtain semantic denoising features. The formula is expressed as follows: ； ； In the formula, Linear represents a linear layer, LN represents layer normalization, and · represents matrix multiplication; Based on the fused 3D query Generate geometric weight matrix and using the geometric weight matrix The local geometric relationships of the sampling points are aggregated to achieve point-dimensional blending, resulting in point-position blended features. The formula is expressed as follows: ； ； In the formula, T represents the matrix transpose; Then, the point-to-point blending features are processed through a linear layer. Aggregation is performed to obtain fusion features. Finally, the fusion features will be... Input the detection head, and the three-dimensional target detection results will be output.

Images