A three-dimensional target detection method based on foreground feature extraction

Abstract

The application discloses a three-dimensional target detection method based on foreground feature extraction, and belongs to the field of three-dimensional target detection, and comprises the following steps: step one, extracting foreground features, the process is as follows: step 1.1) defining a primary feature extraction backbone network; step 1.2) foreground region division; step 1.3) foreground scoring; step 1.4) pyramid feature aggregation; step two, three-dimensional deformable attention, which expands the deformable attention to three-dimensional space, including three-dimensional offset sampling and view transformation, for each query reference point in the three-dimensional space, the model learns a sampling offset around each query point, and projects these reference points into different two-dimensional scales and views; with the help of a two-dimensional target detector, the position of a target object is constrained in certain image regions. M The application effectively reduces background interference and reduces computational complexity, and enhances the extraction and expression ability of the model to feature information.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional target detection, and more specifically to a three-dimensional target detection method based on foreground feature extraction. Background Technology

[0002] In the field of intelligent driving, 3D object detection is a key task for achieving safe driving, aiming to provide vehicles with accurate object localization and classification information. Early 3D object detection methods were mainly based on single-camera technology, which typically relied on the procedural modeling of 2D object detectors. Although these methods have made some progress, their accuracy in predicting 3D object position information remains insufficient due to their inability to fully utilize the camera's geometric parameters and the relationships between multiple views. To address this limitation, researchers have proposed multi-view detection... Figure 3 Multi-view object detection methods. This approach captures information from the surrounding environment from multiple perspectives, more effectively constructing the features of 3D objects. By combining data from different angles, these multi-view methods can more comprehensively understand the shape and spatial location of objects, thereby improving detection accuracy. Furthermore, multi-view... Figure 3 3D object detection technology can also enhance the ability to identify objects in complex scenes. For example, in dense urban environments, acquiring information from multiple perspectives helps to better distinguish between objects that are close to each other, thereby reducing false detections and missed detections. Therefore, multi-view methods not only improve the performance of 3D object detection but also provide stronger reliability for intelligent driving systems in dynamic and complex environments, which is the foundation for achieving highly automated driving.

[0003] Currently, multi-view methods can be broadly categorized into Bird's-Eye-View (BEV) methods and sparse query-based methods. BEV-based methods render multi-view images, transforming perspective features into 3D space to generate BEV features, and then using a 3D detection head to generate a 3D bounding box. Zhang et al. were the first to introduce the DETR model into the field of 3D monocular detection, proposing a special depth-guided mechanism that utilizes depth information in the image and employs an attention mechanism to adaptively extract important features. Zheng introduced the idea of ​​assisted learning, designing multi-keypoint constraints and depth estimation assistance to help train the detection head during the training phase. However, the above methods only use a monocular camera, leading to some targets being occluded or not fully captured, thus affecting detection accuracy. Roddick et al. and Reading et al. transformed dense 2D image features into a BEV spatial view, then fed the BEV features into the detector to complete 3D object detection. Roddick et al. used a 3D-to-2D inverse projection relationship to complete the feature space selection transformation, while Reading et al. used a 2D-to-3D projection and discretized the depth values, while also using image plane transformation into BEV form to improve detection accuracy. However, since the computational cost is proportional to the extent of the three-dimensional space, such methods are difficult to scale to large-scale scenarios.

[0004] To reduce computational cost and apply the model to large-scale scenes, researchers have proposed a sparse query-based approach. This approach uses learnable object queries to aggregate multi-view image features and predicts object bounding boxes based on these query features. The sparse object query-based approach samples image features through queries and directly performs 3D object detection using a decoder, thus avoiding significant computation. Wang et al. projected the query onto the image and then used an attention mechanism to model the relationship between the query and multi-view features. Liu et al. proposed embedding 3D positions in a global system and then using a global cross-attention mechanism to update the query and predict 3D bounding boxes. Building on this, Liu et al. incorporated self-motion into the position embedding by converting the coordinates of the previous frame to the current coordinate system. However, these methods have failed to effectively locate foreground object features in the image. Common foreground object features include shape features, texture features, and edge features, which affects the accuracy of object detection. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes a multi-view method based on foreground feature extraction. Figure 3The proposed 3D object detection method extracts key foreground features through a foreground sampling module, effectively reducing background interference and computational complexity. Then, it fuses multi-scale image features using a cross-attention pyramid method to capture the relationships between objects in the detection scene and their global information. Considering the semantic ambiguity issues that arise in existing methods in dynamic scenes, a 3D deformable attention mechanism is introduced in the feature interaction stage. At the same time, a 2D detector is used to generate reliable object position and depth information, enhancing the model's ability to extract and express feature information.

[0006] This invention provides the following technical solution:

[0007] A 3D target detection method based on foreground feature extraction includes the following steps:

[0008] Step 1: Extract foreground features, the process is as follows:

[0009] Step 1.1) Define the primary feature extraction backbone network;

[0010] Step 1.2) Foreground region division;

[0011] Step 1.3) Foreground scoring;

[0012] Step 1.4) Pyramid feature aggregation;

[0013] Step 2: 3D Deformable Attention. This step extends deformable attention to 3D space, including 3D offset sampling and view transformation. For each query reference point in 3D space, the model learns the surrounding data of each query point. M Each sampling offset is used to project these reference points onto different two-dimensional scales and views; with the help of a two-dimensional object detector, the position of the target object is constrained within certain image regions.

[0014] Furthermore, step 1.1) involves using a ResNet network as the backbone network for extracting basic features from the input image, extracting a multi-view image of one frame from the 3D object detection dataset, and obtaining image features at three different feature extraction scales (1 / 8, 1 / 16, 1 / 32). ,in, Represents the image features at the nth scale from the kth camera's perspective; the ResNet network structure is a deep model that organizes convolutional layers according to a residual structure. The residual structure improves the network's training performance for deep models by connecting the original input data and the convolution calculation results. As a lightweight network, the ResNet network structure has the ability to generalize to various visual detection and segmentation tasks.

[0015] Step 1.2) involves designing a foreground sampling module to filter out irrelevant background, reduce computation, and enhance focus on the main detection target. The input to this module comes from the multi-scale features extracted by the ResNet backbone network in step 1.1). To segment the foreground region, image features are obtained in step 1.1). Then, the feature map at scale s at point (i, j) is... Make the following judgment: revert it to the position of the original image. , Image segmentation scale With different scales s set to {1, 2, 3}, the corresponding scale features are {1 / 8, 1 / 16, 1 / 32}, respectively. Representation of foreground tokens at different scales;

[0016] To obtain the overall structure, shape, and main texture features of the target object, as well as its details, textures, and local features, a distance constraint was set. and restricted areas s = {1, 2, 3} Foreground tokens are obtained from the feature maps at large to small scales by varying s; when The distance is limited within the restricted area and the original image position (x, y) is within the restricted area. Within the specified area, the target is considered a foreground token; otherwise, the token is discarded. `w` refers to the width of the actual 2D bounding box of the foreground object in the image, and `h` refers to the height of the 2D bounding box of the foreground object. The width (w) represents the x-coordinate length of the foreground target. The y-coordinate length, or h-coordinate, of the foreground target represents the height.

[0017] The process in step 1.3) is as follows: After the initial screening in step 1.2), the foreground token of the image is obtained. Then, a top-down scoring module is used to address the importance of features at different levels.

[0018] A scoring module based on an MLP is used to predict the foreground score of each feature map. An MLP is a feedforward neural network with an input layer, hidden layers, and an output layer. Multiple calculations in the hidden layers generate a description of the input data in the global representation space. Considering that high-level feature maps contain richer semantic information and higher resolution, the foreground score, based on high-level semantic information, is used as supplementary information to adjust the feature maps of adjacent low-level semantics. The top-down score adjustment method transmits the foreground score layer by layer through upsampling, effectively utilizing feature information from different levels to improve the accuracy and robustness of image semantic scoring. The foreground feature calculation process is as follows:

[0019] ;

[0020] In the above formula, It is an upsampling function. It is a set of learnable coefficients. Let l be the image feature of the foreground token at the l-th scale, where l=1,2,3. To score the tokens, foreground scoring is performed on the feature maps at each scale. Then, the foreground tokens are further filtered across the feature maps at each scale. The process is represented as follows:

[0021] ;

[0022] ;

[0023] ;

[0024] In the above formula, It's a hyperparameter. It is a sorting and selection function that scores foreground target tokens at the l-th scale. Foreground tokens at the l-th scale After selection, a further screening was conducted to obtain the top [rankings]. A prospect scoring matrix , This is for max pooling.

[0025] The process in step 1.4) is as follows: After foreground feature scoring in step 1.3), an aggregation layer is designed, using a pyramid pooling module to aggregate these foreground features. A 1*1 Conv operation is performed sequentially on each size feature map to reduce the number of channels. Then, bilinear interpolation is used for upsampling, and then the channels are stitched together to aggregate the global and local information of the entire image.

[0026] ;

[0027] The above formula yields the fused features of global context information. , These are the original features of the image;

[0028] Design a cross-attention mechanism that utilizes foreground features of local details. To simultaneously serve as both query q and key k, and at the same time, to fuse features As a value (v), to facilitate interaction between features at different scales, a refined foreground is obtained. As the next attention layer Subsequently, after processing through three cross-attention layers, the foreground features of each scale feature map are fused together to capture high-level contextual information, enhance the semantic foreground, and reduce misjudgments of objects at different distances. The enhanced foreground sampling pyramid feature is shown in the following formula:

[0029] ;

[0030] After the above steps, the foreground features of each scale feature map are obtained after refinement. , T is the transpose symbol, c is the dimension of the feature map at each scale, and the Softmax function is a function that transforms data to between 0 and 1 to prevent gradient explosion.

[0031] Furthermore, the process of step two is as follows:

[0032] Step 2.1) Object query generation: In order to generate object queries from 2D detection, a dynamic query generator is proposed by combining a 2D object detector and a depth information detector.

[0033] Step 2.2) Three-dimensional query feature interaction;

[0034] Step 2.3) Calculate the self-attention feature vector;

[0035] Step 2.4) Calculate the 3D deformable attention feature map;

[0036] Step 2.5) Integrate attention features;

[0037] Step 2.6) Update features;

[0038] Step 2.7) Generate a 3D target detection box.

[0039] The process of step 2.1) is as follows:

[0040] Step 2.1.1) Generation of 2D recognition bounding boxes, the process is as follows:

[0041] The features obtained from foreground sampling in step 1.4) The feature vectors are input into the 2D detector. First, the input 2D image is preprocessed. The preprocessed image is then input into the backbone network of the 2D detector model. A feature pyramid network is used to fuse features from different levels. At the output of the feature pyramid network, multiple prediction heads are used to generate detection results. Each prediction head corresponds to a feature map at a certain scale. The predicted bounding box coordinates are mapped back from the feature map scale to the original image scale. Bounding boxes with confidence scores below a threshold are filtered out, and overlapping bounding boxes are suppressed. Only the bounding boxes with the highest confidence scores are retained. Finally, the foreground object is detected, and the recognized bounding boxes are obtained. ;

[0042] Step 2.1.2) Deep query generation, the process is as follows:

[0043] Then, DepthNet is used to generate the corresponding discrete depth distribution of the image. DepthNet is a deep learning framework that first combines depth information... and the center point of the 2D bounding box ( , The coordinates are then obtained by matrix multiplication using the camera parameter matrix. ,in These represent the four coordinates of the bounding box B; finally, the camera's intrinsic parameters are used. and extrinsic parameters This process transforms the current pixel coordinate system to a 3D reality coordinate system. Camera intrinsic parameters describe the camera's own geometric properties, while camera extrinsic parameters describe the camera's position and orientation in the world coordinate system. Extrinsic parameters are used to convert points in the world coordinate system to the camera coordinate system, thereby generating a series of depth query objects. The process is as described in the following formula:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] After the above steps, the depth query of the two-dimensional object is obtained. Where PE represents sine and cosine positional encoding, i* represents the dimension index of the positional encoding vector, sine function is used at positions 6i* to 6i*+3, cosine function is used at positions 6i*+3 to 6i*+6, and c* represents the embedding dimension.

[0050] The process in step 2.2) is as follows: Three-dimensional deformable attention is employed, utilizing enhanced foreground sampling pyramid features and multi-view efficient feature interaction, with two-dimensional reference points... It is obtained from the following formula:

[0051] ;

[0052] in These are a 3D reference point obtained through MLP from a 3D query and a learnable offset value, representing a 2D reference point projected at different scales and views, and a 3D reference point. Generated from a 3D query, based on the aforementioned 2D reference points. and the corresponding The 3D object query interacts with the foreground sampled feature map. In this way, different features from various viewpoints and scales are aggregated into the 3D query. The 3D deformable attention is expressed as follows:

[0053] ;

[0054] ;

[0055] in, From the foreground features of step 1.4) After summing, A represents the attention weight of the three-dimensional query. This represents the 3D query attention weights at different sampling points. It is a learnable parameter. This is the query result from the nth perspective, where K is the number of sampling points. It is a bilinear interpolation function, where V represents the output result obtained by linearly transforming images from different viewpoints in a 3D query.

[0056] The process in step 2.3) is as follows: The query vector... and positional encoding vector Adding them together yields a vector Q containing depth information. Then, Q and... The input to the self-attention mechanism yields a weighted query vector, expressed as follows:

[0057] ;

[0058] Simultaneously, the original query vector calculated using the self-attention mechanism will be... Add them together and then normalize them to obtain a new vector. ;

[0059] The process in step 2.4) is as follows: The weight A of each three-dimensional reference point is compared with its corresponding eigenvalue. Multiply and sum to get Fine-grained expression ;

[0060] The process in step 2.5) is as follows: The attention features from steps 2.3) and 2.4) are... and The sums are then processed through a multi-layer linear structure with activation functions and normalization operations to obtain the final feature representation. ;

[0061] The process in step 2.6) is as follows: [The steps in step 2.5) are...] Generate query vectors using MLP As input data for step 2.3), and processed The operation of the next iteration. The hyperparameters set are denoted as follows: .

[0062] The process of step 2.7) is as follows:

[0063] Obtain the final 3D target detection features. First, 3D target detection is achieved by setting up a regression head. The regression head includes a backbone network for feature extraction and is processed through multiple fully connected layers. Each layer is followed by an activation function and batch normalization. The output of the regression head includes the center coordinates, size, and orientation angle of the target. The center coordinates are determined by the offset relative to the reference point. The size is ensured to be positive by an exponential function to avoid negative values. The orientation angle is determined by the regression angle and its range is constrained by an activation function. The loss function uses the angle difference Smooth L1 to measure the difference between the model output and the true label.

[0064] Then, a classification head is set up to predict the class probability of the target. The classification head shares the features extracted by the backbone network of the regression head and processes them through multiple fully connected layers. Each layer is followed by an activation function and batch normalization. The output of the classification head is the score of each class. The score is converted into class probability through the softmax function. By setting a confidence threshold, 3D bounding boxes with low recognition scores are filtered out. Finally, the model outputs 3D bounding boxes with center coordinates, size, orientation angle and class, thus completing 3D target detection.

[0065] The beneficial effects of this invention are as follows:

[0066] This invention proposes a foreground extraction method for 3D object detection. By designing a foreground sampling module, it extracts information about the main objects in the image. The module employs a cross-attention pyramid method, processing the foreground features through three cross-attention layers and fusing foreground features from multi-scale feature maps. This suppresses the influence of the background on the model and improves the model's accuracy.

[0067] This invention uses a two-dimensional detector to generate reliable object queries and designs a three-dimensional deformable attention mechanism to capture information features from different views. Simultaneously, it interacts with information features from different time periods to establish long-term temporal dependencies and obtain more effective feature information.

[0068] This invention can effectively improve the accuracy of three-dimensional target detection. Attached Figure Description

[0069] Figure 1 This is a structure diagram for foreground feature extraction.

[0070] Figure 2 This is a flowchart of a deep object query process.

[0071] Figure 3 It is a three-dimensional deformable attention structure diagram. Detailed Implementation

[0072] The present invention will now be further described with reference to the accompanying drawings:

[0073] Reference Figures 1-3 A three-dimensional target detection method based on foreground feature extraction, characterized by the following steps:

[0074] Step 1: Extract foreground features, the process is as follows:

[0075] Step 1.1) Define the primary feature extraction backbone network, the process is as follows:

[0076] Using a ResNet network as the backbone network to extract basic features from the input image, a multi-view image of one frame is taken from a 3D object detection dataset, and image features are obtained at three different feature extraction scales (1 / 8, 1 / 16, 1 / 32). ,in, Representing the image features at the nth scale from the kth camera's perspective, the ResNet network structure is a deep model that organizes convolutional layers according to a residual structure. The residual structure improves the network's training performance for deep models by connecting the original input data and the convolution calculation results. As a lightweight network, the ResNet network structure has the ability to generalize to various visual detection and segmentation tasks.

[0077] Step 1.2) Foreground region division, the process is as follows:

[0078] Considering that when detecting target objects (such as vehicles or crowds), target object features are usually more important than background features, therefore, as Figure 1 As shown, a foreground sampling module was designed to filter out irrelevant background, reduce computation, and enhance focus on the main detection target. The input of this module comes from the multi-scale features extracted by the ResNet backbone network in step 1.1). To segment the foreground region, image features were obtained in step 1.1). Then, the feature map at scale s at point (i, j) is... Make the following judgment: revert it to the position of the original image. , Image segmentation scale With different scales s set to {1, 2, 3}, the corresponding scale features are {1 / 8, 1 / 16, 1 / 32}, respectively. Representation of foreground tokens at different scales;

[0079] In image processing, large-scale features and small-scale features refer to features at different scales (or resolutions) in an image. Understanding and extracting features at different scales is crucial for object detection. Large-scale features typically describe global information or large-scale structures in an image. These features are extracted at lower resolutions, capturing the overall shape, layout, and large-scale patterns of the image. Therefore, in large-scale feature maps, foreground features correspond to the overall structure, shape, and main texture features of the target object, while small-scale feature maps are mainly used to capture its details, textures, and local features. Therefore, to obtain the overall structure, shape, and main texture features of the target object, as well as its details, textures, and local features, this invention sets a distance limit. and restricted areas s = {1, 2, 3} Foreground tokens are obtained from the feature maps at large to small scales by varying the value of s. The distance is limited within the restricted area and the original image position (x, y) is within the restricted area. Within the specified area, the target is considered a foreground token; otherwise, the token is discarded. `w` refers to the width of the actual 2D bounding box of the foreground object in the image, and `h` refers to the height of the 2D bounding box of the foreground object. The width (w) represents the x-coordinate length of the foreground target. The h represents the length of the y-coordinate of the foreground target, i.e., its height.

[0080] Step 1.3) Foreground scoring, the process is as follows:

[0081] After the initial screening in step 1.2), the foreground tokens of the image were obtained. Then, a top-down scoring module is used to address the importance of features at different levels.

[0082] like Figure 1 As shown in the middle section, a scoring module based on a Multi-layer Perceptron (MLP) is used to predict the foreground score of each feature map. MLP is a feedforward neural network with an input layer, hidden layers, and an output layer. It generates a description of the input data in the global representation space through multiple calculations in the hidden layers. Considering that high-level feature maps contain richer semantic information and higher resolution, the foreground score, based on high-level semantic information, is used as supplementary information to adjust the feature maps of adjacent low-level semantics. The top-down score adjustment method transmits the foreground score layer by layer through upsampling, effectively utilizing feature information from different levels to improve the accuracy and robustness of image semantic scoring. The foreground feature calculation process is as follows:

[0083] ;

[0084] In the above formula, It is an upsampling function. Upsampling is used in computer vision tasks. The main function of upsampling is to increase the spatial dimensions (width and height) of the feature map, thereby enabling the network to recover or generate higher resolution output. It is a set of learnable coefficients. Let be the image feature of the foreground token at the l-th scale, where l = 1, 2, 3. Individual by It is generated by MLP. To score the tokens, foreground scoring is performed on the feature maps at each scale. Then, the foreground tokens are further filtered across the feature maps at each scale. The process is represented as follows:

[0085] ;

[0086] ;

[0087] ;

[0088] In the above formula, It's a hyperparameter. It is a sorting and selection function that scores foreground target tokens at the l-th scale. Foreground tokens at the l-th scale After selection, a further screening was conducted to obtain the top [rankings]. A prospect scoring matrix , Max pooling helps improve the computational efficiency, feature robustness, and space invariance of the model by reducing the dimensionality of the feature map and extracting the largest features.

[0089] Step 1.4) Pyramid feature aggregation, the process is as follows:

[0090] After foreground feature scoring in step 1.3), as follows: Figure 1 As shown on the left, this invention designs an aggregation layer (Fuse) that uses a pyramid pooling module to aggregate these foreground features. A 1*1 Conv operation is performed sequentially on each feature map size to reduce the number of channels. In convolutional neural networks (CNNs), a 1*1 Conv operation is typically used to reduce the number of channels in a feature map. Then, bilinear interpolation is used for upsampling. Bilinear interpolation is an interpolation method used in image processing and computer vision. Its main function is to estimate new pixel values ​​by linearly weighting the image pixel values. These values ​​are then concatenated across the channels to aggregate the global and local information of the entire image.

[0091] ;

[0092] The above formula yields the fused features of global context information. , The original features of an image are used for fusion. However, foreground elements are sparse in image features, and the differences in small object features are blurred. Multi-scale feature fusion can easily lead to background points being emphasized while foreground elements are ignored, failing to distinguish effective features and causing biased center position predictions and false detections. Furthermore, low-level features focus on local details, while high-level features capture global contextual information. To highlight foreground features and achieve efficient interaction with global feature fusion, such as... Figure 1 As shown on the right, this invention designs a cross-attention mechanism (CA), which utilizes foreground features of local details. Let it serve as both query (q) and key (k), while simultaneously incorporating fusion features. As a value (v), it facilitates multi-head attention (MH-Attention) interactions between features at different scales. Add & normalize represent residual connections and layer normalization, which work together in various layers of the model to improve training efficiency and performance. FFN represents a feedforward neural network, whose structure consists of nodes at multiple levels that transmit information in specific directions, resulting in a refined foreground. As the next attention layer Subsequently, after processing through three cross-attention layers, the foreground features of each scale feature map are fused together to capture high-level contextual information, enhance the semantic foreground, and reduce misjudgments of objects at different distances. The enhanced foreground sampling pyramid feature is shown in the following formula:

[0093] ;

[0094] After the above steps, the foreground features of each scale feature map are obtained after refinement. , T is the transpose symbol, c is the dimension of each scale feature map, and the Softmax function is a function that transforms data to between 0 and 1 to prevent gradient explosion. Gradient explosion refers to the phenomenon that during backpropagation, the gradient value increases exponentially with the number of network layers, causing the network weights to be updated drastically and making the network unstable.

[0095] Step 2: 3D Deformable Attention

[0096] This invention extends deformable attention to three-dimensional space, including 3D offset sampling and view transformation. For each query reference point in 3D space, the model learns the surrounding environment of each query point. MEach sampling offset is used to project these reference points onto different 2D scales and views. However, attention-based 2D feature enhancement methods suffer from depth ambiguity: when a 3D query is projected onto the camera view, query results with the same projection coordinates (u, v) and different depth values ​​have the same reference points and similar sampling points on the view, which may lead to network convergence difficulties.

[0097] Currently, most research focuses on near-range perception. The initial goal of this invention is to address the performance issues caused by background interference in target detection, while also resolving the problems of error propagation from distant targets and the difficulty of convergence in perceiving distant targets. Therefore, with the assistance of a two-dimensional target detector, this invention constrains the position of the target object within certain image regions, providing valuable reference for locating objects in three-dimensional space. The process is as follows:

[0098] Step 2.1) Object query generation, the process is as follows:

[0099] like Figure 2 As shown, in order to generate object queries from 2D detection, this invention combines a 2D object detector (2Ddection) and a depth information detector (DepthNet) to propose a dynamic query generator, the process of which is as follows:

[0100] Step 2.1.1) Generation of 2D recognition bounding boxes, the process is as follows:

[0101] The features obtained from foreground sampling in step 1.4) The feature vectors are input into the 2D detector. First, the input 2D image is preprocessed, and the preprocessed image is input into the backbone network of the 2D detector model. To better detect targets at different scales, a feature pyramid network is used to fuse features at different levels. At the output of the feature pyramid network, multiple prediction heads are used to generate detection results. Each prediction head corresponds to a feature map at a certain scale. The predicted bounding box coordinates are mapped from the feature map scale back to the original image scale. Bounding boxes with confidence scores below a threshold are filtered out, and overlapping bounding boxes are suppressed. Only the bounding boxes with the highest confidence scores are retained. Finally, the foreground target is detected, and the recognized bounding boxes are obtained. ;

[0102] Step 2.1.2) Deep query generation, the process is as follows:

[0103] Then, DepthNet is used to generate the corresponding discrete depth distribution of the image. DepthNet is a deep learning framework designed for depth estimation using a monocular camera. It first combines depth information... and the center point of the 2D bounding box ( , The coordinates are then obtained by matrix multiplication using the camera parameter matrix. .in These represent the four coordinates of the bounding box B, and finally the camera's intrinsic parameters are used. and extrinsic parameters This process transforms the current pixel coordinate system to a 3D reality coordinate system. The camera's intrinsic and extrinsic parameters describe its geometric characteristics and its relationship to the external world. These parameters are crucial for tasks such as camera calibration, 3D modeling, and stereo vision. The camera's intrinsic parameters describe its own geometric characteristics, while its extrinsic parameters describe its position and orientation in the world coordinate system. The extrinsic parameters are used to convert points in the world coordinate system to the camera coordinate system, thereby generating a series of depth query objects. The process is as described in the following formula:

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] After the above steps, a reliable two-dimensional object depth query is obtained. In this context, PE represents sine and cosine positional encoding, a technique for sequence data processing designed to provide information about each position to the model, helping the neural network understand the relative and absolute positions of elements in the sequence. This concept was first proposed in the Transformer model and has been widely applied to various sequence-related tasks. i* represents the dimension index of the positional encoding vector. To better capture positional information at different granularities, this invention uses a sine function from 6i* to 6i*+3 and a cosine function from 6i*+3 to 6i*+6. c* represents the embedding dimension.

[0111] Step 2.2) Three-dimensional query feature interaction, the process is as follows:

[0112] like Figure 3 As shown, to achieve efficient feature interaction across different scales and views, this invention proposes a 3D deformable attention mechanism. This mechanism utilizes enhanced foreground sampling pyramid features and multiple views to achieve efficient feature interaction, with a 2D reference point... It is obtained from the following formula:

[0113] ;

[0114] in These are the 3D reference points obtained through the MLP (3D Query) and a learnable offset value (3D offset), representing the 2D and 3D reference points projected at different scales and views. Generated from a 3D query, based on the aforementioned 2D reference points. and the corresponding (Corresponding learnable offset values), the 3D object query interacts with the foreground sampled feature map. In this way, different features (n-th view image features) at various viewpoints and scales are aggregated into the 3D query, and the 3D deformable attention is expressed as follows:

[0115] ;

[0116] ;

[0117] in, From the foreground features of step 1.4) After summing, A represents the attention weight of the three-dimensional query. This represents the 3D query attention weights at different sampling points. It is a learnable parameter. This is the query result from the nth perspective, where K is the number of sampling points. It is a bilinear interpolation function. This represents the output result obtained by performing a linear transformation on images from different viewpoints in a 3D query.

[0118] Step 2.3) Calculate the self-attention feature vector, the process is as follows:

[0119] query vector (The initial values ​​include the randomized query object and the depth query object in step 2.1.2) ) and position encoding vector Adding them together yields a vector Q containing depth information. Then, Q and... The input to the self-attention mechanism yields a weighted query vector, expressed as follows:

[0120] ;

[0121] Simultaneously, the original query vector calculated using the self-attention mechanism will be... Add them together and then normalize them to obtain a new vector. ;

[0122] Step 2.4) Calculate the 3D deformable attention feature map. The process is as follows:

[0123] The weight A of each 3D reference point is associated with its corresponding eigenvalue. Multiply and sum to get Fine-grained expression ;

[0124] Step 2.5) Integrate attention features, the process is as follows:

[0125] The attention features in steps 2.3) and 2.4) and The sums are then processed through a multi-layer linear structure with activation functions and normalization operations to obtain the final feature representation. ;

[0126] Step 2.6) Update features, the process is as follows:

[0127] In step 2.5) Generate query vectors using MLP As input data for step 2.3), and processed The operation of the next iteration. The hyperparameters set are denoted as follows: ;

[0128] Step 2.7) Generate the 3D target detection box. The process is as follows:

[0129] Features of multi-view By aggregating 3D deformable attention into 3D queries, each query is obtained. Its rich features, through the combination of self-attention mechanisms And 3D deformable attention mechanism adaptive matching features Obtain the final 3D target detection features. First, 3D object detection is achieved by setting up a regression head. The regression head consists of a backbone network for feature extraction, processed through multiple fully connected layers, each followed by an activation function and batch normalization. The output of the regression head includes the target's center coordinates, size, and orientation angle. The center coordinates are determined by the offset relative to a reference point (such as the scene center or anchor point). The size is determined using an exponential function to ensure a positive output and avoid negative values. The orientation angle is determined by regressing the angle and constrained by an activation function. The loss function uses the angle difference Smooth L1 to measure the difference between the model output and the true label.

[0130] Furthermore, a classification head is set up to predict the class probability of the target. The classification head shares the features extracted by the backbone network of the regression head and processes them through multiple fully connected layers. Each layer is followed by an activation function and batch normalization. The output of the classification head is the score for each class. The softmax function converts the score into class probability. By setting a confidence threshold, 3D bounding boxes with low recognition scores are filtered out. Finally, the model outputs 3D bounding boxes with center coordinates, size, orientation angle, and class, completing 3D target detection. The fully connected layers are responsible for mapping the features extracted from the previous layers into the final output. The activation function performs a non-linear transformation on the input information, and then passes the transformed output information as input information to the next layer of neurons. Batch normalization is a technique used to improve training speed, stability, and performance. It normalizes each mini-batch of data, keeping the distribution of intermediate layer outputs stable during training.

[0131] This embodiment uses the nuScenes dataset to perform performance testing on the present invention, and the process is as follows:

[0132] Step 1: Define the comparison method as follows:

[0133] BEVDet4D is a four-dimensional (4D) object detection method based on Bird's Eye View (BEV) representation. This method captures temporal changes by projecting point cloud data from consecutive frames onto the BEV representation.

[0134] SoloFusion is a 3D target detection method that fuses multimodal data. This method improves detection accuracy by fusing data from different sensors, leveraging the strengths of each sensor.

[0135] PETR is a Transformer-based 3D object detection method. This method converts point cloud data into sequential data through position embedding and uses a Transformer model for feature extraction and object detection.

[0136] BEVDepth is a depth estimation method based on Bird's Eye View (BEV) representation for 3D object detection. This method improves detection accuracy by projecting point cloud data into the BEV representation and incorporating depth information.

[0137] Step 2: Define the evaluation indicators as follows:

[0138] This invention uses mean average precision (mAP) and nuScenes detection score (NDS) as evaluation metrics for 3D object detection tasks:

[0139] ;

[0140] ;

[0141] ;

[0142] ;

[0143] ;

[0144] ;

[0145] ;

[0146] ;

[0147] Where C represents the set of all detection categories. To calculate the average precision for category C, the precision-recall curve is integrated over the precision-recall curve to determine the precision for each category. The NDS is derived from five factors: mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE), and mean Average Attribute Error (mAE), each weighted 10%, while mAP accounts for 50%. mATE measures the Euclidean distance between the center point of the detected bounding box and the center point of the ground truth bounding box. Given N bounding boxes, each with a translation error of... .

[0148] mASE is used to measure the relative error between the measured bounding box and the true bounding box size. Given N measured bounding boxes, the scaling error of each bounding box is... mAOE is used to measure the relative error between the detected bounding box and the true bounding box size. Given N detected bounding boxes, the scaling error of each bounding box is... mAVE is used to measure the relative error between the detected bounding box and the true bounding box size. Given N detected bounding boxes, the scaling error of each bounding box is... mAAE is used to measure the relative error between the measured bounding box and the true bounding box size. Given N measured bounding boxes, the scaling error of each bounding box is... .

[0149] Step 3, Introduction to the dataset, as follows:

[0150] The nuScenes dataset is a large, multimodal dataset consisting of data collected from 6 cameras, 1 LiDAR, and 5 radars. The dataset contains 1000 scenes, each with 20 seconds of video frames, and is fully annotated with 3D bounding boxes every 0.5 seconds.

[0151] Step 4: Evaluation of experimental results, as follows:

[0152] Table 1 shows the results of this invention on the nuScenes validation dataset. First, this invention achieves high performance when using ResNet-50 and ResNet101 as the image feature extraction backbone. In particular, there is a significant improvement in the mAP metric, indicating that this invention has a strong ability to locate objects in 3D space. Specifically, when using ResNet-50 to extract features from lower resolution 256×704 images, this invention improves performance by 8.4% and 6.5% in mAP and NDS, respectively, compared to the BEVDepth method with additional depth supervision. Then, when using ResNet101 to extract features from relatively higher resolution 512×1408 or 900×1600 images, this invention improves performance by 9.5% in mAP and 5.9% in NDS, respectively, compared to the BEVDepth method with additional depth supervision.

[0153] Table 1 shows the results of each method on the nuScenes validation dataset;

[0154] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A three-dimensional target detection method based on foreground feature extraction, characterized in that, The method includes the following steps: Step 1: Extract foreground features, the process is as follows: Step 1.1) Define the primary feature extraction backbone network: Use the ResNet network as the backbone network to extract basic features from the input image. Take a multi-view image of one frame from the 3D object detection dataset, and obtain image features at three different feature extraction scales (1 / 8, 1 / 16, 1 / 32). ; Step 1.2) Foreground Region Segmentation: A foreground sampling module is designed to filter out irrelevant background, reduce computation, and enhance focus on the main detection target. The input of this module comes from the multi-scale features extracted by the backbone network ResNet in Step 1.1). In order to segment the foreground region, after obtaining the image features in Step 1.1), the position of the s-th scale feature map (i, j) is regressed to the position of the original image. Foreground tokens are obtained by setting a limiting distance and a limiting region. When the limiting distance is within the limiting region and the original image position (x, y) is within the region, it is a foreground token; otherwise, the token is discarded. Step 1.3) Foreground scoring: After the initial screening in Step 1.2), foreground tokens of the image are obtained. Then, a top-down scoring module is used to process the importance of features at different levels. An MLP-based scoring module is used to predict the foreground score of each feature map. Foreground tokens at each scale are selected based on the foreground target token score to obtain a further filtered foreground score matrix. Step 1.4) Pyramid Feature Aggregation: After foreground feature scoring in Step 1.3), an aggregation layer is designed, using a pyramid pooling module to aggregate these foreground features. A 1*1 Conv operation is performed sequentially on each feature map of size to reduce the number of channels. Then, bilinear interpolation is used for upsampling, and the channels are stitched together. A cross-attention mechanism is then designed, using the foreground features of local details as query q and key k, and the fused features as value v. After processing through three cross-attention layers, the enhanced foreground sampling pyramid features are obtained. Step 2: 3D Deformable Attention. This step extends deformable attention to 3D space, including 3D offset sampling and view transformation. For each query reference point in 3D space, the model learns the surrounding data of each query point. M Each sampling offset is used, and these reference points are projected onto different two-dimensional scales and views; with the help of a two-dimensional object detector, the position of the target object is constrained within certain image regions, as follows: Step 2.1) Object Query Generation: To generate object queries from 2D detection, a dynamic query generator is proposed, combining a 2D object detector and a depth information detector. The 2D bounding box generation process involves inputting the foreground-sampled features obtained in Step 1.4) into the 2D detector, fusing features at different levels using a feature pyramid network, mapping the predicted bounding box coordinates back to the original image scale, and finally detecting the foreground object and obtaining the recognized bounding box. The depth query generation process involves using DepthNet to generate the corresponding discrete depth distribution of the image, combining depth information and the coordinates of the 2D bounding box center point, and using camera intrinsic and extrinsic parameters to transform the current pixel coordinate system to the 3D real-world coordinate system, thereby generating a series of depth query objects and obtaining the 2D object depth query. Step 2.2) 3D Query Feature Interaction: Employ 3D deformable attention, utilize enhanced foreground sampling pyramid features and achieve efficient feature interaction across multiple views; based on 2D reference points and corresponding learnable offset values, the 3D object query interacts with the foreground sampling feature map, thereby aggregating different features from various perspectives and scales into the 3D query; Step 2.3) Calculate the self-attention feature vector: its input is the query feature from step 2.2), and the updated query vector is generated through the self-attention mechanism and normalization operation; Step 2.4) Calculate the 3D deformable attention feature map: Its input is the query vector updated in step 2.3), and features are extracted from feature maps at different scales through deformable sampling points and weighted summation. The output is the 3D deformable attention feature map. Step 2.5) Fusing attention features: The attention features from Step 2.3) and Step 2.4) are added together and passed through a multi-layer linear structure with activation functions and normalization operations to obtain the final feature representation; Step 2.6) Update features: Use the query vector generated by MLP in step 2.5) as the input data for step 2.3), and perform several iterations of computation; Step 2.7) Generate a 3D target detection box: Its input is the 3D query features from step 2.6), and the detection head outputs the 3D center position, size and orientation angle of the target to generate the final 3D bounding box.

2. The three-dimensional target detection method based on foreground feature extraction as described in claim 1, characterized in that, The process of step 1.1) is as follows: using the ResNet network as the backbone network to extract basic features from the input image, taking a multi-view image of one frame from the 3D object detection dataset, and obtaining image features at three different feature extraction scales (1 / 8, 1 / 16, 1 / 32). ,in, Represents the image features at the nth scale from the kth camera's perspective; the ResNet network structure is a deep model that organizes convolutional layers according to a residual structure. The residual structure improves the network's training performance for deep models by connecting the original input data and the convolution calculation results. As a lightweight network, the ResNet network structure has the ability to generalize to various visual detection and segmentation tasks.

3. The three-dimensional target detection method based on foreground feature extraction as described in claim 1, characterized in that, Step 1.2) involves designing a foreground sampling module to filter out irrelevant background, reduce computation, and enhance focus on the main detection target. The input to this module comes from the multi-scale features extracted by the ResNet backbone network in step 1.1). To segment the foreground region, image features are obtained in step 1.1). Then, the feature map at scale s at point (i, j) is... Perform the following judgment: revert it to the position of the original image. , Image segmentation scale With different scales s set to {1, 2, 3}, the corresponding scale features are {1 / 8, 1 / 16, 1 / 32}, respectively. Representation of foreground tokens at different scales; To obtain the overall structure, shape, and main texture features of the target object, as well as its details, textures, and local features, a distance constraint was set. and restricted areas s = {1, 2, 3} Foreground tokens are obtained from the feature maps at large to small scales by varying s; when The distance is limited within the restricted area and the original image position (x, y) is within the restricted area. Within the specified area, the target is considered a foreground token; otherwise, the token is discarded. `w` refers to the width of the actual 2D bounding box of the foreground object in the image, and `h` refers to the height of the 2D bounding box of the foreground object. The width (w) represents the x-coordinate length of the foreground target. The y-coordinate length, or h-coordinate, of the foreground target represents the height.

4. The three-dimensional target detection method based on foreground feature extraction as described in claim 1, characterized in that, The process in step 1.3) is as follows: the foreground token of the image obtained in step 1.2). The MLP-based scoring module predicts the foreground score for each feature map, performs foreground scoring on feature maps at each scale, and then further filters the foreground tokens on feature maps at each scale.

5. The three-dimensional target detection method based on foreground feature extraction as described in claim 1, characterized in that, The process of step 1.4) is as follows: After the foreground feature scoring in step 1.3), an aggregation layer is designed, and a pyramid pooling module is used to aggregate these foreground features. A 1*1 Conv operation is performed on the feature map of each size in turn to reduce the number of channels. Then, bilinear interpolation is used for upsampling, and then the channels are stitched together to aggregate the global and local information of the entire image. Design a cross-attention mechanism that utilizes foreground features of local details. To simultaneously serve as both query q and key k, and at the same time, to fuse features As a value (v), to facilitate the interaction between features at different scales, a refined foreground is obtained. As the next attention layer ; Subsequently, after processing through three cross-attention layers, the foreground features of each scale feature map are fused together to capture high-level contextual information, enhance semantic foreground information, and reduce misclassification of objects at different distances, resulting in refined foreground features for each scale feature map. .

6. The three-dimensional target detection method based on foreground feature extraction as described in claim 1, characterized in that, The process of step 2.1) is as follows: Step 2.1.1) Generation of 2D recognition bounding boxes, the process is as follows: The features obtained from foreground sampling in step 1.4) The feature vectors are input into the 2D detector. First, the input 2D image is preprocessed. The preprocessed image is then input into the backbone network of the 2D detector model. A feature pyramid network is used to fuse features from different levels. At the output of the feature pyramid network, multiple prediction heads are used to generate detection results. Each prediction head corresponds to a feature map at a certain scale. The predicted bounding box coordinates are mapped back from the feature map scale to the original image scale. Bounding boxes with confidence scores below a threshold are filtered out, and overlapping bounding boxes are suppressed. Only the bounding boxes with the highest confidence scores are retained. Finally, the foreground object is detected, and the recognized bounding boxes are obtained. ; Step 2.1.2) Deep query generation, the process is as follows: The image's discrete depth distribution is generated using DepthNet, first by combining depth information. and the center point of the 2D bounding box ( , The coordinates are then obtained by matrix multiplication using the camera parameter matrix. Finally, the camera's intrinsic parameters are used. and extrinsic parameters This transforms the current pixel coordinate system to a three-dimensional reality coordinate system, thereby generating a series of depth query objects. And calculate the depth query of the two-dimensional object.

7. The three-dimensional target detection method based on foreground feature extraction as described in claim 1, characterized in that, The process of step 2.2) is as follows: Three-dimensional deformable attention is employed, utilizing enhanced foreground sampling pyramid features and multi-view efficient feature interaction, based on two-dimensional reference points. and the corresponding learnable offset values The 3D object query interacts with the foreground sampled feature map, in which different features from various perspectives and scales are aggregated into the 3D query.

8. The three-dimensional target detection method based on foreground feature extraction as described in claim 1, characterized in that, The process in step 2.3) is as follows: The query vector... and position encoding vector Adding them together yields a vector Q containing depth information. Then, Q and... The weighted query vector is obtained as input to the self-attention mechanism, and the original query vector is calculated by the self-attention mechanism. Add them together and then normalize them to obtain a new vector. ; The process in step 2.4) is as follows: The weight A of each three-dimensional reference point is compared with its corresponding eigenvalue. Multiply and sum to get Fine-grained expression ; The process in step 2.5) is as follows: The attention features from steps 2.3) and 2.4) are... and The sums are then processed through a multi-layer linear structure with activation functions and normalization operations to obtain the final feature representation. ; The process in step 2.6) is as follows: [The steps in step 2.5) are...] Generate query vectors using MLP As input data for step 2.3), and after... The operation of the next iteration. The hyperparameters set are denoted as follows: .

9. The three-dimensional target detection method based on foreground feature extraction as described in claim 1, characterized in that, The process in step 2.7) is as follows: obtaining the final three-dimensional target detection features. First, 3D target detection is achieved by setting up a regression head. The regression head includes a backbone network for feature extraction and is processed through multiple fully connected layers. Each layer is followed by an activation function and batch normalization. The output of the regression head includes the center coordinates, size, and orientation angle of the target. The center coordinates are determined by the offset relative to the reference point. The size is ensured to be positive by an exponential function to avoid negative values. The orientation angle is determined by the regression angle and its range is constrained by an activation function. The loss function uses the angle difference Smooth L1 to measure the difference between the model output and the true label. Then, a classification head is set up to predict the class probability of the target. The classification head shares the features extracted by the backbone network of the regression head and processes them through multiple fully connected layers. Each layer is followed by an activation function and batch normalization. The output of the classification head is the score of each class. The score is converted into class probability through the softmax function. By setting a confidence threshold, 3D bounding boxes with low recognition scores are filtered out. Finally, the model outputs 3D bounding boxes with center coordinates, size, orientation angle and class, thus completing 3D target detection.

Images