End-to-end target detection method based on shape correction and feature selection cross attention

Abstract

The invention discloses an end-to-end target detection method based on shape correction and feature selection cross attention, and belongs to the technical field of image recognition methods.The end-to-end target detection method based on shape correction and feature selection cross attention comprises the steps that a backbone network extracts input image features; the encoder further enhances the extracted image features; the decoder iteratively optimizes the query by using a group of learnable relationship between the query reasoning target and the global context of the image; and the detection head outputs a prediction result. According to the method, a feature selection layer is added in the model, image features are screened by using a prediction frame generated by a previous decoder, and the screened features are used for calculating cross attention of a next decoder, so that query is optimized more accurately; and introducing a shape correction method as supervision, and adding the shape correction method as a new loss into the total loss, so that the model pays attention to the shape deviation of the prediction frame and the real frame, thereby more accurately estimating the bounding box.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition methods, and specifically relates to an end-to-end target detection method based on shape correction and feature selection cross-attention. Background Art

[0002] Object detection has become one of the important research topics in the field of computer vision and is widely used in intelligent driving, medical imaging, smart cities and other fields. Existing object detectors are mainly divided into detectors based on convolutional neural networks and detectors based on DETR proposed in recent years. Most object detectors based on convolutional neural networks require manually designed post-processing, such as NMS post-processing and anchor generation;

[0003] DETR provides a new paradigm for object detection, which uses a set of learnable queries to infer the relationship between the object and the global context of the image and outputs the final set of predictions. In recent years, object detectors based on DETR have made significant progress and the detection performance has been continuously improved;

[0004] The study found that DETR ignores the deviation between the predicted box shape and the true box shape, which makes it difficult to estimate the target bounding box very accurately. This is because the bounding box width and height estimated by the model are optimized separately without considering the influence of the bounding box shape. At the same time, DETR lacks target position priors, which leads to inaccurate model predictions. In order to address the impact of shape deviation, an end-to-end target detection method based on shape correction and feature selection cross-attention is provided. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide an end-to-end target detection method based on shape correction and feature selection cross-attention.

[0006] The technical solution adopted to solve the above technical problems is:

[0007] The end-to-end object detection method based on shape correction and feature selection cross-attention includes the following steps:

[0008] Build an end-to-end object detection model;

[0009] The backbone network is responsible for extracting input image features;

[0010] The encoder further enhances the extracted image features;

[0011] The decoder iteratively optimizes the query using a set of learnable query reasoning objectives and the relationship between the global context of the image;

[0012] The detection head outputs the prediction results.

[0013] Furthermore, the construction of the end-to-end target detection model includes: constructing a backbone network for extracting input image features.

[0014] Furthermore, the encoder further enhances the extracted image features, including establishing a multi-layer encoder, the multi-layer encoder has at least six layers, and the multi-layer encoder adopts a series structure.

[0015] Through the above technical solution, the serial structure of multi-layer encoders provides better feature extraction and representation capabilities. Through layer-by-layer abstraction and feature refinement, it can help the model achieve better performance on complex data.

[0016] Furthermore, the decoder uses a set of learnable query reasoning objectives and the relationship between the global context of the image to iteratively optimize the query, including the following specific steps:

[0017] Input query data and encoded image feature data;

[0018] Get the prediction box and classification confidence score;

[0019] Calculate the intersection over union (IoU) of the real box and the predicted box to get the IoU score.

[0020] The classification confidence score and loU score are combined as the basis for screening the TopK prediction boxes, constraining the cross-attention calculation range of the next layer decoder.

[0021] Through the above technical solution, it makes full use of the original DETR decoder to optimize the query layer by layer to make it more accurate, and uses the classification confidence and loU score as the basis for the prediction box constraint of TopK selection. This not only eliminates the mismatch problem between the classification confidence and loU score, but also saves the computational effort caused by the participation of some background area features in the cross-attention calculation.

[0022] Furthermore, the decoder iteratively optimizes the query using a set of learnable query reasoning objectives and the relationship between the global context of the image, and further includes:

[0023] Introduce shape correction loss to supervise the shape of the predicted box;

[0024] Calculate the shape deviation between the predicted box and the true box.

[0025] Through the above technical solution, the loss of the prediction box shape deviation is added to the total loss as the shape correction loss, and the prediction box shape is supervised, so that the prediction box shape is more accurate.

[0026] Further, the cross attention of the i+1th layer decoder is rewritten as: The multi-layer encoder is established and the feature selection layer is introduced, including:

[0027] Rewrite the i+1th layer decoder cross attention as:

[0028] S i =f class (Q i ), Box i =f box (Q i ), IoU i =IoU(Box i , GT) (1)

[0029]

[0030]

[0031] Among them, Q i is the query vector output by the i-th layer, S i It's Q i The classification confidence of the i-th layer prediction box obtained by the classification branch, Box i It's Q i The i-th layer prediction box obtained by the regression branch, IoU i is the IoU value calculated between the i-th layer prediction box and the real box GT, TopkBox i The i-th level classification confidence S i And the IoU with the real box i The product of is the TopK prediction box selected according to, V i+1 It's TopkBox i Local features corresponding to feature map locations.

[0032] Furthermore, a method is introduced to calculate the shape deviation l between the predicted box and the real box. max The algorithm:

[0033] r w =W box / W GT (4)

[0034] r h =H box / H GT (5)

[0035]

[0036] L scale =L1(l max ,1) (9)

[0037] Among them, W box With W GT Indicates the width of the predicted box and the real box, H box With H GTIndicates the height of the predicted box and the true box, r w Represents the ratio of the width of the predicted box to the width of the real box, r h Represents the ratio of the height of the predicted box to the height of the real box, shape deviation l max The maximum deviation between the predicted box and the true box in the width and height directions is measured. This method is only used for positive samples that are successfully matched by the Hungarian algorithm.

[0038] Through the above technical solution, the shape deviation between the predicted box and the real box can be obtained through the formula. When the ratio of the width and height of the predicted box to the width and height of the real box is 1, there is no deviation between the shape of the predicted box and the shape of the real box.

[0039] An end-to-end object detection system based on cross-attention for shape correction and feature selection, including:

[0040] Backbone network module, the backbone network module is responsible for extracting input image features;

[0041] Encoder module, the encoder module further enhances the extracted image features;

[0042] The decoder module iteratively optimizes the query using a set of learnable query reasoning objectives and the relationship between the global context of the image;

[0043] Detection head module,The detection head module is used to output the prediction results.

[0044] The beneficial effects of the present invention are as follows:

[0045] (1) The present invention adds a feature selection layer to the model, uses the prediction box generated by the previous decoder to filter the image features, and the filtered features are used to calculate the cross attention of the next decoder, so as to optimize the query more accurately; introduces a shape correction method as supervision, and adds it as a new loss to the total loss, so that the model pays attention to the shape deviation between the prediction box and the real box, so as to estimate the bounding box more accurately;

[0046] (2) The present invention makes full use of the original DETR decoder to optimize the query layer by layer to make it more accurate through the feature selection layer structure, and performs the prediction box constraint of TopK selection based on the combination of classification confidence and IoU score. This not only eliminates the mismatch problem between classification confidence and IoU score, but also saves the computational complexity caused by the participation of some background area features in cross-attention calculation;

[0047] (3) The present invention introduces the prediction box constraint and the serial structure of the decoder. The prediction box constraint not only brings the prior centered on the target position, but also improves the accuracy of the cross-attention of each layer as the serial structure of the decoder is continuously iteratively optimized. The accurate cross-attention in turn obtains a more accurate prediction box through the iterative optimization of the decoder. The design of the feature selection structure and the addition of the shape correction loss make the optimization of the prediction box more accurate, thereby further improving the performance of the DETR-like model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the overall model detection method flow of the present invention;

[0049] Figure 2 It is a specific structural diagram of the model of the present invention;

[0050] Figure 3 It is a schematic diagram of the process flow of the feature selection layer in the model of the present invention;

[0051] Figure 4 It is a schematic diagram of the process flow of the feature selection layer in the model of the present invention;

[0052] Figure 5 It is a specific flow chart of feature selection of the feature selection layer in the model of the present invention;

[0053] Figure 6 It is a schematic diagram of the process of introducing shape correction by feature selection in the model of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] like Figure 1 - Figure 6 As shown, this embodiment provides an end-to-end object detection method based on shape correction and feature selection cross attention, including the following steps:

[0056] Step 100: Build an end-to-end object detection model;

[0057] Including: building a feature extraction backbone network to extract input image features, among which common backbone networks include VGGNet (developed by the Visual Geometry Group of Oxford University, using multiple 3x3 small-size convolution kernels, a step size of 1, a padding of 1, and a 2x2 maximum pooling layer to reduce the size of the feature map. Its hierarchical structure is simple, and a deep network is constructed by repeatedly stacking simple convolution layers), GoogLeNet (developed by Google, introducing the Inception module, which uses 1x1, 3x3, 5x5 convolutions and 3x3 maximum pooling in parallel to capture features of different scales, and reduces the number of parameters and improves computational efficiency through structural innovation within the network), ResNet (proposed by Microsoft Research, introducing residual learning to solve the gradient vanishing problem in deep network training, and using jump connections to allow gradients to flow directly to earlier layers), DenseNet (each layer is connected to all previous layers to achieve feature reuse, reduce the number of parameters, and further improve parameter efficiency through innovative connection methods), MobileNet (developed by Google, designed for mobile and edge devices, introducing deep separable convolutions to significantly reduce computational effort and model size).

[0058] Specifically, when building a backbone network, first select a suitable backbone network architecture according to task requirements, such as the above-mentioned VGG, ResNet, etc.;

[0059] Then, the basic units of the network are constructed using convolutional layers, pooling layers, batch normalization layers, and activation layers.

[0060] These basic units are then stacked up to form a deep network structure;

[0061] It should be noted that for structures such as ResNet, skip connections or bottleneck layers (BottleneckLayer) need to be introduced.

[0062] Then set the number, size, step size and other parameters of each layer. For structures such as Inception, it is necessary to design multi-path parameter configuration.

[0063] Finally, use a large amount of data sets to train the network, and apply optimization algorithms such as SGD and Adam to adjust the network weights. You can perform transfer learning based on the pre-trained model, freeze some layers, and only train (fine-tune) the last few layers.

[0064] Building a backbone network is an iterative and experimental process that requires continuous adjustment and optimization based on specific application scenarios and performance requirements. As research progresses, new network structures will continue to be proposed.

[0065] Step 200: The backbone network is responsible for extracting input image features. When the image is input into the backbone network, the image features will be extracted.

[0066] Step 300: The encoder further enhances the extracted image features. In computer vision tasks, an encoder refers to a network component that is responsible for extracting and encoding features from input data (such as images). The role of the encoder is to abstract and represent the original data so that subsequent tasks (such as classification, detection, segmentation, etc.) can be more easily processed.

[0067] In this embodiment, a multi-layer encoder is established. The multi-layer encoder usually has six layers and adopts a series structure.

[0068] Specifically, the design of multi-layer encoders facilitates hierarchical feature learning. Multi-layer encoders can gradually extract low-level to high-level features from the original image. The first layer may only capture edges and textures, while deeper layers can recognize more complex shapes and object structures. As the number of layers increases, the encoder can learn more abstract and complex feature representations, which is crucial for understanding image content.

[0069] Each layer increases the nonlinearity of the model through nonlinear activation functions (such as ReLU), which enables the network to capture more complex function mappings. Deep networks can share parameters at different levels, which helps reduce the overall number of parameters of the model while maintaining strong representation capabilities. The multi-layer structure helps the model generalize at different levels and maintain good performance even when facing unseen data.

[0070] Deeper networks are usually better able to generalize to data outside the training set because they can learn more general feature representations. The cascade structure of the multi-layer encoder makes the network design more modular. Each layer can be regarded as an independent processing unit, which is easy to understand and modify. If a stronger model is needed, the encoder can be expanded by adding more layers without completely redesigning the network.

[0071] Multi-layer encoders can effectively reduce the size of feature maps, reduce computation and storage requirements, while retaining key information. In the layer-by-layer processing, the encoder can filter out the most important features and ignore unimportant details, which is very useful for subsequent tasks such as classification or detection. When processing images, different layers can capture features of different scales, which is crucial for understanding multi-scale objects in images (such as objects of different sizes).

[0072] In summary, the serial structure of multi-layer encoders provides better feature extraction and representation capabilities. Through layer-by-layer abstraction and feature refinement, it can help the model achieve better performance on complex data. The specific parameters are designed and adjusted in practical applications.

[0073] Step 400: The decoder uses a set of learnable query reasoning targets and the relationship between the global context of the image to iteratively optimize the query. In the original DETR decoder, the learnable query participates in the global feature cross-attention calculation and infers the global context relationship. The formula is as follows:

[0074]

[0075] Efficient DETR shows that the query will eventually converge to the area around the target. Therefore, focusing the query near the target features is beneficial to the query optimization and model convergence. Based on the above inference, we believe that the query should focus on the vicinity of the target features, while the background area features far away from the target do not contribute much to the query optimization.

[0076] To optimize the above problem, a specific algorithm is provided to rewrite the cross attention of the i+1 layer decoder as follows:

[0077] S i =f class (Q i ), Box i =f box (Q i ), IoU i =IoU(Box i , GT) (1)

[0078]

[0079]

[0080] Among them, Q i is the query vector output by the i-th layer, S i It's Q i The classification confidence of the i-th layer prediction box obtained by the classification branch, Box i It's Q i The i-th layer prediction box obtained by the regression branch, IoU i is the IoU value calculated between the i-th layer prediction box and the real box GT, TopkBox i The i-th level classification confidence S i And the IoU with the real box i The product of is the TopK prediction box selected according to, V i+1 It's TopkBox iLocal features corresponding to feature map locations.

[0081] Reference Figure 4 and Figure 5 , based on the feature of the original DETR decoder to optimize the query layer by layer to make it more accurate, the prediction box constraint of TopK selection is performed based on the combination of classification confidence and loU score. In general, the model only uses the classification confidence score to judge the quality of a prediction box, but this can only prove that the classification of this prediction box is accurate, and the IOU is not necessarily high. This has a mismatch problem, that is, the IOU of a good classification is not high, and a classification with a high IOU is not necessarily good. The present invention combines the classification confidence score and the I0U score, that is, the product of their multiplication (Si*IOUi in the formula) is used as the criterion for judging the quality of the prediction box, which eliminates the mismatch problem between the classification confidence and the loU score. At the same time, the constraint refers to constraining the cross-attention calculation range from the original global feature calculation to the local feature calculation covered by the prediction box. The original cross-attention calculation is in the global feature map feature calculation. In this solution, only the feature map features covered by the selected prediction box participate in the cross-attention calculation, which reduces the number of features involved in the operation, that is, reduces the amount of calculation. Formulas 1-3 and Figure 4 and Figure 5 Shows the process.

[0082] In the process of query optimization of image features, refer to Figure 3 , including the following specific steps:

[0083] Step 310: input query data and encoded image feature data;

[0084] Step 320: Obtain prediction box and classification confidence score;

[0085] Step 330: Calculate the intersection-over-union data of the real box and the predicted box to obtain the loU score;

[0086] Step 340: The classification confidence score and loU score are combined as the basis for screening the TopK prediction boxes, and the cross-attention calculation range of the next layer decoder is constrained.

[0087] When inputting image features, a prediction box is generated. The detection result is gradually optimized by comparing the prediction box with the real box. The decoder adopts a serial structure. The prediction box constraint not only brings the prior centered on the target position, but also is continuously iteratively optimized as the serial structure of the decoder is continuously optimized. Some features are screened and sent to the next layer of decoder to participate in the cross-attention calculation. The prediction box constraint improves the accuracy of each layer of cross-attention. Accurate cross-attention in turn obtains more accurate prediction boxes through iterative optimization of the decoder. The two complement each other and jointly promote the optimization of queries, and finally obtain high-quality prediction boxes.

[0088] In a further embodiment, in the process of decoder query optimization, the following specific steps are included, referring to Figure 6 :

[0089] Step 321: Introduce shape correction loss to supervise the shape of the prediction box;

[0090] Step 322: Calculate the shape deviation between the predicted box and the true box.

[0091] Shape correction is added as a new supervision to the training process. The cross-attention calculation is performed on the feature map features covered by the selected prediction box. The loss of the prediction box shape deviation is added to the total loss as the shape correction loss to supervise the prediction box shape, so that the prediction box shape is more accurate:

[0092] Specifically, a method is introduced to calculate the shape deviation l between the predicted box and the real box. max The algorithm:

[0093] r w =W box / W GT (4)

[0094] r h =H box / H GT (5)

[0095]

[0096] L scale =L1(l max ,1) (9)

[0097] Among them, W box With W GT Indicates the width of the predicted box and the real box, H box With H GT Indicates the height of the predicted box and the true box, r w Represents the ratio of the width of the predicted box to the width of the real box, r h Represents the ratio of the height of the predicted box to the height of the real box, shape deviation l max Measures the maximum deviation between the predicted box and the true box in the width and height directions. This method is only used for positive samples that are successfully matched by the Hungarian algorithm. The shape deviation between the predicted box and the true box can be obtained through the formula. When the ratio of the width and height of the predicted box to the width and height of the true box is 1, there is no deviation between the shape of the predicted box and the shape of the true box.

[0098] Step 500: The detection head outputs the prediction result. The query is optimized through the linear layer to obtain the final output. The detection head includes a parallel classification subnet, a regression subnet and an IoU subnet. The network will accept the feature map as input and generate classification, regression and IoU predictions through three parallel subnets.

[0099] Provide a specific implementation code (taking PyTorch as an example):

[0100]

[0101]

[0102]

[0103]

[0104] In this example, we define a DetectionHead class, which contains three subnets:

[0105] Classification subnet: Outputs the score for each category.

[0106] Regression subnet (regression_subnet): Outputs the location information of the bounding box.

[0107] IoU subnet (iou_subnet): Outputs the IoU value between the predicted bounding box and the true bounding box.

[0108] Each subnet first extracts features through a convolutional layer, and then passes through a fully connected layer for classification or regression prediction. The forward method processes these three subnets in parallel and returns their outputs.

[0109] Please note that the input_tensor size and number of channels in this example are hypothetical and can be adjusted according to the specific network architecture and task requirements. Similarly, the number of input features of the linear layer (e.g., classification subnet: nn.Linear(256,91), regression subnet: MLP(256,256,4,3), IoU subnet: nn.Linear(256,1)) can also be adjusted. This detection head implementation is a basic example and can be adjusted according to the actual feature map size.

[0110] An end-to-end object detection system based on cross-attention for shape correction and feature selection, including:

[0111] Backbone network module 11, the backbone network module 11 is responsible for extracting input image features;

[0112] An encoder module 12, the encoder module 12 further enhances the extracted image features;

[0113] Decoder module 13, decoder module 13 uses a set of learnable query reasoning objectives and the relationship between the global context of the image to iteratively optimize the query;

[0114] The detection head module 14 is used to output the prediction result.

[0115] Provide a carrier for the detection method so that the method can be implemented smoothly. The following are the experimental results after specific implementation:

[0116]

[0117] Table 1: Comparison of SF-DETR with other state-of-the-art methods based on DINO

[0118]

[0119]

[0120] Table 2: Comparison of SF-DETR with other methods on the COCO 2017 validation set under the 12-round training setting

[0121] Note:

[0122] Table 1 gives a comparison between the improved model (DINO version) with 4-scale feature layers based on ResNet50 as the backbone network and other DETR-like methods using single-scale or multi-scale feature maps. Table 1 also includes a comprehensive comparison with the classic two-stage detector FasterRCNN. The data in Table 1 show that the performance of this model exceeds that of the classic target detection model using more training rounds.

[0123] The above also includes the results of comparing the SF-DETR of this model with the baseline model DINO under the same parameter settings under the 12-round training setting. Without increasing the number of parameters, the SF-DETR of this model outperforms the original DINO (+1.0AP).

[0124] Compared with DINO, this model has significant improvements in the three detection standards of large, medium and small, with an increase of +0.8AP respectively. L , +1.4AP M , +1.9AP S .

[0125] At the same time, this model outperforms DINO with 36 rounds of training under the 24-round training setting. Table 2 shows the results of this model on the COCO2017 validation set with 12 rounds of training and the DETR-like methods using single-scale or multi-scale feature maps. The results of FasterRCNN using a 5-scale feature layer are also compared. Under the 12-round training setting, this model also outperforms other classic object detectors in the table.

[0126] Using the same settings as the baseline model DINO, our model also outperforms DINO (+0.4AP, +1.5AP S , +1.0AP M , +1.0AP L ), thanks to the design of the feature selection structure and the addition of shape correction loss, this model optimizes the prediction box more accurately, thereby further improving the performance of the DETR-like model.

[0127] The above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention.

Claims

1. An end-to-end object detection method based on shape correction and feature selection cross-attention, characterized in that: The following steps are involved: Build an end-to-end object detection model; The backbone network is responsible for extracting input image features; The encoder further enhances the extracted image features; The decoder iteratively optimizes the query using a set of learnable query reasoning objectives and the relationship between the global context of the image; The detection head outputs the prediction results.

2. The end-to-end object detection method based on shape correction and feature selection cross attention according to claim 1 is characterized in that: The construction of the end-to-end target detection model includes: constructing a backbone network for extracting input image features.

3. The end-to-end object detection method based on shape correction and feature selection cross attention according to claim 1 is characterized in that: The encoder further enhances the extracted image features, including establishing a multi-layer encoder, the multi-layer encoder has at least six layers, and the multi-layer encoder adopts a series structure.

4. The end-to-end object detection method based on shape correction and feature selection cross attention according to claim 1, characterized in that: The decoder uses a set of learnable query reasoning objectives and the relationship between the global context of the image to iteratively optimize the query, including the following specific steps: Input query data and encoded image feature data; Get the prediction box and classification confidence score; Calculate the intersection over union (IoU) of the real box and the predicted box to get the IoU score. The classification confidence score and the l oU score are combined as the basis for screening the TopK prediction boxes, constraining the cross-attention calculation range of the next layer decoder.

5. The end-to-end object detection method based on shape correction and feature selection cross attention according to claim 4, characterized in that: The decoder iteratively optimizes the query using a set of learnable query reasoning objectives and the relationship between the global context of the image, and also includes: Introduce shape correction loss to supervise the shape of the predicted box; Calculate the shape deviation between the predicted box and the true box.

6. The end-to-end object detection method based on shape correction and feature selection cross attention according to claim 3, characterized in that: Rewrite the i+1th layer decoder cross attention as: S i =f class (Q i ),Box i =f box (Q i ),IoU i =IoU(Box i ,GT) (1) Among them, Q i is the query vector output by the i-th layer, S i It is the classification confidence of the i-th layer prediction box obtained by Qi through the classification branch, Box i It's Q i The i-th layer prediction box obtained by the regression branch, IoU i is the IoU value calculated between the j-th layer prediction box and the real box GT, TopkBox i The i-th level classification confidence S i And the IoU with the real box i The product of is the TopK prediction box selected according to, V i+1 It's TopkBox i Local features corresponding to feature map locations.

7. The end-to-end object detection method based on shape correction and feature selection cross attention according to claim 5, characterized in that: Introduce a method to calculate the shape deviation l between the predicted box and the real box max The algorithm: r w =W box / W GT (4) r h =H box / H GT (5) L scale =L1(l max ,1) (9) Among them, W box With W GT Indicates the width of the predicted box and the real box, H box With H GT Indicates the height of the predicted box and the true box, r w Represents the ratio of the width of the predicted box to the width of the real box, r h Represents the ratio of the height of the predicted box to the height of the real box, shape deviation l max The maximum deviation between the predicted box and the true box in the width and height directions is measured. This method is only used for positive samples that are successfully matched by the Hungarian algorithm.

8. An end-to-end object detection system based on shape correction and feature selection cross-attention, characterized by: include: Backbone network module, the backbone network module is responsible for extracting input image features; Encoder module, the encoder module further enhances the extracted image features; The decoder module iteratively optimizes the query using a set of learnable query reasoning objectives and the relationship between the global context of the image; Detection head module,The detection head module is used to output the prediction results.

Images