Target detection error repair method based on decoupling DETR

Abstract

The invention relates to a target detection error repairing method based on a decoupling DETR, and the method comprises the steps: selecting a target detection basic network, and training the target detection basic network; giving a pre-training trunk and an input image, and obtaining a bounding box through a target detection regression part; according to the input image, obtaining category information of the image through a target detection classification part; and performing error correction on the output of the target detection basic network. The method has the beneficial effects that the regression and classification processes in the DETR model are decoupled, so that errors occurring in the positioning and identification processes can be corrected in a targeted manner. The decoupled model allows the generation of bounding boxes and the prediction of categories to be adjusted more meticulously, thereby significantly improving the accuracy of target detection.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and more specifically, to a target detection error repair method based on decoupling DETR. Background Art

[0002] Object detection, as one of the key tasks in computer vision, aims to identify objects in images and determine their locations. In more traditional object detection methods, such as R-CNN and YOLO, they extract image features based on convolutional neural networks (CNNs), and then perform separate classification and bounding box regression steps. However, these methods usually require complex heuristic operations, such as additional stage processing, anchor boxes, and non-maximum suppression (NMS) to filter redundant detections, making model optimization and interpretation difficult.

[0003] In recent years, the introduction of DETR (DEtection TRansformer) has revolutionized the field of object detection. DETR applies the Transformer architecture originally used for natural language processing tasks to object detection tasks, thus changing the traditional detection method. DETR uses a set-based global loss function to directly predict bounding boxes and category labels, eliminating the need for additional stage processing, anchor boxes, and non-maximum suppression. The core architecture of DETR consists of a backbone network, a Transformer encoder, a Transformer decoder, and a prediction head. The backbone network is used to extract image features, the encoder captures global context through a multi-head self-attention mechanism (MHSA), and the decoder is used to generate target queries and associate queries with specific image regions through a multi-head cross attention mechanism (MHCA).

[0004] Despite the many advantages of DETR, the original DETR model still has some limitations when processing object queries. In traditional DETR, since each query is responsible for both classification (assigning a label to the object) and regression (determining the location and size of the object through the bounding box), the model behavior is difficult to explain, especially when classification or bounding box errors occur, it is difficult to debug and analyze. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and propose a target detection error repair method based on decoupling DETR.

[0006] In the first aspect, a target detection error repair method based on decoupling DETR is provided, including: S1. Select a target detection basic network and train the target detection basic network; the target detection basic network is a DETR model; S2. Given the pre-trained backbone and input image, obtain the bounding box through the object detection regression part; S3, according to the input image, obtaining image category information through the target detection and classification part; S4. Correct the output of the target detection basic network.

[0007] Preferably, S2 comprises: S201, generating a reference bounding box according to the multi-head self-attention part related to the position query in the decoder; S202, generating offsets of each edge in the bounding box according to the relationship between each head in the multi-head mutual attention part of the decoder and the reference bounding box; S203: Calculate a final bounding box according to the reference bounding box and the offsets of each edge in the bounding box.

[0008] Preferably, S3 includes: S301, concatenating the category query representing N categories and the encoder output; S302, the concatenation result in S301 is passed through a multi-head self-attention mechanism to obtain a category query embedding; S303: Use the output of the target detection regression part and the category query embedding to perform inner product to obtain the category of each object in the image.

[0009] Preferably, S4 includes: S401, performing error correction on the target detection regression part, including: bounding box area error correction and bounding box offset error correction; S402, correcting errors in the target detection classification part, including: correcting errors in difficult categories and correcting errors in similar categories.

[0010] Preferably, in S401, the bounding box area correction includes: when the area of ​​the final bounding box does not match the actual area of ​​the object, limiting the area of ​​the reference box by adding an area loss function; the bounding box offset correction includes: when the offset of the bounding box does not match the actual position of the object, correcting the output result of the multi-head mutual attention part by adding an average feature loss function.

[0011] Preferably, in S402, the difficult category correction includes: determining the difficult category according to the training difficulty, and adding a category query corresponding to the difficult category; the similar category correction includes: determining the similar category according to the similarity, and increasing the query distance corresponding to the similar category.

[0012] In a second aspect, a target detection error repair system based on decoupling DETR is provided, which is used to execute any method described in the first aspect, including: A selection module is used to select a target detection basic network and train the target detection basic network; the target detection basic network is a DETR model; The first acquisition module is used to obtain the bounding box through the object detection regression part given the pre-trained backbone and the input image; A second acquisition module is used to acquire image category information according to the input image through the target detection and classification part; The error correction module is used to correct the output of the target detection basic network.

[0013] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0014] In a fourth aspect, an electronic device is provided, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement any method as described in the first aspect.

[0015] The beneficial effects of the present invention are: 1. By decoupling the regression and classification processes in the DETR model, the present invention can correct errors that occur in the positioning and recognition processes in a targeted manner. The decoupled model allows for more detailed adjustments to the generation of bounding boxes and the prediction of categories, thereby significantly improving the accuracy of target detection.

[0016] 2. The traditional DETR model couples classification and regression together, making it difficult to locate the problem when an error occurs. This invention separates these two processes to make the model's decision-making process more transparent, making it easier to understand and explain the model's behavior, especially when debugging and analyzing errors.

[0017] 3. The present invention can more accurately handle problems such as occlusion and the distinction between similar objects by specifically fixing errors in the target detection process, especially in complex scenes. This enables the model to have better generalization capabilities in variable and complex practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flow chart of a target detection error repair method based on decoupling DETR provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of the DETR model provided in the embodiment of the present application; Figure 3 A schematic diagram of the structure of a target detection error repair system based on decoupling DETR provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0020] Embodiment 1: To solve the problems of the prior art, Example 1 of the present application provides a method for repairing target detection errors based on decoupled DETR. The method uses position query and category query to perform regression and classification in target detection respectively, divides the generation of the bounding box into creating a reference box and generating each edge offset based on the mutual attention head and boundary relationship, and can perform special debugging and error correction for the structural characteristics of the decoupled DETR model, which can more accurately predict the position and category of the object.

[0021] like Figure 1 As shown in the figure, the core idea of ​​this method is to decouple the regression and classification processes of DETR target detection, so as to perform targeted error repair on the regression and classification parts respectively. Specifically, this method includes: S1. Select a target detection basic network and train the target detection basic network; the target detection basic network is a DETR model, such as Figure 2 As shown, the input image first passes through the pre-trained backbone and encoder to obtain image features, and then passes through the target detection regression part and the classification part in the decoder to obtain the bounding box and category. The regression part and the classification part are composed of multiple layers of multi-head self-attention and multi-head mutual attention. The present invention uses a DETR-based target detection implementation. DETR is a Transformer-based neural network architecture. Compared with traditional target detection methods, DETR abandons the two steps of anchor boxes and non-maximum suppression (NMS) to achieve a concise end-to-end target detection process. This means that it directly predicts the target category and bounding box from the full image features, reducing the intermediate steps and manually designed components in the detection process.

[0022] S2. Given the pre-trained backbone and input image, obtain the bounding box through the object detection regression part.

[0023] S2 includes: S201. Generate a reference bounding box based on the multi-head self-attention (MHSA) part related to the position query in the decoder.

[0024] Multi-head self-attention is a variant of multi-head attention (MHA). The output formula of multi-head attention is defined as follows:

[0025] Each head The calculation formula is:

[0026] Here Q, K, and V represent query, key, and value matrices respectively. , , is the weight matrix corresponding to the i-th attention head, which is used to transform the input to a different representation space. It is a weight matrix used to concatenate the outputs of all attention heads and transform them again. The multi-head self-attention mechanism is a mechanism introduced in the sequence to enhance the model to capture the relationship between different positions. In the present invention, the multi-head self-attention Q, K, V are all query q, that is:

[0027] S201 leverages the ability of MHSA to capture the relationship between different locations in the input image to generate a preliminary bounding box prediction that takes into account the approximate location and size of objects in the image. Query by location Do long self-attention and multiply it by a reference weight Get, that is:

[0028] S202. Generate offsets of each edge in the bounding box according to the relationship between each head and the reference bounding box in the multi-head mutual attention (MHCA) part of the decoder.

[0029] Multi-head mutual attention is also a variant of multi-head attention. Multi-head mutual attention is similar to multi-head self-attention, except that it uses queries and key-value pairs from two different sequences when calculating attention. This enables the model to establish a connection between two sequences. For example, in the object detection task, one sequence can be a query from the decoder, and the other sequence can be the output of the encoder, representing image features. In the present invention, Q of multi-head mutual attention is the query q, and K and V are the output m of the encoder, that is:

[0030] The present application has observed that the heads of the mutual attention in the original DETR correspond to the boundaries of the bounding box. Therefore, in order to strengthen this correspondence, the present invention designs constraints to explicitly bind the heads and edges to better fix the errors that may occur in the generation of each edge. Furthermore, in S202, the multi-head mutual attention part (MHCA) of the decoder will generate offsets for each edge based on the relationship between each head and the bounding box. In this step, by considering the relationship between the bounding box and the image features, the model can more accurately adjust the position and size of the bounding box to make it closer to the target object. Bounding box offset The calculation is first performed by the location query Perform multi-head mutual attention with the encoder output m and select the head related to the boundary, and then multiply it by the offset weight Get, that is

[0031] in, Represents the selection of the head corresponding to each boundary.

[0032] S203: Calculate a final bounding box according to the reference bounding box and the offsets of each edge in the bounding box.

[0033] In S203, the final bounding box (cx, cy, w, h) is calculated based on the reference bounding box and each boundary offset. Among them, cx and cy are the coordinates of the center point, and w and h are the width and height respectively. This step involves applying the offset to the reference bounding box to correct its position and size, thereby obtaining the final object detection regression bounding box b, that is:

[0034] Through the decoupling process of S201-S203, the object detection model can more effectively utilize image features and position information to predict the accurate bounding box of the object, thereby improving the accuracy and efficiency of detection.

[0035] S3. According to the input image, the category information of the image is obtained through the target detection and classification part.

[0036] In the classification process, previous methods directly use a linear layer for classification, but this method lacks interpretability and also makes it difficult to correct model errors. Therefore, the present invention introduces category query to query all category information in the image.

[0037] Specifically, S3 includes: S301, query the category representing N categories And the encoder output m is spliced.

[0038] S302: The concatenation result in S301 is passed through a multi-head self-attention mechanism to obtain the category query embedding .

[0039] The formula for S302 is:

[0040] Where [:N] means taking the first N sequences.

[0041] S303: Use the output of the target detection regression part (representing each bounding box) and the category query embedding to perform inner product to obtain the category of each object in the image.

[0042] The formula for S303 is:

[0043] Among them, FFN is a feed-forward neural network.

[0044] S4. Correct the output of the target detection basic network.

[0045] Embodiment 2: Based on Example 1, Example 2 of the present application provides a more specific target detection error repair method based on decoupling DETR, including: S1. Select a target detection basic network and train the target detection basic network; the target detection basic network is a DETR model.

[0046] S2. Given the pre-trained backbone and input image, obtain the bounding box through the object detection regression part.

[0047] S3. According to the input image, the category information of the image is obtained through the target detection and classification part.

[0048] S4. Correct the output of the target detection basic network.

[0049] S4 includes: S401, performing error correction on the target detection regression part, including: bounding box area error correction and bounding box offset error correction.

[0050] In S401, the bounding box area error correction includes: when the area of ​​the final bounding box does not match the actual area of ​​the object, limiting the area of ​​the reference box by adding an area loss function.

[0051] For example, after training, it is noticed that the area of ​​the predicted box of some objects does not match the actual area. Specifically, by counting the objects with a confidence greater than 0.8 but an interaction ratio less than 0.5, it is found that in addition to the data set annotation errors, the predicted areas of many objects are too large or too small, which leads to a small interaction ratio. At the same time, the area of ​​the model reference box and the area of ​​the final box are basically the same. Therefore, in order to deal with such errors, the present invention adds an area loss function Limit the area of ​​the reference frame to make it closer to the actual area of ​​the object. The formula is:

[0052] In addition, in order to make the border learned in the multi-head mutual attention consistent with the final border area, the present invention also limits the weight of its multi-head mutual attention, which can improve the performance and interpretability of the model.

[0053] The bounding box offset error correction includes: when the offset of the bounding box does not match the actual position of the object, correcting the output result of the multi-head mutual attention part by adding an average feature loss function.

[0054] For example, after training, it is noticed that the offset of the predicted box of some objects does not match the actual position. In the present invention, the offset of the bounding box is obtained by the MHCA part, so the present invention fixes this problem by analyzing this part. The result of MHCA can be regarded as the area of ​​the attention weight multiplied by the average feature of the image at the position where the attention weight is located, that is:

[0055] By comparing the attention weight area and average features of the wrong examples and positive examples, we found that the area of ​​the wrong examples and negative examples is basically the same, but the head4 in the average features of the wrong examples and negative examples is quite different, and the wrong examples are 10 times larger than the positive examples. Therefore, the present invention adds an average feature loss To correct the size of the average feature of head4:

[0056] S402, correcting errors in the target detection classification part, including: correcting errors in difficult categories and correcting errors in similar categories.

[0057] In S402, the difficulty category error correction includes: determining the difficulty category according to the training difficulty, and adding a category query corresponding to the difficulty category by copying and connecting the original difficulty category query to the end of the original category query.

[0058] For example, after training, it is found that some of the 91 categories in the COCO dataset are difficult to train, such as backpacks, spoons, sandwiches, etc., and the error rate will reach more than 25%. Therefore, the present invention increases the category queries corresponding to these difficult-to-train categories to improve the model's ability to identify difficult categories.

[0059] The similarity category error correction includes: determining a similarity category according to the similarity, and increasing a query distance corresponding to the similarity category.

[0060] For example, after training, it is found that some categories in the COCO dataset are relatively similar, such as trucks and cars, mice and remote controls, etc., with an error rate of more than 60%. Therefore, this application increases the distance between these similar category queries by calculating the L2 distance of the selected similar category queries and adding it to the loss function, so that the model can recognize these similar categories.

[0061] The present invention combines four error correction methods, namely, bounding box area error correction, bounding box offset error correction, difficult category error correction, and similar category error correction, and performs error correction in an integrated manner to obtain a better error correction effect.

[0062] In addition, the present invention conducted experiments on the MSCOCO (Microsoft Common Objects in Context) dataset and compared it with the Conditional DETR model (a variant of the DETR model that speeds up the training of the model) on which the present invention is based. First, 50 rounds of training were performed without error correction, and an AP (average precision) of 41.2 was achieved, which was 0.3 higher than the original ConditionalDETR. After that, the 50-round model was fine-tuned, and the error correction method of the present invention was used. The final effect is shown in Table 1, which is significantly improved compared with the result of 25 rounds of fine-tuning of Conditional DETR, reflecting the effectiveness of the method proposed in the present invention.

[0063] Table 1

[0064] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0065] Embodiment 3: Based on Examples 1 and 2, Example 3 of the present application provides a target detection error repair system based on decoupling DETR, such as Figure 2 As shown, including: A selection module is used to select a target detection basic network and train the target detection basic network; the target detection basic network is a DETR model; The first acquisition module is used to obtain the bounding box through the object detection regression part given the pre-trained backbone and the input image; A second acquisition module is used to acquire image category information according to the input image through the target detection and classification part; The error correction module is used to correct the output of the target detection basic network.

[0066] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiment 1. Therefore, the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be repeated in this application.

Claims

1. A target detection error repair method based on decoupling DETR, characterized in that: include: S1. Selecting a target detection basic network and training the target detection basic network; The target detection basic network is a DETR model; S2. Given the pre-trained backbone and input image, obtain the bounding box through the object detection regression part; S3, according to the input image, obtaining image category information through the target detection and classification part; S4. Correct the output of the target detection basic network.

2. The target detection error repair method based on decoupling DETR according to claim 1 is characterized in that S2 include: S201, generating a reference bounding box according to the multi-head self-attention part related to the position query in the decoder; S202, generating offsets of each edge in the bounding box according to the relationship between each head in the multi-head mutual attention part of the decoder and the reference bounding box; S203: Calculate a final bounding box according to the reference bounding box and the offsets of each edge in the bounding box.

3. The target detection error repair method based on decoupling DETR according to claim 2 is characterized in that S3 include: S301, concatenating the category query representing N categories and the encoder output; S302, the concatenation result in S301 is passed through a multi-head self-attention mechanism to obtain a category query embedding; S303: Use the output of the target detection regression part and the category query embedding to perform inner product to obtain the category of each object in the image.

4. The target detection error repair method based on decoupling DETR according to claim 3 is characterized in that: S4 includes: S401, performing error correction on the target detection regression part, including: bounding box area error correction and bounding box offset error correction; S402, correcting errors in the target detection classification part, including: correcting errors in difficult categories and correcting errors in similar categories.

5. The target detection error repair method based on decoupling DETR according to claim 4 is characterized in that: In S401, the bounding box area correction includes: when the area of ​​the final bounding box does not match the actual area of ​​the object, limiting the area of ​​the reference box by adding an area loss function; the bounding box offset correction includes: when the offset of the bounding box does not match the actual position of the object, correcting the output result of the multi-head mutual attention part by adding an average feature loss function.

6. The target detection error repair method based on decoupling DETR according to claim 5 is characterized in that: In S402, the difficulty category correction includes: determining the difficulty category according to the training difficulty, and adding a category query corresponding to the difficulty category; the similarity category correction includes: determining the similarity category according to the similarity, and adding a query distance corresponding to the similarity category.

7. The target detection error repair system based on decoupling DETR is characterized by: Used to perform the method according to any one of claims 1 to 6, comprising: A selection module is used to select a target detection basic network and train the target detection basic network; the target detection basic network is a DETR model; The first acquisition module is used to obtain the bounding box through the object detection regression part given the pre-trained backbone and the input image; A second acquisition module is used to acquire image category information according to the input image through the target detection and classification part; The error correction module is used to correct the output of the target detection basic network.

8. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 6.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 6.

Images