Conditional DETR image detection method based on image attention weighted enhancement

Abstract

The invention discloses a Conditional DETR image detection method based on graph attention weighting enhancement. The method comprises the steps of preprocessing a data set, constructing a Conditional DETR network, training the Conditional DETR network, storing a model, and testing the Conditional DETR network. The Conditional DETR network model is formed by sequentially connecting a backbone network, an encoder-decoder feature aggregation module and a prediction network in series, the backbone network takes ResNet50 as the backbone network to extract image features, and the image features are used for input of an encoder in the feature aggregation module; the encoder-decoder feature aggregation module is formed by connecting a self-attention mechanism module and a cross attention mechanism module in series; and the prediction network performs classification and regression prediction by using encoder-decoder output features, so that accurate detection of a target is realized. The method has the advantages of high detection precision, strong robustness and the like, and can be used for target detection of images.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to target detection on images. Background Art

[0002] Object detection mainly includes two parts: object classification and position regression. In the fields of object tracking, pedestrian re-identification, and video understanding technology, object detection is the core that affects performance. Since various objects have different appearances, shapes, and postures, and are interfered by factors such as illumination and occlusion during imaging, object detection is still one of the technical problems to be solved in the field of computer vision technology. The core of the problem mainly includes determining the category of a certain area in the image, accurately locating the target, and solving the problem of shape differences among targets with diverse sizes.

[0003] In convolutional neural networks, convolution operations are usually used to extract local features, which have certain limitations in capturing global feature representations. Transformer can reflect complex spatial transformations and long-distance feature dependencies through cascaded attention modules, thereby obtaining a more comprehensive global feature representation. In the process of target detection, the model needs to effectively distinguish between foreground and background, and the semantic connection between foregrounds needs to be considered. Global features can increase the model's field of view and effectively improve target detection performance. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a Conditional DETR image detection method based on graph attention weighted enhancement with high target detection accuracy and strong robustness.

[0005] The technical solution used to solve the above technical problems consists of the following steps:

[0006] (1) Dataset preprocessing

[0007] The COCO 2017 image dataset is divided into a training set and a test set at a ratio of 9:1, and both the training set and the test set have 80 target categories. The images are normalized in red, green and blue channels, and randomly cropped and scaled, with the longest side of the image being no larger than 1333 pixels and the shortest side being no smaller than 800 pixels.

[0008] (2) Building a Conditional DETR Network

[0009] The Conditional DETR network consists of a backbone network, an encoder, a decoder, and a prediction network connected in series.

[0010] The backbone network is ResNet50.

[0011] The encoder is composed of self-attention mechanism module 1, self-attention mechanism module 2, self-attention mechanism module 3, self-attention mechanism module 4, and graph attention mechanism module; self-attention mechanism module 1 is connected in series with self-attention mechanism module 2, self-attention mechanism module 3, and self-attention mechanism module 4 in sequence, and the other output end of self-attention mechanism module 1, self-attention mechanism module 2, self-attention mechanism module 3, and self-attention mechanism module 4 is connected to the graph attention mechanism module.

[0012] The decoder is composed of decoder layer 1, decoder layer 2, decoder layer 3, decoder layer 4, decoder layer 5 and decoder layer 6 connected in series in sequence.

[0013] The decoder layer 1 is composed of a self-attention mechanism module 5, a reference point generation module 1, a semantic transformation module 1, a position transformation module 1, a cross-attention mechanism module 1, and a fully connected layer 3; the output ends of the self-attention mechanism module 1, the reference point generation module 1, the semantic transformation module 1, and the position transformation module 1 are connected to the fully connected layer 3 through the cross-attention mechanism module 1.

[0014] The decoder layer 2 is composed of a self-attention mechanism module 6, a reference point generation module 2, a semantic transformation module 2, a position transformation module 2, a cross-attention mechanism module 2, and a fully connected layer 4; the output ends of the self-attention mechanism module 2, the reference point generation module 2, the semantic transformation module 2, and the position transformation module 2 are connected to the fully connected layer 4 through the cross-attention mechanism module 2, and the input end of the cross-attention mechanism module 2 is connected to the output end of the cross-attention mechanism module 1.

[0015] The structures of the decoder layer 3 and the decoder layer 5 are the same as that of the decoder layer 1 , and the structures of the decoder layer 4 and the decoder layer 6 are the same as that of the decoder layer 2 .

[0016] (3) Training Conditional DETR Network

[0017] 1) Determine the target loss function

[0018] Determine the target loss function Loss according to formula (1):

[0019] The target loss function Loss contains the classification loss function L cls And the regression loss function L box ;

[0020] Loss = L cls +L box (1)

[0021] Determine the classification loss function L according to formula (2): cls :

[0022]

[0023] Where N is the number of targets, M is the number of categories, and N and M are finite positive integers; y ic is the label of target i of category c; p ic is the predicted probability that the observed sample i belongs to category c, p ic ∈(0,1).

[0024] Determine the regression loss function L according to formula (3): box :

[0025]

[0026] Among them, b i Represents the real annotation box coordinates of target i, which is a four-dimensional vector, representing the upper left corner horizontal coordinate, upper left corner vertical coordinate, lower right corner horizontal coordinate, and lower right corner vertical coordinate, respectively. is the predicted annotation box coordinates of target i, in the same format as b i Same as; ||·|| represents the L1 norm, and GIoU(·) represents the function operation of the generalized intersection-over-union ratio of two boxes.

[0027] 2) Determine the evaluation index mAP

[0028] The specific steps are as follows:

[0029] ① Set the intersection-over-union ratio threshold t, which ranges from 0.5 to 0.75. Arrange the prediction boxes of category c in descending order of prediction confidence, take the first k prediction boxes, compare the consistency between the predicted category and the true category, the intersection-over-union ratio between the predicted box and the true box, and calculate the statistical accuracy P. c and recall R c .

[0030] Determine the accuracy P according to formula (4): c :

[0031]

[0032] Determine the recall rate R according to formula (5): c :

[0033]

[0034] Among them, TP c is a true positive, FP c It is a false positive, FN c It is a false negative, TP c , FP c 、FN c are all finite positive integers, and P c , R c TP c , FPc 、FN c Different from 0 at the same time;

[0035] ②Use precision P c and recall R c Draw PR c The curve is determined as the average precision AP of category c according to formula (6). c ;

[0036]

[0037] Among them, i represents the index of the positive sample belonging to category c, K represents the number of positive samples, and R i It represents the rank of positive sample i after sorting by predicted probability from small to large, L represents the number of negative samples, K and L are finite positive integers, and the sum of K and L is equal to k.

[0038] ③ Determine all categories of AP according to formula (7) c The mean mAP of

[0039]

[0040] 3) Training Conditional DETR Network

[0041] The training set is input into the Conditional DETR network for training. The training parameters are: the number of images per batch is 2, the number of training rounds is 50, the learning rate is γ, and the value of γ is 10 -5 or 10 -4 , using the AdamW optimizer, iterate until the target loss function converges.

[0042] (4) Save the model

[0043] During the training of the Conditional DETR network, the weights are continuously updated using a deep learning framework and the weight files are saved.

[0044] (5) Testing the Conditional DETR Network

[0045] Input the test set into the trained Conditional DETR network for testing, load the saved weight file, and obtain the Conditional DETR image target detection result.

[0046] In step (2) of the present invention, the Conditional DETR network is constructed. The self-attention mechanism module 1 of the present invention is composed of a self-attention calculation layer 1 and a normalization layer 1, a fully connected layer 1, a ReLU activation function layer 1, a Dropout layer 1, a fully connected layer 2, a Dropout layer 2, and a normalization layer 2 connected in series in sequence.

[0047] The structures of the self-attention mechanism module 2, self-attention mechanism module 3, self-attention mechanism module 4, self-attention mechanism module 5, and self-attention mechanism module 6 of the present invention are the same as the structure of the self-attention mechanism module 1.

[0048] In step (2) of the present invention, the Conditional DETR network is constructed, the reference point generation module 1 of the decoder is composed of a fully connected layer 5 and a fully connected layer 6 connected in series; the structures of the reference point generation module 2, the reference point generation module 3, the reference point generation module 4, the reference point generation module 5, and the reference point generation module 6 are the same as the structure of the reference point generation module 1.

[0049] In step (2) of the present invention, the Conditional DETR network is constructed, and the cross-attention mechanism module 1 in the decoder is composed of a query vector semantic transformation module 1 and a query vector position transformation module 1, a query vector position embedding transformation module 1, a key vector semantic change module 1, a key vector position transformation module 1, a value vector transformation module 1, and a cross-attention calculation layer 1; wherein the output ends of the query vector semantic transformation module 1, the query vector position transformation module 1, the query vector position embedding transformation module 1, the key vector semantic change module 1, the key vector position transformation module 1, and the value vector transformation module 1 are connected to the cross-attention calculation layer 1.

[0050] The structures of query vector semantic transformation module 1, query vector position transformation module 1, query vector position embedding mapping module 1, key vector semantics 1, key vector position transformation module 1, and value vector transformation module 1 are all single-layer fully connected layers.

[0051] The structures of cross-attention mechanism module 2, cross-attention mechanism module 3, cross-attention mechanism module 4, cross-attention mechanism module 5, and cross-attention mechanism module 6 in the decoder are the same as that of cross-attention mechanism module 1.

[0052] In step (3) of the present invention, in which the Conditional DETR network is trained, 1) in formula (6) of determining the target loss function, i represents the index of the positive sample belonging to category c, K represents the number of positive samples, and R i It indicates the rank of positive sample i after sorting by predicted probability from small to large, L indicates the number of negative samples, and the values ​​of K and L are 2 5 ~2 7 .

[0053] The present invention uses Conditional DETR as the baseline model, decouples the cross-attention calculation in the decoder, and proposes an encoder output feature adaptive enhancement method and a decoder dynamic learning method respectively. The encoder is guided to extract regional features in a divided manner at each layer, so that different layers focus on the extraction of features in different regions and enrich the semantics of the value vector; it effectively avoids the interference of the coarse-grained learning of the first layer of the decoder on the attention weight calculation of the remaining layers, improves the stability of the attention calculation of the remaining layers, and improves the flexibility of the reference point of the prediction box. After the encoder and decoder of the method proposed in the present invention are combined, the small target recognition performance of the Conditional DETR network is improved, making the prediction box more refined. Compared with the prior art, the present invention has the advantages of high target detection accuracy and strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of Example 1 of the present invention.

[0055] Figure 2 It is a structural diagram of the Conditional DETR network.

[0056] Figure 3 yes Figure 2 Schematic diagram of the encoder structure.

[0057] Figure 4 yes Figure 3 Schematic diagram of the structure of self-attention mechanism module 1.

[0058] Figure 5 yes Figure 2 Schematic diagram of the decoder structure.

[0059] Figure 6 yes Figure 5 Schematic diagram of the structure of decoder layer 1 and decoder layer 2.

[0060] Figure 7 yes Figure 6 Schematic diagram of the structure of the reference point generation module 1.

[0061] Figure 8 yes Figure 6 Schematic diagram of the structure of the cross-attention mechanism module 1.

[0062] Fig. 9 It is a visualization diagram of the test results of Example 1 and the comparative experiment. DETAILED DESCRIPTION

[0063] The present invention is further described below with reference to the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0064] Example 1

[0065] Figure 1A flow chart of this embodiment 1 is given. Figure 1 In this example, the Conditional DETR image detection method based on graph attention weighted enhancement consists of the following steps:

[0066] (1) Dataset preprocessing

[0067] The COCO 2017 image dataset is divided into a training set and a test set at a ratio of 9:1, and both the training set and the test set have 80 target categories. The images are normalized in red, green and blue channels, and randomly cropped and scaled, with the longest side of the image being no larger than 1333 pixels and the shortest side being no smaller than 800 pixels.

[0068] (2) Building a Conditional DETR Network

[0069] Figure 2 A schematic diagram of the structure of the Conditional DETR network of this embodiment is given. Figure 2 In this embodiment, the Conditional DETR network is composed of a backbone network, an encoder, a decoder, and a prediction network connected in series in sequence.

[0070] The backbone network of this embodiment is ResNet50.

[0071] Figure 3 Given Figure 2 Schematic diagram of the encoder structure. Figure 3 In the embodiment, the encoder is composed of self-attention mechanism module 1, self-attention mechanism module 2, self-attention mechanism module 3, self-attention mechanism module 4, and graph attention mechanism module; self-attention mechanism module 1 is connected in series with self-attention mechanism module 2, self-attention mechanism module 3, and self-attention mechanism module 4 in sequence, and the other output end of self-attention mechanism module 1, self-attention mechanism module 2, self-attention mechanism module 3, and self-attention mechanism module 4 is connected to the graph attention mechanism module.

[0072] Figure 4 Given Figure 3 Schematic diagram of the structure of the self-attention mechanism module 1. Figure 4 In the embodiment, the self-attention mechanism module 1 is composed of a self-attention calculation layer 1 and a normalization layer 1, a fully connected layer 1, a ReLU activation function layer 1, a Dropout layer 1, a fully connected layer 2, a Dropout layer 2, and a normalization layer 2 connected in series in sequence. The structures of the self-attention mechanism module 2, the self-attention mechanism module 3, and the self-attention mechanism module 4 are the same as the structure of the self-attention mechanism module 1.

[0073] Figure 5 Given Figure 2 Schematic diagram of the decoder structure. Figure 5 In the embodiment, the decoder is composed of decoder layer 1, decoder layer 2, decoder layer 3, decoder layer 4, decoder layer 5, and decoder layer 6 connected in series in sequence;

[0074] Figure 6 Given Figure 5 Schematic diagram of the structure of decoder layer 1 and decoder layer 2. Figure 6 In the embodiment, the decoder layer 1 is composed of a self-attention mechanism module 5, a reference point generation module 1, a semantic transformation module 1, a position transformation module 1, a cross-attention mechanism module 1, and a fully connected layer 3; the output ends of the reference point generation module 1, the semantic transformation module 1, and the position transformation module 1 are connected to the fully connected layer 3 through the cross-attention mechanism module 1.

[0075] The decoder layer 2 of this embodiment is composed of a self-attention mechanism module 6, a reference point generation module 2, a semantic transformation module 2, a position transformation module 2, a cross-attention mechanism module 2, and a fully connected layer 4; the output ends of the self-attention mechanism module 2, the reference point generation module 2, the semantic transformation module 2, and the position transformation module 2 are connected to the fully connected layer 4 through the cross-attention mechanism module 2, and the input end of the cross-attention mechanism module 2 is connected to the output end of the cross-attention mechanism module 1.

[0076] The structures of decoder layer 3 and decoder layer 5 of this embodiment are the same as that of decoder layer 1 , and the structures of decoder layer 4 and decoder layer 6 are the same as that of decoder layer 2 .

[0077] Figure 7 Given Figure 6 Schematic diagram of the reference point generation module 1. Figure 7 In the embodiment, the reference point generation module 1 of the decoder is composed of a fully connected layer 5 and a fully connected layer 6 connected in series; the structures of the reference point generation module 2, the reference point generation module 3, the reference point generation module 4, the reference point generation module 5, and the reference point generation module 6 are the same as the structure of the reference point generation module 1.

[0078] Figure 8 Given Figure 6 Schematic diagram of the cross-attention mechanism module 1. Figure 8 In the decoder of this embodiment, the cross-attention mechanism module 1 is composed of a query vector semantic transformation module 1, a query vector position transformation module 1, a query vector position embedding transformation module 1, a key vector semantic change module 1, a key vector position transformation module 1, a value vector transformation module 1, and a cross-attention calculation layer 1; wherein, the output ends of the query vector semantic transformation module 1, the query vector position transformation module 1, the query vector position embedding transformation module 1, the key vector semantic change module 1, the key vector position transformation module 1, and the value vector transformation module 1 are connected to the cross-attention calculation layer 1.

[0079] In this embodiment, the query vector semantic transformation module 1, query vector position transformation module 1, query vector position embedding mapping module 1, key vector semantics 1, key vector position transformation module 1, and value vector transformation module 1 are all structured as single-layer fully connected layers.

[0080] The structures of cross-attention mechanism module 2, cross-attention mechanism module 3, cross-attention mechanism module 4, cross-attention mechanism module 5, and cross-attention mechanism module 6 in the decoder of this embodiment are the same as the structure of cross-attention mechanism module 1.

[0081] (3) Training Conditional DETR Network

[0082] 1) Determine the target loss function

[0083] The target loss function Loss contains the classification loss function L cls And the regression loss function L box ;

[0084] Determine the target loss function Loss according to formula (1):

[0085] Loss = L cls +L box (1)

[0086] Determine the classification loss function L according to formula (2): cls :

[0087]

[0088] Where N is the number of targets, M is the number of categories, and N and M are finite positive integers; y ic is the label of target i of category c; p ic is the predicted probability that the observed sample i belongs to category c, p ic ∈(0,1), p in this embodiment ic The value is 0.5;

[0089] Determine the regression loss function L according to formula (3): box :

[0090]

[0091] Among them, b i Represents the real annotation box coordinates of target i, which is a four-dimensional vector, representing the upper left corner horizontal coordinate, upper left corner vertical coordinate, lower right corner horizontal coordinate, and lower right corner vertical coordinate, respectively. is the predicted annotation box coordinates of target i, in the same format as b iSame as; ||·|| represents the L1 norm, and GIoU(·) represents the function operation of the generalized intersection-over-union ratio of two boxes.

[0092] 2) Determine the evaluation index mAP

[0093] The specific steps are as follows:

[0094] ① Set the intersection-over-union ratio threshold t, where t is set to 0.5-0.75. In this embodiment, t is set to 0.5 and 0.75 respectively. Arrange the prediction boxes of category c in descending order of prediction confidence, and take the first k prediction boxes, where k is set to 2. 6 ~2 8 In this embodiment, the value of k is 2 7 , compare the consistency of the predicted category and the true category, the intersection ratio of the predicted box and the true box, and the statistical accuracy P c and recall R c .

[0095] Determine the accuracy P according to formula (4): c :

[0096]

[0097] Determine the recall rate R according to formula (5): c :

[0098]

[0099] Among them, TP c is a true positive, FP c It is a false positive, FN c It is a false negative, TP c , FP c 、FN c are all finite positive integers, and P c , R c TP c , FP c 、FN c Not all are 0.

[0100] ②Use precision P c and recall R c Draw PR c Curve, determine the average precision AP of category c according to formula (6) c ;

[0101]

[0102] Among them, i represents the index of the positive sample of category c, K represents the number of positive samples, and R i It indicates the rank of positive sample i after sorting by predicted probability from small to large, L indicates the number of negative samples, and the values ​​of K and L are 25 ~2 7 In this embodiment, the values ​​of K and L are 2 6 , the sum of K and L is equal to k.

[0103] ③ Determine all categories of AP according to formula (7) c The mean mAP of

[0104]

[0105] 3) Training Conditional DETR Network

[0106] The training set is input into the Conditional DETR network for training. The training parameters are: the number of images per batch is 2, the number of training rounds is 50, the learning rate is γ, and γ is 10 -5 or 10 -4 In this embodiment, the value of γ is 10 -5 , using AdamW optimizer, iterate until the target loss function converges;

[0107] (4) Save the model

[0108] During the training of the Conditional DETR network, the weights are continuously updated using a deep learning framework and the weight files are saved;

[0109] (5) Testing the Conditional DETR Network

[0110] Input the test set into the trained Conditional DETR network for testing, load the saved weight file, and obtain the Conditional DETR image target detection result.

[0111] Complete the Conditional DETR image detection method based on graph attention weighted enhancement.

[0112] Example 2

[0113] The Conditional DETR image detection method based on graph attention weighted enhancement in this example consists of the following steps:

[0114] (1) Dataset preprocessing

[0115] This step is the same as in Example 1.

[0116] (2) Building a Conditional DETR Network

[0117] This step is the same as in Example 1.

[0118] (3) Training Conditional DETR Network

[0119] 1) Determine the target loss function

[0120] The target loss function Loss contains the classification loss function L cls And the regression loss function L box ;

[0121] Classification loss function L cls And the regression loss function L box The expression is the same as that in Example 1.

[0122] Determine the classification loss function L according to formula (2): cls , in formula (2), p ic is the predicted probability that the observed sample i belongs to category c, p ic ∈(0,1), p in this embodiment ic The value is 0.1. In formula (2), the meanings and value ranges of other parameters are the same as those in embodiment 1.

[0123] Determine the regression loss function L according to formula (3): box , regression loss function L box The expression of is the same as that in Example 1. In formula (3), the meaning and value range of each parameter are the same as those in Example 1.

[0124] 2) Determine the evaluation index mAP

[0125] The specific steps are as follows:

[0126] ① Set the intersection-over-union ratio threshold t, where t is between 0.5 and 0.75. In this embodiment, t is 0.5. Arrange the prediction boxes of category c in descending order of prediction confidence, and take the first k prediction boxes. k is 2. 6 ~2 8 In this embodiment, the value of k is 2 6 , compare the consistency of the predicted category and the true category, the intersection ratio of the predicted box and the true box, and the statistical accuracy P c and recall R c .

[0127] The other steps of this step are the same as those in Example 1.

[0128] ②Use precision P c and recall R c Draw PR c Curve, determine the average precision AP of category c according to formula (6) c ;

[0129] The expression of formula (6) is the same as that of Example 1.

[0130] In formula (6), K represents the number of positive samples, R i It indicates the rank of positive sample i after sorting by predicted probability from small to large, L indicates the number of negative samples, and the values ​​of K and L are 2 5 ~2 7 In this embodiment, the values ​​of K and L are 2 5 , the sum of K and L is equal to k. The meanings and value ranges of other parameters are the same as those in Example 1.

[0131] The other steps in this step are the same as those in Example 1.

[0132] 3) Training Conditional DETR Network

[0133] The training set is input into the Conditional DETR network for training. The training parameters are: the number of images per batch is 2, the number of training rounds is 50, the learning rate is γ, and γ is 10 -5 or 10 -4 In this embodiment, the value of γ is 10 -4 , using the AdamW optimizer, iterate until the target loss function converges.

[0134] The other steps are the same as those in Example 1. The Conditional DETR image detection method based on graph attention weighted enhancement is completed.

[0135] Example 3

[0136] The Conditional DETR image detection method based on graph attention weighted enhancement in this example consists of the following steps:

[0137] (1) Dataset preprocessing

[0138] This step is the same as in Example 1.

[0139] (2) Building a Conditional DETR Network

[0140] This step is the same as in Example 1.

[0141] (3) Training Conditional DETR Network

[0142] 1) Determine the target loss function

[0143] The target loss function Loss contains the classification loss function L cls And the regression loss function L box

[0144] Classification loss function L cls And the regression loss function L box The expression is the same as that in Example 1.

[0145] Determine the classification loss function L according to formula (2): cls , in formula (2), p ic is the predicted probability that the observed sample i belongs to category c, p ic ∈(0,1), p in this embodiment ic The value is 0.9. In formula (2), the meanings and value ranges of other parameters are the same as those in embodiment 1.

[0146] Determine the regression loss function L according to formula (3): box , regression loss function L box The expression of is the same as that in Example 1. In formula (3), the meaning and value range of each parameter are the same as those in Example 1.

[0147] 2) Determine the evaluation index mAP

[0148] The specific steps are as follows:

[0149] ① Set the intersection-over-union ratio threshold t, which is 0.5 to 0.75. In this embodiment, t is 0.75. Arrange the prediction boxes of category c in descending order of prediction confidence, take the first k prediction boxes, and k is 2 6 ~2 8 In this embodiment, the value of k is 2 8 , compare the consistency of the predicted category and the true category, the intersection ratio of the predicted box and the true box, and the statistical accuracy P c and recall R c .

[0150] The other steps of this step are the same as those in Example 1.

[0151] ②Use precision P c and recall R c Draw PR c Curve, determine the average precision AP of category c according to formula (6) c ;

[0152] The expression of formula (6) is the same as that of Example 1.

[0153] In formula (6), K represents the number of positive samples, R i It indicates the rank of positive sample i after sorting by predicted probability from small to large, L indicates the number of negative samples, and the values ​​of K and L are 2 5 ~2 7 In this embodiment, the values ​​of K and L are 2 7 , the sum of K and L is equal to k. The meanings and value ranges of other parameters are the same as those in Example 1.

[0154] The other steps in this step are the same as those in Example 1.

[0155] 3) Training Conditional DETR Network

[0156] The training set is input into the Conditional DETR network for training. The training parameters are: the number of images per batch is 2, the number of training rounds is 50, the learning rate is γ, and γ is 10 -5 or 10 -4 In this embodiment, the value of γ is 10 -5 , using the AdamW optimizer, iterate until the target loss function converges.

[0157] The other steps are the same as those in Example 1. The Conditional DETR image detection method based on graph attention weighted enhancement is completed.

[0158] In order to verify the beneficial effects of the present invention, the inventors conducted a comparative simulation experiment using the Conditional DETR image detection method based on graph attention weighted enhancement in Example 1 of the present invention and the Conditional DETR method (hereinafter referred to as the comparative experiment). The evaluation code was used to test the model accuracy, and the evaluation function mAP was used as an indicator of the evaluation method. The larger the mAP value of the evaluation function, the better the method. The mAP with an intersection-over-union ratio threshold t of 0.5 and 0.75 was recorded as mAP 50 、mAP 75 , the mAP of small, medium, and large-sized targets is recorded as mAP S 、mAP M and mAP L , small size target area is less than 32 2 , the medium-sized target area is between 32 2 ~96 2 Large target area is greater than 96 2 .

[0159] The experimental results are shown in Table 1.

[0160] Table 1 Example 1 and comparative experimental test results

[0161] Experimental categories mAP <![CDATA[mAP 50 ]]> <![CDATA[mAP 75 ]]> <![CDATA[mAP S ]]> <![CDATA[mAP M ]]> <![CDATA[mAP L ]]> Comparative experiment 40.1 60.2 42.4 19.6 43.5 58.7 Example 1 41.9 62.2 44.3 21.5 45.1 60.4

[0162] As shown in Table 1, the method of the present invention has improved all evaluation indicators compared with the comparative experimental method. Among them, mAP is improved by 1.8% compared with the original model. 50 The mAP is improved by 2% at an intersection-over-union threshold of 0.75. 75 Improved mAP by 1.9% for small-sized targets S Improve mAP of medium-sized objects by 1.9% M Improve mAP of small-sized objects by 1.6%L An increase of 1.7%. Fig. 9 This is a visualization diagram of the test results of Example 1 and the comparative experiment. The first row is the original image, the second row is the comparative experiment test results, and the third row is the test results of Example 1. The green box is the real annotation, and the blue box is the predicted annotation. Fig. 9 This shows that the present invention can detect smaller and more objects in the image, and the prediction and annotation are more precise. This shows that the present invention can achieve more accurate image detection.

Claims

1. A Conditional DETR image detection method based on graph attention weighted enhancement, characterized in that It consists of the following steps: (1) Dataset preprocessing The COCO 2017 image dataset is divided into a training set and a test set at a ratio of 9:

1. Both the training set and the test set have 80 target categories. The images are normalized in red, green and blue channels, and randomly cropped and scaled, with the longest side of the image being no larger than 1333 pixels and the shortest side being no smaller than 800 pixels. (2) Building a Conditional DETR Network The Conditional DETR network is composed of a backbone network, an encoder, a decoder, and a prediction network in series; The backbone network is ResNet50; The encoder is composed of self-attention mechanism module 1, self-attention mechanism module 2, self-attention mechanism module 3, self-attention mechanism module 4, and graph attention mechanism module; self-attention mechanism module 1 is connected in series with self-attention mechanism module 2, self-attention mechanism module 3, and self-attention mechanism module 4 in sequence, and the other output ends of self-attention mechanism module 1, self-attention mechanism module 2, self-attention mechanism module 3, and self-attention mechanism module 4 are connected to the graph attention mechanism module; The decoder is composed of decoder layer 1, decoder layer 2, decoder layer 3, decoder layer 4, decoder layer 5, and decoder layer 6 connected in series in sequence; The decoder layer 1 is composed of a self-attention mechanism module 5, a reference point generation module 1, a semantic transformation module 1, a position transformation module 1, a cross-attention mechanism module 1, and a fully connected layer 3; the output ends of the self-attention mechanism module 1, the reference point generation module 1, the semantic transformation module 1, and the position transformation module 1 are connected to the fully connected layer 3 through the cross-attention mechanism module 1; The decoder layer 2 is composed of a self-attention mechanism module 6, a reference point generation module 2, a semantic transformation module 2, a position transformation module 2, a cross-attention mechanism module 2, and a fully connected layer 4; the output ends of the self-attention mechanism module 2, the reference point generation module 2, the semantic transformation module 2, and the position transformation module 2 are connected to the fully connected layer 4 through the cross-attention mechanism module 2, and the input end of the cross-attention mechanism module 2 is connected to the output end of the cross-attention mechanism module 1; The structures of the decoder layer 3 and the decoder layer 5 are the same as the structure of the decoder layer 1, and the structures of the decoder layer 4 and the decoder layer 6 are the same as the structure of the decoder layer 2; (3) Training Conditional DETR Network 1) Determine the target loss function Determine the target loss function Loss according to formula (1): The target loss function Loss contains the classification loss function L cls And the regression loss function L box ; Loss=L cls +L box (1) Determine the classification loss function L according to formula (2): cls : Where N is the number of targets, M is the number of categories, and N and M are finite positive integers; y ic is the label of target i of category c; p ic is the predicted probability that the observed sample i belongs to category c, p ic ∈(0,1); Determine the regression loss function L according to formula (3): box : Among them, b i Represents the real annotation box coordinates of target i, which is a four-dimensional vector, representing the upper left corner horizontal coordinate, upper left corner vertical coordinate, lower right corner horizontal coordinate, and lower right corner vertical coordinate, respectively. is the predicted annotation box coordinates of target i, in the same format as b i Same; ||·|| represents the L1 norm, GIoU(·) represents the function operation of the generalized intersection-over-union ratio of two boxes; 2) Determine the evaluation index mAP The specific steps are as follows: ① Set the intersection-over-union ratio threshold t, which ranges from 0.5 to 0.

75. Arrange the prediction boxes of category c in descending order of prediction confidence, take the first k prediction boxes, compare the consistency between the predicted category and the true category, the intersection-over-union ratio between the predicted box and the true box, and calculate the statistical accuracy P. c and recall R c ; Determine the accuracy P according to formula (4): c : Determine the recall rate R according to formula (5): c : Among them, TP c is a true positive, FP c It is a false positive, FN c It is a false negative, TP c , FP c 、FN c are all finite positive integers, and P c , R c TP c , FP c 、FN c Different from 0 at the same time; ②Use precision P c and recall R c Draw PR c The curve is determined as the average precision AP of category c according to formula (6). c ; Among them, i represents the index of the positive sample belonging to category c, K represents the number of positive samples, and R i represents the rank of positive sample i after being sorted from small to large by predicted probability, L represents the number of negative samples, K and L are finite positive integers, and the sum of K and L is equal to k; ③ Determine all categories of AP according to formula (7) c The mean mAP of 3) Training Conditional DETR Network The training set is input into the Conditional DETR network for training. The training parameters are: the number of images per batch is 2, the number of training rounds is 50, the learning rate is γ, and the value of γ is 10 -5 or 10 -4 , using AdamW optimizer, iterate until the target loss function converges; (4) Save the model During the training of the Conditional DETR network, the weights are continuously updated using a deep learning framework and the weight files are saved; (5) Testing the Conditional DETR Network Input the test set into the trained Conditional DETR network for testing, load the saved weight file, and obtain the Conditional DETR image object detection results.

2. The Conditional DETR image detection method based on graph attention weighted enhancement according to claim 1, characterized in that: In step (2) of constructing the Conditional DETR network, the self-attention mechanism module 1 is composed of a self-attention calculation layer 1 and a normalization layer 1, a fully connected layer 1, a ReLU activation function layer 1, a Dropout layer 1, a fully connected layer 2, a Dropout layer 2, and a normalization layer 2 connected in series in sequence; The structures of the self-attention mechanism module 2, self-attention mechanism module 3, self-attention mechanism module 4, self-attention mechanism module 5, and self-attention mechanism module 6 are the same as the structure of the self-attention mechanism module 1.

3. The Conditional DETR image detection method based on graph attention weighted enhancement according to claim 1, characterized in that: In step (2) of constructing the Conditional DETR network, the reference point generation module 1 of the decoder is composed of a fully connected layer 5 and a fully connected layer 6 connected in series; the structures of the reference point generation module 2, the reference point generation module 3, the reference point generation module 4, the reference point generation module 5, and the reference point generation module 6 are the same as the structure of the reference point generation module 1.

4. The Conditional DETR image detection method based on graph attention weighted enhancement according to claim 1, characterized in that: In step (2) of constructing the Conditional DETR network, the cross-attention mechanism module 1 in the decoder is composed of a query vector semantic transformation module 1 and a query vector position transformation module 1, a query vector position embedding transformation module 1, a key vector semantic change module 1, a key vector position transformation module 1, a value vector transformation module 1, and a cross-attention calculation layer 1; wherein the output ends of the query vector semantic transformation module 1, the query vector position transformation module 1, the query vector position embedding transformation module 1, the key vector semantic change module 1, the key vector position transformation module 1, and the value vector transformation module 1 are connected to the cross-attention calculation layer 1; The structures of the query vector semantic transformation module 1, the query vector position transformation module 1, the query vector position embedding mapping module 1, the key vector semantics 1, the key vector position transformation module 1, and the value vector transformation module 1 are all single-layer fully connected layers; The structures of the cross-attention mechanism module 2, the cross-attention mechanism module 3, the cross-attention mechanism module 4, the cross-attention mechanism module 5, and the cross-attention mechanism module 6 in the decoder are the same as the structure of the cross-attention mechanism module 1.

5. The Conditional DETR image detection method based on graph attention weighted enhancement according to claim 1, characterized in that: In step (3) of training the Conditional DETR network, 1) in formula (6) of determining the target loss function, i represents the index of the positive sample belonging to category c, K represents the number of positive samples, and R i It indicates the rank of positive sample i after sorting by predicted probability from small to large, L indicates the number of negative samples, and the values ​​of K and L are 2 5 ~2 7 .

Images