Target detection method and system based on improved query selection RT-DRET

Abstract

The invention discloses a target detection method and system based on improved query selection RT-DRET, and the method comprises the steps: obtaining a to-be-detected target data set, and carrying out the data labeling processing, and obtaining a labeled to-be-detected target data set; a query selection module and a multi-scale feature fusion module are introduced, and an improved query selection RT-DRET target detection network model is constructed; and performing target detection processing on the labeled to-be-detected target data set based on an improved query selection RT-DRET target detection network model to obtain a target detection result. According to the embodiment of the invention, the balance between the local features of the target information and the global context information and the relation between the relative position of the prediction frame and the category information can be enhanced, and the efficiency and precision of target detection are improved. The method can be widely applied to the technical field of target detection.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a target detection method and system based on improved query selection RT-DRET. Background Art

[0002] RT-DETR (DETRs Beat YOLOs on Real-time Object Detection) is a Transformer-based object detection model that aims to improve the real-time performance of object detection. However, in the query selection network that supports IoU, the current RT-DETR extracts the relative position and category information of the prediction box through two independent MLP layers. Since these two parts of information do not interact, it is difficult to capture the correlation between the two. Although Varifocal Loss attempts to associate the predicted category information with the IoU of the prediction box in the loss calculation, this association is limited to the loss level and cannot enhance the information interaction at the feature level. Due to the lack of correlation at the feature level, the optimization difficulty increases, and relying on the loss function for optimization will lead to slower training and limited performance improvement. In addition, the CCFM (Cross-scale Communication Feature Module) module in RT-DETR is used for multi-scale feature fusion, but there are two problems: first, feature fusion is only performed in the spatial dimension through convolution, ignoring the complementary information between channels; second, the local receptive field of convolution limits the capture of global context information, resulting in poor performance in complex scenarios.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a target detection method and system based on improved query selection RT-DRET, which can enhance the balance between the local features of the target information and the global context information and the relationship between the relative position of the prediction box and the category information, thereby improving the efficiency and accuracy of target detection.

[0005] To achieve the above objectives, an embodiment of the present application proposes a target detection method based on improved query selection RT-DRET, the method comprising:

[0006] Obtaining a target data set to be detected and performing data annotation processing to obtain an annotated target data set to be detected;

[0007] Introducing query selection module and multi-scale feature fusion module to build an improved query selection RT-DRET target detection network model;

[0008] Based on the improved query selection RT-DRET target detection network model, target detection processing is performed on the labeled target data set to obtain a target detection result.

[0009] In some embodiments, the improved query selection RT-DRET target detection network model includes an encoder module, a query selection module, a multi-scale feature fusion module and a decoder module, wherein the encoder module, the query selection module, the multi-scale feature fusion module and the decoder module are connected in sequence.

[0010] In some embodiments, the query selection module includes a first fully connected layer, a multi-layer perceptual machine layer, a feature cross-fusion module, a loss calculation module and a denoising module, wherein the output end of the first fully connected layer is connected to the input end of the multi-layer perceptual machine layer, the output end of the multi-layer perceptual machine layer is connected to the input end of the feature cross-fusion module, the first output end of the feature cross-fusion module is connected to the input end of the loss calculation module, and the second output end of the feature cross-fusion module is connected to the input end of the denoising module.

[0011] In some embodiments, the feature cross-fusion module includes a first cross-attention layer, a second cross-attention layer, a first normalization layer, a second normalization layer, a second fully connected layer, a third fully connected layer, a first transposed normalization layer and a second transposed normalization layer, wherein the output end of the first cross-attention layer is connected to the input end of the second cross-attention layer, the output end of the second cross-attention layer is respectively connected to the input end of the first normalization layer and the input end of the second normalization layer, the output end of the first normalization layer is connected to the input end of the second fully connected layer, the output end of the second fully connected layer is connected to the input end of the first transposed normalization layer, the output end of the second normalization layer is connected to the input end of the third fully connected layer, and the output end of the third fully connected layer is connected to the input end of the second transposed normalization layer.

[0012] In some embodiments, the multi-scale feature fusion module includes a network division module, a spatial dimension splicing module, a channel dimension splicing module, a transformer module, a depth-separable convolution module and a point multiplication and accumulation calculation module, wherein the network division module, the spatial dimension splicing module, the channel dimension splicing module, the transformer module, the depth-separable convolution module and the point multiplication and accumulation calculation module are connected in sequence.

[0013] In some embodiments, the improved query-based RT-DRET target detection network model is used to perform target detection processing on the labeled target data set to obtain a target detection result, including:

[0014] Inputting the labeled target data set to be detected into the improved query selection RT-DRET target detection network model;

[0015] Based on the improved query, an encoder module of the RT-DRET target detection network model is selected to encode the labeled target data set to obtain a feature map of the target to be detected;

[0016] Based on the query selection module of the improved query selection RT-DRET target detection network model, query selection is performed on the target feature map to be detected to obtain filtered target feature information;

[0017] Based on the improved query selection, a multi-scale feature fusion module of the RT-DRET target detection network model is used to perform multi-scale feature fusion on the screened target feature information to obtain fused target features;

[0018] A decoder module of the RT-DRET target detection network model is selected based on the improved query, the fused target features are decoded, and the target detection result is obtained.

[0019] In some embodiments, the query selection module based on the improved query selection RT-DRET target detection network model performs query selection on the target feature map to be detected to obtain the filtered target feature information, including:

[0020] Inputting the target feature map to be detected into the query selection module;

[0021] Based on the first fully connected layer of the query selection module, the target feature map to be detected is classified to obtain a classified target feature map;

[0022] Based on the multi-layer perceptron layer of the query selection module, channel splitting processing is performed on the classified target feature map to obtain the relative position of the anchor frame of the target feature map and the category information of the target feature map;

[0023] Based on the feature cross fusion module of the query selection module, a feature cross fusion process is performed on the relative position of the anchor frame of the target feature map and the category information of the target feature map to obtain the relative position of the anchor frame after cross fusion and the category information after cross fusion;

[0024] The anchor frame reference position of the target feature map is obtained and combined with the relative position of the anchor frame after the intersection and filtered to obtain the filtered anchor frame absolute position;

[0025] Screening the cross-linked category information to obtain screened category information and combining it with the classified target feature map to obtain screened category features;

[0026] Based on the loss calculation module of the query selection module, performing loss calculation on the absolute position of the filtered anchor box and the filtered category information to obtain a loss calculation result;

[0027] Based on the denoising module of the query selection module, the filtered category features and the loss calculation results are denoised to obtain the filtered target feature information.

[0028] In some embodiments, the feature cross fusion module based on the query selection module performs feature cross fusion processing on the relative position of the anchor frame of the target feature map and the category information of the target feature map to obtain the relative position of the anchor frame after cross fusion and the category information after cross fusion, including:

[0029] Inputting the relative position of the anchor frame of the target feature map and the category information of the target feature map into the feature cross fusion module;

[0030] Based on the first cross attention layer and the second cross attention layer of the feature cross fusion module, the relative position of the anchor frame of the target feature map and the category information of the target feature map are sequentially transposed, mapped and cross attention mechanism operated to obtain the relative position of the anchor frame after preliminary cross and the category information after preliminary cross;

[0031] Based on the first normalization layer and the second normalization layer of the feature cross fusion module, respectively normalizing the relative position of the anchor frame after the preliminary cross and the category information after the preliminary cross to obtain the normalized relative position of the anchor frame and the normalized category information;

[0032] Based on the second fully connected layer and the third fully connected layer of the feature cross fusion module, the normalized anchor box relative positions and the normalized category information are classified to obtain classified anchor box relative positions and classified category information;

[0033] Based on the first transposition normalization layer and the second transposition normalization layer of the feature cross fusion module, the relative position of the classified anchor frame and the classified category information are transposed and normalized to obtain the relative position of the crossed anchor frame and the crossed category information.

[0034] In some embodiments, the multi-scale feature fusion module of the RT-DRET target detection network model based on the improved query selection performs multi-scale feature fusion on the screened target feature information to obtain fused target features, including:

[0035] Inputting the filtered target feature information into the multi-scale feature fusion module;

[0036] Based on the network partitioning module of the multi-scale feature fusion module, the filtered target feature information is partitioned to obtain the partitioned target feature information;

[0037] Based on the spatial dimension splicing module of the multi-scale feature fusion module, the divided target feature information is spliced ​​to obtain target feature information with spatial dimension splicing;

[0038] Based on the channel dimension splicing module of the multi-scale feature fusion module, the target feature information with spatial dimension splicing is spliced ​​to obtain the target feature information with channel dimension splicing;

[0039] Based on the transformer module of the multi-scale feature fusion module, feature transformation is performed on the target feature information with channel dimension splicing to obtain transformed target feature information;

[0040] Based on the depth separable convolution module of the multi-scale feature fusion module, the transformed target feature information is subjected to depth separable convolution processing to obtain the qkv value of the target feature information;

[0041] Based on the point multiplication and accumulation calculation module of the multi-scale feature fusion module, the qkv value of the target feature information is subjected to point multiplication and accumulation calculation to obtain the fused target feature.

[0042] To achieve the above object, another aspect of the embodiment of the present application proposes a target detection system based on improved query selection RT-DRET, the system comprising:

[0043] The first module is used to obtain the target data set to be detected and perform data annotation processing to obtain the annotated target data set to be detected;

[0044] The second module is used to introduce the query selection module and the multi-scale feature fusion module to build an improved query selection RT-DRET target detection network model;

[0045] The third module is used to perform target detection processing on the labeled target data set to be detected based on the improved query selection RT-DRET target detection network model to obtain the target detection result.

[0046] The embodiments of the present application include at least the following beneficial effects: The present application provides a target detection method and system based on improved query selection RT-DRET. The scheme acquires a target data set to be detected and performs data annotation processing, and then introduces a query selection module and a multi-scale feature fusion module to construct an improved query selection RT-DRET target detection network model. The query selection module realizes a query selection method that is more sensitive to the network prediction anchor box, so that the network prediction anchor box information and category information are fully interacted through the feature cross-fusion module, so that the predicted category information and the predicted anchor box information are associated, so that the anchor box corresponding to the query with a high category information score has a high IoU with the real annotation box. For the introduction of the multi-scale feature fusion module, feature information of more dimensions can be captured, the richness and accuracy of feature representation can be improved, and there is a better balance in capturing local features and global context information. Finally, based on the improved query selection RT-DRET target detection network model, target detection processing is performed on the labeled target data set to be detected, which can enhance the balance between the local features of the target information and the global context information and the relationship between the relative position of the prediction box and the category information, thereby improving the efficiency and accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of a target detection method based on improved query selection RT-DRET provided in an embodiment of the present application;

[0048] Figure 2 It is a structural schematic diagram of a target detection system based on improved query selection RT-DRET provided in an embodiment of the present application;

[0049] Figure 3 is a schematic diagram of the structure of the query selection module provided in an embodiment of the present application;

[0050] Figure 4 It is a structural schematic diagram of a feature cross-fusion module provided in an embodiment of the present application;

[0051] Figure 5 It is a feature splitting schematic diagram of a multi-scale feature fusion module provided in an embodiment of the present application;

[0052] Figure 6 It is a structural diagram of the multi-scale feature fusion module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0054] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0055] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0058] 1) Transformer-based object detection model RT-DETR (DETRs Beat YOLOs on Real-time Object Detection): It is a Transformer-based object detection model that aims to improve the real-time performance of object detection. It is an improved version of DETR (End-to-End Object Detection with Transformers) and is the first real-time, end-to-end object detection algorithm. The DETR-related algorithms before it could not achieve real-time detection. Although the YOLO series, which is widely used in the industry, can achieve real-time detection, it still needs to perform non-maximum suppression (NMS) post-processing operations. RT-DETR can efficiently process multi-scale features by designing an efficient hybrid encoder, which can achieve a good trade-off between accuracy and speed.

[0059] There are some deficiencies in the related technologies. For example, in the current RT-DETR, in the iou-aware query selection (support query selection network), two MLP layers are used to obtain the relative position of the prediction box and the category information of the prediction box for the features extracted by the feature extraction network. These two sets of feature information are obtained through two independent structures, so there is no sufficient information interaction between the two sets of feature information, and it is difficult for the network to learn the correlation between the relative position of the prediction box and the category information. VarifocalLoss is introduced in the iou-aware query selection of RT-DETR, the purpose of which is to associate the high score of the category information predicted by the network with the high IoU between the prediction box and the real box. However, after the above analysis, although Varifocal Loss attempts to establish this connection at the loss calculation level, and can associate classification confidence and positioning accuracy to a certain extent, its effect may be subject to the following limitations under the current network structure. First, there is insufficient information interaction. Due to the lack of interaction between position and category information in the feature extraction stage, Varifocal Loss can only be associated in the loss calculation stage, and cannot fundamentally enhance the correlation at the feature level. Second, the difficulty of optimization increases. In the absence of correlation at the feature level, relying solely on the loss function for correlation optimization increases the difficulty of training, resulting in slower model convergence and limited performance improvement.

[0060] In view of this, an object detection method based on improved query selection RT-DRET is provided in an embodiment of the present application. Through a query selection method that is more sensitive to the network prediction anchor box, the network prediction anchor box information and category information can be fully interacted through the feature cross-fusion module, so that the anchor box corresponding to the query with a high category information score and the IoU of the real annotation box are also high, thereby truly giving play to the advantages of the Varifocal Loss loss function. Secondly, a multi-scale feature fusion module is invented. This structure simultaneously fuses features in the channel dimension and the spatial dimension. This dual fusion mechanism can capture feature information of more dimensions and improve the richness and accuracy of feature representation. A combination of convolution and transformer structures is adopted, which has a better balance in capturing local features (convolution) and global context information (transformer).

[0061] Reference Figure 1 , Figure 1 A flowchart of a target detection method based on improved query selection RT-DRET provided by an embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:

[0062] S100, obtaining a target data set to be detected and performing data annotation processing to obtain an annotated target data set to be detected;

[0063] It should be noted that in some embodiments, the goal and purpose of data annotation are first clarified, such as object detection, image classification, etc., and data is further collected. Relevant image or video data is collected according to needs. The data can come from public data sets, crawlers, shooting custom data, etc., and data preprocessing is performed, that is, preprocessing the collected data, such as denoising, resizing, format conversion, etc., to improve data quality, and then select annotation tools. Select appropriate annotation tools or platforms. Commonly used online tools or software include LabelImg, VGG Image Annotator, etc., annotate the data, and use the selected tools to annotate the data, including drawing bounding boxes, classification labels, etc. Ensure the consistency and accuracy of the annotations, and finally store and format them, that is, save the annotation results in a standard format (such as Pascal VOC, COCO, etc.).

[0064] S200, introduce the query selection module and the multi-scale feature fusion module to build an improved query selection RT-DRET target detection network model;

[0065] It should be noted that, in some embodiments, the improved query selection RT-DRET target detection network model includes an encoder module, a query selection module, a multi-scale feature fusion module and a decoder module, wherein the encoder module, the query selection module, the multi-scale feature fusion module and the decoder module are connected in sequence.

[0066] Further, it should be noted that in some specific embodiments, such as Figure 3 As shown, the query selection module includes a first fully connected layer, a multi-layer perception machine layer, a feature cross fusion module, a loss calculation module and a denoising module, wherein the output end of the first fully connected layer is connected to the input end of the multi-layer perception machine layer, the output end of the multi-layer perception machine layer is connected to the input end of the feature cross fusion module, the first output end of the feature cross fusion module is connected to the input end of the loss calculation module, and the second output end of the feature cross fusion module is connected to the input end of the denoising module.

[0067] Among them, Figure 4 As shown, the feature cross-fusion module includes a first cross-attention layer, a second cross-attention layer, a first normalization layer, a second normalization layer, a second fully connected layer, a third fully connected layer, a first transposed normalization layer and a second transposed normalization layer, wherein the output end of the first cross-attention layer is connected to the input end of the second cross-attention layer, the output end of the second cross-attention layer is respectively connected to the input end of the first normalization layer and the input end of the second normalization layer, the output end of the first normalization layer is connected to the input end of the second fully connected layer, the output end of the second fully connected layer is connected to the input end of the first transposed normalization layer, the output end of the second normalization layer is connected to the input end of the third fully connected layer, and the output end of the third fully connected layer is connected to the input end of the second transposed normalization layer.

[0068] Furthermore, it should be noted that in some specific embodiments, Figure 6 As shown, the multi-scale feature fusion module includes a network division module, a spatial dimension splicing module, a channel dimension splicing module, a transformer module, a depth-wise separable convolution module and a point-wise accumulation calculation module, wherein the network division module, the spatial dimension splicing module, the channel dimension splicing module, the transformer module, the depth-wise separable convolution module and the point-wise accumulation calculation module are connected in sequence.

[0069] S300, performing target detection processing on the labeled target data set to be detected based on the improved query selection RT-DRET target detection network model to obtain a target detection result;

[0070] It should be noted that, in some embodiments, step S300 may include steps S310 to S350.

[0071] S310, inputting the labeled target data set to be detected into the improved query selection RT-DRET target detection network model;

[0072] S320, selecting an encoder module of the RT-DRET target detection network model based on the improved query, encoding the labeled target data set to be detected, and obtaining a feature map of the target to be detected;

[0073] S330, based on the query selection module of the improved query selection RT-DRET target detection network model, query and select the feature map of the target to be detected to obtain the filtered target feature information;

[0074] Specifically, step S330 also includes steps S331 to S338.

[0075] S331, input the feature map of the target to be detected into the query selection module; S332, based on the first fully connected layer of the query selection module, classify the feature map of the target to be detected to obtain the classified target feature map; S333, based on the multi-layer perception machine layer of the query selection module, perform channel splitting on the classified target feature map to obtain the relative position of the anchor frame of the target feature map and the category information of the target feature map; S334, based on the feature cross fusion module of the query selection module, perform feature cross fusion on the relative position of the anchor frame of the target feature map and the category information of the target feature map to obtain the relative position of the crossed anchor frame and the category information after the cross ; S335. Obtain the reference position of the anchor frame of the target feature map and combine it with the relative position of the anchor frame after crossing and filter it to obtain the absolute position of the filtered anchor frame; S336. Filter the category information after crossing to obtain the filtered category information and combine it with the classified target feature map to obtain the filtered category features; S337. Based on the loss calculation module of the query selection module, perform loss calculation on the absolute position of the filtered anchor frame and the filtered category information to obtain the loss calculation result; S338. Based on the denoising module of the query selection module, denoise the filtered category features and the loss calculation results to obtain the filtered target feature information.

[0076] Specifically, it should be noted that the selection means selecting the query with good performance from the features F extracted by the network encoder Encoder as the input of the network decoder Decoder, F∈R N×D , where N represents the number of queries and D represents the feature dimension of the query. For the target detection task, good performance is reflected in two aspects: one is the accuracy of the positioning box and the other is the accuracy of the category prediction. In the embodiment of the present invention, the output feature map F obtained from the encoder is obtained through a layer of full connection to obtain F1, and then the relative position of the anchor box R∈R is obtained after the MLP layer with an output dimension of 4+c and the channel split operation.N×4 and category information C∈R N×c , c is the number of categories that the network can predict, and 4 represents the horizontal and vertical coordinates and width and height of the upper left corner of the prediction box. Then, the relative position R of the anchor box and the category information C are fully exchanged through the feature cross-fusion module to obtain the new relative position R′ and category information C′ of the anchor box to alleviate the above-mentioned shortcomings of iou-aware query selection in RT-DETR. The relative position of the anchor box generated by the feature cross-fusion module is added to the anchor box reference coordinates tiled on F to obtain the absolute position of the anchor box predicted by the network. The query selection is filtered according to the score of the category information. For each category information, the category with the highest category score is first found as its predicted category. Then, the 300 predicted categories with the highest confidence are selected as the 300 best queries for all category information, and the top 300 indexes are recorded. The absolute positions of the N anchor boxes determine the 300 filtered anchor boxes according to the top 300 indexes. The feature maps F1 output by the N first fully connected layers determine the filtered category features corresponding to the 300 anchor boxes according to the top 300 indexes. Finally, the filtered anchor boxes and filtered category features are used as the input of the network decoder denoising module. The Varifocal Loss loss function is used to calculate the loss using the filtered anchor boxes and filtered category information. Because the network predicted anchor box information and category information have been fully interacted with each other through the feature cross-fusion module, the Varifocal Loss loss function can be used to associate the predicted category information with the predicted anchor box information, so that the query with a high category information score will have a high IoU with the real annotation box according to the top 300 index, thereby truly giving play to the advantages of the Varifocal Loss loss function.

[0077] Furthermore, it should be noted that step S334 also includes steps S3341 to S3345.

[0078] S3341. Input the relative position of the anchor frame of the target feature map and the category information of the target feature map into the feature cross fusion module; S3342. Based on the first cross attention layer and the second cross attention layer of the feature cross fusion module, perform transposition, mapping and cross attention mechanism operations on the relative position of the anchor frame of the target feature map and the category information of the target feature map in turn to obtain the relative position of the anchor frame after preliminary cross and the category information after preliminary cross; S3343. Based on the first normalization layer and the second normalization layer of the feature cross fusion module, classify the relative position of the anchor frame after preliminary cross and the category information after preliminary cross S3344, based on the second fully connected layer and the third fully connected layer of the feature cross fusion module, the normalized relative position of the anchor frame and the normalized category information are classified respectively to obtain the classified relative position of the anchor frame and the classified category information; S3345, based on the first transposed normalization layer and the second transposed normalization layer of the feature cross fusion module, the classified relative position of the anchor frame and the classified category information are transposed and normalized to obtain the crossed relative position of the anchor frame and the crossed category information.

[0079] Specifically, the feature cross-fusion module first transposes the relative position R of the anchor box and the category information C to obtain R T and C T , followed by R T Map out the query Q, C T The mapping key K and value V are operated by cross attention mechanism to obtain R1, whose expression is as follows:

[0080]

[0081] Where Q1 = R T W q1 , K2=C T W k2 , V2=C T W v2 , W q1 ∈R N×N , W k2 ∈R N×N , W v2 ∈R N×N , W q1 , W k2 and W v2 represents the learnable weight matrix, d k Represents the dimension of the key vector, used to scale the dot product to maintain numerical stability.

[0082] Then R T Map key K and value V, C TThe mapping query Q is operated by the cross attention mechanism to obtain G, which is expressed as follows:

[0083]

[0084] Where, Q2 = C T W q2 , K1=R T W k1 , V1=R T W v1 , W q2 ∈R N×N , W k1 ∈R N×N , W v1 ∈R N×N , W q2 , W k1 and W v1 represents the learnable weight matrix, d k Represents the dimension of the key vector, used to scale the dot product to maintain numerical stability.

[0085] R1 is subjected to the element-by-element addition Add, layer normalization Norm, and fully connected layer FFN operations in the second half of the transformer module, and then transposed to obtain the relative position output R′∈R of the anchor frame of the feature cross-fusion module. N×4 C1 is subjected to the operations of element-by-element addition Add, layer normalization Norm, and full connection layer FFN in the second half of the transformer module, and then transposed to obtain the relative position output C of the anchor frame of the feature cross fusion module. ′ ∈R N×c .

[0086] S340, selecting a multi-scale feature fusion module of the RT-DRET target detection network model based on the improved query, performing multi-scale feature fusion on the screened target feature information, and obtaining a fused target feature;

[0087] Wherein, step S340 also includes steps S341 to S347;

[0088] S341, input the filtered target feature information into the multi-scale feature fusion module; S342, based on the network division module of the multi-scale feature fusion module, divide the filtered target feature information to obtain the divided target feature information; S343, based on the spatial dimension splicing module of the multi-scale feature fusion module, splice the divided target feature information to obtain the target feature information with spatial dimension splicing; S344, based on the channel dimension splicing module of the multi-scale feature fusion module, splice the target feature information with spatial dimension splicing to obtain the target feature information with channel dimension splicing. dimensional splicing target feature information; S345, a transformer module based on a multi-scale feature fusion module, performs feature transformation on the target feature information with channel dimension splicing to obtain the transformed target feature information; S346, a depthwise separable convolution module based on a multi-scale feature fusion module, performs depthwise separable convolution processing on the transformed target feature information to obtain the qkv value of the target feature information; S347, a point multiplication and accumulation calculation module based on a multi-scale feature fusion module, performs point multiplication and accumulation calculation on the qkv value of the target feature information to obtain a fused target feature.

[0089] First of all, it should be noted that the bottleneck structure for fusing multi-scale features in RT-DETR is CCFM (Cross-scale Communication Feature Module). The main purpose of CCFM is to efficiently transfer and fuse information between multi-scale features, thereby enhancing the model's ability to detect objects of different scales. However, it has two disadvantages. First, it mainly fuses features at the spatial level through convolution operations, and does not fully utilize the complementary information between different feature channels. Although the convolution operation can extract spatial features, the information interaction in the channel dimension is relatively limited, so it may not be able to fully fuse the feature information of different channels in the multi-scale feature map, affecting the recognition ability of different categories of objects. Secondly, since CCFM only uses convolutional structures, the receptive field of convolution is local and it is difficult to capture global contextual information. In contrast, the Transformer structure has the advantage of long-distance dependency modeling, which can better capture global features and cross-regional correlations. Therefore, CCFM may perform poorly when dealing with some complex scenes that require global information. Although the convolution operation is computationally efficient, it has poor flexibility and cannot adaptively adjust the feature fusion method according to the dynamic changes of the input features. The Transformer structure can flexibly weigh the feature contributions of different positions and channels through the self-attention mechanism, and fuse information more flexibly. Therefore, CCFM is relatively weak in expression ability, which may lead to insufficient feature fusion effect in complex scenarios. In view of the above-mentioned shortcomings of CCFM, an embodiment of the present invention uses a multi-scale feature fusion structure to simultaneously fuse features in the channel dimension and the spatial dimension, and uses convolution and transformer structures at the same time. This structure simultaneously fuses features in the channel dimension and the spatial dimension. This dual fusion mechanism can capture feature information of more dimensions and improve the richness and accuracy of feature representation. A combination of convolution and transformer structures is adopted, which has a better balance in capturing local features (convolution) and global context information (transformer).

[0090] Specifically, the multi-scale features output by the feature extraction network of the embodiment of the present invention are S3, S4 and S5, S3∈R 4h ×4w×c , S4∈R 2h×2w×2c , S5∈R h×w×4c , c represents the number of channels of S3, h and w represent the height and width of S5. Reduce the resolution of S4 and S5 to the same size as S5. Specifically, divide S4 and S5 into h×w grids respectively, extract the channel features at the same position in all grids of S4, and splice them in the spatial dimension to obtain 4 small features with the same resolution as S5, such as Figure 5The operation of S3 is similar. The channel features of the same position in all grids in S3 are extracted and spliced ​​in the spatial dimension to obtain 16 small features with the same resolution as S5. The 4 small features split from S5 and S4 and the 16 small features split from S3 are spliced ​​in the channel dimension to obtain the feature F5∈R h×w×28c .

[0091] like Figure 6 As shown in the figure, F5 first uses a common transformer module to transform the features of the spatial dimension. Then the transformed F5 is processed by three depth-wise separable convolutions (DWConv) with the same number of channels to obtain the query, key, and value of each channel feature. The output expression is as follows:

[0092] Q={Q0,Q1,...,Q 28c}

[0093] Among them, Q i ∈R h×w×1 , K={K0,K1,...,K 28c}, K i ∈R h×w×1 , V={V0,V1,...,V 28c},V i ∈R h ×w×1 .

[0094] The next step is to calculate the attention coefficient H between the i-th channel and the j-th channel i，j ∈R h×w×1 , which is calculated by taking Q i and K j The channel dimension is concatenated and sent to the convolutional subnetwork to obtain the output H i，j , H i，j ∈R h×w×1 After the relationship between all channels is calculated, for the ith channel, the set of attention coefficients is {H i,0 , H i，1 , ..., H i，28c}, then concat and softmax the third dimension of this set to obtain the attention coefficient H of the i-th channel i , H i ∈R h×w×28c Finally, H i With V i After point multiplication and accumulation, we get the output V′ of the i-th channel after self-attention. i All H i With V iAfter point multiplication and accumulation, we get the output V′∈R after multi-scale feature fusion of features S3, S4 and S5 h×w×28c .

[0095] S350, based on the improved query selection, a decoder module of the RT-DRET target detection network model is used to decode the fused target features to obtain the target detection result.

[0096] In summary, the embodiment of the present invention uses a query selection method and a multi-scale feature fusion module to improve the query selection and CCFM parts of RT-DETR respectively, and the rest of the structure and loss function are consistent with RT DETR.

[0097] Finally, the following is explained in conjunction with the specific implementation environment:

[0098] Taking store sign defect detection as an example, first collect pictures of various store signs, including normal states and various defect states (for example, fading, damage, dirt, etc.). Use the labeling tool LabelMe to label the signs, delineate the defect area, and mark the defect type. Then use the mmdetection framework to implement the RT-DRET model for IoU-sensitive query selection. After dividing the above labeled data into training and test sets according to 8:2, use this model to train the above labeled data. Further convert the trained model into onnx format and deploy it to the mobile phone. After the mobile phone camera shoots the sign data, call the integrated model to obtain all the defect locations and types on the current sign. Finally, after the store receives the defect detection notification, it can arrange maintenance personnel for maintenance.

[0099] Therefore, the advantages of the embodiments of the present invention include:

[0100] 1) A query selection method that is more sensitive to the network predicted anchor box, which allows the network predicted anchor box information and category information to fully interact through the feature cross-fusion module, so that the anchor box corresponding to the query with a high category information score has a high IoU with the real annotation box, thereby truly giving play to the advantages of the Varifocal Loss function.

[0101] 2) A multi-scale feature fusion module, which performs feature fusion in both the channel dimension and the spatial dimension. This dual fusion mechanism can capture feature information in more dimensions and improve the richness and accuracy of feature representation. It uses a combination of convolution and transformer structures, which has a better balance in capturing local features (convolution) and global context information (transformer).

[0102] See also Figure 2The embodiment of the present application also provides a target detection system based on improved query selection RT-DRET, which can implement the above-mentioned target detection method based on improved query selection RT-DRET, and the system includes:

[0103] The first module 201 is used to obtain a target data set to be detected and perform data annotation processing to obtain an annotated target data set to be detected;

[0104] The second module 202 is used to introduce a query selection module and a multi-scale feature fusion module to construct an improved query selection RT-DRET target detection network model;

[0105] The third module 203 is used to perform target detection processing on the labeled target data set to be detected based on the improved query selection RT-DRET target detection network model to obtain a target detection result.

[0106] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0107] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A target detection method based on improved query selection RT-DRET, characterized in that: The method comprises the following steps: Obtaining a target data set to be detected and performing data annotation processing to obtain an annotated target data set to be detected; Introducing query selection module and multi-scale feature fusion module to build an improved query selection RT-DRET target detection network model; Based on the improved query selection RT-DRET target detection network model, target detection processing is performed on the labeled target data set to obtain a target detection result.

2. The method according to claim 1, characterized in that The improved query selection RT-DRET target detection network model includes an encoder module, a query selection module, a multi-scale feature fusion module and a decoder module, wherein the encoder module, the query selection module, the multi-scale feature fusion module and the decoder module are connected in sequence.

3. The method according to claim 2, characterized in that The query selection module includes a first fully connected layer, a multi-layer perceptual machine layer, a feature cross-fusion module, a loss calculation module and a denoising module, wherein the output end of the first fully connected layer is connected to the input end of the multi-layer perceptual machine layer, the output end of the multi-layer perceptual machine layer is connected to the input end of the feature cross-fusion module, the first output end of the feature cross-fusion module is connected to the input end of the loss calculation module, and the second output end of the feature cross-fusion module is connected to the input end of the denoising module.

4. The method according to claim 3, characterized in that The feature cross-fusion module includes a first cross-attention layer, a second cross-attention layer, a first normalization layer, a second normalization layer, a second fully connected layer, a third fully connected layer, a first transposed normalization layer and a second transposed normalization layer, wherein the output end of the first cross-attention layer is connected to the input end of the second cross-attention layer, the output end of the second cross-attention layer is respectively connected to the input end of the first normalization layer and the input end of the second normalization layer, the output end of the first normalization layer is connected to the input end of the second fully connected layer, the output end of the second fully connected layer is connected to the input end of the first transposed normalization layer, the output end of the second normalization layer is connected to the input end of the third fully connected layer, and the output end of the third fully connected layer is connected to the input end of the second transposed normalization layer.

5. The method according to claim 2, characterized in that: The multi-scale feature fusion module includes a network division module, a spatial dimension splicing module, a channel dimension splicing module, a transformer module, a depth-separable convolution module and a point multiplication and accumulation calculation module, wherein the network division module, the spatial dimension splicing module, the channel dimension splicing module, the transformer module, the depth-separable convolution module and the point multiplication and accumulation calculation module are connected in sequence.

6. The method according to claim 1, characterized in that The improved query-based RT-DRET target detection network model is used to perform target detection processing on the labeled target data set to obtain a target detection result, including: Inputting the labeled target data set to be detected into the improved query selection RT-DRET target detection network model; Based on the improved query, an encoder module of the RT-DRET target detection network model is selected to encode the labeled target data set to obtain a feature map of the target to be detected; Based on the query selection module of the improved query selection RT-DRET target detection network model, query selection is performed on the target feature map to be detected to obtain filtered target feature information; Based on the improved query selection, a multi-scale feature fusion module of the RT-DRET target detection network model is used to perform multi-scale feature fusion on the screened target feature information to obtain fused target features; A decoder module of the RT-DRET target detection network model is selected based on the improved query, the fused target features are decoded, and the target detection result is obtained.

7. The method according to claim 6, characterized in that The query selection module based on the improved query selection RT-DRET target detection network model performs query selection on the target feature map to be detected to obtain filtered target feature information, including: Inputting the target feature map to be detected into the query selection module; Based on the first fully connected layer of the query selection module, the target feature map to be detected is classified to obtain a classified target feature map; Based on the multi-layer perceptron layer of the query selection module, the classified target feature map is subjected to channel splitting processing to obtain the relative position of the anchor frame of the target feature map and the category information of the target feature map; Based on the feature cross fusion module of the query selection module, a feature cross fusion process is performed on the relative position of the anchor frame of the target feature map and the category information of the target feature map to obtain the relative position of the anchor frame after cross fusion and the category information after cross fusion; The anchor frame reference position of the target feature map is obtained and combined with the relative position of the anchor frame after the intersection and filtered to obtain the filtered anchor frame absolute position; Screening the cross-linked category information to obtain screened category information and combining it with the classified target feature map to obtain screened category features; Based on the loss calculation module of the query selection module, performing loss calculation on the absolute position of the filtered anchor box and the filtered category information to obtain a loss calculation result; Based on the denoising module of the query selection module, the filtered category features and the loss calculation results are denoised to obtain the filtered target feature information.

8. The method according to claim 7, characterized in that The feature cross fusion module based on the query selection module performs feature cross fusion processing on the relative position of the anchor frame of the target feature map and the category information of the target feature map to obtain the relative position of the anchor frame after cross fusion and the category information after cross fusion, including: Inputting the relative position of the anchor frame of the target feature map and the category information of the target feature map into the feature cross fusion module; Based on the first cross attention layer and the second cross attention layer of the feature cross fusion module, the relative position of the anchor frame of the target feature map and the category information of the target feature map are sequentially transposed, mapped and cross attention mechanism operated to obtain the relative position of the anchor frame after preliminary cross and the category information after preliminary cross; Based on the first normalization layer and the second normalization layer of the feature cross fusion module, respectively normalizing the relative position of the anchor frame after the preliminary cross and the category information after the preliminary cross to obtain the normalized relative position of the anchor frame and the normalized category information; Based on the second fully connected layer and the third fully connected layer of the feature cross fusion module, the normalized anchor box relative positions and the normalized category information are classified to obtain classified anchor box relative positions and classified category information; Based on the first transposition normalization layer and the second transposition normalization layer of the feature cross fusion module, the relative position of the classified anchor frame and the classified category information are transposed and normalized to obtain the relative position of the crossed anchor frame and the crossed category information.

9. The method according to claim 6, characterized in that The multi-scale feature fusion module of the RT-DRET target detection network model based on the improved query selection performs multi-scale feature fusion on the screened target feature information to obtain fused target features, including: Inputting the filtered target feature information into the multi-scale feature fusion module; Based on the network partitioning module of the multi-scale feature fusion module, the filtered target feature information is partitioned to obtain the partitioned target feature information; Based on the spatial dimension splicing module of the multi-scale feature fusion module, the divided target feature information is spliced ​​to obtain target feature information with spatial dimension splicing; Based on the channel dimension splicing module of the multi-scale feature fusion module, the target feature information with spatial dimension splicing is spliced ​​to obtain the target feature information with channel dimension splicing; Based on the transformer module of the multi-scale feature fusion module, feature transformation is performed on the target feature information with channel dimension splicing to obtain transformed target feature information; Based on the depth separable convolution module of the multi-scale feature fusion module, performing depth separable convolution processing on the transformed target feature information to obtain the qkv value of the target feature information; Based on the point multiplication and accumulation calculation module of the multi-scale feature fusion module, the qkv value of the target feature information is subjected to point multiplication and accumulation calculation to obtain the fused target feature.

10. A target detection system based on improved query selection RT-DRET, characterized in that: The system comprises: The first module is used to obtain the target data set to be detected and perform data annotation processing to obtain the annotated target data set to be detected; The second module is used to introduce the query selection module and the multi-scale feature fusion module to build an improved query selection RT-DRET target detection network model; The third module is used to perform target detection processing on the labeled target data set to be detected based on the improved query selection RT-DRET target detection network model to obtain the target detection result.

Images