Small object detection system and method based on hybrid coding and staged training strategy optimization

Abstract

The invention discloses a small object detection system and method based on hybrid coding and staged training strategy optimization, and relates to the technical field of target detection, and the method comprises the steps: constructing an HEST-DETR model based on hybrid coding and a staged training strategy, and enabling the HEST-DETR model to be composed of a backbone network, a hybrid encoder and a decoder; different scale features of the image are extracted through the backbone network to serve as input of a hybrid encoder; enhancing different scale features through a hybrid encoder, and performing multi-scale feature interactive fusion on the enhanced features; and on the basis of the fused features, obtaining initial object query by adopting an IoU-perceived query optimization strategy, carrying out bounding box matching by utilizing an IoU matching strategy, and carrying out updating and optimization through a staged training strategy to obtain position and type information of the small object. Therefore, by the adoption of the small object detection method based on hybrid coding and staged training strategy optimization, detection of the small object is achieved, and the detection performance and efficiency of the model are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a small object detection system and method based on hybrid coding and phased training strategy optimization. Background Art

[0002] Object detection is an important research direction in the field of computer vision. In recent years, object detection models based on convolutional neural networks (CNNs), such as YOLO, have achieved significant performance improvements. These models transform the object detection problem into a regression problem. While improving the detection accuracy, they face some challenges in practical applications. For example, the non-maximum suppression (NMS) method improves the detection accuracy, but limits the detection speed of the model to a certain extent. In addition, the detection accuracy of small objects and the ability to locate objects close to each other are still weak.

[0003] To solve these problems, the DETR model introduces a self-attention mechanism to capture long-distance dependencies and improve the model's expressiveness. However, DETR's encoder and decoder are based on the Transformer architecture, which consumes huge computing resources during training and inference. Due to the characteristics of the Transformer, the model still has certain limitations in the detection accuracy of small and medium objects.

[0004] In the decoder stage, DETR uses a multi-layer cross-attention mechanism to accurately map the query vector to the image features, thereby improving the accuracy of target positioning and recognition. The self-attention mechanism can focus on all positions of the input feature map, enhancing the model's ability to understand global context information, enabling it to capture long-distance dependencies in the image and adapt to complex scenes and target detection.

[0005] Although the self-attention mechanism has many advantages, it also brings some inherent limitations. First, the quadratic computational complexity of the self-attention mechanism greatly increases the computational burden when processing high-resolution images, resulting in a rapid increase in the demand for computing resources. Second, the training process takes a long time, the model converges slowly, and the storage of intermediate feature maps and gradients further increases memory consumption. Although the DETR model has optimized small object detection, its feature aggregation limitation still exists, and it is difficult to completely overcome this bottleneck.

[0006] In addition, the DETR model uses the Hungarian matching algorithm to achieve one-to-one set prediction. However, in the early stage of training, the same query frequently matches different objects at different time points in the same image, resulting in an unstable optimization process. This instability further affects the structural stability of the pre-trained backbone network and the balance of weight distribution. Therefore, it is necessary to provide a method for stage optimization training to ensure the stability and performance of the object detection model. Summary of the invention

[0007] The purpose of the present invention is to provide a small object detection method based on hybrid coding and phased training strategy optimization, which can realize the detection of small objects, effectively improve the detection accuracy and efficiency, accelerate the model convergence speed, and improve the model detection performance.

[0008] To achieve the above object, the present invention provides a small object detection method based on hybrid coding and phased training strategy optimization, comprising the following steps:

[0009] Firstly, based on the hybrid coding and phased training strategy, a HEST-DETR model is constructed, which consists of a backbone network, a hybrid encoder and a decoder.

[0010] Secondly, the backbone network is used to extract different scale features of the image as the input of the hybrid encoder;

[0011] Then, the fine-grained feature enhancement module, the medium-grained channel feature enhancement module, and the high-level semantic information extraction module are used to enhance the features of different scales, and the multi-scale feature fusion module is used to perform multi-scale feature interactive fusion on the enhanced features.

[0012] Then, based on the fused features, an IoU-aware query optimization strategy is used to obtain the initial object query, an IoU matching strategy is used for bounding box matching, and a staged training strategy is used to update and optimize the query to obtain the location and type information of small objects.

[0013] Preferably, the fine-grained feature enhancement module is used to enhance detail features, and the expression is as follows:

[0014] P1,P2=Proj(C1);

[0015] S1=Concat(X_avgPool(P2),Y_avgPool(P2));

[0016] P1,P2=Split(σ(Proj(S1)));

[0017] Y1=σ(Proj(X1))⊙σ(Proj(X1))⊙P2+P1;

[0018] Where P1 and P2 represent the first and second groups of features after convolution projection, respectively. Proj represents convolution projection. X_avgPool represents one-dimensional horizontal global pooling. Y_avgPool represents one-dimensional vertical global pooling. σ and ⊙ represent nonlinear activation and dot multiplication operations, respectively. Split represents the split feature map. Concat represents channel connection. C1 is the extracted detail feature.

[0019] Preferably, the medium-granularity channel feature enhancement module is used to enhance the middle-level features, and the expression is as follows:

[0020] P1,P2=Proj(C2);

[0021] s1=f(avgPool(P2)) T ;

[0022] Y2=σ(Expand(g(Proj(S1)) T ));

[0023] In the formula, C2 represents the extracted middle-level features, f and g represent compression and decompression respectively, and Expand represents the operation of expanding to the same shape as P1.

[0024] Preferably, the multi-scale feature fusion module adopts a generalized feature fusion module to perform multi-scale feature interaction.

[0025] Preferably, the phased training strategy for updating and optimizing includes first freezing the backbone network, training the hybrid encoder and decoder from an initial state, and then training the backbone network according to the learning rate after the hybrid encoder and decoder are trained, as expressed as follows:

[0026]

[0027] In the formula, L, L box , L cls They are total loss, border loss, and label loss respectively; y represents the predicted value and the true value respectively. y={x,c}, where x and c are the true values ​​of the bounding box and label respectively. are the predicted values ​​corresponding to the bounding box and label respectively; α and γ are the hyperparameters of the focal loss, CIou and GIou represent the corresponding IoU calculations, and λ b , c They are regression loss weight and classification loss weight respectively, weight is the classification loss composite weight, and BCE_with_logist is the binary cross entropy metric.

[0028] A small object detection system based on hybrid coding and phased training strategy optimization, comprising:

[0029] The backbone network is used to extract features of different scales of the target;

[0030] A hybrid encoder, including a fine-grained feature enhancement module, a medium-grained channel feature enhancement module, a high-level semantic information extraction module, and a multi-scale feature fusion module, is used to enhance features of different scales and perform multi-scale feature fusion;

[0031] The decoder, including the feature projection module, the IoU query optimization module, the decoding layer, and the one-to-one matching module, is used to optimize the query vector and learn the input features to generate the location and classification information of small objects.

[0032] Therefore, the present invention adopts the above-mentioned small object detection system and method based on hybrid coding and phased training strategy optimization, which has the following technical effects:

[0033] (1) Through a phased training strategy, the feature enhancement of different scales is solved so that the model has the tendency to detect objects of different sizes. At the same time, through the feature enhancement module and the multi-scale feature fusion module, the accuracy of small target detection can be improved, and the fine-grained feature information can be fully utilized to reduce the pollution of high-level information to low-level information in the feature fusion stage, and provide the decoder with fine query vectors and high-quality feature information, so that the DETR model can maximize the response and recognition of small object targets.

[0034] (2) The generalized feature fusion module obtains information through different paths and at different depths, extracts features to achieve cross-scale fusion, improves the richness and robustness of feature representation, and improves the expressiveness and accuracy of the model.

[0035] (3) The convergence is accelerated by a phased training strategy, which avoids the damage of the Hungarian matching instability to the backbone network in the early stage of training, thereby preventing the imbalance of weight distribution. This not only reduces the number of parameters and training costs, but also accelerates the model convergence speed, simplifies the training process, and significantly improves the detection performance.

[0036] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the overall framework of the HEST-DETR model in an embodiment of a small object detection system and method based on hybrid coding and phased training strategy optimization;

[0038] Figure 2 A schematic diagram of a DFE module architecture in an embodiment of a small object detection system and method based on hybrid coding and phased training strategy optimization;

[0039] Figure 3 It is a schematic diagram of an IFE model architecture in an embodiment of a small object detection system and method based on hybrid coding and phased training strategy optimization;

[0040] Figure 4 The invention relates to a GFFM module in a cross-scale feature fusion module in an embodiment of a small object detection system and method based on hybrid coding and phased training strategy optimization. DETAILED DESCRIPTION

[0041] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0042] In this embodiment, a HEST-DETR model is constructed based on hybrid coding and phased training strategies. The model consists of a backbone network, a hybrid encoder and a decoder.

[0043] The hybrid encoder is constructed by combining convolution and Transformer architecture, including fine-grained feature enhancement module (DFE), medium-grained channel feature enhancement module (IFE), high-level semantic information extraction module (HLSIE) and multi-scale feature fusion module (MSFF), so that it can focus on significant multi-scale areas to optimize the detection of small objects and better balance the relationship between speed and performance. The design of this structure can enhance the feature representation through the comprehensive enhancement of multi-level features. The specific processing process is as follows:

[0044] The last three layers of feature maps {C1, C2, C3} extracted by the backbone network are used as the input of the encoder. The information carried by C1 is enhanced by the DFE module. The HLSIE module extracts the global semantic information in C3. The IFE module retains the original information in C2 to ensure the integrity of the information and the richness of the features. The specific expression is:

[0045] Q = K = V = Flatten (C3);

[0046] Y3 = Reshape(Attn(Q,K,V));

[0047] Y2=IFE(C2);

[0048] Y1=DFE(C1);

[0049] In the formula, Y1 represents the enhanced detail feature, Y2 represents the enhanced middle-level feature, Y3 represents the global semantic feature, Flatten represents the flattening operation, Attn represents multi-head self-attention, Reshape represents restoring the feature to the shape of C3, IFE represents middle-level information enhancement, and DFE represents detail feature enhancement.

[0050] like Figure 2As shown in the figure, the DFE module uses two sets of convolutional mapping (Conv 1×1) and selects a set of results to extract features (X-avgPool, Y-avgPool) by adaptive average pooling in the height and width directions; then performs fusion (Concat), CNN convolution (transition) and nonlinear activation (σ) to generate attention weights in the height and width directions; then, these attention weights are multiplied element by element with the input feature map to enhance feature representation; finally, the addition operation is performed with the original feature map retained initially to keep the gradient flow smooth and better act on the weight information in the backbone network. The expression of this process is:

[0051] P1,P2=Proj(C1);

[0052] S1=Concat(X_avgPool(P2),Y_avgPool(P2));

[0053] P1,P2=Split(σ(Proj(S1)));

[0054] Y1=σ(Proj(X1))⊙σ(Proj(X1))⊙P2+P1;

[0055] Where Proj represents convolution projection, X_avgPool represents one-dimensional horizontal global pooling, Y_avgPool represents one-dimensional vertical global pooling, σ and ⊙ represent nonlinear activation and dot multiplication operations respectively, Split represents split feature map, and Concat represents channel connection.

[0056] like Figure 3 As shown in the figure, the IFE module also uses two sets of convolution kernel mapping and selects a set of results. It generates channel attention weights through global average pooling (avgPool) and one-dimensional convolution (Conv1d), and applies them to the original feature map to accurately adjust the weight of each channel to highlight key features, focus on features that are useful for specific tasks, and suppress irrelevant or redundant information, thereby enhancing the ability to capture and represent detailed features. The above structure is used to retain the original information of the input feature map while ensuring the integrity and diversity of the information. The expression of this process is:

[0057] P1,P2=Proj(C2);

[0058] S1=f(avgPool(P2)) T ;

[0059] Y2=σ(Expand(g(Proj(S1)) T ));

[0060] In the formula, f and g represent compression and decompression respectively, and Expand represents the operation of expanding to the same shape as P1.

[0061] In the MSFF module, in order to retain the original information to the maximum extent, the generalized feature fusion module (GFFM) is used to interact with multi-scale features. The GFFM module can obtain feature information from different paths and different depths internally, such as Figure 4 shown.

[0062] The feature map obtained by the above encoder has richer semantic information, which helps the model distinguish the relationship between entities and facilitates subsequent detection and recognition.

[0063] The decoder includes a feature projection module (FeatureProjection), an IoU query optimization module (IouQuerySelector), a decoding layer (DecoderLayersxN) and a one-to-one matching module (onetoonematching).

[0064] Among them, different from traditional query generation, this embodiment is based on the static learnable query proposed by DINO, adopts an IoU-aware query optimization strategy to optimize the initial object query of the decoder, maintains the learnability of the query vector, and generates high-quality query selection, thereby improving model performance.

[0065] During the training process, in order to make it suitable for the detection of small objects, a more sophisticated IoU matching strategy is adopted to obtain accurate feedback signals, so that the model can learn effective features faster, improve the generalization ability of the model, accurately locate detailed information and reduce the instability of the training process. In addition, the CIoU matching strategy is adopted for the bounding box, which not only considers the overlap of the bounding box, but also considers the center point distance and aspect ratio, ensuring shape invariance and improving the sensitivity of target positioning accuracy. The regression of the bounding box is supervised from multiple aspects such as position, shape, and direction to meet the needs of detecting detailed targets, avoiding the instability caused by the traditional hard label method ignoring the overlap between the predicted box and the real box.

[0066] In the decoding layer, a multi-layer cross-attention mechanism is used to accurately map the query vector to the image features, thereby improving the accuracy of target positioning and recognition; the self-attention mechanism enables it to focus on all positions of the input feature map, enhancing the model's ability to understand global contextual information, enabling it to capture long-distance dependencies in the image and adapt to complex scenes and target detection.

[0067] In the training phase, this embodiment adopts a phased training strategy to solve the problem of instability in matching of the DETR model in the early stage of training, which leads to the difficulty of convergence caused by the inconsistency of optimization objectives, so as to accelerate the training of the model and achieve significant results. This strategy is to further solve the difficulty of convergence under strong constraints by preheating some network parameters, specifically:

[0068]

[0069]

[0070] In the formula, L, L box , L cls They are total loss, border loss, and label loss respectively; y represents the predicted value and the true value respectively. y={x,c}, where x and c are the true values ​​of the bounding box and label respectively. are the predicted values ​​corresponding to the bounding box and label respectively; α and γ are the hyperparameters of the focal loss, GIou and GIou represent the corresponding IoU calculations, and λ b , c They are regression loss weight and classification loss weight respectively, weight is the classification loss composite weight, and BCE_with_logist is the binary cross entropy metric.

[0071] In the early stage of training, the instability of bipartite matching (Hungarian matching) will destroy the weight distribution of the model backbone. Based on this, noise is introduced in this embodiment, and the decoder is improved in the hope of accelerating convergence. Inspired by DEYO, a phased training strategy is proposed based on the idea of ​​protecting parameters. This idea treats the encoder and decoder parts in the early stage of model training as black boxes, so that the model implicitly solves the DETR model convergence problem, that is, the unstable state of the early part of the training is limited to the second half of the network, avoiding the wrong gradient caused by matching from being passed to all structures, and completing more training iterations with a shorter training time. The purpose of this is to solve the convergence problem of the model first and then start to improve the training accuracy.

[0072] The phased training strategy can be divided into multiple phases. In this embodiment, it is divided into two phases as follows:

[0073] In the initial stage, the backbone network of the end-to-end detector is frozen, and only the encoder and decoder are allowed to gradually learn from their initial state. The purpose of this is to protect the original distribution of the pre-trained weights of the backbone network. After the encoder and decoder have been fully learned to adapt to the new task requirements and the Hungarian matching has stabilized, the second stage of training begins. The backbone network is unfrozen and the initial training state is set for it, while the rest of the network is trained at the learning rate at the end of the original stage. As the training progresses, the learning rate is gradually reduced to ensure the stability and continuity of the entire training process.

[0074] Through practical verification, the training accuracy of this method reached 53.3% on COCOval2017 and 36.2% on small target detection, surpassing the advanced end-to-end DETR detector of the same scale in small object detection; and, the parameters in the early stage of training were reduced from 48M to 25M, which was reduced by nearly half, significantly speeding up the model training process.

[0075] Therefore, the present invention adopts the above-mentioned small object detection system and method based on hybrid coding and phased training strategy optimization to achieve detection of small objects and significantly improve the detection performance and efficiency of the model.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A small object detection method based on hybrid coding and phased training strategy optimization, characterized in that: The following steps are involved: Firstly, based on the hybrid coding and phased training strategy, a HEST-DETR model is constructed, which consists of a backbone network, a hybrid encoder and a decoder. Secondly, the backbone network is used to extract different scale features of the image as the input of the hybrid encoder; Then, the fine-grained feature enhancement module, the medium-grained channel feature enhancement module, and the high-level semantic information extraction module are used to enhance the features of different scales, and the multi-scale feature fusion module is used to perform multi-scale feature interactive fusion on the enhanced features. Then, based on the fused features, an IoU-aware query optimization strategy is used to obtain the initial object query, an IoU matching strategy is used for bounding box matching, and a staged training strategy is used to update and optimize the query to obtain the location and type information of small objects.

2. The small object detection method based on hybrid coding and phased training strategy optimization according to claim 1 is characterized in that: The expression of the fine-grained feature enhancement module is as follows: P1,P2=Proj(C1); S1=Concat(X_avgPool(P2),Y_avgPool(P2)); P1,P2=Split(σ(Proj(S1))); Y1=σ(Proj(X1))⊙σ(Proj(X1))⊙P2+P1; Where P1 and P2 represent the first and second groups of features after convolution projection, respectively. Proj represents convolution projection. X_avgPool represents one-dimensional horizontal global pooling. Y_avgPool represents one-dimensional vertical global pooling. σ and ⊙ represent nonlinear activation and dot multiplication operations, respectively. Split represents the split feature map. Concat represents channel connection. C1 is the extracted detail feature.

3. The small object detection method based on hybrid coding and phased training strategy optimization according to claim 1 is characterized in that: The expression of the medium-granularity channel feature enhancement module is as follows: P1,P2=Proj(C2); S1=f(avgPool(P2)) T ; Y2=σ(Expand(g(Proj(S1)) T )); In the formula, C2 represents the extracted middle-level features, f and g represent compression and decompression respectively, and Expand represents the operation of expanding to the same shape as P1.

4. The small object detection method based on hybrid coding and phased training strategy optimization according to claim 1 is characterized in that: The multi-scale feature fusion module adopts the generalized feature fusion module for multi-scale feature interaction.

5. The small object detection method based on hybrid coding and phased training strategy optimization according to claim 1 is characterized in that: The phased training strategy for updating and optimization includes freezing the backbone network first, training the hybrid encoder and decoder from the initial state, and then training the backbone network according to the learning rate after the hybrid encoder and decoder are trained. The expression is as follows: In the formula, L, L box , L cls They are total loss, border loss, and label loss respectively; y represents the predicted value and the true value respectively. y={x,c}, where x and c are the true values ​​of the bounding box and label respectively. are the predicted values ​​corresponding to the bounding box and label respectively; α and γ are the hyperparameters of the focal loss, CIou and GIou represent the corresponding IoU calculations, and λ b , c They are regression loss weight and classification loss weight respectively, weight is the classification loss composite weight, and BCE_with_logist is the binary cross entropy metric.

6. A small object detection system based on hybrid coding and phased training strategy optimization, characterized in that: include: The backbone network is used to extract features of different scales of the target; A hybrid encoder, including a fine-grained feature enhancement module, a medium-grained channel feature enhancement module, a high-level semantic information extraction module, and a multi-scale feature fusion module, is used to enhance features of different scales and perform multi-scale feature fusion; Decoder, It includes a feature projection module, an IoU query optimization module, a decoding layer, and a one-to-one matching module, which are used to optimize the query vector, learn the input features, and generate the location and classification information of small objects.

Images