Multi-scale feature learning small target detection method based on improved RTDETR

Abstract

The invention provides a multi-scale feature learning small target detection method based on an improved RTDETR, and the method comprises the steps: carrying out the improvement of a backbone network Backbone of the RTDETR, and carrying out the preliminary feature extraction through a Conv layer, a MaxPool layer and a PRep-Block module; and the PRep-Block module is used for adding the PConv into the Basic Block, and the RepConv is used for replacing the final convolutional layer of the PConv, so that the PRep-Block is obtained. In the CCFM module, in the feature fusion step, a TFE module is adopted to improve a multi-scale feature fusion mode, and a VoVGSCSP module is added after each feature is fused to optimize channel feature representation. And in the final feature fusion step, performing multi-scale feature fusion by using an SSFF module. Compared with the prior art, the model of the invention further improves the expression ability and robustness of the feature map by enhancing the extraction and fusion ability of the multi-scale features, realizes the lightweight of the model, is suitable for the operation of low-end equipment, and improves the recognition precision of the safety equipment in the chemical industry area, especially the detection precision of a small target.

Description

Technical Field

[0001] The invention belongs to the technical field of target recognition of machine vision, and specifically is a small target detection method based on multi-scale feature learning of improved RTDETR. Background Art

[0002] Small target detection is a challenging task in computer vision, especially in the field of target detection and recognition. Small targets occupy fewer pixels in an image and are particularly easy to be overlooked in images of different scales. The change in the depth of the target in the image can cause a significant change in the appearance of the target, and small targets in the distance may become blurred or distorted. In addition, the similarity between the target and the background in a complex background can interfere with the judgment of the detector, significantly increasing the difficulty of detection. In many fields such as security monitoring, autonomous driving, medical imaging, and drone monitoring, the application of target detection technology is becoming increasingly widespread. However, the existing detection scheme based on RTDETR is still difficult to completely solve the problem of small target recognition, and further optimization and improvement are still needed to improve the accuracy and robustness of detection. Summary of the invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a lightweight RTDETR detection method with higher detection accuracy, especially for small target detection accuracy.

[0004] Technical solution: The present invention proposes a small target detection method based on multi-scale feature learning of improved RTDETR, comprising the following steps:

[0005] Step 1: Obtain image data, visualize and annotate it, and build a data set;

[0006] Step 2: Improve the Backbone of the RTDETR feature extraction network, and perform preliminary feature extraction through the Conv layer, MaxPool layer and PRep-Block module; the PRep-Block module is improved based on the BasicBlock of ResNet, combined with the PConv technology of FasterNet, and RepConv is introduced into PConv to obtain a new convolution layer PRepConv, replacing the second 3*3 convolution in BasicBlock. The improved Backbone network outputs the features S2, S3, S4, and S5 of the four stages;

[0007] Step 3: Input the features S2, S3, S4, and S5 of the last four stages extracted by the improved Backbone network into the Encoder module, and process S5 through the GSConv module and the AIFI-LPE module; the GSConv module is used for global information integration and feature enhancement, and the AIFI-LPE module combines the attention mechanism and learnable position encoding, and finally obtains the feature F5 through the Encoder module;

[0008] Step 4: Input the feature maps S2, S3, S4, S5, and F5 of steps 2 and 3 into the CCFM module for PAFPN operation, and output the fused feature F;

[0009] Step 5: Use the IoU-aware Query Selection module of the RTDETR network to predict the feature F to obtain the detection box and perform screening on it;

[0010] Step 6: The feature F filtered in step 5 is input into the Decoder module of RTDETR. Through the multi-layer attention mechanism and prediction head, multiple feature maps are fused and the final detection result is output.

[0011] Furthermore, the specific method of step 1 is:

[0012] Step 1.1: First, obtain the image data of the working area of ​​the chemical enterprise and preprocess the data set by eliminating the background image without target;

[0013] Step 1.2: Use the Labelme annotation tool to annotate the collected image data and divide them into three categories: person, work_clothes, and helmet, to obtain the image dataset C-safe;

[0014] Step 1.3: Divide the dataset into training set, test set and validation set in the ratio of 7:2:1.

[0015] Furthermore, the specific method of the improved Backbone network in step 2 is:

[0016] Step 2.1: After the image is input into the improved Backbone network, three Conv layers and one MaxPool layer are used to extract features and downsample the image in sequence. The spatial size of the image is halved twice, and the number of feature channels increases from the initial 3 channels to 64 channels. The feature map S2 is obtained, and then S2 is input into the first PRep-Block module.

[0017] Step 2.2: The input of the PRep-Block module is x, and the output of the PRep-Block module is f(x). The PRep-Block module uses PConv to process the input features, and uses RepConv to improve PConv. The processing process of the PRep-Block module is shown in the following formula:

[0018] x1=Conv(x)

[0019] x2=PRepConv(x1)

[0020] f(x) = ReLU(x2 + Conv(x))

[0021] Among them, x1 is the output obtained by the convolution layer Conv, and x2 is the output of x1 after PRepConv; x1 is the input of PRepConv, and the input is first processed by the PartialConv filter to process irregular pixels, and then further processed by RepConv to enhance the feature extraction capability; finally, the feature map is input to the multi-layer perceptron MLP for processing, and regularized by DropPath, and finally the output result that integrates the input features is obtained. The processing process of PRepConv is shown in the following formula:

[0022] X partial =PartialConv(x1)

[0023] X′=RepConv(X partial )

[0024] x2 = DropPath(MLP(X′))

[0025] Step 2.3: In the entire improved Backbone network, the feature map passes through four PRep-Block modules successively to obtain four feature maps of different scales, S2, S3, S4, and S5.

[0026] Furthermore, the specific method of the GSConv module in step 3 is:

[0027] The input S5 is adjusted to 256 channels through the GSConv module. The processing process is shown in the following formula:

[0028]

[0029] Among them, C in represents the number of input channels, g represents the number of groupings of channels in the grouped convolution in GSConv, and Y k,i is the i-th output channel of the k-th group of GSConv, X k,j is the jth input channel of the kth group, Wk,ij is the convolution kernel of the kth group, b k,i is the bias of the kth group.

[0030] Furthermore, the specific method of the AIFI-LPE module in step 3 is:

[0031] The output of the GSConv module is used as the input of the AIFI_LPE module, denoted as X. First, the positional encoding P is generated and combined with the flattened feature map X. flatten Add, increase the position information; add the position-encoded feature map X pos Perform Transformer encoding to obtain the encoded feature map X encodedx Finally, the encoded feature map is transposed and restored to its original shape; the processing process of the AIFI-LPE module is shown in the following formula:

[0032] P=LPE(X)

[0033] X pos =X+P

[0034] X flatten = flatten(X pos )

[0035] X encoder = TransformerEncoder(X flatten )

[0036] X output =permute(X encoded T )

[0037] Among them, X represents the input feature map, and the position encoding information is generated by the LPE module. The variables P, D, and L are defined to represent the maximum number of positions, the embedding dimension, and the sequence length, respectively. The position ID is generated from 0 to L-1, which is represented as position ids = [0, 1, 2…, L-1], use the embedding layer to map the position ID to the embedding vector, and define an embedding matrix E∈R P×D , where each row represents the embedding vector of a position ID. Through the lookup operation, the position embedding matrix P is obtained from the embedding matrix E. emb =E[position ids ], where P emb ∈R P×D , at this time X pos Contains the original feature map X and position encoding information P emb ;

[0038] Furthermore, the CCFM module in step 4 introduces the SSFF module and the TFE module to improve the CCFM module. The specific execution process is as follows:

[0039] Step 4.1: The inputs S2, S3, S4, S5, and F5 of the improved CCFM module are denoted as α1, α2, α3, α4, and α5, respectively. The output of the CCFM module is feature F. The processing process of the CCFM module is specifically shown in the following formula:

[0040] F0=TFE(SConv(α5),α2,α3)

[0041] F1=TFE(SConv(F0),α1,α2)

[0042] F2=Fusion(Sconv(F0),DConv(F1))

[0043] F3=Fusion(SConv(α5),DConv(F2))

[0044] F4=Add(F1,SSFF(α2,α3,α4))

[0045] F=Concat(F2,F3,F4)

[0046] Among them, SConv is a GSConv operation with unchanged feature size and changed number of channels; DConv is a GSConv operation with changed feature size and number of channels at the same time; Fusion represents the feature fusion method of the original RT-DETR model, and its actual operation is Concat; Add represents the addition of two tensors element by element, and Concat represents the concatenation operation of feature fusion.

[0047] The TFE module captures the fine spatial information of small objects. It achieves feature fusion and information capture by splicing feature maps of different sizes together in the spatial dimension. The input of the TFE module includes three feature maps of different scales: l, m, and s. The large-scale feature map l is adjusted to the same size as the medium-scale feature map m through adaptive maximum pooling and adaptive average pooling; the small-scale feature map s is upsampled to the same size as m through the nearest neighbor interpolation method; then the processed feature maps l, m, and s are spliced ​​together in the channel dimension to form a new feature map;

[0048] The SSFF module fuses multi-scale feature maps, receives three feature maps of different scales S3, S4, and S5 as input, and adjusts their number of channels to 256 through convolution; upsamples S4 and S5 to the same size as S3 through the nearest neighbor interpolation method, expands the three features in the new dimension, converts them from 3D tensors to 4D tensors, and splices the three expanded tensors together in the depth dimension to form a 4D tensor combine containing multi-scale information; uses 3D convolution to perform convolution operations on the spliced ​​feature maps, and uses a 3D maximum pooling layer to perform pooling operations on the processed feature maps, and finally compresses the 4D tensor in the splicing dimension to return the final 3D feature map.

[0049] Furthermore, the CCFM module is further improved by introducing the GSConv module and the VoVGSCSP module at the end of each TFE module and Fusion. Figure 2 As shown in the figure, since the fused feature map contains information from different levels, VoVGSCSP can enhance the expressive power and information flow of these fused features through dynamic group convolution and Shuffle operations, thereby improving the performance of the network. GSConv (Group Spatial Convolution) is an efficient convolution operation designed to optimize convolution calculation and feature extraction. In the upsampling stage, GSConv is applied to increase the number of channels of the feature map. At this point, the spatial resolution has been restored by the upsampling operation, and GSConv expands the feature representation capability by increasing the number of channels. In the downsampling stage, GSConv is responsible for simultaneously reducing the spatial resolution of the feature map and adjusting the number of channels. GSConv reduces the spatial resolution of the feature map by adopting a convolution with a stride of 2. This operation reduces the spatial size of the feature map while maintaining the effectiveness of the convolution.

[0050] Beneficial effects:

[0051] 1. The present invention pays special attention to improving the accuracy and robustness of detecting personnel and safety equipment in complex scenarios for chemical workers, and can detect in real time. Employee work scenes are complex and diverse, and workers and safety equipment are often small targets and easily interfered by complex background information. By modifying the fixed coding to a learnable and updateable position coding, the model's sensitivity to graph depth information is enhanced; at the same time, a richer feature information fusion method is introduced to improve the accuracy and robustness of detection.

[0052] 2. The present invention introduces PConv into the BasicBlock structure and takes advantage of its advantages in processing occlusion and sparse data to improve the accuracy of target detection in complex backgrounds. At the same time, RepConv is used in PConv to enhance the feature representation capability through convolutions of different paths and different convolution kernel sizes.

[0053] 3. The present invention uses the improved AIFI-LPE and introduces an embedded matrix to encode the position vector in the AIFI generated position coding information, so that the coding layer of RT-DETR abandons the fixed coding method of sine-cosine summation and adopts a more flexible coding strategy. This improvement enables the coding layer to handle nonlinear feature relationships and significantly enhances the model's ability to understand the depth and spatial relationship of image structure.

[0054] 4. The present invention introduces the SSFF module and the TFE module to improve the CCFM module. The TFE module achieves the enhancement of multi-scale feature fusion by cascading large, medium and small-sized feature maps and combining the spatial and channel attention mechanisms. The feature map generated by the TFE module has richer different information and spatial details, so that the model can improve the detection ability of small targets while maintaining the detection performance of medium and large targets. The SSFF module combines the high-level information of the deep feature map with the detailed information of the shallow feature map, thereby establishing a more effective association between the multi-scale feature maps. The combination of the multi-scale processing of the TFE module and the cross-layer feature fusion of the SSFF module enhances the model's adaptability to different scales.

[0055] 5. Based on the SSFF and TFE modules, the present invention further introduces SlimNeck's GSConv and VoVGSCSP structures, aiming to optimize the network architecture and improve the model efficiency. Through lightweight design and feature refinement, SlimNeck effectively reduces the computational complexity and parameter quantity while maintaining the advantages of the SSFF and TFE modules. This not only improves the reasoning speed of the model, but also makes it more suitable for deployment on various hardware platforms. The introduction of SlimNeck achieves a balance between performance and efficiency, so that the present invention has stronger practicality and adaptability while improving the accuracy of target detection. The VoVGSCSP structure enhances the expressiveness of features and improves the computational efficiency of the model by introducing the advantages of the VoVNet and CSP networks. It deeply mines and extracts feature information through the GSBottleneck layer, and uses the CSP structure to divide the feature map into two parts, which are convolved separately and then fused to improve the expressiveness and robustness of the features. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is the overall flow chart of the small target detection method based on multi-scale feature learning of improved RTDETR;

[0057] Figure 2 To improve the network structure diagram of RTDETR;

[0058] Figure 3 It is the structure diagram of PRepConv;

[0059] Figure 4 It is the structural diagram of TFE module;

[0060] Figure 5 This is the SSFF module structure diagram;

[0061] Figure 6 This is the VOVGSCSP module structure diagram;

[0062] Figure 7 This is a graph showing the changes in various indicators during 300 rounds of model training;

[0063] Figure 8 This is a diagram showing the detection effect of the present invention. DETAILED DESCRIPTION

[0064] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0065] Step 1: Obtain images of 11 working areas of chemical companies, visualize and annotate their safety protection equipment, and construct the C-safe dataset.

[0066] Step 1.1: First, obtain the image data of the working area of ​​the chemical enterprise and preprocess the data set by eliminating the background images without targets.

[0067] Step 1.2: Use the Labelme annotation tool to annotate the collected image data, and divide them into three categories: person, work_clothes, and helmet, to obtain the image dataset C-safe.

[0068] Step 1.3: Divide the dataset into training set, test set and validation set in the ratio of 7:2:1.

[0069] Step 2: Improve the Backbone of the RTDETR feature extraction network, and perform preliminary feature extraction through the Conv layer, MaxPool layer and PRep-Block module; the PRep-Block module is improved based on the BasicBlock of ResNet, combined with the PConv technology of FasterNet, and RepConv is introduced in PConv to obtain a new convolutional layer PRepConv, replacing the second 3*3 convolution in BasicBlock. The improved Backbone network outputs the features S2, S3, S4, and S5 of the four stages.

[0070] Step 2.1: After the image is input into the Backbone network, three Conv layers and one MaxPool layer are used to extract features and downsample it in sequence. The spatial size of the image is halved twice, and the number of feature channels is increased from the initial 3 channels to 64 channels. Rich low-level features are extracted to obtain feature map S2, which is then input into the PRep-Block module.

[0071] Step 2.2: The input of the PRep-Block module is x, and the output of the PRep-Block module is f(x), see Figure 1 to Figure 3 , the PRep-Block module uses FasterNet's PConv to process the input features to better handle the situation where some positions are blocked or incomplete. At the same time, the PConv is improved by using RepConv to obtain PRepConv, which fuses multiple convolutional layers into an equivalent convolutional layer through parameter reconstruction technology, which not only improves the accuracy of feature extraction, but also reduces the computational cost and parameter amount in the inference stage, thereby improving the computational efficiency and the ability to handle incomplete input. The processing process of PRep-Block is shown in the following formula:

[0072] x1=Conv(x)

[0073] x2=PRepConv(x1)

[0074] f(x) = ReLU(x2 + Conv(x))

[0075] Among them, -x1 is the output obtained by the convolution layer Conv, and x2 is the output of x1 after PRepConv.

[0076] See also Figure 3, x1 is the input of PRepConv. First, the input is processed through the PartialConv filter to process irregular pixels (such as images with occlusion), and then further processed using RepConv to enhance the feature extraction capability. Finally, the feature map is input to the multi-layer perceptron (MLP) for processing and regularized by DropPath, and finally the output result that integrates the input features is obtained. The processing process is shown in the following formula:

[0077] X partial =PartialConv(x1)

[0078] X′=RepConv(X partial )

[0079] x2 = DropPath(MLP(X′))

[0080] Step 2.3: In the entire backbone network structure, the feature map passes through 4 PRep-Block modules in succession. Each time it passes through a PRep-Block module, the resolution of the feature map decreases, while the number of channels doubles, and four feature maps of different scales, S2, S3, S4, and S5, are obtained in turn. This "resolution reduction and channel number increase" design reflects the feature extraction process from local to global. High-resolution early feature maps (such as S2) retain more spatial details and local information, which is conducive to capturing fine visual features; as the network goes deeper, the increase in the number of channels enables the later feature maps to extract richer semantic features and improve the perception of complex scenes.

[0081] Step 3: Input the features S2, S3, S4, and S5 of the last four stages extracted by the Backbone network into the Encoder module, and process S5 through GSConv and AIFI-LPE to enhance the expressiveness and semantic information of the optimized features. GSConv is used for global information integration and feature enhancement, while AIFI-LPE combines the attention mechanism and learnable position encoding to help improve the correlation between features and the understanding of spatial structure, and finally obtains the feature F5 through the Encoder module.

[0082] Step 3.1: Use GSConv to adjust the number of channels of input S5 to 256. The processing process is shown in the following formula:

[0083]

[0084] Among them, Y k,i is the i-th output channel of the k-th group of GSConv, X k,j is the jth input channel of the kth group, W k,ij is the convolution kernel of the kth group, b k,iis the bias of the kth group.

[0085] Step 3.2: Take the output of step 3.1 as the input of the AIFI_LPE module, denoted as X. First, generate the positional encoding P and compare it with the flattened feature map X. flatten Add, increase the position information; add the position-encoded feature map X pos Perform Transformer encoding to obtain the encoded feature map X encoded ; Finally, the encoded feature map is transposed and restored to its original shape. The processing of the module is shown in the following formula:

[0086] P=LPE(X)

[0087] X pos =X+P

[0088] X flatten = flatten(X pos )

[0089] X encoder = TransformerEncoder(X flatten )

[0090] X output =permute(X encoded T )

[0091] Among them, X represents the input feature map, and the position encoding information is generated by the LPE module. The variables P, D, and L are defined to represent the maximum number of positions, the embedding dimension, and the sequence length, respectively. The position ID is generated from 0 to L-1, which is represented as position ids = [0, 1, 2…, L-1], use the embedding layer to map the position ID to the embedding vector, and define an embedding matrix E∈R P×D , where each row represents the embedding vector of a position ID. Through the lookup operation, the position embedding matrix P is obtained from the embedding matrix E. emb =E[position ids ], where Px mb ∈R P×D , at this time X pos Contains the original feature map X and position encoding information P emb .

[0092] Flatten means to convert X posFlatten into a [B, HxW, C] shape to adapt to the Transformer input format. [B, H×W, C] represents a three-dimensional tensor after flattening the original four-dimensional tensor (batch size, number of channels, height, width) of shape [B, C, H, W], where the spatial dimensions (height and width) of each sample are flattened into one dimension HxW, retaining the number of channels C.

[0093] TransformerEncoder means inputting the feature map into the Transformer encoder to process the input sequence and extract high-level features. TransformerEncoder is composed of multiple encoder layers (TransformerEncoderLayer), each of which includes the following main components: Multi-Head Self-Attention Mechanism: Calculate the dependency between each position in the input sequence and other positions. Through multiple attention heads, the model can capture the correlation in different subspaces. Feedforward Neural Network: Perform nonlinear transformation on the features of each position to enhance the expressive power of the model. Residual Connection and Layer Normalization: Add residual connections to the outputs of the attention mechanism and feedforward neural network, and perform normalization to stabilize the training process and accelerate convergence.

[0094] Permute is used to rearrange the dimensions of a tensor. A tensor of shape [B, H×W, C] is transformed into [B, C, HxW] through permute to match the dimension order required for subsequent operations.

[0095] Step 4: Input the feature maps S2, S3, S4, S5, F5 of steps 2 and 3 into the CCFM module for PAFPN operation, and output the fused feature F.

[0096] The yaml file for the CCFM module to perform multi-scale feature fusion processing is as follows:

[0097] -[-1,1,nn.Upsample,[None,2,'nearest']]#11

[0098] -[5,1,GSConv,[256,1,1,None,1,1,False]]#12input_proj.1

[0099] -[[-1,6,-2],1,Zoom_cat,[]]#13cat backbone P4

[0100] -[-1,3,VoVGSCSP,

[256] ]#14,fpn_blocks.0

[0101] -[-1,1,GSConv,[256,1,1]]#15,Y4,lateral_convs.1

[0102] -[-1,1,nn.Upsample,[None,2,'nearest']]#16

[0103] -[4,1,GSConv,[256,1,1,None,1,1,False]]#17input_proj.0

[0104] -[[-1,5,-2],1,Zoom_cat,[]]#18cat backbone P3

[0105] -[-1,3,VoVGSCSP,

[256] ]#X3(19),fpn_blocks.1

[0106] -[-1,1,GSConv,[256,3,2]]#20,downsample_convs.0

[0107] -[[-1,15],1,Concat,[1]]#21catY4

[0108] -[-1,3,VoVGSCSP,

[256] ]#F4(22),pan_blocks.0

[0109] -[-1,1,GSConv,[256,3,2]]#23,downsample_convs.1

[0110] -[[-1,10],1,Concat,[1]]#24catY5

[0111] -[-1,3,VoVGSCSP,

[256] ]#F5(25),pan_blocks.1

[0112] -[[5,6,8],1,ScalSeq,

[256] ]#26P3,P4,P5

[0113] -[[19,-1],1,Add,[]]#27

[0114] Each line of data in the yaml file contains four parameters, which represent the input of the module or layer, the internal structure or the number of repetitions of the module or layer, the name of the module or layer, and the parameters of the module or layer. In the input of the module or layer, -1 represents the output of the previous layer. The -1 in the first line is the final output F5 of the Backbone network.

[0115] Step 4.1: The inputs S2, S3, S4, S5, and F5 of the CCFM module are respectively denoted as α1, α2, α3, α4, and α5, and the output of the CCFM module is feature F. The processing of the CCFM module is specifically shown in the following formula:

[0116] F0=TFE(SConv(α5),α2,α3)

[0117] F1=TFE(SConv(F0),α1,α2)

[0118] F2=Fusion(Sconv(F0),DConv(F1))

[0119] F3=Fusion(SConv(α5),DConv(F2))

[0120] F4=Add(F1,SSFF(α2,α3,α4))

[0121] F=Concat(F2,F3,F4)

[0122] In the above formula, SConv is a GSConv operation with unchanged feature size and changed number of channels; DConv is a GSConv operation with changed feature size and number of channels at the same time; Fusion represents the feature fusion method of the original RT-DETR model, and its actual operation is Concat; Add represents the addition of two tensors element by element, and Concat represents the concatenation operation of feature fusion, which concatenates multiple tensors along the specified dimension to form a new tensor, thereby fusing feature information of different scales or sources.

[0123] like Figure 3 The main goal of the TFE module is to capture the fine spatial information of small objects. This module achieves feature fusion and information capture by splicing feature maps of different sizes together in the spatial dimension. Its input includes three feature maps of different scales, l, m, and s. The large-size feature map l is adjusted to the same size as the medium-size feature map m through adaptive maximum pooling and adaptive average pooling. The small-size feature map s is upsampled to the same size as m through the nearest neighbor interpolation method. The processed feature maps l, m, and s are then spliced ​​together in the channel dimension to form a new feature map.

[0124] The main goal of the SSFF module is to fuse multi-scale feature maps to enhance the expressiveness and robustness of feature maps. Figure 4 It receives three feature maps S3, S4, and S5 of different scales as input, and adjusts their channel number to 256 through convolution; upsamples S4 and S5 to the same size as S3 through the nearest neighbor interpolation method, expands the three features in the new dimension, converts them from 3D tensors to 4D tensors, and splices the three expanded tensors together in the depth dimension to form a 4D tensor combine containing multi-scale information; uses 3D convolution to perform convolution operations on the spliced ​​feature maps, and uses a 3D maximum pooling layer to perform pooling operations on the processed feature maps, and finally compresses the 4D tensor in the splicing dimension to return the final 3D feature map.

[0125] Step 4.2: At the end of each feature fusion module, that is, at the end of each TFE module and Fusion, the GSConv module and the VoVGSCSP module are introduced to further optimize the feature representation of each channel through the VOVGSCSP module. Figure 2 As shown in the figure, since the fused feature map contains information from different levels, VoVGSCSP can enhance the expressive power and information flow of these fused features through dynamic group convolution and Shuffle operations, thereby improving the performance of the network. GSConv (Group Spatial Convolution) is an efficient convolution operation designed to optimize convolution calculation and feature extraction. In the upsampling stage, GSConv is applied to increase the number of channels of the feature map. At this point, the spatial resolution has been restored by the upsampling operation, and GSConv expands the feature representation capability by increasing the number of channels. In the downsampling stage, GSConv is responsible for simultaneously reducing the spatial resolution of the feature map and adjusting the number of channels. GSConv reduces the spatial resolution of the feature map by adopting a convolution with a stride of 2. This operation reduces the spatial size of the feature map while maintaining the effectiveness of the convolution.

[0126] Figure 2 The yellow and blue blocks have different meanings.

[0127] Yellow squares: indicate VOVGSCSP+upsampling+GSConv adjusts the number of channels;

[0128] Blue squares: Indicates that VOVGSCSP+GSConv performs downsampling and adjusts the number of channels.

[0129] Step 5: Use the IoU-aware Query Selection module of the RTDETR network to predict the feature F to obtain the detection box and perform screening on it.

[0130] Step 6: The feature F filtered in step 5 is input into the Decoder module of RTDETR. Through the multi-layer attention mechanism and prediction head, multiple feature maps are fused and the final detection result is output.

[0131] The above improved RTDETR model, after 300 rounds of model training, shows the change curves of various indicators. Figure 7 The loss function curve during training is smooth and decreases rapidly, indicating that the model does not experience oscillation or instability during the optimization process. The rapid increase in evaluation indicators such as precision, recall, and mean average precision (mAP) indicates that the model can quickly improve performance during the learning process and the training effect is significant. Overall, the model achieved a high performance level in a relatively short period of time and converged quickly, showing the efficiency of the training process. This phenomenon shows that the model has good generalization ability and training efficiency under the current training configuration.

[0132] The experimental environment is configured as Windows system, using NVIDIA A4000 graphics card, and the deep learning framework is PyTorch. The specific configuration is shown in Table 1 below:

[0133] Table 1 Experimental environment configuration

[0134]

[0135] Effect evaluation:

[0136] Calculate the average precision and average recall: The three detection methods are evaluated, and the evaluation results are shown in Table 2 below. In Table 2, YOLOv8 uses only the YOLOv8 network for safety equipment detection, and RTDETR uses the RTEDETR-r18 model for safety equipment detection. After testing on the validation set, the present invention improves the Recall of helmets by 3% and mAP50-95 by 2.6% on the basis of RTDETR; the Recall of person detection is improved by 3.5% and mAP50-95 by 2.6%. The number of model parameters is reduced by 16.38%, and the amount of calculation (GFLOPs) is reduced by 11.95%. Among them, Precision represents the accuracy of the model; mAP50-95 represents the average accuracy of the model at IoU thresholds from 0.50 to 0.95. The algorithm index comparison table is shown in Table 2, which proves the feasibility of the present invention.

[0137] Table 2 Comparison of experimental results of different models

[0138]

[0139] The specific detection effect diagram of the present invention is shown in Figure 8 ,from Figure 8 It can be seen that the detection ability is good under different weather conditions (such as sunny, rainy, foggy, etc.), different lighting conditions (such as strong light, weak light, etc.) and object occlusion (such as partial occlusion, full occlusion), which verifies the stability and reliability of the model in these complex and changing environments.

[0140] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A small target detection method based on multi-scale feature learning of improved RTDETR, characterized in that: The steps include: Step 1: Obtain image data, visualize and annotate it, and build a data set; Step 2: Improve the Backbone of the RTDETR feature extraction network, and perform preliminary feature extraction through the Conv layer, MaxPool layer and PRep-Block module; the PRep-Block module is improved based on the BasicBlock of ResNet, combined with the PConv technology of FasterNet, and RepConv is introduced into PConv to obtain a new convolution layer PRepConv, replacing the second 3*3 convolution in BasicBlock. The improved Backbone network outputs the features S2, S3, S4, and S5 of the four stages; Step 3: Input the features S2, S3, S4, and S5 of the last four stages extracted by the improved Backbone network into the Encoder module, and process S5 through the GSConv module and the AIFI-LPE module; the GSConv module is used for global information integration and feature enhancement, and the AIFI-LPE module combines the attention mechanism and learnable position encoding, and finally obtains the feature F5 through the Encoder module; Step 4: Input the feature maps S2, S3, S4, S5, and F5 of steps 2 and 3 into the CCFM module for PAFPN operation, and output the fused feature F; Step 5: Use the IoU-aware Query Selection module of the RTDETR network to predict the feature F to obtain the detection box and perform screening on it; Step 6: The feature F filtered in step 5 is input into the Decoder module of RTDETR. Through the multi-layer attention mechanism and prediction head, multiple feature maps are fused and the final detection result is output.

2. According to claim 1, a multi-scale feature learning small target detection method based on improved RTDETR is characterized in that: The specific method of step 1 is: Step 1.1: First, obtain the image data of the working area of ​​the chemical enterprise and preprocess the data set by eliminating the background image without target; Step 1.2: Use the Labelme annotation tool to annotate the collected image data and divide them into three categories: person, work_clothes, and helmet, to obtain the image dataset C-safe; Step 1.3: Divide the dataset into training set, test set and validation set in the ratio of 7:2:

1.

3. According to claim 1, a multi-scale feature learning small target detection method based on improved RTDETR is characterized in that: The specific method of the improved Backbone network in step 2 is: Step 2.1: After the image is input into the improved Backbone network, three Conv layers and one MaxPool layer are used to extract features and downsample the image in sequence. The spatial size of the image is halved twice, and the number of feature channels increases from the initial 3 channels to 64 channels. The feature map S2 is obtained, and then S2 is input into the first PRep-Block module. Step 2.2: The input of the PRep-Block module is x, and the output of the PRep-Block module is f(x). The PRep-Block module uses PConv to process the input features, and uses RepConv to improve PConv. The processing process of the PRep-Block module is shown in the following formula: x1=Conv(x) x2=PRepConv(x1) f(x) = ReLU(x2 + Conv(x)) Among them, x1 is the output obtained by the convolution layer Conv, x2 is the output of x1 after PRepConv, and ReLU is the activation function; x1 is the input of PRepConv. First, the input is processed through the PartialConv filter to process irregular pixels, and then RepConv is used for further processing to enhance the feature extraction capability. Finally, the feature map is input into the multi-layer perceptron MLP for processing and regularized by DropPath. Finally, the output result that integrates the input features is obtained. The processing process of PRepConv is shown in the following formula: X partial =PartialConv(x1) X′=RepConv(X partial ) x2 = DropPath(MLP(X′)) Step 2.3: In the entire improved Backbone network, the feature map passes through four PRep-Block modules successively to obtain four feature maps of different scales, S2, S3, S4, and S5.

4. According to claim 1, a multi-scale feature learning small target detection method based on improved RTDETR is characterized in that: The specific method of the GSConv module in step 3 is: The input S5 is adjusted to 256 channels through the GSConv module. The processing process is shown in the following formula: Among them, C in represents the number of input channels, g represents the number of groupings of channels in the grouped convolution in GSConv, and Y k,i is the i-th output channel of the k-th group of GSConv, X k,j is the jth input channel of the kth group, W k,ij is the convolution kernel of the kth group, b k,i is the bias of the kth group.

5. A small target detection method based on multi-scale feature learning of improved RTDETR according to claim 1 or 4, characterized in that: The specific method of the AIFI-LPE module in step 3 is: The output of the GSConv module is used as the input of the AIFI_LPE module, denoted as X. First, the positional encoding P is generated and combined with the flattened feature map X. flatten Add, increase the position information; add the position-encoded feature map X pos Perform Transformer encoding to obtain the encoded feature map X encoded ; Finally, the encoded feature map is transposed and restored to its original shape; the processing process of the AIFI-LPE module is shown in the following formula: P=LPE(X) X pos =X+P emb X flatten =flatten(X pos ) X encoder =TransformerEncoder(X flatten ) F5=exchanges(X encoded T ) Among them, X represents the input feature map, and the position encoding information is generated by the LPE module. The variables P, D, and L are defined to represent the maximum number of positions, the embedding dimension, and the sequence length, respectively. The position ID is generated from 0 to L-1, which is represented as position ids = [0, 1, 2…, L-1], use the embedding layer to map the position ID to the embedding vector, and define an embedding matrix E∈R P×D , where each row represents the embedding vector of a position ID. Through the lookup operation, the position embedding matrix P is obtained from the embedding matrix E. emb =E[position ids ], where P emb ∈R P×D , at this time X pos Contains the original feature map X and position encoding information P emb ; Flatten means to convert X pos Flattened into the shape of [B,H×W,C], [B,H×W,C] represents the three-dimensional tensor obtained by flattening the original four-dimensional tensor of shape [B,C,H,W], where the spatial dimension of each sample is flattened to one dimension H×W, retaining the number of channels C; TransformerEncoder means inputting the feature map into the Transformer encoder to process the input sequence and extract high-level features; permute is used to rearrange the dimensions of the tensor, and the tensor of shape [B,H×W,C] is converted to [B,C,H×W] through permute.

6. According to claim 1, a multi-scale feature learning small target detection method based on improved RTDETR is characterized in that: The CCFM module of step 4 introduces the SSFF module and the TFE module to improve the CCFM module. The specific execution process is as follows: Step 4.1: The inputs S2, S3, S4, S5, and F5 of the improved CCFM module are denoted as α1, α2, α3, α4, and α5, respectively. The output of the CCFM module is feature F. The processing process of the CCFM module is specifically shown in the following formula: F0=TFE(SConv(α5),α2,α3) F1=TFE(SConv(F0),α1,α2) F2=Fusion(Sconv(F0),DConv(F1)) F3=Fusion(SConv(α5),DConv(F2)) F4=Add(F1,SSFF(α2,α3,α4)) F=Concat(F2,F3,F4) Among them, SConv is a GSConv operation with unchanged feature size and changed number of channels; DConv is a GSConv operation with changed feature size and number of channels at the same time; Fusion represents the feature fusion method of the original RT-DETR model, and the actual operation is Concat; Add represents the addition of two tensors element by element, and Concat represents the concatenation operation of feature fusion, which concatenates multiple tensors together along the specified dimension. The TFE module captures the fine spatial information of small objects. It achieves feature fusion and information capture by splicing feature maps of different sizes together in the spatial dimension. The input of the TFE module includes three feature maps of different scales: l, m, and s. The large-scale feature map l is adjusted to the same size as the medium-scale feature map m through adaptive maximum pooling and adaptive average pooling; the small-scale feature map s is upsampled to the same size as m through the nearest neighbor interpolation method; then the processed feature maps l, m, and s are spliced ​​together in the channel dimension to form a new feature map; The SSFF module fuses multi-scale feature maps, receives three feature maps of different scales S3, S4, and S5 as input, and adjusts their number of channels to 256 through convolution; upsamples S4 and S5 to the same size as S3 through the nearest neighbor interpolation method, expands the three features in the new dimension, converts them from 3D tensors to 4D tensors, and splices the three expanded tensors together in the depth dimension to form a 4D tensor combine containing multi-scale information; uses 3D convolution to perform convolution operations on the spliced ​​feature maps, and uses a 3D maximum pooling layer to perform pooling operations on the processed feature maps, and finally compresses the 4D tensor in the splicing dimension to return the final 3D feature map.

7. The method for detecting small targets using multi-scale feature learning based on improved RTDETR according to claim 6, characterized in that: The further improvement of the CCFM module is: introducing a GSConv module and a VoVGSCSP module at the end of each TFE module and Fusion.

Images