A small target detection method, system, device and medium based on a local attention feature correlation mechanism

Abstract

A small target detection method, system, device and medium based on a local attention feature correlation mechanism, the method comprising: data preprocessing, model construction, model training, small target detection; the system, device and medium are used to realize a small target detection method based on a local attention feature correlation mechanism; the application extracts spatial features and semantic features through a residual network, performs feature fusion through a feature pyramid mechanism, reconstructs features based on a local attention feature correlation mechanism, trains a model using a Hungarian matching algorithm, obtains an optimal matching model suitable for a data set, can ensure high accuracy with fewer iteration times, and has the characteristics of low network operation complexity, low training cost, easy model migration, and high model detection performance.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method, system, device and medium for small target detection based on a local attention feature association mechanism. Background Technology

[0002] Small target detection primarily involves accurately detecting the location and classifying the target from complex images, forming the foundation for advanced visual tasks such as image segmentation and action recognition. Small target detection has significant implications in both military and civilian fields. It is fundamental for UAV obstacle avoidance, reconnaissance, and combat, serving as the "eyes" for border defense and maritime early warning, thus possessing crucial military significance. Small target detection also has wide applications in the civilian sector. In medical diagnosis, early lesions and white spots often constitute a small proportion of CT images, making them difficult to detect. Therefore, rapid and accurate detection of lesion areas is essential for early treatment, preventing deterioration, and improving health. Traditional Transformer-based target detection methods mainly rely on self-attention and mutual attention mechanisms in the encoder-decoder, calculating Q, K, and V for each feature vector. For a single feature point, the relationship between it and all other feature vectors needs to be calculated, resulting in a large computational load, reducing the algorithm's real-time performance, and limiting the input image size, which is detrimental to practical applications.

[0003] Patent application CN117132767A discloses a small target detection method, apparatus, device, and readable storage medium. It mainly relates to an improved YOLOv8 target detection algorithm. The improved YOLOv8 target detection algorithm includes a grouping enhancement module, which comprises a grouping module, a bottleneck module, and a spatial attention module. The grouping module groups the input features, the bottleneck module performs residual processing on the grouped features, and the spatial attention module performs spatial perception on the residual processed features. The improved YOLOv8 target detection algorithm is used to perform target detection and recognition on the image to be detected.

[0004] Patent application CN117173551A discloses a scene-adaptive unsupervised underwater weak target detection method and system. The method includes: defining a search space for a neural network architecture; constructing a cell library based on an improved differentiable architecture search method; introducing a MobileNetV3 feature extraction network to construct a RetinaNet network; performing target detection processing on underwater weak targets based on the RetinaNet network; compressing the cell library using underwater weak target feature data as an accuracy detection index; optimizing and classifying the compressed cell library using the OPTICS clustering algorithm; and further optimizing the optimized cell library using a Nas search strategy to obtain the optimal cell library; and performing target detection processing on underwater weak targets based on the optimal cell library. This invention improves the detection efficiency and accuracy of underwater weak targets by constructing a cell library using a custom search space and then performing compression search optimization.

[0005] However, existing technologies have the following problems:

[0006] (1) Current target detection algorithms mainly perform feature fusion but lack the construction of feature relationships. The detection architecture based on convolutional neural networks is difficult to associate features between small targets. Moreover, in complex backgrounds, the spatial features of small targets are easily submerged. Feature reconstruction is needed to highlight the spatial information of small targets.

[0007] (2) The Transformer-based detection framework needs to calculate the relationship between all features, which is computationally intensive, weakens the real-time performance of the algorithm, and increases the hardware load.

[0008] (3) The Transformer-based object detection framework lacks multi-scale feature information when constructing the relationship between features, while multi-scale features often play an important role in object detection. Summary of the Invention

[0009] To overcome the shortcomings of the existing technologies, this invention aims to provide a small target detection method, system, device, and medium based on a local attention feature association mechanism. It employs a residual feature extraction network for feature extraction, utilizes a feature pyramid for feature fusion, and reconstructs features through a local attention feature association mechanism. The Hungarian algorithm is used for prediction, classification, regression, and ground truth matching. This invention combines multi-scale information to improve model detection accuracy, highlights the spatial features of small targets through feature reconstruction, and reduces model computation through a fully connected structure, making the model easier to train, facilitating data transfer, and improving detection robustness.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] A small target detection method based on a local attention feature association mechanism includes the following steps:

[0012] Step 1, Data Preprocessing: Acquire images, randomly select different scaling parameters, cropping sizes and normalization parameters, and perform scaling, random cropping and image normalization on the acquired images in sequence to obtain preprocessed image data;

[0013] Step 2, Model Construction: Construct a small target detection network model based on a local attention feature association mechanism;

[0014] Step 3, Model Training: Input the image data preprocessed in Step 1 into the small target detection network model based on the local attention feature association mechanism constructed in Step 2, use the Hungarian matching algorithm as the training strategy, perform iterative training, optimize the learnable parameters of each layer in the network, and obtain the optimal small target detection network model based on the local attention feature association mechanism.

[0015] Step 4, Small Object Detection: Input the image to be detected into the small object detection network model based on the local attention feature association mechanism trained in Step 3 to obtain the small object detection result of the image to be detected.

[0016] The specific process of data preprocessing in step 1 is as follows:

[0017] Step 1.1: Acquire the image to be predicted;

[0018] Step 1.2: Set different scaling parameters while maintaining the original image aspect ratio during scaling;

[0019] Step 1.3: Set different cropping sizes, use absolute scale for random cropping, and set the height and width of the image;

[0020] Step 1.4: Set different normalization parameters. The image normalization includes three normalization channels, and different normalization means and standard deviations are set for each of the three normalization channels.

[0021] Step 1.5: Randomly select different scaling parameters, cropping size, and normalization parameters, and perform scaling, random cropping, and image normalization on the image to be predicted acquired in Step 1.1 in sequence to obtain preprocessed image data.

[0022] The small target detection network model constructed in step 2 based on the local attention feature association mechanism includes a residual feature extraction network, a feature pyramid, and a local attention feature association mechanism, specifically:

[0023] Residual feature extraction network: extracts feature vectors containing spatial information and feature vectors containing semantic information to obtain multi-scale feature maps;

[0024] Feature pyramid: Multi-scale feature fusion is performed on the multi-scale feature maps extracted by the residual feature extraction network to obtain a fused feature vector;

[0025] Local attention feature association mechanism: Multi-scale feature association is performed on the fused feature vector obtained from the feature pyramid. The offset is predicted through a fully connected network. Feature reconstruction is achieved by combining the self-attention mechanism in the Transformer encoder and the mutual attention mechanism between the encoder and decoder to obtain small target detection results.

[0026] The specific process of model training in step 3 is as follows:

[0027] Step 3.1, Feature Extraction: Input the preprocessed image data obtained in Step 1.5 into the residual feature extraction network, and extract feature vectors containing spatial information and feature vectors containing semantic information in sequence to obtain multi-scale feature maps;

[0028] Step 3.2, Feature Fusion: Input the multi-scale feature map obtained in Step 3.1 into the feature pyramid to perform multi-scale feature fusion and obtain the fused feature vector;

[0029] Step 3.3, Feature Reconstruction: Multi-scale feature association is performed on the fused feature vector obtained in Step 3.2 using the local attention feature association mechanism. The offset is predicted through a fully connected network. Feature reconstruction is achieved by combining the self-attention mechanism in the Transformer encoder and the mutual attention mechanism between the encoder and decoder to obtain the small target detection result.

[0030] Step 3.4: Calculate the classification loss using the Focal loss function based on the prediction results obtained in Step 3.3, calculate the size difference between the predicted box and the ground truth box using the L1 loss function, calculate the intersection-union ratio of the predicted box and the ground truth box using the GIOU loss function, calculate the regression loss together, solve for the Hungarian optimal match, and calculate the overall loss.

[0031] Step 3.5: Repeat steps 3.1-3.4 to train and optimize the small target detection network model based on the local attention feature association mechanism to obtain the optimal small target detection network model based on the local attention feature association mechanism.

[0032] A small target detection system based on a local attention feature association mechanism includes:

[0033] Data preprocessing module: Acquires images, randomly selects different scaling parameters, cropping size and normalization parameters, and sequentially performs scaling, random cropping and image normalization on the acquired images to obtain preprocessed image data;

[0034] Model building module: Constructs a small target detection network model based on a local attention feature association mechanism;

[0035] Model training module: The image data preprocessed by the data preprocessing module is input into the small target detection network model based on the local attention feature association mechanism constructed by the model building module. The Hungarian matching algorithm is used as the training strategy to perform iterative training, optimize the learnable parameters of each layer in the network, and obtain the optimal small target detection network model based on the local attention feature association mechanism.

[0036] Small object detection module: Input the image to be detected into the small object detection network model based on the local attention feature association mechanism trained by the model training module, and obtain the small object detection result of the image to be detected.

[0037] A small target detection device based on a local attention feature association mechanism includes:

[0038] Memory: Used to store the computer program that implements the small target detection method based on the local attention feature association mechanism described above;

[0039] Processor: Used to implement the small target detection method based on local attention feature association mechanism when executing the computer program.

[0040] A computer-readable storage medium:

[0041] The computer-readable storage medium stores a computer program that, when executed by a processor, enables a small target detection method based on a local attention feature association mechanism.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] 1. In step 1 of this invention, several different data preprocessing processes are preset. Compared with the prior art, this ensures that the data has a certain complexity during the training process, improves the robustness of the detection model, improves the sampling module, maintains that the image data before and after preprocessing have the same aspect ratio, and facilitates the reconstruction of spatial information of small targets by subsequent modules.

[0044] 2. In step 3 of this invention, the feature vector is reconstructed based on the local attention feature association mechanism. Compared with the traditional Transformer-based object detection algorithm, the feature information can be utilized more effectively and the object detection effect can be improved through learnable hierarchical position encoding and multi-scale feature architecture.

[0045] 3. In step 3 of this invention, the feature vector is reconstructed based on the local attention feature association mechanism. Compared with the prior art, this enhances the association between features, thereby highlighting the spatial information of small targets and having a certain degree of robustness in complex backgrounds, thus enhancing the reliability of targets.

[0046] 4. In step 3 of this invention, the feature vector is reconstructed based on the local attention feature association mechanism. By utilizing the feedforward neural network and the fully connected structure, compared with the prior art, the number of features used in the feature relationship construction is reduced, the network computation complexity is reduced, the training cost is reduced, and the model is easier to transfer.

[0047] 5. In step 3 of this invention, a Transformer-based prediction architecture is used, and in the embodiment, 100 sets of feature sequences are preset for classification and regression prediction through an attention mechanism. Unlike the training architecture based on convolutional neural networks, no anchor box structure is required, which reduces the overall computational load of the model and the load on the hardware.

[0048] In summary, compared with existing technologies, this invention utilizes multi-scale feature information to improve the richness of feature information; it employs a local attention feature association mechanism for feature reconstruction, highlighting the spatial information of small targets and improving detection robustness in complex backgrounds, while reducing network computational complexity, alleviating training costs, and making the model easier to transfer; the training process uses the Hungarian matching algorithm to improve the model's detection performance. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method of the present invention.

[0050] Figure 2 This is a flowchart of the model training process of the present invention.

[0051] Figure 3 This is a diagram of the residual module architecture in the residual feature extraction network of this invention.

[0052] Figure 4 This is a feature pyramid module architecture diagram of the feature fusion module of the present invention.

[0053] Figure 5 This is an architecture diagram of the encoder based on the local attention feature association mechanism of the present invention.

[0054] Figure 6 This is an architecture diagram of the decoder for the local attention feature association mechanism of the present invention. Detailed Implementation

[0055] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0056] See Figure 1A small target detection method based on a local attention feature association mechanism includes the following steps:

[0057] Step 1, Data Preprocessing: Acquire images, randomly select different scaling parameters, cropping sizes, and normalization parameters, and sequentially scale, randomly crop, and normalize the images to obtain preprocessed image data; specifically:

[0058] Three pre-defined data processing flows are used, and a training strategy of randomly selecting flows is adopted during the model training phase to increase the complexity of the data and improve the robustness of the resulting model. Each data processing flow consists of the following three steps: scaling, where the image scaling ratio is selected from several pre-defined parameters while maintaining the original aspect ratio; random cropping, using absolute scale random cropping with dimensions set to 384 pixels high and 600 pixels wide; and image normalization, where the normalized mean values ​​for the three channels are 123.675, 116.28, and 103.53, with standard deviations of 58.395, 57.12, and 57.375. For an input image, a preprocessing flow is randomly selected to perform different combinations of scaling, random cropping, and normalization parameters on the image.

[0059] Step 2, Model Construction: Construct a small target detection network model based on a local attention feature association mechanism. This model includes a residual feature extraction network, a feature pyramid, and a local attention feature association mechanism, specifically:

[0060] Residual feature extraction network: extracts feature vectors containing spatial information and feature vectors containing semantic information to obtain multi-scale feature maps;

[0061] Feature pyramid: Multi-scale feature fusion is performed on the multi-scale feature maps extracted by the residual feature extraction network to obtain a fused feature vector;

[0062] Local attention feature association mechanism: Multi-scale feature association is performed on the fused feature vector obtained from the feature pyramid. The offset is predicted through a fully connected network. Feature reconstruction is achieved by combining the self-attention mechanism in the Transformer encoder and the mutual attention mechanism between the encoder and decoder to obtain small target detection results.

[0063] Step 3, Model Training: See Figure 2 The image data preprocessed in step 1 is input into the small target detection network model based on the local attention feature association mechanism constructed in step 2. The Hungarian matching algorithm is used as the training strategy for iterative training to optimize the learnable parameters of each layer in the network, thus obtaining the optimal small target detection network model based on the local attention feature association mechanism. Specifically:

[0064] Step 3.1, Feature Extraction: The preprocessed image data obtained in Step 1.5 is input into the residual feature extraction network to sequentially extract feature vectors containing spatial information and feature vectors containing semantic information, resulting in feature maps at four different scales; specifically:

[0065] Step 3.1.1: Extracting feature vectors that mainly contain spatial information. Specifically, the residual network selected in this invention consists of 5 stages. Extracting feature vectors containing spatial information requires stages 1-3. Stage 1 is the preprocessing of the input, including a convolutional layer, normalization, activation function, and global max pooling. Stages 2 and 3 are the feature extraction layers, with 9 and 12 convolutional layers respectively. The outputs of these two stages are the feature maps containing spatial information.

[0066] Step 3.1.2: Extract feature vectors that mainly contain semantic information. Specifically, the feature map obtained in step 3.1.1 is fed into the deep network structure. Skip connections and channel concatenation structures are used to extract features through stages 4 and 5. The number of convolutional layers are 18 and 9, respectively. The output of the two parts is the feature map containing semantic information.

[0067] Step 3.2, Feature Fusion: Input the multi-scale feature map obtained in Step 3.1 into the feature pyramid to perform multi-scale feature fusion, obtaining a fused feature vector; specifically:

[0068] Step 3.2.1: Pass the four feature maps of different scales obtained in Step 3.1 through a convolutional layer with a kernel size of 1*1 to obtain a multi-scale feature map with a uniform number of channels of 256.

[0069] Step 3.2.2: Align the feature map scale with the previous level feature map by upsampling, and add the scale-aligned feature maps in the channel dimension. Output the fused feature vector through concatenation function and upsampling and other structures.

[0070] Step 3.2.3: Input the feature vectors of different levels into a convolutional layer with a kernel size of 3*3 and a stride of 1, and output 4 fused feature vectors of different scales.

[0071] The specific method for feature fusion in this step, which involves adjusting the feature vectors using the feature pyramid module, is as follows:

[0072] The feature pyramid structure in this invention consists of four layers. The inputs to the first to fourth layers come from the outputs of stages 2 to 5 of the residual network. Each layer's input first passes through a convolutional layer with a 1*1 kernel and a stride of 1. Except for the first layer, the remaining layers then pass through an upsampling module, which adds the feature map of the lower layer along the channel dimension for preliminary feature fusion. Subsequently, it passes through a convolutional layer with a 3*3 kernel and a stride of 1. The specific formula is as follows:

[0073] P k =C 3*3 (C 1*1 (R k )+U(C 1*1 (R k+1 )))

[0074] In the formula, P k C represents the output of the k-th layer feature pyramid. 3*3 This represents a convolutional layer with a 3x3 kernel and a stride of 1, C 1*1 U represents a convolutional layer with a 1*1 kernel and a stride of 1, and U(·) represents the upsampling process.

[0075] Step 3.3, Feature Reconstruction: The fused feature vector obtained in Step 3.2 is subjected to multi-scale feature association using a local attention feature association mechanism. The offset is predicted through a fully connected network, and feature reconstruction is achieved by combining the self-attention mechanism in the Transformer encoder and the mutual attention mechanism between the encoder and decoder, thus obtaining the small target detection result; specifically:

[0076] Step 3.3.1: Flatten the four feature vectors with different scales and the same number of channels obtained in Step 3.2 through a convolutional layer with a kernel size of 1*1. For the convenience of subsequent calculations, the four flattened feature sequences are concatenated along the channel dimension to obtain a feature sequence containing multi-scale information. Record the length and width of each scale feature and construct a mask sequence and a position encoding sequence of the same length accordingly.

[0077] Step 3.3.2: Add the feature sequence and position encoding sequence obtained in step 3.3.1 in the channel dimension, introduce hierarchical position encoding, and input it into the encoder. In the encoder, use the concatenated feature sequence as the initial value of Q, K, and V in the Transformer mechanism. Reconstruct the feature sequence Q through the local attention feature association mechanism. Repeat this process multiple times in the encoder and update the multi-scale feature association sequence Q through the feedforward neural network.

[0078] Step 3.3.3: Referring to the mutual attention mechanism between the encoder and decoder in Transformer, the feature sequence obtained by the encoder in step 3.3.2 is fed into the decoder as V. At the same time, according to the preset 100 predicted feature vectors, they are concatenated as Q of the decoder. Through the self-attention mechanism based on Transformer, the query sequence Q is updated. Combined with V obtained by the encoder, the final 100 query vectors are obtained in the decoder through the self-attention mechanism.

[0079] Step 3.3.4: The query vector obtained by the decoder in step 3.3.3 is sent to the classification module and the regression module respectively to generate the target category and prediction box, thus completing the target detection task.

[0080] The specific process for constructing the feature sequence via the local attention feature association mechanism in this step is as follows:

[0081] For an input feature map with dimensions w and h, a flattening operation is first performed to obtain a sequence of length w*h. Then, four feature maps with the same number of channels but different scales obtained from the feature pyramid module are concatenated, which can be defined as:

[0082] X = C[x] o [x1, x2, x3]

[0083] In the formula, C(·) represents the concatenation operation, x i Represents multi-scale feature sequences.

[0084] The small targets obtained through the region proposal network are roughly located within a region, which can be used as a mask for four scales:

[0085] M = [m o [m1, m2, m3]

[0086] In the formula, m i The value is 0 or 1, indicating whether it is a pixel included in the suggested area;

[0087] The process of processing the obtained feature map using a mask can be defined as follows:

[0088]

[0089] Drawing inspiration from the Transformer object detection architecture, attention mechanism weights are obtained through a fully connected network. The process can be defined as follows:

[0090]

[0091] In the formula, w1 represents a learnable fully connected network;

[0092] The selected feature vectors are updated by combining the feature vectors within four preset prediction regions with attention weights. The calculation process can be defined as follows:

[0093]

[0094] In the formula: Δi represents the offset between the actual selected pixel and the preset pixel, which is obtained through training a fully connected network.

[0095] Step 3.4 involves calculating the classification loss using the Focal loss function based on the prediction results obtained in Step 3.3, calculating the size difference between the predicted bounding box and the ground truth box using the L1 loss function, and calculating the intersection-union ratio (IU) between the predicted and ground truth boxes using the GIOU loss function. These steps are then combined to calculate the regression loss. Finally, relevant functions are called to solve for the Hungarian optimal match, and the overall loss is calculated. Specifically:

[0096] Step 3.4.1: In this invention, the TinyPerson sea surface small target human detection dataset is used. The targets to be detected in this dataset are divided into two categories. Therefore, the classification output dimension in step 3.3.4 is (100, 2), that is, 100 query vectors predict the two types of targets. The regression output dimension is (100, 4), that is, 100 query vectors predict the target position. The feature reconstruction process is repeated multiple times in the decoder to obtain 6 sets of predicted values.

[0097] Step 3.4.2: Calculate the classification loss of the 6 sets of feature vectors obtained in Step 3.4.1 using the Focal loss function, calculate the size difference between the predicted box and the ground truth box using the L1 loss function, calculate the intersection-union ratio of the predicted box and the ground truth box using the GIOU loss function, calculate the regression loss together, call the relevant functions to solve the Hungarian optimal matching, calculate the overall loss, and achieve the optimal matching;

[0098] Step 3.5: Repeat steps 3.1-3.4 to train and optimize the small target detection network model based on the local attention feature association mechanism to obtain the optimal small target detection network model based on the local attention feature association mechanism.

[0099] Step 4, Small Object Detection: Input the image to be detected into the small object detection network model based on the local attention feature association mechanism trained in Step 3 to obtain the small object detection result of the image to be detected.

[0100] A small target detection system based on a local attention feature association mechanism includes:

[0101] Data preprocessing module: Acquires images, randomly selects different scaling parameters, cropping size and normalization parameters, and performs scale scaling, random cropping and image normalization on the images in sequence to obtain preprocessed image data;

[0102] Model building module: Constructs a small target detection network model based on a local attention feature association mechanism;

[0103] Model training module: The image data preprocessed by the data preprocessing module is input into the small target detection network model based on the local attention feature association mechanism constructed by the model building module. The Hungarian matching algorithm is used as the training strategy to perform iterative training, optimize the learnable parameters of each layer in the network, and obtain the optimal small target detection network model based on the local attention feature association mechanism.

[0104] Small object detection module: Input the image to be detected into the small object detection network model based on the local attention feature association mechanism trained by the model training module, and obtain the small object detection result of the image to be detected.

[0105] A small target detection device based on a local attention feature association mechanism includes:

[0106] Memory: Used to store the computer program that implements the small target detection method based on the local attention feature association mechanism described above;

[0107] Processor: Used to implement the small target detection method based on local attention feature association mechanism when executing the computer program.

[0108] A computer-readable storage medium:

[0109] The computer-readable storage medium stores a computer program that, when executed by a processor, enables a small target detection method based on a local attention feature association mechanism.

[0110] The application effects of the present invention will be described in detail below with reference to the embodiments.

[0111] See Figure 2 This embodiment uses the TinyPerson small object detection dataset, which divides people into two categories: people in the sea and people on land. The absolute scale of people is less than 36 pixels, and their average proportion in the entire image is 0.012. The normalization parameters are as follows: the normalized mean values ​​for the three channels are 123.675, 116.28, and 103.53, and the standard deviations are 58.395, 57.12, and 57.375, respectively. Three sets of data preprocessing procedures are set up, including scale scaling, random cropping, normalization, and image rotation operations. The ResNet50 network model is used as the feature extraction network. The extracted multi-scale feature maps are fused through the feature pyramid module, and the features are enhanced through the channel mapping module. The features are reconstructed using the local attention-based feature association mechanism invented in this patent.

[0112] The dataset was divided into a training set and a test set. The training set contained 717 images and 25,252 annotated objects, while the test set contained 781 images and 60,698 annotated objects. The Hungarian algorithm was used as the training strategy for 50 epochs.

[0113] Step 1, Data Preprocessing: This invention pre-defines three sets of data processing procedures and employs a training strategy of randomly selecting procedures during the model training phase to increase data complexity and improve the robustness of the resulting model. Each set of data processing procedures consists of the following three steps: scaling, where the image scaling ratio is selected from a set of preset parameters while maintaining the original image aspect ratio during scaling; random cropping, using absolute scale random cropping with dimensions set to 384 pixels high and 600 pixels wide; and image normalization, where the normalized mean values ​​for the three channels are 123.675, 116.28, and 103.53, with standard deviations of 58.395, 57.12, and 57.375. For an input image, a preprocessing procedure is randomly selected to perform different combinations of scaling, random cropping, and normalization parameters on the image.

[0114] Step 2, Model Construction: Construct a small target detection network model based on a local attention feature association mechanism. This model includes a residual feature extraction network, a feature pyramid, and a local attention feature association mechanism, specifically:

[0115] Residual feature extraction network: extracts feature vectors containing spatial information and feature vectors containing semantic information to obtain feature maps at different scales;

[0116] Feature pyramid: Multi-scale feature fusion is performed on the multi-scale feature maps extracted by the residual feature extraction network to obtain a fused feature vector;

[0117] Local attention feature association mechanism: Multi-scale feature association is performed on the fused feature vector obtained from the feature pyramid. The offset is predicted through a fully connected network. Feature reconstruction is achieved by combining the self-attention mechanism in the Transformer encoder and the mutual attention mechanism between the encoder and decoder to obtain small target detection results.

[0118] Step 3, Model Training: Input the preprocessed image data from Step 1 into the small object detection network model based on the local attention feature association mechanism constructed in Step 2. Use the Hungarian matching algorithm as the training strategy for iterative training to optimize the learnable parameters of each layer in the network, thereby obtaining the optimal small object detection network model based on the local attention feature association mechanism. Specifically:

[0119] Step 3.1, Feature Extraction: Input the preprocessed image data from Step 1 into the residual feature extraction network to perform feature extraction and obtain feature maps at four different scales;

[0120] See Figure 3 The specific process of feature extraction in step 3.1 is as follows:

[0121] Step 3.1.1: Extract feature vectors containing spatial information of small targets. Specifically, this includes: loading a pre-trained residual network model, extracting image features through the ResNet50 residual network, and the outputs of stages 2 and 3 being shallow network outputs, which mainly contain spatial information.

[0122] The specific process of extracting feature maps containing spatial information using residual networks is as follows: ResNet50 consists of a series of stacked residual blocks, which can be divided into 5 stages. After each feature extraction stage, the feature map size is halved, while the number of channels is doubled. During feature extraction, image data is sequentially passed through a convolutional layer with a kernel size of 7*7 and a stride of 2, and a max pooling layer with a kernel size of 3*3 and a stride of 2. Then, it is input into stage 2 of the residual network. Stage 2 mainly contains three convolutional layers with kernel sizes of 1*1, 3*3, and 1*1. Stage 3 mainly contains four convolutional layers with kernel sizes of 1*1, 3*3, and 1*1. Figure 3 The diagram shows the basic structure of the residual module. The residual network structure includes an identity mapping layer, which is responsible for fitting the residuals relative to the original network and correcting the overall bias of the model. A residual block consists of a direct mapping part and a residual part, and can be represented as follows:

[0123] R(x)=F(x)+x

[0124] In the formula, F(x) represents the output of the residual network module, x represents the sample dataset, and R(x) represents the input of the final residual network.

[0125] Step 3.1.2: Extract feature vectors containing semantic information of small targets. Specifically, the features obtained in step 3.1.1 are fed into the deep network. The outputs of stage 4 and stage 5 are used as the outputs of the deep network, which mainly contain semantic information, resulting in feature vectors of four scales.

[0126] The specific process of extracting feature maps containing semantic information using residual networks is as follows: the output of stage 3 is sent to stage 4, which mainly contains six convolutional layers with kernels of 1*1, 3*3, and 1*1. Stage 5 mainly contains three convolutional layers with kernels of 1*1, 3*3, and 1*1, and finally four feature maps are obtained.

[0127] Step 3.2, Feature Fusion: The multi-scale feature maps extracted in Step 2 are processed using a feature pyramid to perform multi-scale feature fusion;

[0128] See Figure 4 The specific process of feature fusion in step 3.2 is as follows:

[0129] Step 3.2.1: Input the four feature maps of different scales from stage 2 to stage 5 obtained in step 3.1 into the four levels S2-S5 of the feature pyramid. At this time, the number of channels of the four feature maps are 256, 512, 1024 and 2048 respectively, and the scale is gradually halved.

[0130] Step 3.2.2: Feed S2-S5 into a convolutional layer with a kernel size of 1*1, a stride of 1, and 256 output channels to obtain four feature maps with gradually halved scales and the same number of channels. Then, upsample the high-level feature maps to align their scales with the lower-level feature maps and add them in the channel dimension. Except for the fourth layer, which is not added, the rest complete the preliminary feature fusion to obtain four fused feature maps M2-M5 with gradually halved scales and the same number of channels.

[0131] Step 3.2.3: Feed the summed feature maps M2 to M5 into a convolutional layer with a kernel size of 3*3, a stride of 1, and 256 output channels to obtain the final feature pyramid fusion feature map. Then, perform subsequent feature association based on these four feature maps at different scales. This process can be specifically expressed by the following formula:

[0132] P k =C 3*3 (C 1*1 (R k )+U(C 1*1 (R k+1 )))

[0133] In the formula, P k C represents the output of the k-th layer feature pyramid. 3*3 This represents a convolutional layer with a 3x3 kernel and a stride of 1, C 1*1 U represents a convolutional layer with a 1*1 kernel and a stride of 1, and U(·) represents the upsampling process.

[0134] Step 3.3, Feature Reconstruction: The feature vector obtained in Step 3.2 is subjected to multi-scale feature association using a local attention feature association mechanism. The offset is predicted through a fully connected network, and feature reconstruction is achieved by combining the self-attention mechanism in the Transformer encoder and the mutual attention mechanism between the encoder and decoder.

[0135] See Figure 5 The specific process of feature reconstruction in step 3.3 is as follows:

[0136] Step 3.3.1: Feature reconstruction is performed using the local attention feature association mechanism of this invention. This module consists of an encoder and a decoder, which include a self-attention mechanism and a Transformer-based mutual attention mechanism. First, a feature sequence is constructed. The four-scale feature maps containing spatial and semantic information obtained in Step 3.2 are flattened using a convolutional layer with a kernel size of 1*1 and a stride of 1. The feature fusion module unifies the number of channels at different scales, allowing the flattened feature sequences to be concatenated along the channel dimension, as expressed by the formula:

[0137] X = C 1*1 (x o )·C 1*1 (x1)·C 1*1 (x2)·C 1*1 (x3)

[0138] In the formula, C 1*1 (·) indicates a convolutional layer with a 1*1 kernel, and · indicates splicing along the channel dimension.

[0139] Step 3.3.2: Based on the length and width of each segment of the constructed feature sequence, construct 4-scale sine and cosine position codes of equal length. Simultaneously, to distinguish the equally proportioned positions at different levels, a hierarchical position code is introduced. To focus on the importance of different levels to the reconstructed features, learnable parameters are added to the hierarchical position code. The resulting hierarchical position code and sine / cosine position code channel numbers are unified and added to the feature sequence, which can be expressed by the formula:

[0140]

[0141] In the formula, P i Indicates the positional encoding of sine and cosine. The symbol '·' indicates hierarchical encoding, and '·' indicates splicing along the channel dimension.

[0142] Step 3.3.3: Complete the multi-scale feature reconstruction of the query vector in the encoder part, refer to... Figure 5The query vector obtained in step 3.3.2 is assigned to K and V. The query vector dimension is now (1, 256, N), where 1 represents the batch size, 256 represents the feature dimension, and N is the length of the concatenated feature sequence. Subsequently, a fully connected network is used to predict the offset, attention weights, and V. The offset dimension is (1, 2, 4, 4, 8, N), representing 1 batch, 2 representing the x and y coordinates of the offset, 4 representing 4 scales and 4 preset offset points, and 8 representing the number of prediction heads. The attention weight dimension is (1, 4, 4, 8, N), representing 1 batch, 4 scales, 4 coordinates, and 8 detection heads. The weights and offsets at different levels are calculated using a fully connected network, and then fed back to the query vector via deformable convolutions and a feedforward neural network to find the optimal feature association relationship for each feature point at different levels.

[0143] Step 3.3.4: The final detection vector is output in the decoder section. The optimal query vector obtained in Step 3.3.3 is assigned to V through a mutual attention mechanism. At this point, a query sequence of 100 objects is initialized. The query sequence is updated through the self-attention mechanism in the decoder, with dimensions (100, 256), where 100 is the preset query sequence and 256 is the feature dimension. Through the mutual attention mechanism, combined with V obtained in Step 3.3.3, and referencing... Figure 6 A fully connected network is used to predict offsets, attention weights, and the final detection vector. The offset dimension is (1, 2, 4, 4, 8, 100), meaning 1 batch, 2 represents the x and y coordinates of the offset, 4 scales, 4 preset offset points, and 8 prediction heads. The attention weight dimension is (1, 4, 4, 8, 100), meaning 1 batch, 4 scales, 4 coordinate attention weights, and 8 detection heads. A final detection vector with dimension (256, 100) is obtained through deformable convolution, activation functions, and a feedforward neural network.

[0144] Step 3.3.5: The final detection vector obtained by the decoder in step 3.3.4 is sent to the classification module and the regression module respectively to generate a category prediction feature sequence with dimensions (2, 100) and a location prediction feature sequence with dimensions (4, 100), where 2 represents the two types of targets contained in the TinyPerson dataset and 4 represents the predicted target location.

[0145] Step 3.4: Subsequently, the focal classification loss function is used in conjunction with the obtained vector to calculate the class loss. The L1 loss function is used in conjunction with the obtained vector to calculate the difference between the detected bounding box size and the ground truth bounding box. The GIOU loss function is used in conjunction with the obtained vector to calculate the intersection-union ratio (IUR) between the predicted and actual bounding box positions. The optimal match between the predicted and ground truth positions is obtained by combining the classification and regression losses from 100 different positions using the Hungarian matching algorithm, thus completing the object detection task.

[0146] Step 3.5: Train the small object detection network model that combines residual networks, feature pyramids, and local attention feature association mechanisms. Use the Hungarian matching algorithm as the training strategy and the Transformer prediction mechanism. No anchor boxes are needed during training, reducing training costs. Repeat steps 3.3-3.4 to optimize the learnable parameters of each layer in the network, obtaining the optimal small object detection network model based on the local attention feature association mechanism.

[0147] Step 4, Small Object Detection: Input the image to be detected into the small object detection network model based on the local attention feature association mechanism trained in Step 3 to obtain the small object detection result of the image to be detected.

[0148] Using the Windows 10 operating system and PyCharm software, comparative experiments were conducted on different detection networks. The specific experimental conditions are shown in Table 1.

[0149] Table 1 Experimental conditions

[0150] Experimental conditions parameter graphics card 4090(24g) Optimizer Adamw, SGD Learning rate 2e-4, attenuation 1e-5 batchsize 4 loss function Focalloss, L1loss, GIOU Location coding Sine and cosine position coding

[0151] In Table 1, batchsize represents the number of samples selected in one training session. The learning rate gradually decreases with each training epoch. Two optimizers were used in the experiment, and both achieved similar detection results. Table 2 shows the detection performance of the model of this invention under different epochs. It can be seen that the model only needs 50 iterations to achieve good detection performance. As the number of iterations increases, the detection performance does not change much, indicating that the model converges quickly, has few parameters, and has a certain degree of real-time performance.

[0152] Table 2 Epoch Ablation Experiment

[0153]

[0154] Secondly, validation was performed on the Tinyperson dataset with an IOU confidence level of 0.5. Comparative experiments were conducted on nine neural network models with different parameter scales: YOLOX-L, Cascade Mask R-CNN, TridentNet, RetinaNet, Faster R-CNN, Scratch, Mask R-CNN, Swin, and EmpiricalAttention. The F1-Score represents the balance between accuracy and recall. The accuracy, recall, and F1 score of each model are shown in Table 3, which respectively demonstrate the detection accuracy of the two types of targets. It can be seen that the model constructed in this invention significantly improves the detection performance compared to other models.

[0155] Table 3 Experimental Results

[0156]

[0157]

[0158] In summary, this invention extracts spatial and semantic features through a residual network, performs feature fusion using a feature pyramid mechanism, reconstructs features using a local attention feature association mechanism, and trains the model using the Hungarian matching algorithm to obtain the optimal matching model adapted to the dataset. Furthermore, the small object detection method proposed in this invention is easy to train, maintains high accuracy with fewer iterations, and features low network computational complexity, low training cost, easy model transferability, and high detection performance.

Claims

1. A small target detection method based on a local attention feature association mechanism, characterized in that, Includes the following steps: Step 1, Data Preprocessing: Acquire images, randomly select different scaling parameters, cropping sizes and normalization parameters, and perform scaling, random cropping and image normalization on the acquired images in sequence to obtain preprocessed image data; Step 2, Model Construction: Construct a small target detection network model based on a local attention feature association mechanism; Step 3, Model Training: Input the image data preprocessed in Step 1 into the small target detection network model based on the local attention feature association mechanism constructed in Step 2, use the Hungarian matching algorithm as the training strategy, perform iterative training, optimize the learnable parameters of each layer in the network, and obtain the optimal small target detection network model based on the local attention feature association mechanism. Step 4, Small Object Detection: Input the image to be detected into the small object detection network model based on the local attention feature association mechanism trained in Step 3 to obtain the small object detection result of the image to be detected; The small target detection network model constructed in step 2 based on the local attention feature association mechanism includes a residual feature extraction network, a feature pyramid, and a local attention feature association mechanism, specifically: Residual feature extraction network: extracts feature vectors containing spatial information and feature vectors containing semantic information to obtain multi-scale feature maps; Feature pyramid: Multi-scale feature fusion is performed on the multi-scale feature maps extracted by the residual feature extraction network to obtain a fused feature vector; Local attention feature association mechanism: Multi-scale feature association is performed on the fused feature vector obtained from the feature pyramid. The offset is predicted through a fully connected network. Feature reconstruction is achieved by combining the self-attention mechanism in the Transformer encoder and the mutual attention mechanism between the encoder and decoder to obtain small target detection results. The specific process of model training in step 3 is as follows: Step 3.1, Feature Extraction: Input the preprocessed image data obtained in Step 1.5 into the residual feature extraction network, and extract feature vectors containing spatial information and feature vectors containing semantic information in sequence to obtain multi-scale feature maps; Step 3.2, Feature Fusion: Input the multi-scale feature map obtained in Step 3.1 into the feature pyramid to perform multi-scale feature fusion and obtain the fused feature vector; Step 3.3, Feature Reconstruction: Multi-scale feature association is performed on the fused feature vector obtained in Step 3.2 using the local attention feature association mechanism. The offset is predicted through a fully connected network. Feature reconstruction is achieved by combining the self-attention mechanism in the Transformer encoder and the mutual attention mechanism between the encoder and decoder to obtain the small target detection result. Step 3.4: Calculate the classification loss using the Focal loss function based on the prediction results obtained in Step 3.3, calculate the size difference between the predicted box and the ground truth box using the L1 loss function, calculate the intersection-union ratio of the predicted box and the ground truth box using the GIOU loss function, calculate the regression loss together, solve for the Hungarian optimal match, and calculate the overall loss. Step 3.5: Repeat steps 3.1-3.4 to train and optimize the small target detection network model based on the local attention feature association mechanism to obtain the optimal small target detection network model based on the local attention feature association mechanism.

2. The small target detection method based on a local attention feature association mechanism according to claim 1, characterized in that, The specific process of data preprocessing in step 1 is as follows: Step 1.1: Acquire the image to be predicted; Step 1.2: Set different scaling parameters while maintaining the original image aspect ratio during scaling; Step 1.3: Set different cropping sizes, use absolute scale for random cropping, and set the height and width of the image; Step 1.4: Set different normalization parameters. The image normalization includes three normalization channels, and different normalization means and standard deviations are set for each of the three normalization channels. Step 1.5: Randomly select different scaling parameters, cropping size, and normalization parameters, and perform scaling, random cropping, and image normalization on the image to be predicted acquired in Step 1.1 in sequence to obtain preprocessed image data.

3. A small target detection system based on a local attention feature association mechanism, used to implement the method of claim 1, characterized in that, include: Data preprocessing module: Acquires images, randomly selects different scaling parameters, cropping size and normalization parameters, and sequentially performs scaling, random cropping and image normalization on the acquired images to obtain preprocessed image data; Model building module: Constructs a small target detection network model based on a local attention feature association mechanism; Model training module: The image data preprocessed by the data preprocessing module is input into the small target detection network model based on the local attention feature association mechanism constructed by the model building module. The Hungarian matching algorithm is used as the training strategy to perform iterative training, optimize the learnable parameters of each layer in the network, and obtain the optimal small target detection network model based on the local attention feature association mechanism. Small object detection module: Input the image to be detected into the small object detection network model based on the local attention feature association mechanism trained by the model training module, and obtain the small object detection result of the image to be detected.

4. A small target detection device based on a local attention feature association mechanism, characterized in that, include: Memory: for storing a computer program that implements the small target detection method based on a local attention feature association mechanism as described in any one of claims 1-2; Processor: Used to implement the small target detection method based on local attention feature association mechanism as described in any one of claims 1-2 when executing the computer program.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of a small target detection method based on a local attention feature association mechanism as described in any one of claims 1-2.

Images