A small-size target detection method based on dynamic anchor frame and transformer

Abstract

The application discloses a small-size target detection method based on a dynamic anchor frame and a transformer, belongs to the field of image processing, pattern recognition and computer vision, and solves the problems of low detection precision and low network efficiency of an existing small-size target detection task based on a transformer. The application sequentially constructs a backbone network module, an encoder module, a decoder module and a prediction output module as a target detection network; based on public COCO image data, small-size target sample image data is preprocessed; the preprocessed small-size target sample image data is input into a small-size target detection network with initialized parameters for training, and a trained small-size target detection network is obtained; a small-size target image to be predicted is input into the trained small-size target detection network, and through forward propagation, the prediction frame position and category information of the small-size target are output in an end-to-end manner.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of target detection in computer vision, and particularly relates to a small-size target detection method based on dynamic anchor frame and Transformer. BACKGROUND

[0002] With the continuous development of large-scale parallel computing devices and the continuous evolution of neural network related algorithms, computer vision technology based on deep learning is increasingly widely used. Computer vision technology uses electronic devices to identify and measure targets through specific algorithms, and further processes them. Target detection extracts high-level semantic information that adapts to the environment from the underlying data and processes it into clearer images or texts that contain more obvious content using a computer. Computer vision mainly includes three fields: image classification, target detection, and target segmentation. Target detection aims to detect all target positions and their corresponding categories in a given image using specific algorithms or models. Due to the different sizes, colors, and positions of the targets to be detected, as well as the influence of other interference factors in the picture, target detection has always been a research difficulty and focus in the field of computer vision.

[0003] Currently, the target detection method based on deep learning mainly uses convolutional neural networks to extract features from input images. Since images contain different levels of features, and high-level features depend on low-level features, convolutional neural networks can extract abstract features layer by layer through hierarchical extraction, so convolutional neural networks have a significant advantage in processing images. In addition, the convolution kernel in the convolutional neural network has inductive bias such as locality and translation invariance, which can effectively capture the local information of the input image. At the same time, due to the limitations of convolution operation, convolutional neural networks cannot effectively understand the global information of the input image and lack the ability to model the dependency between features, so they cannot fully utilize the context information.

[0004] Compared with the CNN-based target detection method, the use of the Transformer-based target detection method for target detection tasks has many advantages such as not needing to manually set candidate boxes and post-processing such as non-maximum suppression, but this emerging method still has some deficiencies that need to be addressed. For example, the Transformer-based target detection method needs to calculate cross-attention for the features of the entire image, resulting in a quadratic relationship between the time and space complexity of the model and the number of input data, excessive network computation, slow convergence, etc. At the same time, the large number of parameters of the model results in a large amount of calculation during forward reasoning, and the high computational cost makes it difficult to apply it to devices with limited computing and storage resources.

[0005] In order to obtain a more sufficient feature map representation, different levels of feature maps are aggregated to aggregate context information, improve the performance and stability of the target detection method, especially the target detection method based on the Transformer, on small size targets, and improve the efficiency of the model, which is an important and challenging goal. SUMMARY

[0006] In order to overcome the shortcomings of the prior art, the present application proposes a small size target detection method based on dynamic anchor frame and Transformer. The technology applies the Transformer method in deep learning and dynamic anchor frame structure, and further optimizes the small size target existing in the COCO dataset.

[0007] The small size target detection method based on dynamic anchor frame and Transformer of the present application comprises the following steps:

[0008] Step 1: Construct a small size target detection network;

[0009] Step 2: Extract small size target sample image data based on the COCO dataset, and perform image enhancement preprocessing on the extracted small target sample image data to obtain a training set;

[0010] Step 3: Train the small size target detection network on the training set, use multiple data enhancement strategies during training, and optimize the parameters through the validation set to obtain the trained small size target detection network;

[0011] Step 4: Load the trained small size target detection network parameters into the dynamic anchor frame Transformer model, then test the detection results on the validation set, and detect the subsequent collected test images.

[0012] Further, the small size target detection network comprises a sequentially connected backbone network module, an encoder module, a decoder module and a prediction output module; the backbone network module is used to extract the features of the small target and output a preliminary feature map; the encoder module is used to further extract the features based on the preliminary feature map and output intermediate values to the decoder module; the decoder module calculates the intermediate values based on sparse cross attention and outputs features to the prediction output module; the prediction output module performs prediction frame coordinate regression and prediction frame class classification based on the candidate frame.

[0013] Further, the backbone network module comprises an input layer, a first group of convolution layers, a maximum pooling layer, a second group of convolution layers, a third group of convolution layers, a fourth group of convolution layers and a fifth group of convolution layers connected in sequence, wherein the input size of the input layer is 513x513 image data; the first group of convolution layers comprises two parts of 1 7x7 convolution operation and 1 nonlinear activation function operation in sequence, the second group of convolution layers comprises 9 convolution layers, a nonlinear activation layer and an average pooling layer in sequence, the third group of convolution layers comprises 12 convolution layers, a nonlinear activation layer and an average pooling layer in sequence, the fourth group of convolution layers comprises 69 convolution layers, a nonlinear activation layer and an average pooling layer in sequence, and the fifth group of convolution layers comprises 9 convolution layers, a nonlinear activation layer and an average pooling layer in sequence, wherein each convolution layer in the second group of convolution layers to the fifth group of convolution layers sequentially undergoes 1 1x1 convolution, 1 3x3 convolution and 1 1x1 convolution operation;

[0014] The encoder module comprises six encoders, which are Transformer encoders, and the encoder module is stacked by six identical encoders, each of which has two sub-layers, the first sub-layer is a multi-head self-attention aggregation, and the second sub-layer is a position-based feedforward neural network. Each sub-layer adopts a residual link. The encoder adds the serialized feature map and the position encoding to obtain the query Q and the key value K, which are added to the feature map after passing through a multi-head self-attention layer and are normalized, and then pass through a feedforward network to obtain the output of a single encoder and serve as the input of the next encoder. After passing through the six identical encoder structures, the output of the encoder part is obtained.

[0015] The decoder module: the input of the decoder is composed of the output of the encoder, position encoding and query. The dimension of the query is (300, 4), the first dimension is the number of predefined target queries, and the second dimension is the number of hidden layers. That is, according to the encoded features of the encoder, the decoder converts the query into a target. Then calculate the cross attention with the output of the encoder, and after passing through six identical decoder structures, the output of the decoder part is obtained. The dynamic anchor box with size information is used as the query in the decoder, and the output of each layer of the decoder is changed into the offset of the coordinates of the anchor box, and the anchor box is iteratively updated. At the same time, a grouping mixed assignment strategy is used to optimize the small size target detection query.

[0016] Prediction output module: continue to calculate according to the features output by the decoder, mainly composed of feedforward neural network and full connection layer. The feedforward neural network layer is mainly divided into two parts, one part predicts the category, and the other part predicts the position. The branch of predicting the category is composed of a hidden layer with a linear layer of dimension 512. Since there is a background class (empty class), the output dimension is the number of categories plus 1. The other branch of the feedforward neural network that predicts the position is mainly composed of 3 hidden layers with a linear layer of dimension 512, and both branches will pass through a sigmoid activation function.

[0017] Further, the image enhancement preprocessing in step 2 includes normalization processing, random flipping, random cropping and image size adjustment.

[0018] Further, the plurality of data enhancement strategies in step 3 refer to Mosaic data enhancement and GridMask data enhancement.

[0019] Beneficial effects, introduce the hierarchical structure of the feature pyramid and sparse attention calculation, convert dense attention calculation to sparse attention calculation. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is the network structure diagram and detailed module diagram of the application

[0021] Figure 2 is the structure diagram of the sparse attention calculation module

[0022] Figure 3 is the structure diagram of the Transformer decoder module

[0023] Figure 4 is the mAP index comparison diagram of the algorithm of the application and the original algorithm

[0024] Figure 5 is the training process index comparison diagram of the algorithm of the application

[0025] Figure 6 is the detection result comparison diagram of the algorithm of the application and the original algorithm DETAILED DESCRIPTION

[0026] The application will be further described below in conjunction with the drawings.

[0027] A small size target detection method based on dynamic anchor frame and Transformer, comprising the following steps:

[0028] Step 1: Constructing a small size target detection network

[0029] The small size target detection network comprises a backbone network module, an encoder module, a decoder module and a prediction output module connected in sequence.

[0030] The backbone network module is configured to extract features of small targets and output a preliminary feature map;

[0031] The encoder module is configured to further extract features based on the preliminary feature map and output intermediate values to the decoder module; the decoder module is configured to calculate the intermediate values based on sparse cross attention and output features to the prediction output module; the prediction output module is configured to perform prediction box coordinate regression and prediction box class classification based on the candidate box. After the construction, the parameters of the small-size target detection and recognition network are randomly initialized.

[0032] The backbone network module comprises an input layer, a first group of convolutional layers, a maximum pooling layer, a second group of convolutional layers, a third group of convolutional layers, a fourth group of convolutional layers and a fifth group of convolutional layers connected in sequence, wherein the input size of the input layer is 513x513 image data; the first group of convolutional layers comprises two parts of 1 7x7 convolution operation and 1 nonlinear activation function operation in sequence, the second group of convolutional layers comprises 9 convolutional layers, a nonlinear activation layer and an average pooling layer in sequence, the third group of convolutional layers comprises 12 convolutional layers, a nonlinear activation layer and an average pooling layer in sequence, the fourth group of convolutional layers comprises 69 convolutional layers, a nonlinear activation layer and an average pooling layer in sequence, and the fifth group of convolutional layers comprises 9 convolutional layers, a nonlinear activation layer and an average pooling layer in sequence, wherein each convolutional layer in the second group of convolutional layers to the fifth group of convolutional layers sequentially undergoes 1 1x1 convolution, 1 3x3 convolution and 1 1x1 convolution operation.

[0033] The encoder module comprises six encoders, which are Transformer encoders, and the encoder module is stacked by 6 identical encoders, each of which has two sub-layers, the first sub-layer is a multi-head self-attention aggregation, and the second sub-layer is a position-based feedforward neural network. Each sub-layer adopts a residual link. The encoder adds the serialized feature map and the position encoding to obtain the query Q and the key value K, adds and normalizes the feature map after passing through a multi-head self-attention layer, and obtains the output of a single encoder through a feedforward network, and the output is used as the input of the next encoder. After passing through 6 identical encoder structures, the output of the encoder part is obtained, as shown in Figure 2 .

[0034] The decoder module: the input of the decoder is composed of the output of the encoder, the position encoding and the query. The dimension of the query is (300, 4), the first dimension is the number of predefined target queries, and the second dimension is the number of hidden layers. That is, according to the encoded features of the encoder, the decoder converts 300 queries into 300 targets. The decoder calculates the first half of the multi-head self-attention in the same way as the encoder, and then calculates the cross-attention with the output of the encoder. After passing through 6 identical decoder structures, the output of the decoder part is obtained, as shown inFigure 3 The dynamic anchor box with size information is taken as a query in the decoder to provide better spatial prior information, and the output of each layer of the decoder is changed to the offset of the coordinates of the anchor box, and the anchor box is iteratively updated. A grouping mixed allocation strategy is used to optimize the foreground-background (positive-negative) sample imbalance problem in small size target detection queries.

[0035] The prediction output module: continue to calculate according to the features output by the decoder, mainly composed of a feedforward neural network and a fully connected layer. The feedforward neural network layer is mainly divided into two parts, one part predicts the category and the other part predicts the position. The branch of predicting the category is composed of a hidden layer with a dimension of 512 linear layers. Since there is a background class (empty class), the output dimension is the number of categories plus 1. The other branch of the feedforward neural network that predicts the position is mainly composed of 3 hidden layers with a dimension of 512 linear layers, and both branches will pass through a sigmoid activation function.

[0036] The transmission process of the detected picture in the small size target detection method based on the dynamic anchor box of the Transformer proposed in the application is as shown in Figure 1

[0037] Step 2: Extract small size target sample image data based on the COCO dataset, and perform image enhancement preprocessing on the extracted small target sample image data to obtain a training set;

[0038] The COCO dataset includes more than 80 categories such as pedestrians, bicycles, electric vehicles, cars, trucks, buses, and fire hydrants.

[0039] The image enhancement preprocessing includes normalization processing, random flipping, random cropping, image size adjustment, etc. The image normalization mean is [123.67, 116.28, 103.53], and the image normalization standard deviation is [58.39, 57.12, 57.37].

[0040] Step 3: Train the small size target detection network on the training set, adopt multiple data enhancement strategies during training, and optimize the parameters through the validation set to obtain the trained small size target detection network;

[0041] The multiple data enhancement strategies refer to Mosaic data enhancement and GridMask data enhancement.

[0042] ​The specific implementation process of training the small-size target detection network is as follows: the preprocessed small-size target sample image data is input into the small-size target detection network with initialized parameters, forward propagation is performed to obtain regression and classification results, the loss of the regression and classification results is calculated according to the small-size target sample image data, the loss is used for reverse propagation to update the parameters of the small-size target detection network, and when the iteration condition is reached, the trained small-size target detection network is obtained.

[0043] The model parameter optimization algorithm during the reverse propagation is SGD, the learning rate during the unfreezing training is 1e-5, the number of batches of images during the freezing training is 2, the probability of randomly discarding neurons for preventing overfitting is 0.1, and the number of hidden layers of the encoder-decoder is 256.

[0044] Step 4: loading the best small-size target detection network parameters trained into the dynamic anchor box Transformer model, then testing the detection results on the validation set, and detecting the subsequent collected test images.

[0045] The present application optimizes the feature extraction backbone network output feature map, sparse attention mechanism, dynamic anchor box, loss function calculation and the like on the basis of the original Transformer algorithm, proposes a small-size target detection method based on the dynamic anchor box of the Transformer, and effectively enhances the detection accuracy of the network model on small-size target objects. The detection effect is as shown in Figure 4 Compared with the original Transformer algorithm, the mAP index is greatly improved. The training of the model is also more stable, and the loss function during the training process is as shown in Figure 5 The visualization comparison chart of the detection results is as shown in Figure 6 .

[0046] The above is only a specific embodiment of the present application, any feature disclosed in the description can be replaced by other equivalent or similar purpose alternative features unless specifically described, and all features disclosed or all steps in the method or process can be combined in any way except for mutually exclusive features or steps.

Claims

1. A method for detecting small-sized targets based on dynamic anchor frames and Transformer, characterized in that, Includes the following steps: Step 1: Construct a small-size target detection network; The small target detection network consists of a backbone network module, an encoder module, a decoder module, and a prediction output module connected in sequence. The backbone network module is used to extract features from small targets and output preliminary feature maps; The encoder module is used to further extract features based on the initial feature map and output intermediate values ​​to the decoder module; The decoder module calculates features from intermediate values ​​based on sparse cross-attention and outputs them to the prediction output module. The prediction output module performs prediction box coordinate regression and prediction box category classification based on the candidate boxes. Dynamic anchor boxes with size information are used as queries in the decoder, and the output of each decoder layer is changed to the coordinate offset of the anchor box, and the anchor box is iteratively updated; at the same time, a grouped hybrid allocation strategy is used to optimize small-sized target detection queries. Step 2: Extract small-sized target sample image data from the COCO dataset, and perform image enhancement preprocessing on the extracted small target sample image data to obtain the training set; Step 3: Train the small-size object detection network on the training set. During training, various data augmentation strategies are used, and the parameters are tuned using the validation set to obtain the trained small-size object detection network. Step 4: Load the trained small-size object detection network parameters into the dynamic anchor box Transformer model, then test the detection results on the validation set, and perform detection on subsequently acquired test images.

2. The method for detecting small targets based on dynamic anchor frames and Transformer according to claim 1, characterized in that, The backbone network module comprises an input layer, a first set of convolutional layers, a max pooling layer, a second set of convolutional layers, a third set of convolutional layers, a fourth set of convolutional layers, and a fifth set of convolutional layers connected in sequence. The input layer contains 513x513 image data. The first set of convolutional layers consists of a 7x7 convolution operation and a non-linear activation function operation. The second set of convolutional layers consists of 9 convolutional layers, a non-linear activation layer, and an average pooling layer. The third set of convolutional layers consists of 12 convolutional layers, a non-linear activation layer, and an average pooling layer. The fourth set of convolutional layers consists of 69 convolutional layers, a non-linear activation layer, and an average pooling layer. The fifth set of convolutional layers consists of 9 convolutional layers, a non-linear activation layer, and an average pooling layer. Each convolutional layer from the second to the fifth set of convolutional layers undergoes a 1x1 convolution, a 3x3 convolution, and a 1x1 convolution operation in sequence. The encoder module includes six encoders, which are Transformer encoders. The encoder module is composed of six identical encoders stacked together. Each encoder has two sub-layers. The first sub-layer is a multi-head self-attention convergence layer. The second sub-layer is a position-based feedforward neural network. Each sub-layer uses residual connections. The encoder sums the serialized feature map with the position encoding to obtain the query Q and the key value K. After passing through a multi-head self-attention layer, it is summed with the feature map and normalized. Then, it passes through a feedforward network to obtain the output of a single encoder, which is used as the input of the next encoder. After passing through the six identical encoder structures, the output of the encoder part is obtained. Decoder module: The input of the decoder consists of the encoder output, positional encoding, and query; where the query dimension is (300, 4), the first dimension is the number of predefined target queries, and the second dimension is the number of hidden layers; that is, according to the features encoded by the encoder, the decoder transforms the query into the target; then cross-attention is calculated with the encoder output, and the decoder output is obtained after passing through 6 identical decoder structures; The prediction output module continues to calculate based on the features output by the decoder, and consists of a feedforward neural network and a fully connected layer. The feedforward neural network layer is divided into two parts: one part predicts the category and the other part predicts the location. The branch predicting the category consists of a linear layer with a hidden layer of dimension 512. Since there is a background class, the output dimension is the number of categories plus 1. The other branch of the feedforward neural network predicting the location consists of a linear layer with three hidden layers of dimension 512. Both branches pass through a sigmoid activation function.

3. The method for detecting small-sized targets based on dynamic anchor frames and Transformer according to claim 1, characterized in that, The image enhancement preprocessing described in step 2 includes normalization, random flipping, random cropping, and image resizing.

4. The method for detecting small-sized targets based on dynamic anchor frames and Transformer according to claim 1, characterized in that, The various data augmentation strategies mentioned in step 3 refer to the use of Mosaic data augmentation and GridMask data augmentation.

Images