A position encoding method for object detection based on reparameterized convolution

Abstract

The present invention discloses a target detection position encoding method based on re-parameterized convolution. The method first performs sliding window upsampling on the image vector to obtain the expanded image vector W out . Then extract the W after sliding window upsampling out The convolution position coding information and the absolute position information of the image mask are extracted. Finally, the convolution position coding information and the absolute position information are fused through the position coding and extraction module to obtain the fused image position coding features, completing the target detection position coding. This invention fully utilizes the image feature information and the mask, better utilizing the original features of the image, thereby improving the detection capability of the entire model.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and in particular relates to a target detection position encoding method based on reparameterized convolution. Background Art

[0002] The Transformer model has been widely used in various computer tasks, including large language models such as ChatGPT and GPT4. It has also been used in visual tasks. For example, Vision Transform pioneered the application of the Transform structure to visual tasks, and DETR proposed an end-to-end detection framework that does not require additional structural design. DINO, as a subsequent improver, introduced scalable Transformer operation blocks and denoising technology, greatly enhancing the speed and accuracy of DETR's object detection. However, the Transformer model does not carry the positional feature information of the image, so researchers have proposed various position encoding methods to compress the two-dimensional coordinate information of the image into a one-dimensional vector, which can then be entered into the Transformer model for operation to achieve better results.

[0003] Currently, there are three typical visual position encoding methods: absolute position encoding, relative position encoding, and convolution-based position encoding. In absolute position encoding, each position in a sequence has a fixed position vector. This means that for sequences or images of the same size, the generated encoding is identical. Relative position encoding was inspired by researchers' discovery that position information is lost during self-attention calculations. Therefore, an offset is added to each self-attention calculation. Convolution-based position encoding replaces sine and cosine encoding with convolution. However, unlike conventional position encoding, convolution-based position encoding is placed between each Transformer block, rather than generated during the embedding process before the Transformer. Furthermore, all of these position encodings are based on the original image extraction, rather than the extracted image feature vector. Conventional DETR position encoding, on the other hand, is generated on a mask vector, eliminating the need for post-feature extraction image feature vector information and resulting in significant loss of image feature information. The target detection position encoding method based on reparameterized convolution accepts the image feature vector information after feature extraction, and also accepts the Mask vector to fuse and generate position encoding information. Summary of the Invention

[0004] To address the problem that conventional position coding is not effective on target detection heads, this paper proposes a target detection position coding method based on reparameterized convolution. The present invention is mainly divided into three parts:

[0005] In the first part, in order to solve the problem of the different number of information channels output by its feature extraction layer, the present invention proposes a new sliding window upsampling module. This module uses a sliding window method for calculation. First, the length of the original array is calculated, and then the two sides of the array are properly padded to ensure that the window can slide completely at the edge. Then, an S is set as the width of the window, and the original array is subjected to sliding window processing. In each window, the average value of the elements in the window is calculated, and this average value is inserted into the corresponding position of the original array. In this way, a new array that has been upsampled can be obtained. Its length is twice that of the original array, but the average value is inserted inside the window, thereby achieving the upsampling effect.

[0006] In the second part, in order to solve the problem, the target detection head has two inputs: Mask and image feature vector W in However, only the mask is used for position encoding, which means that the position encoding does not contain the original feature information of the image. This paper proposes a reparameterized convolutional coding module, which consists of three sections. The first section performs reparameterized convolution encoding on the image feature vector, the second section performs absolute position encoding on the mask, and the last section uses the output of the first and second sections to perform a fusion extraction position encoding operation using reparameterized convolution again.

[0007] In the third part, in order to improve the model quality, prevent gradient explosion and numerical stability, the three parts of the previous part are fused, normalized and activated.

[0008] A method for object detection position encoding based on reparameterized convolution is specifically implemented including the following steps:

[0009] Step (1). Transform the image vector W in , the size is (B, C, H, W), where B represents the number of images in each batch, C represents the number of channels of image features, H and W represent the width and height of the image respectively, and the sliding window upsampling process is performed. First, the image feature W is detected. in The size of the second dimension is the size of the number of channels C, and then the sliding window upsampling module is used to expand the second dimension of the image vector to size M to obtain the expanded image vector W out , its size is (B,M,H,W), and M is generally 2048.

[0010] Step (2). Extract W after sliding window upsampling out The convolution position encoding information W Conv , and extract the absolute position information W of the image mask Mask APE .

[0011] For Wout Use reparameterized convolution operation, which extracts the convolution kernel parameters W after training. b After a series of parameter changes, the convolution kernel parameter W is obtained t , and use W t As the final convolution kernel, the convolution operation is performed. At the same time, the convolution has the ability to change the number of channels to obtain the position encoding information W after the first extraction. Conv , its size is (B,N,H,W), and N is generally 256.

[0012] Perform absolute position encoding on the mask. First, calculate the cumulative sum matrix M according to the size of the mask (B, H, W). Its size is also (B, H, W). Perform numerical regularization on the cumulative sum matrix. Then use the temperature hyperparameter module to smooth the numerical distribution of the matrix M. Then perform size mapping to make it a vector of (B, 1, H, W). Finally, copy and expand the second dimension to N, and for each element in the array, calculate the ratio between its value and the array index (from 0 to N) to obtain the absolute position encoding W. APE , whose size is (B,N,H,W).

[0013] Step (3). Encode the convolution position information W Conv and absolute position information W APE , by fusing the position coding and re-extraction module, the fused image position coding feature W is obtained cpe , completing the target detection position encoding.

[0014] First, the absolute position is encoded W APE And the position encoding information W after convolution conv Add together to get the fused feature W f . Secondly, W f The reparameterized convolution operation is performed again. The reparameterized convolution here does not transform the number of channels, so as to obtain W fc Then batch normalization is performed on W fc Normalize the values ​​in the channel dimension, which is the second dimension, to ensure stability and accelerated convergence during model training. The number of channels in the batch normalization layer used here is N. Finally, the batch normalized feature W batchnorm Input into the activation function ReLu to obtain the final fused image position encoding feature W cpe .

[0015] Beneficial effects of the present invention:

[0016] The model of the present invention is composed of a sliding window upsampling module, a reparameterized convolutional coding module, a fusion and reparameterized convolutional coding module, and a regular fusion activation module. The sliding window upsampling module performs upsampling expansion processing by perceiving the vector, thereby determining the dimensionality of the subsequent vector and thereby improving the resolution or spatial information. The reparameterized convolutional coding module extracts the depth position information in the image by reparameterizing the image feature vector itself while scaling the dimension to N. Another part, the absolute position coding module, is the information that performs absolute position coding on the normalized cumulative sum generated according to the size of the mask. Finally, there is the fusion and reparameterized convolutional coding module, which uses reparameterized convolution fusion to process the position coding of the above two parts. At the same time, for the stability of model training and reasoning, the present invention uses batch normalization to normalize the data to a normal distribution, and finally connects a linear rectification activation function to enhance the nonlinear ability of the model. The reparameterized convolutional coding module proposed in the present invention makes full use of image feature information and mask, and uses convolution to embed position coding information into the vector in advance, thereby better utilizing the original features of the image, thereby improving the detection ability of the entire model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Position encoding method for target detection based on reparameterized convolution;

[0018] Figure 2 Sliding window upsampling module. DETAILED DESCRIPTION

[0019] The present invention proposes a target detection position encoding method based on reparameterized convolution. Figure 1As shown in the figure, the overall structure consists of a sliding window upsampling module, a re-reference convolutional encoding module, an absolute position encoding module, and a fused re-reference convolutional module. First, through the sliding window upsampling module, the four image feature vectors of different dimensions are aligned to the same dimension. The method is to upsample the image vector to obtain a unified, aligned image feature vector to be processed. Next, re-reference convolution is performed on the image. Unlike general convolution, re-reference convolution incorporates more learnable parameters into the model during training and trains them to achieve better performance. During inference, these parameters are merged into the parameters of the original model to avoid introducing additional inference costs. Then batch normalization is used to perform numerical stabilization on the convolved image features, and the linear rectification activation function is used to expand the fitting capability of the encoding module. At the same time, the cumulative sum is calculated by obtaining the vector shape of the Mask, smoothed by the temperature hyperparameter, and the same numerical stabilization is performed by normalization. Secondly, the vector after the reparameterized convolution of the image features and the vector after the absolute position encoding are added together, and the fused re-reference convolution module is used to perform position encoding extraction on the fused preliminary extracted image position encoding features again. Finally, batch normalization and linear rectification functions are used to perform the same numerical stabilization. In this way, by increasing the original processing of the image features and fusing the original absolute position encoding and image features, the position encoding will have stronger position prior features.

[0020] Next, the implementation steps will be described in detail with reference to the accompanying drawings.

[0021] Step (1). Figure 2 As shown in the figure, the sliding window upsampling module upsamples the feature vectors of four layers with different dimensions to the same dimension. in , the size is (B, C, H, W) and the expansion upsampling is performed to increase the size of the image or feature map, thereby improving the resolution or spatial information.

[0022] Step (1.1) First, the sliding window upsampling module uses the sliding window to calculate the inserted element list. S is the size of the sliding window, P is the padding of the element list to meet the calculation needs of the edge elements, and ST is the stride size, that is, the distance the sliding window moves each time. inp W in The vector formed by filling the front and back with 0 elements is:

[0023] W inp =[0]*P / / 2+W in +[0]*(PP / / 2)

[0024] Where P / / 2 means P divides 2.

[0025] Step (1.2) uses a sliding window to upsample the vector Ww The calculation formula is:

[0026]

[0027] Where W inp [:,i:i+S,:] means traversing and selecting W inp The elements from i to i+S in the second dimension are summed and averaged.

[0028] Step (1.3) alternately combines the upsampled vector W w and W in Get the final result W out , the formula is:

[0029] W out [:,2i,:,:]=W in [:,i,:,:]

[0030] W out [:,2i+1,:,:]=W w [:,i,:,:]

[0031] Where W in Insert into W out In the even-numbered rows, W w Insert into W out In the odd-numbered rows, the final W is formed out .

[0032] Step (2). The position information extraction module is divided into two parts: the process of extracting image position features by reparameterized convolution, and the process of obtaining absolute position encoding information by using cumulative sum and temperature smoothing.

[0033] Step (2.1). First, use reparameterized convolution to extract image position information features.

[0034] Step (2.1.1). First, the weight parameter of the initial convolution is W b , K is the convolution kernel size. Then use the reparameterized transformation T(.), the formula is:

[0035] W t =W b *W r +W b

[0036] in to complete the reparameterization transformation.

[0037] Step (2.1.2). Use the new convolution kernel weight W t For the W obtained in step (1), outPerform convolution operation to obtain the output tensor W Conv .

[0038] Step (2.2). Calculate the absolute position encoding information of the image

[0039] Step (2.2.1). Get the valid area of ​​the image through the Mask vector. In Mask, valid is 1 and invalid is 0. Mask can be regarded as a two-dimensional matrix M with a size of H high and W wide.

[0040] Step (2.2.2). Calculate the cumulative sum in the directions of image H and W respectively. The formula is:

[0041]

[0042] X is x ij The matrix composed of y ij The matrix composed of ij is the element in the i-th row and j-th column of the matrix M.

[0043] Step (2.2.3). Manually regularize the obtained X and Y. The calculation formula is:

[0044]

[0045] n is the last element of the H dimension in Y, that is, the one-way sum of the cumulative sums in step (2.2.2), and the same operation is performed in the X direction.

[0046] Step (2.2.4). Use the temperature hyperparameter to smooth X and Y. Temperature is the smoothing coefficient used to smooth a set of values, and is generally set to 10000. The formula is:

[0047]

[0048] in Indicates integer division of X and rounding. Do the same for Y.

[0049] Step (2.2.5). Finally, concatenate X and Y in the third dimension, that is, the dimension of C, to obtain the final feature vector W APE .

[0050] Step (3). Use the linear rectification function and batch normalization function to perform conventional normalization and activation processing to obtain the final position encoding output W cpe , the formula is:

[0051] W cpe =Relu(BatchNorm2d(W conv +WAPE ))

[0052] The experimental results are shown in Table 1. AP (Average Precision) is called average accuracy. It is calculated by integrating the area enclosed by the PR curve of each category and the coordinate axis. The larger the area, the higher the average accuracy of the model, and the overall precision and recall are also relatively high. 50 and AP 75 They represent the AP values ​​corresponding to IoU thresholds of 0.5 and 0.75 respectively. S 、AP M 、AP L Respectively indicate that the target area is less than 32 2 Small targets with an area greater than 32 2 Less than 96 2 Medium targets and areas larger than 96 2 The AP value corresponding to the large target.

[0053] Table 1

[0054]

[0055] This paper uses the End-to-End Object Detection with Improved Denoising Anchors (DINO) method as a baseline model to compare its performance with other existing image position encoding methods. Training is performed at half-scale (6 epochs), using a ResNet-50 network as the backbone. Comparative results on the MS COCO dataset are shown in the figure, showing that the present invention enables DINO to achieve an absolute gain of +0.7 in AP. This results in significant improvements for common objects, such as medium-sized objects, and a smaller improvement for large objects.

[0056] The performance improvement achieved by this invention is superior to the current DETR series of target detection methods, such as native DETR: an end-to-end target detection method based on Transformer; Deformable DETR: a target detection method that uses a variable Attention module to accelerate convergence; Conditional DETR: a target detection method that improves the query generation method in DETR; DINO: a target detection method that denoises and inserts position information into the query.

[0057] The present invention uses reparameterized convolution to extract image position features and also uses reparameterized convolution to fuse the information after absolute position encoding, thereby enhancing the model's ability to process image position information. In the subsequent calculation process, more encoded information is integrated into the vector, making the target detection results more accurate, that is, obtaining higher classification confidence and IoU score.

Claims

1. A target detection position encoding method based on reparameterized convolution, characterized in that: The steps include: Step (1). Perform sliding window upsampling on the image vector to obtain the expanded image vector W out , the specific process is: image vector W in , the size is (B, C, H, W), where B represents the number of images in each batch, C represents the number of channels of image features, H and W represent the width and height of the image respectively, and the sliding window upsampling process is performed; The upsampling process is as follows: first, the image feature W is detected. in The size of the second dimension, that is, the size of the number of channels C, is then uniformly expanded to size M using a sliding window upsampling module on the second dimension of the image vector to obtain the expanded image vector W out , whose size is (B,M,H,W); Step (2). Extract W after sliding window upsampling out The convolution position encoding information W Conv , and extract the absolute position information W of the image mask Mask APE ; The extraction out The convolution position encoding information W Conv The details are as follows: For W out Use the reparameterized convolution operation to extract the trained convolution kernel parameters W b After changing the parameters, the convolution kernel parameters W are obtained t , and use W t As the final convolution kernel, perform the convolution operation; at the same time, obtain the position coding information W after the first extraction Conv , whose size is (B,N,H,W); The absolute position information W of the extracted image mask Mask APE The details are as follows: Perform absolute position encoding on the Mask. First, calculate the cumulative sum matrix M according to the size of the Mask (B, H, W). Its size is (B, H, W). Perform numerical regularization on the cumulative sum matrix. Then use the temperature hyperparameter module to smooth the numerical distribution of the matrix M. Then perform size mapping to make it a vector of (B, 1, H, W). Finally, copy and expand the second dimension to N, and for each element in the array, calculate the ratio between its value and the index of the array to obtain the absolute position encoding W. APE , whose size is (B,N,H,W); Step (3). Encode the convolution position information W Conv and absolute position information W APE , by fusing the position coding and re-extraction module, the fused image position coding feature W is obtained cpe , completing the target detection position encoding.

2. The target detection position encoding method based on reparameterized convolution according to claim 1, characterized in that: The specific process of step (3) is as follows: First, the absolute position is encoded W APE And the position encoding information W after convolution Conv Add together to get the fused feature W f ; Secondly, W f Perform the reparameterized convolution operation again. The reparameterized convolution here does not transform the number of channels to obtain W fc ; Then, batch normalization is performed on W fc Normalize the values ​​in the channel dimension, and the number of channels of the batch normalization layer used is N; Finally, the batch normalized features W batchnorm Input into the activation function ReLu to obtain the final fused image position encoding feature W cpe .

Images