Lightweight detection method for point target of optical image

Abstract

The invention discloses an optical image point target lightweight detection method, which utilizes a single-layer attention module to carry out point target position and confidence prediction on sparse space points on an image, and reduces two indexes of network parameter quantity and floating point operation times by 3-4 orders of magnitude compared with a current common target detection network. And the method has a relatively great application value on a satellite-borne or airborne information processing platform with limited computing resources. The thermal infrared satellite image civil aircraft detection task verifies that the method can effectively detect the dark point-shaped civil aircraft. According to the method, a deformable attention mechanism based on space sparse sampling is adopted, the calculation process is further optimized, and attention feature calculation is achieved with a small calculation amount.

Description

Technical Field

[0001] The invention relates to the field of optical image target detection, and in particular to a lightweight detection method for optical image point targets. Background Art

[0002] If the optical monitoring device is far away from the object to be measured (i.e., the target), or the object to be measured (i.e., the target) is close to or smaller than the image resolution, the object to be measured (i.e., the target) will appear as a dot on the image. Traditional point detection methods are mainly based on image filtering methods, which use the grayscale abnormality of target imaging and the slow grayscale variation of the background area to suppress the background area. This type of algorithm requires fewer computing resources and is widely used in civil fields such as civil aviation monitoring, object recognition, and image detection. In addition, the detection algorithm based on sparse low-rank reconstruction will consider small targets to be sparse and the slowly varying background to be low-rank, and the target can be separated from the background using low-rank reconstruction. The method based on tensor analysis expands the optical image to a three-dimensional tensor space, and separates the target from the overall low-rank background from the perspective of tensor decomposition. These methods rely on prior knowledge of the target and manual setting of key parameters, which are difficult to adapt to complex scenes and are prone to false alarms and missed alarms. In recent years, deep learning-based methods have become a new research hotspot in the field of target detection. These methods can automatically learn effective features from a large amount of data, avoid the manual setting of feature templates, and have a certain degree of adaptability and generalization for complex scenes. The current target detection network is mainly aimed at targets with large size and shape information. First, the visual basic model based on convolutional neural network is used to extract texture and shape features from the optical image, and then the detection head module is used to detect and locate the target from the feature map containing texture, shape and depth characteristics output by the visual basic model. However, point targets occupy a small number of pixels in the optical image and have very few shape features, so the existing detection network has great redundancy.

[0003] Taking the 30-meter spatial resolution thermal infrared image collected by the public Sustainable Development Science Satellite 1 as an example, the size of civil aircraft is close to the ground sampling interval of the thermal infrared payload, and appears as a dark spot. Figure 1 The enlarged image on the right shows that the radiation energy of civil aircraft is distributed in the 7×7 pixel space and is mainly concentrated in the 2×2 pixel space. The existing target detection network structure related to target feature extraction and recognition does not take into account the characteristics of point targets with few spatially correlated pixels, and the existing target detection network structure has a large redundancy.

[0004] The terms involved in the present invention are explained as follows:

[0005] Visual base models: refers to a class of deep learning models that are designed to understand and interpret the visual content in image or video data. These models are usually based on convolutional neural networks or variants, as well as attention-based models that have become popular in recent years. The purpose of visual base models is to extract meaningful feature representations from raw pixels, which can be used to solve various computer vision tasks such as image classification, object detection, semantic segmentation, pose estimation, etc.

[0006] Detection head module: refers to a class of deep learning models that are designed to understand and interpret the visual content in image or video data. These models are usually based on convolutional neural networks or variants, as well as attention-based models that have become popular in recent years. The purpose of the visual base model is to extract meaningful feature representations from raw pixels, which can be used to solve various computer vision tasks such as image classification, object detection, semantic segmentation, pose estimation, etc.

[0007] Deformable attention: It was first proposed in Deformable DETR (Zhu X, Su W, Lu L, et al. DeformableDETR: Deformable Transformers for end-to-end object detection [J]. arxiv preprint arxiv: 2010.04159, 2020.), which samples key points by applying a learnable offset on the feature map instead of using fixed-position sampling points. In this way, the model can adaptively adjust its focus in different objects and scenes, thereby improving the accuracy and efficiency of target detection. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a lightweight detection method for optical image point targets in view of the shortcomings of the existing technology, which can effectively detect dark point targets on a satellite or airborne information processing platform with limited computing resources.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is: a lightweight detection method of optical image point targets, comprising the following steps:

[0010] S1, preprocessing and channel expansion of the input single-channel or multi-channel optical image;

[0011] S2, sparsely sample the pixels on the feature image after channel expansion to obtain query points and feature vectors;

[0012] S3, calculating a low-dimensional query vector corresponding to the query point;

[0013] S4, calculating the positions of several feature sampling points of the query point from the low-dimensional query vector, sampling several feature vectors from the feature image using bilinear interpolation, multiplying each sampled feature vector by the corresponding attention weight and summing them up to obtain an attention feature vector;

[0014] S5, using the attention feature vector to update the feature vector of the query point;

[0015] S6. Use the updated feature vector to predict the confidence of the existence of the target at the query point and the target rectangular prediction box.

[0016] The present invention uses a simplified deformable attention layer and a few linear layers and other components to build a detection model for point targets. Compared with the current common target detection network, the number of network parameters and the number of floating-point operations are reduced by 3-4 orders of magnitude, which has great application value in terminal-side deployment reasoning with limited computing resources.

[0017] In the present invention, the expression of the feature image F after channel expansion is: F = linear (I); linear represents the linear layer, Among them, H represents the image height, W represents the image width, and C represents the number of image channels.

[0018] In step S2 of the present invention, the specific implementation process of obtaining the query point includes: dividing the feature image into a number of non-overlapping image blocks, and selecting a pixel point with a relatively fixed position in each image block as a query point.

[0019] In step S3 of the present invention, the low-dimensional query vector expression is:

[0020] q i,j = linear(f i,j +p i,j );

[0021] in, c<C, c is the query vector dimension, q i,j is the query vector corresponding to the query point (i, j), i represents the vertical position, j represents the horizontal position, Represents the feature vector on the feature image calculated using sinusoidal coding The position encoding of , linear represents the linear layer.

[0022] In step S4 of the present invention, the attention feature vector a of the query point (i, j) is i,j The calculation formula is:

[0023]

[0024] in, M is the number of heads in the attention module, K is the number of sampling points of each attention head, m and k are the indexes of the attention head and sampling points, A m,k Represents the attention weight of each sampling point, Δi m,k and Δj m,k Indicates the sampling point relative to f i,j The position offset, g(F,Δi m,k, Δj m,k ) represents the feature pixels sampled from the feature image F using bilinear interpolation.

[0025] Position offset Δi m,k and Δj m,k The calculation formula is:

[0026]

[0027] A m,k The range is [0,1], satisfying

[0028] In step S5, the updated feature vector f′ of the query point i,j Expressed as: f′ i,j =norm(f i,j +a i,j ); norm represents the normalization layer.

[0029] In step S6, the calculation formula of the target rectangular prediction box b is: Among them, Sigmoid represents the Sigmoid activation function, MLP represents the multi-layer perceptron, and f′ i,j is the updated feature vector of the query point (i, j), x0 and y0 are the centers of the anchor box, and w0 and h0 are the width and height of the anchor box.

[0030] In step S6, the calculation formula of the target confidence α is: α = Sigmoid[MLP(f′ i,j )].

[0031] Compared with the prior art, the present invention has the following beneficial effects: the present invention uses a single-layer attention module to predict the position and confidence of point targets at sparse spatial points on an image, which reduces the network parameter quantity and the number of floating-point operations by 3-4 orders of magnitude compared with the current common target detection network, and has great application value on satellite-borne or airborne information processing platforms with limited computing resources. It has been verified in the task of detecting civil aircraft in thermal infrared satellite images that the method of the present invention can effectively detect civil aircraft in the form of dark dots. The method of the present invention adopts a deformable attention mechanism based on spatial sparse sampling and further optimizes the calculation process, realizing attention feature calculation with a smaller amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 For civil aircraft on the optical image of the Sustainable Development Science Satellite 1;

[0033] Figure 2 This is a schematic diagram of the network structure of a point target detection method according to an embodiment of the present invention;

[0034] Figure 3 This is the inference result of the method of the embodiment of the present invention on the real point target. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] An embodiment of the present invention provides a lightweight optical image point target detection network, comprising the following steps:

[0037] S1. Preprocessing and channel expansion of the input single-channel or multi-channel optical image;

[0038] The input image is a single-band image or a multi-band optical image. Where d, H and W are the image channels, length and width respectively. In order to allow the network to have sufficient feature representation space, the number of image channels needs to be expanded:

[0039] F=linear(I)

[0040] Where linear represents the linear layer, is the image after dimensional expansion. Since there are very few point target shape features in a single frame image, it is no longer necessary to use the visual basic model to extract the deep feature meaning of the image, so the dimensional expansion image F is directly regarded as the network feature map.

[0041] S2. Sparsely sample the pixels in the feature image after dimension expansion to obtain query points and feature vectors;

[0042] Point targets are usually separated by a certain number of pixels, so it is not necessary to use each pixel as a query point for point target prediction. This method uses sparse sampling to select query points and query vectors, and divides the feature map F into several non-overlapping image blocks of size 2×2. Pixels with fixed relative positions on each image block are selected as query points.

[0043] S3. Calculate the low-dimensional query vector corresponding to the query point;

[0044] For each query point, this method uses a simplified deformable attention module to calculate its attention feature, which can reflect the neighborhood spatial features of the query point. For any query point (i, j), the feature vector on the feature map is Use sinusoidal encoding to calculate its position encoding Then we get the query vector embedded with the feature pixel location information

[0045] q i,j = linear(f i,j +p i,j )

[0046] Wherein, i represents the vertical position, j represents the horizontal position, and c < C. Different from the original deformable attention, the method of this embodiment uses a linear layer to reduce the dimension of the query vector to reduce the computational complexity of subsequent sampling point position mapping.

[0047] S4. Calculate the positions of several feature sampling points of the query point from the low-dimensional query vector, and then use bilinear interpolation to sample several feature vectors from the feature image, multiply each sampled feature vector by the corresponding attention weight and sum them to obtain the attention feature vector;

[0048] The attention feature vector of the query point (i, j) is calculated by the following formula:

[0049]

[0050] in, M is the number of heads of the attention module, K is the number of sampling points of each attention head, and m and k are the indexes of the attention heads and sampling points. m,k Represents the attention weight of each sampling point, which is a learnable variable with a range of [0,1], satisfying Δi m,k and Δj m,k Indicates the sampling point relative to f i,j The position offset of g(F,Δi m,k ,Δj m,k ) represents the feature pixel sampled from the feature map F using bilinear interpolation. m,k and Δj m,k All are by q i,j The sampling point positions obtained by linear mapping:

[0051]

[0052] In the original deformable attention, the attention weight A m,k By q i,jAfter linear mapping, this method sets the attention weights as a set of learnable parameters, and all query points share a set of attention weights, further reducing the computational complexity of the attention module.

[0053] S5. Update the feature vector of the query point using the attention feature vector;

[0054] After calculating the attention feature vector a i,j Then, the feature pixel f i,j To update:

[0055] f′ i,j =norm(f i,j +a i,j )

[0056] Among them, norm represents the layer normalization layer.

[0057] S6. Use the updated feature vector to predict the confidence of the existence of the target at the query point and the target rectangular prediction box.

[0058] Get the vector f′ that combines local spatial features and pixel features i,j After that, it is sent to the confidence prediction module and the position prediction module respectively to obtain the confidence of the existence of the target near the pixel position and the target rectangular prediction box respectively. In order to facilitate network training, in f i,j An initial anchor box is preset at the pixel position Among them, x0 and y0 are the centers of the anchor boxes, set to f i,j The corresponding pixel center, w0 and h0 are the width and height of the anchor box, which are set to 0.01 of the image width and height respectively. i,j Send it to the position prediction module to get the predicted rectangular box b of the target: Among them, Sigmoid represents the Sigmoid activation function, and MLP represents the MultiLayer Perceptron, which is composed of multiple linear layers and ReLU activation functions. Considering that the center coordinates, width and length of the normalized target rectangular box are in the range of (0,1), b0 is first subjected to an inverse Sigmoid operation, and then the position offset prediction output by the MLP is added and then mapped to (0,1) using the Sigmoid activation function. We then use the confidence prediction module to obtain the confidence α of the target prediction box: α=Sigmoid[MLP(f′ i,j )], α ranges from (0,1), and the closer it is to 1, the greater the possibility that the predicted box belongs to the target.

[0059] This method is verified on the public Sustainable Development Science Satellite 1 thermal infrared imagery for civil aircraft detection. The training set and test set are constructed by simulation on the public Sustainable Development Science Satellite 1 thermal infrared imagery. The training set contains 30,000 training images, and the test set contains 3,000 test images. In addition, based on the training of simulation data, this method is verified on the Sustainable Development Science Satellite 1 thermal infrared imagery containing real civil aircraft.

[0060] The experimental settings are as follows: the feature dimension C is set to 64, c is set to 8, the number of heads of the attention module M is set to 32, and the number of sampling points of each attention head is set to 4. The network input image size is 256×256, the number of training iterations is set to 50, the batch size is set to 8, the Adam optimizer is used, the learning rate is initialized to 0.0001, and the learning rate is reduced to the original 0.1 after 40 rounds.

[0061] Evaluation indicators:

[0062] AP, AP20: AP, AP20 are evaluation indicators in the COCO dataset, where AP is the average of the average precision (AP) under 10 intersection-over-union ratio thresholds ranging from 0.50 to 0.95, and AP20 is the average precision under the intersection-over-union ratio threshold of 0.20.

[0063] Recall rate: The ratio of the number of correctly predicted targets to the number of all true targets when the intersection-over-union ratio threshold is 0.20.

[0064] Accuracy: The ratio of the number of correctly predicted targets to the number of all predicted targets when the intersection-over-union ratio threshold is 0.20.

[0065] Experimental results:

[0066] Table 1 Comparison of detection performance and network scale of different methods

[0067]

[0068] The network size of the proposed method is small, the number of parameters is only 37.1K, and the number of floating-point operations is only 46.2M. Without the use of model compression operations such as network pruning and network quantization, the proposed method is 3 to 4 orders of magnitude lower than the current detection network in terms of the number of parameters and floating-point operations, and the detection performance of civil aircraft is close to that of the current detection network. The proposed method is inferred on the thermal infrared images of the Sustainable Development Science Satellite 1 containing real civil aircraft, and can effectively detect real civil aircraft, verifying the practicality of the proposed method.

[0069] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0070] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A lightweight detection method for optical image point targets, characterized in that: The following steps are involved: S1, preprocessing and channel expansion of the input single-channel or multi-channel optical image; S2, sparsely sample the pixels on the feature image after channel expansion to obtain query points and feature vectors; S3, calculating a low-dimensional query vector corresponding to the query point; S4, calculating the positions of several feature sampling points of the query point from the low-dimensional query vector, sampling several feature vectors from the feature image using bilinear interpolation, multiplying each sampled feature vector by the corresponding attention weight and summing them up to obtain an attention feature vector; S5, using the attention feature vector to update the feature vector of the query point; S6. Use the updated feature vector to predict the confidence of the existence of the target at the query point and the target rectangular prediction box.

2. The optical image point target lightweight detection method according to claim 1, characterized in that: The expression of the feature image F after channel expansion is: F = linear (I); linear represents the linear layer, in, H represents the image height, W represents the image width, and C represents the number of image channels.

3. The optical image point target lightweight detection method according to claim 1, characterized in that: In step S2, the specific implementation process of obtaining the query point includes: dividing the feature image into a number of non-overlapping image blocks, and selecting a pixel point with a relatively fixed position in each image block as a query point.

4. The optical image point target lightweight detection method according to claim 1, characterized in that: In step S3, the low-dimensional query vector expression is: in, c<C, c is the query vector dimension, q i,j is the query vector corresponding to the query point (i, j), i represents the vertical position, j represents the horizontal position, Represents the feature vector on the feature image calculated using sinusoidal coding The position encoding of , linear represents the linear layer.

5. The optical image point target lightweight detection method according to claim 4, characterized in that: In step S4, the attention feature vector a of the query point (i, j) is i,j The calculation formula is: in, M is the number of heads in the attention module, K is the number of sampling points of each attention head, m and k are the indexes of the attention head and sampling points, A m,k Represents the attention weight of each sampling point, Δi m,k and Δj m,k Indicates the sampling point relative to f i,j The position offset, g(F,Δi m,k ,Δj m,k ) represents the feature pixels sampled from the feature image F using bilinear interpolation.

6. The optical image point target lightweight detection method according to claim 5, characterized in that: Position offset Δi m,k and Δj m,k The calculation formula is:

7. The optical image point target lightweight detection method according to claim 5, characterized in that: A m,k The range is [0,1], satisfying 8. The optical image point target lightweight detection method according to claim 4, characterized in that: In step S5, the updated feature vector f′ of the query point i,j Expressed as: f′ i,j =norm(f i,j +a i,j ); norm represents the normalization layer.

9. The optical image point target lightweight detection method according to claim 4, characterized in that: In step S6, the calculation formula of the target rectangular prediction box b is: Among them, Sigmoid represents the Sigmoid activation function, MLP represents the multi-layer perceptron, and f′ i,j is the updated feature vector of the query point (i, j), x0 and y0 are the centers of the anchor box, and w0 and h0 are the width and height of the anchor box.

10. The optical image point target lightweight detection method according to claim 9, characterized in that: In step S6, the calculation formula of the target confidence α is: α = Sigmoid[MLP(f′ i,j )].

Images