A fine-grained aircraft target detection method based on scene-sample relationship prior

Abstract

The application relates to a fine-grained airplane target detection method based on a scene-sample relationship prior, which comprises the following steps: inputting pictures containing airplane samples into a backbone network to establish multi-scale feature maps; global position coding is established, and key region features on the feature maps are obtained through a region proposal network, and then boundary box regression and classification are performed; an attention matrix between a scene and an object is constructed, and the key region features added with the attention matrix are converted into a query vector, an index vector and a key value vector; the query vector, the index vector and the key value vector and the global position coding are used to obtain a relationship matrix between targets through a Transformer; and the key region features added with the relationship matrix are used to optimize the classification results of each boundary box. The application can realize fine-grained airplane target detection in a remote sensing image, enhance the detection capability for high-similarity samples, and improve the detection precision of airplane targets.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of aircraft target fine-grained detection, and particularly relates to a fine-grained aircraft target detection method based on scene-sample relationship prior. BACKGROUND

[0002] Aircraft target fine-grained detection is a computer graphics and computer vision technology, which aims to obtain the location and model of aircraft targets from input images. It allows us to learn the features of different models of aircraft through a deep learning network, correctly find these aircraft in the image, and distinguish their models, although these aircraft are very similar and difficult to distinguish with the naked eye.

[0003] Traditional target detection mainly relies on methods such as hand-designed features, sliding windows, classifiers, and target representation and description to obtain the location and category of the target. However, these methods are usually limited by the quality of the input image, the similarity of the samples, and the complexity of the scene, resulting in problems such as difficulty in obtaining the location of the target or misclassification of the category of the target. With the development of remote sensing technology, the resolution of ground object details in aerial images is getting higher and higher. This makes it possible to detect aircraft and identify fine-grained types. With the development of remote sensing technology and the rapid development of the field of computer vision, there is an increasing demand for higher precision target detection results.

[0004] Aircraft target fine-grained detection technology involves complex algorithms and models, such as one-stage detection methods based on convolutional neural networks (CNN) and two-stage methods based on region convolutional neural networks (RCNN). These methods can learn the features in the input image, infer the key regions in the image, and then classify and regress the key regions to obtain the target detection result. Although aircraft target fine-grained detection technology has made significant progress, there are still challenges in some applications, such as the possibility of inaccuracy when dealing with similar samples of different categories in complex scenes. Therefore, it is still important to improve and optimize aircraft target fine-grained detection technology, especially in the fields of military reconnaissance, aviation traffic, etc.

[0005] For aircraft target fine-grained target detection method research, the deficiencies of existing technology mainly manifest in the following aspects: (1) existing target detection methods only rely on key region features to obtain target location and category, ignoring scene prior information, and are difficult to learn the feature differences between similar aircraft categories; (2) existing target detection methods use CNN or RCNN to extract image features, ignoring the prior information provided by other aircraft samples in the same scene, and are difficult to capture long-distance sample correlation features. SUMMARY

[0006] In order to overcome the shortcomings of the prior art, the present application provides a fine-grained aircraft target detection method based on scene-sample relationship prior, which improves the training speed and inference efficiency of the model on the basis of the most advanced model, while maintaining a lower parameter amount of the neural network, and realizes fine-grained detection of aircraft targets.

[0007] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] A fine-grained aircraft target detection method based on scene-sample relationship prior comprises the following steps:

[0009] Step (1) inputting a picture containing an aircraft sample to a backbone network as an image, and establishing a multi-scale feature map by using a feature pyramid network;

[0010] Step (2) establishing a global position code for the multi-scale feature map established in step (1), and obtaining key region features of the feature map (these features refer to region features that may contain targets in the image) by a region proposal network, and sending the key region features into a regression network and a classification network to perform boundary box regression and classification of the key region;

[0011] Step (3) constructing an attention matrix between a scene and an object for the key region features obtained in step (2), and using a convolution network to convert the key region features with the attention matrix into a query vector, an index vector and a key value vector;

[0012] Step (4) obtaining a relationship matrix between targets by a Transformer neural network by using the global position code of step (2) for the query vector, the index vector and the key value vector of step (3);

[0013] Step (5) reclassifying the boundary boxes of the key region by using the key region features in step (2) for the relationship matrix of step (4), and optimizing the classification results of each boundary box.

[0014] Further, the step (1) comprises:

[0015] mapping the input image to a feature space by using a ResNet backbone network:

[0016] feature=ResNet(I)

[0017] wherein I is an input picture, usually with a size of HxWx3, wherein H and W are the length and width of the image. The input picture obtains the features of the entire picture by ResNet(I), which is an abstract representation of the input image, usually with a size of (H / 32)x(W / 32)x C, wherein C is the total channel number of the feature map. Then the feature map is sent into an FPN (feature pyramid network) to establish a multi-scale feature map:

[0018] {P2,P3,P4,P5}=FPN(feature)

[0019] Where P2, P3, P4, P5 are features of different scales.

[0020] Further, the step (2) comprises:

[0021] For each feature map, a position encoding PE is constructed:

[0022]

[0023]

[0024] Where i = 0, 1, …, C / 4-1 is the index of the feature map channel dimension, C is the total number of feature map channels, and pos is the result of the position coordinates (x, y) after encoding; the multi-scale feature map is obtained by RPN (region proposal network) to obtain the key region, and then the key region is sent into the regression and classification network to obtain the preliminary detection result:

[0025] ROI = RPN(P2, P3, P4, P5, P6)

[0026] pred_bboxes = reg(ROI)

[0027] pred_cls = cls(ROI)

[0028] Where ROI is the feature of the key region in the picture, reg is the bounding box regression network, cls is the bounding box classification network, pred_bboxes is the bounding box regression, and pred_cls is the classification result of the bounding box.

[0029] Further, the step (3) comprises:

[0030] A attention matrix M between the scene and the object is constructed:

[0031] M = Φ(S·F img T )·Ψ(F img ·F obj T )

[0032] Where Φ and Ψ represent linear transformation functions realized by 1x1 convolution, F img and F obj represent image features with position encoding and key region features with position encoding, and T is a transpose function.

[0033] Then, the query vector, index vector and key value vector are obtained by a convolution layer:

[0034] Q i = FFN1([M, F obj ])

[0035] K i = FFN2([M, F obj ])

[0036] V i = FFN3([M, F obj ])

[0037] where FFN1, FFN2, FFN3 are three fully connected layers, Q i is the query vector, K i is the index vector, V i is the key vector, i = 0, 1,..., n, and n is the number of key regions.

[0038] Further, the step (4) comprises:

[0039] Constructing the attention matrix between objects in the same scene:

[0040]

[0041] where Q i , K i , V i are the results obtained in step (3), D = (C + L) / h is the number of channels of the attention head, h is the number of each attention head, L is a hyperparameter, softmax() is an activation function, and head i represents the score of the i-th attention head, and there are n attention heads in total.

[0042] The obtained attention matrix is input into the classification network to obtain the optimized classification result of the detection frame:

[0043] pred_cls refine = FFN(F obj + [head1,..., head h ]W O )

[0044] where W O is the linear transformation weight of the attention head; pred_cls refine is the result of re-classification of the detection frame, which is an n x K matrix, where K is the number of aircraft categories.

[0045] Further, the step (5) comprises:

[0046] The primary detection frame and the classification result are respectively sent into a smooth L1 loss function and a focal loss function, the optimized classification result is sent into a cross entropy loss function, the sum of the three is taken as a total loss function, and the weight of the third branch is controlled by a parameter λ:

[0047]

[0048] L = L focal + L reg + λL ce

[0049] Wherein, when k is equal to the true value, Y k = 1, otherwise Y k = 0; L ce is a cross entropy loss function, L reg is a smooth absolute value loss function, and L focal is a class imbalance loss function.

[0050] Compared with the prior art, the present application has the beneficial effects that:

[0051] (1) The lightweight scene context attention module is adopted in the present application, which expands the receptive field of the model and strengthens the utilization of the prior information of the remote sensing scene by the neural network on the basis of maintaining a low parameter amount of the algorithm.

[0052] (2) The present application uses the Transformer to extract the relationship between the targets in the scene, and optimizes the target classification result by using the correlation between the targets in the scene on the basis of the prior information of the scene, thereby significantly reducing the number of misclassified samples.

[0053] In summary, the method adopted in the present application has a simple principle and can achieve the purpose of obtaining the position and type of the aircraft target in the remote sensing image. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a flow chart of a fine-grained aircraft target detection method based on scene-sample relationship prior according to the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0056] As shown in the drawings, Figure 1 a fine-grained aircraft target detection method based on scene-sample relationship prior according to the present application is specifically implemented as follows:

[0057] Step 1, input the picture containing the aircraft sample as an image to the backbone network, and use the feature pyramid network to establish a multi-scale feature map. According to the size of the input image and the size of the target, a multi-scale feature map is constructed to express the image features with fewer parameters:

[0058] Use the ResNet backbone network to map the input image to the feature space:

[0059] feature=ResNet(I)

[0060] Where I is the input image, and the feature of the entire picture is obtained through ResNet(I), which is an abstract representation of the input image, and then sent to FPN (feature pyramid network) to establish a multi-scale feature map:

[0061] {P2,P3,P4,P5}=FPN(feature)

[0062] Where {P2,P3,P4,P5} are features of different scales (W / 4×H / 4,W / 8×H / 8,W / 16×H / 16,W / 32×H / 32), and W and H are the length and width of the original image.

[0063] The high-dimensional feature map has fewer parameters and can efficiently describe the semantic information in the remote sensing image, but it may ignore the small-scale target features; the low-dimensional feature map contains small-scale target features, but has a large number of parameters and low computational efficiency. FPN uses a multi-scale feature fusion method to significantly improve the scale robustness of feature expression without significantly increasing the computational complexity.

[0064] Step 2, use the multi-scale feature map to establish a position encoding PE at each feature point to provide a unique representation for each position so that the subsequent Transformer (multi-layer self-attention network) can distinguish the information of different positions:

[0065]

[0066]

[0067] Where i=0,1,...,C / 4-1 is the index of the channel dimension of the feature map, C is the total number of channels of the feature map, and pos is the result of the position coordinates (x,y) after encoding. The multi-scale feature map is obtained through RPN (region proposal network) to obtain the key area, and then the key area is sent to the regression and classification network to obtain the preliminary detection result:

[0068] ROI=RPN(P2,P3,P4,P5,P6)

[0069] pred_bboxes = reg(ROI)

[0070] pred_cls = cls(ROI)

[0071] where ROI is the feature of the key region in the picture, P6 is the feature, reg is the bounding box regression network, cls is the bounding box classification network, pred_bboxes is the regression result of the bounding box, and pred_cls is the classification result of the bounding box. Usually, this is achieved by adding a fully connected layer (or multiple fully connected layers) at the last layer of the network, and the output of the fully connected layer is the bounding box of the target candidate region and the probability distribution of the candidate region class.

[0072] Step 3, using a set of learnable scene priori representation S and the obtained image feature F img , and the candidate region feature F obj , to construct the attention matrix M between the scene and the object, and capture the overall context feature relationship of the scene:

[0073] M = Φ(S·F img T )·Ψ(F img ·F obj T )

[0074] where Φ and Ψ represent linear transformation functions realized by 1×1 convolution, F img and F obj represent image features with position encoding and key region features with position encoding, and the superscript T represents the transpose of the matrix.

[0075] Then, through a fully connected network, the key region feature with the attention matrix is converted into a one-dimensional query vector, index vector and key value vector:

[0076] Q i = FFN1([M,F obj ])

[0077] K i = FFN2([M,F obj ])

[0078] V i = FFN3([M,F obj ])

[0079] where FFN1, FFN2, FFN3 are three different learnable fully connected layers, Q i is the query vector, K i is the index vector, and V iThe key value vector is i=0, 1,...,n, n is the number of key regions, and the FFN forward calculation is only performed once in the entire detection process, thereby significantly reducing the calculation amount and reducing the training time of the neural network.

[0080] Step 4, using the query vector, the index vector and the key value vector, the attention matrix between the target and the target in the same scene is obtained through the Transformer neural network, and the method is as follows:

[0081]

[0082] Wherein, Q i ,K i ,V i is the result obtained in step (3), D=(C+L) / h is the number of channels of the attention head, h is the number of each attention head, L is a hyperparameter, softmax() is an activation function, head i represents the score of the i-th attention head, and there are n attention heads in total.

[0083] Then, the obtained attention matrix is sent into the classification network to obtain the optimized classification result of the detection frame:

[0084] pred_cls refine =FFN(F obj +[head1,...,head h ]W O )

[0085] Wherein, W O is the linear transformation weight of the attention head; pred_cls refine is the reclassification result of the detection frame, and the size is n×K matrix, wherein K is the number of aircraft categories.

[0086] Step 5, according to the attention matrix of step (4), the key region feature ROI in step (2) is used to reclassify the key region bounding box, and the classification result of each bounding box is optimized, and the method is as follows:

[0087] The preliminary detection frame and the classification result are respectively sent into the smooth absolute value loss function and the category imbalance loss function, the optimized classification result is sent into the cross entropy loss function, the sum of the three is taken as the total loss function, and the weight of the third branch is controlled by the parameter λ.

[0088]

[0089] L=L focal +L reg +λL ce

[0090] where Y = 1 when k is equal to the true value, otherwise Y = 0; L k k ce is a cross-entropy loss function, L reg is a smooth absolute value loss function, L focal is a class imbalance loss function.

[0091] Therefore, the application can be used for multi-class aircraft targets in remote sensing images, uses the relationship between the scene and the target as prior knowledge, improves the detection accuracy of the neural network, and obtains high-quality fine-grained detection results.

[0092] The contents not described in detail in the specification of the application belong to the common knowledge of the skilled in the art. Although the above describes the specific embodiments of the application, so that the skilled in the art can understand the application, and it should be clear that the application is not limited to the scope of the specific embodiments, and for the skilled in the art, as long as various changes are within the spirit and scope of the application defined and determined by the appended claims, all the application and creation using the concept of the application are included in the protection.​​

Claims

1. A fine-grained aircraft target detection method based on scene-sample relationship prior, characterized in that, Includes the following steps: Step (1) Input the image containing the aircraft sample into the backbone network and use the feature pyramid network to build a multi-scale feature map; Step (2) Establish global location encoding for the multi-scale feature map established in step (1), and obtain the key region features of the feature map through the region proposal network, and send them into the regression network and classification network to perform bounding box regression and classification of the key regions; the key region features refer to the region features of the image containing the target; Step (3) Based on the key region features obtained in step (2), construct an attention matrix between the scene and the object, and use a convolutional network to convert the key region features added to the attention matrix into query vectors, index vectors and key value vectors. Step (4) Using the query vector, index vector and key value vector from step (3), the global position encoding from step (2) is used to obtain the relationship matrix between targets through a multi-layer self-attention network; Step (5) Based on the relation matrix in step (4), the key region features in step (2) are used to reclassify the bounding boxes of the key regions and optimize the classification results of each bounding box.

2. The fine-grained aircraft target detection method based on scene-sample relationship prior as described in claim 1, characterized in that: Step (1) includes: Using the ResNet backbone network, the input image is mapped to the feature space: ； in, It is the result of mapping the input image to the feature space. The input image has a size of [size missing]. Where H and W are the length and width of the image; the input image is processed through... Extract the features of the entire image; it is an abstract representation of the input image, with a size of [size missing]. Where C is the total number of channels in the feature map; then the feature map is fed into a feature pyramid network to build a multi-scale feature map: ； in, Features at different scales.

3. The fine-grained aircraft target detection method based on scene-sample relationship prior as described in claim 2, characterized in that: Step (2) includes: Construct a positional encoding (PE) for each feature map: ； in, is the index of the feature map channel dimension, and C is the total number of channels in the feature map. Position coordinates The results after encoding; the multi-scale feature map is passed through a region proposal network to obtain key region features, and then the key region features are fed into a regression and classification network to obtain preliminary detection results: ； in, Features of key areas in the image. For bounding box regression network, For bounding box classification networks, The regression yields the bounding box. This represents the classification result of the bounding box.

4. The fine-grained aircraft target detection method based on scene-sample relationship prior as described in claim 1, characterized in that: Step (3) includes: Construct the attention matrix M between the scene and the objects: ； in, and This represents a linear transformation function implemented by a 1×1 convolution. and Indicates image features with added positional coding and key region features with added positional coding, superscript To represent the transpose of a matrix; Then, the query vector, index vector, and key value vector are obtained through convolutional layers: ； in, It consists of three fully connected layers. For query vector, For index vectors, For key value vectors, , The number of key areas.

5. The fine-grained aircraft target detection method based on scene-sample relationship prior as described in claim 4, characterized in that: Step (4) includes: Construct the attention matrix between objects in the same scene: ； in, To obtain the result of step (3), It is the number of channels in the attention head. The number of attention heads per attention head It's a hyperparameter. It is an activation function. This represents the score of the i-th attention head, in total One point of attention; The obtained attention matrix is ​​fed into the classification network to obtain the classification result after the detection boxes are optimized: ； in, It is a linear transformation weight of the attention head; This is the result of the reclassification of the detection box, with a size of The matrix, where This represents the number of aircraft categories.

6. The fine-grained aircraft target detection method based on scene-sample relationship prior as described in claim 5, characterized in that: Step (5) includes: The initial detection boxes and classification results are fed into the smoothed L1 loss function and the focal loss function, respectively. The optimized classification result is then fed into the cross-entropy loss function. The sum of these three loss functions is used as the total loss function, and parameters are used to calculate the final loss. Controlling the weight of the third branch: ； in, For coefficients, when When equal to the truth value ,otherwise k represents the serial number; Let cross-entropy be the loss function. For smoothing absolute value loss function, This is the class imbalance loss function.

Images