A long-tail distribution fine-grained aircraft recognition method

Abstract

The present invention discloses a long-tail distribution fine-grained aircraft recognition method, comprising: designing a dynamic feature hallucination module, synthesizing hallucination samples based on the dynamic feature hallucination module to introduce additional data variance, enhance the representation of tail categories in feature space, and obtain long-tail distribution tail category sample data; designing a contrast learning module, extracting discriminant features of the long-tail distribution tail category sample data by maximizing inter-class distance and minimizing intra-class distance based on the contrast learning module, detecting fine-grained targets of aircraft targets in remote sensing images according to the discriminant features, and obtaining detection results. The present invention introduces additional data variance by synthesizing hallucination samples through the dynamic feature hallucination module, enhances the representation of tail categories in feature space, extracts discriminant features by maximizing inter-class distance and minimizing intra-class distance, and realizes accurate recognition of aircraft.

Description

Technical Field

[0001] The invention belongs to the field of optical remote sensing aircraft target detection, and in particular relates to a long-tail distribution fine-grained aircraft recognition method. Background Art

[0002] Automatic analysis and understanding of remote sensing images have important application scenarios in reality, such as disaster detection, urban monitoring, traffic control, etc. in the civil field, strategic deployment and earth observation in the military field, etc. As an important application scenario of intelligent understanding of remote sensing images, intelligent detection of aircraft in remote sensing images aims to accurately locate and identify aircraft targets, which has important research significance for airport flight monitoring.

[0003] In recent years, with the development of deep learning theory and computing resources, methods based on convolutional neural networks (CNNs) have made great progress in remote sensing aircraft target detection. These methods can be summarized as one-stage methods and two-stage methods based on whether they have a region generation network (RPN). One-stage detectors directly perform bounding box regression and target classification, and two-stage detectors require two key steps: the first step is to generate proposals through a region generation network, and the second step is to refine the proposals for regression and classification. Remote sensing scene images are usually captured from satellites with a wide field of view. Due to the change in acquisition height, objects in different remote sensing images have different scales, and targets are usually densely distributed, with uncertain target orientations. In order to detect targets in remote sensing images in a targeted manner, researchers have proposed many directional target detectors based on one-stage and two-stage target detectors in natural images. Typical object-oriented detectors based on one-stage detectors are single-category alignment networks S2A-Net, R3Det, ​​SCRDet and other similar detectors. Two-stage object-oriented detectors have high detection accuracy, but slow detection speed. This type of oriented detectors such as RoI Transformer, ReDet and OrientedRCNN are developed from two-stage detectors.

[0004] However, convolutional neural network (CNN) based detectors still have difficulty in accurately identifying aircraft in fine-grained long-tailed distribution aircraft detection tasks due to the following problems.

[0005] 1) Unbalanced category distribution: Different categories of aircraft appear at different frequencies in real-world scenarios, so aircraft datasets usually show imbalanced distribution between classes. Due to the small number of tail category samples, CNN cannot extract rich tail category features, so it will overfit the tail category with insufficient training examples, resulting in poor detection accuracy.

[0006] 2) Small inter-class differences and large intra-class differences: Different types of aircraft at the model level show slight inter-class differences. At the same time, due to different coatings on the same type of aircraft and geometric distortion of remote sensing images, the same type of aircraft has large intra-class differences. Therefore, general object detectors cannot learn distinguishable features to correctly classify aircraft, resulting in a decrease in fine-grained aircraft recognition performance.

[0007] The above two problems affect each other. The small difference between classes and the large difference within classes pose a major challenge to the detector in the process of learning discriminative detail features. The imbalanced distribution of categories makes it more difficult for the training process to provide enough data for feature learning of rare classes, which seriously degrades the detection performance. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a long-tail distribution fine-grained aircraft recognition method, comprising:

[0009] Designing a dynamic feature hallucination module, synthesizing hallucination samples based on the dynamic feature hallucination module to introduce additional data variance, enhance the representation of tail categories in the feature space, and obtain long-tail distribution tail category sample data;

[0010] A contrastive learning module is designed. Based on the contrastive learning module, discriminant features of the tail category sample data of the long-tail distribution are extracted by maximizing the inter-class distance and minimizing the intra-class distance. Fine-grained targets of aircraft targets in remote sensing images are detected according to the discriminant features to obtain detection results.

[0011] Preferably, the process of introducing additional data variance by synthesizing hallucinated samples based on the dynamic feature hallucination module to enhance the representation of tail categories in the feature space comprises:

[0012] Capturing the feature distribution of each category in real time based on the dynamic feature hallucination module, and then generating training features of the selected category according to the number of tail categories;

[0013] Synthesizing hallucinated features based on the training features of the selected categories enhances the representation of the tail categories in the feature space.

[0014] Preferably, the process of capturing the feature distribution of each category in real time based on the dynamic feature hallucination module and then generating the training features of the selected category according to the number of tail categories includes:

[0015] Construct a baseline OrientedRCNN model consisting of a backbone network Resnet50, FPN, a region proposal network RPN, and a detection head, randomly obtain images as input through the backbone network Resnet50, and generate corresponding feature maps;

[0016] The feature map is convolved by the region proposal network RPN, and the dynamic feature hallucination module generates a fixed number of region proposals that overlap with the GT bounding box using a non-learnable bounding box generator;

[0017] Next, the RRoIAlign layer pools the region proposal into a fixed size according to the feature map, and then encodes the region proposal through the RoI feature extractor to obtain d-dimensional RoI features;

[0018] Finally, the RoI features are fed into the classification, regression and CLM modules, and the CLM module increases the fine-grained object class distance by comparing prototype and sample features.

[0019] Preferably, the process of the dynamic feature hallucination module using a non-learnable bounding box generator to generate a fixed number of region proposals that overlap with the GT bounding box comprises:

[0020] The bounding box generator uses coordinate operations to randomly convert the GT bounding boxes in the image into valid proposals. For a batch of training images I, the box coordinates of each instance in the training image I are converted by the designed box generator. As input, dense proposals are generated for each instance in the training image I, where (x1, y1) and (x2, y2) represent the coordinates of the upper left corner and the lower right corner, respectively.

[0021] Preferably, the box coordinates of each instance in the training image I are converted by a designed box generator As input, the formula for generating dense proposals for each instance in the training image I is:

[0022] (1)

[0023] in is randomly selected, w and h are the frame width and frame height respectively, .

[0024] Preferably, the RRoIAlign layer pools the region proposal into a fixed size according to the feature map, and then encodes the region proposal through a RoI feature extractor, and the process of obtaining a d-dimensional RoI feature further includes:

[0025] Encoding the region proposal into RoI features based on the RoIAlign layer and the RoI feature extractor, collecting online feature distribution of prototypes and variances;

[0026] Calculate the mean of the features of each category i based on the dynamic feature hallucination module and variance , and then use the defined exponential moving average function to change the corresponding prototype and variance ;

[0027] Finally, a sampling probability is assigned to each category i based on the dynamic feature hallucination module To ensure the prominence of the tail category, the sampling probability With long-term indicators Inversely proportional;

[0028] Using the sampling probability ,The dynamic feature hallucination module randomly selects c categories, continuously updates the feature distribution through the re-parameterization method, and generates m hallucination features for each category i.

[0029] Preferably, the exponential moving average function expression defined is:

[0030]

[0031] in, represents the weight factor;

[0032] The sampling probability The formula expression is:

[0033]

[0034] in is the frequency of instances of each category;

[0035]

[0036] Preferably, the contrast loss function of the contrast learning module is an InfoNCE+ loss function based on contrast predictive coding, and the formula expression is:

[0037]

[0038] Among them, the molecule is the current sample set The representation vector of the positive sample, Indicates that in the same batch The number of positive samples, represents the function used to calculate the similarity between two samples; is the temperature hyperparameter; for the denominator, Represents the total number of valid samples, including positive and negative samples in the current batch.

[0039] Preferably, the selection of the positive samples and negative samples is based on the sampling frequency, using a ratio Constrain the number of positive and negative samples in the current sample; assuming the batch size is , the number of foreground regions is ; Traverse samples, for each ,in , Represents the category label, and based on the remaining The category labels of samples are positive samples and negative samples, and the formula is:

[0040]

[0041] Preferably, based on the sampling frequency, a ratio The process of constraining the number of positive and negative samples in the current sample is based on the following rules:

[0042] If the length of pos_set is less than 1 or the length of neg_set is less than 3, the current RoI feature is not included in the loss calculation;

[0043] The number of positive samples selected from pos_set does not exceed , is the maximum number of positive samples used to constrain;

[0044] at last, The loss is calculated and added to the total loss In the calculation of:

[0045]

[0046] in, and represents regression and classification losses, Indicates InfoNCE+ loss.

[0047] Compared with the prior art, the present invention has the following advantages and technical effects:

[0048] The dynamic feature hallucination module (FHM) designed in the present invention introduces additional data variance by synthesizing hallucinated samples, thereby enhancing the representation of tail categories in the feature space.

[0049] The contrastive learning module designed by the present invention extracts discriminative features by maximizing the inter-class distance and minimizing the intra-class distance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0051] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0052] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0053] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0054] With the continuous development of satellite technology, satellite imaging capabilities have gradually improved, providing a data basis for remote sensing image target detection tasks. With the continuous development of deep learning, remote sensing image target detection based on deep learning has made great progress. However, facing the class imbalance and fine-grained problems of data in real scenes, methods based on deep learning are extremely challenging. To this end, this embodiment proposes a long-tail distribution aircraft fine-grained recognition method based on dynamic feature synthesis and contrastive learning. First, in view of the small number of sample data in the tail category of the long-tail distribution, a dynamic feature hallucination module (FHM) is proposed. By synthesizing hallucinated samples to introduce additional data variance, the representation of the tail category in the feature space is enhanced; secondly, a contrastive learning module is designed for large intra-class differences and small inter-class differences. The discriminant features are extracted by maximizing the inter-class distance and minimizing the intra-class distance, which is used for fine-grained target detection of aircraft targets in remote sensing images.

[0055] like Figure 1 As shown, this embodiment provides a long-tail distribution fine-grained aircraft recognition method, including:

[0056] Designing a dynamic feature hallucination module, synthesizing hallucination samples based on the dynamic feature hallucination module to introduce additional data variance, enhance the representation of tail categories in the feature space, and obtain long-tail distribution tail category sample data;

[0057] A contrastive learning module is designed. Based on the contrastive learning module, discriminant features of the tail category sample data of the long-tail distribution are extracted by maximizing the inter-class distance and minimizing the intra-class distance. Fine-grained targets of aircraft targets in remote sensing images are detected according to the discriminant features to obtain detection results.

[0058] The extreme scarcity of tail training samples in long-tail distribution data hinders the classifier's ability to identify this class. Most previous studies have used various resampling methods to increase the number of training samples for the tail category. However, the sampling method only increases the number of samples without increasing the diversity of samples. This method, which ignores the inclusion of additional data changes, hinders further improvement of the tail category.

[0059] To overcome this problem, a simple yet effective Feature Hallucination Module (FHM) is proposed to enhance the representation in the feature space, especially the representation of the tail categories, by synthesizing hallucinated features to improve the diversity of the data. FHM first captures the feature distribution of each category in real time, and then generates training features for the selected category based on the number of tail categories.

[0060] Furthermore, the process of introducing additional data variance by synthesizing hallucination samples based on the dynamic feature hallucination module and enhancing the representation of the tail categories in the feature space includes:

[0061] Capturing the feature distribution of each category in real time based on the dynamic feature hallucination module, and then generating training features of the selected category according to the number of tail categories;

[0062] Synthesizing hallucinated features based on the training features of the selected categories enhances the representation of the tail categories in the feature space.

[0063] The process of capturing the feature distribution of each category in real time based on the dynamic feature hallucination module and then generating the training features of the selected category according to the number of tail categories includes:

[0064] Construct a baseline Oriented RCNN model, which consists of a backbone network Resnet50, FPN, a region proposal network RPN, and a detection head. The backbone network Resnet50 randomly obtains images as input and generates corresponding feature maps; the backbone network Resnet50 randomly obtains images as input and generates corresponding feature maps. The region proposal network (RPN) performs convolution operations on the feature map. At the same time, the FHM uses a non-learnable bounding box generator to generate region proposals with a large overlap with the GT bounding box, generating a fixed number of region proposals. Next, the RRoIAlign layer first pools these region proposals to a fixed size according to the feature map, and then encodes them into d-dimensional RoI features h through a RoI feature extractor composed of two fully connected layers (FC). Finally, these RoI features are fed into the classification, regression, and CLM modules. The CLM module increases the fine-grained object class distance by comparing prototype and sample features.

[0065] Specifically, FHM uses a non-learnable bounding box generator to generate region proposals with a large overlap with the GT bounding boxes. Compared with RPN, the bounding box generator uses coordinate operations to randomly transform the GT bounding boxes in the image into valid proposals. That is, for a batch of training images I, the designed box generator converts the box coordinates of each instance in I As input, (x1, y1), (x2, y2) represent the coordinates of the upper left corner and the lower right corner respectively. Then, it generates 16 dense proposals for each instance in I, whose positions are slightly offset as follows:

[0066] (1)

[0067] in is randomly selected, w and h are the frame width and frame height respectively, that is, Obviously, each proposal is slightly different from the other 15 proposals in terms of foreground and background, which is beneficial to increase the sample variance.

[0068] After that, the subsequent RoIAlign layer and RoI feature extractor encode the region proposal into RoI features, not for classification and regression, but to collect online feature distribution, which includes prototype and variance. Specifically, FHM calculates the mean of the features for each category i and variance , and then use the exponential moving average function defined in equations (2) and (3) to change the corresponding prototype and variance .

[0069]

[0070] in, represents the weight factor, the value of β determines the previous value (μ i and σ i ) and the new value (u i and v i ) compared with the weight;

[0071] Finally, FHM is achieved by assigning a sampling probability to each class i To ensure the prominence of the tail category, the sampling probability With long-term indicators Inversely proportional:

[0072]

[0073] in is the frequency of instances of each category.

[0074] Using the above sampling probability, FHM randomly selects c categories and continuously updates the feature distribution through the reparameterization method to generate m hallucination features for each category i:

[0075]

[0076] Therefore, FHM dynamically enhances data diversity by introducing novel hallucinated features, especially for tail categories, thus alleviating the problem of insufficient features for tail categories.

[0077] Furthermore, the contrastive learning method benefits from the advantage of expanding the difference between classes and reducing the difference within classes, and can improve the recognition ability of the detector between classes with small differences. Therefore, this embodiment proposes a CLM module to deal with the problem of difference between classes.

[0078] Unlike unsupervised / self-supervised contrastive learning tasks, fine-grained aircraft object detection has true value boxes during the training phase, so there is no need to generate positive samples by rotating, scaling, and cropping the original samples. Therefore, the key point of introducing contrastive learning methods into aircraft detection tasks lies in the selection of contrastive loss functions and the selection of positive and negative samples. In this embodiment, the InfoNCE+ loss function based on contrastive predictive coding is used to optimize the feature distribution between different categories. The formula is as follows:

[0079]

[0080] Among them, the molecule is the current sample set The representation vector of the positive sample, Indicates that in the same batch The number of positive samples. Represents the function used to calculate the similarity between two samples. is a temperature hyperparameter set to 0.1. For the denominator, Represents the total number of valid samples, including positive and negative samples in the current batch.

[0081] The selection of positive and negative samples takes into account the sampling frequency of 1:3 and adopts the ratio To constrain the number of positive and negative samples of the current sample. Specifically, assuming the batch size is , the number of foreground regions is . Traverse this samples, for each ,in , Represents the category label, and based on the remaining The category labels of samples select positive samples and negative samples:

[0082]

[0083] In proportion After obtaining positive and negative samples from formula (7), the following rules should be followed:

[0084] 1) If the length of pos_set is less than 1 or the length of neg_set is less than 3, the current RoI should not be included in the loss calculation.

[0085] 2) The number of positive samples selected from pos_set should not exceed , Used to constrain the maximum number of positive samples.

[0086] Again, batches + Losses can be calculated and added to the total loss In the calculation of:

[0087]

[0088] in, and denotes the regression and classification losses, which is consistent with the original design of the detector used. express loss. α is a hyperparameter that represents Add to The weight when , the weight is set to 0.2.

[0089] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A long-tail distribution fine-grained aircraft recognition method, characterized in that: include: Designing a dynamic feature hallucination module, synthesizing hallucination samples based on the dynamic feature hallucination module to introduce additional data variance, enhance the representation of tail categories in the feature space, and obtain long-tail distribution tail category sample data; Design a contrastive learning module, extract discriminant features of the sample data of the tail category of the long-tail distribution by maximizing the inter-class distance and minimizing the intra-class distance based on the contrastive learning module, detect fine-grained targets of aircraft targets in remote sensing images according to the discriminant features, and obtain detection results; The process of synthesizing hallucinated samples based on the dynamic feature hallucination module to introduce additional data variance and enhance the representation of the tail categories in the feature space includes: Capturing the feature distribution of each category in real time based on the dynamic feature hallucination module, and then generating training features of the selected category according to the number of tail categories; Synthesizing hallucinated features based on the training features of the selected categories to enhance the representation of the tail categories in the feature space; The process of capturing the feature distribution of each category in real time based on the dynamic feature hallucination module and then generating the training features of the selected category according to the number of tail categories includes: Construct a baseline Oriented RCNN model consisting of a backbone network Resnet50, FPN, a region proposal network RPN, and a detection head, randomly obtain images as input through the backbone network Resnet50, and generate corresponding feature maps; The feature map is convolved by the region proposal network RPN, and the dynamic feature hallucination module generates a fixed number of region proposals that overlap with the GT bounding box using a non-learnable bounding box generator; Next, the RRoIAlign layer pools the region proposal into a fixed size according to the feature map, and then encodes the region proposal through the RoI feature extractor to obtain d-dimensional RoI features; Finally, the RoI features are fed into the classification, regression and CLM modules. The CLM module increases the fine-grained distance between target classes by comparing the features of positive and negative samples. The process of the dynamic feature hallucination module using a non-learnable bounding box generator to generate a fixed number of region proposals that overlap with the GT bounding box includes: The bounding box generator uses coordinate operations to randomly convert the GT bounding boxes in the image into valid proposals. For a batch of training images I, the box coordinates of each instance in the training image I are converted by the designed box generator. As input, dense proposals are generated for each instance in the training image I, where (x1, y1) and (x2, y2) represent the coordinates of the upper left corner and the lower right corner, respectively.

2. The long-tail distribution fine-grained aircraft identification method according to claim 1 is characterized in that: The box coordinates of each instance in the training image I are converted by the designed box generator As input, the formula for generating dense proposals for each instance in the training image I is: (1) in is randomly selected, w and h are the frame width and frame height respectively, .

3. The long-tail distribution fine-grained aircraft identification method according to claim 1 is characterized in that: The RRoIAlign layer pools the region proposal into a fixed size according to the feature map, and then encodes the region proposal through the RoI feature extractor to obtain the d-dimensional RoI feature. The process also includes: Encoding the region proposal into RoI features based on the RoIAlign layer and the RoI feature extractor, collecting online feature distribution of prototypes and variances; Calculate the mean of the features of each category i based on the dynamic feature hallucination module and variance , and then use the defined exponential moving average function to change the corresponding prototype and variance ; Finally, a sampling probability is assigned to each category i based on the dynamic feature hallucination module To ensure the prominence of the tail category, the sampling probability With long-term indicators Inversely proportional; Using the sampling probability ,The dynamic feature hallucination module randomly selects c categories, continuously updates the feature distribution through the re-parameterization method, and generates m hallucination features for each category i.

4. The long-tail distribution fine-grained aircraft identification method according to claim 3 is characterized in that: The exponential moving average function expression defined is: （2） （3） in, represents the weight factor; The sampling probability The formula expression is: （4） in is the frequency of instances of each category; （5）。 5. The long-tail distribution fine-grained aircraft identification method according to claim 1 is characterized in that: The contrast loss function of the contrast learning module is the InfoNCE+ loss function based on contrast prediction coding, and the formula expression is: （6） Among them, the molecule is the current sample set The representation vector of the positive sample, Indicates that in the same batch The number of positive samples, represents the function used to calculate the similarity between two samples; is the temperature hyperparameter; for the denominator, Represents the total number of valid samples, including positive and negative samples in the current batch.

6. The long-tail distribution fine-grained aircraft identification method according to claim 5 is characterized in that: The selection of positive and negative samples is based on the sampling frequency, using a ratio Constrain the number of positive and negative samples in the current sample; assuming the batch size is , the number of foreground regions is ; Traverse samples, for each ,in , Represents the category label, and based on the remaining The category labels of samples are positive samples and negative samples, and the formula is: （7）。 7. The long-tail distribution fine-grained aircraft identification method according to claim 6 is characterized in that: Based on the sampling frequency, the ratio The process of constraining the number of positive and negative samples in the current sample is based on the following rules: If the length of pos_set is less than 1 or the length of neg_set is less than 3, the current RoI feature is not included in the loss calculation; The number of positive samples selected from pos_set does not exceed , is the maximum number of positive samples used to constrain; at last, The loss is calculated and added to the total loss In the calculation of: in, and represents regression and classification losses, Indicates InfoNCE+ loss.

Images