A ViT-based aerial image recognition method

By using ViT model in aerial image recognition combined with data augmentation and label smoothing technology, the problems of small data volume and noise samples in aerial image recognition are solved, the recognition accuracy and model scalability are improved, and more accurate aerial images are achieved Identify.

CN114842343BActive Publication Date: 2024-11-01WUHAN UNIV OF TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210541111.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-17
Publication Date
2024-11-01
Estimated Expiration
2042-05-17

AI Technical Summary

Technical Problem

Aerial image recognition faces the problems of small data volume, common noise samples and difficult samples. The prior art is difficult to effectively utilize the internal information of aerial images, resulting in insufficient recognition accuracy.

Method used

ViT-based aerial image recognition method is adopted, and ViT is used as an image feature encoder, combining online data enhancement and distinctive label smoothing, and the model is trained using cross-entropy loss function and distinctive contrast loss function to avoid overfitting.

Benefits of technology

It improves the accuracy of aerial image recognition, enhances the scalability and feature capture capabilities of the model, avoids overfitting of noise data, and achieves more accurate recognition results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114842343B_ABST
    Figure CN114842343B_ABST
Patent Text Reader

Abstract

The present invention discloses an aerial image recognition method based on ViT, comprising the steps of: S1, collecting aerial image data sets, constructing training sets, verification sets and test sets; S2, expanding the data volume of the training sets; S3, constructing an aerial image recognition model based on ViT; S4, inputting the expanded training set into the recognition model, performing discriminative label smoothing on the labels corresponding to the images, and simultaneously adopting a cross entropy loss function and a discriminative contrast loss function to train the model, updating the recognition model through a back propagation algorithm, and selecting the optimal aerial image recognition model; S5, testing the recognition performance of the model through a test set. The present invention obtains an aerial image recognition model based on ViT with stronger feature learning ability by performing discriminative label smoothing on the labels corresponding to the images, and simultaneously adopting a cross entropy loss function and a discriminative contrast loss function to supervise the training process of the model. The model has the advantages of high recognition rate and strong scalability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning algorithms and image processing, and in particular to an aerial image recognition method based on ViT. Background Art

[0002] Aerial image recognition refers to identifying the category of an aerial image given one. With the increasing maturity of aviation technology, the resolution of aerial images is increasing, and aerial images are playing an increasingly important role in people's daily lives. Tasks such as natural disaster detection, urban planning, resource exploration, and thematic map production are inseparable from aerial image recognition, so accurate recognition of aerial images is of great value.

[0003] Although there is a large amount of aerial image data, the number of datasets available for model training is small and the quality is not high. The labeled datasets are even rarer, and the problems of noisy samples and difficult samples are common. In addition, aerial images are mostly overhead images, with the characteristics of wide imaging range, large scale changes, and sparse changes in objects in the scene. Therefore, compared with natural images, aerial image recognition has the difficulties of small data volume and complex background.

[0004] At present, most solutions to the above problems are centered around building targeted lightweight deep learning algorithms, which have not been extended to more diverse aerial images and have limitations. In addition, most of these methods use the cross entropy loss of learning label information to supervise the model, without considering the internal information of the aerial image itself. Summary of the invention

[0005] In view of the shortcomings of the background technology, the present invention proposes an aerial image recognition method based on ViT, which utilizes the advantages of ViT (Vision Transformer) in capturing long-distance dependencies and dynamic adaptive modeling capabilities, uses ViT as the feature encoder of the image to capture significant semantic features, and makes improvements based on ViT so that it can make full use of limited aerial image data for training and avoid overfitting the noise in the image.

[0006] To achieve the above object, the present invention designs an aerial image recognition method based on ViT, which is special in that the method comprises the following steps:

[0007] S1) Collect the aerial image dataset and obtain the required original aerial image x i and its corresponding category label y i , divide the training set, validation set and test set into proportional quantities, which are used for subsequent model training, validation and evaluation respectively, where the training set is recorded as B is the number of images in the training set;

[0008] S2) performing online data enhancement on the training set images so that each image in the training set generates M different enhanced images. The number of images in the training set after the enhancement is B*M, which is recorded as

[0009] S3) construct an aerial image recognition model based on ViT;

[0010] S4) the training set The image is input into the ViT-based aerial image recognition model, the labels corresponding to the images are discriminatively smoothed, and the model is trained by using a cross entropy loss function and a discriminative contrast loss function, the recognition model is updated by a back propagation algorithm, and the optimal aerial image recognition model is selected by using the validation set in step S1);

[0011] S5) using the test set of step S1) to test the recognition performance of the aerial image recognition model to obtain a final model recognition accuracy rate; when the model recognition accuracy rate reaches a set threshold, the image to be recognized is input into the aerial image recognition model for recognition; otherwise, returning to step S3) until the model recognition accuracy rate reaches the set threshold.

[0012] Preferably, in step S2), the input image is randomly cropped to 224*224 pixels and then randomly flipped horizontally, and then the image is enhanced using an image enhancement strategy, and finally an expanded training set is obtained, which is recorded as

[0013] Preferably, the image enhancement strategy in step S2) includes one or more combinations of the following operations: normalizing the image, performing random color distortion and Gaussian blur in sequence, automatic enhancement, random enhancement, randomly selecting an image enhancement operation each time, then randomly determining its enhancement amplitude, and enhancing the image, and randomly erasing a rectangular area from the image without changing the original label of the image.

[0014] Preferably, the ViT-based aerial image recognition model in step S3) is composed of an encoder F(·), a classification head G(·) and a projection head P(·) used only in the training phase:

[0015] The encoder F(·) is composed of ViT pre-trained on the dataset and is used to learn and encode the global features of the image. In the input feature encoder F(·), the first token of the encoder is used as The global feature representation h i ;

[0016] The classification head G(·) consists of MLP layers, and its structure is fully connected layer FC-activation function Tanh-fully connected layer FC. The number of output neurons in the MLP layer is the total number of categories of aerial images in the current dataset.

[0017] The projection head P(·) is only used in the training phase of the model. Its function is to represent the encoded global feature h i Mapped into the potential space where contrast loss is applied, its structure is fully connected layer FC—activation function ReLU—fully connected layer FC.

[0018] Preferably, in step S4), the discriminative label smoothing is performed on the label corresponding to the image, which means that the discriminative label smoothing is performed on the image according to the discrete probability value output by the model and the current training stage, and then the smoothed label is used to calculate the cross entropy loss function value, and the expression is as follows:

[0019]

[0020] Where, L CE is the cross entropy loss function value, K is the total number of categories in the aerial image dataset; is the initial label probability distribution of the i-th sample, that is, for the correct label category is 1, otherwise it is 0; It is the discrete probability distribution output by the model, which refers to the model's predicted probability of the i-th sample in the k-th class. γ.(s) is a smooth variable.

[0021] Preferably, the smooth variable γ.(s) is composed of two smooth variables γ hard (s) and γ simple (s) are used to control the smoothing weights of difficult samples and simple samples in different training stages, respectively. The expressions are as follows:

[0022]

[0023] γ simple (s) = (γ hard (s)+γ bias )*0.5 (1+s / I)

[0024] Among them, s∈{1…I} is the number of iterations of the current training, I is the total number of iterations; γ max is the maximum value of the smoothing weight corresponding to the difficult sample, γ min is the minimum value; bias is the deviation value of the smoothing weights of difficult samples and simple samples; Refers to the smooth interpolation function, which is expressed as follows:

[0025]

[0026] Among them, Comb: is the number of permutations and combinations, which represents the total number of ways to take out n elements from N+n elements, and N is used to control the smoothing rate.

[0027] Preferably, in the i-th sample The probability of K classes output by the model in the classification of difficult or simple samples When the maximum value is greater than 0.8 and the second largest value is less than 0.2, it is considered to be a simple sample, otherwise, it is classified as a difficult sample. The corresponding smooth variables are selected and the cross entropy loss function value is calculated.

[0028] Preferably, when the cross entropy loss function and the discriminative contrast loss function are used simultaneously to train the model in step S4), the total loss value L is calculated according to the following formula:

[0029] L=L CE +β*L DCL

[0030] Where, L CE is the cross entropy loss function, L DCL is the discriminative contrast loss function, and β is the weight coefficient, which is used to adjust the importance of the discriminative contrast loss function.

[0031] The expression of the discriminative contrast loss function is as follows:

[0032]

[0033]

[0034] In the formula, B*M is the total number of samples in the training set, is an indicator function that is 1 if and only if the input condition is met, and Among samples belonging to the same category, S i represents a set of samples enhanced by the same image, C i Indicates other situations, Representation and Sample Samples of the same type but obtained by different image enhancements The dot product ratio of Representation and Sample Samples of the same type and obtained from the same image enhancement The dot product ratio of , τ>0 is the temperature parameter, ε is the similarity threshold, 1≥ε>0.

[0035] The present invention also proposes a ViT-based aerial image recognition computer device, which includes a memory, a processor, and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps in the above method.

[0036] The present invention further proposes a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned ViT-based aerial image recognition method when executed by a processor.

[0037] The beneficial effects of the present invention are:

[0038] 1. High recognition rate: In order to solve the problem that the amount of trainable data in aerial image recognition is small and easily leads to overfitting of deep learning algorithms, the present invention adopts discriminative label smoothing to promote the model to learn sufficiently good feature information while not overfitting the distribution of noise data.

[0039] 2. Strong scalability: The ViT-based aerial image recognition method of the present invention has a high degree of versatility in principle and can be applied to different types of aerial image recognition tasks by selecting appropriate training data according to actual needs.

[0040] 3. Reasonable data structure: The present invention designs a discriminative label smoothing term and a discriminative supervised contrast loss to learn a more compact and reasonable data structure; thereby, a ViT-based aerial image recognition model with a stronger ability to capture significant features is trained, making the recognition of aerial images more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is an overall flow chart of an aerial image recognition method based on ViT of the present invention;

[0042] Figure 2 This is a demonstration diagram of a random enhancement module in an embodiment of the present invention;

[0043] Figure 3 Schematic diagram of a model for event recognition in aerial images in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to specifically illustrate the purpose, technical scheme, advantages and feasibility of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific examples described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0045] This embodiment takes event recognition in aerial images as a scenario to explain in detail a ViT-based aerial image recognition method provided by the present invention.

[0046] like Figure 1 As shown, the ViT-based aerial image recognition method proposed in the present invention is applied to event recognition tasks in aerial images. The detailed steps of the method are as follows:

[0047] Step S1: Collect event recognition data sets in aerial images and obtain aerial images x i and its corresponding event label y i In this embodiment, the ERA aerial image event recognition dataset is selected. The dataset contains 2864 sample images of 25 event categories. The already divided training set and test set are directly used, and the original training set is randomly divided into the training set and the validation set in a ratio of 9:1. The training set is recorded as B is the number of images in the training set.

[0048] Step S2: Construct a random data augmentation module to expand the amount of training data. Input the training set images in step S1 into the random augmentation module for online data augmentation. In the random augmentation module, the input image is first randomly cropped to 224*224 pixels and then randomly flipped horizontally. Then, six commonly used image augmentation strategies for the current visual task are selected, including (1) BaseAugment (normalization of the image only); (2) SimAugment (random color distortion and Gaussian blur are performed in sequence, and additional sparse image distortion operations may be performed at the end of the sequence); (3) AutoAugment (automatic augmentation); (4) RandAugment (random augmentation); (5) TrivialAugment (randomly select an image augmentation operation each time, then randomly determine its augmentation amplitude, and enhance the image); (6) RandomErasing (randomly erase a rectangular area from the image without changing the original label of the image). That is, given an image in the training set, M (6 ≥ M ≥ 0) strategies are randomly selected from the above six strategies to enhance the image, and finally the expanded training set is obtained, which is recorded as In this embodiment, M is 4, such as Figure 2 shown.

[0049] Step S3: Construct an aerial image recognition model based on ViT. The model structure is as follows: Figure 3 As shown. The model consists of an encoder F(·), a classification head G(·), and a projection head P(·) that is only used in the training phase:

[0050] The encoder F(·) is composed of ViT pre-trained on the ImageNet dataset, which is used to learn and encode the global features of the image. Specifically, the encoder F(·) consists of two parts: a linear layer and a transformer encoder. The linear layer is used to embed the image into a representation; the transformer encoder consists of a multi-head self-attention layer and a multi-layer perceptron block to learn the global features of the image. LayerNorm normalization is applied before each block, and residual connection is applied after each block. The training image In the input feature encoder F(·), the first token of the last layer of transformer encoder is used as The global feature representation h i . Then h i Input into the classifier and projector to calculate the total loss value.

[0051] The classification head G(·) is composed of MLP layers, and its structure is “fully connected layer FC—activation function Tanh—fully connected layer FC”. The number of output neurons in the MLP layer is the total number of categories of aerial images in the current data set, which is 25 in this embodiment.

[0052] The projection head P(·) is only used in the training phase of the model. Its function is to transform the encoded representation h i Mapped to the latent space where contrast loss is applied, its structure is “fully connected layer FC—activation function ReLU—fully connected layer FC”, and the number of output neurons in the MLP layer is 128.

[0053] Step S4: The training set in step S2 The image is input into the recognition model constructed in step S3, and then the labels corresponding to the images are discriminatively smoothed. At the same time, the model is trained using the cross entropy loss function and the discriminative contrast loss function. The recognition model is updated through the back propagation algorithm, and the model with the best recognition accuracy on the verification set in step S1 is selected as the final trained recognition model.

[0054] Among them, discriminative label smoothing is performed on the labels corresponding to the image, which means that according to the discrete probability value output by the model and the current training stage, the discriminative label smoothing of the image is performed, and then the smoothed label is used to calculate the cross entropy loss function value, which is expressed as follows:

[0055]

[0056] Where K is the total number of categories in the aerial image dataset; is the initial label probability distribution of the i-th sample, that is, for the correct label category is 1, otherwise it is 0; It is the discrete probability distribution output by the model, which refers to the model's predicted probability of the i-th sample in the k-th class.

[0057] Compared with natural image datasets, it usually costs more to obtain annotated aerial images, so the scale of aerial image datasets is generally small, which can easily lead to overfitting of the model on the training data. Although traditional label smoothing can alleviate model overfitting to a certain extent, it will have the risk of underfitting when the dataset size is small. Therefore, a smoothing variable γ.(s) is proposed to control the smoothing weight, and different smoothing weight values ​​are assigned according to the changes in the model training stage. Specifically, γ.(s) is composed of two smoothing variables γ hard (s) and γ simple (s) are used to control the smoothing weights of difficult samples and simple samples in different training stages, respectively. The expressions are as follows:

[0058]

[0059] γ simple (s) = (γ hard (s)+γ bias )*0.5 (1+s / I)

[0060] Among them, s∈{1…I} is the number of iterations of the current training, I is the total number of iterations; γ max is the maximum value of the smoothing weight corresponding to the difficult sample. Similarly, γ min is the minimum value; bias is the deviation value of the smoothing weights of difficult samples and simple samples; Refers to the smooth interpolation function. Its expression is as follows:

[0061]

[0062] Among them, Comb: represents the number of permutations and combinations, for example It refers to the total number of ways to extract n elements from N+n elements without considering the extraction order. N is used to control the smoothing rate, and is 1 in this embodiment.

[0063] In the i-th sample The probability of K classes output by the model in the classification of difficult or simple samples When the maximum value is greater than 0.8 and the second largest value is less than 0.2, it is considered to be a simple sample, otherwise it is classified as a difficult sample. The corresponding smoothing function is selected to calculate the cross entropy loss function value.

[0064] The cross entropy loss function and the discriminative contrast loss function are used to train the model at the same time, and the total loss value L is calculated according to the following formula:

[0065] L=L CE +β*L DCL

[0066] Where, L CE is the cross entropy loss function as described in claim 5, L DCL is the discriminative contrast loss function, and β is the weight coefficient, which is used to adjust the importance of the discriminative contrast loss function.

[0067] Among them, the discriminative contrast loss function is expressed as follows:

[0068]

[0069]

[0070]

[0071] Compared with natural images, aerial images have greater intra-class variation and inter-class similarity. Even samples of the same type have certain differences. After random enhancement, such differences will be further enhanced. Therefore, by further distinguishing whether images of the same type are obtained by enhancing the same image, the above discriminative contrast loss function is proposed. Specifically, B*M is the total number of samples in the training set, is an indicator function that is 1 if and only if the input condition is true. Among samples belonging to the same category, S i represents a set of samples enhanced by the same image, C i Indicates other situations. Representation and Sample Samples of the same type but obtained by different image enhancements The dot product ratio of Representation and Sample Samples of the same type and obtained from the same image enhancement τ>0 is the temperature parameter, and ε(1≥ε>0) is the similarity threshold.

[0072] Step S5: Input the test set images of step S1 into the trained recognition model, and compare the predicted category output by the model with the real category to obtain the final recognition accuracy. When the model recognition accuracy reaches the set threshold, the image to be recognized is input into the aerial image recognition model for recognition; otherwise, return to step S3) until the model recognition accuracy reaches the set threshold.

[0073] Based on the above method, the present invention also proposes a ViT-based aerial image recognition computer device, which includes a memory, a processor, and program instructions stored in the memory for the processor to execute, wherein the processor executes the program instructions to implement the steps in the above ViT-based aerial image recognition method.

[0074] The present invention further proposes a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned ViT-based aerial image recognition method when executed by a processor.

[0075] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

[0076] It should be understood that, for those skilled in the art, the principles of the present invention and the above description can be improved or transformed, or the method provided by the present invention can be applied to similar aerial image recognition tasks, and all these improvements and transformations should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A ViT-based aerial image recognition method, characterized in that: The method comprises the following steps: S1) Collect the aerial image data set to obtain the required original aerial image x i and its corresponding category label y i , divide the training set, validation set and test set into proportional quantities, which are used for subsequent model training, validation and evaluation respectively, where the training set is recorded as B is the number of images in the training set; S2) performing online data enhancement on the training set images so that each image in the training set generates M different enhanced images. The number of images in the training set after the enhancement is B*M, which is recorded as S3) construct an aerial image recognition model based on ViT; S4) the training set The image is input into the ViT-based aerial image recognition model, the labels corresponding to the images are discriminatively smoothed, and the model is trained by using a cross entropy loss function and a discriminative contrast loss function, the recognition model is updated by a back propagation algorithm, and the optimal aerial image recognition model is selected by using the validation set in step S1); S5) using the test set of step S1) to test the recognition performance of the aerial image recognition model to obtain a final model recognition accuracy, and when the model recognition accuracy reaches a set threshold, inputting the image to be recognized into the aerial image recognition model for recognition; Otherwise, return to step S3) until the model recognition accuracy reaches the set threshold.

2. The ViT-based aerial image recognition method according to claim 1, characterized in that: Step S2) randomly crops the input image into 224*224 pixels and then randomly flips it horizontally. Then, the image is enhanced using the image enhancement strategy, and finally the expanded training set is obtained, which is recorded as 3. The ViT-based aerial image recognition method according to claim 2, characterized in that: The image enhancement strategy in step S2) includes one or more combinations of the following operations: normalizing the image, performing random color distortion and Gaussian blur in sequence, automatic enhancement, random enhancement, randomly selecting an image enhancement operation each time, then randomly determining its enhancement amplitude, and enhancing the image, and randomly erasing a rectangular area from the image without changing the original label of the image.

4. The ViT-based aerial image recognition method according to claim 1, characterized in that: The ViT-based aerial image recognition model in step S3) consists of an encoder F(·), a classification head G(·) and a projection head P(·) used only in the training phase: The encoder F(·) is composed of ViT pre-trained on the dataset and is used to learn and encode the global features of the image. In the input feature encoder F(·), the first token of the encoder is used as The global feature representation h i ; The classification head G(·) consists of MLP layers, and its structure is fully connected layer FC-activation function Tanh-fully connected layer FC. The number of output neurons in the MLP layer is the total number of categories of aerial images in the current dataset. The projection head P(·) is only used in the training phase of the model. Its function is to represent the encoded global feature h i Mapped into the potential space where contrast loss is applied, its structure is fully connected layer FC—activation function ReLU—fully connected layer FC.

5. The ViT-based aerial image recognition method according to claim 1, characterized in that: In step S4), the labels corresponding to the images are discriminatively smoothed, which means that the images are discriminatively smoothed according to the discrete probability values ​​output by the model and the current training stage, and then the smoothed labels are used to calculate the cross entropy loss function value, which is expressed as: Where, L CE is the cross entropy loss function value, K is the total number of categories in the aerial image dataset; is the initial label probability distribution of the i-th sample, that is, for the correct label category is 1, otherwise it is 0; is the discrete probability distribution output by the model, which refers to the model's predicted probability of the i-th sample in the k-th class, γ · (s) is a smooth variable.

6. The ViT-based aerial image recognition method according to claim 5, characterized in that: The smooth variable γ · (s) consists of two smooth variables γ hard (s) and γ simple (s) are used to control the smoothing weights of difficult samples and simple samples in different training stages, respectively. The expressions are as follows: c simple (s)=(γ hard (s)+γ bias )*0.5 (1+s / I) Among them, s∈{1...I} is the number of iterations of the current training, I is the total number of iterations; γ max is the maximum value of the smoothing weight corresponding to the difficult sample, γ min is the minimum value; bias is the deviation value of the smoothing weights of difficult samples and simple samples; Refers to the smooth interpolation function, which is expressed as follows: Among them, Comb is the number of permutations and combinations, which means the total number of ways to take out n elements from N+n elements, and N is used to control the smoothing rate.

7. The ViT-based aerial image recognition method according to claim 6, characterized in that: In the i-th sample The probability of K classes output by the model in the classification of difficult or simple samples When the maximum value is greater than 0.8 and the second largest value is less than 0.2, it is considered to be a simple sample, otherwise, it is classified as a difficult sample. The corresponding smooth variables are selected and the cross entropy loss function value is calculated.

8. The ViT-based aerial image recognition method according to claim 6, characterized in that: When the cross entropy loss function and the discriminative contrast loss function are used to train the model in step S4), the total loss value L is calculated according to the following formula: L=L CE +β*L DCL Where, L CE is the cross entropy loss function, L DCL is the discriminative contrast loss function, β is the weight coefficient, which is used to adjust the importance of the discriminative contrast loss function; The expression of the discriminative contrast loss function is as follows: In the formula, B*M is the total number of samples in the training set, is an indicator function that is 1 if and only if the input condition is met, and Among samples belonging to the same category, S i represents a set of samples enhanced by the same image, C i Indicates other situations, Representation and Sample Samples of the same type but obtained by different image enhancements The dot product ratio of Representation and Sample Samples of the same type and obtained from the same image enhancement The dot product ratio of , τ>0 is the temperature parameter, ε is the similarity threshold, 1≥ε>0.

9. A ViT-based aerial image recognition computer device, comprising a memory, a processor, and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps in the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Remote sensing image building target efficient extraction method based on attention mechanism

    CN113780149A

  • Pollen image classification method based on cross attention distillation Transformer

    CN113887610A

  • Semantic soft label image recognition method and device for limited and unbalanced data

    CN114419394A

  • Computer Vision Systems and Methods for End-to-End Training of Convolutional Neural Networks Using Differentiable Dual-Decomposition Techniques

    US20210182675A1