Remote sensing image classification method and device, equipment and storage medium

By employing a multi-level Transformer structure and wavelet transform techniques, the problem of convolutional neural networks struggling to capture long-range contextual relationships in high-resolution remote sensing image classification was solved, achieving higher classification accuracy and efficiency.

CN118918371APending Publication Date: 2024-11-08WUHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410970400.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing convolutional neural networks struggle to capture long-range contextual relationships between images in high-resolution remote sensing image classification, resulting in high classification error rates.

Method used

A multi-level Transformer structure is adopted, which combines a multi-head attention layer and a tree-like wavelet frequency selection layer to enhance the image. Through the multi-head attention mechanism and wavelet transform, different frequency and long-range and short-range features of the image are captured to generate accurate feature maps for classification.

Benefits of technology

It improves the accuracy and efficiency of remote sensing image classification and significantly enhances the average accuracy on remote sensing datasets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918371A_ABST
    Figure CN118918371A_ABST
Patent Text Reader

Abstract

The invention discloses a remote sensing image classification method and device, equipment and a storage medium, and belongs to the technical field of image processing. The first remote sensing image is input to a multi-level Transform structure, a first feature map of the first remote sensing image is obtained, and each level of Transform is used for generating a first feature map; and classifying the first remote sensing image on the basis of a first feature map output by the last level of Transform in the multi-level Transform structure. The method can improve the accuracy degree of remote sensing image classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a remote sensing image classification method, device, equipment and storage medium. Background Art

[0002] With the development of technology, the resolution of remote sensing images is getting higher and higher. When classifying high-resolution remote sensing images, for example, when using conventional convolutional neural networks to implement remote sensing image classification, convolutional neural networks cannot capture the long-range contextual relationship between patches of high-resolution remote sensing images, which makes it easy to make mistakes when using convolutional neural networks to implement remote sensing image classification. Summary of the invention

[0003] The present disclosure provides a remote sensing image classification method, device, equipment and storage medium, which can improve the accuracy of remote sensing image classification. The technical solution at least includes the following solutions:

[0004] In a first aspect, a remote sensing image classification method is provided, including: obtaining a first remote sensing image; inputting the first remote sensing image into a multi-level Transformer structure to obtain a first feature map of the first remote sensing image, each level of Transformer is used to generate a first feature map, the first Transformer includes a lifting scheme multi-head attention layer, the first Transformer is any level of Transformer in the multi-level Transformer structure, the lifting scheme multi-head attention layer is used to adopt a lifting scheme in the channel dimension based on the first remote sensing image to obtain a first feature distributed along the channel dimension, based on the first feature, determine a first weight, based on the first remote sensing image, adopt a multi-head attention mechanism to obtain a second feature, and weight the second feature based on the first weight to obtain a third feature, the third feature is used to generate a first feature map of the first Transformer; based on the first feature map output by the last level Transformer in the multi-level Transformer structure, classify the first remote sensing image.

[0005] Optionally, the first Transformer also includes a tree-structured wavelet frequency selection layer, the output of which is connected to the input of the multi-head attention layer of the lifting scheme; the tree-structured wavelet frequency selection layer is used to: obtain the keys and values ​​of the first remote sensing image; downsample the keys and values ​​of the first remote sensing image to obtain the downsampled keys and values ​​of the first remote sensing image.

[0006] Optionally, the method of downsampling the key of the first remote sensing image is the same as the method of downsampling the value of the first remote sensing image, and the tree-like wavelet frequency selection layer is used to implement downsampling of the key and value of the first remote sensing image in the following manner to obtain the key and value of the first remote sensing image after downsampling: performing an L-level two-dimensional discrete wavelet transform on the key of the first remote sensing image to obtain a first frequency feature set, wherein the first frequency feature set includes multiple frequency features; constructing the multiple frequency features into a tree-like wavelet structure according to the resolution of the frequency features, and the tree-like wavelet frequency selection layer is used to implement downsampling of the key and value of the first remote sensing image in the following manner to obtain the key and value of the first remote sensing image after downsampling. The wavelet structure includes a tree root and multiple branches, the frequency features in each branch are different, and the frequency features in each branch are located at different multiple levels, the frequency features with the same resolution are located in the same level, and each level corresponds to a wavelet coefficient threshold; starting from the tree root, the multiple frequency features in the tree-like wavelet structure are traversed, and during the traversal process, the frequency features where noise interference is located are cut off to obtain a modified tree-like wavelet structure; based on the modified tree-like wavelet structure, the key after downsampling of the first remote sensing image is obtained; wherein L is an integer, and L is greater than 2.

[0007] Optionally, the multi-head attention layer of the lifting scheme is also used to obtain a query of the first remote sensing image. The multi-head attention layer of the lifting scheme is used to implement the second feature obtained based on the first remote sensing image by using a multi-head attention mechanism in the following manner: the query of the first remote sensing image, the key of the first remote sensing image after downsampling, and the value of the first remote sensing image after downsampling are processed by a multi-head attention mechanism to obtain multiple head features of the first remote sensing image, the multi-head attention mechanism includes multiple heads, and the output of each head is a head feature; the multiple head features are spliced ​​in the channel dimension to obtain the second feature.

[0008] Optionally, the lifting scheme multi-head attention layer is used to implement the first remote sensing image in the following manner: based on the first remote sensing image, a lifting scheme is adopted in the channel dimension to obtain a first feature distributed along the channel dimension, and a fourth feature is obtained. The fourth feature is obtained by converting the first remote sensing image into a one-dimensional sequence. After obtaining the first sequence, the first sequence is subjected to layer normalization processing; based on the parity of the channel of the fourth feature, the fourth feature is subjected to splitting processing, prediction processing and update processing to obtain a first prediction result and a first update result. The first prediction result is the output of the prediction processing, and the first update result is the output of the update processing; the first feature is determined based on the first prediction result and the first update result.

[0009] Optionally, the splitting processing, prediction processing and update processing of the fourth feature based on the parity of the channel to obtain a first prediction result and a first update result includes: using the following formula to implement the splitting processing of the fourth feature:

[0010] x odd , x eυen =Split(x n )

[0011] Among them, x odd is the odd channel feature in the fourth feature, x eυen is the even channel feature in the fourth feature, x n For the fourth feature, Split(x n ) is to split the fourth feature;

[0012] The following formula is used to implement the prediction process of the fourth feature:

[0013] I H =x odd -P(x eυen )=x odd -Tanh(conυ(x eυen ))

[0014] Among them, I H is the first prediction result, P is the prediction operator, Tanh() is the Tanh nonlinear activation function, conυ() is the convolution operation, Tanh(conυ(x eυen )) is to perform convolution processing on the even channel feature in the fourth feature and then input it into the Tanh nonlinear activation function;

[0015] The following formula is used to implement the updating process of the fourth feature:

[0016] I L =x eυen +U(x odd )=x eυen +Tanh(conυ(x odd ))

[0017] Among them, I L is the first update result, U is the update operator, Tanh(conυ(x odd )) is to perform convolution processing on the odd channel feature in the fourth feature and then input it into the Tanh nonlinear activation function;

[0018] The determining the first feature based on the first prediction result and the first update result includes:

[0019] The following formula is used to implement the determination of the first feature based on the first prediction result and the first update result:

[0020] Y=Concat(I L , I H )

[0021] Wherein, Y is the first feature, Concat(I L , I H ) is to concatenate the first prediction result and the second prediction result in the channel dimension.

[0022] Optionally, the first Transformer further includes a feedforward network layer, the output of the multi-head attention layer of the lifting scheme is connected to the input of the feedforward network layer, and the feedforward network layer is used to generate the first feature map based on the third feature using the following formula;

[0023] x out =FFN(IDWT(K w , V w )+z)

[0024] Among them, x out is the output of the feedforward network layer, the output of the feedforward network layer is used to expand and rearrange to obtain the first feature map, FFN() is the feedforward network layer, K w is the key after the first remote sensing image is downsampled, V w is the value of the first remote sensing image after downsampling, IDWT(K w , V w ) is a two-dimensional discrete wavelet inverse transform performed on the key after downsampling of the first remote sensing image and the value after downsampling of the first remote sensing image, and z is the third feature.

[0025] In the second aspect, a remote sensing image classification device is also provided, including: an acquisition module for acquiring a first remote sensing image; an input module for inputting the first remote sensing image into a multi-level Transformer structure to obtain a first feature map of the first remote sensing image, each level of Transformer is used to generate a first feature map, the first Transformer includes a lifting scheme multi-head attention layer, the first Transformer is any level of Transformer in the multi-level Transformer structure, the lifting scheme multi-head attention layer is used to adopt a lifting scheme in the channel dimension based on the first remote sensing image to obtain a first feature distributed along the channel dimension, based on the first feature, determine a first weight, based on the first remote sensing image, adopt a multi-head attention mechanism to obtain a second feature, and weight the second feature based on the first weight to obtain a third feature, the third feature is used to generate a first feature map of the first Transformer; a classification module is used to classify the first remote sensing image based on the first feature map output by the last level Transformer in the multi-level Transformer structure.

[0026] In a third aspect, a computer device is also provided, comprising: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor, thereby executing the remote sensing image classification method described in the above embodiment.

[0027] In a fourth aspect, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor, thereby executing the remote sensing image classification method described in the above embodiment.

[0028] In a fifth aspect, a computer program product is provided, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method described in the first aspect is implemented.

[0029] The beneficial effects brought by the technical solution provided by the embodiments of the present disclosure include at least:

[0030] In the disclosed embodiment, the lifting scheme multi-head attention layer in the first Transformer, in the frequency dimension, can focus on different parts of the input and capture more levels of features, which are the second features, because of the use of the multi-head attention mechanism; in the spatial dimension (that is, the channel dimension), the lifting scheme is adopted to obtain the first feature distributed along the channel dimension, and the second feature is weighted by the first feature (the first weight is calculated based on the first feature, so weighting the second feature by the first weight is equivalent to weighting the second feature by the first feature), so that the multi-level Transformer structure has the ability to interact with cross-combinations of different frequencies and long and short ranges. In this way, the third feature obtained based on the multi-level Transformer structure can more accurately classify the first remote sensing image. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 A flowchart of a remote sensing image classification method provided by an exemplary embodiment of the present disclosure is shown;

[0033] Figure 2 A flowchart of a remote sensing image classification method provided by another exemplary embodiment of the present disclosure is shown;

[0034] Figure 3 This is a schematic diagram of the structure of the first Transformer;

[0035] Figure 4 It is a schematic diagram of the tree wavelet frequency selection layer;

[0036] Figure 5 This is a schematic diagram of the multi-head attention layer of the improvement scheme;

[0037] Figure 6 It is a schematic diagram of the multi-level Transformer structure;

[0038] Figure 7 This is a schematic diagram of the workflow of the first Transformer;

[0039] Figure 8 A schematic diagram of the structure of a remote sensing image classification device provided by an exemplary embodiment of the present disclosure is shown;

[0040] Fig. 9It is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] Unless otherwise defined, the technical or scientific terms used herein shall have the usual meanings understood by persons of ordinary skill in the field to which the present disclosure belongs. The words "first", "second", "third" and similar words used in the patent application specification and claims of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not indicate a quantitative limitation, but rather indicate the presence of at least one. Words such as "include" or "comprise" and similar words mean that the elements or objects appearing before "include" or "comprise" cover the elements or objects listed after "include" or "comprise" and their equivalents, and do not exclude other elements or objects. Words such as "connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0042] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0043] To facilitate understanding of the embodiments of the present disclosure, the two-dimensional discrete wavelet transform and the lifting scheme are first explained below:

[0044] Discrete Wavelet Transform (DWT) is a branch of Fourier Transform. DWT can capture both the time domain and frequency domain information of the signal. When DWT is applied to a two-dimensional image, it is called a two-dimensional discrete wavelet transform, which means that the two-dimensional image is subjected to two DWTs in the row and column dimensions, thereby generating four sub-images LL, LH, HL and HH with half the height and width and half the resolution. These four sub-images represent the low-frequency features, vertical high-frequency features, horizontal high-frequency features and diagonal high-frequency features of the two-dimensional image.

[0045] DWT includes the first generation DWT and the second generation DWT. The second generation DWT introduces an improvement plan based on the first generation DWT.

[0046] The typical lifting scheme is mainly implemented by three core steps: splitting, prediction, and updating. Since the prediction and update operators are nonlinear, a nonlinear wavelet can be constructed based on the first generation DWT based on the lifting scheme. Nonlinear wavelets can well preserve the structural information of the image, thereby effectively extracting the vertical and horizontal features of the image.

[0047] Figure 1A flowchart of a remote sensing image classification method provided by an exemplary embodiment of the present disclosure is shown. The method can be executed by a computer device. Figure 1 , the method comprising:

[0048] In step 101, a first remote sensing image is acquired.

[0049] Optionally, the first remote sensing image is any remote sensing image that needs to be classified.

[0050] In step 102, the first remote sensing image is input into a multi-stage Transformer structure to obtain a first feature map of the first remote sensing image.

[0051] Each level of Transformer is used to generate a first feature map.

[0052] The first Transformer includes a lifting scheme multi-head attention layer. The first Transformer is any level Transformer in the multi-level Transformer structure. The lifting scheme multi-head attention layer is used to adopt a lifting scheme in the channel dimension based on the first remote sensing image to obtain a first feature distributed along the channel dimension, determine a first weight based on the first feature, adopt a multi-head attention mechanism based on the first remote sensing image to obtain a second feature, and weight the second feature based on the first weight to obtain a third feature. The third feature is used to generate a first feature map of the first Transformer.

[0053] Transformer is a deep learning model architecture for natural language processing (NLP) and other sequence-to-sequence tasks.

[0054] In the disclosed embodiment, the multi-level Transformer structure is a replacement for the backbone network of the RetinaNet detection network. The backbone network of the conventional RetinaNet detection network is ResNet, while in the disclosed embodiment, the backbone network of the RetinaNet detection network is replaced by the multi-level Transformer structure in step 102. The detection network can be called a RetinaNet detection network based on the multi-level Transformer structure.

[0055] In the embodiment of the present disclosure, when the input of the RetinaNet detection network based on the multi-stage Transformer structure is the first remote sensing image, the output of each level of the Transformer in the multi-stage Transformer structure is a first feature map.

[0056] Optionally, before executing step 101, the method also includes: establishing a benchmark remote sensing image data set to be classified; and using random sampling to divide the remote sensing images in the benchmark remote sensing image data set to be classified into a training set (Strain), a validation set (Sval) and a test set (Stest) in a non-overlapping manner according to a set ratio.

[0057] Among them, the remote sensing images in the training set are used to train the network parameters in the RetinaNet detection network based on the multi-level Transformer structure; the remote sensing images in the validation set are used to adjust the network parameters in the process of training the RetinaNet detection network based on the multi-level Transformer structure; the test set is used to test the trained RetinaNet detection network based on the multi-level Transformer structure to determine whether the classification of the RetinaNet detection network based on the multi-level Transformer structure is accurate. At this time, the network parameters of the RetinaNet detection network based on the multi-level Transformer structure have been fixed, and the output of the RetinaNet detection network based on the multi-level Transformer structure can be compared with the real label to test the RetinaNet detection network based on the multi-level Transformer structure.

[0058] Optionally, the set ratio may be 6:2:2.

[0059] In this case, the first remote sensing image may be any remote sensing image in a training set, a validation set, or a test set.

[0060] When training the RetinaNet detection network based on the multi-level Transformer structure through the training set, the network parameters can be learned through the back propagation and stochastic gradient algorithm of CNN (Convolutional Neural Networks). The training cycle can also be set, and the loss function can be made as small as possible and convergence can be achieved through continuous training, and finally the training can be completed.

[0061] In step 103, the first remote sensing image is classified based on the first feature map output by the last level Transformer in the multi-level Transformer structure.

[0062] In the disclosed embodiment, the lifting scheme multi-head attention layer in the first Transformer, in the frequency dimension, can focus on different parts of the input and capture more levels of features, which are the second features, because of the use of the multi-head attention mechanism; in the spatial dimension (that is, the channel dimension), the lifting scheme is adopted to obtain the first feature distributed along the channel dimension, and the second feature is weighted by the first feature (the first weight is calculated based on the first feature, so weighting the second feature by the first weight is equivalent to weighting the second feature by the first feature), so that the multi-level Transformer structure has the ability to interact with cross-combinations of different frequencies and long and short ranges. In this way, the third feature obtained based on the multi-level Transformer structure can more accurately classify the first remote sensing image.

[0063] Figure 2 A flowchart of a remote sensing image classification method provided by another exemplary embodiment of the present disclosure is shown. The method can be executed by a computer device. Figure 2 , the method comprising:

[0064] In step 201, a first remote sensing image is acquired.

[0065] The relevant contents of step 201 refer to the aforementioned step 101, and the detailed description is omitted here.

[0066] In step 202, the first remote sensing image is input into a RetinaNet detection network based on a multi-level Transformer structure.

[0067] In the disclosed embodiment, the RetinaNet detection network based on the multi-stage Transformer structure includes a backbone network, a feature pyramid, and a detection head connected in sequence.

[0068] The backbone network is a multi-level Transformer structure. In the embodiment of the present disclosure, each level of the Transformer structure in the multi-level Transformer structure is the same, and the first Transformer is any level of the Transformer in the multi-level Transformer. r Any level of Transformation in the structure r Provide explanation.

[0069] Optionally, the multi-level Transformer structure is a 4-level Transformer. Alternatively, in the multi-level Transformer structure, the number of levels is greater than 4, for example, it can be a 5-level Transformer. r , Level 6 Transformerr , 7-level Transformer, etc.

[0070] Figure 3 This is a schematic diagram of the structure of the first Transformer, see Figure 3 The first Transformer includes an embedding layer, a tree wavelet frequency selection layer, a lifting scheme multi-head attention layer, and a feedforward network layer connected in sequence. r Each layer is described separately.

[0071] Embedding layer

[0072] In the embodiment of the present disclosure, the embedding layer includes a Patch embedding layer and a position embedding layer, and the embedding layer is used to convert the first remote sensing image into a one-dimensional sequence to obtain a first sequence.

[0073] For the first remote sensing image x, its size is x∈R (H×W×C) , that is, the length of the first remote sensing image is H, the width is W, and the number of channels is C. When the first remote sensing image is input into the embedding layer, it is first input into the position embedding layer, and then into the patch embedding layer.

[0074] The position embedding layer is obtained using relative position encoding. The patch embedding layer performs a sliding window convolution operation on the first remote sensing image to obtain non-overlapping patches, where the step size of the sliding window is the size of the convolution kernel.

[0075] There are many methods for implementing the position embedding layer and the patch embedding layer in the related technologies, so detailed description is omitted here.

[0076] The final first sequence x p is a one-dimensional vector sequence, the size of the first sequence is x p ∈R (N×D) , that is, the first sequence is a one-dimensional vector sequence with N patches and D dimensions of hidden channels.

[0077] Optionally, before inputting the first sequence into the tree-structured wavelet frequency selection layer, the first sequence needs to be subjected to layer normalization processing. For example, the first sequence may be input into a layer normalization layer, so as to obtain a fourth feature that is stable after normalization.

[0078] The layer normalization process can be expressed by formula (1).

[0079] x n =LN(x p ) (1)

[0080] In formula (1), x n is the fourth feature, LN() is the layer normalization layer, xp is the first sequence, LN(x p ) indicates that the first sequence is input to the layer normalization layer. There are many implementation methods of the layer normalization layer in the related art, which are omitted here. The fourth feature can be rearranged into a feature of h×w×D, that is, x n ∈R (h×w×D) , indicating that the length of the fourth feature is h, the width is w, and the number of channels is D.

[0081] There are many methods for implementing the layer normalization layer in the related art, and the detailed description is omitted here.

[0082] After the fourth feature is obtained, the fourth feature can be passed through three linear transformation layers to obtain the three features of query, key and value of the first remote sensing image, so as to facilitate the subsequent processing of the three features.

[0083] The process of passing the fourth feature through three linear transformation matrices respectively to obtain the three features of the first remote sensing image, namely, the query, the key and the value can be expressed by formula (2).

[0084] Q, K, V = x n w q , x n w k , x n w υ (2)

[0085] In formula (2), Q is the query, K is the key, V is the value, and w q is the linear transformation matrix corresponding to the query, w k is the linear transformation matrix corresponding to the key, w υ is the linear transformation matrix corresponding to the value, and the meanings of other parameters in formula (2) are the same as those in formula (1), and detailed description is omitted here.

[0086] The keys and values ​​can be rearranged into h×w×D features, i.e. K, V∈R (h×w×D) , indicating that the length of the key and value is h, the width is w, and the number of channels is D.

[0087] Tree-structured wavelet frequency selection layer

[0088] The tree-like wavelet frequency selection layer is used to obtain the key and value of the first remote sensing image; the key and value of the first remote sensing image are downsampled to obtain the downsampled key and value of the first remote sensing image.

[0089] After the fourth feature passes through three linear transformation matrices respectively, the three features of query, key and value of the first remote sensing image obtained can be input into the tree wavelet frequency selection layer. Accordingly, the tree wavelet frequency selection layer can realize the acquisition of the key and value of the first remote sensing image.

[0090] Optionally, the way of downsampling the key of the first remote sensing image is the same as the way of downsampling the value of the first remote sensing image. In this case, the tree wavelet frequency selection layer is used to implement downsampling of the key and value of the first remote sensing image by adopting the following steps ad to obtain the key and value of the first remote sensing image after downsampling:

[0091] Step a: performing an L-level two-dimensional discrete wavelet transform on the key of the first remote sensing image to obtain a first frequency feature set.

[0092] The first frequency feature set includes multiple frequency features. L is an integer, and L is greater than 2. In the following description, L is 3.

[0093] When performing a three-level two-dimensional discrete wavelet transform on the key of the first remote sensing image, the first-level two-dimensional discrete wavelet transform can transform the key of the first remote sensing image into four sub-images, LL1, LH1, HL1 and HH1, whose length, width and resolution are halved relative to the key of the first remote sensing image. The second-level two-dimensional discrete wavelet transform can transform LL1 into four sub-images, LL2, LH2, HL2 and HH2, whose length, width and resolution are halved relative to LL1. Similarly, the third-level two-dimensional discrete wavelet transform can transform LL2 into four sub-images, LL3, LH3, HL3 and HH3, whose length, width and resolution are halved relative to LL2.

[0094] Therefore, when L is 3, the first frequency feature set includes ten frequency features: LL3, LH3, HL3, HH3, LH2, HL2, HH2, LH1, HL1 and HH1.

[0095] Step b: constructing multiple frequency features into a tree-like wavelet structure according to the resolution of the frequency features.

[0096] The tree-like wavelet structure includes a root and multiple branches. The frequency features in each branch are different, and the frequency features in each branch are located at different multiple levels. Frequency features with the same resolution are located at the same level, and each level corresponds to a wavelet coefficient threshold.

[0097] Figure 4 is a schematic diagram of the tree wavelet frequency selection layer. Figure 4 Part (a) is a schematic diagram of constructing a tree-like wavelet structure. Figure 4 In part (a), L is 3. In the constructed tree-like wavelet structure, LH3, LH3, and LH1 are on the same branch, that is, they are all vertical high-frequency features; HL3, HL2, and HL1 are on the same branch, that is, they are all horizontal high-frequency features; HH3, HH2, and HH1 are on the same branch, that is, they are all diagonal high-frequency features. It can be seen that frequency features in the same direction are on the same branch.

[0098] Since frequency features of the same resolution are located in the same level, LL3, LH3, HL3, and HH3 are in the same level (level 1), LH2, HL2, and HH2 are in the same level (level 2), and LH1, HL1, and HH1 are in the same level (level 3).

[0099] It can be seen that multiple levels are arranged in an arrangement order. In the arrangement order, multiple levels are arranged in order from low to high resolution, and the level with the lowest resolution is connected to the root. The root is a low-frequency frequency feature. When L is 3, the root is LL3.

[0100] The wavelet coefficient threshold corresponding to each level is calculated using formula (3) to formula (4).

[0101]

[0102] In formula (3), T0 represents the wavelet coefficient threshold corresponding to the first level, Indicates log2Max(c i,j ) is rounded down, c i,j is the wavelet coefficient of the pixel with coordinates (i, j) in level 1. Here, the wavelet coefficient of a pixel is the value obtained after the pixel is transformed by wavelet.

[0103]

[0104] In formula (4), T k represents the wavelet coefficient threshold of the k+1th level, k is an integer, k is greater than 0 and less than L; T k-1 Indicates the wavelet coefficient threshold of the kth level.

[0105] Step c, starting from the root, traverse multiple frequency features in the tree-like wavelet structure, and in the traversal process, cut off the frequency features where the noise interference is located to obtain a modified tree-like wavelet structure.

[0106] Optionally, during traversal, the traversal is performed in the order in which the multiple levels are arranged.

[0107] In the embodiment of the present disclosure, the frequency characteristics of the noise interference are determined in the following manner:

[0108] During the traversal process, if the sum of the wavelet coefficients of each pixel in the first frequency feature is greater than the wavelet coefficient threshold corresponding to the level where the first frequency feature is located, the first frequency feature is retained. If the sum of the wavelet coefficients of each pixel in the first frequency feature is less than or equal to the wavelet coefficient threshold corresponding to the level where the first frequency feature is located, the branch where the first frequency feature is located is pruned from the first frequency feature, and the pruned frequency feature is the frequency feature where the noise interference is located. Among them, the first frequency feature is any frequency feature in the first frequency feature set.

[0109] Figure 4 Part (b) is a schematic diagram of the modified tree-like wavelet structure. Figure 4 As shown in part (b), the sum of the wavelet coefficients of each pixel in HH3 is less than or equal to the wavelet coefficient threshold corresponding to the level where HH3 is located (that is, the first level), so the branch is pruned starting from HH3, that is, HH3, HH2, and HH1 are all frequency features where noise interference is located and need to be pruned. Similarly, LH1 is also a frequency feature where noise interference is located and is pruned.

[0110] Step d: obtaining a downsampled key of the first remote sensing image based on the modified tree-like wavelet structure.

[0111] When implementing step d, the frequency features in the modified tree-like wavelet structure may be inversely transformed to obtain the key after the first remote sensing image is downsampled.

[0112] When performing the inverse transformation, the frequency features in the modified tree-like wavelet structure are inversely transformed in order from low to high resolution, and the frequency features with the same resolution are used for inverse transformation, thereby obtaining frequency features with doubled resolution. The inverse transformation process can be expressed by formula (5).

[0113]

[0114] Since the frequency characteristics in the modified tree-like wavelet structure have three different resolutions, three two-dimensional discrete wavelet inverse transforms are performed in formula (5). In formula (5), The lengths of LL3, LH3 and HL3 are Width The number of channels is D. IDWT() represents the two-dimensional inverse discrete wavelet transform, IDWT(LL3, LH3, HL3) represents the two-dimensional inverse discrete wavelet transform of LL3, LH3 and HL3, and y1 is the result of the first inverse transform. The lengths of y1, LH2 and HL2 are Width The number of channels is D. IDWT(y1, LH2, HL2) represents the two-dimensional inverse discrete wavelet transform of y1, LH2 and HL2, and y2 is the result of the second inverse transform. It means that the length of y2 and HL1 is Width The number of channels is D. IDWT(y2, HL1) represents the two-dimensional discrete wavelet inverse transform of y2 and HL1, and y is the result of the third inverse transform.

[0115] Figure 4 Part (c) of is a schematic diagram of the inverse transformation, see Figure 4 As can be seen from part (c) of the figure, the resolution of the output result of each inverse transform is doubled compared to the resolution before the inverse transform.

[0116] Here, since steps ad are to downsample the key of the first remote sensing image, the y finally obtained in step d is actually the key after downsampling of the first remote sensing image. w When downsampling the value of the first remote sensing image, it is only necessary to replace the key in the above steps ad with the value. At this time, the y finally obtained in step d is actually the value of the first remote sensing image after downsampling. w express.

[0117] In the disclosed embodiment, through the tree-structured wavelet frequency selection layer, on the one hand, it is possible to filter out the noise in the keys and values ​​of the first remote sensing image, thereby improving the accuracy of subsequent classification of the first remote sensing image, and on the other hand, it is possible to reduce the resolution of the first remote sensing image, thereby speeding up the classification of the first remote sensing image.

[0118] Improvement scheme multi-head attention layer

[0119] Optionally, the lifting scheme multi-head attention layer is used to adopt a lifting scheme in the channel dimension based on the first remote sensing image to obtain a first feature distributed along the channel dimension, determine a first weight based on the first feature, and adopt a multi-head attention mechanism based on the first remote sensing image to obtain a second feature, and weight the second feature based on the first weight to obtain a third feature, and the third feature is used to generate a first feature map of the first Transformer.

[0120] Optionally, the multi-head attention layer of the lifting scheme is also used to obtain a query of the first remote sensing image. In this case, the multi-head attention layer of the lifting scheme is used to implement the second feature based on the first remote sensing image by using the multi-head attention mechanism using the following steps ef:

[0121] In step e, the query of the first remote sensing image, the key after downsampling of the first remote sensing image, and the value after downsampling of the first remote sensing image are processed using a multi-head attention mechanism to obtain multiple head features of the first remote sensing image.

[0122] The multi-head attention mechanism includes multiple heads, and the output of each head is a head feature. Here, the multi-head attention mechanism is to process the input with multiple groups of self-attention mechanisms, and the process of each group of self-attention mechanisms is a head. The output of a self-attention mechanism is the head feature corresponding to the self-attention mechanism.

[0123] Optionally, the number of heads in the first Transformer can be calculated by the number of heads in the first Transformer in the multi-stage Transformer. r The level in the structure.

[0124] In a multi-stage Transformer structure, the number of heads in each stage of the Transformer can be different.

[0125] When the image is input into the multi-level Transformer structure, as the image is processed by each level of Transformer in sequence, the number of channels of the query of the first remote sensing image, the key after downsampling of the first remote sensing image, and the value after downsampling of the first remote sensing image that each level of Transformer needs to process gradually increases. Usually, the number of heads needs to be divisible by the number of channels. Therefore, as the level of the first Transformer in the multi-level Transformer structure increases, the number of heads can gradually increase. r Compared to the first Transforme r It is the fourth level Transforme r , the number of heads can be greater than the number of heads when the first Transformer is a first-level Transformer.

[0126] Optionally, the number of heads in each level of Transformer may be set based on experience, which is not limited in the embodiments of the present disclosure.

[0127] Optionally, step e is implemented by formula (6).

[0128]

[0129] In formula (6), head j is the head feature of the jth head, j = 1, 2, ..., N h , N his the number of heads in the first Transformer, Softmax() represents the Softmax normalization function, Indicates that Input Softmax normalization function. Q j represents the query of the first remote sensing image input to the jth head, represents the key of the first remote sensing image input to the jth head, represents the value of the first remote sensing image input to the jth head, D h Represents the dimensions of each head.

[0130] Step f: concatenate multiple head features in the channel dimension to obtain a second feature.

[0131] Optionally, step e is implemented by formula (7).

[0132]

[0133] In formula (7), m is the second characteristic, To concatenate all heads in the channel dimension, w o is the linear transformation matrix. The meanings of other parameters in formula (7) are the same as those in formula (6), and detailed description is omitted here.

[0134] The following description assumes that the number of heads in the first Transformer is 4.

[0135] Figure 5 It is a schematic diagram of the multi-head attention layer of the improvement scheme. Figure 5 Part (a) of FIG. 1 is a schematic diagram of step ef, as shown in FIG. Figure 5 As shown in part (a), the number of heads is 4. For the jth head j The head features are concatenated in the channel dimension.

[0136] Optionally, the lifting scheme multi-head attention layer is used to implement the lifting scheme in the channel dimension by adopting the following step gi to obtain a first feature distributed along the channel dimension, and determine a first weight based on the first feature:

[0137] Step g, obtaining the fourth feature.

[0138] The relevant content of the fourth feature can be found in the aforementioned formula (1), and the detailed description is omitted here.

[0139] Step h: based on the parity of the channel of the fourth feature, the fourth feature is split, predicted and updated to obtain a first prediction result and a first update result.

[0140] The first prediction result is the output of the prediction process, and the first update result is the output of the update process.

[0141] Optionally, the fourth feature is split using formula (8):

[0142] x odd , x eυen =Split(x n ) (8)

[0143] In formula (8), x odd is the odd channel feature in the fourth feature, x eυen is the even channel feature in the fourth feature, x n The fourth feature is Split(x n ) is to split the fourth feature. Before splitting, the size of the fourth feature is x n ∈R (h ×w×D) After splitting the fourth feature in the channel dimension, the size of the odd channel feature and the even channel feature can be expressed as x odd , That is, the length of the odd channel feature and the even channel feature is h, the width is w, and the number of channels is That is to say, the number of channels of the features after splitting is halved compared to the number of channels of the features before splitting.

[0144] Optionally, the fourth feature is predicted using formula (9):

[0145] I H =x odd -P(x eυen )=x odd -Tanh(conυ(x eυen )) (9)

[0146] In formula (9), I H is the first prediction result, P is the prediction operator, Tanh() is the Tanh nonlinear activation function, conυ() is the convolution operation, Tanh(conυ(x eυen )) is to convolve the even channel feature in the fourth feature and then input it into the Tanh nonlinear activation function. The meanings of other parameters in formula (9) are the same as those in formula (8), and detailed description is omitted here. The size of the first prediction result can be expressed as That is, the length of the first prediction result is h, the width is w, and the number of channels is

[0147] In the disclosed embodiment, the first prediction result is a high-frequency feature in the channel dimension.

[0148] Optionally, the fourth feature is updated using formula (10):

[0149] I L =x eυen +U(x odd )=x eυen +Tanh(conυ(x odd )) (10)

[0150] In formula (10), I L is the first update result, U is the update operator, Tanh(conυ( xodd )) is to convolve the odd channel features in the fourth feature and then input them into the Tanh nonlinear activation function. The meanings of other parameters in formula (10) are the same as those in formula (8) and formula (9), and detailed description is omitted here. The size of the first update result can be expressed as That is, the length of the first update result is h, the width is w, and the number of channels is

[0151] In the embodiment of the present disclosure, the first update result is a low-frequency feature in the channel dimension.

[0152] Exemplarily, in the process of predicting the fourth feature and updating the fourth feature, the convolution kernel may be a 3x3 convolution kernel, and the step size of the convolution operation may be 1.

[0153] Since the number of heads in the first Transformer is 4, the number of head features is also 4. When weighting the 4 head features based on the first feature, the first feature needs to have 4 parts so that it can correspond one-to-one with the 4 head features to weight the 4 head features respectively.

[0154] If step h is only performed once, only the frequency characteristics of the first prediction result and the first update result can be obtained, rather than the frequency characteristics of four parts. Therefore, in the embodiment disclosed herein, step h needs to be performed once for the first prediction result and the first update result respectively, that is, step h is performed once with the first prediction result as the fourth feature, so that the second prediction result and the second update result of the first prediction result can be obtained, which is equivalent to decomposing the first prediction result into two parts of the frequency characteristics. The same is true for the first update result. Step h is performed once with the first update result as the fourth feature, so that the third prediction result and the third update result of the first update result can be obtained, which is equivalent to decomposing the first update result into two parts of the frequency characteristics. Finally, the frequency characteristics of the four parts of the second prediction result, the second update result, the third prediction result and the third update result can be obtained.

[0155] Step i: determining a first feature based on the first prediction result and the first update result.

[0156] Optionally, when the number of heads in the first Transformer is 2, formula (11) is used to determine the first feature based on the first prediction result and the first update result:

[0157] Y=Concat(I L , I H ) (11)

[0158] In formula (11), Y is the first feature, Concat(I L , I H ) is to concatenate the first prediction result and the second prediction result in the channel dimension.

[0159] When the number of heads in the first Transformer is 2, the first feature only needs to concatenate the first prediction result and the first update result. When the number of heads in the first Transformer is 4, the first feature needs to concatenate the second prediction result, the second update result, the third prediction result, and the third update result.

[0160] Figure 5 Part (b) of is a schematic diagram of step gi, such as Figure 5 As shown in part (b), first, the fourth feature x n Execute step h once to obtain the first prediction result I H and the first update result I L Since the number of heads is 4, the number of frequency features of the two parts is insufficient. H Execute step h once, so that the first prediction result I H Decomposed into two parts of frequency features; on the other hand, the first update result I L Execute step h once, so that the first update result I L After being decomposed into two frequency features, we can finally get four frequency features. Finally, we can concatenate these four frequency features to get the first feature Y.

[0161] Optionally, the multi-head attention layer of the lifting scheme is used to determine the first weight based on the first feature in the following manner: input the first feature into the head importance module to obtain the first weight.

[0162] The head importance module includes multiple weight parameters, and the number of the multiple weight parameters is the same as the number of frequency features in the first feature. For example, if there are 4 parts of frequency features in the first feature, the head importance module includes 4 weight parameters. Each weight parameter is used to weight the frequency features of one part in the first feature. Since the head features output by multiple heads are finally spliced ​​into a second feature, the weighted frequency features of multiple parts here will also be spliced ​​into the first weight.

[0163] In the embodiment of the present disclosure, there are the following three implementation methods for the head importance module. When implementing the head importance module, you only need to use any one of the following three implementation methods.

[0164] The first method is to implement the head importance module through manual design.

[0165] In the embodiment of the present disclosure, the manual design method is to manually design multiple weight parameters.

[0166] Since the importance of features of different frequencies is different, low-frequency features represent the main body of the image, and high-frequency features represent the contour and noise of the image. Therefore, when designing weight parameters, the weight parameters of low-frequency features can be higher than those of high-frequency features.

[0167] Exemplarily, the first feature includes the frequency features of the second prediction result, the second update result, the third prediction result and the third update result. The frequency features of the four parts can be sorted from low frequency to high frequency as follows: the third update result, the third prediction result, the second update result and the second prediction result. Then, according to the sorting from low frequency to high frequency, the weight parameters can be designed to be 0.8, 0.5, 0.3 and 0.2 respectively.

[0168] The head importance module can be realized relatively easily by adopting the manual design method.

[0169] The second method is to implement the head importance module through network learning.

[0170] In implementation, a convolutional neural network may be set to learn the channel importance of each head feature. In the process of training the convolutional neural network, multiple weight factors are set for learning, for example, 4 weight parameters are set for learning.

[0171] The network learning method can effectively provide the accuracy of each weight parameter in the head importance module.

[0172] The third method is to implement the head importance module by calculating the channel correlation within each frequency feature.

[0173] During calculation, the frequency feature of each part of the first feature is calculated separately, and the frequency feature of each part includes multiple channels. For example, the third update result includes multiple channels. During calculation, it is necessary to calculate the correlation between the multiple channels in the third update result, and then take the average value of all the calculated correlations (or the maximum value or the minimum value), and the average value is the weight parameter of the third update result. For the frequency features of other parts, the weight parameters can also be calculated in the same way.

[0174] After calculating a plurality of weight parameters, the plurality of weight parameters are used to weight the frequency characteristics of the plurality of parts of the first characteristic respectively. The weighting is to multiply a certain weight parameter by the frequency characteristic corresponding to the weight parameter.

[0175] Optionally, the multi-head attention layer of the lifting scheme is used to adopt formula (12) to weight the second feature based on the first weight to obtain a third feature.

[0176] z=HM(Y)×m(12)

[0177] In formula (12), z is the third feature, HM() represents the head importance module, and HM(Y) represents inputting the first feature into the head importance module. The meanings of other parameters in formula (12) are the same as those in formula (7) and formula (11), and detailed description is omitted here.

[0178] Figure 5 Part (c) of is a schematic diagram of weighting the second feature. Figure 5 As shown in part (c), after the first feature is input into the head importance module, a first weight is obtained, and the second feature is weighted by the first weight to obtain a third feature.

[0179] Feedforward Network Layer

[0180] Optionally, the feedforward network layer is used to generate a first feature map based on the third feature using formula (13).

[0181] x out =FFN(IDWT(K w , V w )+z) (13)

[0182] In formula (13), x out is the output of the feedforward network layer, FFN() is the feedforward network layer, K w is the key after the first remote sensing image is downsampled, V w is the value after downsampling of the first remote sensing image, IDWT(K w , V w) is to perform a two-dimensional discrete wavelet inverse transform on the key after downsampling of the first remote sensing image and the value after downsampling of the first remote sensing image, which can be used To indicate that z is the third characteristic.

[0183] In the Before the third feature is input into the feedforward network layer, it is also necessary to After adding the third feature, the layer normalization is performed, for example, input to the layer normalization layer. In addition, in the conventional Transformer structure, After adding the first feature and the third feature, the first sequence needs to be combined with The third feature is added before it can be input into the layer normalization layer.

[0184] The output x of the feedforward network layer out It is essentially the same as the first feature map, but the representation is different. The output x of the feedforward network layer out It can be restored to the first feature map by expanding and rearranging.

[0185] Figure 6 This is a schematic diagram of the multi-level Transformer structure. Figure 6 As shown, the schematic diagram of the multi-level Transformer structure includes a total of 4 levels of Transformer r , each level Transforme r The internal structure is the same. As the first remote sensing image is processed by each level of Transformer in the multi-level Transformer in turn, the first feature map output by each level of Transformer becomes smaller and smaller, thereby reducing the resolution of the first remote sensing image while effectively extracting the features of the first remote sensing image. In this way, the resolution of the first feature map output by the last level of Transformer is low enough, and the subsequent structure of the RetinaNet detection network can process low-resolution images more accurately and quickly, thereby improving the efficiency and accuracy of remote sensing image classification.

[0186] Figure 7 This is a schematic diagram of the workflow of the first Transformer. Figure 7 As shown in the figure, it can be seen that in the first Transformer, the tree wavelet frequency selection layer, the lifting scheme multi-head attention layer and the feedforward network layer belong to Transformer r Encoder structure.

[0187] In step 203, the first remote sensing image is classified based on the first feature map output by the last level Transformer in the multi-level Transformer structure.

[0188] When implementing step 203, it is only necessary to input the first feature map output by the last level Transformer into the subsequent structure of the RetinaNet detection network.

[0189] Among them, the RetinaNet detection network includes: a backbone network, a feature pyramid, a classification subnetwork, and a bounding box regression subnetwork connected in sequence. The first feature map output by the last level Transformer is equivalent to the output of the backbone network. It is sequentially input into the feature pyramid, the classification subnetwork, and the bounding box regression subnetwork to classify the first remote sensing image. There are many implementation methods for the feature pyramid, the classification subnetwork, and the bounding box regression subnetwork in the related art, so detailed description is omitted here.

[0190] The RetinaNet detection network based on the multi-level Transformer structure in the embodiment of the present disclosure is compared with the RetinaNet detection network (Swin-Tiny structure and PVT-Tiny structure) whose backbone network is the common Transformer structure and whose backbone network is the common ResNet50. On the remote sensing dataset DOTA-v1.0, the average precision mean mAP of the method of the present invention is 6.3%, 1.6%, and 3.1% higher than that of ResNet50, SwinTransformer-Tiny, and PVT-Tiny, respectively. On the remote sensing dataset HRSID, the average precision mean mAP of the method of the present invention is 7.4%, 1.8%, and 4.8% higher than that of ResNet50, SwinTransformer-Small, and PVT-Small, respectively.

[0191] In the comparative experiment, the difference between the RetinaNet detection network based on the multi-level Transformer structure in the embodiment of the present disclosure and other RetinaNet detection networks is that the backbone network of the RetinaNet detection network based on the multi-level Transformer structure in the embodiment of the present disclosure is a multi-level Transformer structure, while the backbone networks of other RetinaNet detection networks in the comparative experiment are ResNet50, Swin-Tiny structure and PVT-Tiny structure, and all other settings are the same.

[0192] In the disclosed embodiment, the RetinaNet detection network based on the multi-level Transformer structure can fully utilize the respective advantages of wavelet transform and Transformer global modeling, and can effectively capture the long-range dependency between two targets in remote sensing scenes. At the same time, the multi-resolution downsampling capability of the wavelet transform is used, which significantly reduces the additional overhead required for subsequent multi-head attention mechanism calculations and reduces video memory. The RetinaNet detection network based on the multi-level Transformer structure can effectively extract features of images or feature maps, and perform multi-dimensional modeling of targets in images, effectively improving the average accuracy of the detection network in the benchmark remote sensing image target detection task.

[0193] In specific implementation, the above process can be automatically run using computer software technology, and the system device of the running method should also be within the protection scope of the present invention.

[0194] The following are device embodiments of the present application. For details not described in detail in the device embodiments, reference may be made to the above method embodiments.

[0195] Figure 8 A schematic diagram of a remote sensing image classification device provided by an exemplary embodiment of the present disclosure is shown. Figure 8 The remote sensing image classification device 800 includes: an acquisition module 801, an input module 802 and a classification module 803.

[0196] The acquisition module 801 is used to acquire a first remote sensing image.

[0197] The input module 802 is used to input the first remote sensing image into the multi-level Transformer structure to obtain the first feature map of the first remote sensing image. Each level of Transformer is used to generate a first feature map. The first Transformer includes a lifting scheme multi-head attention layer. The first Transformer is any level of Transformer in the multi-level Transformer structure. The lifting scheme multi-head attention layer is used to adopt a lifting scheme in the channel dimension based on the first remote sensing image to obtain a first feature distributed along the channel dimension. Based on the first feature, a first weight is determined. Based on the first remote sensing image, a multi-head attention mechanism is used to obtain a second feature. The second feature is weighted based on the first weight to obtain a third feature. The third feature is used to generate the first feature map of the first Transformer.

[0198] The classification module 803 is used to classify the first remote sensing image based on the first feature map output by the last level Transformer in the multi-level Transformer structure.

[0199] It should be noted that: when the remote sensing image classification device provided in the above embodiment performs remote sensing image classification, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the remote sensing image classification device provided in the above embodiment and the remote sensing image classification method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0200] The division of modules in the embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present disclosure may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0201] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a terminal device (which can be a personal computer, a mobile phone, or a communication device, etc.) or a processor (processor) to perform all or part of the steps of the method of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0202] Fig. 9 Schematic diagram of the structure of the computer device provided by the embodiment of the present disclosure. Fig. 9 As shown, the computer device 900 includes: a processor 901 and a memory 902 .

[0203] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0204] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction, which is used to be executed by the processor 901 to implement the remote sensing image classification method provided in the embodiment of the present disclosure.

[0205] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the computer device 900, and the computer device 900 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0206] The embodiment of the present disclosure also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a computer device, the computer device can execute the remote sensing image classification method provided in the embodiment of the present disclosure.

[0207] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which implements the remote sensing image classification method provided in the embodiments of the present disclosure when the computer program / instruction is executed by a processor.

[0208] The above description is only an optional embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A remote sensing image classification method, characterized in that: The method comprises: acquiring a first remote sensing image; Input the first remote sensing image into a multi-level Transformer structure to obtain a first feature map of the first remote sensing image, each level of Transformer is used to generate a first feature map, the first Transformer includes a lifting scheme multi-head attention layer, the first Transformer is any level of Transformer in the multi-level Transformer structure, the lifting scheme multi-head attention layer is used to adopt a lifting scheme in the channel dimension based on the first remote sensing image to obtain a first feature distributed along the channel dimension, based on the first feature, determine a first weight, based on the first remote sensing image, adopt a multi-head attention mechanism to obtain a second feature, and weight the second feature based on the first weight to obtain a third feature, the third feature is used to generate a first feature map of the first Transformer; The first remote sensing image is classified based on a first feature map output by a last-stage Transformer in the multi-stage Transformer structure.

2. The method according to claim 1, characterized in that The first Transformer further includes a tree-like wavelet frequency selection layer, and the output of the tree-like wavelet frequency selection layer is connected to the input of the multi-head attention layer of the lifting scheme; The tree-structured wavelet frequency selection layer is used to: Obtaining the key and value of the first remote sensing image; Downsampling is performed on the keys and values ​​of the first remote sensing image to obtain the downsampled keys and values ​​of the first remote sensing image.

3. The method according to claim 2, characterized in that The manner of downsampling the key of the first remote sensing image is the same as the manner of downsampling the value of the first remote sensing image. The tree-structured wavelet frequency selection layer is used to downsample the key and value of the first remote sensing image in the following manner to obtain the downsampled key and value of the first remote sensing image: Performing an L-level two-dimensional discrete wavelet transform on the key of the first remote sensing image to obtain a first frequency feature set, wherein the first frequency feature set includes a plurality of frequency features; According to the resolution of the frequency features, the multiple frequency features are constructed into a tree-like wavelet structure, wherein the tree-like wavelet structure includes a root and multiple branches, the frequency features in each branch are different, and the frequency features in each branch are respectively located at different multiple levels, and the frequency features with the same resolution are located at the same level, and each level corresponds to a wavelet coefficient threshold; Starting from the tree root, traversing the multiple frequency features in the tree-like wavelet structure, and in the traversal process, cutting off the frequency features where the noise interference is located, so as to obtain a modified tree-like wavelet structure; Based on the modified tree-like wavelet structure, obtaining a downsampled key of the first remote sensing image; Wherein, L is an integer greater than 2.

4. The method according to claim 3, characterized in that The multi-head attention layer of the lifting scheme is also used to obtain a query of the first remote sensing image, The multi-head attention layer of the lifting scheme is used to implement the second feature obtained based on the first remote sensing image by using the multi-head attention mechanism in the following manner: Processing the query of the first remote sensing image, the key after downsampling of the first remote sensing image, and the value after downsampling of the first remote sensing image using a multi-head attention mechanism to obtain a plurality of head features of the first remote sensing image, wherein the multi-head attention mechanism includes a plurality of heads, and an output of each of the heads is one of the head features; The plurality of head features are spliced ​​in the channel dimension to obtain the second feature.

5. The method according to any one of claims 1 to 4, characterized in that: The lifting scheme multi-head attention layer is used to implement the first remote sensing image based on the first remote sensing image, adopting the lifting scheme in the channel dimension to obtain the first feature distributed along the channel dimension in the following manner: Acquire a fourth feature, wherein the fourth feature is obtained by converting the first remote sensing image into a one-dimensional sequence, obtaining a first sequence, and then performing layer normalization processing on the first sequence; Based on the parity of the channel of the fourth feature, the fourth feature is subjected to splitting processing, prediction processing and update processing to obtain a first prediction result and a first update result, wherein the first prediction result is an output of the prediction processing, and the first update result is an output of the update processing; The first feature is determined based on the first prediction result and the first update result.

6. The method according to claim 5, characterized in that The method of performing splitting processing, prediction processing and updating processing on the fourth feature based on the parity of the channel to obtain a first prediction result and a first update result includes: The following formula is used to implement the splitting process of the fourth feature: x odd ,x even =Split(x n ) Among them, x odd is the odd channel feature in the fourth feature, x even is the even channel feature in the fourth feature, x n For the fourth feature, Split(x n ) is to split the fourth feature; The following formula is used to implement the prediction process of the fourth feature: I H =x odd -P(x even )=x odd -Tanh(conv(x even )) Among them, I H is the first prediction result, P is the prediction operator, Tanh() is the Tanh nonlinear activation function, conv() is the convolution operation, Tanh(conv(x even )) is to perform convolution processing on the even channel feature in the fourth feature and then input it into the Tanh nonlinear activation function; The following formula is used to implement the updating process of the fourth feature: I L =x even +U(x odd )=x even +Tanh(conv(x odd )) Among them, I L is the first update result, U is the update operator, Tanh(conv(x odd )) is to perform convolution processing on the odd channel feature in the fourth feature and then input it into the Tanh nonlinear activation function; The determining the first feature based on the first prediction result and the first update result includes: The following formula is used to implement the determination of the first feature based on the first prediction result and the first update result: Y=Concat(I L ,I H ) Wherein, Y is the first feature, Concat(I L , I H ) is to concatenate the first prediction result and the second prediction result in the channel dimension.

7. The method according to any one of claims 2 to 4, characterized in that: The first Transformer further includes a feedforward network layer, the output of the multi-head attention layer of the lifting scheme is connected to the input of the feedforward network layer, and the feedforward network layer is used to generate the first feature map based on the third feature using the following formula; x out =FFN(IDWT(K w ,In w )+z) Among them, x out is the output of the feedforward network layer, the output of the feedforward network layer is used to expand and rearrange to obtain the first feature map, FFN () is the feedforward network layer, K w is the key after the first remote sensing image is downsampled, V w is the value of the first remote sensing image after downsampling, IDWT(K w , V w ) is a two-dimensional discrete wavelet inverse transform performed on the key after downsampling of the first remote sensing image and the value after downsampling of the first remote sensing image, and z is the third feature.

8. A remote sensing image classification device, characterized in that: The device comprises: An acquisition module, used for acquiring a first remote sensing image; An input module, used for inputting the first remote sensing image into a multi-level Transformer structure to obtain a first feature map of the first remote sensing image, each level of Transformer is used to generate a first feature map, the first Transformer includes a lifting scheme multi-head attention layer, the first Transformer is any level of Transformer in the multi-level Transformer structure, the lifting scheme multi-head attention layer is used to adopt a lifting scheme in the channel dimension based on the first remote sensing image to obtain a first feature distributed along the channel dimension, based on the first feature, determine a first weight, based on the first remote sensing image, adopt a multi-head attention mechanism to obtain a second feature, and based on the first weight, weight the second feature to obtain a third feature, and the third feature is used to generate a first feature map of the first Transformer; A classification module is used to classify the first remote sensing image based on the first feature map output by the last level Transformer in the multi-level Transformer structure.

9. A computer device, characterized in that: The computer device comprises: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the method according to any one of claims 1 to 7.