Multi-layer fusion picture classification method and system based on long-range dependence mechanism

Abstract

The application discloses a kind of multi-layer fusion picture classification method and system based on long-distance dependence mechanism, method includes: using multi-core grouping convolution to construct multi-scale pyramid feature extraction model, carries out multi-scale feature extraction;Adopt long-distance attention mechanism, utilize the multi-head attention mechanism selected by step in visual converter, dynamically learn position information and multilayer perceptron, extract long dependence context information;Adopt convolution and pooling, unify different hierarchical feature scales in middle;Using visual converter attention mechanism, fuse the features of different levels in middle;The feature after fusion in middle is used as guidance, and the global feature is learned using distillation loss guidance;According to the global feature learned, final picture classification is carried out.The application can effectively handle the classification problem of high inter-class similarity, large intra-class difference and different target scale, improve the classification accuracy and classification speed of small sample picture.

Description

Technical Field

[0001] This invention relates to remote sensing scene classification technology, and in particular to a multi-layer fusion image classification method and system based on a long-distance dependency mechanism. Background Technology

[0002] Remote sensing image scene classification refers to inferring semantic labels based on the content of remote sensing scenes. It has become a research hotspot and important topic in the field of remote sensing image interpretation in recent years. It plays a crucial role in land resource planning, urban planning, and traffic control. Most remote sensing scene images are taken by satellites, and with the continuous advancement of remote sensing imaging technology, these images often contain complex and diverse terrain features. This presents a classification challenge due to high inter-class similarity, large intra-class differences, and significant scale variations in remote sensing images.

[0003] In recent years, deep learning methods have made rapid progress in the field of remote sensing image scene classification, resulting in increasingly rich feature representations. Deep feature extraction methods using neural networks have gradually replaced earlier manually designed methods, focusing on extracting richer feature representations and exploring feature fusion and enhancement. Xiaoqiang Lu et al. [Xiaoqiang Lu et al., “A feature aggregation convolutional neural network for remote sensing scene classification,” IEEE Trans. Geosci. Remote. Sens., vol. 57, no. 10, pp. 7894–7906, 2019.] discovered that features from different layers in a convolutional neural network contain different spatial and semantic information. By aggregating intermediate features (convolutional features and fully connected layer features), they significantly improved the accuracy of remote sensing classification. Shi Cuiping et al. [A Multi-Branch Feature Fusion Remote Sensing Scene Image Classification Method Based on Attention Mechanism: CN202110192358.9 [P]. 2021-05-28.] proposed a strategy for fusing multi-branch features. After extracting feature information from multiple branches, an attention mechanism is used to further extract the branch information, and finally, multiple feature segments are fused. With the advancement of deep learning, visual transducers have emerged and achieved outstanding results in natural language processing. Subsequently, they have been widely applied to visual tasks. Hao Siyuan et al. [A Two-Stream SwinTransformer Remote Sensing Scene Classification Method: CN202210372827.X [P]. 2022-08-23.] extracted the original features and edge features respectively using two Swin Transformers with identical structures, and then fused the two features to improve classification performance.

[0004] The methods described above consider the complexity of remote sensing image scene classification from different perspectives. The first two methods enhance the feature representation capabilities of convolutional networks by utilizing feature fusion and attention mechanisms, while the last method uses a visual converter to capture long-range dependencies. However, they do not effectively combine the advantages of both methods and fail to compensate for the shortcomings of convolutional neural networks in capturing global features using long-range feature capture. Summary of the Invention

[0005] The purpose of this invention is to propose a multi-layer fusion image classification method and system based on long-distance dependency mechanism. It makes full use of the rich semantic features of the intermediate layer and the long-distance dependency relationship to handle the classification problem of high inter-class similarity, large intra-class difference and large scale difference in image classification, and has excellent classification performance.

[0006] The technical solution to achieve the purpose of this invention is as follows: In a first aspect, this invention provides a multi-layer fusion image classification method based on a long-distance dependency mechanism, comprising the following steps:

[0007] The first step is to construct a data preprocessing module using downsampling and layer normalization to achieve image preprocessing;

[0008] The second step involves using multi-kernel grouped convolutional blocks to group the feature maps, with each group independently using different convolutional kernels, to construct a multi-scale pyramid feature extraction module and achieve multi-scale feature extraction.

[0009] The third step is to take the three middle layers of features extracted by the feature learning module in the second step as input and send them to the long-distance attention mechanism module. The long-distance dependency features of each layer are learned by using the multi-head attention mechanism selected by step size, dynamic position learning and multilayer perceptron in the visual converter.

[0010] The fourth step is to construct a scale normalization module using convolution and pooling operations to normalize the scale of long-distance dependency feature maps in the intermediate layers and unify them to the same size features.

[0011] The fifth step involves the multi-layer feature fusion module using the attention mechanism principle of the visual converter to calculate the similarity between features of the same size in the first and second layers, and adding them to the features in the third layer to obtain the fused features.

[0012] In the sixth step, the classification module sequentially performs pooling, L2 regularization, full connection, and Softmax classifier on the fused features to obtain various classification scores. It also sequentially performs pooling, full connection, and Softmax classifier on the global features to obtain various classification scores. Furthermore, it calculates the hash relationship between the global features and the fused features through knowledge distillation loss, updates the loss, guides the global features to learn, and ultimately obtains the final classification.

[0013] Furthermore, in the first step, a data preprocessing module is used, which includes downsampling and layer normalization operations, to achieve data preprocessing. The specific process is as follows:

[0014] First, the image H×W×N is preprocessed, where H represents the image height, W represents the image width, and N represents the number of channels. Then, a convolutional layer with a kernel size of 7*7, a number of convolutions of size C, and a stride of 2 is passed through it. Layer normalization is then performed to achieve a 2x downsampling operation, and the image is adjusted to H / 2×W / 2×C, where C is the feature dimension.

[0015] Furthermore, the features obtained from the first deep preprocessing step are used as input, and a multi-scale pyramid structure is constructed using multi-kernel grouped convolutional blocks to extract intermediate features at multiple scales. The specific process is as follows:

[0016] (1) The composition of a multi-kernel grouped convolutional block is as follows:

[0017] First, a convolutional layer with a 1x1 kernel, a stride of 1, and padding of 0 is used to adjust the dimension, followed by normalization and activation. Next, grouped convolutions are performed, where the feature maps are grouped, each group using an independent convolutional kernel with a stride of 2, and then the feature maps from each group are concatenated, followed by normalization and activation. A 1x1 convolutional layer is then used to restore the input dimension to four times its original size, followed by normalization and activation. Finally, the output is added to the initial input tensor, which has been downsampled by pooling, processed by a 1x1 convolutional layer, and normalized, and adjusted to four times its original size.

[0018] (2) A total of four stages of multi-kernel grouped convolutional block stacking are performed. In the first stage, the multi-kernel grouped convolutional blocks are stacked 3 times, divided into 4 groups {G1,G2,G3,G4}, with a kernel size of {K1}. 2 K2 2 K3 2 K4 2 The output size is H / 4×W / 4×4C; in the second stage, it goes through a one-dimensional adjustment convolutional layer with a dimension of 2C, and multi-kernel grouped convolutional blocks are stacked 4 times, divided into 3 groups {G1,G2,G3}, with a kernel size of {K1}. 2 K2 2 K3 2 The output size is H / 8×W / 8×8C; in the third stage, after a one-dimensional adjustment convolutional layer with a dimension of 4C, multi-kernel grouped convolutional blocks are stacked 6 times, divided into 2 groups {G1,G2}, with a kernel size of {K1}. 2 K2 2The output size is H / 16×W / 16×16C; in the fourth stage, after a one-dimensional adjustment convolutional layer with a dimension of 8C, the multi-kernel grouped convolutional blocks are stacked 3 times and divided into 1 group G1 with a kernel size of 3*3 and an output size of H / 32×W / 32×32C.

[0019] The outputs of stages one, two, and three in this step serve as inputs to the long-distance attention mechanism module, while the fourth stage serves as input to the global feature classification module.

[0020] Furthermore, the features extracted in stages one, two, and three of the second-step multi-scale pyramid feature extraction module are used as input. Utilizing the multi-head attention and multi-layer perceptron of the visual converter, long-distance dependency features at each level are learned. The specific process is as follows:

[0021] (1) Flatten and regularize the feature map of size H / 4×W / 4×4C to become (H / 4·W / 4)×4C. Then perform size deformation, i.e., select the feature map according to the step size S, to obtain H·W / (16·S). 2 )×S 2 ×4C, then through a fully connected layer with multi-head attention and size deformation, we obtain (H / 4·W / 4) / S 2 ×n×S 2 ×4C / n, where n is the number of multi-head attention; next is (H / 4·W / 4) / S 2 ×n×S 2 ×4C / n and (H / 4·W / 4) / S 2 ×n×4C / n×S 2 Multiplying the tensors yields the feature size as (H / 4·W / 4) / S. 2 ×n×S 2 ×S 2 ;

[0022] (2) Simultaneously generate a window of [1-s,s] to learn dynamic position information, and then perform (H / 4·W / 4) / S again. 2 ×n×S 2 ×S 2 With (H / 4·W / 4) / S 2 ×n×S 2 Multiplying the tensor by ×4C / n and then performing dimensional deformation yields a dimension of (H / 4·W / 4)×4C;

[0023] (3) Then it is added to the initial flattened layer and fed into a multilayer sensor, passing through a fully connected layer, an activation layer, a dropout layer, a fully connected layer, and a dropout layer in sequence to obtain a size of (H / 4·W / 4)×4C.

[0024] (4) Finally, the (H / 4·W / 4)×4C dimension is transformed into H / 4×W / 4×4C, and then added to the output feature H / 4×W / 4×4C of the first stage to finally obtain the output H / 4×W / 4×4C.

[0025] (5) The steps in the second and third stages are the same as those in the first stage.

[0026] Furthermore, the long-range dependency features extracted in the third step are used as input, and a scale normalization module is constructed using convolution and pooling operations to normalize the scale of the long-range dependency feature maps of the first three stages of the intermediate layer, unifying them to the same size. The specific process is as follows:

[0027] (1) First, a 1*1 convolution is used to uniformly adjust the feature dimensions of the three stages to 8C. At this time, the size is H / 4×W / 4×8C, H / 8×W / 8×8C, and H / 16×W / 16×8C respectively.

[0028] (2) Next, using 4*4 and 2*2 pooling kernels respectively, pooling operations are performed on features with size H / 4×W / 4×8C and H / 8×W / 8×8C to unify them into features with size H / 16×W / 16×8C.

[0029] Furthermore, in the fifth step, the multi-layer feature fusion module mainly consists of flattening, size deformation, fully connected layers, and tensor multiplication and addition. Utilizing the principle of a visual converter, it calculates the similarity relationship between features of the same size in the first and second layers, and adds it to the features of the third layer to obtain the fused features. The specific process is as follows:

[0030] (1) Flatten the H / 16×W / 16×8C features of the first and second stages after scaling to become (H / 16·W / 16)×8C;

[0031] (2) Next, the features flattened in the first stage are fed into the fully connected layer of the multi-head attention and the size deformation is performed to obtain a tensor q1 with a size of (H / 16·W / 16)×n×8C / n, where n is the number of multi-head attention; the features flattened in the second stage are fed into the multi-head attention and the full connection and size deformation are performed twice to obtain two tensors k1 and v1 with a size of (H / 16·W / 16)×n×8C / n.

[0032] (3) Multiply q1 and k1 by tensor, then multiply by v1 through the Softmax activation function and the dropout layer, and adjust the size deformation to H / 16×W / 16×8C through the fully connected layer, the dropout layer and the size deformation.

[0033] (4) Finally, add the H / 16×W / 16×8C feature from the third stage to obtain the final fusion feature H / 16×W / 16×8C.

[0034] Furthermore, the classification module sequentially performs pooling, L2 regularization, fully connected processing, and a Softmax classifier on the features fused in step five to obtain classification scores for each category. It also sequentially performs pooling, fully connected processing, and a Softmax classifier on the global features from stage four of step two to obtain classification scores for each category. Finally, it calculates the hash relationship between the global features and the aggregated features using knowledge distillation loss, updates the loss, and guides the global features in learning, thereby obtaining the final classification. The specific process is as follows:

[0035] (1) Using the output of step 5 as input, perform 1*1 pooling, L2 regularization, and a fully connected layer sequentially to obtain the network output logits. t A Softmax classifier is then connected to it, and the classification scores for each class are obtained under the parameter T=t. tsoft The calculation formula is as follows;

[0036]

[0037] Where z i The logits value for the i-th class is given by cn, the total number of output classes is given by T, and p is a parameter. i Let be the probability distribution of the i-th class;

[0038] (2) Using the output of the second step as input, perform 1*1 pooling and fully connected layer operations sequentially to obtain the network output logits. s A Softmax classifier is then connected to it, and the classification scores S for each class are obtained under parameters T=t and T=1, respectively. ssoft S shard ;

[0039] (3) S tsoft With S ssoft The loss_KD is obtained by performing KL divergence calculation, and the calculation formula is as follows;

[0040] Where U and V are probability distributions;

[0041] (4) The overall loss function is calculated as follows. The loss is updated accordingly to guide the learning of global features and obtain the final classification.

[0042] loss=(1-λ)×S shard +λ×T×T×loss_KD, where λ and T are parameters.

[0043] Secondly, the present invention provides a multi-layer fusion image classification system based on a long-distance dependency mechanism, comprising:

[0044] The data preprocessing module utilizes downsampling and layer normalization to construct the data preprocessing module and achieve image preprocessing;

[0045] The multi-scale pyramid feature extraction module will use multi-kernel grouped convolutional blocks to group the feature maps and use different convolutional kernels for each group to achieve multi-scale feature extraction.

[0046] The long-range attention mechanism module utilizes the multi-head attention mechanism based on long-range window selection, dynamic position learning, and multilayer perceptron in the visual converter to learn the long-range dependency features at each level.

[0047] The scale normalization module utilizes convolution and pooling operations to construct a scale normalization module, which normalizes the long-distance dependency feature maps of the intermediate layers to the same size feature.

[0048] The multi-layer feature fusion module is mainly composed of fully connected layers. It uses the attention mechanism principle of the visual converter to calculate the similarity between features of the same size in the first and second layers, and adds them to the features in the third layer to obtain the fused features.

[0049] The classification module uses the fused features as guidance and calculates the hash relationship between the global features and the fused features through knowledge distillation loss. It updates the loss, guides the global features to learn, and then performs the final image classification.

[0050] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0051] Fourthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0052] Compared with existing technologies, the significant features of this invention are: (1) a multi-core grouped convolutional network is constructed to form a multi-scale pyramid network, and feature extraction is performed through pre-training, resulting in richer feature extraction; (2) a long-distance attention mechanism module utilizes the multi-head attention mechanism based on long-distance step size selection, dynamic position learning, and multilayer perceptron in the visual converter to obtain long-distance dependencies at each level; (3) the principle of the visual converter's attention mechanism is used to calculate the dependencies between different layers to obtain fused features; (4) the fused features are used as guidance, and the hash relationship between global features and fused features is calculated through knowledge distillation loss to update the loss, guide global features to learn, and achieve final image classification. The long-distance dependency capture capability of the visual converter is combined with the rich semantic information of the intermediate layers of the convolutional neural network to improve classification accuracy.

[0053] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0054] Figure 1 This is a structural diagram of the method of the present invention.

[0055] Figure 2 This is a diagram of the structure of a multi-core grouped convolutional block.

[0056] Figure 3 It is a structural diagram of learning location information.

[0057] Figure 4 This is a diagram of a multilayer perceptron structure.

[0058] Figure 5 This is a classification confusion matrix diagram of the AID30 dataset with 50% partitioning by the method of this invention.

[0059] Figure 6 This is a classification confusion matrix diagram of the NWPU-RESISC45 dataset with 20% partitioning by the method of this invention. Detailed Implementation

[0060] Compared to existing methods in the background art, this invention proposes a multi-layer fusion image classification method and system based on a long-distance dependency mechanism. It applies a pre-trained multi-scale pyramid convolutional network as a feature extractor, combining a visual converter's multi-head attention mechanism based on long-distance window selection, dynamic position learning, and a multi-layer perceptron to extract long-distance semantic relationships in the intermediate layers. The intermediate layer features are then fused through the visual converter, and distillation loss is used to guide global feature learning for final classification, thereby improving classification accuracy.

[0061] The following is combined with Figure 1 The implementation process of the present invention will be described in detail.

[0062] A multi-layer fusion image classification method based on long-distance dependency mechanism includes the following steps:

[0063] The first step involves using a data preprocessing module, which includes operations such as cropping, flipping, standardization, downsampling, and layer normalization, to perform data preprocessing. The specific process is as follows:

[0064] (1) Perform data augmentation on the original image, crop it to a size of 224×224, randomly flip the image horizontally with a probability of 0.5, and then convert the image data into a standard normal distribution mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225]. The standardization formula is as follows:

[0065] input[channel]=(input[channel]-mean[channel]) / std(channel)

[0066] For an image of 224×224×3, a convolutional layer with a kernel size of 7*7, a number of convolutions of 64, and a stride of 2 is passed through it, and layer normalization is performed to achieve a 2x downsampling operation, which adjusts the image to 112×112×64, where 64 is the feature dimension;

[0067] The second step involves using the features from the deep preprocessing in the first step as input, and constructing a multi-scale pyramid structure using multi-kernel grouped convolutional blocks to extract intermediate features at multiple scales. The following section combines... Figure 2 Explain the specific process:

[0068] (1) The composition of a multi-kernel grouped convolutional block is as follows:

[0069] First, a 1x1 convolutional layer with a stride of 1 and zero padding is used to adjust the dimension, followed by normalization and activation. Next, grouped convolutions are performed, where the feature maps are grouped, each group using an independent convolutional kernel with a stride of 2. The feature maps from each group are then concatenated, followed by normalization and activation. Finally, a 1x1 convolutional layer is used to restore the input dimension to four times its original size, again followed by normalization and activation. Finally, this is added to the initial input tensor, which has been downsampled by pooling layers, processed by the 1x1 convolutional layer, and normalized, and adjusted to four times its original size, to obtain the final output.

[0070] (2) A total of four stages of multi-kernel grouped convolutional block stacking are performed. In the first stage, the multi-kernel grouped convolutional blocks are stacked 3 times, divided into 4 groups {1, 4, 8, 16}, with kernel sizes of {3*3, 5*5, 7*7, 9*9} and output sizes of 56×56×256. In the second stage, after passing through a one-dimensional adjustment convolutional layer with a dimension of 128, the multi-kernel grouped convolutional blocks are stacked 4 times, divided into 3 groups {1, 4, 8}, with kernel sizes of {3*3, 5*5, 7*7}. The output size is 28×28×512; in the third stage, after passing through a one-dimensional adjustment convolutional layer with a dimension of 256, multi-kernel grouped convolutional blocks are stacked 6 times, divided into 2 groups {1,4}, with kernel size {3*3,5*5}, and the output size is 14×14×1024; in the fourth stage, after passing through a one-dimensional adjustment convolutional layer with a dimension of 512, multi-kernel grouped convolutional blocks are stacked 3 times, divided into 1 group {1}, with kernel size 3*3, and the output size is 7×7×2048;

[0071] The outputs of stages one, two, and three in this step serve as inputs to the long-distance attention mechanism module, while the fourth stage serves as input to the global feature classification module.

[0072] The third step involves using the features extracted in stages one, two, and three of the multi-scale pyramid feature extraction module in the second step as input. Then, leveraging the multi-head attention and multi-layer perceptron of the visual converter, long-distance dependency features at each level are learned. The specific process is as follows:

[0073] (1) The 56×56×256 feature map extracted in the first stage of the multi-scale pyramid feature extraction module is flattened and regularized to become 3136×256. Then, the size is deformed, i.e., the feature map is selected according to the window stride size of 4, resulting in 196×16×256. Then, through the fully connected layer of multi-head attention and size deformation, 196×4×16×64 is obtained, where 4 is the number of multi-head attention. Next, the tensors 196×4×16×64 and 196×4×64×16 are multiplied to obtain the feature size of 196×4×16×16.

[0074] (2) Simultaneously generate a window of [-3,4] to learn dynamic position information. Multiply the 196×4×16×16 and 196×4×16×64 tensors again and perform size deformation to obtain a size of 3136×256. The dynamic position information learning structure diagram is as follows. Figure 3 As shown;

[0075] (3) Then it is added to the initial flattened layer and fed into the multilayer sensor, passing through the fully connected layer, activation layer, discard layer, fully connected layer and discard layer in sequence to obtain a size of 3136×256.

[0076] (4) Finally, the 3136×256 size is transformed into 56×56×256, and then added to the output feature of the first stage, 56×56×256, to finally obtain the output 56×56×256.

[0077] (5) The 28×28×512 feature map extracted in the second stage of the multi-scale pyramid feature extraction module is flattened and regularized to become 784×512. Then, the size is deformed, i.e., the feature map is selected according to the stride size of 4, to obtain 49×16×512. Then, through the fully connected layer of multi-head attention and size deformation, 49×4×16×128 is obtained, where 4 is the number of multi-head attention. Next, the tensors 49×4×16×128 and 49×4×128×16 are multiplied to obtain the feature size of 49×4×16×16.

[0078] (6) Simultaneously generate a window of [-3,4] to learn dynamic position information, multiply the tensors of 49×4×16×16 and 49×4×16×128 again and perform size deformation to obtain a size of 784×512.

[0079] (7) Then, it is added to the initial flattened result and fed into a multilayer sensor, such as... Figure 4As shown, by sequentially passing through a fully connected layer, an active layer, a dropout layer, a fully connected layer, and a dropout layer, a size of 784×512 is obtained;

[0080] (8) Finally, the 784×512 size is transformed into 28×28×512, and then added to the output feature of the second stage, 28×28×512, to finally obtain the output 28×28×512.

[0081] (9) The 14×14×1024 feature map extracted in the third stage of the multi-scale pyramid feature extraction module is flattened and regularized to become 196×1024. Then, the size is deformed, i.e., the feature map is selected according to the window stride size of 7, to obtain 4×49×1024. Then, through the fully connected layer of multi-head attention and size deformation, 4×4×49×256 is obtained, where 4 is the number of multi-head attention. Next, the tensors 4×4×49×256 and 4×4×256×49 are multiplied to obtain the feature size of 4×4×49×49.

[0082] (10) Simultaneously generate a window of [-6,7] to learn dynamic position information, multiply the tensors of 4×4×49×49 and 4×4×49×256 again and perform size deformation to obtain a size of 196×1024.

[0083] (11) Then it is added to the initial flattened layer and fed into a multilayer perceptron, passing through a fully connected layer, an activation layer, a discard layer, a fully connected layer, and a discard layer in sequence to obtain a size of 196×1024.

[0084] (12) Finally, the 196×1024 size is transformed into 14×14×1024, and then added to the output feature of the three stages, 14×14×1024, to finally obtain the output 14×14×1024.

[0085] The fourth step involves using the long-range dependency features extracted in the third step as input, and constructing a scale normalization module using convolution and pooling operations. This module normalizes the scale of the long-range dependency feature maps from the first three stages of the intermediate layer, unifying them to the same size. The specific process is as follows:

[0086] (1) First, a 1*1 convolution is used to uniformly adjust the feature dimensions of the three stages to 512. At this time, the size is 56×56×512, 28×28×512, and 14×14×512 respectively.

[0087] (2) Next, using 4*4 and 2*2 pooling kernels respectively, pooling operations are performed on features with size 56×56×512 and 28×28×512 to unify them into features with size 14×14×512.

[0088] The fifth step, the multi-layer feature fusion module, mainly consists of flattening, size deformation, fully connected layers, and tensor multiplication and addition. Utilizing the principle of a visual converter, it calculates the similarity relationship between features of the same size in the first and second layers, and adds this similarity relationship to the features in the third layer to obtain the fused features. The specific process is as follows:

[0089] (1) Flatten the 14×14×512 features of the first and second stages after scaling to 196×512.

[0090] (2) Next, the features flattened in the first stage are fed into a fully connected layer of multi-head attention and subjected to size deformation to obtain a tensor q1 with a size of 196×1×256, where 1 is the number of multi-head attention. The features flattened in the second stage are fed into multi-head attention and subjected to two full connections and size deformations to obtain two tensors k1 and v1 with a size of 196×1×256.

[0091] (3) Multiply q1 and k1 by tensor, then multiply by v1 through the Softmax activation function and the dropout layer, and adjust the size to 14×14×512 through the fully connected layer, the dropout layer and the size deformation.

[0092] (4) Finally, add it to the 14×14×512 feature of the third stage to obtain the final fused feature 14×14×512.

[0093] In step six, the classification module sequentially performs pooling, L2 regularization, fully connected processing, and a Softmax classifier on the features fused in step five to obtain classification scores for each category. It also sequentially performs pooling, fully connected processing, and a Softmax classifier on the global features from stage four of step two to obtain classification scores for each category. Furthermore, it calculates the hash relationship between the global features and the aggregated features using knowledge distillation loss, updates the loss, and guides the global features in learning, ultimately obtaining the final classification. The specific process is as follows:

[0094] (1) Using the output of step 5 as input, perform 1*1 pooling, L2 regularization, and a fully connected layer sequentially to obtain the network output logits. t A Softmax classifier is then connected to it, and the classification scores for each class are obtained at T=15. tsoft The calculation formula is as follows;

[0095]

[0096] Where z i The logits value for the i-th class is given by cn, the total number of output classes is given by T, and p is a parameter. i Let be the probability distribution of the i-th class;

[0097] (2) Using the output of the second step as input, perform 1*1 pooling and fully connected layer operations sequentially to obtain the network output logits. s A Softmax classifier is then connected to it, and the classification scores for each class are obtained at T=15 and T=1, respectively. ssoft S shard ;

[0098] (3) S tsoft With S ssoft The loss_KD is obtained by performing KL divergence calculation, and the calculation formula is as follows;

[0099] Where U and V are probability distributions;

[0100] (4) The overall loss function is calculated as follows. The loss is updated accordingly to guide the learning of global features and obtain the final classification.

[0101] loss=(1-λ)×S shard +λ×T×T×loss_KD, where the λ parameter is set to 0.2 and the T parameter is set to 15.

[0102] Based on the same concept, the present invention also provides a multi-layer fusion image classification system based on a long-distance dependency mechanism, comprising:

[0103] The data preprocessing module utilizes downsampling and layer normalization to construct the data preprocessing module and achieve image preprocessing;

[0104] The multi-scale pyramid feature extraction module will use multi-kernel grouped convolutional blocks to group the feature maps and use different convolutional kernels for each group to achieve multi-scale feature extraction.

[0105] The long-range attention mechanism module utilizes the multi-head attention mechanism based on long-range window selection, dynamic position learning, and multilayer perceptron in the visual converter to learn the long-range dependency features at each level.

[0106] The scale normalization module utilizes convolution and pooling operations to construct a scale normalization module, which normalizes the long-distance dependency feature maps of the intermediate layers to the same size feature.

[0107] The multi-layer feature fusion module is mainly composed of fully connected layers. It uses the attention mechanism principle of the visual converter to calculate the similarity between features of the same size in the first and second layers, and adds them to the features in the third layer to obtain the fused features.

[0108] The classification module uses the aggregated features as guidance and calculates the hash relationship between the global features and the fused features through knowledge distillation loss. It updates the loss, guides the global features to learn, and then performs the final image classification.

[0109] The specific implementation methods of the above modules correspond to the first to sixth steps of the aforementioned image classification method, and will not be repeated here.

[0110] Furthermore, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned image classification method.

[0111] Furthermore, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned image classification method.

[0112] This invention employs a long-range dependency mechanism, fully extracting the rich long-range dependency features from the intermediate layers of the convolutional neural network. It then integrates these features using the attention mechanism of the visual converter and effectively guides global learning of contextual features. By using a pre-trained network for feature extraction, the feature learning capability is enhanced, effectively solving the problem of insufficient global feature extraction and improving the classification accuracy and speed of images.

[0113] The effects of this invention can be further illustrated by the following simulation experiments:

[0114] Simulation conditions

[0115] The simulation experiment used two sets of optical remote sensing image data: the AID dataset and the NWPU-RESISC45 dataset. All images in the AID dataset were released by Wuhan University and Huazhong University of Science and Technology, containing 30 scene classes with approximately 220-420 images per class, totaling 10,000 images, used as a 50% training set. The NWPU-RESISC45 dataset was created by Northwestern Polytechnical University, containing 31,500 images covering 45 scene classes, with 700 images per class. The data covers more than 100 countries and regions globally, making it quite large. Except for islands, lakes, and mountains with lower spatial resolution, the spatial resolution of most scene classes ranges from 0.2 to 30 meters. This dataset also considers the influence of different weather, seasons, lighting, and other natural conditions, resulting in rich image variations in background and occlusion. Using a 20% training set, the training and test sets contain 6300 and 25200 images respectively, with each image being 256×256 pixels in size. In the experiments, all images from the AID and NWPU-RESISC45 datasets were resized to 224*224. Overall classification accuracy was used as the evaluation metric for both sets of experiments. Furthermore, the comparison methods included: basic convolutional neural networks such as GoogLeNet, VGG_16, PyConvResNet, Self-attention-based deep feature fusion (SAFF), Capsule Networks (CapsNet), Attention Consistent Networks (ACNet), Vision Transformers (ViT), and Pyramid Vision Transformers (PVT).

[0116] In the experiments, the Adam optimizer was used for the deep separable convolutional feature learning network with an initial learning rate of 0.001, which was divided by 10 after 60 epochs. The momentum and weight decay rates were 0.9 and 1e-4, respectively. Furthermore, the network was trained for the first 50 epochs of AID and NWPU-RESISC45. Other network hyperparameter configurations are summarized in Table 1. Simulation experiments were all conducted on a Linux operating system using Python 3.8 + PyTorch 1.8 + CUDA 11.2.

[0117] Table 1 Network Hyperparameter Configuration

[0118]

[0119]

[0120]

[0121]

[0122]

[0123] Analysis of Simulation Experiment Results

[0124] Tables 2 and 3 show the classification accuracy (%) of the method of the present invention in simulation experiments on the NWPU-RESISC45 and AID datasets.

[0125] Table 2 Classification results of different methods on the AID dataset

[0126]

[0127] Table 3 Classification results of different methods on the NWPU-RESISC45 dataset

[0128]

[0129]

[0130] The experimental results show that the method of this invention can significantly improve the classification accuracy of both datasets. On the AID dataset with a 50% split, the classification accuracy of the method of this invention is 97.40 ± 0.19%. The classification confusion map obtained by the method of this invention is shown below. Figure 5 As shown. On the 20% split of the AID dataset, the classification accuracy of the method of this invention is 95.25±0.13%. Compared with other methods, the method of this invention has made a significant breakthrough in improving the classification accuracy of the more complex target class, such as resorts. This is due to the combination of the long-distance attention mechanism module in this invention, which can extract long-distance dependency information more comprehensively. On the 20% split of the NWPU-RESISC45 dataset, the average accuracy of the method of this invention is 94.65±0.16%. The classification confusion map obtained by the method of this invention is shown below. Figure 6 As shown, on the 10% split of the NWPU-RESISC45 dataset, the average accuracy of the method of this invention is 92.93 ± 0.12%. Compared with other methods, the method of this invention achieves better classification results, mainly due to multi-layer feature fusion using the attention mechanism of the visual converter, and the use of multi-layer feature fusion to guide global feature extraction. The above results fully demonstrate that the method of this invention can effectively learn the depth feature information of remote sensing images, improve the classification accuracy of remote sensing scene images, and has high classification performance.

Claims

1. A multi-layer fusion image classification method based on a long-distance dependency mechanism, characterized in that, Includes the following steps: The first step is to construct a data preprocessing module using downsampling and layer normalization to achieve image preprocessing; The second step involves using multi-kernel grouped convolutional blocks to group the feature maps and use different convolutional kernels for each group to construct a multi-scale pyramid feature extraction module, thereby achieving multi-scale feature extraction. The third step is to take the three middle layers of features extracted by the feature learning module in the second step as input and send them to the long-distance attention mechanism module. The long-distance dependency features of each layer are learned by using the multi-head attention mechanism selected by step size, dynamic position learning and multilayer perceptron in the visual converter. The fourth step is to construct a scale normalization module using convolution and pooling operations to normalize the scale of long-distance dependency feature maps in the intermediate layers and unify them to the same size features. The fifth step involves the multi-layer feature fusion module using the attention mechanism principle of the visual converter to calculate the similarity between features of the same size in the first and second layers, and adding them to the features in the third layer to obtain the fused features. In the sixth step, the classification module sequentially performs pooling, L2 regularization, full connection, and Softmax classifier on the fused features to obtain various classification scores. It also sequentially performs pooling, full connection, and Softmax classifier on the global features to obtain various classification scores. Furthermore, it calculates the hash relationship between the global features and the fused features through knowledge distillation loss, updates the loss, guides the global features to learn, and ultimately obtains the final classification.

2. The multi-layer fusion image classification method based on long-distance dependency mechanism according to claim 1, characterized in that, The first step involves using a data preprocessing module, which includes downsampling and layer normalization operations, to perform data preprocessing. The specific process is as follows: First, the image H×W×N is preprocessed, where H represents the image height, W represents the image width, and N represents the number of channels. Then, a convolutional layer with a kernel size of 7*7, a number of convolutions of size C, and a stride of 2 is passed through it. Layer normalization is then performed to achieve a 2x downsampling operation, and the image is adjusted to H / 2×W / 2×C, where C is the feature dimension.

3. The multi-layer fusion image classification method based on long-distance dependency mechanism according to claim 2, characterized in that, Using the features obtained from the first deep preprocessing step as input, a multi-scale pyramid structure is constructed using multi-kernel grouped convolutional blocks to extract intermediate features at multiple scales. The specific process is as follows: (1) The composition of a multi-kernel grouped convolutional block is as follows: First, a convolutional layer with a 1x1 kernel, a stride of 1, and padding of 0 is used to adjust the dimension, followed by normalization and activation. Next, grouped convolutions are performed, where the feature maps are grouped, each group using an independent convolutional kernel with a stride of 2, and then the feature maps from each group are concatenated, followed by normalization and activation. A 1x1 convolutional layer is then used to restore the input dimension to four times its original size, followed by normalization and activation. Finally, the output is added to the initial input tensor, which has been downsampled by pooling, processed by a 1x1 convolutional layer, and normalized, and adjusted to four times its original size. (2) A total of four stages of multi-kernel grouped convolutional block stacking are performed. In the first stage, the multi-kernel grouped convolutional blocks are stacked 3 times, divided into 4 groups {G1,G2,G3,G4}, with a kernel size of {K1}. 2 K2 2 K3 2 K4 2 The output dimensions are H / 4×W / 4×4C. In the second stage, after a one-dimensional adjustment convolutional layer with a dimension of 2C, multi-kernel grouped convolutional blocks are stacked four times, divided into three groups {G1, G2, G3}, with a kernel size of {K1}. 2 K2 2 K3 2 The output dimensions are H / 8×W / 8×8C. In the third stage, after a one-dimensional adjustment convolutional layer with a dimension of 4C, multi-kernel grouped convolutional blocks are stacked 6 times, divided into 2 groups {G1, G2}, with a kernel size of {K1}. 2 K2 2 The output size is H / 16×W / 16×16C; in the fourth stage, after a one-dimensional adjustment convolutional layer with a dimension of 8C, the multi-kernel grouped convolutional blocks are stacked 3 times and divided into 1 group G1 with a kernel size of 3*3 and an output size of H / 32×W / 32×32C. The outputs of stages one, two, and three in this step serve as inputs to the long-distance attention mechanism module, while the fourth stage serves as input to the global feature classification module.

4. The multi-layer fusion image classification method based on long-distance dependency mechanism according to claim 3, characterized in that, The features extracted in stages one, two, and three of the second-step multi-scale pyramid feature extraction module are used as input. The multi-head attention and multi-layer perceptron of the visual converter are then used to learn the long-distance dependency features at each level. The specific process is as follows: (1) Flatten and regularize the feature map of size H / 4×W / 4×4C to become (H / 4·W / 4)×4C. Then perform size deformation, i.e., select the feature map according to the step size S, to obtain H·W / (16·S). 2 )×S 2 ×4C, then through a fully connected layer with multi-head attention and size deformation, we obtain (H / 4·W / 4) / S 2 ×n×S 2 ×4C / n, where n is the number of multi-head attention; next is (H / 4·W / 4) / S 2 ×n×S 2 ×4C / n and (H / 4·W / 4) / S 2 ×n×4C / n×S 2 Multiplying the tensors yields the feature size as (H / 4·W / 4) / S. 2 ×n×S 2 ×S 2 ; (2) Simultaneously generate a window of [1-s,s] to learn dynamic position information, and then perform (H / 4·W / 4) / S again. 2 ×n×S 2 ×S 2 With (H / 4·W / 4) / S 2 ×n×S 2 Multiplying the tensor by ×4C / n and then performing dimensional deformation yields a dimension of (H / 4·W / 4)×4C; (3) Then it is added to the initial flattened layer and fed into a multilayer sensor, passing through a fully connected layer, an activation layer, a dropout layer, a fully connected layer, and a dropout layer in sequence to obtain a size of (H / 4·W / 4)×4C. (4) Finally, the (H / 4·W / 4)×4C dimension is transformed into H / 4×W / 4×4C, and then added to the output feature H / 4×W / 4×4C of the first stage to finally obtain the output H / 4×W / 4×4C. (5) The steps in the second and third stages are the same as those in the first stage.

5. The multi-layer fusion image classification method based on long-distance dependency mechanism according to claim 4, characterized in that, Using the long-range dependency features extracted in the third step as input, a scale normalization module is constructed using convolution and pooling operations to normalize the scale of the long-range dependency feature maps in the first three stages of the intermediate layer, unifying them to the same size. The specific process is as follows: (1) First, a 1*1 convolution is used to uniformly adjust the feature dimensions of the three stages to 8C. At this time, the size is H / 4×W / 4×8C, H / 8×W / 8×8C, and H / 16×W / 16×8C respectively. (2) Next, using 4*4 and 2*2 pooling kernels respectively, pooling operations are performed on features with size H / 4×W / 4×8C and H / 8×W / 8×8C to unify them into features with size H / 16×W / 16×8C.

6. The multi-layer fusion image classification method based on long-distance dependency mechanism according to claim 5, characterized in that, The fifth step, the multi-layer feature fusion module, mainly consists of flattening, size deformation, fully connected layers, and tensor multiplication and addition. Utilizing the principle of a visual converter, it calculates the similarity relationship between features of the same size in the first and second layers, and adds this similarity relationship to the features in the third layer to obtain the fused features. The specific process is as follows: (1) Flatten the H / 16×W / 16×8C features of the first and second stages after scaling to become (H / 16·W / 16)×8C; (2) Next, the features flattened in the first stage are fed into the fully connected layer of multi-head attention and size deformation to obtain a tensor q1 with size (H / 16·W / 16)×n×8C / n, where n is the number of multi-head attention. The features flattened in the second stage are fed into multi-head attention, and two full connections and size deformations are performed to obtain two tensors k1 and v1 with a size of (H / 16·W / 16)×n×8C / n. (3) Multiply q1 and k1 by tensor, then multiply by v1 through the Softmax activation function and the dropout layer, and adjust the size deformation to H / 16×W / 16×8C through the fully connected layer, the dropout layer and the size deformation. (4) Finally, add the H / 16×W / 16×8C feature from the third stage to obtain the final fusion feature H / 16×W / 16×8C.

7. The multi-layer fusion image classification method based on long-distance dependency mechanism according to claim 6, characterized in that, The classification module sequentially performs pooling, L2 regularization, fully connected processing, and a Softmax classifier on the features fused in step five to obtain classification scores for each category. It also sequentially performs pooling, fully connected processing, and a Softmax classifier on the global features from stage four of step two to obtain classification scores for each category. Furthermore, it calculates the hash relationship between the global features and the aggregated features using knowledge distillation loss, updates the loss, and guides the global features in learning, thereby obtaining the final classification. The specific process is as follows: (1) Using the output of step 5 as input, perform 1*1 pooling, L2 regularization, and a fully connected layer sequentially to obtain the network output logits. t A Softmax classifier is then connected to it, and the classification scores for each class are obtained under the parameter T=t. tsoft The calculation formula is as follows; Where z i The logits value for the i-th class is given by cn, the total number of output classes is given by T, and p is a parameter. i Let be the probability distribution of the i-th class; (2) Using the output of the second step as input, perform 1*1 pooling and fully connected layer operations sequentially to obtain the network output logits. s A Softmax classifier is then connected to it, and the classification scores S for each class are obtained under parameters T=t and T=1, respectively. ssoft S shard ; (3) S tsoft With S ssoft The loss_KD is obtained by performing KL divergence calculation, and the calculation formula is as follows; Where U and V are probability distributions; (4) The overall loss function is calculated as follows. The loss is updated accordingly to guide the learning of global features and obtain the final classification. loss=(1-λ)×S shard +λ×T×T×loss_KD, where λ and T are parameters.

8. A multi-layer fusion image classification system based on a long-distance dependency mechanism, characterized in that, include: The data preprocessing module utilizes downsampling and layer normalization to construct the data preprocessing module and achieve image preprocessing; The multi-scale pyramid feature extraction module will use multi-kernel grouped convolutional blocks to group the feature maps and use different convolutional kernels for each group to achieve multi-scale feature extraction. The long-distance attention mechanism module takes the three middle layers of features extracted by the multi-scale pyramid feature extraction module as input, and uses the multi-head attention mechanism based on long-distance step size selection, dynamic position learning and multilayer perceptron in the visual converter to learn the long-distance dependency features of each level. The scale normalization module utilizes convolution and pooling operations to construct a scale normalization module, which normalizes the long-distance dependency feature maps of the intermediate layers to the same size feature. The multi-layer feature fusion module is mainly composed of fully connected layers. It uses the attention mechanism principle of the visual converter to calculate the similarity between features of the same size in the first and second layers, and adds them to the features in the third layer to obtain the fused features. The classification module sequentially performs pooling, L2 regularization, fully connected processing, and a Softmax classifier on the fused features to obtain various classification scores. It also sequentially performs pooling, fully connected processing, and a Softmax classifier on the global features to obtain various classification scores. Furthermore, it calculates the hash relationship between the global features and the fused features through knowledge distillation loss, updates the loss, guides the global features to learn, and ultimately obtains the final classification.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program performs the steps of the method described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-7.

Images