A remote sensing image fine-grained target recognition method based on local multi-scale super-resolution

Abstract

The present application relates to a kind of remote sensing image fine-grained target identification method based on local multi-scale super resolution, it is related to remote sensing image technical field, solve the technical problem that fine-grained target identification method precision is not high in prior art, big computing cost, comprising the following steps: constructing remote sensing image target area positioning network, original remote sensing image is input into remote sensing image target area positioning network, the rough detection of target of interest is done to input image and local area positioning;Adaptive multi-scale restoration and super-resolution reconstruction network are constructed, and the local area slice image that remote sensing image target area positioning network exports is done multi-scale restoration and super-resolution reconstruction;Multi-scale remote sensing image fine-grained identification network is constructed, and the multi-scale local slice image of high resolution is done fine-grained target identification.The method of the present application is by the multi-scale restoration and super-resolution reconstruction of local area to remote sensing image, and the computing cost is smaller, and the calculation speed is faster, can improve identification efficiency and identification precision.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image technology, and in particular to a method for fine-grained target recognition in remote sensing images based on local multi-scale super-resolution. Background Technology

[0002] Remote sensing images, with their advantages of wide coverage, diverse scenes, and short acquisition cycles, play a crucial role in intelligent interpretation in fields such as smart cities, land planning, and disaster monitoring. Fine-grained target recognition in remote sensing images, as a key technology in intelligent interpretation, is widely applied in aerospace observation, transportation planning, and national defense security. With the rapid development of deep learning technology and the arrival of the highly intelligent information age, fine-grained target recognition in remote sensing images is gradually evolving towards intelligentization. Domestic and international scholars have introduced deep learning into remote sensing image target recognition, optimizing algorithm design and model training to improve the efficiency and accuracy of target recognition.

[0003] Remote sensing images are easily affected by factors such as lighting and weather, resulting in unstable image quality, especially under conditions of low solar altitude and poor lighting. Whether it's the complex weather conditions leading to intelligent interpretation under low light or severe weather conditions, both manifest as fine-grained target recognition problems in the field of intelligent interpretation. Low-quality images suffer from significant loss of effective information regarding target contours and colors, hindering the extraction of effective features by target recognition methods. Furthermore, compared to the imaging field of remote sensing images, there are many small targets, leading to significant scale variations in the targets to be identified. Complex backgrounds in remote sensing images also negatively impact the effectiveness of fine-grained target recognition methods. Therefore, it is necessary to improve the imaging quality and resolution of remote sensing images to provide richer and more effective information for fine-grained target recognition.

[0004] Existing target recognition methods based on image super-resolution reconstruction mostly involve performing super-resolution reconstruction on the entire remote sensing image to obtain a high-resolution remote sensing image, which is then input into the target recognition network. This type of method, which enhances low-quality images before target interpretation, can provide higher accuracy in target recognition, but it requires greater computational resources and time, resulting in a higher computational cost. Furthermore, this type of method only considers target features in the super-resolution high-resolution image, neglecting target features in the original resolution image, thus having certain limitations. Summary of the Invention

[0005] This invention aims to address the technical problems of low accuracy and high computational cost in existing fine-grained target recognition methods, and provides a fine-grained target recognition method for remote sensing images based on local multi-scale super-resolution.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A method for fine-grained target recognition in remote sensing images based on local multi-scale super-resolution is described, comprising the following steps:

[0008] Step 1: Construct a remote sensing image target region localization network. Input the original remote sensing image into the remote sensing image target region localization network to perform coarse detection of the target of interest and local region localization on the input image.

[0009] Step 2: Construct an adaptive multi-scale restoration and super-resolution reconstruction network to perform multi-scale image restoration and super-resolution reconstruction on the local area slice images output by the remote sensing image target area localization network.

[0010] Step 3: Construct a multi-scale remote sensing image fine-grained recognition network to perform fine-grained target recognition on the high-resolution multi-scale local slice image output in Step 2.

[0011] In the above technical solution, in step 1,

[0012] The remote sensing image target region localization network adopts a dual-branch attention mechanism consisting of a channel attention mechanism branch and a spatial attention mechanism branch. It extracts multi-scale features from the original remote sensing image and performs weighted fusion of the multi-scale features to achieve the localization of local candidate regions in the original remote sensing image.

[0013] In the above technical solution, step 1 specifically includes:

[0014] Step 11: Construct a feature pyramid network to extract multi-scale feature maps from the original remote sensing images. And ResNet was selected as the backbone network of the feature pyramid network;

[0015] Step 12: Construct a dual-branch attention mechanism by analyzing the multi-scale feature maps of the input image. Different weights are assigned to filter image features. This dual-branch attention mechanism consists of a spatial attention mechanism branch and a channel attention mechanism branch.

[0016] Step 13: Construct a channel attention mechanism branch to process the input multi-scale feature map in the spatial dimension. Perform global max pooling and global average pooling to obtain the feature map. and :

[0017] ;

[0018] in, This represents the feature map obtained after global average pooling. This represents the feature map obtained after global max pooling. This represents the global average pooling operation. This represents the global maximum pooling operation;

[0019] Step 14: Transfer the feature map and The data is fed into a multilayer perceptron with shared weights to learn the features and importance of the channel dimension. The outputs of the multilayer perceptron are summed and then passed through a sigmoid function. Mapping activation yields the channel attention weight matrix. :

[0020] ;

[0021] in, This represents the sigmoid function. This represents a multilayer perceptron;

[0022] Step 15: Construct a spatial attention mechanism branch, processing the input multi-scale feature map along the channel dimension. Perform global mean pooling and global max pooling to obtain the feature map. and :

[0023] ;

[0024] in, This represents the feature map after global average pooling. This represents the feature map after global max pooling;

[0025] Step 16: Use convolutional layers From feature maps and Features are extracted and processed by the sigmoid function. Mapping activation yields the spatial attention weight matrix. :

[0026] ;

[0027] in, Indicates the convolution operation;

[0028] Step 17: Convert the channel attention weight matrix of the channel attention mechanism branch. Spatial attention weight matrix of the spatial attention mechanism branch Application to multi-scale feature maps The above process involves assigning feature weights to the feature channels and spatial regions of the input image to obtain a weighted feature map.

[0029] Step 18: Fuse the weighted feature maps output by the spatial attention mechanism branch and the channel attention mechanism branch to obtain a fused feature map. Use convolutional layers to extract features from the fused feature map to obtain the output of the remote sensing image target area localization network.

[0030] Step 19: Perform location information fusion and update on candidate regions containing overlapping areas;

[0031] For candidate regions A and B that contain overlapping areas, perform candidate region localization optimization:

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] In the formula, and These are the coordinates of the top left and bottom right corners of candidate region A, respectively. and These are the coordinates of the top left and bottom right corners of candidate region B, respectively. and These represent the confidence levels of candidate region A and candidate region B, respectively. and The coordinates of the upper left and lower right corners of the candidate region C are updated respectively, and multiple local region slices with non-overlapping areas are cropped from the original remote sensing image based on the positioning information.

[0037] In the above technical solution, in step 18, the output of the remote sensing image target area localization network includes: multiple candidate areas, location information, and confidence level.

[0038] In the above technical solution, in step 2,

[0039] An adaptive multi-scale restoration and super-resolution reconstruction network extracts key features from local region slice images, establishes an adaptive super-scale selection mechanism, and performs multi-scale image restoration and super-resolution reconstruction on local region slice images.

[0040] In the above technical solution, step 2 specifically includes:

[0041] Step 21: Construct a shallow feature extraction module for a given low-resolution local region slice image. The image is then input into a shallow feature extraction module composed of convolutional layers to extract features and generate shallow features of the image. :

[0042] ;

[0043] in, Indicates the convolution operation;

[0044] Step 22: Construct a deep feature extraction network to further extract deep features from the image; utilize multiple residual convolutional modules composed of residual blocks and convolutional layers to form a long-range deep feature extraction network, employing a self-attention mechanism to capture global information of the image, performing local attention and cross-window interaction on image information; simultaneously, utilize convolutional layers to introduce region boundary information and neighboring pixel information, assisting in the fusion of shallow and deep image information and the restoration of image details, as shown in the following formula:

[0045] ;

[0046] In the formula, It can be 8, 16, or 32. The value range is an integer between 6 and 12; Indicates the first The first residual convolutional module The output of the residual channel attention submodule It is the first Attention weight matrix for each residual channel attention submodule. Indicates the first In the residual convolution module, the first The output of the residual channel attention submodule It is the first Attention weight matrix for each residual channel attention submodule. It is the first attention weight matrix of the deep feature extraction network. Indicates the first The first residual convolutional module The output of each residual channel attention submodule; Representing the deep features of the image; Step 23: Constructing a local region boundary feature extraction network; Using convolutional kernels of different sizes, extracting edge features of local region slice images from the original remote sensing image, and fusing multiple features to provide more effective information for slice reconstruction, as shown in the following formula:

[0047] ;

[0048] In the formula, This represents a low-resolution target image. This represents a 1x1 convolution operation. This represents a 3x3 convolution operation. This represents a 5x5 convolution operation. This represents a 7x7 convolution operation. This indicates that a splicing operation is performed along the channel dimension. Represents the edge information features of an image;

[0049] Step 24: Construct a weighted feature fusion mechanism; establish long skip connections to fuse shallow, deep, and edge features of the image, and simultaneously use an attention mechanism to assign different weights to the shallow, deep, and edge features of the image, as shown in the following formula:

[0050] ;

[0051] In the formula, , , Weights are assigned to the attention mechanism features of shallow features, deep features, and edge features, respectively. For image fusion features;

[0052] Step 25: Construct a scale-adaptive upsampling module; adaptively select the super-resolution scale based on the target local region size output by the remote sensing image target region localization network, perform multi-scale upsampling and restoration super-resolution on the image fusion features, and reconstruct a multi-scale high-resolution enhanced remote sensing image of the target region; for N candidate regions in the image, cluster the candidate regions according to the size of the local region slice image, divide the target slice into three groups: S, M, and L, and adaptively select three scales to perform multi-scale upsampling and restoration reconstruction on the target slice. The scale selection method is shown in the following formula:

[0053] ;

[0054] In the formula, , and These represent the maximum target slice sizes for groups L, M, and S, respectively. The minimum target slice size corresponding to group S. , , , All are scaling factors. 、 The values ​​are all 2, 4 or 8 respectively; 、 The values ​​were all 0.25 or 0.5 respectively;

[0055] Step 26: Construct a loss function to constrain the training of the adaptive multi-scale restoration and super-resolution reconstruction network; construct a global content loss function to provide supervision for image consistency and texture detail reconstruction; construct a region edge loss function to constrain the extraction of local region boundary features; the loss function for adaptive multi-scale restoration and super-resolution reconstruction. as follows:

[0056] ;

[0057] In the formula, For image reconstruction content loss function, For the region edge loss function, The weighting coefficients of the region edge loss function are represented. Indicates the reconstructed image. Represents a high-resolution image. Represents a regular term. This indicates gradient extraction.

[0058] In the above technical solution, in step 3,

[0059] A multi-scale remote sensing image fine-grained recognition network combines intra-feature map attention and inter-feature map attention mechanisms to perform feature extraction and feature fusion on multi-scale local slice images, thereby achieving fine-grained target recognition.

[0060] In the above technical solution, step 3 specifically includes:

[0061] Step 31: Construct a multi-scale local image feature extraction module; select ResNet as the backbone network of the multi-scale local image feature extraction module to extract features from multi-scale local slice images and obtain feature maps. , and ;

[0062] Step 32: Construct an intra-feature map attention mechanism for the feature map. , and Different weights are assigned internally to filter image features;

[0063] Step 33: Process the feature map , and Perform global average pooling and global maximum pooling along the channel dimension, as shown in the following formula:

[0064] ;

[0065] in, Representing feature maps, Number the feature maps. This represents the global average pooling operation. This represents the global maximum pooling operation. This represents the feature map after average pooling. This represents the feature map after max pooling;

[0066] Step 34: Use convolutional layers Feature extraction is performed on the feature maps after global max pooling and global average pooling, and then processed by the sigmoid function. Mapping activation yields the attention weight matrix within the feature map. :

[0067] ;

[0068] in, This represents the convolution operation. Represents the sigmoid function;

[0069] Step 35: Construct an attention mechanism between feature maps; combine image feature maps of different scales according to the channel dimension, and perform global mean pooling and global maximum pooling on them in the spatial dimension:

[0070] ;

[0071] in, This represents the combined image feature maps at different scales. This represents the feature map after global average pooling. This represents the feature map after global max pooling;

[0072] Step 36: Feed the feature maps after global max pooling and global average pooling into a multilayer perceptron, and sum the outputs of the multilayer perceptron, then pass the sum through the sigmoid function. Mapping activation yields the attention weight matrix between feature maps. :

[0073] ;

[0074] in, This represents a multilayer perceptron;

[0075] Step 37: Utilize the attention weight matrix within the feature map Weights are assigned to feature maps at a single scale, utilizing the attention weight matrix between feature maps. Assign weights to feature maps for fusion, and then redistribute the weights and fuse the features across the feature maps.

[0076] Step 38: Use convolutional layers to further extract features from the fused feature map to obtain the output of the multi-scale remote sensing image fine-grained recognition network.

[0077] In the above technical solution, in step 38, the output of the multi-scale remote sensing image fine-grained recognition network includes: the fine-grained category of the target of interest, the target location, and the confidence level.

[0078] The present invention has the following beneficial effects:

[0079] The fine-grained target recognition method for remote sensing images based on local multi-scale super-resolution of the present invention improves recognition efficiency by performing multi-scale restoration and super-resolution reconstruction on local regions of remote sensing images, which has low computational cost and fast computation speed.

[0080] The present invention provides a method for fine-grained target recognition of remote sensing images based on local multi-scale super-resolution. This method extracts multi-scale features from multiple local region restoration and super-resolution reconstructed images and performs fusion recognition on the multi-scale features. It fully considers the role of multi-scale feature fusion in fine-grained target recognition and can improve the accuracy of fine-grained target recognition. Attached Figure Description

[0081] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0082] Figure 1 This is a schematic diagram of the network architecture of the fine-grained target recognition network proposed in the fine-grained target recognition method for remote sensing images based on local multi-scale super-resolution of the present invention.

[0083] Figure 2 A schematic diagram of a network for locating target regions in a remotely sensed image.

[0084] Figure 3 This is a schematic diagram of an adaptive multi-scale restoration and super-resolution reconstruction network.

[0085] Figure 4 This is a schematic diagram of a fine-grained recognition network for multi-scale remote sensing images. Detailed Implementation

[0086] The inventive concept of this invention is as follows:

[0087] The present invention provides a fine-grained target recognition method for remote sensing images based on local multi-scale super-resolution. This method is based on local multi-scale super-resolution reconstruction of remote sensing images. It extracts effective features for target recognition from the multi-scale restored and reconstructed images of local remote sensing images, fuses the multi-scale recognition features, and inputs the fused features into the fine-grained target recognition network module to improve the accuracy of fine-grained target recognition.

[0088] The fine-grained target recognition network of the method of this invention consists of a remote sensing image target region localization network, an adaptive multi-scale restoration and super-resolution reconstruction network, and a multi-scale remote sensing image fine-grained recognition network. (See also...) Figure 1Specifically, the remote sensing image target region localization network is used to extract features from the input raw remote sensing image and search for and locate local candidate regions in the image. The adaptive multi-scale restoration and super-resolution reconstruction network consists of a local region boundary feature extraction network and a multi-scale restoration and super-resolution reconstruction module. The multi-scale restoration and super-resolution reconstruction module restores and super-reconstructs local candidate region images to obtain multi-scale high-resolution images of the local regions. The local region boundary feature extraction network extracts features from the local region boundaries based on the local region slice location information in the remote sensing image, guiding the multi-scale restoration and super-resolution reconstruction of the local region image. The multi-scale remote sensing image fine-grained recognition network includes a multi-scale local image feature extraction module, a multi-scale feature fusion module, and a fine-grained target recognition module. The multi-scale local image feature extraction module extracts features from the high-resolution multi-scale local restoration and reconstruction image; the multi-scale feature fusion module fuses the features of the multi-scale images; the fused feature map is input into the fine-grained target recognition module, which outputs the fine-grained target recognition result of the remote sensing image. This invention achieves high-efficiency and high-precision fine-grained target recognition of remote sensing images by performing multi-scale restoration and super-resolution reconstruction on local candidate regions in remote sensing images, extracting features from multi-scale high-resolution local region images, and performing feature fusion and target recognition.

[0089] The fine-grained target recognition method for remote sensing images based on local multi-scale super-resolution of the present invention can solve the problems of low accuracy and high computational cost of existing fine-grained target recognition methods.

[0090] The present invention will now be described in detail with reference to the accompanying drawings.

[0091] The fine-grained target recognition method for remote sensing images based on local multi-scale super-resolution of this invention proposes a fine-grained target recognition network consisting of a remote sensing image target region localization network, an adaptive multi-scale restoration and super-resolution reconstruction network, and a multi-scale remote sensing image fine-grained recognition network. The network architecture is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0092] Step 1: Construct a remote sensing image target region localization network. Input the original remote sensing image into the network, and use it to perform coarse detection of the target of interest (ROI) and local region localization. The local region refers to a candidate region in the input image that contains the ROI. The architecture of the remote sensing image target region localization network is as follows: Figure 2 As shown, the input image for this network is the original remote sensing image ( Figure 1 and 2 In this study, the original remote sensing image is simply referred to as the remote sensing image. The network output is a slice image of the target local region, and the location information of this slice image within the original remote sensing image. Step 1 specifically includes the following steps:

[0093] Step 11: Construct a feature pyramid network to extract multi-scale feature maps from the original remote sensing images. ResNet was chosen as the backbone network of the feature pyramid network.

[0094] Step 12: Construct a dual-branch attention mechanism to filter image features by assigning different weights to the multi-scale features of the input image. This dual-branch attention mechanism consists of a spatial attention mechanism branch and a channel attention mechanism branch.

[0095] Step 13: Construct a channel attention mechanism branch to process the input multi-scale feature map in the spatial dimension. Perform global max pooling and global average pooling to obtain the feature map. and :

[0096] (1)

[0097] in, This represents the feature map obtained after global average pooling. This represents the feature map obtained after global max pooling. This represents the global average pooling operation. This represents the global maximum pooling operation; where, It stands for the first letter of "channel," representing channel attention mechanism, to distinguish it from spatial attention mechanism mentioned below;

[0098] Step 14: Transfer the feature map and The data is fed into a multilayer perceptron with shared weights to learn the features and importance of the channel dimension. The outputs of the multilayer perceptron are summed and then passed through a sigmoid function. Mapping activation yields the channel attention weight matrix. :

[0099] (2)

[0100] in, This represents the sigmoid function. This represents a multilayer perceptron;

[0101] Step 15: Construct a spatial attention mechanism branch, processing the input multi-scale feature map along the channel dimension. Perform global mean pooling and global max pooling to obtain the feature map. and :

[0102] (3)

[0103] in, This represents the feature map after global average pooling. This represents the feature map after global max pooling; where, It stands for the first letter of "space," representing spatial attention mechanism, to distinguish it from the channel attention mechanism mentioned above.

[0104] Step 16: Use convolutional layers From feature maps and Features are extracted and processed by the sigmoid function. Mapping activation yields the spatial attention weight matrix. :

[0105] (4)

[0106] in, Indicates the convolution operation;

[0107] Step 17: Convert the channel attention weight matrix of the channel attention mechanism branch. Spatial attention weight matrix of the spatial attention mechanism branch Application to multi-scale feature maps The input image feature channels and input image spatial region features are weighted to obtain a weighted feature map.

[0108] Step 18: Fuse the weighted feature maps output by the spatial attention mechanism branch and the channel attention mechanism branch to obtain a fused feature map. Use convolutional layers to extract features from the fused feature map to obtain the output of the remote sensing image target area localization network, including multiple candidate regions, location information and confidence scores.

[0109] Step 19: Perform location information fusion and update for candidate regions containing overlapping areas. For candidate regions A and B containing overlapping areas, perform candidate region location optimization:

[0110] (5)

[0111] (6)

[0112] (7)

[0113] (8)

[0114] In the formula, and These are the coordinates of the top left and bottom right corners of candidate region A, respectively. and These are the coordinates of the top left and bottom right corners of candidate region B, respectively. and These represent the confidence levels of candidate region A and candidate region B, respectively. and The coordinates of the upper left and lower right corners of the candidate region C are updated respectively, which outputs the local region's location information. Based on the location information, multiple local region slices with non-overlapping areas are cropped from the original remote sensing image.

[0115] Step 2: Construct an adaptive multi-scale restoration and super-resolution reconstruction network to perform multi-scale restoration and super-resolution reconstruction on the local area slice images output by the remote sensing image target area localization network. The adaptive multi-scale restoration and super-resolution reconstruction network consists of a multi-scale local image feature extraction module, a local region boundary feature extraction network, and a scale adaptive selection upsampling module. See the network structure below. Figure 1 and 3 As shown. The multi-scale local image feature extraction module and the scale adaptive selection upsampling module constitute the multi-scale restoration and super-resolution reconstruction module. Based on the adaptive super-resolution magnification, it performs feature extraction, image restoration, and super-resolution reconstruction on the local region slice image, outputting a high-resolution multi-scale local slice image. The local region boundary feature extraction network extracts the surrounding edge information of the region from the original remote sensing image based on the local region slice image's location information in the original remote sensing image, providing real-world information guidance for local region image reconstruction and enhancing the accuracy and robustness of image restoration and super-resolution reconstruction. Step 2 specifically includes the following steps:

[0116] Step 21: Construct a shallow feature extraction module. For a given low-resolution local region slice image... The image is then input into a shallow feature extraction module composed of convolutional layers to extract features and generate shallow features of the image. :

[0117] (9)

[0118] in, Indicates the convolution operation;

[0119] Step 22: Construct a deep feature extraction network to further extract deep features from the image. A long-range deep feature extraction network is constructed using multiple residual convolutional modules consisting of residual blocks and convolutional layers. A self-attention mechanism is employed to capture global information of the image, performing local attention and cross-window interaction on image information. Simultaneously, convolutional layers introduce region boundary information and neighboring pixel information to assist in the fusion of shallow and deep image information and the restoration of image details, as shown in the following equation:

[0120] (10)

[0121] In the formula, It can be 8, 16, or 32. The value of is an integer between 6 and 12. In this specific embodiment, It is 8. It is 6; Indicates the first The first residual convolutional module The output of the residual channel attention submodule It is the first The attention weight matrix of each residual channel attention submodule. Indicates the first In the residual convolution module, the first The output of the residual channel attention submodule It is the first Attention weight matrix for each residual channel attention submodule. It is the first attention weight matrix of the deep feature extraction network. Indicates the first The first residual convolutional module The output of each residual channel attention submodule; Represents the deep features of an image;

[0122] In other specific implementations, It can also be 16 or 32; The value range is an integer between 6 and 12, that is, except for 6, in other specific implementations, It can also take the values ​​7, 8, 9, 10, 11 or 12.

[0123] Step 23: Construct a local region boundary feature extraction network. Edge features of local region slices are extracted from the original remote sensing image using convolutional kernels of different sizes. Multiple features are then fused to provide more effective information for slice reconstruction, as shown in the following equation:

[0124] (11)

[0125] In the formula, This represents a low-resolution target image. This represents a 1x1 convolution operation. This represents a 3x3 convolution operation. This represents a 5x5 convolution operation. This represents a 7x7 convolution operation. This indicates that a splicing operation is performed along the channel dimension. Represents the edge information features of an image.

[0126] Step 24: Construct a weighted feature fusion mechanism. Establish long-skip connections to fuse shallow, deep, and edge features of the image. Simultaneously, use an attention mechanism to assign different weights to the shallow, deep, and edge features of the image, as shown in the following formula:

[0127] (12)

[0128] In the formula, , , Weights are assigned to the attention mechanism features of shallow features, deep features, and edge features, respectively. These are image fusion features.

[0129] Step 25: Construct a scale-adaptive upsampling module. Based on the target local region size output by the remote sensing image target region localization network, the super-resolution scale is adaptively selected. Multiple scales of upsampling and restoration super-resolution are performed on the image fusion features to reconstruct a multi-scale, high-resolution enhanced remote sensing image of the target region. For N candidate regions in the image, the candidate regions are clustered according to the size of the local region slice image. The target slices are divided into three groups: S, M, and L. Three scales are adaptively selected for multi-scale upsampling and restoration reconstruction of the target slices. The scale selection method is shown in the following formula:

[0130] (13)

[0131] In the formula, , and These represent the maximum target slice sizes for groups L, M, and S, respectively. The minimum target slice size corresponding to group S. , , , All are scale scaling factors, set to , In other specific implementations, 、 The values ​​can also be 4 or 8; 、 The values ​​can also be 0.25;

[0132] Step 26: Construct a loss function to constrain the training of the adaptive multi-scale restoration and super-resolution reconstruction network. A global content loss function is constructed to provide supervision for image consistency and texture detail reconstruction; a region edge loss function is constructed to constrain the extraction of local region boundary features. The loss function of the adaptive multi-scale restoration and super-resolution reconstruction network is as follows:

[0133] (14)

[0134] In the formula, For image reconstruction content loss function, For the region edge loss function, The weighting coefficients of the region edge loss function are represented. Indicates the reconstructed image. Represents a high-resolution image. Represents a regular term. This indicates gradient extraction.

[0135] Step 3: Construct a multi-scale remote sensing image fine-grained recognition network to process the high-resolution multi-scale local tile images output from Step 2 (i.e., Figure 3 and 4 Fine-grained target recognition is performed on multi-scale, high-resolution restored images. See also Figure 1 The multi-scale remote sensing image fine-grained recognition network comprises a multi-scale local image feature extraction module, a multi-scale feature fusion module, and a fine-grained target recognition module. It achieves fine-grained target recognition by extracting target features from multi-scale local slice images and fusing these features. Specifically, the multi-scale remote sensing image fine-grained recognition network combines intra-feature map attention mechanisms and inter-feature map attention mechanisms to perform feature extraction and feature fusion on multi-scale local slice images, achieving fine-grained target recognition. The network structure is as follows: Figure 4 As shown. Step 3 specifically includes the following steps:

[0136] Step 31: Construct a multi-scale local image feature extraction module. ResNet is selected as the backbone network for this module to extract features from multi-scale local slice images, resulting in feature maps. , and .

[0137] Step 32: Construct an intra-feature map attention mechanism for the feature map. , and Different weights are assigned internally to filter image features.

[0138] Step 33: Process the feature map , and Perform global average pooling and global maximum pooling along the channel dimension, as shown in the following formula:

[0139] (15)

[0140] in, Representing feature maps, Number the feature maps. This represents the global average pooling operation. This represents the global maximum pooling operation. This represents the feature map after average pooling. This represents the feature map after max pooling;

[0141] Step 34: Use convolutional layers Feature extraction is performed on the feature maps after global max pooling and global average pooling, and then processed by the sigmoid function. Mapping activation yields the attention weight matrix within the feature map. :

[0142] (16)

[0143] in, This represents the convolution operation. Represents the sigmoid function;

[0144] Step 35: Construct an attention mechanism between feature maps. Combine image feature maps of different scales according to the channel dimension, and perform global mean pooling and global maximum pooling on them in the spatial dimension:

[0145] (17)

[0146] in, This represents the combined image feature maps at different scales. This represents the feature map after global average pooling. This represents the feature map after global max pooling;

[0147] Step 36: Feed the feature maps after global max pooling and global average pooling into a multilayer perceptron, and sum the outputs of the multilayer perceptron, then pass the sum through the sigmoid function. Mapping activation yields the attention weight matrix between feature maps. :

[0148] (18)

[0149] in, This represents a multilayer perceptron;

[0150] Step 37: Utilize the attention weight matrix within the feature map Weights are assigned to feature maps at a single scale, utilizing the attention weight matrix between feature maps. Weights are assigned to feature maps for fusion, and weights are redistributed and features are fused from each feature map.

[0151] Step 38: Further feature extraction is performed on the fused feature map using convolutional layers to obtain the output of the multi-scale remote sensing image fine-grained recognition network, including the fine-grained category of the target of interest, the target location, and the confidence level.

[0152] The present invention provides a fine-grained target recognition method for remote sensing images based on local multi-scale super-resolution. The remote sensing image target region localization network adopts a dual-branch attention mechanism to extract features from the remote sensing image and realize the localization of key target regions. The adaptive multi-scale restoration and super-resolution reconstruction network adopts a residual structure to perform multi-scale image restoration and super-resolution reconstruction on local region images based on the localization information of key regions. The multi-scale remote sensing image fine-grained recognition network adopts an attention mechanism to extract features from multi-scale local images and uses a feature-level fusion strategy to realize fine-grained target recognition.

[0153] The fine-grained target recognition method for remote sensing images based on local multi-scale super-resolution of the present invention improves recognition efficiency by performing multi-scale restoration and super-resolution reconstruction on local regions of remote sensing images, which has low computational cost and fast computation speed.

[0154] The present invention provides a method for fine-grained target recognition of remote sensing images based on local multi-scale super-resolution. This method extracts multi-scale features from multiple local region restoration and super-resolution reconstructed images and performs fusion recognition on the multi-scale features. It fully considers the role of multi-scale feature fusion in fine-grained target recognition and can improve the accuracy of fine-grained target recognition.

[0155] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for fine-grained target recognition in remote sensing images based on local multi-scale super-resolution, characterized in that, It is a fine-grained target recognition method for remote sensing images based on multi-scale super-resolution reconstruction of local regions, including the following steps: Step 1: Construct a remote sensing image target region localization network. Input the original remote sensing image into the remote sensing image target region localization network to perform coarse detection of the target of interest and local region localization on the input image. Step 2: Construct an adaptive multi-scale restoration and super-resolution reconstruction network to perform multi-scale image restoration and super-resolution reconstruction on the local area slice images output by the remote sensing image target area localization network. Step 3: Construct a multi-scale remote sensing image fine-grained recognition network to perform fine-grained target recognition on the high-resolution multi-scale local slice image output in Step 2; Step 2 is as follows: Step 21: Construct a shallow feature extraction module for a given low-resolution local region slice image. The image is then input into a shallow feature extraction module composed of convolutional layers to extract features and generate shallow features of the image. : ； in, Indicates the convolution operation; Step 22: Construct a deep feature extraction network to further extract deep features from the image; utilize multiple residual convolutional modules composed of residual blocks and convolutional layers to form a long-range deep feature extraction network, employing a self-attention mechanism to capture global information of the image, performing local attention and cross-window interaction on image information; simultaneously, utilize convolutional layers to introduce region boundary information and neighboring pixel information, assisting in the fusion of shallow and deep image information and the restoration of image details, as shown in the following formula: ； In the formula, It can be 8, 16, or 32. The value range is an integer between 6 and 12; Indicates the first The first residual convolutional module The output of the residual channel attention submodule It is the first Attention weight matrix for each residual channel attention submodule. Indicates the first In the residual convolution module, the first The output of the residual channel attention submodule It is the first Attention weight matrix for each residual channel attention submodule. It is the first attention weight matrix of the deep feature extraction network. Indicates the first The first residual convolutional module The output of each residual channel attention submodule; Represents the deep features of an image; Step 23: Construct a local region boundary feature extraction network; use convolutional kernels of different sizes to extract edge features of local region slice images from the original remote sensing images, and fuse multiple features to provide more effective information for slice reconstruction, as shown in the following formula: ； In the formula, This represents a low-resolution target image. This represents a 1x1 convolution operation. This represents a 3x3 convolution operation. This represents a 5x5 convolution operation. This represents a 7x7 convolution operation. This indicates that a splicing operation is performed along the channel dimension. Represents the edge information features of an image; Step 24: Construct a weighted feature fusion mechanism; establish long skip connections to fuse shallow, deep, and edge features of the image, and simultaneously use an attention mechanism to assign different weights to the shallow, deep, and edge features of the image, as shown in the following formula: ； In the formula, , , Weights are assigned to the attention mechanism features of shallow features, deep features, and edge features, respectively. For image fusion features; Step 25: Construct a scale-adaptive sampling module; adaptively select the super-resolution scale based on the target local region size output by the remote sensing image target region localization network, perform multi-scale sampling and restoration super-resolution on the image fusion features, and reconstruct a multi-scale high-resolution enhanced remote sensing image of the target region; for N candidate regions in the image, cluster and group the candidate regions according to the size of the local region slice image, divide the target slice into three groups: S, M, and L, and adaptively select three scales to perform multi-scale sampling and restoration reconstruction on the target slice. The scale selection method is shown in the following formula: ； In the formula, , and These represent the maximum target slice sizes for groups L, M, and S, respectively. The minimum target slice size corresponding to group S. , , , All are scaling factors. , The values ​​are all 2, 4 or 8 respectively; , The values ​​were all 0.25 or 0.5 respectively; Step 26: Construct a loss function to constrain the training of the adaptive multi-scale restoration and super-resolution reconstruction network; construct a global content loss function to provide supervision for image consistency and texture detail reconstruction; construct a region edge loss function to constrain the extraction of local region boundary features; the loss function for adaptive multi-scale restoration and super-resolution reconstruction. as follows: ； In the formula, For image reconstruction content loss function, For the region edge loss function, The weighting coefficients of the region edge loss function are represented. Indicates the reconstructed image. Represents a high-resolution image. Represents a regular term. This indicates gradient extraction.

2. The method for fine-grained target recognition in remote sensing images according to claim 1, characterized in that, In step 1, The remote sensing image target region localization network adopts a dual-branch attention mechanism consisting of a channel attention mechanism branch and a spatial attention mechanism branch. It extracts multi-scale features from the original remote sensing image and performs weighted fusion of the multi-scale features to achieve the localization of local candidate regions in the original remote sensing image.

3. The method for fine-grained target recognition in remote sensing images according to claim 2, characterized in that, Step 1 is as follows: Step 11: Construct a feature pyramid network to extract multi-scale feature maps from the original remote sensing images. And ResNet was selected as the backbone network of the feature pyramid network; Step 12: Construct a dual-branch attention mechanism by analyzing the multi-scale feature maps of the input image. Different weights are assigned to filter image features. This dual-branch attention mechanism consists of a spatial attention mechanism branch and a channel attention mechanism branch. Step 13: Construct a channel attention mechanism branch to process the input multi-scale feature map in the spatial dimension. Perform global max pooling and global average pooling to obtain the feature map. and : ； in, This represents the feature map obtained after global average pooling. This represents the feature map obtained after global max pooling. This represents the global average pooling operation. This represents the global maximum pooling operation; Step 14: Transfer the feature map and The data is fed into a multilayer perceptron with shared weights to learn the features and importance of the channel dimension. The outputs of the multilayer perceptron are summed and then passed through a sigmoid function. Mapping activation yields the channel attention weight matrix. : ； in, This represents the sigmoid function. This represents a multilayer perceptron; Step 15: Construct a spatial attention mechanism branch, processing the input multi-scale feature map along the channel dimension. Perform global mean pooling and global max pooling to obtain the feature map. and : ； in, This represents the feature map after global average pooling. This represents the feature map after global max pooling; Step 16: Use convolutional layers From feature maps and Features are extracted and processed by the sigmoid function. Mapping activation yields the spatial attention weight matrix. : ； in, Indicates the convolution operation; Step 17: Convert the channel attention weight matrix of the channel attention mechanism branch. Spatial attention weight matrix of the spatial attention mechanism branch Application to multi-scale feature maps The above process involves assigning feature weights to the feature channels and spatial regions of the input image to obtain a weighted feature map. Step 18: Fuse the weighted feature maps output by the spatial attention mechanism branch and the channel attention mechanism branch to obtain a fused feature map. Use convolutional layers to extract features from the fused feature map to obtain the output of the remote sensing image target area localization network. Step 19: Perform location information fusion and update on candidate regions containing overlapping areas; For candidate regions A and B that contain overlapping areas, perform candidate region localization optimization: ； ； ； ； In the formula, and These are the coordinates of the top left and bottom right corners of candidate region A, respectively. and These are the coordinates of the top left and bottom right corners of candidate region B, respectively. and These represent the confidence levels of candidate region A and candidate region B, respectively. and The coordinates of the upper left and lower right corners of the candidate region C are updated respectively, and multiple local region slices with non-overlapping areas are cropped from the original remote sensing image based on the positioning information.

4. The method for fine-grained target recognition in remote sensing images according to claim 3, characterized in that, In step 18, the output of the remote sensing image target area localization network includes: multiple candidate areas, location information, and confidence level.

5. The method for fine-grained target recognition in remote sensing images according to claim 1, characterized in that, In step 2, An adaptive multi-scale restoration and super-resolution reconstruction network extracts key features from local region slice images, establishes an adaptive super-scale selection mechanism, and performs multi-scale image restoration and super-resolution reconstruction on local region slice images.

6. The method for fine-grained target recognition in remote sensing images according to claim 1, characterized in that, In step 3, A multi-scale remote sensing image fine-grained recognition network combines intra-feature map attention and inter-feature map attention mechanisms to perform feature extraction and feature fusion on multi-scale local slice images, thereby achieving fine-grained target recognition.

7. The method for fine-grained target recognition in remote sensing images according to claim 6, characterized in that, Step 3 specifically involves: Step 31: Construct a multi-scale local image feature extraction module; select ResNet as the backbone network of the multi-scale local image feature extraction module to extract features from multi-scale local slice images and obtain feature maps. , and ; Step 32: Construct an intra-feature map attention mechanism for the feature map. , and Different weights are assigned internally to filter image features; Step 33: Process the feature map , and Perform global average pooling and global maximum pooling along the channel dimension, as shown in the following formula: ； in, Representing feature maps, Number the feature maps. This represents the global average pooling operation. This represents the global maximum pooling operation. This represents the feature map after average pooling. This represents the feature map after max pooling; Step 34: Use convolutional layers Feature extraction is performed on the feature maps after global max pooling and global average pooling, and then processed by the sigmoid function. Mapping activation yields the attention weight matrix within the feature map. : ； in, This represents the convolution operation. Represents the sigmoid function; Step 35: Construct an attention mechanism between feature maps; combine image feature maps of different scales according to the channel dimension, and perform global mean pooling and global maximum pooling on them in the spatial dimension: ； in, This represents the combined image feature maps at different scales. This represents the feature map after global average pooling. This represents the feature map after global max pooling; Step 36: Feed the feature maps after global max pooling and global average pooling into a multilayer perceptron, and sum the outputs of the multilayer perceptron, then pass the sum through the sigmoid function. Mapping activation yields the attention weight matrix between feature maps. : ； in, This represents a multilayer perceptron; Step 37: Utilize the attention weight matrix within the feature map Weights are assigned to feature maps at a single scale, utilizing the attention weight matrix between feature maps. Assign weights to feature maps for fusion, and then redistribute the weights and fuse the features across the feature maps. Step 38: Use convolutional layers to further extract features from the fused feature map to obtain the output of the multi-scale remote sensing image fine-grained recognition network.

8. The method for fine-grained target recognition in remote sensing images according to claim 7, characterized in that, In step 38, the output of the multi-scale remote sensing image fine-grained recognition network includes: fine-grained category of the target of interest, target location, and confidence level.

Images