Remote Sensing Image Segmentation Method and System Based on Multi-Scale Fusion Attention

Abstract

The present invention discloses a remote sensing image segmentation method and system based on multi-scale fusion attention. The method includes: 1) multi-scale feature representation (MFR); 2) feature fusion attention (FFA); 3) feature refinement (FRM); 4) evaluation. The method of the present invention enables the model to gradually refine high-resolution features during the process of learning remote sensing images, and effectively alleviates the problems of misclassification and missed classification caused by complex image backgrounds and large differences in class sizes. Thereby improving the segmentation effect of remote sensing images.

Description

Technical Field

[0001] The present invention belongs to the field of computer science and artificial intelligence, and specifically relates to a remote sensing image segmentation method and system based on multi-scale fusion attention, which is used for the segmentation task of high-resolution remote sensing images. Background Art

[0002] With the rapid development of remote sensing technology, remote sensing equipment is constantly updated, and the resolution and data volume of remote sensing images are increasing. As a result, the application of remote sensing images has developed into scene classification, object detection, target segmentation, etc.

[0003] Semantic segmentation of high-resolution remote sensing images assigns labels to each class pixel by pixel in the image to obtain the location and semantic information of the object. It is a pixel-level classification. Remote sensing image segmentation is widely used in fields such as urban road monitoring, building detection, and vegetation extraction. Compared with natural scenes, geospatial target segmentation is more challenging in high-resolution remote sensing images. In many natural scene categories, different object categories usually have strong interdependence, while remote sensing images are different from natural scenes and do not need to rely on too many target categories. Generally, in high-resolution remote sensing images, objects will have large size changes, and there are certain difficulties in the positioning and identification of targets compared to natural scenes.

[0004] At present, due to the complex background and high resolution of remote sensing images, traditional segmentation methods such as clustering, wavelet analysis, support vector machine, threshold method, etc. cannot accurately extract category information from remote sensing images. With the emergence of deep convolutional neural networks, in recent years, the research on semantic segmentation methods based on deep learning has also been developed in the field of remote sensing. In 2015, Long et al. proposed the fully convolutional neural network FCN, which is the first end-to-end fully convolutional neural network. Compared with the traditional convolutional neural network, he replaced the fully connected layer in the network with the convolutional layer, which can effectively complete the image segmentation task. After that, Ronneberger et al. designed the U-net network, which is a U-shaped network model used for medical image segmentation. Its encoding and decoding structures correspond to each other, which can effectively combine high-level and low-level feature information to prevent the loss of detail information. Badrinarayanan et al. proposed SegNet, which transferred the maximum pooling index to the decoder to improve the resolution of segmentation. Zhao et al. proposed the pyramid scene parsing network PSPNet, which uses the pyramid network architecture to aggregate features between different convolutional layers and mine global context information. Later, DeeplabV3+ proposed by Chen et al. used Atos convolution to obtain multi-scale contextual information, and while obtaining a larger receptive field, it would not bring too much computational effort.

[0005] Although high-resolution remote sensing segmentation technology has achieved certain success in recent years, in the long run, there are still limitations. First, most of the remote sensing images are high-resolution images, and the objects in the images usually have different sizes. For example, from a top-down perspective, we find that buildings in the view occupy most of the pixel area, while small objects such as cars and vegetation only occupy a small area. It is difficult for a receptive field of a single size to obtain object features with sufficient representational ability, which leads to the multi-scale phenomenon. Second, the background in remote sensing images is complex, and the difference between the foreground categories and the background is large, which easily causes the problem of misidentifying similar targets. Finally, the segmentation results of high-resolution remote sensing images are often unclear and a large amount of noise is generated near the object boundaries. Summary of the Invention

[0006] The present invention provides a method that solves the problems disclosed in the background technology.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] Remote sensing image segmentation method based on multi-scale fusion attention:

[0009] Obtain dataset images, input the dataset images into a pre-constructed ResNet-50 residual network to obtain deep semantic feature information, and then input the obtained feature information into a pre-constructed multi-scale feature representation module. The multi-scale feature representation module performs feature aggregation through dilated convolutions with different dilation rates;

[0010] Reduce the dimension of the deep semantic feature information and the output feature information of the multi-scale feature representation module, and then input them into a pre-constructed feature fusion module FFA. The feature fusion module FFA fuses the dimension-reduced features in order from high layer to low layer;

[0011] Input the fused features into a pre-constructed feature refinement module. The feature refinement module reduces the number of channels, reduces the computational amount, and performs two refinements with different receptive fields to gradually refine the feature map.

[0012] Furthermore:

[0013] The multi-scale feature representation module performs feature aggregation through dilated convolutions with different dilation rates and outputs a feature matrix M4.

[0014]

[0015] where M represents the output of the last layer of ResNet-50; It represents a dilated convolution operation with a convolution kernel size of 3 and a dilation rate of 1; BN represents further batch normalization; ReLU represents a non-linear activation layer; GAP represents a global average pooling layer; UP represents upsampling to the original image size; Concat represents adding each feature in the channel dimension.

[0016] Furthermore:

[0017] The deep semantic feature information is the feature matrix M1, the feature matrix M2, and the feature matrix M3;

[0018] Reducing the dimension of the feature matrix M1 gives:

[0019] T1 = ReLU(BN(C 1×1 (M1)))

[0020] Reducing the dimension of the feature matrix M2 gives:

[0021] T2 = ReLU(BN(C 1×1 (M2)))

[0022] Reducing the dimension of the feature matrix M3 gives:

[0023] T3 = ReLU(BN(C 1×1 (M3)))

[0024] Reducing the dimension of the feature matrix M4 gives:

[0025] T4 = ReLU(BN(C 1×1 (M4)))

[0026] In the formula, C 1×1 represents a convolution operation with a convolution kernel size of 1; BN represents further batch normalization to accelerate the training speed; ReLU represents a non-linear activation layer, which is used to reduce the mutual dependence of parameters and alleviate the overfitting phenomenon.

[0027] Furthermore:

[0028] The process of the feature fusion module FFA fusing the dimension-reduced features in order from high layer to low layer is as follows: The feature fusion module FFA includes three fusion modules. Input T4 and T3 into the first fusion module. T4 is used as the high-level semantic feature of the first fusion module, and T3 is used as the low-level semantic feature. The feature matrix T4 first undergoes an upsampling operation to obtain a feature matrix with the same size as T3, and the two are fused through an addition operation to obtain a fused feature matrix I3; Finally, I3 is dimension-reduced using a convolution with a convolution kernel size of 3, and a feature matrix K3 with the same size as T3 is output:

[0029] I3 = Concat(UP(T4), T3)

[0030] K3 = ReLU(BN(C 3×3 (I3)))

[0031] Wherein, UP represents upsampling, restoring T4 to the size of T3, Concat is an addition operation, adding the high-level features and the low-level features according to the number of channels, and finally reducing the number of channels to the size of the low-level feature T3 through a convolution operation;

[0032] Then, global average pooling is used to guide the low-level feature information. Assuming the high-level feature is T4 = [t1, t2,... t c1 ∈ R c1×h×w , and the low-level feature is K3 = [k1, k2,... k c2 ∈ R c2×h×w , a global average pooling operation is performed on the high-level semantic feature T4 to obtain global information, and the obtained global information is represented as A ∈ R c1×1×1 , and this process is expressed as:

[0033]

[0034] Wherein, h and w respectively represent the length and width of the feature map, and A k is the feature weight value of the k-th layer;

[0035] After A is dimension-reduced by a 1×1 convolution, a feature matrix with a length and width of 1 and a number of channels of C2 is obtained. The weight values are divided between [0, 1] through the Sigmoid activation function to generate a weight vector V ∈ R c2×1×1 , the weight vector is multiplied by the low-level feature K3, followed by a 1×1 convolution operation, BN batch normalization, and ReLU activation. This process is expressed as:

[0036] U3 = ReLU(BN(C 1×1 (K3 × V)))

[0037] Wherein, U3 represents the output of the shallow features after the product of the weight vector;

[0038] Finally, an upsampling operation is performed on the high-level feature T4 to make the elements on T4 correspond one-to-one with the low-level feature map T3. Then, the two feature maps are directly added according to the corresponding pixels, and the formula is as follows:

[0039] P3 = UP(T4) + U3

[0040] Similarly, the high-level feature P3 and the low-level feature T2 are input into the second fusion module to obtain the high-level feature P2, and the high-level feature P2 and the low-level feature T1 are input into the third fusion module to finally obtain the high-level feature P1.

[0041] Correspondingly, a remote sensing image segmentation system based on multi-scale fusion attention includes:

[0042] Multi-scale Feature Representation Module MFR: It includes the Residual Network ResNet-50. The dataset images are input into the Residual Network ResNet-50 to obtain deep semantic feature information, and then the obtained feature information is input for feature aggregation through dilated convolutions with different dilation rates.

[0043] Feature Fusion Module FFA: It is used to reduce the dimension of the deep semantic feature information and the output feature information of the multi-scale feature representation module, and then fuse the dimension-reduced features in order from high-level to low-level.

[0044] Feature Refinement Module FRM: It is used to input the fused features into a pre-constructed one to reduce the number of channels and computational amount, and perform two refinements with different receptive fields to gradually refine the feature map.

[0045] Furthermore:

[0046] The multi-scale feature representation module MFR performs feature aggregation through dilated convolutions with different dilation rates and outputs the feature matrix M4.

[0047]

[0048] Where M represents the output of the last layer of ResNet-50. represents the dilated convolution operation with a convolution kernel size of 3 and a dilation rate of 1; BN represents further batch normalization; ReLU represents the non-linear activation layer; GAP represents the global average pooling layer; UP represents upsampling to the original image size; Concat represents adding each feature in the channel dimension.

[0049] Furthermore:

[0050] The deep semantic feature information output by the Residual Network ResNet-50 is the feature matrix M1, the feature matrix M2, and the feature matrix M3.

[0051] The dimension of the feature matrix M1 is reduced to obtain:

[0052] T1 = ReLU(BN(C 1×1 (M1)))

[0053] The dimension of the feature matrix M2 is reduced to obtain:

[0054] T2 = ReLU(BN(C 1×1 (M2)))

[0055] The dimension of the feature matrix M3 is reduced to obtain:

[0056] T3 = ReLU(BN(C 1×1 (M3)))

[0057] The dimensionality reduction of the feature matrix M4 results in:

[0058] T4 = ReLU(BN(C 1×1 (M4)))

[0059] In the formula, C 1×1 represents a convolution operation with a kernel size of 1; BN represents further batch normalization to accelerate the training speed; ReLU represents a non-linear activation layer, which is used to reduce the mutual dependence of parameters and alleviate the overfitting phenomenon.

[0060] Furthermore:

[0061] The process of the feature fusion module FFA fusing the dimensionality-reduced features in order from high-level to low-level is as follows: The feature fusion module FFA includes three fusion modules. Input T4 and T3 into the first fusion module. T4 serves as the high-level semantic feature of the first fusion module, and T3 serves as the low-level semantic feature. The feature matrix T4 first undergoes an upsampling operation to obtain a feature matrix with the same size as T3, and the two are fused through an addition operation to obtain the fused feature matrix I3; finally, I3 is dimensionally reduced using a convolution with a kernel size of 3, and a feature matrix K3 of the same size as T3 is output:

[0062] I3 = Concat(UP(T4), T3)

[0063] K3 = ReLU(BN(C 3×3 (I3)))

[0064] In the formula, UP represents upsampling, restoring T4 to the size of T3. Concat is an addition operation, adding the high-level feature and the low-level feature according to the number of channels, and finally reducing the number of channels to the size of the low-level feature T3 through a convolution operation.

[0065] Then, global average pooling is used to guide the low-level feature information. Assume the high-level feature is T4 = [t1, t2,... t c1 ∈ R c1×h×w , and the low-level feature is K3 = [k1, k2,... k c2 ∈ R c2×h×w . A global average pooling operation is performed on the high-level semantic feature T4 to obtain global information, and the obtained global information is represented as A ∈ R c1×1×1 . This process is expressed as:

[0066]

[0067] In the formula, h and w respectively represent the length and width of the feature map, and A k is the feature weight value of the k-th layer;

[0068] A is reduced in dimension through a 1×1 convolution to obtain a feature matrix with a length and width of 1 and a channel number of C2. The weight values are divided between [0,1] through the Sigmoid activation function to generate a weight vector V∈R c2×1×1 , the weight vector is multiplied by the underlying feature K3, followed by a 1×1 convolution operation, BN batch normalization, and ReLU activation. This process is expressed as:

[0069] U3 = ReLU(BN(C 1×1 (K3×V)))

[0070] In the formula, U3 represents the output after the product of the shallow feature and the weight vector;

[0071] Finally, an upsampling operation is performed on the high-level feature T4 to make the elements on T4 correspond one-to-one with the low-level feature map T3. Then, the two feature maps are directly added according to the corresponding pixels. The formula is as follows:

[0072] P3 = UP(T4)+U3

[0073] Similarly, the high-level feature P3 and the underlying feature T2 are input into the second fusion module to obtain the high-level feature P2, and the high-level feature P2 and the underlying feature T1 are input into the third fusion module to finally obtain the high-level feature P1.

[0074] Correspondingly: The one or more programs include instructions that, when executed by a computing device, cause the computing device to execute any of the above methods.

[0075] Correspondingly, a computing device includes:

[0076] One or more processors, one or more memories, and one or more programs, where the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors. The one or more programs include instructions for executing any of the above methods.

[0077] The beneficial effects achieved by the present invention:

[0078] The remote sensing image segmentation method and system based on multi-scale fusion attention of the present invention enable the model to more efficiently learn the features between different layers during the learning process, effectively alleviate the misclassification problem for complex backgrounds, and at the same time reduce the high-resolution boundary noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 is a schematic diagram of the overall structure of this method;

[0080] Figure 2 is a schematic diagram of the structure of the multi-scale feature representation module;

[0081] Figure 3 Schematic diagram of the feature fusion attention module;

[0082] Figure 4 Schematic diagram of the feature refinement module;

[0083] Figure 5 Schematic diagram of the visualization result of the method of the present invention on the Potsdam dataset of high - resolution remote sensing images.

[0084] Figure 6 Schematic diagram of the visualization result of the method of the present invention on the Vaihingen dataset of high - resolution remote sensing images. Detailed implementation manners

[0085] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0086] Embodiment 1:

[0087] A high - resolution remote sensing image segmentation system based on multi - scale fusion, as Figure 1 shown, is composed of a multi - scale feature representation module, a feature fusion attention module, and a feature refinement module.

[0088] 1) Multi - scale feature representation: Extract image multi - scale feature information from the given dataset according to the following method:

[0089] As Figure 2 shown, first input the dataset image into the ResNet - 50 residual network to obtain deep semantic feature information, and then input the obtained feature information into the multi - scale feature representation module. This module performs feature aggregation through dilated convolutions with different dilation rates. The specific process is represented by the following formula.

[0090]

[0091] In the formula, M represents the output of the last layer of ResNet - 50; represents a dilated convolution operation with a convolution kernel size of 3 and a dilation rate of 1; BN represents further batch normalization; ReLU represents a non - linear activation layer; GAP represents global average pooling layer; UP represents up - sampling to the original image size; Concat represents adding each feature in the channel dimension.

[0092] 2) Feature fusion attention: Perform dimensionality reduction operations on the feature matrices M1, M2, and M3 output by each residual block of ResNet - 50 and the output feature matrix M4 of the multi - scale feature representation, and then input them into three feature fusion modules FFA in sequence, as Figure 3As shown in the figure, the specific steps are as follows:

[0093] 2.1) Use a convolution with a kernel size of 1 to reduce the dimension of the input features:

[0094] First, extract the feature matrix M4 extracted in step 1 and the outputs M1, M2, and M3 of the three convolutional layers of ResNet50. Before inputting into this module, use a convolution kernel of size 1 to reduce the dimension. The corresponding number of channels after dimension reduction are [256, 512, 1024, 2048] respectively. This process can be expressed by the following formula:

[0095] T1 = ReLU(BN(C 1×1 (M1)))

[0096] T2 = ReLU(BN(C 1×1 (M2)))

[0097] T3 = ReLU(BN(C 1×1 (M3)))

[0098] T4 = ReLU(BN(c 1×1 (M4)))

[0099] In the formula, C 1×1 represents a convolution operation with a kernel size of 1; BN represents further batch normalization to accelerate the training speed; ReLU represents a non - linear activation layer to reduce the mutual dependence of parameters and alleviate the overfitting phenomenon.

[0100] 2.2) Fuse and output the dimension - reduced features in order from high - level to low - level:

[0101] First, input T4 and T3 into the first fusion module FFA. T4 serves as the high - level semantic feature of this module, and T3 serves as the low - level semantic feature. The feature matrix T4 first undergoes an up - sampling operation to obtain a feature matrix with the same size as T3. In the module, the two are fused through an addition operation to obtain the fused feature matrix I3; finally, use a convolution with a kernel size of 3 to reduce the dimension of I3 and output a feature matrix K3 with the same size as T3:

[0102] I3 = Concat(UP(T4), T3)

[0103] K3 = ReLU(BN(C 3×3 (I3)))

[0104] In the formula, UP represents up - sampling, which restores T4 to the size of T3. Concat is an addition operation that adds the high - level feature and the low - level feature according to the number of channels. Finally, the number of channels is reduced to the size of the low - level feature T3 through a convolution operation.

[0105] Then, global average pooling is used to guide the underlying feature information. Assume the high-level feature is T4 = [t1, t2, … t c1 ∈ R c1×h×w and the underlying feature is K3 = [k1, k2, … k c2 ∈ R c2×h×w . We perform global average pooling operation on the high-level semantic feature T4 to obtain global information, and the obtained global information is denoted as A ∈ R c1×1×1 . This process can be expressed as:

[0106]

[0107] where h and w respectively represent the length and width of the feature map, and A k is the feature weight value of the k-th layer.

[0108] After A is reduced in dimension by a 1×1 convolution, a feature matrix with a length and width of 1 and a channel number of C2 is obtained. The weight values are divided between [0, 1] through the Sigmoid activation function to generate a weight vector V ∈ R c2×1×1 . The weight vector is multiplied by the underlying feature K3 to enhance the correlation between channels. Then, operations such as 1×1 convolution operation, BN batch normalization, and ReLU activation are performed. The specific implementation is as follows:

[0109] U3 = ReLU(BN(C 1×1 (K3 × V)))

[0110] where U3 represents the output of the shallow feature after the product of the weight vector.

[0111] Finally, an upsampling operation is performed on the high-level feature T4 to make the elements on T4 correspond one by one to the low-level feature map T3. Then, the two feature maps are directly added according to the corresponding pixels, and the formula is as follows:

[0112] P3 = UP(T4) + U3

[0113] 2.3) Repeat the above two steps, input the high-level feature P3 and the underlying feature T2 into the second fusion module to obtain P2, and input the high-level feature P2 and the underlying feature T1 into the third fusion module to finally obtain P1.

[0114] 3) Feature refinement: Perform channel ladder refinement operation before upsampling.

[0115] Such as Figure 4As shown in the figure, the first layer of this module is a 1×1 convolution, which we use to reduce the number of channels and computational complexity. At the same time, it can also combine information from all channels. Next is a basic residual unit in the residual network. The 3×3 and 5×5 convolutional layers in this residual unit are equivalent to performing two refinement operations with different receptive fields. In this way, the goal of gradually refining the feature map can be achieved.

[0116] 4) Evaluation: As Figure 5 , 6 shown, experiments conducted on the publicly available Potsdam dataset and Vaihingen dataset indicate that our method has a certain effect on remote sensing image segmentation, effectively alleviating the problems of misclassification and missed classification caused by complex image backgrounds and large differences in class sizes.

[0117] Application Example 1:

[0118] This application example adopts the remote sensing image segmentation method based on multi-scale fusion attention in Example 1 and applies it to the datasets Potsdam and Vaihingen of high-resolution remote sensing images to verify its effectiveness. The experimental results are shown in Tables 1, 2, and 3.

[0119] As shown in Table 1 below, considering the meanings of the evaluation metrics MIoU, OA, and F1-score, this method has significant effects on the remote sensing segmentation dataset Potsdam. The multi-scale feature representation module (MFR), feature fusion attention module (FFA), and feature refinement module (FRM) added to the network all have certain effects. Compared with the Baseline, the MIoU and OA of FFANet have increased by 2.6% and 1.6% respectively, and the IoU of each category has also increased accordingly. This proves that the network can effectively improve the segmentation accuracy. To further prove the effectiveness of our network for remote sensing segmentation images, we visualized each stage of the network ablation experiment. Figure 1 Shows the segmentation results of remote sensing images at each stage of the network. As Figure 1 shown in the second row of pictures, when only ResNet-50 is used for segmentation, the feature extraction is incomplete, there are breaks in categories such as light blue low vegetation, and the recognition rates of two categories with similar features, trees and low vegetation, are poor. After adding the MFR and FFA modules, the feature extraction is optimized and the misclassification problem is alleviated. When using the three modules MFR, FFA, and FRM simultaneously, we find that the edge noise of each category is reduced and the segmentation edge results are clearer.

[0120] Table 1 Ablation experiment of FFANet on the Potsdam dataset

[0121] Table 1 Ablation experiment of FFANet on Potsdam dataset

[0122]

[0123] Note: The bold font represents the optimal result.

[0124] As shown in Table 2. To evaluate the network more comprehensively, we compared the model with five existing classical networks on the Potsdam dataset. The MIoU and OA metrics of FCN are the lowest because it uses VGG16 as the backbone network, which has weaker feature extraction ability compared to ResNet-50. PSPNet uses the pyramid pooling module to extract context information, and DeeplabV3+ uses dilated convolution to increase the receptive field, both of which improve the segmentation accuracy to a certain extent. However, these networks do not consider the connection between shallow features and deep features in the feature fusion part, but only directly concatenate them in the channel dimension, resulting in poor extraction ability for the background features of complex remote sensing images. The proposed FFANet network uses the attention mechanism to establish the connection between different feature layers, and both the MIoU and OA metrics exceed these semantic segmentation models. The MIoU, OA, and F1-socre of FFANet reach 74.4%, 87.7%, and 84.9% respectively.

[0125] Table 2 Comparative experiment of FFANet on Potsdam dataset

[0126] Table 2 Ablation experiment of FFANet on Potsdam data set

[0127]

[0128] Note: The bold font represents the optimal result.

[0129] To verify the generalization ability of the model, ablation experiments were carried out on the Vaihingen dataset. The experimental results are shown in Table 3. From the data in the table, it can be seen that the three modules of the network also have certain effects on the Vaihingen dataset. The visualization results are as Figure 3 shown. Due to the lack of multi-scale feature representation and feature fusion, the Baseline has incomplete feature extraction and poor segmentation results. While the multi-scale feature representation module (MFR), feature fusion attention module (FFA), and feature refinement module (FRM) enable the network to segment targets that could not be recognized originally, correct misclassifications, and improve the segmentation accuracy.

[0130] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

[0131] A computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform a remote sensing image segmentation method based on multi-scale fusion attention.

[0132] A computing device, comprising one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing a remote sensing image segmentation method based on multi-scale fusion attention.

[0133] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0135] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, thereby the instructions executed on the computer or other programmable apparatus provide steps for realizing the processing in the flow Figure 1 one process or a plurality of processes and / or blocks Figure 1 steps of the functions specified in one block or a plurality of blocks.

[0137] The above are only embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are included within the scope of the claims of the present invention pending approval of the application.

Claims

1. A remote sensing image segmentation method based on multi-scale fusion attention, characterized in that: Obtain dataset images, input the dataset images into a pre-constructed Residual Network ResNet-50 to obtain deep semantic feature information, the deep semantic feature information being feature matrices M1, M2, M3, and M. Then input the obtained feature matrix M into a pre-constructed multi-scale feature representation module, and the multi-scale feature representation module performs feature aggregation through dilated convolutions with different dilation rates; Output feature matrix M4, Wherein, M represents the output of the last layer of ResNet-50; represents a dilated convolution operation with a convolution kernel size of 3 and a dilation rate of 1; BN represents further batch normalization; ReLU represents a non-linear activation layer; GAP represents a global average pooling layer; UP represents upsampling to the original image size; Concat represents adding each feature in the channel dimension; Respectively perform dimensionality reduction on the feature matrices M1, M2, M3 output by the Residual Network ResNet-50 and the feature matrix M4 output by the multi-scale feature representation module to obtain T1, T2, T3, and T4, and then input them into a pre-constructed feature fusion module FFA. The process of the feature fusion module FFA fusing the dimensionality-reduced features in order from high-level to low-level is as follows: The feature fusion module FFA includes three fusion modules. Input T4 and T3 into the first fusion module. T4 is used as the high-level feature of the first fusion module, and T3 is used as the low-level feature. First perform an upsampling operation on T4 to obtain a feature matrix of the same size as T3, and fuse the two through an addition operation to obtain a fused feature matrix I3. Finally, use a convolutional operator with a kernel size of 3 to reduce the dimensionality of I3 and output a feature matrix K3 of the same size as T3: I3 = Concat(UP(T4), T3) K3 = ReLU(BN(C 3×3 (I3))) Wherein, UP represents upsampling, which restores T4 to the size of T3, Concat is an addition operation that adds up the high-level features and low-level features channel by channel, and C 3×3 is a convolution operator with a convolution kernel size of 3; Then, the global average pooling is used to guide the underlying feature information, and the high-level feature is set as T4 = [t1, t2, … t c1 ∈ R c1 ×h×w , and the underlying feature is K3 = [k1, k2, … k c2 ∈ R c2×h×w , a global average pooling operation is performed on the high-level feature T4 to obtain global information, and the obtained global information is represented as A ∈ R c1×1×1 , and this process is expressed as: where h and w respectively represent the length and width of the high-level feature T4, and A k the value of the k-th layer in the global information, where k = 1, 2... c1; A is reduced in dimension through a 1×1 convolution to obtain a feature matrix with a length and width of 1 and a number of channels of C2. The weight values are divided between [0,1] through the Sigmoid activation function to generate a weight vector V∈R c2×1×1 , the weight vector is multiplied by the underlying feature K3, followed by a 1×1 convolution operation, BN batch normalization, and ReLU activation. This process is expressed as: U3 = ReLU(BN(C 1×1 (K3 × V))) where U3 represents the output after the underlying features are multiplied by the weight vector, and C 1×1 is a convolution operator with a convolution kernel size of 1; Finally, perform an upsampling operation on the high-level feature T4 so that the elements on T4 correspond one by one to U3. Then, directly add the high-level feature T4 after the upsampling operation and U3 according to the corresponding pixels to obtain P3. The formula is as follows: P3 = UP(T4) + U3 Similarly, input the high-level feature P3 and the low-level feature T2 into the second fusion module to obtain the high-level feature P2, and input the high-level feature P2 and the low-level feature T1 into the third fusion module to finally obtain the high-level feature P1; Input the fused high-level feature P1 into a pre-constructed feature refinement module. The feature refinement module consists of a 1×1 convolutional layer and a residual unit. Use the 1×1 convolutional layer to reduce the number of channels. The 3×3 and 5×5 convolutional layers of the residual unit are equivalent to performing two refinement operations with different receptive fields to obtain a refined output feature.

2. The remote sensing image segmentation method based on multi-scale fusion attention according to claim 1, characterized in that: Perform dimensionality reduction on the feature matrix M1 to obtain: T1 = ReLU(BN(C 1×1 (M1))) Perform dimensionality reduction on the feature matrix M2 to obtain: T2 = ReLU(BN(C 1×1 (M2))) Perform dimensionality reduction on the feature matrix M3 to obtain: T3 = ReLU(BN(C 1×1 (M3))) Perform dimensionality reduction on the feature matrix M4 to obtain: T4 = ReLU(BN(C 1×1 (M4))) where C 1×1 represents a convolution operation with a convolution kernel size of 1; BN represents further batch normalization to accelerate the training speed; ReLU represents a non-linear activation layer.

3. Remote sensing image segmentation system based on multi-scale fusion attention, characterized in that: Including: A multi-scale feature representation module MFR, used to obtain dataset images, input the dataset images into a pre-constructed Residual Network ResNet-50 to obtain deep semantic feature information, the deep semantic feature information being feature matrices M1, M2, M3, and M. Then input the obtained feature matrix M into a pre-constructed multi-scale feature representation module, and the multi-scale feature representation module performs feature aggregation through dilated convolutions with different dilation rates; Output feature matrix M4, Wherein, M represents the output of the last layer of ResNet-50; represents a dilated convolution operation with a convolution kernel size of 3 and a dilation rate of 1; BN represents further batch normalization; ReLU represents a non-linear activation layer; GAP represents a global average pooling layer; UP represents upsampling to the original image size; Concat represents adding each feature in the channel dimension; The feature matrices M1, M2, M3 output by the ResNet-50 residual network and the feature matrix M4 output by the multi-scale feature representation module are respectively dimension-reduced to obtain T1, T2, T3, and T4, and then input into the pre-constructed feature fusion module FFA. Feature fusion module FFA: The process of sequentially fusing the dimension-reduced features in the order from high-level to low-level is as follows: The feature fusion module FFA includes three fusion modules. Input T4 and T3 into the first fusion module. T4 serves as the high-level feature of the first fusion module, and T3 serves as the low-level feature. First, perform an upsampling operation on T4 to obtain a feature matrix with the same size as T3, and fuse the two through an addition operation to obtain the fused feature matrix I3. Finally, use a convolutional operator with a kernel size of 3 to reduce the dimension of I3 and output a feature matrix K3 with the same size as T3: I3 = Concat(UP(T4), T3) K3 = ReLU(BN(C 3×3 (I3))) Wherein, UP represents upsampling, which restores T4 to the size of T3, Concat is an addition operation that adds the high-level features and the low-level features according to the number of channels, and C 3×3 is a convolution operator with a convolution kernel size of 3; Then, the global average pooling is used to guide the low-level feature information, and the high-level feature is set as T4 = [t1, t2, … t c1 ∈ R c1 ×h×w , and the low-level feature is K3 = [k1, k2, … k c2 ∈ R c2×h×w . The global average pooling operation is performed on the high-level feature T4 to obtain the global information, and the obtained global information is represented as A ∈ R c1×1×1 . This process is expressed as: Where h and w respectively represent the length and width of the high-level feature T4, and A k The value of the k-th layer in the global information, where k = 1, 2... c1; A is reduced in dimension through a 1×1 convolution to obtain a feature matrix with a length and width of 1 and a channel number of C2. The weight values are divided between [0,1] through the Sigmoid activation function to generate a weight vector V∈R C2×1×1 , the weight vector is multiplied by the underlying feature K3, followed by a 1×1 convolution operation, BN batch normalization, and ReLU activation. This process is expressed as: U3 = ReLU(BN(C 1×1 (K3 × V))) where U3 represents the output after the underlying features are multiplied by the weight vector, and C 1×1 is a convolution operator with a convolution kernel size of 1; Finally, perform an upsampling operation on the high-level feature T4 so that the elements on T4 correspond one-to-one with U3. Then, directly add the high-level feature T4 after the upsampling operation and U3 according to the corresponding pixels to obtain P3. The formula is as follows: P3 = UP(T4) + U3 Similarly, input the high-level feature P3 and the low-level feature T2 into the second fusion module to obtain the high-level feature P2, and input the high-level feature P2 and the low-level feature T1 into the third fusion module to finally obtain the high-level feature P1; Input the fused high-level feature P1 into the pre-constructed feature refinement module. Feature refinement module FRM: The feature refinement module consists of a 1×1 convolutional layer and a residual unit. Use the 1×1 convolutional layer to reduce the number of channels. The 3×3 and 5×5 convolutional layers of the residual unit are equivalent to performing two refinement operations with different receptive fields to obtain the refined output feature.

4. The remote sensing image segmentation system based on multi-scale fusion attention according to claim 3, wherein: Perform dimension reduction on the feature matrix M1 to obtain: T1 = ReLU(BN(C 1×1 (M1))) Perform dimension reduction on the feature matrix M2 to obtain: T2 = ReLU(BN(C 1×1 (M2))) Perform dimension reduction on the feature matrix M3 to obtain: T3 = ReLU(BN(C 1×1 (M3))) Perform dimension reduction on the feature matrix M4 to obtain: T4 = ReLU(BN(C 1×1 (M4))) In the formula, C 1×1 represents a convolution operation with a convolution kernel size of 1; BN represents further batch normalization to accelerate the training speed; ReLU represents a non-linear activation layer.

5. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions that, when executed by a computing device, cause the computing device to execute any one of the methods according to claims 1 to 2.

6. A computing device, characterized in that, Include: One or more processors, one or more memories, and one or more programs, where the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors. The one or more programs include instructions for executing any one of the methods according to claims 1 to 2.

Images