A semantic segmentation method of an optical remote sensing image

Abstract

The application provides a semantic segmentation method of an optical remote sensing image, and relates to the technical field of image processing, and the method comprises the following steps: adopting a downsampling module to perform downsampling processing on a target pixel size optical remote sensing image to be processed to obtain a first feature map and obtain first normalized features; adopting a linear attention module to perform linear attention processing on the first normalized features, adopting a second normalization layer to normalize target global features to obtain second normalized features; adopting a multilayer perceptron to perform feature mapping to obtain a second feature map with the same pixel size as the first feature map; adopting an upsampling module to perform upsampling processing on the second feature map to obtain a target feature map with a target pixel size; and adopting a feature segmentation module to perform feature segmentation on the target feature map to obtain a semantic segmentation map of the optical remote sensing image. The application improves the semantic segmentation calculation accuracy, reduces the calculation complexity, and reduces the demand for computing resources.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a semantic segmentation method for optical remote sensing images. Background Technology

[0002] Semantic segmentation of optical remote sensing images is a crucial task in the fields of remote sensing and computer vision. Its goal is to assign a corresponding category label to each pixel and generate a fine-grained pixel-level segmentation map. This technology has significant value in various application areas such as urban planning, environmental monitoring, and disaster assessment. However, achieving high-precision segmentation faces considerable challenges due to the complex terrain structures, diverse scale variations, and rich detail information often presented in remote sensing images.

[0003] Currently, deep learning technology is mainly used for image segmentation of optical remote sensing images. Convolutional operations can capture local spatial relationships in images, such as edges, corners, and textures, which are crucial information in image segmentation. Through multiple layers of convolution and pooling, the network can gradually abstract more complex features, thereby recognizing objects in images at different scales.

[0004] However, deep learning techniques have limitations in capturing global spatial information and handling long-range dependencies in images, especially in high-resolution remote sensing image scenarios, where computational efficiency is a significant issue. It cannot guarantee both image processing accuracy and reduced computational complexity while processing high-resolution remote sensing images. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a semantic segmentation method for optical remote sensing images, which improves the accuracy of semantic segmentation calculation, reduces computational complexity, and lowers the demand for computing resources.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, one embodiment of this application provides a semantic segmentation method for optical remote sensing images, the method comprising:

[0008] An image segmentation network model is applied, comprising: a feature extraction module and a feature segmentation module. The feature extraction module includes: a downsampling module, a linear attention network, and an upsampling module. The linear attention network includes: a first normalization layer, a linear attention module, a second normalization layer, and a multilayer perceptron. The method includes:

[0009] The downsampling module is used to downsample the optical remote sensing image of the target pixel size to be processed, and a first feature map is obtained.

[0010] The first feature map is normalized using the first normalization layer to obtain the first normalized feature.

[0011] The linear attention module is used to perform linear attention processing on the first normalized feature to obtain the target global feature with contextual information;

[0012] The target global features are normalized using the second normalization layer to obtain the second normalized features;

[0013] The multilayer perceptron is used to perform feature mapping on the second normalized feature to obtain a second feature map with the same pixel size as the first feature map;

[0014] The upsampling module is used to upsample the second feature map to obtain a target feature map of the target pixel size;

[0015] The target feature map is segmented using the feature segmentation module to obtain the semantic segmentation map of the optical remote sensing image.

[0016] Optionally, the linear attention module includes: a linear processing layer, a first aggregation layer, a second aggregation layer, a third aggregation layer, a fourth aggregation layer, a first attention calculation layer, and a second attention calculation layer; the step of using the linear attention module to perform linear processing on the first normalized feature to obtain the target global feature with contextual information includes:

[0017] The first normalized feature is linearly processed by the linear processing layer to obtain a first linear query vector, a first linear key vector, and a first linear value vector corresponding to the first linear key vector.

[0018] The first aggregation layer is used to aggregate the first global feature corresponding to the first linear query vector and the first linear key vector to obtain a first aggregated feature with contextual information.

[0019] The first attention calculation layer is used to perform attention calculation on the first aggregated features to obtain the first attention matrix;

[0020] The second aggregation layer is used to aggregate and calculate the first attention matrix and the first linear value vector to obtain the second global feature.

[0021] The activation function features of the first linear query vector and the first global features are aggregated and calculated using the third aggregation layer to obtain the second aggregated features;

[0022] The second attention calculation layer is used to perform attention calculation on the second aggregated features to obtain the second attention matrix;

[0023] The fourth aggregation layer is used to aggregate and calculate the second attention matrix and the first global feature to obtain the target global feature.

[0024] Optionally, the downsampling module includes: multiple convolutional layers, each of which corresponds to a multiple pixel size;

[0025] The downsampling module processes the optical remote sensing image of the target pixel size to be processed to obtain a first feature map, including:

[0026] The optical remote sensing image is sequentially convolved using the multiple convolutional layers to obtain a first feature map of the multiple pixel sizes;

[0027] The step of normalizing the first feature map using the first normalization layer to obtain the first normalized feature includes:

[0028] The first normalization layer is used to normalize the first feature map with the smallest pixel size to obtain the first normalized feature.

[0029] Optionally, the upsampling module includes: multiple deconvolutional layers; the multiple deconvolutional layers respectively correspond to the multiple pixel sizes; the step of using the upsampling module to upsample the second feature map to obtain a target feature map of the target pixel size includes:

[0030] The second feature map is deconvolved sequentially using the multiple deconvolution layers to obtain a third feature map of multiple pixel sizes, and the third feature map with the largest pixel size is determined as the target feature map.

[0031] Optionally, the upsampling module further includes: multiple gating modules, each corresponding to one of the multiple pixel sizes; the method further includes:

[0032] For each gated module, feature fusion is performed on the first feature map and the second feature map corresponding to the pixel size of each gated module to obtain the fused feature map corresponding to the pixel size;

[0033] The step of sequentially deconvolutionally processing the second feature map using the multiple deconvolution layers to obtain a third feature map of multiple pixel sizes, and determining the third feature map with the largest pixel size as the target feature map, includes:

[0034] Each deconvolutional layer is used to deconvolve the fused feature map of the corresponding pixel size to obtain the third feature map of the next pixel size.

[0035] Optionally, each gating module includes: a first mixing unit, a second mixing unit, a weighting function unit, and a connection function unit; the step of using each gating module to perform feature fusion on the first feature map and the second feature map of the corresponding pixel size to obtain the fused feature map of the corresponding pixel size includes:

[0036] The weight function unit is used to calculate the weights of the first feature map and the second feature map of the corresponding pixel size to obtain the first weight parameter;

[0037] Using the first mixing unit, a weighted calculation is performed based on the first weight parameter and the first feature map of the corresponding pixel size to obtain a first weighted feature map;

[0038] The second mixing unit is used to perform a weighted calculation based on the second weight parameter corresponding to the first weight parameter and the second feature map of the corresponding pixel size to obtain a second weighted feature map; wherein the sum of the first weight parameter and the second weight parameter is 1;

[0039] The first weighted feature map and the second weighted feature map are concatenated using the connection function unit to obtain a fused feature map of the corresponding pixel size.

[0040] Optionally, each gating module includes: a first convolutional unit, a second convolutional unit, a first mixing unit, a second mixing unit, a weighting function unit, and an addition unit;

[0041] The step of employing each gating module and fusing the first and second feature maps corresponding to the pixel size of each gating module to obtain the fused feature map of the corresponding pixel size includes:

[0042] The first convolutional unit and the second convolutional unit are used to perform convolution processing on the first feature map and the second feature map of the corresponding pixel size, respectively, to obtain the first mapping feature and the second mapping feature;

[0043] The weight function unit is used to calculate the weights of the first mapping feature and the second mapping feature to obtain the first weight parameter;

[0044] Using the first hybrid unit, a weighted calculation is performed based on the first weight parameter and the first mapping feature to obtain a first weighted feature map;

[0045] Using the second hybrid unit, a weighted calculation is performed based on the second weight parameter corresponding to the first weight parameter and the second mapping feature to obtain a second weighted feature map; wherein, the sum of the first weight parameter and the second weight parameter is 1;

[0046] The addition unit is used to perform addition processing on the first weighted feature map and the second weighted feature map to obtain the fused feature map of the corresponding pixel size.

[0047] Optionally, each gating module includes: a first convolutional unit, a second convolutional unit, a first mixing unit, a second mixing unit, a weighting function unit, and a connection function unit;

[0048] The step of employing each gating module and fusing the first and second feature maps corresponding to the pixel size of each gating module to obtain the fused feature map of the corresponding pixel size includes:

[0049] The first convolutional unit and the second convolutional unit are used to perform convolution processing on the first feature map and the second feature map of the corresponding pixel size, respectively, to obtain the first mapping feature and the second mapping feature;

[0050] The weight function unit is used to calculate the weights of the first mapping feature and the second mapping feature to obtain the first weight parameter;

[0051] Using the first mixing unit, a weighted calculation is performed based on the first weight parameter and the first feature map of the corresponding pixel size to obtain a first weighted feature map;

[0052] Using the first mixing unit, a second weighted feature map is obtained by weighting the second weight parameter corresponding to the first weight parameter and the second feature map of the corresponding pixel size; wherein, the sum of the first weight parameter and the second weight parameter is 1;

[0053] The first weighted feature map and the second weighted feature map are concatenated using the connection function unit to obtain a fused feature map of the corresponding pixel size.

[0054] Optionally, the feature segmentation module includes: a feature mapping unit and a generation unit;

[0055] The step of using the feature segmentation module to perform feature segmentation on the target feature map to obtain the semantic segmentation map of the optical remote sensing image includes:

[0056] The feature mapping unit performs a convolution operation on the target feature map according to a preset number of categories to obtain a new feature map. Each pixel in the new feature map has a corresponding category label and the probability of the corresponding category.

[0057] The generation unit generates the semantic segmentation map based on the new feature map.

[0058] Optionally, before the downsampling module performs downsampling processing on the optical remote sensing image of the target pixel size to be processed to obtain the first feature map, the method further includes:

[0059] The original remote sensing image is cropped to obtain an optical remote sensing image of the target pixel size.

[0060] Secondly, another embodiment of this application provides a semantic segmentation device for optical remote sensing images, applied to an image segmentation network model. The image segmentation network model includes a feature extraction module and a feature segmentation module. The feature extraction module includes a downsampling module, a linear attention network, and an upsampling module. The linear attention network includes a first normalization layer, a linear attention module, a second normalization layer, and a multilayer perceptron. The device includes:

[0061] The first sampling module is used to perform downsampling processing on the optical remote sensing image of the target pixel size to be processed by the downsampling module to obtain the first feature map;

[0062] The first processing module is used to normalize the first feature map using the first normalization layer to obtain the first normalized feature.

[0063] The second processing module is used to perform linear attention processing on the first normalized feature using the linear attention module to obtain the target global feature with contextual information.

[0064] The third processing module is used to normalize the target global features using the second normalization layer to obtain the second normalized features;

[0065] The mapping module is used to perform feature mapping on the second normalized feature using the multilayer perceptron to obtain a second feature map with the same pixel size as the first feature map.

[0066] The second sampling module is used to perform upsampling processing on the second feature map using the upsampling module to obtain a target feature map of the target pixel size;

[0067] The segmentation module is used to perform feature segmentation on the target feature map using the feature segmentation module to obtain the semantic segmentation map of the optical remote sensing image.

[0068] Thirdly, another embodiment of this application provides a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.

[0069] Fourthly, another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of any of the methods described in the first aspect above.

[0070] The beneficial effects of this application are:

[0071] This application provides a semantic segmentation method for optical remote sensing images, applied to an image segmentation network model. A downsampling module downsamples the optical remote sensing image at the target pixel size to obtain a first feature map. A first normalization layer normalizes the first feature map to obtain a first normalized feature. A linear attention module performs linear attention processing on the first normalized feature to obtain a global feature with contextual information. A second normalization layer normalizes the global feature to obtain a second normalized feature. A multilayer perceptron performs feature mapping on the second normalized feature to obtain a second feature map with the same pixel size as the first feature map. An upsampling module upsamples the second feature map to obtain a target feature map at the target pixel size. A feature segmentation module performs feature segmentation on the target feature map to obtain a semantic segmentation map of the optical remote sensing image. This application includes a linear attention module for linear processing of the first normalized feature, which effectively handles long-distance dependencies in optical remote sensing images. It utilizes multi-scale semantic segmentation to handle complex spatial relationships, improving computational accuracy, reducing computational complexity, and lowering the demand for computational resources. Attached Figure Description

[0072] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a schematic diagram of the structure of the first image segmentation network model provided in the embodiments of this application;

[0074] Figure 2 A flowchart illustrating a semantic segmentation method for optical remote sensing images provided in an embodiment of this application;

[0075] Figure 3 This is a schematic diagram of the structure of an image segmentation network model provided in an embodiment of this application;

[0076] Figure 4 A flowchart illustrating the process of determining global features of a target in a semantic segmentation method for optical remote sensing images provided in this application embodiment;

[0077] Figure 5 This is a schematic diagram of the structure of a downsampling module in an image segmentation network model provided in an embodiment of this application;

[0078] Figure 6 A flowchart illustrating the determination of the first normalized feature in a semantic segmentation method for optical remote sensing images provided in this application embodiment;

[0079] Figure 7 This is a schematic diagram of the structure of an upsampling module in an image segmentation network model provided in an embodiment of this application;

[0080] Figure 8 A schematic diagram of the upsampling module in another image segmentation network model provided in this application embodiment;

[0081] Figure 9 A schematic diagram illustrating the process of determining the third feature map in a semantic segmentation method for remote sensing images provided in this application embodiment;

[0082] Figure 10 A schematic diagram of the gating module in the first image segmentation network model provided in this application embodiment;

[0083] Figure 11 This is a schematic diagram illustrating the process of determining a fused feature map in a semantic segmentation method for optical remote sensing images provided in an embodiment of this application.

[0084] Figure 12 A schematic diagram of the gating module in the second image segmentation network model provided in this application embodiment;

[0085] Figure 13 A schematic diagram illustrating the process of determining the fused feature map in the second semantic segmentation method for optical remote sensing images provided in this application embodiment;

[0086] Figure 14 A schematic diagram of the gating module in the third image segmentation network model provided in this application embodiment;

[0087] Figure 15 A schematic diagram illustrating the process of determining the fused feature map in the third semantic segmentation method for optical remote sensing images provided in this application embodiment;

[0088] Figure 16 This is a schematic diagram of the feature segmentation module in an image segmentation network model provided in an embodiment of this application;

[0089] Figure 17 This is a schematic diagram illustrating the process of generating a semantic segmentation map in a semantic segmentation method for optical remote sensing images provided in an embodiment of this application.

[0090] Figure 18 A schematic diagram of a semantic segmentation device for optical remote sensing images provided in an embodiment of this application;

[0091] Figure 19 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0093] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0094] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0095] To clearly describe the semantic segmentation method for optical remote sensing images provided in the embodiments of this application, the semantic segmentation method for optical remote sensing images will be described below in conjunction with several accompanying drawings. The semantic segmentation method for optical remote sensing images in this application is applied to an image segmentation network model, which can be a convolutional neural network model. Figure 1 A schematic diagram of the structure of the first image segmentation network model provided in the embodiments of this application is shown below. Figure 1As shown, the image segmentation network model includes a feature extraction module 100 and a feature segmentation module 200. The feature extraction module 100 includes a downsampling module 101, a linear attention network 102, and an upsampling module 103. The linear attention network 102 includes a first normalization layer 1021, a linear attention module 1022, a second normalization layer 1023, and a multilayer perceptron 1024. Specifically, the feature extraction module 100 extracts low-level and high-level features from the optical remote sensing image, and the feature segmentation module 200 determines the label of each pixel in the optical remote sensing image. The downsampling module 101 converts the image from a high-resolution image to a low-resolution image. The linear attention network 102 extracts the target global features of the image. The upsampling module 103 converts the image from a low-resolution image to a high-resolution image. The first normalization layer 1021 normalizes the pixel values ​​in the image. The linear attention module 1022 extracts global features with contextual information from the image. The second normalization layer 1023 is used to normalize the global features of the extracted image that have contextual information. The multilayer perceptron 1024 is used to learn data features and perform feature transformation.

[0096] Figure 2 A flowchart illustrating a semantic segmentation method for optical remote sensing images provided in this application embodiment is shown below. Figure 2 As shown, the method includes:

[0097] Step 201: Use a downsampling module to downsample the optical remote sensing image of the target pixel size to be processed, and obtain the first feature map.

[0098] The downsampling module is used to convert an optical remote sensing image with a target pixel size into a low-resolution optical remote sensing image. The pixel size of the first feature map is smaller than the target pixel size. In this application, there can be multiple first feature maps, and this embodiment does not limit this. When there are multiple first feature maps, the pixel sizes of the multiple first feature maps are different and all are smaller than the target pixel size.

[0099] Optionally, a downsampling module is used to downsample the optical remote sensing image of the target pixel size to be processed, so that the optical remote sensing image of the target pixel size becomes a low-resolution optical remote sensing image, which is the first feature map.

[0100] Step 202: Normalize the first feature map using the first normalization layer to obtain the first normalized feature.

[0101] The first normalization layer is used to normalize the pixel values ​​in the first feature map.

[0102] Optionally, the first feature map can be normalized using formula (1).

[0103]

[0104] Where, x (i) Let be the pixel value of the i-th pixel in the first feature map. μ is the normalized value of the i-th pixel. B The mean of the pixel values ​​in the first feature map. γ is the variance of pixel values ​​in the first feature map, γ is the training scaling parameter of the first normalization layer, and β is the training translation parameter of the first normalization layer.

[0105] Step 203: Apply linear attention to the first normalized features using a linear attention module to obtain target global features with contextual information.

[0106] The linear attention module is used to extract global features with contextual information from the image. Contextual information refers to the relationships between pixels in the optical remote sensing image of the target pixel size and the association between each first normalized feature and the entire scene. The target global features are the features of the optical remote sensing image of the target pixel size, and these global features can represent the attributes of the entire optical remote sensing image of the target pixel size.

[0107] Optionally, a linear attention module is used to perform linear attention processing on the first normalized feature, and the first normalized feature interacts with the context to obtain the target global feature with contextual information.

[0108] Step 204: Use the second normalization layer to normalize the global features of the target, and obtain the second normalized features.

[0109] The second normalization layer is used to normalize the global features of the target.

[0110] Optionally, a second normalization layer is used to normalize the global features of the target to obtain the second normalized features, or a second normalization layer is used to normalize both the global features of the target and the first feature map to obtain the second normalized features.

[0111] Step 205: Use a multilayer perceptron to perform feature mapping on the second normalized features to obtain a second feature map with the same pixel size as the first feature map.

[0112] Among them, the multilayer perceptron is used to learn data features and perform feature transformation.

[0113] Optionally, a multilayer perceptron is used to perform feature mapping on the second normalized features, and the second normalized features are compressed and transformed to obtain a second feature map with the same pixel size as the first feature map, so that the second feature map can be used for subsequent convolution operations and feature extraction.

[0114] Step 206: Upsample the second feature map using an upsampling module to obtain the target pixel size.

[0115] Mark the feature map.

[0116] The upsampling module is used to convert low-resolution images into high-resolution images.

[0117] Optionally, an upsampling module is used to upsample the second feature map to obtain a target feature map with the target pixel size. The target pixel size is the target pixel size of the optical remote sensing image that is the same as the target pixel size to be processed.

[0118] Step 207: Use the feature segmentation module to perform feature segmentation on the target feature map to obtain the semantic segmentation map of the optical remote sensing image.

[0119] The feature segmentation module is used to determine the analogy of each pixel in the optical remote sensing image. The semantic segmentation map of the optical remote sensing image is an image in which each pixel in the optical remote sensing image is classified and labeled according to its corresponding land cover category.

[0120] Optionally, a feature segmentation module can be used to segment the target feature map, which can effectively extract semantic information from the optical remote sensing image, thereby obtaining a semantic segmentation map of the optical remote sensing image based on the semantic information.

[0121] This application provides a semantic segmentation method for optical remote sensing images, applied to an image segmentation network model. A downsampling module downsamples the optical remote sensing image at the target pixel size to obtain a first feature map. A first normalization layer normalizes the first feature map to obtain a first normalized feature. A linear attention module performs linear attention processing on the first normalized feature to obtain a target global feature with contextual information. A second normalization layer normalizes the target global feature to obtain a second normalized feature. A multilayer perceptron performs feature mapping on the second normalized feature to obtain a second feature map with the same pixel size as the first feature map. An upsampling module upsamples the second feature map to obtain a target feature map at the target pixel size. A feature segmentation module performs feature segmentation on the target feature map to obtain a semantic segmentation map of the optical remote sensing image. This application includes a linear attention module for linear processing of the first normalized feature, which effectively handles long-distance dependencies in remote sensing images. It utilizes multi-scale semantic segmentation to handle complex spatial relationships, improving computational accuracy, reducing computational complexity, and lowering the demand for computational resources.

[0122] Based on the above embodiments, the linear attention module includes: a linear processing layer, a first aggregation layer, a second aggregation layer, a third aggregation layer, a fourth aggregation layer, a first attention calculation layer, and a second attention calculation layer. Figure 3 This is a schematic diagram of the structure of an image segmentation network model provided in an embodiment of this application, such as... Figure 3 As shown, the linear attention module 1022 includes: a linear processing layer 10221, a first aggregation layer 10222, a second aggregation layer 10223, a third aggregation layer 10224, a fourth aggregation layer 10225, a first attention calculation layer 10226, and a second attention calculation layer 10227. The linear processing layer 10221 performs a linear transformation on the first normalized features, mapping the first normalized features to the linear attention module. The first aggregation layer 10222, the second aggregation layer 10223, the third aggregation layer 10224, and the fourth aggregation layer 10225 are all used for aggregation calculations. The first attention calculation layer 10226 and the second attention calculation layer 10227 are used for attention calculations.

[0123] Therefore, this application also provides a process for determining global features of a target in a semantic segmentation method for optical remote sensing images. Figure 4 This is a flowchart illustrating the determination of global target features in a semantic segmentation method for optical remote sensing images provided in this application embodiment. For example, in step 203 above, a linear attention module is used to linearly process the first normalized features to obtain global target features with contextual information, including:

[0124] Step 401: Perform linear processing on the first normalized feature using a linear processing layer to obtain the first linear query vector, the first linear key vector, and the first linear value vector corresponding to the first linear key vector.

[0125] Here, the first linear query vector is Query(Q), which is used to extract relevant information in the linear processing layer, representing the context information to be queried. The first linear key vector is Key(K), which is used to match with the first linear query vector to determine the relevance of each input feature to the current query. The first linear value vector is Value(V), which contains the actual information of the input features and is used to update the actual output of the linear processing layer.

[0126] Step 402: The first aggregation layer is used to aggregate the first global feature and the first linear key vector corresponding to the first linear query vector to obtain the first aggregated feature with contextual information.

[0127] The first global feature corresponding to the first linear query vector is a new feature vector generated based on the first linear query vector. The first global feature includes context information, and the number of first global features can be flexibly set; this embodiment does not limit this. The generation strategy can be to apply an additional weight matrix to the first linear query vector for linear transformation, or to apply a nonlinear function to the first linear query vector, etc.; this embodiment does not limit this. The aggregation calculation is mainly used to calculate the dot product of the first global feature and the first linear key vector.

[0128] Optionally, before using the first aggregation layer to perform aggregation calculation on the first global feature and the first linear key vector corresponding to the first linear query vector, it is also necessary to optimize the first linear key vector using a nonlinear function.

[0129] Optionally, a first global feature corresponding to the first linear query vector is generated through the first linear query vector, the first linear key vector is optimized through a nonlinear function, and the first global feature and the optimized first key vector are aggregated using a first aggregation layer to obtain a first aggregated feature with contextual information.

[0130] Step 403: Use the first attention calculation layer to perform attention calculation on the first aggregated feature to obtain the first attention matrix.

[0131] Optionally, the first attention calculation layer is used to perform attention calculation on the first aggregated feature to obtain the first attention matrix. Specifically, the first attention matrix can be calculated using formula (2).

[0132]

[0133] Here, softmax is the first attention calculation, GQ is the first aggregated feature with contextual information, and K... T The matrix representing the first linear key vectors is obtained through the first normalized eigenvalue transformation. d k Let S be the dimension of the matrix of the first linear key vector, which is mainly used as a scaling factor. Let S be the first attention matrix, where n×d represents the dimension of S.

[0134] Step 404: Use the second aggregation layer to aggregate and calculate the first attention matrix and the first linear value vector to obtain the second global feature.

[0135] Optionally, the first attention matrix and the first linear value vector are aggregated and calculated through the second aggregation layer. Specifically, the second global feature is obtained by performing a dot product calculation on the first attention matrix and the first linear value vector using formula (3).

[0136] GV∈R n×d =S×V (3)

[0137] Where GV is the second global feature, S is the first attention matrix, V is the first linear value vector, and n×d in GV is the dimension of GV.

[0138] Step 405: The activation function features of the first linear query vector and the first global features are aggregated and calculated using the third aggregation layer to obtain the second aggregated features.

[0139] The activation function is the ReLU function, the first global feature is a new feature vector generated based on the first linear query vector, the first global feature includes context information, and the aggregation calculation is the dot product calculation.

[0140] Optionally, the second aggregated feature is obtained by performing a dot product calculation using the ReLU function features of the first linear query vector of the third aggregation layer and the first global feature generated based on the first linear query vector.

[0141] Step 406: Use the second attention calculation layer to perform attention calculation on the second aggregated features to obtain the second attention matrix.

[0142] Optionally, a second attention calculation layer is used to perform attention calculation on the second aggregated features to obtain a second attention matrix. Specifically, the first attention matrix can be calculated using formula (4).

[0143]

[0144] Here, softmax is the second attention calculation, Q×GQ. T This is a second aggregation feature with contextual information. k Let U be the dimension of the matrix containing the first linear key vector, which is mainly used as a scaling factor. U is the first attention matrix, and N×n in U represents the dimension of U.

[0145] Step 407: Use the fourth aggregation layer to aggregate and calculate the second attention matrix and the first global feature to obtain the target global feature.

[0146] Optionally, the second attention matrix and the first global feature are aggregated and calculated through the fourth aggregation layer. Specifically, the target global feature is obtained by calculating the dot product of the second attention matrix and the first global feature using formula (5).

[0147] O∈R N×d =U×GV (5)

[0148] Where O represents the target global feature, U represents the first attention matrix, GV represents the first global feature, and N×d in O represents the dimension of O.

[0149] For example, combined Figure 3This document describes the processing flow of a linear processing layer provided in an embodiment of this application. The first normalized feature undergoes linear processing by the linear processing layer 10221 to obtain a first linear query vector, a first linear key vector, and a first linear value vector corresponding to the first linear key vector. The first aggregation layer 10222 aggregates the first global feature corresponding to the first linear query vector and the activation function feature of the first linear key vector to obtain a first aggregated feature with contextual information. The first attention calculation layer 10226 performs attention calculation on the first aggregated feature to obtain a first attention matrix. The second aggregation layer 10223 aggregates the first attention matrix and the first linear value vector to obtain a second global feature. The third aggregation layer 10224 aggregates the activation function feature of the first linear query vector and the first global feature to obtain a second aggregated feature. The second attention calculation layer 10226 performs attention calculation on the second aggregated feature to obtain a second attention matrix. The fourth aggregation layer 10225 aggregates the second attention matrix and the first global feature to obtain a target global feature.

[0150] In this embodiment, by setting multiple linear processing layers, multiple aggregation layers, and multiple attention calculation layers, the learning ability of the image segmentation network model can be enhanced by introducing activation functions and a first linear query vector to generate new feature vectors with contextual information. This helps the model better capture dependencies when processing sequential data, thereby improving the model's performance and generalization ability. This application obtains two attention matrices through two attention calculation layers, which reduces the computational complexity of the image segmentation network model. Furthermore, it can extract and aggregate global information from the first linear key vector and the first linear value vector, capturing global dependencies in the first normalized features and providing rich contextual information for subsequent calculations. The aggregation calculation of the second attention matrix and the first global feature can be broadcast to effectively interact the first normalized features with global information. This further enhances the image segmentation network model's ability to capture long-distance dependencies in the input first normalized features, providing rich feature representations for the model's output. Simultaneously, it reduces computational difficulty, preserves information exchange in the first normalized features, maintains the integrity of information transmission when processing large amounts of data, and achieves near-linear computational complexity when processing large amounts of data, significantly improving the processing efficiency of the image segmentation network model.

[0151] Based on the above embodiments, the downsampling module includes: multiple convolutional layers, each corresponding to a multiple pixel size. Figure 5 This is a schematic diagram of the structure of a downsampling module in an image segmentation network model provided in an embodiment of this application, as shown below. Figure 5As shown, the downsampling module 101 includes multiple convolutional layers 1011. Each of the multiple convolutional layers 1011 corresponds to a multiple pixel size.

[0152] Therefore, this application provides a process for determining the first normalized feature in a semantic segmentation method for optical remote sensing images. Figure 6 A flowchart illustrating the determination of the first normalized feature in a semantic segmentation method for optical remote sensing images provided in this application embodiment is shown below. Figure 6 As shown, in step 201 above, the downsampling module processes the optical remote sensing image of the target pixel size to be processed to obtain the first feature map, including:

[0153] Step 601: Perform convolution processing on the optical remote sensing image sequentially using multiple convolutional layers to obtain a first feature map with multiple pixel sizes.

[0154] Multiple convolutional layers are used to obtain a first feature map of multiple pixel sizes.

[0155] For example, when there are three convolutional layers, the input optical remote sensing image has a pixel size of 512×512 and is a 3-channel image; after the first convolutional layer, the resulting first feature map has a pixel size of 256×256 and 128 channels; after the second convolutional layer, the resulting first feature map has a pixel size of 128×128 and 256 channels; after the third convolutional layer, the resulting first feature map has a pixel size of 64×64 and 512 channels. This application only uses three convolutional layers as an example, and does not limit the specific number of convolutional layers or the number of downsampling operations. The number of channels in the first feature map is determined according to the specific application scenario and is not limited to the above embodiment.

[0156] The first feature map is normalized using a first normalization layer to obtain the first normalized features, including:

[0157] Step 602: The first feature map with the smallest pixel size is normalized using the first normalization layer to obtain the first normalized feature.

[0158] The first feature map with the smallest pixel size is the first feature map obtained after the last convolutional layer. According to the example above, the first feature map with the smallest pixel size is the first feature map with a pixel size of 64×64 and a channel size of 512.

[0159] Optionally, before the first feature map with the smallest pixel size enters the normalization layer, it needs to pass through a hidden layer and a linear mapping layer. The specific pixel values ​​of each pixel in the first feature map with the smallest pixel size after passing through the linear mapping layer are normalized to obtain the feature corresponding to the first feature map, which is the first normalized feature.

[0160] In this embodiment, multiple convolutional layers are used to sequentially convolve the optical remote sensing images to obtain first feature maps of multiple pixel sizes. A first normalization layer is then used to normalize the first feature map with the smallest pixel size to obtain a first normalized feature. By performing convolution processing on the optical remote sensing images, this application reduces the computational load, thereby lowering the computational complexity of the image segmentation network model and accelerating processing. Normalizing the first feature map ensures that the dimensions of each pixel in the first feature map are the same, reducing the impact of differences in feature scales, making the model more stable, and improving model accuracy.

[0161] Based on the above embodiments, the upsampling module includes: multiple deconvolution layers. Figure 7 This is a schematic diagram of the structure of an upsampling module in an image segmentation network model provided in an embodiment of this application, as shown below. Figure 7 As shown, the upsampling module 103 includes multiple deconvolution layers 1031, each corresponding to a multiple pixel size.

[0162] Therefore, this application provides a process for determining the target feature map in a semantic segmentation method for optical remote sensing images, such as the upsampling module used in step 104 above to upsample the second feature map to obtain the target feature map of the target pixel size, including:

[0163] Multiple deconvolutional layers are used to sequentially deconvolve the second feature map to obtain a third feature map with multiple pixel sizes, and the third feature map with the largest pixel size is determined as the target feature map.

[0164] Multiple deconvolutional layers are used to upsample the second feature map, thereby obtaining a third feature map of multiple pixel sizes.

[0165] For example, when there are three deconvolution layers, the second input feature map has a pixel size of 64×64 and 512 channels. After the first deconvolution layer, the third feature map has a pixel size of 128×128 and 256 channels; after the second deconvolution layer, the third feature map has a pixel size of 256×256 and 128 channels; and after the third deconvolution layer, the third feature map has a pixel size of 512×512 and 64 channels. This application only uses three deconvolution layers as an example and does not limit the specific number of deconvolution layers or the number of downsampling operations. The number of channels in the third feature map is determined according to the specific application scenario and is not limited to the above embodiment.

[0166] Optionally, before sequentially deconvolutionally processing the second feature map using multiple deconvolution layers, the second feature map is passed through a convolutional normalization layer and an activation function. Then, the second feature map is deconvolutionally processed (upsampling) to obtain multiple third feature maps of different pixel sizes, and the third feature map with the largest pixel size is determined as the target feature map. The number of third feature maps of different pixel sizes is the same as the number of first feature maps of different pixel sizes, and their pixel sizes are also the same.

[0167] In this embodiment, multiple deconvolution layers are sequentially applied to the second feature map to obtain third feature maps of multiple pixel sizes, and the third feature map with the largest pixel size is determined as the target feature map. This application can recover image details at different pixel sizes during the deconvolution process, which helps to preserve and fuse multi-scale contextual information in the final feature map.

[0168] Based on the above embodiments, the upsampling module also includes: multiple gating modules. Figure 8 A schematic diagram of the upsampling module in another image segmentation network model provided in this application embodiment is shown below. Figure 8 As shown, the upsampling module 103 includes: multiple gate modules 1032, each corresponding to a multiple pixel size.

[0169] Therefore, this application provides a process for determining the third feature map in a semantic segmentation method for optical remote sensing images. Figure 9 This is a flowchart illustrating the process of determining the third feature map in a semantic segmentation method for remote sensing images provided in this application embodiment, as shown below. Figure 9 As shown, based on the above method of sequentially deconvolving the second feature map with multiple deconvolution layers to obtain a third feature map of multiple pixel sizes, and determining the third feature map with the largest pixel size as the target feature map, this method further includes:

[0170] Step 901: Using each gated module, perform feature fusion on the first feature map and the second feature map corresponding to the pixel size of each gated module to obtain the fused feature map corresponding to the pixel size.

[0171] Each gated fusion module corresponds to a different fusion size. The gated fusion module fuses the first and second feature maps of the corresponding pixel size for each gated module. The number of gated modules is the same as the number of convolutional and deconvolutional layers. At this point, the pixel size of the gated fusion module, the pixel size of the first feature map, and the pixel size of the second feature map are all the same. The size of the fused feature map obtained by the gated fusion module is also the pixel size corresponding to the gated fusion module. The first feature map is a low-level feature map obtained through downsampling, and the second feature map is a high-level feature map obtained through upsampling.

[0172] Optionally, for each gated module, feature fusion is performed on the first feature map and the second feature map corresponding to the pixel size of each gated module. That is, feature fusion is performed on the first feature map and the second feature map with the same size as the gated module in multiple first feature maps and multiple second feature maps to obtain a fused feature map of the corresponding pixel size.

[0173] As described above, multiple deconvolution layers are sequentially applied to the second feature map to obtain third feature maps of multiple pixel sizes. The third feature map with the largest pixel size is then determined as the target feature map, including:

[0174] Step 902: Perform deconvolution processing on the fused feature map of the corresponding pixel size using each deconvolution layer to obtain the third feature map of the next pixel size.

[0175] The next pixel size is a pixel size that is larger than the corresponding pixel size.

[0176] Optionally, each deconvolution layer is used to perform deconvolution processing on the fused feature map of the corresponding pixel size. That is, the fused feature map of the corresponding pixel size is upsampled by the deconvolution layer to obtain the third feature map of the next pixel size of the fused feature map.

[0177] In this embodiment, each gating module fuses the first and second feature maps corresponding to the pixel size of each gating module to obtain a fused feature map of the corresponding pixel size. Each deconvolution layer then performs deconvolution processing on the fused feature map of the corresponding pixel size to obtain a third feature map of the next pixel size. This application can obtain a third feature map with a larger pixel size. By superimposing it with the feature maps of lower layers, the upsampling quality can be improved and artifacts reduced. It can simultaneously utilize high-level semantic information and low-level detail information to enrich the feature representation.

[0178] Based on the above embodiments, each gating module includes: a first mixing unit, a second mixing unit, a weighting function unit, and a connection function unit. Figure 10 A schematic diagram of the gating module in the first image segmentation network model provided in this application embodiment is shown below. Figure 10 As shown, each gating module 1032 includes: a first mixing unit 10321, a second mixing unit 10322, a weighting function unit 10323, and a connection function unit 10324. The first mixing unit 10321 is used to calculate the feature map based on weight parameters. The second mixing unit 10322 is also used to calculate the feature map based on weight parameters. The weighting function unit 10323 is used to determine the corresponding weight parameters. The connection function unit 10324 is used to concatenate the weighted feature maps.

[0179] Therefore, this application provides a process for determining the fused feature map in a semantic segmentation method for optical remote sensing images. Figure 11 This is a schematic diagram illustrating the process of determining the fused feature map in a semantic segmentation method for optical remote sensing images provided in an embodiment of this application, as shown below. Figure 11 As shown, in step 1001 above, each gating module is used to perform feature fusion on the first feature map and the second feature map corresponding to the pixel size of each gating module, to obtain a fused feature map corresponding to the pixel size, including:

[0180] Step 1101: Use a weight function unit to calculate the weights of the first feature map and the second feature map of the corresponding pixel size to obtain the first weight parameter.

[0181] Optionally, a weighting function unit is used to calculate the weights of the first and second feature maps of corresponding pixel sizes. Specifically, the first weight parameter is calculated based on the input first and second feature maps using the logic function in formula (6), where the logic function can be the Sigmoid function. The value of the first weight parameter is between 0 and 1. The first weight parameter reflects the importance of the first feature map in the fused feature map.

[0182] W = sigmoid(X1 + X2) (6)

[0183] Where X1 is the first feature map, X2 is the second feature map, and W is the first weight parameter.

[0184] Step 1102: Using the first hybrid unit, perform weighted calculations based on the first weight parameters and the first feature map of the corresponding pixel size to obtain the first weighted feature map.

[0185] Optionally, a first hybrid unit is used to perform weighted calculation based on the first weight parameter and the first feature map of the corresponding pixel size. Specifically, the first weighted feature map is obtained by calculating the first feature map of the corresponding pixel size according to the first weight parameter using formula (7).

[0186] X1′=W·X1(7)

[0187] Where X1′ is the first weighted feature map, W is the first weight parameter, and X1 is the first feature map.

[0188] Step 1103: Using the second hybrid unit, perform weighted calculations based on the second weight parameter corresponding to the first weight parameter and the second feature map of the corresponding pixel size to obtain the second weighted feature map.

[0189] The sum of the first weight parameter and the second weight parameter is 1. The second weight parameter can be determined by subtracting the first weight parameter from 1.

[0190] Optionally, a second hybrid unit is used to perform weighted calculation based on the second weight parameter corresponding to the first weight parameter and the second feature map of the corresponding pixel size. Specifically, the second weight parameter is calculated based on the first weight parameter, and the second weighted feature map is calculated based on the second weight parameter and the second feature map using formula (8).

[0191] X′2=(1-W)·X2(8)

[0192] Where X′2 is the second weighted feature map, W is the first weight parameter, and X2 is the second feature map.

[0193] Step 1104: Use a connection function unit to concatenate the first weighted feature map and the second weighted feature map to obtain a fused feature map of the corresponding pixel size.

[0194] Optionally, a connection function unit is used to concatenate the first weighted feature map and the second weighted feature map. Specifically, the first weighted feature map and the second weighted feature map are concatenated according to formula (9) to obtain a fused feature map of the corresponding pixel size.

[0195] Y = X1′⊕X′2(9)

[0196] Where Y is the fused feature map, X1′ is the first weighted feature map, and X2′ is the second weighted feature map.

[0197] In this embodiment, the first mixing unit, the second mixing unit, the weighting function unit, and the connection function unit in the gating module adaptively adjust the importance of features at different scales in the final fusion process according to the characteristics of the input first feature map and the second feature map, making the fusion process more flexible and the fusion result more accurate.

[0198] Based on the above embodiments, each gating module includes: a first convolutional unit, a second convolutional unit, a first mixing unit, a second mixing unit, a weighting function unit, and an addition unit. Figure 12 A schematic diagram of the gating module in the second image segmentation network model provided in this application embodiment is shown below. Figure 12As shown, each gating module 1032 includes: a first convolutional unit 10325, a second convolutional unit 10326, a first mixing unit 10321, a second mixing unit 10322, a weighting function unit 10323, and an addition unit 10327. The first mixing unit 10321 is used to calculate the feature map based on weight parameters. The second mixing unit 10322 is also used to calculate the feature map based on weight parameters. The weighting function unit 10323 is used to determine the corresponding weight parameters. The first convolutional unit 10325 is used to perform a convolution operation on the input feature map. The second convolutional unit 10326 is used to perform a convolution operation on the input feature map. The addition unit 10327 is used to perform addition processing on the feature map.

[0199] Therefore, this application provides a process for determining the fused feature map in a semantic segmentation method for optical remote sensing images. Figure 13 This is a schematic diagram illustrating the process of determining the fused feature map in the second semantic segmentation method for optical remote sensing images provided in this application embodiment, as shown below. Figure 13 As shown, in step 1001 above, each gating module is used to perform feature fusion on the first feature map and the second feature map corresponding to the pixel size of each gating module, to obtain a fused feature map corresponding to the pixel size, including:

[0200] Step 1301: Use the first convolution unit and the second convolution unit to perform convolution processing on the first feature map and the second feature map of the corresponding pixel size, respectively, to obtain the first mapping feature and the second mapping feature.

[0201] The convolution process is a 1×1 convolution operation.

[0202] Optionally, the first convolution unit and the second convolution unit are used to perform convolution processing on the first feature map and the second feature map of the corresponding pixel size, respectively. Specifically, the first mapping feature is obtained by performing convolution processing on the first feature map of the corresponding pixel size using formula (10), and the second mapping feature is obtained by performing convolution processing on the second feature map of the corresponding pixel size using formula (11).

[0203] m1 = Conv 1×1 (X1) (10)

[0204] m2=Conv 1×1 (X2) (11)

[0205] Where M1 is the first mapping feature, X1 is the first feature map, M2 is the second mapping feature, X2 is the second feature map, and Conv 1×1 This is a 1×1 convolution operation.

[0206] Step 1302: Use a weight function unit to calculate the weights of the first mapping feature and the second mapping feature to obtain the first weight parameter.

[0207] Optionally, a weight function unit is used to calculate the weights of the first and second mapping features. Specifically, the weights are calculated using the logic function in formula (12) based on the first and second mapping features to obtain the first weight parameter. The logic function can be a Sigmoid function. The value of the first weight parameter is between 0 and 1. The first weight parameter reflects the importance of the first feature map in the fused feature map.

[0208] W = Sigmoid(M1 + M2) (12)

[0209] Where W is the first weight parameter, M1 is the first mapping feature, and M2 is the second mapping feature.

[0210] Step 1303: Using the first hybrid unit, perform weighted calculations based on the first weight parameters and the first mapping features to obtain the first weighted feature map.

[0211] Optionally, a first hybrid unit is used to perform weighted calculations based on a first weight parameter and a first mapping feature to determine the weight of the first mapping feature, thereby obtaining a first weighted feature map.

[0212] Step 1304: Using the second hybrid unit, perform weighted calculations based on the second weight parameter corresponding to the first weight parameter and the second mapping feature to obtain the second weighted feature map.

[0213] The sum of the first weight parameter and the second weight parameter is 1. The second weight parameter can be determined by subtracting the first weight parameter from 1.

[0214] Optionally, a second hybrid unit is used to perform weighted calculations based on the second weight parameter and the second mapping feature corresponding to the first weight parameter to determine the second weight parameter, and to determine the second weighted feature map based on the second weight parameter and the second mapping feature.

[0215] Step 1305: Use an addition unit to perform addition processing on the first weighted feature map and the second weighted feature map to obtain a fused feature map of the corresponding pixel size.

[0216] Optionally, an addition unit is used to add the first weighted feature map and the second weighted feature map. Specifically, the first weighted feature map and the second weighted feature map are added using formula (13) to obtain a fused feature map of the corresponding pixel size.

[0217] Y = W·M1 + (1-W)·M2 (13)

[0218] Where W is the first weight parameter, M1 is the first mapping feature, M2 is the second mapping feature, W·M1 is the first weighted feature map, (1-W)·M2 is the second weighted feature map, and Y is the fusion feature map.

[0219] In this embodiment, the gating module includes a first convolutional unit, a second convolutional unit, a first mixing unit, a second mixing unit, a weighting function unit, and an addition unit. Based on the characteristics of the input first and second feature maps, the importance of features at different scales in the final fusion process is adaptively adjusted, making the fusion process more flexible and the fusion result more accurate.

[0220] Based on the above embodiments, each gating module includes: a first convolutional unit, a second convolutional unit, a first mixing unit, a second mixing unit, a weighting function unit, and an addition unit. Figure 14 A schematic diagram of the gating module in the third image segmentation network model provided in this application embodiment is shown below. Figure 14 As shown, each gating module 1032 includes: a first convolutional unit 10324, a second convolutional unit 10325, a first mixing unit 10321, a second mixing unit 10322, a weighting function unit 10323, and a connection function unit 10324. The functions of the specific units are the same as in the above embodiments, and will not be repeated here.

[0221] Therefore, this application provides a process for determining the fused feature map in a semantic segmentation method for optical remote sensing images. Figure 15 A flowchart illustrating the process of determining the fused feature map in the third semantic segmentation method for optical remote sensing images provided in this application embodiment is shown below. Figure 15 As shown, in step 1001 above, each gating module is used to perform feature fusion on the first feature map and the second feature map corresponding to the pixel size of each gating module, to obtain a fused feature map corresponding to the pixel size, including:

[0222] Step 1501: Use the first convolution unit and the second convolution unit to perform convolution processing on the first feature map and the second feature map of the corresponding pixel size, respectively, to obtain the first mapping feature and the second mapping feature.

[0223] Optionally, this step can be obtained by calculating using the above formulas (10) and (11).

[0224] M1 = Conv 1×1 (X1) (10)

[0225] M2 = Conv 1×1 (X2) (11)

[0226] Where M1 is the first mapping feature, X1 is the first feature map, M2 is the second mapping feature, X2 is the second feature map, and Comv 1×1 This is a 1×1 convolution operation.

[0227] Step 1502: Use a weight function unit to calculate the weights of the first mapping feature and the second mapping feature to obtain the first weight parameter.

[0228] Alternatively, this step can be calculated using the formula (12) above.

[0229] W = Sigmoid(M1 + M2) (12)

[0230] Where W is the first weight parameter, M1 is the first mapping feature, and M2 is the second mapping feature.

[0231] Step 1503: Using the first hybrid unit, perform weighted calculations based on the first weight parameters and the first feature map of the corresponding pixel size to obtain the first weighted feature map.

[0232] Optionally, a first hybrid unit is used to perform weighted calculations based on the first weight parameter and the first feature map of the corresponding pixel size using the above formula (7) to determine the weight of the first feature map of the corresponding pixel size, thereby obtaining the first weighted feature map.

[0233] Step 1504: Using the second hybrid unit, perform weighted calculations based on the second weight parameter corresponding to the first weight parameter and the second feature map to obtain the second weighted feature map.

[0234] The sum of the first weight parameter and the second weight parameter is 1. The second weight parameter can be determined by subtracting the first weight parameter from 1.

[0235] Optionally, a second hybrid unit is used to perform weighted calculations based on the second weight parameters and the second feature map of the corresponding pixel size using the above formula (8) to determine the weight of the second feature map of the corresponding pixel size, thereby obtaining the second weighted feature map.

[0236] Step 1505: Use a connection function unit to concatenate the first weighted feature map and the second weighted feature map to obtain a fused feature map of the corresponding pixel size.

[0237] Optionally, the first weighted feature map and the second weighted feature map are spliced ​​together using the above formula (9) by using a connection function unit to obtain a fused feature map of the corresponding pixel size.

[0238] In this embodiment, the gating module comprises a first convolutional unit, a second convolutional unit, a first mixing unit, a second mixing unit, a weighting function unit, and a connection function unit. Based on the characteristics of the input first and second feature maps, the importance of features at different scales in the final fusion process is adaptively adjusted, making the fusion process more flexible and the fusion result more accurate.

[0239] Based on the above embodiments, the feature segmentation module includes a feature mapping unit and a generation unit. Figure 16 This is a schematic diagram of the feature segmentation module in an image segmentation network model provided in an embodiment of this application, as shown below. Figure 16 As shown, the feature segmentation module 200 includes a feature mapping unit 2001 and a generation unit 2002. The feature mapping unit 2001 is used to extract higher-level semantic information and map it to category labels, while the generation unit 2002 is used to generate the corresponding semantic segmentation map.

[0240] Therefore, this application provides a process for generating semantic segmentation maps in a semantic segmentation method for optical remote sensing images. Figure 17 This is a schematic diagram illustrating the process of generating a semantic segmentation map in a semantic segmentation method for optical remote sensing images provided in an embodiment of this application, as shown below. Figure 17 As shown, in step 207 above, the feature segmentation module performs feature segmentation on the target feature map to obtain a semantic segmentation map of the optical remote sensing image, including:

[0241] Step 1701: The feature mapping unit performs a convolution operation on the target feature map according to the preset number of categories to obtain a new feature map. The features of each pixel in the new feature map have corresponding category labels and the probability of the corresponding category.

[0242] The preset number of categories is determined based on actual conditions, and this embodiment does not impose any restrictions on it. The size of the target feature map is H×W×C, where H×W are the height and width, respectively, and C is the number of channels, which is usually equal to the number of target categories. The number of category labels for the features of each pixel in the new feature map is consistent with the preset number of categories.

[0243] Optionally, a new feature map can be obtained by using the feature mapping unit to perform a convolution operation on the target feature map according to the preset number of categories through formula (14).

[0244] F′=Conv(F,K) (14)

[0245] Where F is the target feature map, F′ is the new feature map, K is the convolution kernel, and Conv is the convolution operation.

[0246] Optionally, the probability of each pixel's feature for different category labels can be calculated using an activation function according to formula (15). The activation function can be the softmax activation function.

[0247]

[0248] Where, P(c|F′) i F′ represents the probability that the feature of a pixel at position i in the target feature map belongs to the analogy label c. i,c is the value of the pixel at position i in the target feature map corresponding to category c, and j is the number of pixels.

[0249] Step 1702: Generate a semantic segmentation map using the generation unit based on the new feature map.

[0250] Optionally, the label category with the highest probability for each pixel is determined by formula (16) as the predicted category label for each pixel. A corresponding semantic segmentation map is generated based on the predicted category label for each pixel. In the semantic segmentation map, each pixel is assigned a category label to represent a different region, and the value of each pixel is the index of the corresponding category label. The size of the semantic segmentation map is the same as the target size of the optical remote sensing image of the target pixel size to be processed.

[0251]

[0252] in, Let P(c|F′) be the predicted class label of the pixel at position i in the target feature map. i argmax represents the probability that the feature of a pixel at position i in the target feature map belongs to the analog label c, and argmax is the maximum probability calculation function.

[0253] In this embodiment, a semantic segmentation map is generated by a feature mapping unit and a generation unit. The feature mapping unit performs a convolution operation on the target feature map according to a preset number of categories to obtain a new feature map. Each pixel in the new feature map has a corresponding category label and the probability of the corresponding category. The category label of the pixel is determined according to the category label with the highest probability, which ensures the accuracy of the generated semantic segmentation map and improves the precision of the generated semantic segmentation map.

[0254] Based on the above embodiments, this application also provides a process for determining the target pixel size of an optical remote sensing image in a semantic segmentation method for optical remote sensing images. Before step 201 above, where a downsampling module is used to downsample the optical remote sensing image of the target pixel size to be processed to obtain the first feature map, the method further includes:

[0255] The original remote sensing image is cropped to obtain an optical remote sensing image of the target pixel size.

[0256] The cropping process can use a sliding window of 512×512 pixels to crop the entire image area, ensuring that no details of the optical remote sensing image are missed.

[0257] Optionally, when cropping the original remote sensing image, the sliding window is moved with a fixed step size s to obtain optical remote sensing images of multiple target pixel sizes. The value of s can be adjusted according to the characteristics of the specific dataset; the step size is usually smaller than the size of the sliding window to ensure appropriate overlap between adjacent cropping regions.

[0258] For example, if the original remote sensing image size is H×W and the easing window size is K×K, then the number of cropped images N can be calculated using formula (17).

[0259]

[0260] Where H and W are the dimensions of the original remote sensing image, K is the size of the sliding window, and s is the step size of the sliding window.

[0261] Optionally, edge areas that cannot be divided evenly during the cropping process can be filled or scaled to ensure consistent image size.

[0262] For example, the insufficient parts are filled or scaled by using bilinear interpolation method through formula (17) to ensure that all image segments have the same size.

[0263] Inew(x,y)=∑ x,y I(x,y)·max(0,1-|x′-x|)·max(0,1-|y′-y|) (17)

[0264] Among them, I new (x,y) represents the pixel value of the scaled optical remote sensing image, I(x,y) is the pixel value of the original optical remote sensing image, and x,y and x′,y′ are the coordinates in the original image and the scaled image, respectively.

[0265] Optionally, during the training of the image segmentation network model, after cropping the preset dataset, a total of 3,300 optical remote sensing images of size 512×512 are obtained. These images are then divided into a training set and a test set according to a certain ratio. The training set contains 2,805 images of 512×512 pixels, used for model training and optimization. The test set contains 495 images of 512×512 pixels, used for model performance evaluation and testing, ensuring that the model has good generalization ability on unseen data. The preset dataset can be the Vaihingen dataset, which includes 33 high-resolution images in the near-infrared, red, and green bands, with a ground sampling distance of 9 cm and different land cover types, such as water surfaces, buildings, vegetation, and vehicles.

[0266] In the embodiments of the application, the original remote sensing image is cropped to obtain an optical remote sensing image with the target pixel size. This ensures that the pixel size of the input image segmentation network model is consistent, and the original remote sensing image is processed in batches, reducing the amount of computation, improving the image processing speed, and saving computing resources.

[0267] Based on the same inventive concept, this application also provides a semantic segmentation device for optical remote sensing images corresponding to the semantic segmentation method for optical remote sensing images. Since the principle of the device in this application is similar to the semantic segmentation method for optical remote sensing images described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0268] Figure 18 This is a schematic diagram of a semantic segmentation device for optical remote sensing images provided in an embodiment of this application. The device is applied to an image segmentation network model, which includes a feature extraction module and a feature segmentation module. The feature extraction module includes a downsampling module, a linear attention network, and an upsampling module. The linear attention network includes a first normalization layer, a linear attention module, a second normalization layer, and a multilayer perceptron. The device includes:

[0269] The first sampling module 1801 is used to perform downsampling processing on the optical remote sensing image of the target pixel size to be processed by the downsampling module to obtain the first feature map;

[0270] The first processing module 1802 is used to normalize the first feature map using a first normalization layer to obtain the first normalized feature.

[0271] The second processing module 1803 is used to perform linear attention processing on the first normalized features using a linear attention module to obtain target global features with contextual information.

[0272] The third processing module 1804 is used to normalize the target global features using the second normalization layer to obtain the second normalized features;

[0273] The mapping module 1805 is used to perform feature mapping on the second normalized features using a multilayer perceptron to obtain a second feature map with the same pixel size as the first feature map.

[0274] The second sampling module 1806 is used to perform upsampling processing on the second feature map using the upsampling module to obtain a target feature map of the target pixel size;

[0275] The segmentation module 1807 is used to perform feature segmentation on the target feature map using the feature segmentation module to obtain the semantic segmentation map of the optical remote sensing image.

[0276] In one possible implementation, the linear attention module includes: a linear processing layer, a first aggregation layer, a second aggregation layer, a third aggregation layer, a fourth aggregation layer, a first attention calculation layer, and a second attention calculation layer. The second processing module 1803 is specifically used to: perform linear processing on the first normalized feature using the linear processing layer to obtain a first linear query vector, a first linear key vector, and a first linear value vector corresponding to the first linear key vector.

[0277] The first aggregation layer is used to aggregate and calculate the first global feature and the first linear key vector corresponding to the first linear query vector to obtain the first aggregated feature with contextual information.

[0278] A first attention calculation layer is used to perform attention calculation on the first aggregated feature to obtain the first attention matrix;

[0279] A second aggregation layer is used to aggregate and calculate the first attention matrix and the first linear value vector to obtain the second global feature.

[0280] The activation function features of the first linear query vector and the first global features are aggregated and calculated using the third aggregation layer to obtain the second aggregated features;

[0281] A second attention calculation layer is used to perform attention calculation on the second aggregated features to obtain the second attention matrix;

[0282] The fourth aggregation layer is used to aggregate and calculate the second attention matrix and the first global feature to obtain the target global feature.

[0283] In one possible implementation, the downsampling module includes: multiple convolutional layers, each corresponding to a multiple pixel size; and a first sampling module 1801, specifically used to: sequentially perform convolution processing on the optical remote sensing image using multiple convolutional layers to obtain a first feature map of multiple pixel sizes.

[0284] In one possible implementation, the first processing module 1802 is specifically used to: normalize the first feature map with the smallest pixel size using a first normalization layer to obtain the first normalized feature.

[0285] In one possible implementation, the upsampling module includes: multiple deconvolution layers; the multiple deconvolution layers correspond to multiple pixel sizes respectively; the second sampling module 1806 is specifically used to: sequentially perform deconvolution processing on the second feature map using multiple deconvolution layers to obtain a third feature map of multiple pixel sizes, and determine the third feature map with the largest pixel size as the target feature map.

[0286] In one possible implementation, the upsampling module further includes: multiple gate modules, each corresponding to a multiple pixel size; the second sampling module 1806 is further configured to: use each gate module to perform feature fusion on the first feature map and the second feature map corresponding to the pixel size of each gate module to obtain a fused feature map corresponding to the pixel size.

[0287] In one possible implementation, the second sampling module 1806 is specifically used to: perform deconvolution processing on the fused feature map of the corresponding pixel size using each deconvolution layer to obtain a third feature map of the next pixel size.

[0288] In one possible implementation, each gate module includes: a first mixing unit, a second mixing unit, a weighting function unit, and a connection function unit; the second sampling module 1806 is specifically used to: use the weighting function unit to calculate the weights of the first feature map and the second feature map of the corresponding pixel size to obtain the first weight parameter;

[0289] The first hybrid unit is used to perform weighted calculation based on the first weight parameters and the first feature map of the corresponding pixel size to obtain the first weighted feature map;

[0290] A second hybrid unit is used to perform weighted calculations based on the second weight parameter corresponding to the first weight parameter and the second feature map of the corresponding pixel size to obtain a second weighted feature map; wherein the sum of the first weight parameter and the second weight parameter is 1;

[0291] The first weighted feature map and the second weighted feature map are concatenated using a connection function unit to obtain a fused feature map of the corresponding pixel size.

[0292] In one possible implementation, each gating module includes: a first convolutional unit, a second convolutional unit, a first mixing unit, a second mixing unit, a weighting function unit, and an addition unit; the second sampling module 1806 is specifically used to: use each gating module to perform feature fusion on the first feature map and the second feature map of the corresponding pixel size of each gating module to obtain a fused feature map of the corresponding pixel size, including:

[0293] The first convolutional unit and the second convolutional unit are used to perform convolution processing on the first feature map and the second feature map of the corresponding pixel size, respectively, to obtain the first mapping feature and the second mapping feature;

[0294] The first weight parameter is obtained by using a weight function unit to calculate the weights of the first and second mapping features.

[0295] The first hybrid unit is used to perform weighted calculation based on the first weight parameter and the first mapping feature to obtain the first weighted feature map;

[0296] A second hybrid unit is used to perform weighted calculations based on the second weight parameter corresponding to the first weight parameter and the second mapping feature to obtain a second weighted feature map; wherein the sum of the first weight parameter and the second weight parameter is 1;

[0297] An additive unit is used to add the first weighted feature map and the second weighted feature map to obtain a fused feature map of the corresponding pixel size.

[0298] In one possible implementation, each gate module includes: a first convolutional unit, a second convolutional unit, a first mixing unit, a second mixing unit, a weighting function unit, and a connection function unit; the second sampling module 1806 is specifically used to: perform convolution processing on the first feature map and the second feature map of the corresponding pixel size using the first convolutional unit and the second convolutional unit respectively, to obtain the first mapping feature and the second mapping feature;

[0299] The first weight parameter is obtained by using a weight function unit to calculate the weights of the first and second mapping features.

[0300] The first hybrid unit is used to perform weighted calculation based on the first weight parameters and the first feature map of the corresponding pixel size to obtain the first weighted feature map;

[0301] A first hybrid unit is used to perform weighted calculations based on the second weight parameter corresponding to the first weight parameter and the second feature map of the corresponding pixel size to obtain a second weighted feature map; wherein, the sum of the first weight parameter and the second weight parameter is 1;

[0302] The first weighted feature map and the second weighted feature map are concatenated using a connection function unit to obtain a fused feature map of the corresponding pixel size.

[0303] In one possible implementation, the feature segmentation module includes a feature mapping unit and a generation unit; the segmentation module 1807 is specifically used to: use the feature mapping unit to perform a convolution operation on the target feature map according to a preset number of categories to obtain a new feature map, wherein the features of each pixel in the new feature map have corresponding category labels and the probability of the corresponding category;

[0304] The generation unit generates a semantic segmentation map based on the new feature map.

[0305] In one possible implementation, the first sampling module 1801 is further configured to: crop the original remote sensing image to obtain an optical remote sensing image of the target pixel size.

[0306] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0307] This application also provides a computer device. Figure 19 A schematic diagram of a computer device provided in this application embodiment includes: a processor 1901 and a memory 1902. The memory 1902 stores machine-readable instructions executable by the processor 1901. When the computer device is running, the processor 1901 communicates with the memory 1902. When the machine-readable instructions are executed by the processor 1901, the steps of the semantic segmentation method for optical remote sensing images described above are performed.

[0308] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the semantic segmentation method for optical remote sensing images described above.

[0309] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0310] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0311] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A semantic segmentation method for optical remote sensing images, characterized in that, An image segmentation network model is applied, comprising: a feature extraction module and a feature segmentation module. The feature extraction module includes: a downsampling module, a linear attention network, and an upsampling module. The linear attention network includes: a first normalization layer, a linear attention module, a second normalization layer, and a multilayer perceptron. The linear attention module includes: a linear processing layer, a first aggregation layer, a second aggregation layer, a third aggregation layer, a fourth aggregation layer, a first attention calculation layer, and a second attention calculation layer. The method includes: The downsampling module is used to downsample the optical remote sensing image of the target pixel size to be processed, and a first feature map is obtained. The first feature map is normalized using the first normalization layer to obtain the first normalized feature. The first normalized feature is linearly processed by the linear processing layer to obtain a first linear query vector, a first linear key vector, and a first linear value vector corresponding to the first linear key vector. The first aggregation layer is used to aggregate the first global feature corresponding to the first linear query vector and the first linear key vector to obtain a first aggregated feature with contextual information. The first attention calculation layer is used to perform attention calculation on the first aggregated features to obtain the first attention matrix; The second aggregation layer is used to aggregate and calculate the first attention matrix and the first linear value vector to obtain the second global feature. The activation function features of the first linear query vector and the first global features are aggregated and calculated using the third aggregation layer to obtain the second aggregated features; The second attention calculation layer is used to perform attention calculation on the second aggregated features to obtain the second attention matrix; The fourth aggregation layer is used to aggregate and calculate the second attention matrix and the first global feature to obtain the target global feature; The target global features are normalized using the second normalization layer to obtain the second normalized features; The multilayer perceptron is used to perform feature mapping on the second normalized feature to obtain a second feature map with the same pixel size as the first feature map; The upsampling module is used to upsample the second feature map to obtain a target feature map of the target pixel size; The target feature map is segmented using the feature segmentation module to obtain the semantic segmentation map of the optical remote sensing image.

2. The semantic segmentation method for optical remote sensing images according to claim 1, characterized in that, The downsampling module includes: multiple convolutional layers, each of which corresponds to a multiple pixel size; The downsampling module processes the optical remote sensing image of the target pixel size to be processed to obtain a first feature map, including: The optical remote sensing image is sequentially convolved using the multiple convolutional layers to obtain a first feature map of the multiple pixel sizes; The step of normalizing the first feature map using the first normalization layer to obtain the first normalized feature includes: The first normalization layer is used to normalize the first feature map with the smallest pixel size to obtain the first normalized feature.

3. The semantic segmentation method for optical remote sensing images according to claim 2, characterized in that, The upsampling module includes: multiple deconvolutional layers; the multiple deconvolutional layers respectively correspond to the multiple pixel sizes; the step of using the upsampling module to upsample the second feature map to obtain a target feature map of the target pixel size includes: The second feature map is deconvolved sequentially using the multiple deconvolution layers to obtain a third feature map of multiple pixel sizes, and the third feature map with the largest pixel size is determined as the target feature map.

4. The semantic segmentation method for optical remote sensing images according to claim 3, characterized in that, The upsampling module further includes: multiple gate modules, each corresponding to one of the multiple pixel sizes; the method further includes: For each gated module, feature fusion is performed on the first feature map and the second feature map corresponding to the pixel size of each gated module to obtain the fused feature map corresponding to the pixel size; The step of sequentially deconvolutionally processing the second feature map using the multiple deconvolution layers to obtain a third feature map of multiple pixel sizes, and determining the third feature map with the largest pixel size as the target feature map, includes: Each deconvolutional layer is used to deconvolve the fused feature map of the corresponding pixel size to obtain the third feature map of the next pixel size.

5. The semantic segmentation method for optical remote sensing images according to claim 4, characterized in that, Each gating module includes: a first mixing unit, a second mixing unit, a weighting function unit, and a connection function unit; the step of using each gating module to perform feature fusion on the first feature map and the second feature map of the corresponding pixel size to obtain the fused feature map of the corresponding pixel size includes: The weight function unit is used to calculate the weights of the first feature map and the second feature map of the corresponding pixel size to obtain the first weight parameter; Using the first mixing unit, a weighted calculation is performed based on the first weight parameter and the first feature map of the corresponding pixel size to obtain a first weighted feature map; The second mixing unit is used to perform a weighted calculation based on the second weight parameter corresponding to the first weight parameter and the second feature map of the corresponding pixel size to obtain a second weighted feature map; wherein the sum of the first weight parameter and the second weight parameter is 1; The first weighted feature map and the second weighted feature map are concatenated using the connection function unit to obtain a fused feature map of the corresponding pixel size.

6. The semantic segmentation method for optical remote sensing images according to claim 4, characterized in that, Each gate module includes: a first convolutional unit, a second convolutional unit, a first mixing unit, a second mixing unit, a weighting function unit, and an addition unit; The step of employing each gating module and fusing the first and second feature maps corresponding to the pixel size of each gating module to obtain the fused feature map of the corresponding pixel size includes: The first convolutional unit and the second convolutional unit are used to perform convolution processing on the first feature map and the second feature map of the corresponding pixel size, respectively, to obtain the first mapping feature and the second mapping feature; The weight function unit is used to calculate the weights of the first mapping feature and the second mapping feature to obtain the first weight parameter; Using the first hybrid unit, a weighted calculation is performed based on the first weight parameter and the first mapping feature to obtain a first weighted feature map; Using the second hybrid unit, a weighted calculation is performed based on the second weight parameter corresponding to the first weight parameter and the second mapping feature to obtain a second weighted feature map; wherein, the sum of the first weight parameter and the second weight parameter is 1; The addition unit is used to perform addition processing on the first weighted feature map and the second weighted feature map to obtain the fused feature map of the corresponding pixel size.

7. The semantic segmentation method for optical remote sensing images according to claim 4, characterized in that, Each gate module includes: a first convolutional unit, a second convolutional unit, a first hybrid unit, a second hybrid unit, a weighting function unit, and a connection function unit; The step of employing each gating module and fusing the first and second feature maps corresponding to the pixel size of each gating module to obtain the fused feature map of the corresponding pixel size includes: The first convolutional unit and the second convolutional unit are used to perform convolution processing on the first feature map and the second feature map of the corresponding pixel size, respectively, to obtain the first mapping feature and the second mapping feature; The weight function unit is used to calculate the weights of the first mapping feature and the second mapping feature to obtain the first weight parameter; Using the first mixing unit, a weighted calculation is performed based on the first weight parameter and the first feature map of the corresponding pixel size to obtain a first weighted feature map; Using the first mixing unit, a second weighted feature map is obtained by weighting the second weight parameter corresponding to the first weight parameter and the second feature map of the corresponding pixel size; wherein, the sum of the first weight parameter and the second weight parameter is 1; The first weighted feature map and the second weighted feature map are concatenated using the connection function unit to obtain a fused feature map of the corresponding pixel size.

8. The semantic segmentation method for optical remote sensing images according to claim 1, characterized in that, The feature segmentation module includes: a feature mapping unit and a generation unit; The step of using the feature segmentation module to perform feature segmentation on the target feature map to obtain the semantic segmentation map of the optical remote sensing image includes: The feature mapping unit performs a convolution operation on the target feature map according to a preset number of categories to obtain a new feature map. Each pixel in the new feature map has a corresponding category label and the probability of the corresponding category. The generation unit generates the semantic segmentation map based on the new feature map.

9. The semantic segmentation method for optical remote sensing images according to claim 1, characterized in that, Before the downsampling module performs downsampling processing on the optical remote sensing image of the target pixel size to be processed to obtain the first feature map, the method further includes: The original remote sensing image is cropped to obtain an optical remote sensing image of the target pixel size.

Images