Sub-pixel mapping method based on local-overall abundance attention mechanism

Abstract

The invention discloses a sub-pixel mapping method based on a local-overall abundance attention mechanism, and the method comprises the steps: solving a mixed pixel problem through employing an end-to-end network assisted by abundance semantic information, and achieving the multi-scale information supervision; introducing an abundance attention mechanism, estimating a context relationship by using a generated abundance map, and fusing the context relationship into a generation process of a sub-pixel classification map; a local-overall spatial information fusion strategy is used, local information fusion is firstly carried out to ensure a local positioning effect, and then global information fusion is carried out to ensure the consistency of global context. Compared with the existing method, the method provided by the invention is good in interpretability, high in drawing precision and wide in application range.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and relates to a hyperspectral image sub-pixel mapping technology. Specifically, the present invention relates to a sub-pixel mapping method based on a local-global abundance attention mechanism. Background Art

[0002] In recent decades, hyperspectral images have been widely used in land resource utilization, ecological environment protection and urban planning due to their rich spatial and spectral information. The spatial resolution of currently used hyperspectral images is limited, which leads to the possibility of multiple objects in one pixel, which is called the mixed pixel problem. Sub-pixel mapping is a technology that divides a pixel into multiple sub-pixels and assigns a ground object label to each sub-pixel.

[0003] In the past decade, although end-to-end convolutional neural networks based on deep learning have achieved good sub-pixel mapping results, their ability to extract spatial context is insufficient. On the one hand, in the process of feature extraction, the convolution structure pays more attention to low-frequency information, causing the loss of spatial high-frequency texture information; on the other hand, remote sensing images have end-member variability, and the spectral information of the same type of objects can be different, making it difficult for convolutional networks to maintain intra-class consistency. Using prior information about objects to assist the network in extracting spatial context information is a feasible method. For example, SIMNet integrates the abundance image into the end-to-end network through the "Scale and Shift" matrix multiplication and addition structure, proving that prior information about objects can guide the network to extract spatial detail information. HiSMNet uses low-frequency abundance maps as prior knowledge to force the network to pay attention to high-frequency details. SCNet adds an encoding auxiliary branch in the feature extraction stage, and integrates the abundance map as prior knowledge into the main branch through splicing.

[0004] There are still two problems to be solved for the above-mentioned methods of assisting localization with abundance semantic information. First, in practical applications, it is not always easy to obtain abundance maps. We need to incorporate the generation process of abundance maps into the network structure to form a truly end-to-end network. Second, these methods integrate the spatial prior information of ground objects into the backbone network on a global scale. Directly introducing global spatial prior information does not emphasize the difference in correlation between end elements at different distances, which may cause the network to ignore local texture detail information. We need a more effective information fusion method. Summary of the invention

[0005] In view of the existing technical defects, the present invention proposes a sub-pixel mapping method based on the local-global abundance attention mechanism. The sub-pixel classification map obtained by the present invention has higher positioning accuracy, richer local texture structure and better global consistency.

[0006] This paper combines two sub-pixel localization methods and proposes a multi-task network that can achieve unmixing and localization in parallel. The abundance map obtained by unmixing is used to guide the localization process through the "local-global" abundance attention mechanism. This network takes into account the physical connection between the low-resolution abundance map and the high-resolution feature distribution map, and uses a common feature extraction unit to mine high-dimensional feature maps from the original remote sensing image. Then, the auxiliary branch reduces the dimension of the features to obtain the abundance map; the abundance map is divided into different regions, and the local context is fused into the high-dimensional feature map of the main branch; next, global information fusion is performed to ensure the consistency of the global context; finally, the final sub-pixel localization result is obtained through the upsampling module and the classification module.

[0007] The technical solution of the present invention comprises the following steps:

[0008] On the one hand, the present invention provides a sub-pixel mapping method based on a local-global abundance attention mechanism, comprising the following steps:

[0009] Design a feature extraction module to process the original low-resolution spectrum map and extract a high-dimensional feature map with the same resolution as the original low-resolution spectrum map;

[0010] Design auxiliary branches to transform the high-dimensional feature map into an abundance map;

[0011] Design a local-global abundance attention module, which takes high-dimensional feature maps and abundance maps as inputs, first locally fuses abundance information using matrix multiplication to aggregate local features; then globally performs weighted operations on regions to obtain feature maps that are globally consistent in space;

[0012] Design upsampling modules and classification modules, and connect the feature extraction module, local-global abundance attention module, upsampling module and classification module in series to form a backbone sub-pixel mapping network;

[0013] A loss function is designed to train the backbone network and auxiliary branches, and finally an end-to-end mapping of the low-resolution spectral image to the high-resolution sub-pixel localization image is formed.

[0014] Furthermore, the feature extraction module includes an encoder and a decoder, wherein the encoder completes the extraction of low-level semantic information to high-level semantic information of the original low-resolution spectrogram, and the spatial resolution gradually decreases; the decoder gradually increases the spatial resolution while ensuring the number of channels, completes multi-scale feature fusion through jump connections, and outputs a high-dimensional feature map with the same spatial resolution as the input.

[0015] Furthermore, the feature extraction module design method is as follows:

[0016] An encoder E is established. The encoder E uses the structure of the ResNet-50 backbone network and makes changes to the original ResNet-50: a 3×3 convolution kernel is used instead of a 7×7 convolution kernel to improve the problem of spatial information loss, and the downsampling rate is changed from 32 to 8; downsampling is performed three times only in the backbone layer, the maximum pooling stage, and the second layer in the initial input feature extraction stage, and the downsampling rate is 2 each time, and the third and fourth layers are not downsampled; the fully connected layer of ResNet-50 is cancelled, and the output of the fourth layer of ResNet-50 is used as the encoder output;

[0017] A decoder D is established, and the decoder D uses bilinear interpolation to complete the resolution transformation; in the upsampling process, the output result of the encoder E is upsampled once with an upsampling rate of 2, and then the sampling result and the maximum pooling result of the encoder are spliced ​​after 1×1 convolution kernel channel, batch regularization and ReLU activation function operations; then the operation result is subjected to 3×3 convolution, batch regularization and ReLU activation function operations, and an upsampling with an upsampling rate of 2; then the operation result and the maximum pooling result of the encoder are spliced ​​with the result of 1×1 convolution kernel channel, batch regularization and ReLU activation function operations; the operation result is subjected to 3×3 convolution, batch regularization and ReLU activation function operations again, and an upsampling with an upsampling rate of 2.

[0018] Furthermore, the auxiliary branches are designed as follows:

[0019] Design a 1×1 convolution kernel to transform the high-dimensional feature map into an abundance map.

[0020] Furthermore, the method of designing the local-global abundance attention module is as follows:

[0021] The abundance map is divided into P×P non-overlapping local blocks, and a local abundance attention map is obtained after linear mapping;

[0022] The high-dimensional feature map is divided into P×P non-overlapping local blocks, and a local high-dimensional feature map is obtained after linear mapping;

[0023] The local abundance attention map and the local high-dimensional feature map are used to fuse the spatial context into the features using matrix multiplication to obtain the first fused features of all local blocks;

[0024] Considering the spatial weights of different local attention maps, the local abundance attention map is processed through the mapping matrix according to the weight of each local block, and the local abundance map is obtained after weighting;

[0025] The first fusion feature and the local abundance map are fused to obtain a high-dimensional feature vector fused with local context information and global context information;

[0026] Restore the high-dimensional feature vector to a three-dimensional expression form according to the inverse operation of segmentation to obtain a first output;

[0027] The first output and the input high-dimensional feature map are added to obtain the second output, which is a feature map that integrates the semantic information of the abundance map.

[0028] Furthermore, the abundance map is divided into P×P non-overlapping local blocks, and a local abundance attention map is obtained after linear mapping, which also includes

[0029] Linearly map the local blocks of each abundance map to obtain a linear local abundance map;

[0030] Transpose the linear local abundance map to obtain a transposed linear local abundance map;

[0031] The local abundance attention map is obtained by matrix multiplying the linear local abundance map and the transposed linear local abundance map.

[0032] Furthermore, the high-dimensional feature map and the feature map that incorporates the semantic information of the abundance map are connected together as the input of the upsampling module and the classification module to obtain the sub-pixel mapping result.

[0033] X=Conν 1×1 (Up(Conν 3×3 (Cat[F on ,F o ])))

[0034] Among them, F in is a high-dimensional feature map, F o is a feature map that integrates the semantic information of the abundance map, Cat[] is a connection function for constructing a multidimensional array, Conv 3×3 is the 3x3 convolution operation for information fusion, Up(·) is the upsampling operation of the upsampling module; Conv 1×1 It is the 1×1 convolution operation of the classification module.

[0035] On the other hand, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned sub-pixel mapping method when executing the program.

[0036] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned sub-pixel mapping method when executed by a processor.

[0037] On the other hand, the present invention provides a computer program product, comprising a computer program, which implements the above-mentioned sub-pixel mapping method when executed by a processor.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] (1) This paper proposes a new end-to-end sub-pixel mapping network, which uses auxiliary branches to mine semantic information and integrates it into the features extracted by the backbone network. The semantic information is no longer provided externally, but learned by the network, which enables errors to be corrected through back propagation;

[0040] (2) This paper proposes a new abundance attention mechanism as an information fusion method, which uses the abundance map to extract the correlation between pixels and integrates the correlation into the high-dimensional features of the backbone network;

[0041] (3) The present invention effectively enhances the inter-class differences of features in different contexts and the intra-class consistency in the global context through local-global information fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a sub-pixel mapping method based on a local-global abundance attention mechanism of the present invention.

[0043] Figure 2 It is the overall framework of the sub-pixel mapping model of the present invention.

[0044] Figure 3 This is a feature extraction network structure diagram of the present invention.

[0045] Figure 4 Schematic diagram of the structure of the local-global abundance attention module in an embodiment of the present invention.

[0046] Figure 5 It is the qualitative result of the comparative experiment of the present invention. DETAILED DESCRIPTION

[0047] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0048] The method of the present invention uses the hyperspectral WHU-OHS dataset as the data for comparative experiments. This dataset consists of 7793 hyperspectral images of size 512×512, with a spatial resolution of 10m and a total of 24 types of ground feature labels. The images are collected from 40 different regions in China, and each image has a pixel-level label. The dataset covers 32 bands from visible light to near infrared, with an average spectral resolution of 15nm.

[0049] like Figure 1The embodiment of the present invention provides a sub-pixel mapping method based on the local-global abundance attention mechanism. The obtained sub-pixel mapping model based on the local-global abundance attention mechanism is as follows: Figure 2 As shown, the following steps are included:

[0050] Step 1: Design a feature extraction module to process the original low-resolution spectrum and extract a high-dimensional feature map F with the same resolution as the original low-resolution spectrum. in ;

[0051] Step 2: Design an auxiliary branch to transform the high-dimensional feature map into an abundance map A;

[0052] Step 3: Design a local-global abundance attention module to in and the abundance map A as input, firstly use matrix multiplication to fuse the abundance information locally, so as to aggregate the local features; then perform weighted operations on the regions globally to obtain the feature map F0 with global consistency in space;

[0053] Step 4: Design an upsampling module and a classification module, and connect the feature extraction module, the local-global abundance attention module, the upsampling module, and the classification module in series to form a backbone sub-pixel mapping network;

[0054] Step 5: Design a loss function to train the backbone network and auxiliary branches, and finally form an end-to-end mapping from the low-resolution spectral image to the high-resolution sub-pixel positioning image.

[0055] In step 1, the original image I (resolution size is H×W) is extracted by the feature extraction module to obtain a high-dimensional feature map F in (resolution size is H×W), the feature extraction module can be replaced by a suitable feature extraction network in the prior art. The present invention provides an exemplary description, wherein the feature extraction module includes an encoder and a decoder, such as Figure 3 As shown, the specific implementation method is as follows;

[0056] Step 1.1, establish encoder E, which uses the structure of ResNet-50 backbone network and makes changes to the original ResNet-50: use 3×3 convolution kernel instead of 7×7 convolution kernel to improve the problem of spatial information loss; change the downsampling rate from 32 to 8, and perform downsampling three times only in the initial input feature extraction stage (backbone layer), the subsequent maximum pooling stage and the second layer, each downsampling rate is 2, and the third and fourth layers are not downsampled; cancel the fully connected layer of ResNet-50, and use the output of the fourth layer of ResNet-50 as the encoder output; in Figure 3 In the above figure, the structure of each layer of the encoder is consistent with that of the original ResNet-50, and will not be repeated here;

[0057] Step 1.2, establish a decoder D, and the decoder D uses bilinear interpolation to complete the resolution transformation. In the upsampling process, the output result of the encoder E is upsampled once with an upsampling rate of 2, and then the sampling result and the encoder maximum pooling result are spliced ​​after 1×1 convolution kernel channel, batch normalization and ReLU activation function operations; the above operation results are subjected to 3×3 convolution, batch normalization and ReLU activation function operations, and upsampled once with an upsampling rate of 2; the above operation results are spliced ​​with the results of 1×1 convolution kernel channel, batch normalization and ReLU activation function operations of the encoder maximum pooling result; the above operation results are subjected to 3×3 convolution, batch normalization and ReLU activation function operations again, and upsampled once with an upsampling rate of 2. Through the above operations, the decoder continuously increases the spatial resolution while ensuring the feature dimension, realizing multi-scale feature fusion.

[0058] like Figure 2 As shown, in step 2, the specific implementation method of step 2 is as follows;

[0059] Design a 1×1 convolution kernel to transform the high-dimensional feature map into an abundance map A∈R C×H×W , R represents the abundance map set, C is the number of channels, and H×W is the resolution size.

[0060] In step 3, if Figure 2 and Figure 4 As shown in Figure 2, the method for designing the local-global abundance attention module is as follows:

[0061] Step 3.1: Divide the abundance map A into P×P non-overlapping local blocks, where P represents the segmentation ratio. 2 local blocks, the i-th local block is represented by A (i) , each local block size is h×w, where After linear mapping, we get the local abundance attention map M A , the specific process is as follows:

[0062] Step 3.1.1: local block A of the abundance map (i) Perform linear mapping to obtain a linear local abundance map The i-th linear local abundance map is denoted as

[0063] Step 3.1.2: After transposing the linear local abundance map, the transposed linear local abundance map is obtained.

[0064] Step 3.1.3: Matrix multiply the linear local abundance map and the transposed linear local abundance map to obtain the local abundance attention map. The formula is as follows:

[0065]

[0066] is the i-th linear local abundance map; W q is a linear local abundance map The linear mapping matrix, W k is the transposed linear local abundance map The linear mapping matrix, W q , W k ∈R HW×HW ; represents the local abundance attention map of the i-th local block, A sequential label for a local block.

[0067] Step 3.2: Transform the output of the feature extraction network through a 3×3 convolution kernel to obtain a high-dimensional feature map F. in ∈R K×H×W Divide into P×P non-overlapping local blocks and expand them into a one-dimensional form in space K is the weight, and the size of each local block is h×w, where After linear mapping, we get a local high-dimensional feature map The linear mapping process is expressed as follows:

[0068]

[0069] is the local high-dimensional feature map corresponding to the i-th local block, W v To realize the linear mapping matrix of local high-dimensional feature maps, is the i-th local block of the input.

[0070] Step 3.3: Use matrix multiplication to fuse the spatial context into the features of the local abundance attention map and the local high-dimensional feature map to obtain the first fused features of all local blocks. The process is shown as follows:

[0071]

[0072] Among them, W l The mapping matrix is ​​used to fuse the local attention map and the high-dimensional feature map so that different local blocks have the same local context estimation. is the first fusion feature, which represents the result of weighting the high-dimensional feature map according to the attention relationship. i ranges from 0 to P 2 -1, traverse all local blocks once and complete the fusion of local features of all sub-regions (local blocks).

[0073] Step 3.4: Consider the spatial weights of different local attention maps According to the weight of each local block, the local abundance attention map is processed through the mapping matrix, and the local abundance map is obtained after weighting

[0074] Considering the spatial weights of different local attention maps, we have

[0075] M g =W g (AvgPool 1×1 W l (M A ))

[0076] in, represents the local abundance attention map, Represents the mapping matrix W g The weight of each sub-region after enhancement, AvgPool represents the average pooling operation;

[0077] The spatial weight expression of the local block of the i-th high-dimensional feature map is as follows:

[0078]

[0079] is the spatial weight of the local block of the i-th high-dimensional feature map, W g is the mapping matrix;

[0080] Will Replicate in two dimensions to get the local abundance map

[0081] Step 3.5: Fuse the first fused feature with the local abundance map to obtain a high-dimensional feature vector fused with local context and global context information; the expression is as follows:

[0082]

[0083] represents the high-dimensional feature vector corresponding to the i-th local block, represents the local abundance map corresponding to the ith local block, The first fused feature corresponding to the i-th local block.

[0084] Step 3.6: Convert the high-dimensional feature vector According to the reverse operation of segmentation, rearrange it into a three-dimensional form to obtain the first output F g ∈R K×H×W ;

[0085] Step 3.7: In order to make the network more powerful without changing the main structure of the network, the first output F gAnd the high-dimensional feature map F of the input in The second output is obtained by adding them together, which is the feature map F0 that integrates the semantic information of the abundance map. The expression is as follows:

[0086] F0=F g +F in

[0087] In step 4, an upsampling module and a classification module are designed, and the feature extraction module, the local-global abundance attention module, the upsampling module and the classification module are connected in series to form a backbone sub-pixel mapping network;

[0088] Specifically, the high-dimensional feature map F in It is connected with the feature map F0 that integrates the semantic information of the abundance map as the input of the upsampling module and the classification module to obtain the sub-pixel mapping result.

[0089] X=Conv 1×1 (Up(Conv 3×3 (Cat[F in ,F0])))

[0090] Among them, F in is a high-dimensional feature map, F0 is a feature map that integrates the semantic information of the abundance map, Cat[] is a connection function for constructing a multidimensional array, Conv 3×3 () is a 3x3 convolution operation for information fusion, Up() is an upsampling operation of the upsampling module; Conv 1×1 () is the 1×1 convolution operation of the classification module.

[0091] In step 5, the cross entropy loss is used as the loss L between the sub-pixel mapping result X and the result Xgt in the test set ce (), and use the root mean square error as the difference between the abundance map output result A in the auxiliary branch and the result A in the test set gt The loss between mse (), the final loss function can be expressed as L = L ce (X,X gt )+λL mse (A,A gt )

[0092] Among them, X gt A is obtained by converting the feature labels into one-hot codes. gt By downsampling X gt It is obtained that λ is an empirical parameter used to balance the impact of the two losses.

[0093] In specific implementation, the design scheme of the hyperspectral sub-pixel mapping network based on the local-global abundance attention mechanism can be implemented by software. In order to objectively measure the fusion performance of the method proposed in the present invention, the performance of each method is evaluated from both qualitative and quantitative aspects. Qualitative evaluation is a subjective evaluation method that depends on human visual perception. A good positioning result should have smooth inter-class texture and high intra-class consistency. Quantitative evaluation objectively evaluates the positioning performance through some statistical indicators. This paper selects three quantitative indicators widely used in the field of sub-pixel mapping, such as F1 score (F1 Score, F1), overall accuracy (Overall Accuracy, OA), and mean intersection over Union (Mean Intersection over Union, mIoU). F1 can comprehensively evaluate the accuracy and precision of each category. The higher the F1, the more balanced the accuracy and precision of the mapping results in each category. OA measures the ratio of predicted correct pixels to total pixels. From the perspective of pixels, the higher the OA, the better the positioning accuracy. mIoU measures the average value of the overlap between each category and the real area. The larger the mIoU, the better the visual positioning effect.

[0094] Qualitative comparison: 7 typical results are selected to qualitatively demonstrate the performance of each algorithm, such as Figure 5 As shown. Seven methods were selected for comparison with the method of the present invention, namely SSNET (Synthetic Surveillance through Regression Network, an algorithm based on deep learning), SRMCNN (A Style-based Recalibration Module for Convolutional Neural Networks, residual convolutional network), SRMCNN-ESPCN (residual convolution-super resolution joint algorithm), DeepRSM model jointly proposed by Microsoft and Tsinghua University, super-resolution reconstruction model SRCNN-SPM, text matching network SIMNet and displacement convolution network SCNet.

[0095] SSNET, SRMCNN and SRMCNN-ESPCN have simple structures and few parameters, but still have good mapping effects, which is caused by the imbalance of data sets. For DeepSRM and SRCNN-SPM, due to the use of more complex abundance generation networks to obtain reliable abundance information as input, these two models have better positioning performance. However, in Figure (1), although the inter-class differences between farmland and urban built-up areas are extremely large, DeepSRM cannot distinguish the types of objects on the class boundary. This is because the two spectra are aliased in the mixed pixels at low resolution, and the discriminative features disappear; for the intra-class situation, due to the variability of endmembers, the network can easily confuse objects with similar average spectra, such as the shoal and farmland here. Therefore, in order to solve the problem of mixed pixels between classes and the problem of endmember variation within classes, it is necessary to use object information to assist network positioning. Among the three abundance-assisted positioning methods, SIMNet has better positioning effect than SCNet, which reflects the effectiveness of the abundance information fusion method through "shift and scale" in hyperspectral datasets. The network of the present invention uses local abundance attention fusion to ensure the clarity of boundary texture between classes, and uses global abundance fusion to ensure the consistency of semantics within a class, thus achieving the best positioning effect. In general, the method proposed in the present invention is superior to these latest methods in qualitative comparison.

[0096] Quantitative comparison: The quantitative comparison results on the WHU-OHS dataset are shown in Table 1. The proposed method achieves a Mean F1 of 63.23, a mIoU (semantic segmentation index) of 50.20%, and an OA (overall accuracy) of 76.87% on the test set, which is 2.08, 1.58%, and 0.68% higher than the optimal network, and has the best comprehensive performance. In general, the proposed method is also quantitatively superior to these comparison algorithms.

[0097] Table 1 Quantitative results of comparative tests, bold indicates the best

[0098]

[0099] The above description is only a preferred embodiment of the present invention and does not limit the technical scope of the present invention. Therefore, any slight modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A sub-pixel mapping method based on local-global abundance attention mechanism, It is characterized in that The steps include: Design a feature extraction module to process the original low-resolution spectrum map and extract a high-dimensional feature map with the same resolution as the original low-resolution spectrum map; Design auxiliary branches to transform the high-dimensional feature map into an abundance map; Design a local-global abundance attention module, which takes high-dimensional feature maps and abundance maps as inputs, first locally fuses abundance information using matrix multiplication to aggregate local features; then globally performs weighted operations on regions to obtain feature maps that are globally consistent in space; Design upsampling modules and classification modules, and connect the feature extraction module, local-global abundance attention module, upsampling module and classification module in series to form a backbone sub-pixel mapping network; A loss function is designed to train the backbone network and auxiliary branches, and finally an end-to-end mapping of the low-resolution spectral image to the high-resolution sub-pixel localization image is formed.

2. According to the sub-pixel mapping method based on the local-global abundance attention mechanism of claim 1, It is characterized in that The feature extraction module includes an encoder and a decoder, wherein the encoder completes the extraction of low-level semantic information to high-level semantic information of the original low-resolution spectrogram, and the spatial resolution gradually decreases; the decoder gradually increases the spatial resolution while ensuring the number of channels, completes multi-scale feature fusion through jump connections, and outputs a high-dimensional feature map with the same spatial resolution as the input.

3. According to the sub-pixel mapping method based on the local-global abundance attention mechanism of claim 2, It is characterized in that The feature extraction module design method is as follows: An encoder E is established. The encoder E uses the structure of the ResNet-50 backbone network and makes changes to the original ResNet-50: a 3×3 convolution kernel is used instead of a 7×7 convolution kernel to improve the problem of spatial information loss, and the downsampling rate is changed from 32 to 8; downsampling is performed three times only in the backbone layer, the maximum pooling stage, and the second layer in the initial input feature extraction stage, and the downsampling rate is 2 each time, and no downsampling is performed in the third and fourth layers; Cancel the fully connected layer of ResNet-50 and use the output of the fourth layer of ResNet-50 as the encoder output; Establish a decoder D, which uses bilinear interpolation to complete the resolution transformation; in the upsampling process, the output result of the encoder E is upsampled once with an upsampling rate of 2, and then the sampling result and the maximum pooling result of the encoder are spliced ​​after 1×1 convolution kernel channel, batch normalization and ReLU activation function operations; The calculation result is then subjected to 3×3 convolution, batch normalization, and ReLU activation function operations, and an upsampling with an upsampling rate of 2 is performed; the calculation result is then concatenated with the encoder's maximum pooling result by 1×1 convolution kernel channel, batch normalization, and ReLU activation function operations; the calculation result is once again subjected to 3×3 convolution, batch normalization, and ReLU activation function operations, and an upsampling with an upsampling rate of 2 is performed.

4. According to the sub-pixel mapping method based on the local-global abundance attention mechanism of claim 2, It is characterized in that The design auxiliary branches are as follows: Design a 1×1 convolution kernel to transform the high-dimensional feature map into an abundance map.

5. According to the sub-pixel mapping method based on the local-global abundance attention mechanism of claim 2, It is characterized in that The method for designing the local-global abundance attention module is as follows: The abundance map is divided into P×P non-overlapping local blocks, and a local abundance attention map is obtained after linear mapping; The high-dimensional feature map is divided into P×P non-overlapping local blocks, and a local high-dimensional feature map is obtained after linear mapping; The local abundance attention map and the local high-dimensional feature map are used to fuse the spatial context into the features using matrix multiplication to obtain the first fused features of all local blocks; Considering the spatial weights of different local attention maps, the local abundance attention map is processed through the mapping matrix according to the weight of each local block, and the local abundance map is obtained after weighting; The first fusion feature and the local abundance map are fused to obtain a high-dimensional feature vector fused with local context information and global context information; Restore the high-dimensional feature vector to a three-dimensional expression form according to the inverse operation of segmentation to obtain a first output; The first output and the input high-dimensional feature map are added together to obtain the second output, which is a feature map that incorporates the semantic information of the abundance map.

6. The sub-pixel mapping method based on the local-global abundance attention mechanism according to claim 5, It is characterized in that The abundance map is divided into P×P non-overlapping local blocks, and a local abundance attention map is obtained after linear mapping, which also includes linear mapping the local blocks of each abundance map to obtain a linear local abundance map; Transpose the linear local abundance map to obtain a transposed linear local abundance map; The local abundance attention map is obtained by matrix multiplying the linear local abundance map and the transposed linear local abundance map.

7. The sub-pixel mapping method based on the local-global abundance attention mechanism according to claim 5, It is characterized in that The high-dimensional feature map and the feature map that incorporates the semantic information of the abundance map are connected together as the input of the upsampling module and the classification module to obtain the sub-pixel mapping result. X=Conv 1×1 (Up(Conv 3×3 (Cat[F in ,F o ]))) Among them, F in is a high-dimensional feature map, F o is a feature map that integrates the semantic information of the abundance map, Cat[] is a connection function for constructing a multidimensional array, Conv 3×3 is a 3x3 convolution operation for information fusion, Up() is an upsampling operation of the upsampling module; Conv 1×1 It is the 1×1 convolution operation of the classification module.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the sub-pixel mapping method according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the sub-pixel mapping method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the sub-pixel mapping method according to any one of claims 1 to 6 is implemented.

Images