Sub-pixel mapping method based on local-global abundance attention mechanism

Abstract

This paper discloses a sub-pixel mapping method based on a local-global abundance attention mechanism. This method uses an end-to-end network, assisted by abundance semantic information, to address the mixed pixel problem and achieve multi-scale information supervision. The method introduces an abundance attention mechanism, uses the generated abundance map to estimate contextual relationships, and integrates this into the generation of sub-pixel classification maps. Furthermore, a local-global spatial information fusion strategy is employed, first performing local information fusion to ensure local positioning, and then performing global information fusion to ensure global context consistency. Compared to existing methods, this proposed method offers improved interpretability, high mapping accuracy, and a wide range of applicability.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and relates to a hyperspectral image sub-pixel mapping technology. Specifically, the present invention relates to a sub-pixel mapping method based on a local-global abundance attention mechanism. Background Art

[0002] In recent decades, hyperspectral imagery, due to its rich spatial and spectral information, has been widely used in land resource utilization, ecological and environmental protection, and urban planning. However, the spatial resolution of currently used hyperspectral imagery is limited, resulting in the presence of multiple objects within a single pixel, a phenomenon known as the mixed pixel problem. Subpixel mapping involves dividing a pixel into multiple subpixels and assigning a feature label to each subpixel.

[0003] While deep learning-based end-to-end convolutional neural networks have achieved impressive sub-pixel mapping results over the past decade, their ability to extract spatial context has been limited. Firstly, during feature extraction, convolutional structures prioritize low-frequency information, resulting in a loss of high-frequency spatial texture information. Secondly, remote sensing images exhibit end-member variability, meaning the spectral information of the same feature can vary, making it difficult for convolutional networks to maintain intra-class consistency. Using prior information about features to aid the network in extracting spatial context is a viable approach. For example, SIMNet integrates abundance images into an end-to-end network using a "Scale and Shift" matrix multiplication-addition structure, demonstrating that prior information about features can guide the network in extracting spatial detail. HiSMNet uses low-frequency abundance maps as prior knowledge, forcing the network to focus on high-frequency details. SCNet adds an auxiliary encoding branch to the feature extraction stage, integrating the abundance map as prior knowledge into the main branch through concatenation.

[0004] Two issues remain to be addressed for the aforementioned methods that use abundance semantic information to assist localization. First, in practice, abundance maps are not always readily available. We need to incorporate the generation of these maps into the network structure to achieve a truly end-to-end network. Second, these methods integrate global spatial prior information into the backbone network. Directly incorporating this global prior information fails to emphasize the differences in correlations between end-members at different distances, potentially causing the network to overlook local texture details. More effective information fusion methods are needed. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this paper proposes a sub-pixel mapping method based on a local-global abundance attention mechanism. The resulting sub-pixel classification map has higher positioning accuracy, richer local texture structure, and better global consistency.

[0006] This paper integrates two sub-pixel localization methods and proposes a multi-task network that performs unmixing and localization in parallel. The unmixed abundance map guides the localization process through a "local-global" abundance attention mechanism. This network considers the physical connection between the low-resolution abundance map and the high-resolution feature distribution map, utilizing a shared feature extraction unit to mine high-dimensional feature maps from the original remote sensing image. An auxiliary branch then reduces the feature dimensionality to produce an abundance map. The abundance map is then segmented into distinct regions, and the local context is incorporated into the high-dimensional feature map of the main branch. Global information fusion is then performed to ensure global context consistency. Finally, an upsampling module and a classification module are used to obtain the final sub-pixel localization result.

[0007] The technical solution of the present invention comprises the following steps:

[0008] In one aspect, the present invention provides a sub-pixel mapping method based on a local-global abundance attention mechanism, comprising the following steps:

[0009] Design a feature extraction module to process the original low-resolution spectrum and extract a high-dimensional feature map with the same resolution as the original low-resolution spectrum;

[0010] Design auxiliary branches to transform the high-dimensional feature map into an abundance map;

[0011] We design a local-global abundance attention module that takes a high-dimensional feature map and an abundance map as input. We first perform matrix multiplication to fuse abundance information locally to aggregate local features. We then perform a weighted operation on the regions globally to obtain a spatially consistent feature map.

[0012] Design an upsampling module and a classification module, and connect the feature extraction module, the local-global abundance attention module, the upsampling module, and the classification module in series to form a backbone sub-pixel mapping network;

[0013] A loss function is designed to train the backbone network and auxiliary branches, and ultimately form an end-to-end mapping from low-resolution spectral images to high-resolution sub-pixel localization maps.

[0014] Furthermore, the feature extraction module includes an encoder and a decoder, wherein the encoder completes the extraction of low-level semantic information of the original low-resolution spectrogram to high-level semantic information, and the spatial resolution gradually becomes smaller; the decoder gradually increases the spatial resolution while ensuring the number of channels, completes multi-scale feature fusion through jump connections, and outputs a high-dimensional feature map with the same spatial resolution as the input.

[0015] Furthermore, the feature extraction module design method is as follows:

[0016] Establish an encoder E that uses the ResNet-50 backbone network structure and makes some modifications to the original ResNet-50: using 3×3 convolution kernels instead of 7×7 convolution kernels to improve the problem of spatial information loss, and changing the downsampling rate from 32 to 8; downsampling is performed only three times in the backbone layer of the initial input feature extraction stage, the maximum pooling stage, and the second layer, with a downsampling rate of 2 each time; the third and fourth layers are not downsampled; the fully connected layer of ResNet-50 is removed, and the output of the fourth layer of ResNet-50 is used as the encoder output;

[0017] Establish a decoder D, which uses bilinear interpolation to complete the resolution conversion; in the upsampling process, the output result of the encoder E is upsampled once with an upsampling rate of 2, and the sampling result and the maximum pooling result of the encoder are spliced ​​after 1×1 convolution kernel channel, batch normalization and ReLU activation function operations; then the operation result is subjected to 3×3 convolution, batch normalization and ReLU activation function operations, and an upsampling rate of 2 is performed; then the operation result and the maximum pooling result of the encoder are spliced ​​with the results of 1×1 convolution kernel channel, batch normalization and ReLU activation function operations; the operation result is again subjected to 3×3 convolution, batch normalization and ReLU activation function operations, and an upsampling rate of 2 is performed.

[0018] Furthermore, the auxiliary branches are designed as follows:

[0019] Design a 1×1 convolution kernel to transform the high-dimensional feature map into an abundance map.

[0020] Furthermore, the method of designing the local-global abundance attention module is as follows:

[0021] The abundance map is divided into P×P non-overlapping local blocks, and a local abundance attention map is obtained after linear mapping;

[0022] The high-dimensional feature map is divided into P×P non-overlapping local blocks, and a local high-dimensional feature map is obtained after linear mapping;

[0023] The local abundance attention map and the local high-dimensional feature map are used to fuse the spatial context into the features using matrix multiplication to obtain the first fused features of all local blocks;

[0024] Considering the spatial weights of different local attention maps, the local abundance attention map is processed through the mapping matrix according to the weight of each local block, and the local abundance map is obtained after weighting;

[0025] The first fusion feature and the local abundance map are fused to obtain a high-dimensional feature vector fused with local context and global context information;

[0026] Restore the high-dimensional feature vector into a three-dimensional expression according to the inverse operation of segmentation to obtain the first output;

[0027] The second output is obtained by adding the first output and the input high-dimensional feature map, which is a feature map that integrates the semantic information of the abundance map.

[0028] Furthermore, the abundance map is divided into P×P non-overlapping local blocks, and a local abundance attention map is obtained after linear mapping, which also includes

[0029] Perform linear mapping on the local blocks of each abundance map to obtain a linear local abundance map;

[0030] Transpose the linear local abundance map to obtain a transposed linear local abundance map;

[0031] The local abundance attention map is obtained by matrix multiplying the linear local abundance map and the transposed linear local abundance map.

[0032] Furthermore, the high-dimensional feature map and the feature map that integrates the semantic information of the abundance map are connected together as the input of the upsampling module and the classification module to obtain the sub-pixel mapping result.

[0033] X=Conν 1×1 (Up(Conν 3×3 (Cat[F on ,F o ])))

[0034] Among them, F in is a high-dimensional feature map, F o is a feature map that integrates the semantic information of the abundance map, Cat[] is a connection function that constructs a multidimensional array, Conv 3×3 is the 3x3 convolution operation for information fusion, Up(·) is the upsampling operation of the upsampling module; Conv 1×1 It is the 1×1 convolution operation of the classification module.

[0035] On the other hand, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned sub-pixel mapping method when executing the program.

[0036] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned sub-pixel mapping method when executed by a processor.

[0037] On the other hand, the present invention provides a computer program product, comprising a computer program, which implements the above-mentioned sub-pixel mapping method when executed by a processor.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] (1) This paper proposes a novel end-to-end sub-pixel mapping network that uses auxiliary branches to mine semantic information and integrates it into the features extracted by the backbone network. The semantic information is no longer provided externally but is learned by the network, which enables errors to be corrected through backpropagation.

[0040] (2) This paper proposes a new abundance attention mechanism as an information fusion method, which uses the abundance map to extract the correlation between pixels and integrates the correlation into the high-dimensional features of the backbone network;

[0041] (3) The present invention effectively enhances the inter-class differences of features in different contexts and the intra-class consistency in the global context through local-global information fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the sub-pixel mapping method based on the local-global abundance attention mechanism of the present invention.

[0043] Figure 2 It is the overall framework of the sub-pixel mapping model of the present invention.

[0044] Figure 3 This is the feature extraction network structure diagram of the present invention.

[0045] Figure 4 Schematic diagram of the structure of the local-global abundance attention module in an embodiment of the present invention.

[0046] Figure 5 This is the qualitative result of the comparative experiment of the present invention. DETAILED DESCRIPTION

[0047] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0048] The method described in this paper uses the hyperspectral WHU-OHS dataset as data for comparative experiments. This dataset consists of 7,793 512×512 hyperspectral images with a spatial resolution of 10 meters and 24 types of ground feature labels. The images are collected from 40 different regions in China, and each image is labeled at the pixel level. The dataset covers 32 bands from visible light to near-infrared light, with an average spectral resolution of 15 nm.

[0049] like Figure 1, the embodiment of the present invention provides a sub-pixel mapping method based on the local-global abundance attention mechanism, and the obtained sub-pixel mapping model based on the local-global abundance attention mechanism is as follows Figure 2 As shown, the following steps are included:

[0050] Step 1: Design a feature extraction module to process the original low-resolution spectrum and extract a high-dimensional feature map F with the same resolution as the original low-resolution spectrum. in ;

[0051] Step 2: Design an auxiliary branch to perform channel transformation on the high-dimensional feature map to obtain the abundance map A;

[0052] Step 3: Design a local-global abundance attention module to use the high-dimensional feature map F in Taking the abundance map A as input, we first use matrix multiplication to fuse the abundance information locally to aggregate local features. Then we perform a weighted operation on the region globally to obtain a feature map F0 that is globally consistent in space.

[0053] Step 4: Design an upsampling module and a classification module, and connect the feature extraction module, the local-global abundance attention module, the upsampling module, and the classification module in series to form a backbone sub-pixel mapping network;

[0054] Step 5: Design a loss function to train the backbone network and auxiliary branches, and finally form an end-to-end mapping from the low-resolution spectral image to the high-resolution sub-pixel positioning image.

[0055] In step 1, the original image I (resolution size is H×W) is extracted by the feature extraction module to obtain a high-dimensional feature map F in (resolution size is H×W), the feature extraction module can be replaced by a suitable feature extraction network in the prior art. The present invention provides an exemplary description, wherein the feature extraction module includes an encoder and a decoder, such as Figure 3 As shown, the specific implementation method is as follows;

[0056] Step 1.1: Establish encoder E. The encoder E uses the structure of the ResNet-50 backbone network and makes changes to the original ResNet-50: use 3×3 convolution kernel instead of 7×7 convolution kernel to improve the problem of spatial information loss; change the downsampling rate from 32 to 8, and perform downsampling three times only in the initial input feature extraction stage (backbone layer), the subsequent maximum pooling stage and the second layer, each downsampling rate is 2, and the third and fourth layers are not downsampled; cancel the fully connected layer of ResNet-50, and use the output of the fourth layer of ResNet-50 as the encoder output; in Figure 3 In the example, the structure of each layer of the encoder is consistent with the original ResNet-50 and will not be repeated here;

[0057] Step 1.2: Establish a decoder D, which uses bilinear interpolation to achieve resolution conversion. During the upsampling process, the output of encoder E is upsampled once with an upsampling rate of 2. The sampling result is then concatenated with the encoder's maximum pooling result after a 1×1 convolution kernel channel, batch normalization, and ReLU activation function operation. The above operation result is then subjected to a 3×3 convolution, batch normalization, and ReLU activation function operation, and then upsampled once with an upsampling rate of 2. The above operation result is then concatenated with the encoder's maximum pooling result after a 1×1 convolution kernel channel, batch normalization, and ReLU activation function operation. The above operation result is then subjected to a 3×3 convolution, batch normalization, and ReLU activation function operation, and then upsampled once with an upsampling rate of 2. Through these operations, the decoder continuously increases spatial resolution while maintaining feature dimensionality, achieving multi-scale feature fusion.

[0058] like Figure 2 As shown, in step 2, the specific implementation of step 2 is as follows;

[0059] Design a 1×1 convolution kernel to transform the high-dimensional feature map into the abundance map A∈R C×H×W , R represents the abundance map set, C is the number of channels, and H×W is the resolution size.

[0060] In step 3, if Figure 2 and Figure 4 As shown in Figure 2, the method for designing the local-global abundance attention module is as follows:

[0061] Step 3.1: Divide the abundance map A into P×P non-overlapping local blocks, where P represents the segmentation ratio. 2 local blocks, the i-th local block is represented by A (i) , each local block size is h×w, where After linear mapping, the local abundance attention map M is obtained A The specific process is as follows:

[0062] Step 3.1.1, the abundance map local block A (i) Perform linear mapping to obtain a linear local abundance map The i-th linear local abundance map is denoted as

[0063] Step 3.1.2: Transpose the linear local abundance map to obtain the transposed linear local abundance map.

[0064] Step 3.1.3: Perform matrix multiplication on the linear local abundance map and the transposed linear local abundance map to obtain the local abundance attention map. The formula is as follows:

[0065]

[0066] is the i-th linear local abundance map; W q is a linear local abundance map The linear mapping matrix, W k is the transposed linear local abundance map The linear mapping matrix, W q 、W k ∈R HW×HW ; represents the local abundance attention map of the i-th local block, is the sequential label of the local block.

[0067] Step 3.2: Transform the output of the feature extraction network through a 3×3 convolution kernel to obtain a high-dimensional feature map F. in ∈R K×H×W Divide into P×P non-overlapping local blocks and expand them into a one-dimensional form in space K is the weight, and the size of each local block is h×w, where After linear mapping, the local high-dimensional feature map is obtained The linear mapping process is expressed as follows:

[0068]

[0069] is the local high-dimensional feature map corresponding to the i-th local block, W v To realize the linear mapping matrix of local high-dimensional feature maps, is the i-th local block of the input.

[0070] Step 3.3: Use matrix multiplication to fuse the spatial context into the features of the local abundance attention map and the local high-dimensional feature map to obtain the first fused features of all local blocks. The process is shown as follows:

[0071]

[0072] Among them, W l The mapping matrix is ​​used to fuse the local attention map and the high-dimensional feature map so that different local blocks have the same local context estimation. is the first fusion feature, which represents the result of weighting the high-dimensional feature map according to the attention relationship. i ranges from 0 to P 2 -1, traverse all local blocks and complete the fusion of local features of all sub-regions (local blocks).

[0073] Step 3.4: Consider the spatial weights of different local attention maps According to the weight of each local block, the local abundance attention map is processed through the mapping matrix, and the weighted local abundance map is obtained

[0074] Considering the spatial weights of different local attention maps, we have

[0075] M g =W g (AvgPool 1×1 W l (M A ))

[0076] in, represents the local abundance attention map, Represents the mapping matrix W g The weight of each sub-region after enhancement, AvgPool represents the average pooling operation;

[0077] The spatial weight expression of the local block of the i-th high-dimensional feature map is as follows:

[0078]

[0079] is the spatial weight of the local block of the i-th high-dimensional feature map, W g is the mapping matrix;

[0080] Will Replicate in two dimensions to get a local abundance map

[0081] Step 3.5: Fuse the first fused feature with the local abundance map to obtain a high-dimensional feature vector that is a fusion of local context and global context information; the expression is as follows:

[0082]

[0083] Represents the high-dimensional feature vector corresponding to the i-th local block, represents the local abundance map corresponding to the i-th local block, The first fusion feature corresponding to the i-th local block.

[0084] Step 3.6: High-dimensional feature vector According to the inverse operation of segmentation, rearrange it into a three-dimensional form to obtain the first output F g ∈R K×H×W ;

[0085] Step 3.7: In order to make the network have stronger representation ability and not change the main structure of the network, the first output F gAnd the input high-dimensional feature map F in The second output is obtained by adding them together, which is the feature map F0 that integrates the semantic information of the abundance map. The expression is as follows:

[0086] F0=F g +F in

[0087] In step 4, the upsampling module and the classification module are designed, and the feature extraction module, the local-global abundance attention module, the upsampling module and the classification module are connected in series to form a backbone sub-pixel mapping network;

[0088] Specifically, the high-dimensional feature map F in It is connected with the feature map F0 that integrates the semantic information of the abundance map as the input of the upsampling module and the classification module to obtain the sub-pixel mapping result.

[0089] X=Conv 1×1 (Up(Conv 3×3 (Cat[F in ,F0])))

[0090] Among them, F in is a high-dimensional feature map, F0 is a feature map that integrates the semantic information of the abundance map, Cat[] is a connection function for constructing a multidimensional array, Conv 3×3 () is the 3x3 convolution operation for information fusion, Up() is the upsampling operation of the upsampling module; Conv 1×1 () is the 1×1 convolution operation of the classification module.

[0091] In step 5, the cross entropy loss is used as the loss L between the sub-pixel mapping result X and the test set result Xgt ce (), and use the root mean square error as the difference between the abundance map output result A in the auxiliary branch and the result A in the test set. gt The loss between mse (), the final loss function can be expressed as L = L ce (X,X gt )+λL mse (A,A gt )

[0092] Among them, X gt A is obtained by converting the feature labels into a one-hot code. gt By downsampling X gt It is obtained that λ is an empirical parameter used to balance the impact of the two losses.

[0093] In practice, a design scheme for a hyperspectral sub-pixel mapping network based on a local-global abundance attention mechanism can be implemented using software. To objectively measure the fusion performance of the proposed method, the performance of each method was evaluated qualitatively and quantitatively. Qualitative evaluation is a subjective assessment method that relies on human visual perception. Good localization results should have smooth inter-class texture and high intra-class consistency. Quantitative evaluation objectively evaluates localization performance through statistical metrics. This paper selects three quantitative metrics widely used in the field of sub-pixel mapping: F1 score (F1), overall accuracy (OA), and mean intersection over union (mIoU). F1 comprehensively evaluates the accuracy and precision of each category. A higher F1 indicates a more balanced accuracy and precision of the mapping result across categories. OA measures the ratio of correctly predicted pixels to total pixels. From a pixel perspective, a higher OA indicates better localization accuracy. mIoU measures the average overlap between each category and the ground truth. A higher mIoU indicates better visual localization.

[0094] Qualitative comparison: 7 typical results were selected to qualitatively demonstrate the performance of each algorithm, such as Figure 5 Seven methods were selected for comparison with the method of the present invention, namely SSNET (Synthetic Surveillance through Regression Network, a deep learning-based algorithm), SRMCNN (A Style-based Recalibration Module for Convolutional Neural Networks, residual convolutional network), SRMCNN-ESPCN (residual convolution-super-resolution joint algorithm), DeepRSM model jointly proposed by Microsoft and Tsinghua University, super-resolution reconstruction model SRCNN-SPM, text matching network SIMNet and shift convolution network SCNet.

[0095] SSNET, SRMCNN and SRMCNN-ESPCN have simple structures and few parameters, but still have good mapping effects. This is due to the imbalance of the dataset categories. For DeepSRM and SRCNN-SPM, due to the use of a more complex abundance generation network to obtain reliable abundance information as input, these two models have better positioning performance. However, in Figure (1), although the inter-class differences between farmland and urban built-up areas are extremely large, DeepSRM cannot distinguish the types of land objects on the class boundary. This is because the two spectra are aliased in the mixed pixels at low resolution, and the discriminative features disappear; for the intra-class situation, due to the variability of end members, the network can easily confuse land objects with similar average spectra, such as the shallows and farmland here. Therefore, in order to solve the problem of mixed pixels between classes and the problem of end member variation within classes, it is necessary to use land object information to assist network positioning. Among the three abundance-assisted positioning methods, SIMNet has better positioning effect than SCNet, which reflects the effectiveness of the abundance information fusion method through "shift and scale" in hyperspectral datasets. The proposed network uses local abundance attention fusion to ensure the clarity of boundary textures between classes and global abundance fusion to ensure semantic consistency within classes, achieving the best localization results. Overall, the proposed method outperforms these state-of-the-art methods in qualitative comparison.

[0096] Quantitative Comparison: Quantitative comparison results on the WHU-OHS dataset are shown in Table 1. The proposed method achieves a Mean F1 score of 63.23, an mIoU (semantic segmentation metric) of 50.20%, and an Overall Accuracy of 76.87% on the test set, exceeding the optimal network by 2.08, 1.58%, and 0.68%, respectively, demonstrating optimal overall performance. Overall, the proposed method demonstrates quantitative advantages over these comparison algorithms.

[0097] Table 1 Quantitative results of comparative tests, bold indicates the best

[0098]

[0099] The above description is merely a preferred embodiment of the present invention and does not limit the technical scope of the present invention. Therefore, any minor modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A sub-pixel mapping method based on local-global abundance attention mechanism, characterized by: The steps include: Design a feature extraction module to process the original low-resolution spectrum and extract a high-dimensional feature map with the same resolution as the original low-resolution spectrum; Design auxiliary branches to transform the high-dimensional feature map into an abundance map; We design a local-global abundance attention module that takes a high-dimensional feature map and an abundance map as input. We first perform matrix multiplication to fuse abundance information locally to aggregate local features. We then perform a weighted operation on the regions globally to obtain a spatially consistent feature map. Design an upsampling module and a classification module, and connect the feature extraction module, the local-global abundance attention module, the upsampling module, and the classification module in series to form a backbone sub-pixel mapping network; A loss function is designed to train the backbone network and auxiliary branches, and ultimately form an end-to-end mapping from low-resolution spectral images to high-resolution sub-pixel localization images; The method for designing the local-global abundance attention module is as follows: Split the abundance map into After linear mapping of non-overlapping local blocks, the local abundance attention map is obtained; Divide the high-dimensional feature map into After linear mapping of non-overlapping local blocks, a local high-dimensional feature map is obtained; The local abundance attention map and the local high-dimensional feature map are used to fuse the spatial context into the features using matrix multiplication to obtain the first fused features of all local blocks; Considering the spatial weights of different local attention maps, the local abundance attention map is processed through the mapping matrix according to the weight of each local block, and the local abundance map is obtained after weighting; The first fusion feature and the local abundance map are fused to obtain a high-dimensional feature vector fused with local context and global context information; Restore the high-dimensional feature vector into a three-dimensional expression according to the inverse operation of segmentation to obtain the first output; The second output is obtained by adding the first output and the input high-dimensional feature map, which is a feature map that integrates the semantic information of the abundance map.

2. The sub-pixel mapping method based on the local-global abundance attention mechanism according to claim 1, characterized in that: The feature extraction module includes an encoder and a decoder, wherein the encoder completes the extraction of low-level semantic information from the original low-resolution spectrogram to high-level semantic information, and the spatial resolution gradually decreases; the decoder gradually increases the spatial resolution while ensuring the number of channels, completes multi-scale feature fusion through jump connections, and outputs a high-dimensional feature map with the same spatial resolution as the input.

3. The sub-pixel mapping method based on the local-global abundance attention mechanism according to claim 2, characterized in that: The feature extraction module design method is as follows: Building the encoder E , the encoder E Using the ResNet-50 backbone network structure, the original ResNet-50 was modified: 3×3 convolution kernels were used instead of 7×7 convolution kernels to improve the problem of spatial information loss, and the downsampling rate was changed from 32 to 8. Downsampling was only performed three times: in the backbone layer of the initial input feature extraction stage, in the maximum pooling stage, and in the second layer, with a downsampling rate of 2 each time. No downsampling was performed on the third and fourth layers. Cancel the fully connected layer of ResNet-50 and use the fourth layer output of ResNet-50 as the encoder output; Building a decoder D , the decoder D Use bilinear interpolation to complete the resolution transformation; in the process of upsampling, The encoder E The output result of the encoder is upsampled with an upsampling rate of 2, and then the sampling result is combined with the maximum pooling result of the encoder by 1 The convolution kernel channel, batch normalization and ReLU activation function are operated and then concatenated; the operation result is then subjected to 3×3 convolution, batch normalization and ReLU activation function operation, and an upsampling with an upsampling rate of 2 is performed; the operation result is then combined with the encoder maximum pooling result by 1 The results of the convolution kernel channel, batch normalization, and ReLU activation function operations are spliced ​​together; the operation results are subjected to another 3×3 convolution, batch normalization, and ReLU activation function operation, and then upsampled with an upsampling rate of 2.

4. The sub-pixel mapping method based on the local-global abundance attention mechanism according to claim 2, characterized in that: The design auxiliary branches are as follows: Design a Convolution kernel, which transforms the high-dimensional feature map into an abundance map.

5. The sub-pixel mapping method based on the local-global abundance attention mechanism according to claim 1, characterized in that: Split the abundance map into After linear mapping, the local abundance attention map is obtained, which also includes Perform linear mapping on the local blocks of each abundance map to obtain a linear local abundance map; Transpose the linear local abundance map to obtain a transposed linear local abundance map; The local abundance attention map is obtained by matrix multiplying the linear local abundance map and the transposed linear local abundance map.

6. The sub-pixel mapping method based on the local-global abundance attention mechanism according to claim 4, characterized in that: The high-dimensional feature map and the feature map that integrates the semantic information of the abundance map are connected together as the input of the upsampling module and the classification module to obtain the sub-pixel mapping result. in, is a high-dimensional feature map, is a feature map that integrates the semantic information of the abundance map. To construct a concatenation function for a multidimensional array, is the 3x3 convolution operation used for information fusion, It is the upsampling operation of the upsampling module; is the 1 × 1 convolution operation of the classification module.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the sub-pixel mapping method according to any one of claims 1 to 6 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sub-pixel mapping method according to any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the sub-pixel mapping method according to any one of claims 1 to 6 is implemented.

Images