A remote sensing image feature enhancement method and system

Abstract

The present invention relates to the field of image processing technology, and in particular to a method and system for feature enhancement of remote sensing images. The method comprises: constructing a feature enhancement model, the feature enhancement model comprising a bidirectional adaptive local-global feature encoder, a decoder and a reconstruction unit; the bidirectional adaptive local-global feature encoder is used to extract multi-resolution local-global features of fine-resolution images and multi-resolution local-global features of coarse-resolution images; by inputting an image to be processed into the bidirectional adaptive local-global feature encoder, shallow local features and local-global features of multiple resolutions are extracted; the shallow local features and the local-global features of multiple resolutions are gradually fused from low resolution to high resolution through a decoder to generate fused features of ideal resolution, and then the fused features of ideal resolution are generated into a high-resolution image through a reconstruction unit; the present invention can enhance the expression of multi-scale local and global features of remote sensing images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a remote sensing image feature enhancement method and system. Background Art

[0002] Images obtained by a single sensor cannot simultaneously meet the requirements of high spatial resolution, high spectral resolution, and high temporal resolution. Multi-source remote sensing image fusion technology is an effective way to obtain multispectral images with high spatial resolution (spatial-spectral fusion) and multispectral images with high temporal and spatial resolution (spatial-temporal fusion). However, most current deep learning models can only achieve spatial-spectral fusion or spatial-temporal fusion, and the generalization ability of the model is limited.

[0003] In addition, the targets in remote sensing images vary in size, and single-scale features are insufficient to express information about targets of multiple sizes. The limited receptive field of convolution in CNN has the advantage of acquiring local features, but has limited expression of long-distance dependencies, while Transformer can establish long-distance dependencies but its complexity grows quadratically with spatial resolution. The huge difference in resolution between multi-source remote sensing images and sensor bias make it difficult to reconstruct texture, details, and change information of coarse-resolution images. Most models directly extract features of coarse-resolution images by upsampling coarse-resolution images to fine-resolution images. Such a huge increase in resolution at one time will bring many uncertainties to the fusion. For example, the resolution ratio between Landsat images and MODIS images in spatiotemporal fusion is 16. Spatial-spectral fusion lacks reference images for supervised training of deep learning models, and the down-resolution training model leads to poor fusion performance on full-resolution data.

[0004] Therefore, it is necessary to provide a solution to enhance the expression of multi-scale local and global features of remote sensing images and reduce the spectral information loss and detail distortion that may be caused by increasing the resolution multiple by a large factor. Summary of the invention

[0005] The present invention aims to provide a remote sensing image feature enhancement method and system, which can enhance the expression of multi-scale local and global features of remote sensing images.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a method for enhancing features of a remote sensing image, the method comprising the following steps:

[0008] S100, constructing a feature enhancement model, wherein the feature enhancement model includes a bidirectional adaptive local-global feature encoder, a decoder, and a reconstruction unit; wherein the bidirectional adaptive local-global feature encoder is used to extract multi-resolution local-global features of a fine-resolution image and multi-resolution local-global features of a coarse-resolution image;

[0009] S200, obtaining an image to be processed, inputting the image to be processed into the bidirectional adaptive local-global feature encoder, extracting shallow local features and local-global features of multiple resolutions; wherein the image to be processed includes a fine-resolution image and a coarse-resolution image;

[0010] S300, the shallow local features and the local-global features of multiple resolutions are gradually fused from low resolution to high resolution through a decoder to generate fused features of ideal resolution, and then the fused features of ideal resolution are generated into a high-resolution image through a reconstruction unit.

[0011] Optionally, the bidirectional adaptive local-global feature encoder comprises two encoder branches, each of which comprises an MDconv module, an RMDCT module corresponding to N+1 resolution levels, and a sampling module;

[0012] In S200, the step of inputting the image to be processed into the bidirectional adaptive local-global feature encoder to extract shallow local features and local-global features of multiple resolutions includes:

[0013] S210, inputting the image to be processed into the MDconv module to extract shallow local features of the image to be processed;

[0014] S220 , extracting local-global features of N+1 resolution levels from the image to be processed through RMDCT modules of N+1 resolution levels and N sampling operations.

[0015] Optionally, in S210, inputting the image to be processed into an MDconv module to extract shallow local features of the image to be processed includes:

[0016] The shallow local features of the image to be processed are extracted by the MDconv module, and the mathematical expression is as follows:

[0017]

[0018] Among them, SF0 and SC N Represent the shallow local features of the fine-resolution image and the coarse-resolution image, respectively. represents a dilated convolution operation with a kernel size of 3 and a dilation rate of i, FI i and CIi Respectively indicated by The features of the extracted fine-resolution image and coarse-resolution image, represents the convolution operation with a convolution kernel of 1, κ lrelu (·) represents the leaky linear rectification function, κ concat (·) indicates the splicing operation along the channel direction.

[0019] Optionally, in S220, extracting local-global features of N+1 resolution levels from the image to be processed through RMDCT modules of N+1 resolution levels and N sampling operations includes:

[0020] The local-global features of N+1 resolution levels are extracted from the image to be processed by the following formula:

[0021] F0=κ RMDCT (SF0);

[0022] F n =κ RMDCT (DF n-1 )n=1,…,N;

[0023]

[0024] C N =κ RMDCT (SC N );

[0025] C n-1 =κ RMDCT (UC n )n=1,…,N;

[0026]

[0027] Among them, κ RMDCT (·) represents the function processed by the RMDCT module, F n represents the local-global features extracted from the fine-resolution image by the RMDCT module of the nth resolution level of an encoder branch of the bidirectional adaptive local-global feature encoder, C n represents the local-global features extracted from the coarse-resolution image by the RMDCT module at the nth resolution level of another encoder branch, DF n Indicates F n The local-global downsampling features obtained by downsampling operation, κ unshuffle (·) represents the downsampling operation, UC n Indicates C n The local-global upsampling feature obtained by upsampling operation, κ shuffle(·) represents upsampling operation, Represents a convolution operation with a kernel of 3; where n = 0, 1, …, N.

[0028] Optionally, the RMDCT module includes a residual-MDconv module and an MDCT module, the MDCT module is a multi-head deep convolution Transformer module, and the residual-MDconv module and the MDCT module constitute a residual structure; the residual-MDconv module adopts residual connection to avoid gradient disappearance, and is used to extract and enhance local information; the MDCT module includes a layer normalization processing module, a multi-head deep convolution attention module, an enhanced forward transfer network and a residual connection;

[0029] The RMDCT module processing process includes:

[0030] S510, performing layer normalization operation on the input image Z of the MDCT module to obtain normalized feature Z LN , for the normalized feature Z LN A 1×1 convolutional layer aggregates pixel-level cross-channel context and a 3×3 deep convolutional coding channel separates spatial context and generates queries, keys, and values; wherein the input image Z of the MDCT module of the first resolution level is the shallow local feature, and the input image Z of the subsequent MDCT module is the local-global down-sampled feature or the local-global up-sampled feature output by the previous resolution level;

[0031] The layer normalization operation and the process of generating queries, keys, and values ​​are expressed as follows:

[0032] Z LN =κ LN (Z);

[0033]

[0034] Among them, κ LN is the layer normalization operation, Z and Z LN is the input image of the MDCT module and the normalized features after the input image layer is normalized, Q, K, and V are the generated query, key, and value, respectively. and Indicates that the mapping generates Q with a depth convolution operation of 3×3 kernel and a convolution operation with a kernel of 1; and Indicates that the mapping generates K with a depth convolution operation of 3×3 kernel and a convolution operation with a kernel of 1; and Indicates that the mapping generates V by a depthwise convolution operation with a kernel of 3×3 and a convolution operation with a kernel of 1;

[0035] S520, transform the query Q and the key K to obtain Q r and K r , and perform dot product and Softmax operations to generate a transposed attention map A, and obtain the global attention weight of the correlation between the query Q and the key K; wherein, the process expression of generating the transposed attention map A is as follows:

[0036]

[0037] Among them, A is the transposed attention image, Q r , K r It is obtained by transforming Q and K respectively; is a learnable parameter used to adjust the amplitude;

[0038] S530, transform the value V to obtain V r , the V r Multiplying with the transposed attention map A to obtain the corrected features helps to obtain useful local context; finally, the corrected features are deformed and feature interconnection is achieved through 1×1 convolution to obtain the interconnected features Z A , the interconnection feature Z A Perform residual connection with the input image Z to generate the first local-global feature Z′; wherein the generated interconnected feature Z A The process expression of the first local-global feature Z′ is as follows:

[0039]

[0040] Z′=Z+Z A ;

[0041] Among them, Z A is the interconnected feature after convolution self-attention correction, represents a convolution operation with a kernel of 1, κ r is the deformation operation, V r is obtained by deformation operation on V, and Z′ is the first local-global feature output after normalization, convolution self-attention and residual connection;

[0042] S540, the first local-global feature Z′ generates a second local-global feature Z″ through layer normalization operation, enhanced forward transfer network and residual connection; wherein the process expression of generating the second local-global feature Z″ is as follows:

[0043]

[0044] Among them, Z″ is the second local-global feature generated after layer normalization operation, enhanced forward transmission network, and residual connection, κ GELUis the Gaussian error linear unit activation function;

[0045] S550, performing residual connection on the second local-global feature Z″ generated by the MDCT module and the input image operated by the residual-MDconv module to generate the local-global feature of the nth resolution level RMDCT module in the bidirectional adaptive local-global feature encoder.

[0046] In a second aspect, an embodiment of the present invention provides a remote sensing image feature enhancement system, the system comprising:

[0047] at least one processor;

[0048] at least one memory for storing at least one program;

[0049] When the at least one program is executed by the at least one processor, the at least one processor implements the feature enhancement method of the remote sensing image as described in any one of the above items.

[0050] The beneficial effects of the present invention are as follows: the present invention discloses a method and system for feature enhancement of remote sensing images. The present invention extracts low-resolution and high-low resolution local-global features through a bidirectional local-global feature encoder, and performs feature association on the shallow local features of the input image and the corresponding local-global features at multiple resolutions to obtain an associated feature image, which can enhance the expression of multi-scale local and global features of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 A flow chart of a remote sensing image feature enhancement method in an embodiment of the present invention is shown.

[0053] Figure 2 A structural diagram of a bidirectional adaptive local-global feature encoder in an embodiment of the present invention is shown.

[0054] Figure 3 A structural diagram of the MDconv module in an embodiment of the present invention is shown.

[0055] Figure 4 The structure diagram of the down-sampling module and the up-sampling module in the embodiment of the present invention is shown.

[0056] Figure 5The structure diagram of the RMDCT module in the embodiment of the present invention is shown.

[0057] Figure 6 A structural diagram of a multi-head deep convolutional attention module and an enhanced forward transfer network in an embodiment of the present invention is shown.

[0058] Figure 7 A block diagram of a remote sensing image feature enhancement system in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0059] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0060] Among the related technologies, remote sensing images with high temporal and spatial resolution play a very important role in monitoring coastal ecological environment changes, marine resource monitoring, and urban dynamic planning. However, due to the limitations of sensor technology and other factors, the acquired images are mutually restricted in time, space, and spectral resolution, which to a certain extent limits the application of remote sensing images. Multi-source remote sensing fusion technology integrates multi-source complementary information to provide a convenient and effective solution for generating high-resolution images. At present, a large number of remote sensing image fusion methods have been developed, such as space-spectrum fusion, space-time fusion, and other technologies.

[0061] Spatial-spectral fusion aggregates the complementary spatial and spectral information between multi-source remote sensing images to generate multispectral images with high spatial resolution. Spatiotemporal fusion aggregates dual-source low-temporal-high spatial resolution images and high-temporal-low spatial resolution images to obtain high-temporal-spatial resolution images. However, most current deep learning models can only achieve spatial-spectral fusion or spatiotemporal fusion, and the generalization ability of the models is limited.

[0062] In addition, the targets in remote sensing images vary in size, and single-scale features are insufficient to express information about targets of multiple sizes. The limited receptive field of convolution in CNN has the advantage of acquiring local features, but has limited expression of long-distance dependencies, while Transformer can establish long-distance dependencies but its complexity grows quadratically with spatial resolution. The huge difference in resolution between multi-source remote sensing images and sensor bias make it difficult to reconstruct texture, details, and change information of coarse-resolution images. Most models directly extract features of coarse-resolution images by upsampling coarse-resolution images to fine-resolution images. Such a huge increase in resolution at one time will bring many uncertainties to the fusion. For example, the resolution ratio between Landsat images and MODIS images in spatiotemporal fusion is 16. Spatial-spectral fusion lacks reference images for supervised training of deep learning models, and the down-resolution training model leads to poor fusion performance on full-resolution data.

[0063] Taking the above challenges into consideration, the present invention uses a small resolution factor to gradually improve the spatial resolution, and can bidirectionally and adaptively extract multi-resolution features and adjust the resolution ratio according to the resolution characteristics of the coarse-fine image. Considering the advantages of CNN in extracting local features and Transformer in establishing long-distance dependencies, the deep convolutional Transformer in the Restormer model reduces the amount of calculation and is suitable for high-resolution images; Uformer also enhances local contextual information through a locally enhanced forward transfer module. The present invention also proposes a residual multi-head deep convolutional Transformer (RMDCT) module to enhance the expression of multi-scale local and global features of remote sensing images.

[0064] Combination Figure 1 and Figure 2 The embodiment of the present invention provides a method for enhancing the features of a remote sensing image, the method comprising the following steps:

[0065] S100, constructing a feature enhancement model, wherein the feature enhancement model includes a bidirectional adaptive local-global feature encoder, a decoder, and a reconstruction unit; wherein the bidirectional adaptive local-global feature encoder is used to extract multi-resolution local-global features of a fine-resolution image and multi-resolution local-global features of a coarse-resolution image;

[0066] S200, obtaining an image to be processed, inputting the image to be processed into the bidirectional adaptive local-global feature encoder, extracting shallow local features and local-global features of multiple resolutions; wherein the image to be processed includes a fine-resolution image and a coarse-resolution image;

[0067] S300, the shallow local features and the local-global features of multiple resolutions are gradually fused from low resolution to high resolution through a decoder to generate fused features of ideal resolution, and then a high-resolution image is generated by the fused features of ideal resolution through a reconstruction unit.

[0068] Specifically, the shallow local features and the local-global features of multiple resolutions are fused step by step from low resolution to high resolution through a decoder to generate fused features of ideal resolution; the fused features of ideal resolution are input into a reconstruction unit to generate a high-resolution image. The decoder and the reconstruction unit can be implemented using models such as convolution or Transformer in related technologies.

[0069] refer to Figures 2 to 6 In the figure, FI represents a fine resolution image, CI represents a coarse resolution image, and HI represents a high resolution image generated by the model of the present invention. n and C n (n=0,1,……,N) represent the local-global features of the extracted nth resolution, that is, the deep-level local-global features of the nth resolution level. The size of F0 is the same as that of the fine-resolution image. N The size of F0 and F n The ratio of the resolution is τ n , τ is the downsampling factor each time and τ=2 is set. n Indicates the generated intermediate image at the nth resolution level.

[0070] MDconv stands for the dilated convolution module, the RMDCT module is the residual multi-head deep convolution Transformer module, and BALGE stands for the bidirectional adaptive local-global feature encoder.

[0071] In some embodiments, the bidirectional adaptive local-global feature encoder includes N+1 resolution levels; the bidirectional adaptive local-global feature encoder includes two encoder branches, each of which includes an MDconv module, an RMDCT module corresponding to the N+1 resolution levels, and a sampling module;

[0072] Specifically, the bidirectional adaptive local-global feature encoder includes a first encoder branch and a second encoder branch, the first encoder branch includes a first sub-MDconv module, a first RMDCT module corresponding to N+1 resolution levels, and a downsampling module, and the second encoder branch includes a second sub-MDconv module, a second RMDCT module corresponding to N+1 resolution levels, and an upsampling module;

[0073] In S200, the step of inputting the image to be processed into the bidirectional adaptive local-global feature encoder to extract shallow local features and local-global features of multiple resolutions includes:

[0074] S210, inputting the image to be processed into the MDconv module to extract shallow local features corresponding to the image to be processed;

[0075] S220 , extracting local-global features of N+1 resolution levels from the image to be processed through RMDCT modules of N+1 resolution levels and N sampling operations.

[0076] Specifically, the first branch of the encoder extracts the deep local-global features F of the fine-resolution image through the first RMDCT module of the N+1 resolution level n , for the deep level local-global feature F n Perform N downsampling operations to obtain the local-global feature DF n The second branch of the encoder extracts the deep local-global features C of the coarse resolution image through the second RMDCT module of N+1 resolution levels n , for the deep level local-global feature C n Perform N upsampling operations to obtain local-global features UC n .

[0077] like Figure 2 As shown in the figure, the MDconv module is used to extract multi-scale shallow local features of fine-resolution images and coarse-resolution images. Its structure is as follows Figure 3 As shown in the figure, the multi-scale features of the fine-resolution image and the coarse-resolution image are extracted in parallel by convolutions with dilation rates of 1, 2, and 3, respectively. The multi-scale features are then concatenated and preliminarily fused and the number of features is adjusted by 1×1 convolution. Convolution (k3, r1) means that the convolution kernel is 3×3 and the dilation rate is 1; the dilation rates of convolution (k3, r2) and convolution (k3, r3) are 2 and 3, respectively.

[0078] The mathematical expression of the MDconv module for processing the image to be processed is as follows:

[0079]

[0080] Among them, SF0 and SC N Respectively represent the extraction of shallow local features of fine-resolution images and coarse-resolution images, represents a dilated convolution operation with a kernel size of 3 and a dilation rate of i, FI i and CI i Respectively indicated by The features of the extracted fine-resolution image and coarse-resolution image, represents the convolution operation with a convolution kernel of 1, κ lrelu (·) represents the leaky linear rectification (LReLU) function operation, κ concat (·) indicates the splicing operation along the channel direction.

[0081] In order to obtain the spatial details and spectral information of the fine-resolution image and the coarse-resolution image for fusion at multiple resolutions, a structure combining the RMDCT module with downsampling (the first branch of the encoder) and a structure combining the RMDCT module with upsampling (the second branch of the encoder) are designed. Figure 2 As shown, after N+1 RMDCT module processing and N downsampling processing, the local-global features of the fine-resolution image at N+1 resolutions are obtained; similarly, after N+1 RMDCT modules and N upsampling, the local-global features of the coarse-resolution image at N+1 resolutions are obtained. The structures of downsampling and upsampling are shown in Figure 4 As shown in the figure, the downsampling operation and upsampling operation are implemented by pixel-unshuffle operation and pixel-shuffle operation respectively, and the kernel size of convolution (k3) is 3, which is used to adjust the number of channels; the mathematical expression of the above implementation process is as follows:

[0082] F0=κ RMDCT (SF0);

[0083] F n =κ RMDCT (DF n-1 )n=1,…,N;

[0084]

[0085] C N =κ RMDCT (SC N );

[0086] C n-1 =κ RMDCT (UC n )n=1,…,N;

[0087]

[0088] Among them, κ RMDCT (·) represents the function of the RMDCT module (Residual Multi-Head Deep Convolution Transformer), F n represents the deep local-global features extracted from the fine-resolution image by the RMDCT module of the nth resolution level of the first branch of the encoder, C nrepresents the deep local-global features extracted from the coarse-resolution image by the RMDCT module of the nth resolution level of the second branch of the encoder, DF n Indicates F n The local-global downsampling features obtained by downsampling operation, κ unshuffle (·) represents the downsampling operation, UC n Indicates C n The local-global upsampling feature obtained by upsampling operation, κ shuffle (·) represents upsampling operation, Represents a convolution operation with a kernel of 3; where n = 0, 1, …, N.

[0089] In some embodiments, the RMDCT module includes a residual-MDconv module and an MDCT module, the MDCT module is a multi-head deep convolution Transformer module, and the residual-MDconv module and the MDCT module constitute a residual structure; the residual-MDconv module uses residual connection to avoid gradient disappearance, and is used to extract and enhance local information; the MDCT module includes a layer normalization processing module, a multi-head deep convolution attention module, an enhanced forward transfer network and a residual connection;

[0090] Specifically, the structure of the RMDCT module is as follows: Figure 5 As shown in the figure, the RMDCT module includes a residual-MDconv module and an MDCT module (multi-head deep convolution Transformer module), and the two constitute a residual structure. The residual-MDconv module is used to extract and enhance local information, and use residual connections to avoid gradient vanishing. Due to the limitation of the local receptive field of CNN, the global structure is ignored, and only extracting local features is not enough to reconstruct high-quality images. Therefore, the MDCT module is introduced to collaboratively learn long-distance dependencies within the image. The MDCT module includes layer normalization (LayerNorm), a multi-head deep convolution attention module (MDCA), an enhanced forward transfer network (EFFN) and residual connections. The multi-head deep convolution attention module is the same as the multi-head deep transposed attention structure in the Restormer model. In order to make up for the defect that ordinary self-attention cannot obtain local features and the computational complexity grows with the square of the image size, it is not suitable for high-resolution images, so deep convolution is introduced to enhance local features to generate a global attention map.

[0091] Specific as Figure 5 and Figure 6 As shown, the process of the RMDCT module includes:

[0092] First, the input image Z of the MDCT module is normalized to obtain the normalized feature Z LN, for the normalized feature Z LN A 1×1 convolutional layer aggregates pixel-level cross-channel context and a 3×3 deep convolutional coding channel separates spatial context and generates queries, keys, and values; wherein the input image Z of the MDCT module of the first resolution level is the shallow local feature, and the input image Z of the MDCT module of the nth resolution level is the local-global downsampled feature DF output by the previous resolution level n-1 Or local-global upsampling feature UC n ; The implementation process of layer normalization operation and generation of query, key and value is as follows:

[0093] Z LN =κ LN (Z);

[0094]

[0095] Among them, κ LN is the layer normalization operation, Z and Z LN is the input image of the MDCT module and the normalized features after the input image layer is normalized, Q, K, and V are the generated query, key, and value, respectively. and Indicates that the mapping generates Q with a depth convolution operation of 3×3 kernel and a convolution operation with a kernel of 1; and Indicates that the mapping generates K with a depth convolution operation of 3×3 kernel and a convolution operation with a kernel of 1; and Indicates that the mapping generates V by a depthwise convolution operation with a kernel of 3×3 and a convolution operation with a kernel of 1;

[0096] Next, transform the query Q and key K to obtain Q r and K r , and perform dot product and Softmax operations to generate a transposed attention map A, and obtain the global attention weight of the correlation between the query Q and the key K; wherein, the process expression of generating the transposed attention map A is as follows:

[0097]

[0098] Among them, A is the transposed attention image, Q r , K r It is obtained by transforming Q and K respectively; is a learnable parameter used to adjust the amplitude;

[0099] Then transform the value V to get V r , the V rMultiplying with the transposed attention map A to obtain the corrected features helps to obtain useful local context; finally, the corrected features are deformed and feature interconnection is achieved through 1×1 convolution to obtain the interconnected features Z A , the interconnection feature Z A Perform residual connection with the input image Z to generate the first local-global feature Z′; wherein the generated interconnected feature Z A The process expression of the first local-global feature Z′ is as follows:

[0100]

[0101] Z′=Z+Z A ;

[0102] Among them, Z A is the interconnected feature after convolution self-attention correction, represents a convolution operation with a kernel of 1, κ r is the deformation operation, V r is obtained by deformation operation on V, and Z′ is the first local-global feature output after normalization, convolution self-attention and residual connection;

[0103] Then, the first local-global feature Z′ generates a second local-global feature Z″ through layer normalization operation, enhanced forward transfer network and residual connection; in addition, a 3×3 deep convolutional coding is used in the enhanced forward transfer network to enhance the local context; wherein, the process expression of generating the second local-global feature Z″ is as follows:

[0104]

[0105] Among them, Z″ is the second local-global feature generated after layer normalization operation, enhanced forward transmission network, and residual connection, κ GELU is the Gaussian Error Linear Unit (GELU) activation function;

[0106] Finally, the second local-global feature Z″ generated by the MDCT module and the input image operated by the residual-MDconv module are residually connected to generate the local-global feature of the nth resolution level RMDCT module of the bidirectional adaptive local-global feature encoder.

[0107] With the collaboration of the residual-MDconv module and MDCT, the RMDCT module can obtain the global correlation of local features and regions, which helps to reconstruct high-quality remote sensing images.

[0108] and Figure 1 Corresponding to the method, refer to Figure 7, an embodiment of the present invention provides a remote sensing image feature enhancement system, comprising:

[0109] at least one processor;

[0110] at least one memory for storing at least one program;

[0111] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0112] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0113] In addition, an embodiment of the present invention further discloses a computer program product or a computer program, which is stored in a computer-readable storage medium. A processor of a computer device can read the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above method. Similarly, the contents of the above method embodiment are all applicable to the storage medium embodiment, and the functions specifically implemented by the storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method embodiment.

[0114] It will be appreciated by those skilled in the art that all or some of the methods disclosed above, the system can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0115] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A method for enhancing the features of a remote sensing image, characterized in that: The method comprises the following steps: S100, constructing a feature enhancement model, wherein the feature enhancement model includes a bidirectional adaptive local-global feature encoder, a decoder, and a reconstruction unit; wherein the bidirectional adaptive local-global feature encoder is used to extract multi-resolution local-global features of a fine-resolution image and multi-resolution local-global features of a coarse-resolution image; S200, obtaining an image to be processed, inputting the image to be processed into the bidirectional adaptive local-global feature encoder, extracting shallow local features and local-global features of multiple resolutions; wherein the image to be processed includes a fine resolution image and a coarse resolution image; S300, fusing the shallow local features and the local-global features of multiple resolutions step by step from low resolution to high resolution via a decoder to generate fused features of an ideal resolution, and then generating a high-resolution image from the fused features of the ideal resolution via a reconstruction unit; The bidirectional adaptive local-global feature encoder comprises two encoder branches, each of which comprises an MDconv module, an RMDCT module corresponding to N+1 resolution levels, and a sampling module; In S200, the step of inputting the image to be processed into the bidirectional adaptive local-global feature encoder to extract shallow local features and local-global features of multiple resolutions includes: S210, inputting the image to be processed into the MDconv module to extract shallow local features of the image to be processed; S220, extracting local-global features of N+1 resolution levels from the image to be processed through RMDCT modules of N+1 resolution levels and N sampling operations; In S210, the step of inputting the image to be processed into the MDconv module and extracting shallow local features of the image to be processed includes: The shallow local features of the image to be processed are extracted by the MDconv module, and the mathematical expression is as follows: Among them, SF0 and SC N Represent the shallow local features of the fine-resolution image and the coarse-resolution image, respectively. represents a dilated convolution operation with a kernel size of 3 and a dilation rate of i, FI i and CI i Respectively indicated by The features of the extracted fine-resolution image and coarse-resolution image, represents the convolution operation with a convolution kernel of 1, κ lrelu (·) represents the leaky linear rectification function, κ concat (·) represents the splicing operation along the channel direction, FI represents the fine-resolution image, and CI represents the coarse-resolution image; In S220, extracting local-global features of N+1 resolution levels from the image to be processed through RMDCT modules of N+1 resolution levels and N sampling operations includes: The local-global features of N+1 resolution levels are extracted from the image to be processed by the following formula: F0=κ RMDCT (SF0); F n =κ RMDCT (DF n-1 )n=1,…,N; C N =κ RMDCT (SC N ); C n-1 =k RMDCT (UC n )n=1,…,N; Among them, κ RMDCT (·) represents the function processed by the RMDCT module, F n represents the local-global features extracted from the fine-resolution image by the RMDCT module of the nth resolution level of an encoder branch of the bidirectional adaptive local-global feature encoder, C n represents the local-global features extracted from the coarse-resolution image by the RMDCT module at the nth resolution level of another encoder branch, DF n Indicates F n The local-global downsampling features obtained by downsampling operation, κ unshuffle (·) represents the downsampling operation, UC n Indicates C n The local-global upsampling feature obtained by upsampling operation, κ shuffle (·) represents upsampling operation, represents a convolution operation with a kernel of 3, where n = 0, 1, ..., N; The RMDCT module includes a residual-MDconv module and an MDCT module, wherein the MDCT module is a multi-head deep convolution Transformer module, and the residual-MDconv module and the MDCT module constitute a residual structure; the residual-MDconv module adopts residual connection to avoid gradient disappearance, and is used to extract and enhance local information; the MDCT module includes a layer normalization processing module, a multi-head deep convolution attention module, an enhanced forward transfer network and a residual connection; The RMDCT module processing process includes: S510, performing layer normalization operation on the input image Z of the MDCT module to obtain normalized feature Z LN , for the normalized feature Z LN A 1×1 convolutional layer aggregates pixel-level cross-channel context and a 3×3 deep convolutional coding channel separates spatial context and generates queries, keys, and values; wherein the input image Z of the MDCT module of the first resolution level is the shallow local feature, and the input image Z of the subsequent MDCT module is the local-global down-sampled feature or the local-global up-sampled feature output by the previous resolution level; The layer normalization operation and the process of generating queries, keys, and values ​​are expressed as follows: WITH LN =κ LN (WITH); Among them, κ LN is the layer normalization operation, Z and Z LN is the input image of the MDCT module and the normalized features after the input image layer is normalized, Q, K, and V are the generated query, key, and value, respectively. and Indicates that the mapping generates Q with a depth convolution operation of 3×3 kernel and a convolution operation with a kernel of 1; and Indicates that the mapping generates K with a depth convolution operation of 3×3 kernel and a convolution operation with a kernel of 1; and Indicates that the mapping generates V by a depthwise convolution operation with a kernel of 3×3 and a convolution operation with a kernel of 1; S520, transform the query Q and the key K to obtain Q r and K r , and perform dot product and Softmax operations to generate a transposed attention map A, and obtain the global attention weight of the correlation between the query Q and the key K; wherein, the process expression of generating the transposed attention map A is as follows: Among them, A is the transposed attention image, Q r , K r It is obtained by transforming Q and K respectively; is a learnable parameter used to adjust the amplitude; S530, transform the value V to obtain V r , the V r Multiplying with the transposed attention map A to obtain the corrected features helps to obtain useful local context; finally, the corrected features are deformed and feature interconnection is achieved through 1×1 convolution to obtain the interconnected features Z A , the interconnection feature Z A Perform residual connection with the input image Z to generate the first local-global feature Z′; wherein the generated interconnected feature Z A The process expression of the first local-global feature Z′ is as follows: Z′=Z+Z A ; Among them, Z A is the interconnected feature after convolution self-attention correction, represents a convolution operation with a kernel of 1, κ r is the deformation operation, V r is obtained by deformation operation on V, and Z′ is the first local-global feature output after normalization, convolution self-attention and residual connection; S540, the first local-global feature Z′ generates a second local-global feature Z″ through layer normalization operation, enhanced forward transfer network and residual connection; wherein the process expression of generating the second local-global feature Z″ is as follows: Among them, Z″ is the second local-global feature generated after layer normalization operation, enhanced forward transmission network, and residual connection, κ GELU is the Gaussian error linear unit activation function; S550, performing residual connection on the second local-global feature Z″ generated by the MDCT module and the input image operated by the residual-MDconv module to generate the local-global feature of the nth resolution level RMDCT module in the bidirectional adaptive local-global feature encoder.

2. A remote sensing image feature enhancement system, characterized in that: The system comprises: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the feature enhancement method for remote sensing images as claimed in claim 1.

Images