A remote sensing image super-resolution reconstruction method based on weight sharing multi-scale multi-expert decision

Abstract

The application discloses a remote sensing image super-resolution reconstruction method based on weight sharing and multi-scale multi-expert decision-making, which comprises image rotation, super-resolution expert network, super-resolution result of the rotated super-resolution expert network, average of four super-resolution expert network super-resolution results, feature vector splicing and fusion expert network; the image rotation comprises the following steps: after inputting an original low-resolution remote sensing image, aligning and rotating 0 degrees, 90 degrees, 180 degrees and 270 degrees, as shown in the following formula: I0=R O t0(I), I 90 =R O t 90 (I), I 180 =R O t 180 (I), I 270 =R O t 270 (I), wherein I is the original low-resolution remote sensing image. The super-resolution reconstruction system can solve the problems of the limitation of single expert decision-making and the imbalance of model parameter quantity and effect, so that the super-resolution reconstruction mechanism of multi-expert decision-making and the multi-scale weight sharing convolution module are introduced in the super-resolution reconstruction process, and more abundant detailed texture information can be obtained by using the mechanism of multi-expert decision-making.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image technology, and specifically to a weight-sharing, multi-scale, multi-expert decision-making method for super-resolution reconstruction of remote sensing images. Background Technology

[0002] Remote sensing imagery, as a common technique for observing the Earth's surface, is widely used in various technical fields such as military reconnaissance, disaster detection, climate early warning, and agriculture. However, due to the resolution limitations of the optical sensors carried by flying platforms, the acquired remote sensing images suffer from limited resolution, leading to problems such as difficulty in identifying surface features or poor display quality. Traditional super-resolution upscaling techniques typically use bilinear interpolation to increase the resolution of remote sensing images, but this often suffers from severe jagged edges and blurring.

[0003] Existing technologies mostly employ deep learning methods for super-resolution reconstruction, which can effectively solve the problems caused by bilinear interpolation and obtain better super-resolution reconstruction results.

[0004] Deep learning methods typically employ a single-branch network structure, considering only a single expert's decision-making to achieve super-resolution reconstruction. While this effectively addresses the jagged edges problem in traditional bilinear interpolation, it fails to adequately recover detailed image information. Furthermore, the quality of existing deep learning-based super-resolution reconstruction results largely depends on model depth; increasing model depth leads to a massive number of parameters, significantly hindering model training and inference, and making deployment on devices difficult due to performance limitations. Therefore, there is an urgent need to design a weight-sharing, multi-scale, multi-expert decision-making method for remote sensing image super-resolution reconstruction to address these issues. Summary of the Invention

[0005] The purpose of this invention is to provide a weight-sharing, multi-scale, multi-expert decision-making method for super-resolution reconstruction of remote sensing images, in order to overcome the above-mentioned shortcomings in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This includes image rotation, super-resolution expert network, super-resolution results of rotated super-resolution expert network, calculating the average of super-resolution results of four super-resolution expert networks, feature vector concatenation, and fusion expert network.

[0008] The image rotation includes aligning and rotating the original low-resolution remote sensing image by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, as shown below: I0 = R O t0(I), I 90 =R O t 90 (I), I 180= R O t 180 (I), I 270 = R O t 270 (I), wherein I is an original low-resolution remote sensing image, R O t0(·), R O t 90 (·), R O t 180 (·), R O t 270 (·) are respectively 0-degree, 90-degree, 180-degree and 270-degree clockwise rotation operations of the image, I O , I 90 , I 180 , I 270 are respectively 0-degree, 90-degree, 180-degree and 270-degree clockwise rotation result images.

[0009] The super-resolution reconstruction system can introduce a super-resolution reconstruction mechanism of multi-expert decision and a multi-scale weight sharing convolution module in the super-resolution reconstruction process to solve the problems of limitations of single expert decision and imbalance of model parameter quantity and effect, so that more rich detail texture information can be obtained by using the mechanism of multi-expert decision, and the model performance is effectively increased without increasing too many model parameter quantities by using the multi-scale weight sharing convolution module.

[0010] The super-resolution expert network includes a super-resolution reconstruction result, a shallow feature extraction, a multi-scale weight sharing convolution module, a convolution layer, an upward projection module and a super-resolution reconstruction module, the super-resolution reconstruction result includes inputting the images rotated by 0 degrees, 90 degrees, 180 degrees and 270 degrees into four super-resolution expert networks respectively to obtain four different super-resolution reconstruction results. The network structures of the four super-resolution expert networks are the same, only the inputs are different, so that the outputs are different, the rotated images are input into the four super-resolution expert networks to obtain the corresponding super-resolution reconstruction results, as follows: U0 = ExpertSR(I0), U 90 = ExpertSR(I 90 ), U 180 = ExpertSR(I 180 ), U 270 = ExpertSR(I 270 ), wherein U0, U 90 , U 180 , U 270The super-resolution expert network super-resolution reconstruction result images rotated by 0 degrees, 90 degrees, 180 degrees and 270 degrees, respectively, ExpertSR is the super-resolution expert network, next, the super-resolution expert network is introduced in detail, the network structures of the four super-resolution expert networks are the same, therefore, only the super-resolution expert network rotated by 0 degrees is introduced.

[0011] The shallow feature extraction includes shallow feature extraction on the low-resolution remote sensing image rotated by 0 degrees, as follows: F S = conv(conv(I0)), wherein, F S is the extracted shallow feature vector, and conv(·) is a 3*3 convolution operation.

[0012] The multi-scale weight sharing convolution module includes inputting the extracted shallow feature vector into three multi-scale weight sharing convolution modules arranged in sequence, to obtain a deep feature vector with sufficient expression of feature information, as follows: F d = MSCB(MSCB(MSCB(F s ))), wherein, F d is the extracted deep feature vector, MSCB(·) is a multi-scale weight sharing convolution module, specifically, the multi-scale weight sharing convolution module MSCB includes two sub-modules (SCB1, SCB2), the weights of three basic modules (CB1, CB2, CB3) in each sub-module are shared, and the relationship between the modules is as follows: MSCB(X) = SCB2(SCB1(X)), SCB(X) = X + CB3(X + CB2(x + CB1(X))), X is a placeholder for a feature vector, and is introduced to describe the feature relationship between the modules. SCB1 and SCB2 are two sub-modules in the multi-scale weight sharing convolution module MSCB, the weights of three basic modules (CB1, CB2, CB3) in SCB1 are shared, and the weights of three basic modules (CB1, CB2, CB3) in SCB2 are shared. The network structures of the three basic modules (CB1, CB2, CB3) are the same, and only the basic module CB1 is expanded and introduced in detail, two different scale convolutions are performed on the feature vector input into the basic module, as follows: X2 = conv1(X1), X3 = conv2(X1), wherein, X1 is the feature vector input into the basic module CB1, X2 and X3 are convolution results obtained by 3*3 and 5*5 convolution operations, respectively; conv1(·) is a 3*3 convolution operation, and conv2(·) is a 5*5 convolution operation; the two convolution results are feature spliced, and then a 5*5 convolution operation is performed, as follows: X c = (X2, X3), X4 = conv2(X c ), wherein, X cThe vector after splicing the feature vectors X2 and X3, and (·, ·) is a feature splicing operation; X4 is a convolution result obtained by a 5*5 convolution operation, and conv2(·) is a 5*5 convolution operation. Finally, the feature vector X4 is subjected to a 1*1 convolution operation to obtain the output result of the module CB1, as follows: X5 = conv3(X4), X5 is a convolution result obtained by a 1*1 convolution operation, and conv3(·) is a 1*1 convolution operation.

[0013] The convolution layer includes three sequentially arranged multi-scale weight sharing convolution modules, to obtain a deep feature vector in which feature information is fully expressed, and the deep feature vector is input into two consecutive 3*3 convolution layers, as follows: Wherein, F d is a deep feature vector output by the multi-scale weight sharing convolution module, is a convolution result obtained by two consecutive 5*5 convolution operations, and conv(·) is a 3*3 convolution operation.

[0014] The up-projection module is at least used to complete the upsampling operation of the above output deep feature vector. The up-projection module includes upsampling the feature vector to a resolution size after sufficient reconstruction, performing 3*3 convolution on the deep feature vector output in the upsampling process, and then using an Unpooling function to complete the upsampling operation, as follows: Wherein, is a deep feature vector after upsampling, Unpooling(·) is an nn.MaxUnpool2d function in pythorch, and conv(·) is a 3*3 convolution operation, which is used to finely process the result of upsampling to obtain a more accurate upsampling result, as follows: Wherein, is a final deep feature vector output by the up-projection module, and conv(·) is a 3*3 convolution operation.

[0015] The super-resolution reconstruction module includes performing bilinear interpolation on the original low-resolution remote sensing image, and then adding a learned residual vector to obtain a super-resolution reconstruction result output by the super-resolution expert network, as follows: Wherein, U0 is a super-resolution reconstruction result image of the super-resolution expert network rotated by 0 degrees, BIC(·) is a bilinear interpolation operation, and conv(·) is a 3*3 convolution operation.

[0016] The super-resolution result of the rotation super-resolution expert network includes rotating the super-resolution reconstruction result image of the super-resolution expert network to the original direction, as follows: E0 = IR O t0(U0), E 90 = IR Ot 90 (U 90 ), E 180 = IR O t 180 (U 180 ), E 270 = IR O t 270 (U 270 ), wherein U0, U 90 , U 180 , U 270 are respectively super-resolution result images of the super-resolution expert network with input of rotation 0 degrees, 90 degrees, 180 degrees, and 270 degrees low-resolution images, IR O t0(·), IR O t 90 (·), IR O t 180 (·), IR O t 270 (·) are respectively images rotated 0 degrees, 90 degrees, 180 degrees, and 270 degrees in the counterclockwise direction, E0, E 90 , E 180 , E 270 are respectively result images of U0, U 90 , U 180 , U 270 rotated 0 degrees, 90 degrees, 180 degrees, and 270 degrees in the counterclockwise direction.

[0017] The four super-resolution expert network super-resolution results include rotating the four super-resolution results back to the original direction to obtain the average, as follows: E = (E0 + E 90 + E 180 + E 270 ) / 4, wherein E is the average obtained, E0, E 90 , E 180 , E 270 are respectively result images of U0, U 90 , U 180 , U 270 rotated 0 degrees, 90 degrees, 180 degrees, and 270 degrees in the counterclockwise direction, and the feature vector splicing includes performing feature splicing operation on the four super-resolution results rotated back to the original direction and the average obtained, as follows: E c = (E, E0, E 90 , E 180 , E 270 ).

[0018] The fusion expert network includes the feature vector E cThe input fusion expert network is fully fused to obtain a residual vector required by super-resolution reconstruction, the fusion expert network is similar to the super-resolution expert network in structure, only two parts are different: one is the upward projection module, the fusion expert network does not contain the module, and the super-resolution expert network contains; two is the residual network superimposed vector, the super-resolution expert network superimposes the bilinear interpolation result of the initial low-resolution remote sensing image, and the fusion expert network superimposes the average E of the four super-resolution expert network super-resolution results, first, the feature vector E c Shallow feature extraction is performed as follows: D s = covn(covco(E c )), wherein D s is the extracted shallow feature vector, and conv(·) is a 3*3 convolution operation. s Three sequentially arranged multi-scale weight sharing convolution modules are inputted to obtain a deep feature vector fully expressing feature information, as follows: D d = MSCB(MSCB(MSCB(D s ))), wherein D d is the extracted deep feature vector, and MSCB(·) is a multi-scale weight sharing convolution module. d Two consecutive 3*3 convolution layers are inputted, as follows: wherein D d is the deep feature vector output by the three sequentially arranged multi-scale weight sharing convolution modules, is a convolution result obtained by two consecutive 3*3 convolution operations, and conv(·) is a 3*3 convolution operation. The average E of the four super-resolution expert network super-resolution results is added to obtain the super-resolution reconstruction result output by the method, as follows: wherein OUT is the super-resolution reconstruction result image output by the method.

[0019] In the above technical solution, the weight sharing multi-scale multi-expert decision remote sensing image super-resolution reconstruction method provided by the application has the following beneficial effects:

[0020] The super-resolution reconstruction system can solve the problems of limitations of single expert decision and imbalance of model parameter quantity and effect, thereby introducing a multi-expert decision super-resolution reconstruction mechanism and a multi-scale weight sharing convolution module in the super-resolution reconstruction process, which can obtain more rich detail texture information by using the multi-expert decision mechanism, and effectively increase the model performance without increasing too many model parameter quantities by using the multi-scale weight sharing convolution module. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0022] Figure 1 The flowchart provided by the embodiment of the present application is a weight-sharing multi-scale multi-expert decision remote sensing image super-resolution reconstruction method.

[0023] Figure 2 The super-resolution expert network flowchart provided by the embodiment of the present application is a weight-sharing multi-scale multi-expert decision remote sensing image super-resolution reconstruction method.

[0024] Figure 3 The fusion expert network flowchart provided by the embodiment of the present application is a weight-sharing multi-scale multi-expert decision remote sensing image super-resolution reconstruction method.

[0025] Figure 4 The multi-scale weight-sharing convolution module flowchart provided by the embodiment of the present application is a weight-sharing multi-scale multi-expert decision remote sensing image super-resolution reconstruction method.

[0026] Figure 5 The basic module flowchart provided by the embodiment of the present application is a weight-sharing multi-scale multi-expert decision remote sensing image super-resolution reconstruction method.

[0027] Figure 6 The upward projection module flowchart provided by the embodiment of the present application is a weight-sharing multi-scale multi-expert decision remote sensing image super-resolution reconstruction method.

[0028] Figure 7 The result comparison diagram of the present application. DETAILED DESCRIPTION

[0029] In order to make those skilled in the art better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings.

[0030] As shown in Figures 1-6 the present application provides a weight-sharing multi-scale multi-expert decision remote sensing image super-resolution reconstruction method.

[0031] including image rotation, super-resolution expert network, super-resolution result of rotated super-resolution expert network, mean value of four super-resolution expert network super-resolution results, feature vector splicing and fusion expert network;

[0032] Image rotation includes aligning and rotating 0 degrees, 90 degrees, 180 degrees, and 270 degrees after inputting the original low-resolution remote sensing image, as shown below: I0=R O t0(I), I 90 =R O t 90 (I), I 180 =R O t 180 (I), I 270 =R O t 270 (I), wherein I is an original low-resolution remote sensing image, R O t0(·), R O t 90 (·), R O t 180 (·), R O t 270 (·) are respectively rotating the image 0 degrees, 90 degrees, 180 degrees, and 270 degrees in the clockwise direction, I0, I 90 , I 180 , I 270 are respectively the result images rotated 0 degrees, 90 degrees, 180 degrees, and 270 degrees in the clockwise direction.

[0033] The super-resolution reconstruction system can introduce a multi-expert decision super-resolution reconstruction mechanism and a multi-scale weight sharing convolution module in the super-resolution reconstruction process to solve the limitations of single expert decision and the imbalance of model parameter quantity and effect, so that more rich detail texture information can be obtained by using the multi-expert decision mechanism, and the model performance is effectively increased without increasing too many model parameters by using the multi-scale weight sharing convolution module.

[0034] The super-resolution reconstruction system can use the multi-expert decision super-resolution reconstruction mechanism to rotate the original image in four directions, then use four super-resolution expert networks to output the corresponding super-resolution reconstruction results respectively, and then use a fusion expert network to fuse the super-resolution results of the four super-resolution expert networks to obtain more rich and clear super-resolution reconstruction results. In addition, the multi-scale weight sharing convolution module is introduced into the basic framework in the expert network, the weights in the basic modules in the same sub-module are shared, which increases the depth of the network and reduces the parameter quantity of the model itself, and can fully extract deep features. In addition, different scale convolution modules are used in the basic modules used, and the feature information of each scale in the image is fully extracted.

[0035] Referring to Figure 2The super-resolution expert network of the embodiment includes a super-resolution reconstruction result, a shallow feature extraction, a multi-scale weight sharing convolution module, a convolution layer, an upward projection module and a super-resolution reconstruction module. The super-resolution reconstruction result includes inputting images rotated by 0 degrees, 90 degrees, 180 degrees and 270 degrees into four super-resolution expert networks respectively to obtain four different super-resolution reconstruction results. The four super-resolution expert networks have the same network structure, but different inputs, resulting in different outputs. The images after rotation are input into the four super-resolution expert networks to obtain the corresponding super-resolution reconstruction results, as follows: U0=ExpertSR(I0), U 90 =ExpertSR(I 90 ), U 180 =ExpertSR(I 180 ), U 270 =ExpertSR(I 270 ), wherein U0, U 90 , U 180 , U 270 are super-resolution expert network super-resolution reconstruction result images rotated by 0 degrees, 90 degrees, 180 degrees and 270 degrees respectively, and ExpertSR is a super-resolution expert network. Next, the super-resolution expert network is described in detail. The four super-resolution expert networks have the same network structure, so only the super-resolution expert network rotated by 0 degrees is described.

[0036] The shallow feature extraction includes shallow feature extraction of the low-resolution remote sensing image rotated by 0 degrees, as follows: F S =conv(conv(I0)), wherein F S is a shallow feature vector extracted, and conv(·) is a 3*3 convolution operation.

[0037] Referring to Figure 4 , the multi-scale weight sharing convolution module of the embodiment includes inputting the extracted shallow feature vector into three multi-scale weight sharing convolution modules arranged in sequence to obtain a deep feature vector with sufficient feature information, as follows: F d =MSCB(MSCB(MSCB(F s ))), wherein F dTo extract the deep feature vector, MSCB(·) is a multi-scale weight sharing convolution module. Specifically, the multi-scale weight sharing convolution module MSCB includes two sub-modules (SCB1, SCB2), and the weights of three basic modules (CB1, CB2, CB3) in each sub-module are shared. The relationship between the modules is as follows: MSCB(X) = SCB2(SCB1(X)), SCB(X) = X + CB3(X + CB2(x + CB1(X))), X is a placeholder for a feature vector, which is introduced to describe the feature relationship between modules. SCB1 and SCB2 are two sub-modules in the multi-scale weight sharing convolution module MSCB, and the weights of three basic modules (CB1, CB2, CB3) in SCB1 are shared, and the weights of three basic modules (CB1, CB2, CB3) in SCB2 are shared. The network structures of the three basic modules (CB1, CB2, CB3) are the same, and only one basic module CB1 will be expanded in detail. Two different scale convolutions are performed on the feature vector input into the basic module, as follows: X2 = conv1(X1), X3 = conv2(X1), where X1 is the feature vector input into the basic module CB1, X2 and X3 are the convolution results obtained by 3*3 and 5*5 convolution operations respectively; conv1(·) is a 3*3 convolution operation, and conv2(·) is a 5*5 convolution operation. The two convolution results are feature spliced and then subjected to a 5*5 convolution operation, as follows: X c = (X2, X3), X4 = conv2(X c ), where X c is the vector after splicing the feature vectors X2 and X3, (·, ·) is a feature splicing operation; X4 is the convolution result obtained by a 5*5 convolution operation, and conv2(·) is a 5*5 convolution operation. Finally, the feature vector X4 is subjected to a 1*1 convolution operation to obtain the output result of the module CB1, as follows: X5 = conv3(X4), X5 is the convolution result obtained by a 1*1 convolution operation, and conv3(·) is a 1*1 convolution operation.

[0038] The convolution layer includes three multi-scale weight sharing convolution modules arranged in sequence, and a deep feature vector with sufficient expression of feature information is obtained, which is input into two consecutive 3*3 convolution layers, as follows: where F d is a deep feature vector output by a multi-scale weight sharing convolution module, are convolution results obtained by two consecutive 5*5 convolution operations, and conv(·) is a 3*3 convolution operation.

[0039] The up-projection module is used at least for completing the up-sampling operation of the deep feature vector output above, and includes up-sampling the feature vector to the resolution size after sufficient reconstruction, performing 3*3 convolution on the deep feature vector output above, and then using an Unpooling function to complete the up-sampling operation, as follows: wherein, is the deep feature vector after up-sampling, Unpooling(·) is an nn.MaxUnpool2d function in pythorch, and conv(·) is a 3*3 convolution operation for fine processing of the up-sampled result to obtain a more accurate up-sampled result, as follows: wherein, is the final deep feature vector output by the up-projection module, and conv(·) is a 3*3 convolution operation.

[0040] The super-resolution reconstruction module includes performing bilinear interpolation on the original low-resolution remote sensing image, and then adding the learned residual vector to obtain the super-resolution reconstruction result output by the super-resolution expert network, as follows: wherein, U0 is the super-resolution reconstruction result image of the super-resolution expert network rotated by 0 degrees, BIC(·) is a bilinear interpolation operation, and conv(·) is a 3*3 convolution operation.

[0041] The rotation super-resolution expert network super-resolution result includes rotating the super-resolution expert network super-resolution reconstruction result image to the original direction, as follows: E0=IR O t0(U0), E 90 =IR O t 90 (U 90 ), E 180 =IR O t 180 (U 180 ), E 270 =IR O t 270 (U 270 ), wherein, U0, U 90 , U 180 , U 270 are super-resolution expert network super-resolution reconstruction result images input with low-resolution images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively, IR O t0(·), IR O t 90 (·), IR O t 180 (·), IR O t 270(·) are respectively the images rotated by 0, 90, 180, 270 degrees in the counterclockwise direction, E0, E 90 , E 180 , E 270 are respectively the images rotated by 0, 90, 180, 270 degrees in the counterclockwise direction. 90 , U 180 , U 270 .

[0042] The four super-resolution expert network super-resolution result mean value includes four super-resolution results rotated back to the original direction, and the mean value is calculated as follows: E = (E0+E 90 +E 180 +E 270 ) / 4, wherein E is the calculated mean value, E0, E 90 , E 180 , E 270 are respectively the images rotated by 0, 90, 180, 270 degrees in the counterclockwise direction. 90 , U 180 , U 270 . c The feature vector splicing includes splicing the four super-resolution results rotated back to the original direction and the calculated mean value after the mean value is calculated, and the splicing operation is as follows: E 90 = (E, E0, E 180 , E 270 ).

[0043] Referring to Figure 3 , the fusion expert network of the embodiment includes inputting the feature vector E c after the feature splicing operation into the fusion expert network, fully fusing it to obtain the residual error vector required for super-resolution reconstruction. The fusion expert network is similar in structure to the super-resolution expert network, but there are some differences in two parts: one is the upward projection module, and the fusion expert network does not contain the module, while the super-resolution expert network contains it; the second is the vector stacked by the residual error network, and the super-resolution expert network stacks the bilinear interpolation result of the initial low-resolution remote sensing image, while the fusion expert network stacks the mean value E of the four super-resolution expert network super-resolution results. First, the spliced feature vector E c is subjected to shallow feature extraction as follows: D s = covn(covn(E c )), wherein D s is the extracted shallow feature vector, and conv(·) is a 3*3 size convolution operation. The extracted shallow feature vector D s is input into three multi-scale weight shared convolution modules arranged in sequence to obtain a deep feature vector fully expressing feature information as follows: D dMSCB(MSCB(MSCB(D s ))),wherein D d is a deep feature vector extracted, MSCB(·) is a multi-scale weight sharing convolution module, then the deep feature vector F d is input into two successive 3*3 convolution layers, as follows: wherein D d is a deep feature vector output by three successive multi-scale weight sharing convolution modules, is a convolution result obtained by two successive 3*3 convolution operations, conv(·) is a 3*3 convolution operation, finally, the mean E of four super-resolution expert network super-resolution results is added to to obtain a super-resolution reconstruction result output by the method, as follows: wherein OUT is a super-resolution reconstruction result image output by the method.

[0044] Referring to Figure 7 , the bilinear interpolation result image is (a) and the super-resolution result image is (b), it can be seen that the definition of the super-resolution result image (b) is obviously better than that of the bilinear interpolation result image (a).

[0045] Working principle: firstly, the original low-resolution remote sensing image is respectively rotated clockwise by 0 degrees, 90 degrees, 180 degrees and 270 degrees, and then input into four super-resolution expert networks to obtain four different super-resolution reconstruction results; secondly, the super-resolution reconstruction results rotated clockwise by 0 degrees, 90 degrees, 180 degrees and 270 degrees are respectively rotated counterclockwise by 0 degrees, 90 degrees, 180 degrees and 270 degrees to rotate them back; then, the mean of the four rotated super-resolution reconstruction results is calculated; the four rotated super-resolution reconstruction results and the calculated mean are input into a fusion expert network after feature splicing, and the residual vector required for super-resolution reconstruction is obtained by fully fusing them; finally, the mean of the super-resolution reconstruction results is added to the residual vector to obtain the final super-resolution reconstruction result.

[0046] The above only describes certain exemplary embodiments of the present application in a descriptive manner, without doubt, for ordinary skilled in the art, the described embodiments can be modified in various ways without departing from the spirit and scope of the present application. Therefore, the above figures and descriptions are illustrative in nature and should not be understood as limiting the scope of protection of the claims of the present application.

Claims

1. A weight-sharing, multi-scale, multi-expert decision-making method for super-resolution reconstruction of remote sensing images, characterized in that, This includes image rotation, super-resolution expert network, super-resolution results of rotated super-resolution expert network, calculating the average of super-resolution results of four super-resolution expert networks, feature vector concatenation, and fusion expert network; The image rotation includes aligning and rotating the original low-resolution remote sensing image by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, as shown below: , , , , in, The original low-resolution remote sensing image. , , , These operations rotate the image clockwise by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. , , , The images are rotated clockwise by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. The super-resolution expert network includes super-resolution reconstruction results, shallow feature extraction, a multi-scale weight-sharing convolution module, convolutional layers, an up-projection module, and a super-resolution reconstruction module. The super-resolution reconstruction results are obtained by inputting the rotated images (0 degrees, 90 degrees, 180 degrees, and 270 degrees) into four super-resolution expert networks to obtain four different super-resolution reconstruction results. The four super-resolution expert networks use the same network structure, but differ in their inputs, resulting in different outputs. The rotated image is input into the four super-resolution expert networks to obtain the corresponding super-resolution reconstruction results, as shown below: , , , ,in , , , The images show the super-resolution reconstruction results from the super-resolution expert network after rotations of 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. For super-resolution expert networks.

2. The method for super-resolution reconstruction of remote sensing images with weight sharing, multi-scale, and multi-expert decision-making, as described in claim 1, is characterized in that... The shallow feature extraction includes performing shallow feature extraction on low-resolution remote sensing images rotated by 0 degrees, as shown below: ,in, To extract shallow feature vectors, This is a 3x3 convolution operation.

3. The method for super-resolution reconstruction of remote sensing images with weight sharing, multi-scale, and multi-expert decision-making, as described in claim 2, is characterized in that... The multi-scale weight-shared convolution module includes inputting the extracted shallow feature vector into three sequentially arranged multi-scale weight-shared convolution modules to obtain a deep feature vector that fully expresses the feature information. As shown below: ,in, To extract deep feature vectors, This is a multi-scale weight-sharing convolution module. Specifically, it's a multi-scale weight-sharing convolution module. Includes two sub-modules ( , ), the three basic modules in each submodule ( , , The weights of the modules are shared, and the relationships between the modules are as follows: = , , The eigenvector is introduced to describe the feature relationships between modules; and Multi-scale weight-sharing convolutional module Two sub-modules in The three basic modules ( , , The weights of ) are shared. The three basic modules ( , , The weights of the components are shared. Three basic modules ( , , The network structure is the same as that of the other two, and we will only use one of the basic modules here. To elaborate further, two convolutions of different scales are performed on the feature vectors of the input base module, as shown below: , ,in, For input basic modules eigenvectors, , The results are convolutional operations of sizes 3*3 and 5*5, respectively. For a 3x3 convolution operation, For a 5x5 convolution operation, the features of the two convolution results are concatenated, and then a 5x5 convolution operation is performed, as shown below: , ,in, Feature vector The concatenated vector, ( () represents the feature splicing operation; This is the convolution result obtained from a 5x5 convolution operation. A convolution operation of size 5x5; Finally, the feature vector This module is obtained after a 1x1 convolution operation. The output is shown below: , This is the convolution result obtained from a 1x1 convolution operation. A convolution operation of size 1*1.

4. The method for super-resolution reconstruction of remote sensing images with weight sharing, multi-scale, and multi-expert decision-making, as described in claim 3, is characterized in that... The convolutional layer comprises three sequentially arranged multi-scale weight-shared convolutional modules to obtain a deep feature vector that fully expresses the feature information. This vector is then input into two consecutive 3x3 convolutional layers, as shown below: ,in, This is the deep feature vector output by the multi-scale weight-sharing convolution module. This is the convolution result obtained from two consecutive 5x5 convolution operations. This is a 3x3 convolution operation.

5. The method for super-resolution reconstruction of remote sensing images with weight sharing, multi-scale, and multi-expert decision-making, as described in claim 4, is characterized in that... The upprojection module is used at least to perform upsampling operations on the deep feature vectors output above. The upprojection module includes upsampling the feature vectors to the fully reconstructed resolution size, performing a 3*3 convolution on the deep feature vectors output from the above process, and then using the Unpooling function to complete the upsampling operation, as shown below: ,in, This is the upsampled deep feature vector. ( () refers to the nn.MaxUnpool2d function in Pythorch. For a 3x3 convolution operation, the upsampling result is refined to obtain a more accurate upsampling result, as shown below: ,in, This is the final deep feature vector output by the upward projection module. This is a 3x3 convolution operation.

6. The method for super-resolution reconstruction of remote sensing images with weight sharing, multi-scale, and multi-expert decision-making, as described in claim 5, is characterized in that... The super-resolution reconstruction module includes bilinear interpolation of the original low-resolution remote sensing image, followed by adding the learned residual vector to obtain the super-resolution reconstruction result output by the super-resolution expert network, as shown below: ,in, The image shows the super-resolution reconstruction result from a super-resolution expert network rotated by 0 degrees. This is a bilinear interpolation operation. This is a 3x3 convolution operation.

7. The method for super-resolution reconstruction of remote sensing images with weight sharing, multi-scale, and multi-expert decision-making, as described in claim 6, is characterized in that... The super-resolution result of the rotated super-resolution expert network includes rotating the super-resolution reconstruction result image of the super-resolution expert network back to its original orientation, as shown below: , , , ,in, , , , The images shown are the super-resolution reconstruction results from a super-resolution expert network, where the input images are low-resolution images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees. , , , These operations rotate the image counterclockwise by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. , , , They are respectively for , , , The resulting images after rotating counterclockwise by 0 degrees, 90 degrees, 180 degrees, and 270 degrees.

8. The method for super-resolution reconstruction of remote sensing images with weight sharing, multi-scale, and multi-expert decision-making, as described in claim 7, is characterized in that... The calculation of the average super-resolution results of the four super-resolution expert networks includes averaging the four super-resolution results rotated back to the original direction, as shown below: / 4, of which, To obtain the mean, , , , They are respectively for , , , The resulting images are rotated counterclockwise by 0 degrees, 90 degrees, 180 degrees, and 270 degrees. The feature vector concatenation includes, after calculating the mean, performing a feature concatenation operation on the four super-resolution results rotated back to the original direction and the calculated mean, as shown below: , , , .

9. The method for super-resolution reconstruction of remote sensing images with weight sharing, multi-scale, and multi-expert decision-making, as described in claim 8, is characterized in that... The fusion expert network includes feature vectors obtained after feature concatenation operations. The input fusion expert network is fully fused to obtain the residual vector required for super-resolution reconstruction. The fusion expert network has a similar structure to the super-resolution expert network, except for two parts: First, there is the upward projection module, which is not included in the fusion expert network, but is included in the super-resolution expert network. Secondly, the vectors superimposed by the residual networks are different. The super-resolution expert network superimposes the bilinear interpolation results of the initial low-resolution remote sensing image, while the fusion expert network superimposes the average of the super-resolution results of the four super-resolution expert networks. ; First, the concatenated feature vectors Shallow feature extraction is performed as follows: ,in, To extract shallow feature vectors, A 3x3 convolution operation will extract the shallow feature vector. By inputting three sequentially arranged multi-scale weight-shared convolutional modules, we obtain a deep feature vector that fully expresses the feature information, as shown below: ,in, To extract deep feature vectors, For multi-scale weight-sharing convolutional modules, then, the deep feature vectors are... Input two consecutive 3x3 convolutional layers, as shown below: ,in, The deep feature vectors are output by three sequential multi-scale weight-shared convolutional modules. This is the convolution result obtained from two consecutive 3x3 convolution operations. The convolution operation is performed in 3x3 form. Finally, the super-resolution results of the four super-resolution expert networks are averaged. Plus The super-resolution reconstruction results obtained by this method are shown below: ,in, This is the super-resolution reconstruction result image output by this method.

Images