Weight-sharing multi-scale multi-expert decision remote sensing image super-division reconstruction method

Abstract

The invention discloses a weight-sharing multi-scale multi-expert decision remote sensing image super-division reconstruction method. The method comprises the steps of image rotation, super-division of an expert network, rotation of super-division results of the super-division expert network, solving of a mean value of the super-division results of the four super-division expert networks, feature vector splicing and fusion of the expert network. Image rotation comprises that after an original low-resolution remote sensing image is input, the image is aligned and rotated by 0 degree, 90 degrees, 180 degrees and 270 degrees, and the image rotation is shown as I0 = ROt0 (I), I90 = ROt90 (I), I180 = ROt180 (I) and I270 = ROt270 (I), and I is the original low-resolution remote sensing image. According to the invention, a super-resolution reconstruction mechanism of multi-expert decision and a multi-scale weight sharing convolution module are introduced in a super-resolution reconstruction process by aiming at the limitation of single expert decision and the imbalance of model parameter quantity and effect through the arranged super-resolution reconstruction system; and richer detail texture information can be obtained by utilizing a multi-expert decision-making mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing images, and in particular to a remote sensing image super-resolution reconstruction method with multi-scale and multi-expert decision-making based on weight sharing. Background Art

[0002] As a common technical means of observing the earth's surface, remote sensing images are widely used in military reconnaissance, disaster detection, climate warning, agricultural breeding and other technical fields. However, due to the resolution limitation of the optical sensor carried by the flight platform, the resolution of the obtained remote sensing images is limited, resulting in problems such as difficulty in identifying surface objects or poor display quality. Traditional super-resolution magnification technology usually uses bilinear interpolation to magnify the resolution of remote sensing images, but it often suffers from severe aliasing and blurring problems.

[0003] Existing technical means mostly use deep learning methods for super-resolution reconstruction, which can effectively solve the problems caused by bilinear interpolation and can obtain better super-resolution reconstruction results.

[0004] Since deep learning methods usually use a single-branch structure in the network structure, only a single expert is considered to make decisions to achieve super-resolution reconstruction. Although it can effectively solve the aliasing problem in traditional bilinear interpolation, it cannot restore the detailed information in the image well. In addition, the excellence of existing super-resolution reconstruction results based on deep learning depends on the depth of the model. However, the increase in model depth will lead to a huge amount of parameters, which will cause great obstacles to model training and reasoning, and it will be difficult to transplant when deployed on the device segment due to performance reasons. Therefore, it is urgent to design a weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method to solve the above-mentioned deficiencies in the prior art.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] Including image rotation, super-resolution expert network, rotating super-resolution expert network super-resolution results, finding the average of super-resolution results of four super-resolution expert networks, feature vector splicing and fusion expert network;

[0008] The image rotation includes inputting the original low-resolution remote sensing image, aligning and rotating it by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, as shown below: I0=R O t0(I),I 90 =R O t 90 (I),I 180=R O t 180 (I),I 270 =R O t 270 (I), where I is the original low-resolution remote sensing image, R O t0(·), R O t 90 (·), R O t 180 (·), R O t 270 (·) respectively rotate the image clockwise by 0, 90, 180, and 270 degrees, I O ,I 90 ,I 180 ,I 270 The resulting images are rotated 0, 90, 180, and 270 degrees clockwise respectively.

[0009] The super-resolution reconstruction system can address the limitations of single expert decision-making and the imbalance between model parameter quantity and effect, thereby introducing a super-resolution reconstruction mechanism of multiple expert decisions and a multi-scale weight sharing convolution module in the super-resolution reconstruction process. The multi-expert decision mechanism can be used to obtain richer detailed texture information, and the multi-scale weight sharing convolution module can be used to effectively increase model performance without increasing too many model parameters.

[0010] The super-resolution expert network includes super-resolution reconstruction results, shallow feature extraction, multi-scale weight sharing convolution module, convolution layer, upward projection module and super-resolution reconstruction module. The super-resolution reconstruction results include inputting images rotated by 0 degrees, 90 degrees, 180 degrees and 270 degrees into four super-resolution expert networks to obtain four different super-resolution reconstruction results. The four super-resolution expert networks use the same network structure, but the input is different, resulting in different outputs. After the rotated image is input into the four super-resolution expert networks, the corresponding super-resolution reconstruction results are obtained, as shown below: U0 = ExpertSR (I0), U 90 =ExpertSR(I 90 ), U 180 =ExpertSR(I 180 ), U 270 =ExpertSR(I 270 ), where U0, U 90 , U 180 , U 270The images are the super-resolution reconstruction results of the super-resolution expert network rotated by 0, 90, 180 and 270 degrees respectively. ExpertSR is the super-resolution expert network. Next, the super-resolution expert network is introduced in detail. The four super-resolution expert networks use the same network structure, so only the super-resolution expert network rotated by 0 degree is introduced.

[0011] The shallow feature extraction includes performing shallow feature extraction on a low-resolution remote sensing image rotated 0 degrees, as shown below: S =conv(conv(I0)), where F S To extract shallow feature vectors, conv(·) is a convolution operation of size 3*3.

[0012] The multi-scale weight sharing convolution module includes inputting the extracted shallow feature vector into three multi-scale weight sharing convolution modules arranged in sequence to obtain a deep feature vector that fully expresses the feature information, as shown below: d =MSCB(MSCB(MSCB(F s ))), where F d In order to extract the deep feature vector, MSCB(·) is a multi-scale weight sharing convolution module. Specifically, the multi-scale weight sharing convolution module MSCB includes two sub-modules (SCB1, SCB2). The weights of the three basic modules (CB1, CB2, CB3) in each sub-module are shared. The relationship between the modules is as follows: MSCB(X)=SCB2(SCB1(X)), SCB(X)=X+CB3(X+CB2(x+CB1(X))), X is the representative feature vector, which is introduced to describe the feature relationship between modules. SCB1 and SCB2 are two sub-modules in the multi-scale weight sharing convolution module MSCB. The weights of the three basic modules (CB1, CB2, CB3) in SCB1 are shared, and the weights of the three basic modules (CB1, CB2, CB3) in SCB2 are shared. The network structures of the three basic modules (CB1, CB2, and CB3) are the same. Now we will only introduce one of the basic modules CB1 in detail. Two convolutions of different scales are performed on the feature vectors of the input basic modules, as shown below: X2=conv1(X1), X3=conv2(X1), where X1 is the feature vector of the input basic module CB1, X2 and X3 are the convolution results obtained by the convolution operations of 3*3 and 5*5 respectively; conv1(·) is a convolution operation of 3*3, and conv2(·) is a convolution operation of 5*5. The two convolution results are feature concatenated, and then a 5*5 convolution operation is performed, as shown below: X c =(X2,X3),X4=conv2(X c ), where X cThe vector after the feature vectors X2 and X3 are concatenated, (·, ·) is the feature concatenation operation; X4 is the convolution result obtained by the convolution operation of size 5*5, and conv2(·) is the convolution operation of size 5*5. Finally, the feature vector X4 is subjected to a 1*1 convolution operation to obtain the output result of the module CB1, as shown below: X5=conv3(X4), X5 is the convolution result obtained by the convolution operation of size 1*1, and conv3(·) is the convolution operation of size 1*1.

[0013] The convolution layer includes three sequentially arranged multi-scale weight sharing convolution modules to obtain a deep feature vector that fully expresses the feature information, which is input into two consecutive 3*3 convolution layers, as shown below: Among them, F d is the deep feature vector output by the multi-scale weight-sharing convolution module, is the convolution result obtained by two consecutive 5*5 convolution operations, and conv(·) is a 3*3 convolution operation.

[0014] The upward projection module is at least used to complete the upsampling operation of the deep feature vector outputted above. The upward projection module includes upsampling the feature vector to a fully reconstructed resolution size, performing a 3*3 convolution on the deep feature vector outputted by the above process, and then using the Unpooling function to complete the upsampling operation, as shown below: in, is the deep feature vector after upsampling, Unpooling(·) is the nn.MaxUnpool2d function in pytorch, conv(·) is a 3*3 convolution operation, and the upsampling result is refined to obtain a more accurate upsampling result, as shown below: in, is the final deep feature vector output by the upward projection module, and conv(·) is a 3*3 convolution operation.

[0015] The super-resolution reconstruction module includes bilinear interpolation of the original low-resolution remote sensing image, and then adding the learned residual vector to obtain the super-resolution reconstruction result output by the super-resolution expert network, as shown below: Among them, U0 is the super-resolution reconstruction result image of the super-resolution expert network rotated by 0 degrees, BIC(·) is the bilinear interpolation operation, and conv(·) is the convolution operation of size 3*3.

[0016] The rotating super-resolution expert network super-resolution result includes rotating the super-resolution expert network super-resolution reconstruction result image to the original direction, as shown below: E0 = IR O t0(U0),E 90 =IR Ot 90 (U 90 ), E 180 =IR O t 180 (U 180 ), E 270 =IR O t 270 (U 270 ), where U0, U 90 , U 180 , U 270 The super-resolution reconstruction result images of the super-resolution expert network are respectively input as low-resolution images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees. O t0(·),IR O t 90 (·),IR O t 180 (·),IR O t 270 (·) respectively rotate the image counterclockwise by 0, 90, 180, and 270 degrees, E0, E 90 、E 180 、E 270 For U0 and U 90 , U 180 , U 270 The resulting image is rotated 0, 90, 180, and 270 degrees counterclockwise.

[0017] The method of calculating the average of the super-resolution results of the four super-resolution expert networks includes rotating the four super-resolution results back to the original direction to calculate the average, as shown below: E = (E0 + E 90 +E 180 +E 270 ) / 4, where E is the mean, E0, E 90 、E 180 、E 270 For U0 and U 90 , U 180 , U 270 The result image is rotated 0, 90, 180, and 270 degrees counterclockwise. The feature vector splicing includes performing a feature splicing operation on the four super-resolution results rotated back to the original direction and the average after obtaining the average, as shown below: c =(E, E0, E 90 、E 180 、E 270 ).

[0018] The fusion expert network includes the feature vector E after the feature concatenation operation cInput the fusion expert network and fully fuse it to obtain the residual vector required for super-resolution reconstruction. The fusion expert network is similar to the super-resolution expert network in structure, but differs in two parts: one is the upward projection module, which is not included in the fusion expert network, while the super-resolution expert network does; the other is the vector superimposed by the residual network. The super-resolution expert network superimposes the result of bilinear interpolation of the initial low-resolution remote sensing image, while the fusion expert network superimposes the mean E of the super-resolution results of the four super-resolution expert networks. First, the concatenated feature vector E c Perform shallow feature extraction as follows: D s = covn(covco(E c )), where D s In order to extract the shallow feature vector, conv(·) is a 3*3 convolution operation, and the extracted shallow feature vector D s Input three multi-scale weight-sharing convolution modules arranged in sequence to obtain a deep feature vector that fully expresses the feature information, as shown below: d =MSCB(MSCB(MSCB(D s ))), where D d To extract the deep feature vector, MSCB(·) is a multi-scale weight sharing convolution module. Then, the deep feature vector F d Input two consecutive 3*3 convolutional layers as follows: Among them, D d is the deep feature vector output by three sequential multi-scale weight-sharing convolutional modules, is the convolution result obtained by two consecutive 3*3 convolution operations, conv(·) is a 3*3 convolution operation, and finally, the mean E of the super-resolution results of the four super-resolution expert networks is added to The super-resolution reconstruction result output by this method is obtained as follows: Among them, OUT is the super-resolution reconstruction result image output by this method.

[0019] In the above technical solution, the present invention provides a weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method, which has the following beneficial effects:

[0020] The super-resolution reconstruction system can address the limitations of single expert decision-making and the imbalance between model parameter quantity and effect, thereby introducing a super-resolution reconstruction mechanism of multiple expert decisions and a multi-scale weight sharing convolution module in the super-resolution reconstruction process. The multi-expert decision mechanism can be used to obtain richer detailed texture information, and the multi-scale weight sharing convolution module can be used to effectively increase model performance without increasing too many model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0022] Figure 1 A flow chart of an embodiment of a weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method provided by the present invention.

[0023] Figure 2 A schematic diagram of a super-resolution expert network flow chart is provided for an embodiment of a weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method of the present invention.

[0024] Figure 3 A schematic diagram of a fusion expert network flow chart is provided for an embodiment of a remote sensing image super-resolution reconstruction method for multi-scale multi-expert decision-making with weight sharing in the present invention.

[0025] Figure 4 A schematic diagram of a multi-scale weight-sharing convolution module flow chart is provided for an embodiment of a weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method of the present invention.

[0026] Figure 5 A schematic diagram of the basic module flow chart provided for an embodiment of a weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method of the present invention.

[0027] Figure 6 A schematic diagram of an upward projection module flow chart is provided for an embodiment of a weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method of the present invention.

[0028] Figure 7 It is a schematic diagram for comparing the results of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0030] like Figure 1-6 As shown, an embodiment of the present invention provides a weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method.

[0031] Including image rotation, super-resolution expert network, rotating super-resolution expert network super-resolution results, finding the average of super-resolution results of four super-resolution expert networks, feature vector splicing and fusion expert network;

[0032] Image rotation includes inputting the original low-resolution remote sensing image and aligning and rotating it by 0, 90, 180, and 270 degrees, as shown below: I0 = R O t0(I),I 90 =R O t 90 (I),I 180 =R O t 180 (I),I 270 =R O t 270 (I), where I is the original low-resolution remote sensing image, R O t0(·), R O t 90 (·), R O t 180 (·), R O t 270 (·) respectively rotate the image clockwise by 0, 90, 180, and 270 degrees, I0, I 90 ,I 180 ,I 270 The resulting images are rotated 0, 90, 180, and 270 degrees clockwise respectively.

[0033] The super-resolution reconstruction system can address the limitations of single expert decision-making and the imbalance between model parameter quantity and effect, thereby introducing a super-resolution reconstruction mechanism of multiple expert decisions and a multi-scale weight sharing convolution module in the super-resolution reconstruction process. The multi-expert decision mechanism can be used to obtain richer detailed texture information, and the multi-scale weight sharing convolution module can be used to effectively increase model performance without increasing too many model parameters.

[0034] The super-resolution reconstruction system can utilize the super-resolution reconstruction mechanism of multi-expert decision-making, rotate the original image in four directions, and use four super-resolution expert networks to output the corresponding super-resolution reconstruction results respectively, and then use a fusion expert network to fuse the super-resolution results of the above four super-resolution expert networks to obtain richer and clearer super-resolution reconstruction results. In addition, a multi-scale weight sharing convolution module is introduced in the basic framework of the expert network, and the weights in the basic modules within the same submodule are shared, which not only increases the depth of the network but also reduces the number of parameters of the model itself, and can fully extract deep features. In addition, in the basic modules used, convolution modules of different scales are used to fully extract the feature information of each scale in the image.

[0035] Reference Figure 2The super-resolution expert network of this embodiment includes super-resolution reconstruction results, shallow feature extraction, multi-scale weight sharing convolution module, convolution layer, upward projection module and super-resolution reconstruction module. The super-resolution reconstruction results include inputting images rotated by 0 degrees, 90 degrees, 180 degrees and 270 degrees into four super-resolution expert networks respectively to obtain four different super-resolution reconstruction results. The four super-resolution expert networks use the same network structure, but the input is different, resulting in different outputs. After the rotated image is input into the four super-resolution expert networks, the corresponding super-resolution reconstruction results are obtained, as shown below: U0 = ExpertSR (I0), U 90 =ExpertSR(I 90 ), U 180 =ExpertSR(I 180 ), U 270 =ExpertSR(I 270 ), where U0, U 90 , U 180 , U 270 The images are the super-resolution reconstruction results of the super-resolution expert network rotated by 0, 90, 180 and 270 degrees respectively. ExpertSR is the super-resolution expert network. Next, the super-resolution expert network is introduced in detail. The four super-resolution expert networks use the same network structure, so only the super-resolution expert network rotated by 0 degree is introduced.

[0036] Shallow feature extraction includes shallow feature extraction of low-resolution remote sensing images rotated 0 degrees, as shown below: S =conv(conv(I0)), where F S To extract shallow feature vectors, conv(·) is a convolution operation of size 3*3.

[0037] Reference Figure 4 The multi-scale weight sharing convolution module of this embodiment includes inputting the extracted shallow feature vector into three multi-scale weight sharing convolution modules arranged in sequence to obtain a deep feature vector that fully expresses the feature information, as shown below: d =MSCB(MSCB(MSCB(F s ))), where F dIn order to extract the deep feature vector, MSCB(·) is a multi-scale weight sharing convolution module. Specifically, the multi-scale weight sharing convolution module MSCB includes two sub-modules (SCB1, SCB2). The weights of the three basic modules (CB1, CB2, CB3) in each sub-module are shared. The relationship between the modules is as follows: MSCB(X)=SCB2(SCB1(X)), SCB(X)=X+CB3(X+CB2(x+CB1(X))), X is the representative feature vector, which is introduced to describe the feature relationship between modules. SCB1 and SCB2 are two sub-modules in the multi-scale weight sharing convolution module MSCB. The weights of the three basic modules (CB1, CB2, CB3) in SCB1 are shared, and the weights of the three basic modules (CB1, CB2, CB3) in SCB2 are shared. The network structures of the three basic modules (CB1, CB2, and CB3) are the same. Now we will only introduce one of the basic modules CB1 in detail. Two convolutions of different scales are performed on the feature vectors of the input basic modules, as shown below: X2=conv1(X1), X3=conv2(X1), where X1 is the feature vector of the input basic module CB1, X2 and X3 are the convolution results obtained by the convolution operations of 3*3 and 5*5 respectively; conv1(·) is a convolution operation of 3*3, and conv2(·) is a convolution operation of 5*5. The two convolution results are feature concatenated, and then a 5*5 convolution operation is performed, as shown below: X c =(X2,X3),X4=conv2(X c ), where X c The vector after the feature vectors X2 and X3 are concatenated, (·, ·) is the feature concatenation operation; X4 is the convolution result obtained by the convolution operation of size 5*5, and conv2(·) is the convolution operation of size 5*5. Finally, the feature vector X4 is subjected to a 1*1 convolution operation to obtain the output result of the module CB1, as shown below: X5=conv3(X4), X5 is the convolution result obtained by the convolution operation of size 1*1, and conv3(·) is the convolution operation of size 1*1.

[0038] The convolution layer includes three sequentially arranged multi-scale weight-sharing convolution modules to obtain a deep feature vector that fully expresses the feature information, which is input into two consecutive 3*3 convolution layers, as shown below: Among them, F d is the deep feature vector output by the multi-scale weight-sharing convolution module, is the convolution result obtained by two consecutive 5*5 convolution operations, and conv(·) is a 3*3 convolution operation.

[0039] The upward projection module is at least used to complete the upsampling operation of the deep feature vector output above. The upward projection module includes upsampling the feature vector to a fully reconstructed resolution size, performing a 3*3 convolution on the deep feature vector output by the above process, and then using the Unpooling function to complete the upsampling operation, as shown below: in, is the deep feature vector after upsampling, Unpooling(·) is the nn.MaxUnpool2d function in pytorch, conv(·) is a 3*3 convolution operation, and the upsampling result is refined to obtain a more accurate upsampling result, as shown below: in, is the final deep feature vector output by the upward projection module, and conv(·) is a 3*3 convolution operation.

[0040] The super-resolution reconstruction module includes bilinear interpolation of the original low-resolution remote sensing image, and then adds the learned residual vector to obtain the super-resolution reconstruction result output by the super-resolution expert network, as shown below: Among them, U0 is the super-resolution reconstruction result image of the super-resolution expert network rotated by 0 degrees, BIC(·) is the bilinear interpolation operation, and conv(·) is the convolution operation of size 3*3.

[0041] Rotating the super-resolution expert network super-resolution result includes rotating the super-resolution expert network super-resolution reconstruction result image to the original direction, as shown below: E0 = IR O t0(U0),E 90 =IR O t 90 (U 90 ), E 180 =IR O t 180 (U 180 ), E 270 =IR O t 270 (U 270 ), where U0, U 90 , U 180 , U 270 The super-resolution reconstruction result images of the super-resolution expert network are respectively input as low-resolution images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees. O t0(·),IR O t 90 (·),IR O t 180 (·),IR O t 270(·) respectively rotate the image counterclockwise by 0, 90, 180, and 270 degrees, E0, E 90 、E 180 、E 270 For U0 and U 90 , U 180 , U 270 The resulting image is rotated 0, 90, 180, and 270 degrees counterclockwise.

[0042] Find the average of the super-resolution results of the four super-resolution expert networks, including the average of the four super-resolution results rotated back to the original direction, as shown below E = (E0 + E 90 +E 180 +E 270 ) / 4, where E is the mean, E0, E 90 、E 180 、E 270 For U0 and U 90 , U 180 , U 270 The result image is rotated 0, 90, 180, and 270 degrees counterclockwise. The feature vector splicing includes the feature splicing operation of the four super-resolution results rotated back to the original direction and the average after calculating the average, as shown below: c =(E, E0, E 90 、E 180 、E 270 ).

[0043] Reference Figure 3 The fusion expert network of this embodiment includes the feature vector E after the feature concatenation operation c Input the fusion expert network and fully fuse it to get the residual vector required for super-resolution reconstruction. The structure of the fusion expert network is similar to that of the super-resolution expert network, but there are two differences: one is the upward projection module, which is not included in the fusion expert network, but included in the super-resolution expert network; the other is the vector superimposed by the residual network. The super-resolution expert network superimposes the result of bilinear interpolation of the initial low-resolution remote sensing image, while the fusion expert network superimposes the mean E of the super-resolution results of the four super-resolution expert networks. First, the concatenated feature vector E is c Perform shallow feature extraction as shown below s = covn(covn(E c )), where D s In order to extract the shallow feature vector, conv(·) is a 3*3 convolution operation, and the extracted shallow feature vector D s Input three multi-scale weight-sharing convolution modules arranged in sequence to obtain a deep feature vector that fully expresses the feature information, as shown below: d=MSCB(MSCB(MSCB(D s ))), where D d To extract the deep feature vector, MSCB(·) is a multi-scale weight sharing convolution module. Then, the deep feature vector F d Input two consecutive 3*3 convolutional layers as follows: Among them, D d is the deep feature vector output by three sequential multi-scale weight-sharing convolutional modules, is the convolution result obtained by two consecutive 3*3 convolution operations, conv(·) is a 3*3 convolution operation, and finally, the mean E of the super-resolution results of the four super-resolution expert networks is added to The super-resolution reconstruction result output by this method is obtained as follows: Among them, OUT is the super-resolution reconstruction result image output by this method.

[0044] Reference Figure 7 , the bilinear interpolation result of this embodiment is shown in (a), and the super-resolution result is shown in (b). It can be seen that the clarity of the super-resolution result (b) is obviously better than that of the bilinear interpolation result (a).

[0045] Working principle: First, the original low-resolution remote sensing image is rotated clockwise by 0, 90, 180, and 270 degrees, respectively, and then input into four super-resolution expert networks to obtain four different super-resolution reconstruction results; then, the super-resolution reconstruction results rotated by 0, 90, 180, and 270 degrees clockwise are rotated counterclockwise by 0, 90, 180, and 270 degrees, respectively, and then rotated back; then, the average of the four rotated super-resolution reconstruction results is calculated; the four rotated super-resolution reconstruction results and the calculated average are feature-spliced ​​and input into the fusion expert network, which is fully fused to obtain the residual vector required for super-resolution reconstruction; finally, the mean of the super-resolution reconstruction result is added to the residual vector to obtain the final super-resolution reconstruction result.

[0046] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method, characterized in that: Including image rotation, super-resolution expert network, rotating super-resolution expert network super-resolution results, finding the average of four super-resolution expert network super-resolution results, feature vector splicing and fusion expert network; The image rotation includes inputting the original low-resolution remote sensing image, aligning and rotating it by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, as shown below: I0=R O t0(I)I 90 =R O t 90 (I)I 180 =R O t 180 (I),I 270 =R O t 270 (I) Among them, I is the original low-resolution remote sensing image, R O t0(·), R O t 90 (·), R O t 180 (·), R O t 270 (·) respectively rotate the image clockwise by 0, 90, 180, and 270 degrees, I0, I 90 ,I 180 ,I 270 The resulting images are rotated 0, 90, 180, and 270 degrees clockwise respectively.

2. According to the weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method of claim 1, it is characterized in that: The super-resolution expert network includes a super-resolution reconstruction result, a shallow feature extraction, a multi-scale weight sharing convolution module, a convolution layer, an upward projection module and a super-resolution reconstruction module. The super-resolution reconstruction result includes obtaining images rotated by 0 degrees, 90 degrees, 180 degrees and 270 degrees, respectively inputting them into four super-resolution expert networks to obtain four different super-resolution reconstruction results. The four super-resolution expert networks use the same network structure, but the input is different, which leads to different outputs. The rotated image is input into the four super-resolution expert networks to obtain the corresponding super-resolution reconstruction results, as shown below: U0 = ExpertSR (I0), U 90 =ExpertSR(I 90 ), U 180 =ExpertSR(I 180 ),U 270 =ExpertSR(I 270 ), where U0, U 90 , U 180 , U 270 They are the super-resolution expert network super-resolution reconstruction result images rotated 0, 90, 180, and 270 degrees respectively, and ExpertSR is the super-resolution expert network.

3. The weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method according to claim 2 is characterized in that: The shallow feature extraction includes performing shallow feature extraction on a low-resolution remote sensing image rotated 0 degrees, as shown below: S =conv(conv(I0))where, F S To extract shallow feature vectors, conv(·) is a convolution operation of size 3*3.

4. The weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method according to claim 3 is characterized in that: The multi-scale weight sharing convolution module includes inputting the extracted shallow feature vector into three multi-scale weight sharing convolution modules arranged in sequence to obtain a deep feature vector that fully expresses the feature information; As shown below: d =MSCB(MSCB(MSCB(F s ))), where F d In order to extract the deep feature vector, MSCB(·) is a multi-scale weight sharing convolution module. Specifically, the multi-scale weight sharing convolution module MSCB includes two sub-modules (SCB1 and SCB2). The weights of the three basic modules (CB1, CB2, and CB3) in each sub-module are shared. The relationship between the modules is as follows: MSCB(X)=SCB2(SCB1(X)), SCB(X)=X+CB3(X+CB2(x+CB1(X))), where X is the representative feature vector introduced to describe the feature relationship between modules; SCB1 and SCB2 are two sub-modules in the multi-scale weight sharing convolution module MSCB. The weights of the three basic modules (CB1, CB2, and CB3) in SCB1 are shared, and the weights of the three basic modules (CB1, CB2, and CB3) in SCB2 are shared; The network structures of the three basic modules (CB1, CB2, and CB3) are the same. Now we will only introduce one of the basic modules CB1 in detail. Two convolutions of different scales are performed on the feature vectors of the input basic modules, as shown below: X2=conv1(X1), X3=conv2(X1), where X1 is the feature vector of the input basic module CB1, X2 and X3 are the convolution results obtained by the convolution operations of 3*3 and 5*5 respectively; conv1(·) is a convolution operation of 3*3, and conv2(·) is a convolution operation of 5*5. The two convolution results are feature concatenated, and then a 5*5 convolution operation is performed, as shown below: X c =(X2,X3),X4=conv2(X c ), where X c The concatenated vector of feature vectors X2 and X3, (·, ·) is the feature concatenation operation; X4 is the convolution result obtained by the convolution operation of size 5*5, and conv2(·) is the convolution operation of size 5*5: Finally, the feature vector X4 is subjected to a 1*1 convolution operation to obtain the output result of the module CB1, as shown below: X5=conv3(X4), where X5 is the convolution result obtained by a convolution operation of size 1*1, and conv3(·) is a convolution operation of size 1*1.

5. The weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method according to claim 4 is characterized in that: The convolution layer includes three sequentially arranged multi-scale weight sharing convolution modules to obtain a deep feature vector that fully expresses the feature information, which is input into two consecutive 3*3 convolution layers, as shown below: Among them, F d is the deep feature vector output by the multi-scale weight-sharing convolution module, is the convolution result obtained by two consecutive 5*5 convolution operations, and conv(·) is a 3*3 convolution operation.

6. The weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method according to claim 5 is characterized in that: The upward projection module is at least used to complete the upsampling operation of the deep feature vector outputted above. The upward projection module includes upsampling the feature vector to a fully reconstructed resolution size, performing a 3*3 convolution on the deep feature vector outputted by the above process, and then using the Unpooling function to complete the upsampling operation, as shown below: in, is the deep feature vector after upsampling, Unpooling(·) is the nn.MaxUnpool2d function in pytorch, conv(·) is a 3*3 convolution operation, and the upsampling result is refined to obtain a more accurate upsampling result, as shown below: in, is the final deep feature vector output by the upward projection module, and conv(·) is a 3*3 convolution operation.

7. The weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method according to claim 6 is characterized in that: The super-resolution reconstruction module includes bilinear interpolation of the original low-resolution remote sensing image, and then adding the learned residual vector to obtain the super-resolution reconstruction result output by the super-resolution expert network, as shown below: Among them, U0 is the super-resolution reconstruction result image of the super-resolution expert network rotated by 0 degrees, BIC(·) is the bilinear interpolation operation, and conv(·) is the convolution operation of size 3*3.

8. The weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method according to claim 7 is characterized in that: The rotating super-resolution expert network super-resolution result includes rotating the super-resolution expert network super-resolution reconstruction result image to the original direction, as shown below: E0 = IR O t0(U0),E 90 =IR O t 90 (U 90 ), E 180 =IR O t 180 (U 180 ), E 270 =IR O t 270 (U 270 ), where U0, U 90 , U 180 , U 270 The super-resolution reconstruction result images of the super-resolution expert network are respectively input as low-resolution images rotated 0 degrees, 90 degrees, 180 degrees, and 270 degrees. O t0(·),IR O t 90 (·),IR O t 180 (·),IR O t 270 (·) respectively rotate the image counterclockwise by 0, 90, 180, and 270 degrees, E0, E 90 、E 180 、E 270 For U0 and U 90 , U 180 , U 270 The resulting image is rotated 0, 90, 180, and 270 degrees counterclockwise.

9. The weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method according to claim 8, characterized in that: The method of calculating the average of the super-resolution results of the four super-resolution expert networks includes rotating the four super-resolution results back to the original direction to calculate the average, as shown below: E = (E0 + E 90 +E 180 +E 270 ) / 4, where E is the mean, E0, E 90 、E 180 、E 270 For U0 and U 90 , U 180 , U 270 The result image is rotated 0, 90, 180, and 270 degrees counterclockwise. The feature vector splicing includes performing a feature splicing operation on the four super-resolution results rotated back to the original direction and the average after obtaining the average, as shown below: c =(E, E0, E 90 、E 180 、E 270 ).

10. The weight-sharing multi-scale multi-expert decision-making remote sensing image super-resolution reconstruction method according to claim 9, characterized in that: The fusion expert network includes the feature vector E after the feature concatenation operation c The fusion expert network is input and fully fused to obtain the residual vector required for super-resolution reconstruction. The fusion expert network is similar to the super-resolution expert network in structure, but differs in two parts: The first is the upward projection module, which is not included in the fusion expert network but included in the super-resolution expert network; The second is the vector superimposed by the residual network. The super-resolution expert network superimposes the result of bilinear interpolation of the initial low-resolution remote sensing image, while the fusion expert network superimposes the mean E of the super-resolution results of the four super-resolution expert networks. First, the concatenated feature vector E c Perform shallow feature extraction as follows: D s = covn(covn(E c )), where D s In order to extract the shallow feature vector, conv(·) is a 3*3 convolution operation, and the extracted shallow feature vector D s Input three multi-scale weight-sharing convolution modules arranged in sequence to obtain a deep feature vector that fully expresses the feature information, as shown below: d =MSCB(MSCB(MSCB(D s ))), where D d To extract the deep feature vector, MSCB(·) is a multi-scale weight sharing convolution module. Then, the deep feature vector F d Input two consecutive 3*3 convolutional layers as follows: Among them, D d is the deep feature vector output by three sequential multi-scale weight-sharing convolutional modules, is the convolution result obtained by two consecutive 3*3 convolution operations, conv(·) is a 3*3 convolution operation, and finally, the mean E of the super-resolution results of the four super-resolution expert networks is added to The super-resolution reconstruction result output by this method is obtained as follows: Among them, OUT is the super-resolution reconstruction result image output by this method.

Images