Remote sensing image fusion method and device based on quaternary convolutional neural network

Abstract

The invention discloses a remote sensing image fusion method and device based on a quaternary convolutional neural network, and belongs to the technical field of remote sensing images, and the method comprises the steps: carrying out the up-sampling of a preset multiple of a multispectral image, and obtaining a first image; the first image is subjected to feature enhancement through a first feature enhancement module to obtain a first feature, the panchromatic image is subjected to feature enhancement through a second feature enhancement module to obtain a second feature, and the first feature enhancement module and the second feature enhancement module are consistent in structure; and finally, fusing the first feature and the second feature through a fusion module of the quaternary convolutional neural network to obtain a to-be-output remote sensing image, so that more feature details can be effectively reserved, the resolution of the output remote sensing image can be improved, and the remote sensing image fusion is more accurate.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing images, and in particular relates to a remote sensing image fusion method and device based on a quaternary convolutional neural network. Background Art

[0002] In recent years, with the successful launch of optical earth observation satellites, a large amount of data that can be used in various research fields has been provided for the research of remote sensing images, which has promoted the vigorous development of various work fields. Specifically in the remote sensing system, satellites can obtain two types of images in different ways, namely multispectral images and panchromatic images. Among them, multispectral images have multiple channels, can obtain rich color information, and have high spectral resolution, but their spatial resolution is low, while panchromatic images are single-channel and have a large spatial resolution, but they cannot display color information. There are solutions for fusing the two remote sensing images in the existing technology, but the fusion of the existing solutions is not accurate enough, and the feature details are lost a lot.

[0003] Therefore, how to fuse remote sensing images more accurately while retaining more feature details and improving the resolution of the output remote sensing images is a technical problem to be solved by those skilled in the art. Summary of the invention

[0004] The purpose of the present invention is to solve the technical problems in the prior art that when two remote sensing images are fused, more feature details are lost and the resolution of the fused remote sensing image is not high enough.

[0005] To achieve the above technical objectives, on the one hand, the present invention provides a remote sensing image fusion method based on a quaternary convolutional neural network, the method comprising: The multispectral image is upsampled by a preset multiple to obtain a first image; The first image is feature enhanced by a first feature enhancement module to obtain a first feature, and the full-color image is feature enhanced by a second feature enhancement module to obtain a second feature, wherein the first feature enhancement module and the second feature enhancement module have the same structure; The first feature and the second feature are fused through the fusion module of the quaternary convolutional neural network to obtain the remote sensing image to be output.

[0006] Further, the first feature enhancement module includes a first layer convolution module, a second layer convolution module, a third layer convolution module, a first layer stacking module, a fourth layer convolution module, a second layer stacking module, a fifth layer convolution module and an output module connected in sequence, wherein the first layer convolution module is specifically a convolution unit and an activation function with a step size of 1, the second layer convolution module includes a first convolution block and a second convolution block, the first convolution block is specifically a convolution unit and an activation function with a step size of 1, the second convolution block is specifically a convolution unit and an activation function with a step size of 2, the third layer convolution module includes a third convolution block, a fourth convolution block, a fifth convolution block, a sixth convolution block, a seventh convolution block and an eighth convolution block, the third convolution block is specifically a convolution unit and an activation function with a step size of 2, the fourth convolution block is specifically a convolution unit and an activation function with a step size of 1, the fifth convolution block is specifically a convolution unit and an activation function with a step size of 4, the sixth convolution block is specifically a convolution unit and an activation function with a step size of 1, the seventh convolution block Specifically, it is a transposed convolution and an activation function with a step size of 2, the eighth convolution block is specifically a convolution unit and an activation function with a step size of 2, the fourth layer convolution module includes a ninth convolution block, a tenth convolution block, an eleventh convolution block, a twelfth convolution block, a thirteenth convolution block and a fourteenth convolution block, wherein the ninth convolution block is specifically a transposed convolution and an activation function with a step size of 2, the tenth convolution block is specifically a convolution unit and an activation function with a step size of 1, and the eleventh convolution block is specifically a convolution unit and an activation function of 2, The twelfth convolution block is specifically a convolution unit and an activation function with a step size of 1, the thirteenth convolution block and the fourteenth convolution block are both transposed convolution and activation functions with a step size of 2, the fifth-layer convolution module includes a fifteenth convolution block and a sixteenth convolution block, the fifteenth convolution block is specifically a convolution unit and an activation function with a step size of 1, the sixteenth convolution block is specifically a transposed convolution and an activation function with a step size of 2, and the output module is specifically a stacked sub-module, a convolution unit with a step size of 2, and an activation function connected in sequence.

[0007] Further, the step of performing feature enhancement on the first image by a first feature enhancement module to obtain a first feature specifically includes: Input the first image into the first convolution module to obtain feature data A1; Input the feature data A1 into the first convolution block and the second convolution block respectively to obtain corresponding feature data B1 and feature data B2; The feature data B1 is respectively input into the third convolution block, the fourth convolution block and the fifth convolution block to obtain the corresponding feature data C1, feature data C2 and feature data C3, and the feature data B2 is respectively input into the sixth convolution block, the seventh convolution block and the eighth convolution block to obtain the corresponding feature data D1, feature data D2 and feature data D3; The characteristic data C1 and the characteristic data D1 are input into the first layer stacking module for stacking to obtain characteristic data E1, the characteristic data C2 and the characteristic data D2 are input into the first layer stacking module for stacking to obtain characteristic data E2, and the characteristic data C3 and the characteristic data D3 are input into the first layer stacking module for stacking to obtain characteristic data E3; The feature data E1 is respectively input into the ninth convolution block and the tenth convolution block to obtain corresponding feature data F1 and feature data F2, the feature data E2 is respectively input into the eleventh convolution block and the twelfth convolution block to obtain corresponding feature data F3 and feature data F4, and the feature data E3 is respectively input into the thirteenth convolution block and the fourteenth convolution block to obtain corresponding feature data F5 and feature data F6; Input the feature data F1, feature data F4 and feature data F5 into the second layer stacking module for stacking to obtain feature data F7, input the feature data F2, feature data F3 and feature data F6 into the second layer stacking module for stacking to obtain feature data F8; Input the feature data F7 into the fifteenth convolution block to obtain feature data H1, and input the feature data F8 into the sixteenth convolution block to obtain feature data H2; The feature data H1 and the feature data H2 are input into the output module for processing to obtain the first feature of the output.

[0008] Furthermore, the fusing of the first feature and the second feature to obtain the remote sensing image to be output specifically includes: The first feature is processed by the attention module and then divided into multiple groups of sub-features, and the second feature is processed by spatial feature enhancement to obtain a second feature to be fused; Input each group of sub-features into the corresponding convolution channel for convolution mapping and spectral feature enhancement to obtain the corresponding first sub-feature to be fused, wherein the structures of all convolution channels are consistent; Stacking all the first sub-features to be fused to obtain a first feature to be fused; The first feature to be fused is convolved with the second feature to be fused to obtain a remote sensing image to be output.

[0009] Furthermore, the method also includes updating the network parameters of the quaternary convolutional neural network based on the output of the first feature enhancement module, the output of the second feature enhancement module and the output of the fusion module at the end of each training during the training phase of the quaternary convolutional neural network.

[0010] On the other hand, the present invention also provides a remote sensing image fusion device based on a quaternary convolutional neural network, the device comprising: A sampling module, used to upsample the multispectral image by a preset multiple to obtain a first image; A first feature enhancement module, used for performing feature enhancement on the first image to obtain a first feature; A second feature enhancement module, used for performing feature enhancement on the full-color image to obtain a second feature; The quaternary convolutional neural network is used to fuse the first feature and the second feature to obtain the remote sensing image to be output.

[0011] The present invention provides a remote sensing image fusion method and device based on a quaternary convolutional neural network. Compared with the prior art, the method firstly performs upsampling of a multispectral image by a preset multiple to obtain a first image; then performs feature enhancement on the first image by a first feature enhancement module to obtain a first feature, and performs feature enhancement on the full-color image by a second feature enhancement module to obtain a second feature, wherein the first feature enhancement module and the second feature enhancement module have the same structure; finally, the first feature and the second feature are fused by a fusion module of a quaternary convolutional neural network to obtain a remote sensing image to be output, which can effectively retain more feature details and improve the resolution of the output remote sensing image, thereby making the remote sensing image fusion more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 The figure is a schematic diagram of the process of the remote sensing image fusion method based on the quaternary convolutional neural network provided in the embodiment of this specification; Figure 2 Shown is a schematic diagram of the structure of a remote sensing image fusion device based on a quaternary convolutional neural network provided in an embodiment of this specification. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0015] like Figure 1The flowchart of the remote sensing image fusion method based on the quaternary convolutional neural network provided by the embodiment of this specification is shown. Although this specification provides the method operation steps or device structures shown in the following embodiments or drawings, the method or device may include more or fewer operation steps or module units after partial merger based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure shown in the embodiments or drawings of this specification. When the method or module structure is applied in an actual device, server or terminal product, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiment or drawings (for example, in an environment of parallel processors or multi-threaded processing, or even in an implementation environment of distributed processing and server clusters).

[0016] The remote sensing image fusion method based on the quaternary convolutional neural network provided in the present embodiment can be applied to terminal devices such as clients and servers, such as Figure 1 As shown, the method specifically comprises the following steps: Step S101: up-sample the multispectral image by a preset multiple to obtain a first image.

[0017] Specifically, the panchromatic image and multispectral image in this application are both original full-size images. The panchromatic image can be represented as a Full-scale PAN image. , the multispectral image can be represented as Full-scale LRMS image, .

[0018] Step S102: performing feature enhancement on the first image by a first feature enhancement module to obtain a first feature, and performing feature enhancement on the full-color image by a second feature enhancement module to obtain a second feature, wherein the first feature enhancement module and the second feature enhancement module have the same structure.

[0019] Specifically, the panchromatic image and the multispectral image are respectively enhanced by two preset feature enhancement modules, so as to obtain the spatial feature, that is, the second feature, and the spectral feature, that is, the first feature.

[0020] In an embodiment of the present application, the first feature enhancement module includes a first layer convolution module, a second layer convolution module, a third layer convolution module, a first layer stacking module, a fourth layer convolution module, a second layer stacking module, a fifth layer convolution module and an output module connected in sequence, wherein the first layer convolution module is specifically a convolution unit and an activation function with a step size of 1, the second layer convolution module includes a first convolution block and a second convolution block, the first convolution block is specifically a convolution unit and an activation function with a step size of 1, the second convolution block is specifically a convolution unit and an activation function with a step size of 2, the third layer convolution module includes a third convolution block, a fourth convolution block, a fifth convolution block, a sixth convolution block, a seventh convolution block and an eighth convolution block, the third convolution block is specifically a convolution unit and an activation function with a step size of 2, the fourth convolution block is specifically a convolution unit and an activation function with a step size of 1, the fifth convolution block is specifically a convolution unit and an activation function with a step size of 4, the sixth convolution block is specifically a convolution unit and an activation function with a step size of 1, the seventh convolution block is The convolution block is specifically a transposed convolution and an activation function with a step size of 2, the eighth convolution block is specifically a convolution unit and an activation function with a step size of 2, and the fourth layer convolution module includes a ninth convolution block, a tenth convolution block, an eleventh convolution block, a twelfth convolution block, a thirteenth convolution block and a fourteenth convolution block, wherein the ninth convolution block is specifically a transposed convolution and an activation function with a step size of 2, the tenth convolution block is specifically a convolution unit and an activation function with a step size of 1, and the eleventh convolution block is specifically a convolution unit and an activation function with a step size of 2. The twelfth convolution block is specifically a convolution unit and an activation function with a step size of 1, the thirteenth convolution block and the fourteenth convolution block are both transposed convolution and activation functions with a step size of 2, the fifth-layer convolution module includes a fifteenth convolution block and a sixteenth convolution block, the fifteenth convolution block is specifically a convolution unit and an activation function with a step size of 1, the sixteenth convolution block is specifically a transposed convolution and an activation function with a step size of 2, and the output module is specifically a stacked sub-module, a convolution unit with a step size of 2, and an activation function connected in sequence.

[0021] Specifically, the step of performing feature enhancement on the first image by a first feature enhancement module to obtain the first feature specifically includes: Input the first image into the first convolution module to obtain feature data A1; Input the feature data A1 into the first convolution block and the second convolution block respectively to obtain corresponding feature data B1 and feature data B2; The feature data B1 is respectively input into the third convolution block, the fourth convolution block and the fifth convolution block to obtain the corresponding feature data C1, feature data C2 and feature data C3, and the feature data B2 is respectively input into the sixth convolution block, the seventh convolution block and the eighth convolution block to obtain the corresponding feature data D1, feature data D2 and feature data D3; The characteristic data C1 and the characteristic data D1 are input into the first layer stacking module for stacking to obtain characteristic data E1, the characteristic data C2 and the characteristic data D2 are input into the first layer stacking module for stacking to obtain characteristic data E2, and the characteristic data C3 and the characteristic data D3 are input into the first layer stacking module for stacking to obtain characteristic data E3; The feature data E1 is respectively input into the ninth convolution block and the tenth convolution block to obtain corresponding feature data F1 and feature data F2, the feature data E2 is respectively input into the eleventh convolution block and the twelfth convolution block to obtain corresponding feature data F3 and feature data F4, and the feature data E3 is respectively input into the thirteenth convolution block and the fourteenth convolution block to obtain corresponding feature data F5 and feature data F6; Input the feature data F1, feature data F4 and feature data F5 into the second layer stacking module for stacking to obtain feature data F7, input the feature data F2, feature data F3 and feature data F6 into the second layer stacking module for stacking to obtain feature data F8; Input the feature data F7 into the fifteenth convolution block to obtain feature data H1, and input the feature data F8 into the sixteenth convolution block to obtain feature data H2; The feature data H1 and the feature data H2 are input into the output module for processing to obtain the first feature of the output.

[0022] Specifically, the convolution unit and activation function in each convolution block are used to map the input feature data. The activation function is specifically the PReLU activation function. The multi-scale convolution layer in the first feature enhancement module and the second feature enhancement module can extract information of different scales from the image. Compared with a single scale, the enhancement module extracts richer information from the image, which is helpful for the fusion of subsequent information, thereby further improving the accuracy of the fused image.

[0023] Step S103: fusing the first feature and the second feature through a fusion module of a quaternion convolutional neural network to obtain a remote sensing image to be output.

[0024] Specifically, the MS enhanced features are input to the fusion module in the quaternary convolutional neural network, and the MS enhanced features are the first features after feature enhancement of the first image, corresponding to the multispectral image, and then up-sampled at different scales. The quaternary convolutional fusion module enhances the fused features and integrates them with the PAN enhanced features, which are the second features and correspond to the full-color image. After convolution mapping, the fused features are processed by the channel attention block, which consists of adaptive global average pooling, linear layers, and activation functions. Then, the feature channel is divided into four branches. Each branch is subjected to multi-scale group convolution mapping and then input to the convolution block. The convolution block performs convolution mapping and normalization on the features, and then combines them with the original features and processes them using the LeakyReLU activation function. Then, the features from the four convolution blocks are merged, and the merged features are integrated with the PAN enhanced features from the PAN enhanced feature module. The spatial attention module facilitates this integration, which performs a convolution operation on the PAN enhanced features.

[0025] The first feature is processed by the attention module and then divided into multiple groups of sub-features, and the second feature is processed by spatial feature enhancement to obtain a second feature to be fused; Input each group of sub-features into the corresponding convolution channel for convolution mapping and spectral feature enhancement to obtain the corresponding first sub-feature to be fused, wherein the structures of all convolution channels are consistent; Stacking all the first sub-features to be fused to obtain a first feature to be fused; The first feature to be fused is convolved with the second feature to be fused to obtain a remote sensing image to be output.

[0026] Specifically, that is to say, the first feature and the second feature are first processed by the attention module, and the spatial feature is enhanced. Then, the first feature processed by the attention module is divided into multiple groups of sub-features, and each group of sub-features is input into the corresponding convolution channel for convolution mapping and normalization. Then, the group of sub-features is subjected to spectral feature enhancement by the spectral feature enhancement module to obtain the first sub-feature to be fused. Then, all the first sub-features to be fused are stacked and integrated to obtain the first feature to be fused and convoluted again, and then stacked and integrated with the second feature after spatial feature enhancement, that is, the second feature to be fused, to obtain the final remote sensing image to be output.

[0027] In addition, during the training phase of the quaternary convolutional neural network, at the end of each training, the network parameters of the quaternary convolutional neural network are updated based on the output of the first feature enhancement module, the output of the second feature enhancement module and the output of the fusion module.

[0028] In order to more accurately describe the present application scheme and evaluate its effect, the present application scheme is compared with the BDSD (band-dependent spatial detail) transformation method, AWLP (additive wavelet luminance proportional) transformation method, MTF-GLP-CBD (modulation transfer function Generalized Laplacian Pyramid with contextbased decision) transformation method and Enhanced-PanNet method in the prior art. The comparison results are reflected in the following evaluation indicators: 1. The global fusion index Q4 reflects the overall similarity between the reference image and the fused image in space and spectrum. The closer the value of Q4 is to 1, the better the fusion result.

[0029] 2. Root mean square error RMSE, which represents the square root of the ratio of the square of the deviation between the predicted value and the true value to the number of observations n. The smaller the value, the better the fusion result.

[0030] 3. The global comprehensive error index ERG, based on RMSE, considers the scale relationship between the fused image and the observed image. The interval is [0,1]. The closer the index is to 1, the better.

[0031] 4. Spectral arc SAM, which indicates the degree of spectrum distortion. The closer it is to 0, the better the fusion result.

[0032] 5. The overall image quality index UIQI indicates the closeness between the fused image and the reference image. The closer it is to 1, the better the fusion result.

[0033] The comparison results are shown in Table 1 below: Table 1 Usage scenarios BDSD AWLP MTF-GLP-CBD PNN Enhanced-PanNet This application plan Q4 0.7948 0.8081 0.726 0.7683 0.789 0.9688 RMSE 30.7153 25.5502 29.4144 25.252 21.3259 19.1374 ERG 1.7835 1.4781 1.7087 2.0521 1.5442 1.3646 SAM 5.5667 4.9367 4.6384 4.4735 4.6647 3.6353 UIQ 0.9435 0.9528 0.9156 0.9407 0.9569 0.9642 It can be seen from Table 1 that the global fusion index Q4 and the overall image quality index UIQI are greater than the evaluation values ​​of the prior art, and the root mean square error RMSE, the global error score ERG and the spectral arc SAM are all smaller than the evaluation values ​​of the prior art. The above evaluation values ​​are better than the evaluation values ​​of the prior art. It can be seen that most of the objective evaluation indicators of the present invention are better than the objective evaluation indicators of the prior art.

[0034] Based on the above-mentioned remote sensing image fusion method based on a quaternary convolutional neural network, one or more embodiments of this specification also provide a platform and terminal for remote sensing image fusion based on a quaternary convolutional neural network. The platform or terminal may include a device, software, module, plug-in, server, client, etc. using the method described in the embodiments of this specification and combined with the necessary implementation hardware. Based on the same innovative concept, the system in one or more embodiments provided in the embodiments of this specification is as described in the following embodiments. Since the implementation scheme and method for solving the problem of the system are similar, the implementation of the specific system in the embodiments of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. The terms "unit" or "module" used below can implement a combination of software and / or hardware with predetermined functions. Although the system described in the following embodiments is preferably implemented in software, hardware and a combination of software and hardware are also possible and conceived.

[0035] Specifically, Figure 2 is a schematic diagram of the module structure of an embodiment of a remote sensing image fusion device based on a quaternary convolutional neural network provided in this specification, such as Figure 2 As shown, the remote sensing image fusion device based on the quaternary convolutional neural network provided in this specification includes: The sampling module 201 is used to upsample the multispectral image by a preset multiple to obtain a first image; A first feature enhancement module 202, configured to perform feature enhancement on the first image to obtain a first feature; A second feature enhancement module 203, used to perform feature enhancement on the full-color image to obtain a second feature; The quaternary convolutional neural network 204 is used to fuse the first feature and the second feature to obtain the remote sensing image to be output. It should be noted that the above system can also include other implementation methods according to the description of the corresponding method embodiment. The specific implementation method can refer to the description of the corresponding method embodiment above, which will not be repeated here.

[0036] The present application also provides an electronic device, including: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the method provided in the above embodiment.

[0037] The electronic device provided in the embodiment of the present application stores executable instructions of the processor in a memory. When the processor executes the executable instructions, it can first upsample the multispectral image by a preset multiple to obtain a first image; then perform feature enhancement on the first image through a first feature enhancement module to obtain a first feature, and perform feature enhancement on the full-color image through a second feature enhancement module to obtain a second feature, and the first feature enhancement module and the second feature enhancement module have the same structure; finally, the first feature and the second feature are fused through a fusion module of a quaternary convolutional neural network to obtain a remote sensing image to be output, which can effectively retain more feature details and improve the resolution of the output remote sensing image, so that the remote sensing image fusion is more accurate.

[0038] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0039] The methods or devices described in the above embodiments provided in this specification can implement business logic through computer programs and record them on storage media, and the storage media can be read and executed by computers to achieve the effects of the solutions described in the embodiments of this specification, such as: The multispectral image is upsampled by a preset multiple to obtain a first image; The first image is feature enhanced by a first feature enhancement module to obtain a first feature, and the full-color image is feature enhanced by a second feature enhancement module to obtain a second feature, wherein the first feature enhancement module and the second feature enhancement module have the same structure; The first feature and the second feature are fused through the fusion module of the quaternary convolutional neural network to obtain the remote sensing image to be output.

[0040] The storage medium may include a physical device for storing information, which is usually a medium that digitizes information and then stores it in an electrical, magnetic or optical manner. The storage medium may include: a device that stores information in an electrical energy manner, such as various memories, such as RAM, ROM, etc.; a device that stores information in a magnetic energy manner, such as a hard disk, a floppy disk, a magnetic tape, a magnetic core memory, a magnetic bubble memory, a USB flash drive; a device that stores information in an optical manner, such as a CD or a DVD. Of course, there are other readable storage media, such as quantum memory, graphene memory, etc.

[0041] The embodiments of this specification are not limited to complying with industry communication standards, standard computer resource data update and data storage rules, or the situations described in one or more embodiments of this specification. Certain industry standards or slightly modified implementation plans based on the implementation described in the custom method or embodiment can also achieve the same, equivalent or similar, or predictable implementation effects after deformation of the above-mentioned embodiments. The embodiments obtained by using these modified or deformed data acquisition, storage, judgment, processing methods, etc. can still fall within the scope of the optional implementation plans of the embodiments of this specification.

[0042] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a computer-readable medium such as a microprocessor or processor and a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, ATMEL AT91SAM, MICROCHIP PIC18F26K20, and SILICONE LABS C8051F320. The memory controller can also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0043] The device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. There may be other divisions in actual implementation, such as multiple units or plug-ins may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interface, indirect coupling or communication connection of the device or unit, which may be electrical, mechanical or other forms.

[0044] These computer program instructions may also be loaded onto a computer or other programmable resource data updating device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0045] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0046] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A remote sensing image fusion method based on a quaternary convolutional neural network, characterized in that: The method comprises: The multispectral image is upsampled by a preset multiple to obtain a first image; The first image is feature enhanced by a first feature enhancement module to obtain a first feature, and the full-color image is feature enhanced by a second feature enhancement module to obtain a second feature, wherein the first feature enhancement module and the second feature enhancement module have the same structure; The first feature and the second feature are fused through the fusion module of the quaternary convolutional neural network to obtain the remote sensing image to be output.

2. The remote sensing image fusion method based on a quaternary convolutional neural network as claimed in claim 1, characterized in that: The first feature enhancement module includes a first layer convolution module, a second layer convolution module, a third layer convolution module, a first layer stacking module, a fourth layer convolution module, a second layer stacking module, a fifth layer convolution module and an output module connected in sequence, wherein the first layer convolution module is specifically a convolution unit and an activation function with a step size of 1, the second layer convolution module includes a first convolution block and a second convolution block, the first convolution block is specifically a convolution unit and an activation function with a step size of 1, the second convolution block is specifically a convolution unit and an activation function with a step size of 2, the third layer convolution module includes a third convolution block, a fourth convolution block, a fifth convolution block, a sixth convolution block, a seventh convolution block and an eighth convolution block, the third convolution block is specifically a convolution unit and an activation function with a step size of 2, the fourth convolution block is specifically a convolution unit and an activation function with a step size of 1, the fifth convolution block is specifically a convolution unit and an activation function with a step size of 4, the sixth convolution block is specifically a convolution unit and an activation function with a step size of 1, and the seventh convolution block is specifically The eighth convolution block is specifically a convolution unit and an activation function with a step size of 2, the fourth convolution module includes a ninth convolution block, a tenth convolution block, an eleventh convolution block, a twelfth convolution block, a thirteenth convolution block and a fourteenth convolution block, wherein the ninth convolution block is specifically a transposed convolution and an activation function with a step size of 2, the tenth convolution block is specifically a convolution unit and an activation function with a step size of 1, the eleventh convolution block is specifically a convolution unit and an activation function with a step size of 2, the The twelfth convolution block is specifically a convolution unit and an activation function with a step size of 1, the thirteenth convolution block and the fourteenth convolution block are both transposed convolutions and activation functions with a step size of 2, the fifth-layer convolution module includes a fifteenth convolution block and a sixteenth convolution block, the fifteenth convolution block is specifically a convolution unit and an activation function with a step size of 1, the sixteenth convolution block is specifically a transposed convolution and an activation function with a step size of 2, and the output module is specifically a stacked sub-module, a convolution unit with a step size of 2, and an activation function connected in sequence.

3. The remote sensing image fusion method based on a quaternary convolutional neural network as claimed in claim 2, characterized in that: The step of performing feature enhancement on the first image by a first feature enhancement module to obtain the first feature specifically includes: Input the first image into the first convolution module to obtain feature data A1; Input the feature data A1 into the first convolution block and the second convolution block respectively to obtain corresponding feature data B1 and feature data B2; The feature data B1 is respectively input into the third convolution block, the fourth convolution block and the fifth convolution block to obtain the corresponding feature data C1, feature data C2 and feature data C3, and the feature data B2 is respectively input into the sixth convolution block, the seventh convolution block and the eighth convolution block to obtain the corresponding feature data D1, feature data D2 and feature data D3; The characteristic data C1 and the characteristic data D1 are input into the first layer stacking module for stacking to obtain characteristic data E1, the characteristic data C2 and the characteristic data D2 are input into the first layer stacking module for stacking to obtain characteristic data E2, and the characteristic data C3 and the characteristic data D3 are input into the first layer stacking module for stacking to obtain characteristic data E3; The feature data E1 is respectively input into the ninth convolution block and the tenth convolution block to obtain corresponding feature data F1 and feature data F2, the feature data E2 is respectively input into the eleventh convolution block and the twelfth convolution block to obtain corresponding feature data F3 and feature data F4, and the feature data E3 is respectively input into the thirteenth convolution block and the fourteenth convolution block to obtain corresponding feature data F5 and feature data F6; Input the feature data F1, feature data F4 and feature data F5 into the second layer stacking module for stacking to obtain feature data F7, input the feature data F2, feature data F3 and feature data F6 into the second layer stacking module for stacking to obtain feature data F8; Input the feature data F7 into the fifteenth convolution block to obtain feature data H1, and input the feature data F8 into the sixteenth convolution block to obtain feature data H2; The feature data H1 and the feature data H2 are input into the output module for processing to obtain the first feature of the output.

4. The remote sensing image fusion method based on a quaternary convolutional neural network as claimed in claim 1, characterized in that: The fusing of the first feature and the second feature to obtain the remote sensing image to be output specifically includes: The first feature is processed by the attention module and then divided into multiple groups of sub-features, and the second feature is processed by spatial feature enhancement to obtain a second feature to be fused; Input each group of sub-features into the corresponding convolution channel for convolution mapping and spectral feature enhancement to obtain the corresponding first sub-feature to be fused, wherein the structures of all convolution channels are consistent; Stacking all the first sub-features to be fused to obtain a first feature to be fused; The first feature to be fused is convolved with the second feature to be fused to obtain a remote sensing image to be output.

5. The remote sensing image fusion method based on a quaternary convolutional neural network as claimed in claim 1, characterized in that: The method also includes updating the network parameters of the quaternary convolutional neural network based on the output of the first feature enhancement module, the output of the second feature enhancement module and the output of the fusion module at the end of each training phase of the quaternary convolutional neural network.

6. A remote sensing image fusion device based on a quaternary convolutional neural network, characterized in that: The device comprises: A sampling module, used to upsample the multispectral image by a preset multiple to obtain a first image; A first feature enhancement module, used for performing feature enhancement on the first image to obtain a first feature; A second feature enhancement module, used for performing feature enhancement on the full-color image to obtain a second feature; The quaternary convolutional neural network is used to fuse the first feature and the second feature to obtain the remote sensing image to be output.

Images