Panchromatic sharpening method, device, equipment, medium and product

Abstract

The invention provides a panchromatic sharpening method, device and equipment, a medium and a product, and the method comprises the steps: carrying out the super-resolution reconstruction of a low-resolution multispectral image through a super-resolution network, and obtaining a super-resolution image; a high-resolution panchromatic image, the super-resolution image and an original first component are mapped through a mapping network, an optimized first component is generated, and the original first component is obtained through GS transformation based on the high-resolution panchromatic image and the super-resolution image; based on the optimized first component, determining a trained super-resolution network and a mapping network; and carrying out panchromatic sharpening on to-be-processed data based on the trained super-resolution network and mapping network.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing data processing, and specifically to a panchromatic sharpening method, apparatus, device, medium and product. Background Art

[0002] The panchromatic and multispectral fusion (pansharpening) of remote sensing images is a key technology aimed at enhancing the spatial and spectral information of images. High-resolution panchromatic images (HRPan) are usually single-band images with high spatial resolution, while low-resolution multispectral images (LRMS) provide rich spectral information but have low spatial resolution. Therefore, panchromatic sharpening is to fuse the two to obtain high-resolution multispectral images (HRMS). However, the current panchromatic sharpening methods have low performance, and the obtained fusion results are relatively poor. Summary of the Invention

[0003] In view of the above problems, the present invention provides a panchromatic sharpening method, apparatus, device, medium and product, which can at least solve one of the above problems.

[0004] According to the first aspect of the present invention, a panchromatic sharpening method based on the GS transform is provided, and the method includes:

[0005] Performing super-resolution reconstruction on the low-resolution multispectral image using a super-resolution network to obtain a super-resolution image;

[0006] Using a mapping network to map the high-resolution panchromatic image, the super-resolution image and the original first component to generate an optimized first component, where the original first component is obtained by performing a GS transform based on the high-resolution panchromatic image and the super-resolution image;

[0007] Based on the optimized first component, determining the trained super-resolution network and mapping network;

[0008] Based on the trained super-resolution network and mapping network, performing panchromatic sharpening on the data to be processed.

[0009] According to an embodiment of the present invention, the obtaining of the original first component based on the high-resolution panchromatic image and the super-resolution image includes:

[0010] Taking the high-resolution panchromatic image as the first band and the super-resolution image as the subsequent bands to perform a GS transform to obtain the original first component.

[0011] According to an embodiment of the present invention, determining the trained super-resolution network and mapping network based on the optimized first component includes:

[0012] Performing an inverse GS transform by replacing the original first component with the optimized first component to obtain a fused multi-band image;

[0013] Taking the second and subsequent bands of the fused multi-band image as a high-spatial-resolution multi-spectral image;

[0014] Constructing a spatial loss function based on the high-resolution panchromatic image and the high-spatial-resolution multi-spectral image, and constructing a spectral loss function based on the low-resolution multi-spectral image LRMS and the high-spatial-resolution multi-spectral image;

[0015] Jointly training the super-resolution network and the mapping network through the spatial loss function and the spectral loss function to obtain the trained super-resolution network and mapping network.

[0016] According to an embodiment of the present invention, the method further includes:

[0017] Constructing at least one set of training data pairs, where each training data pair includes a multi-spectral image and a downsampled image, and the downsampled image is obtained by downsampling the multi-spectral image, and the training data pair is used to train the super-resolution network.

[0018] According to an embodiment of the present invention, the downsampled image is obtained by downsampling the multi-spectral image, which includes:

[0019] Performing downsampling on the low-resolution multi-spectral image by a preset multiple to obtain the downsampled image, and the preset multiple is the spatial resolution ratio of the high-resolution panchromatic image to the low-resolution multi-spectral image.

[0020] According to an embodiment of the present invention, the method further includes:

[0021] Inputting the downsampled image into the super-resolution network to output a prediction result;

[0022] Calculating the similarity between the prediction result and the low-resolution multi-spectral image as the loss function of the super-resolution network;

[0023] Inputting the low-resolution multi-spectral image into the super-resolution network to obtain a super-resolution image.

[0024] According to a second aspect of the present invention, there is provided a panchromatic sharpening device based on the GS transform, and the device includes:

[0025] A super-resolution module, configured to perform super-resolution reconstruction on a low-resolution multispectral image using a super-resolution network to obtain a super-resolution image;

[0026] A mapping module, configured to map a high-resolution panchromatic image, the super-resolution image, and an original first component using a mapping network to generate an optimized first component, where the original first component is obtained based on the high-resolution panchromatic image and the super-resolution image;

[0027] A determination module, configured to determine a trained super-resolution network and a mapping network based on the optimized first component.

[0028] A sharpening module, configured to perform panchromatic sharpening on data to be processed based on the trained super-resolution network and mapping network.

[0029] According to the panchromatic sharpening method, apparatus, device, medium, and product provided by embodiments of the present invention, after the GS transform, the relationship between the high-resolution panchromatic image and the optimized first component is not a simple linear transformation relationship. In traditional methods, the high-resolution panchromatic image is usually linearly mapped to the optimized first component. However, embodiments of the present invention utilize the non-linear expression ability of a deep neural network and use a mapping network to map the high-resolution panchromatic image to the optimized first component, taking into account the complex mapping relationship between the two and improving the fusion effect. Description of the Drawings

[0030] Through the following description of embodiments of the present invention with reference to the drawings, the above content and other objects, features, and advantages of the present invention will become clearer. In the drawings:

[0031] Figure 1 Schematically shows a flowchart of a panchromatic sharpening method based on the GS transform according to an embodiment of the present invention;

[0032] Figure 2 Schematically shows an architecture diagram of a panchromatic sharpening method based on the GS transform according to an embodiment of the present invention;

[0033] Figure 3 Schematically shows a structural diagram of a panchromatic sharpening apparatus based on the GS transform according to an embodiment of the present invention;

[0034] Figure 4 Schematically shows a block diagram of an electronic device suitable for implementing a panchromatic sharpening method based on the GS transform according to an embodiment of the present invention. Detailed Embodiments

[0035] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In the following detailed description, for the sake of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present invention. However, it is obvious that one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present invention.

[0036] The terms used herein are merely for describing specific embodiments and are not intended to limit the present invention. The terms "comprising", "including" and the like used herein indicate the presence of the described features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0037] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0038] In cases where expressions similar to "at least one of A, B, and C, etc." are used, generally, it should be interpreted according to the meaning that those skilled in the art usually understand such expressions (for example, "a system having at least one of A, B, and C" should include, but not be limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0039] Currently, panchromatic sharpening methods are mainly divided into four categories: 1) component substitution methods (CS), 2) multiresolution analysis (MRA), 3) variational optimization (VO), and 4) deep learning (DL)-based fusion algorithms. Generally, the first three methods are considered traditional methods. The performance of traditional methods is low, the obtained fusion results are relatively poor, and traditional methods usually can only complete linear mapping. In the GS fusion method, the mapping from HRPan to the first component of GS is linear, but in fact, the relationship between the two is not a simple linear mapping. Therefore, the performance of the traditional component substitution method is relatively low.

[0040] In deep learning-based methods, supervised methods use simulated datasets for training. Since simulated datasets cannot fully reflect the situation of real datasets, there is a problem of scale generalization. Unsupervised methods utilize real datasets. However, such methods require fine-tuning of hyperparameters. Methods combining supervised and unsupervised approaches use the same network to train on simulated and real datasets, and the training strategy is relatively complex.

[0041] Figure 1 A flowchart of a panchromatic sharpening method based on the GS transform according to an embodiment of the present invention is schematically shown.

[0042] As Figure 1 shown, the panchromatic sharpening method based on the GS transform includes operations S110 - S140.

[0043] In operation S110, a super-resolution network is used to perform super-resolution reconstruction on a low-resolution multispectral image to obtain a super-resolution image.

[0044] The super-resolution network (SRNet) is used to upscale a low-resolution multispectral image to a high resolution, generating a super-resolution image that is spatially aligned with a high-resolution panchromatic image. SRNet can adopt lightweight super-resolution networks such as ESPCN, SRCNN, EDSR, etc.

[0045] In operation S120, a mapping network is used to map the high-resolution panchromatic image, the super-resolution image, and the original first component to generate an optimized first component.

[0046] Among them, the original first component is obtained by performing a GS transform on the high-resolution panchromatic image and the super-resolution image.

[0047] The mapping network can be a deep neural network such as U-Net, ResNet, etc. The mapping network is used to non-linearly map the high-resolution panchromatic image and the original first component to a better component, that is, the optimized first component.

[0048] In operation S130, based on the optimized first component, a trained super-resolution network and a mapping network are determined.

[0049] In operation S140, based on the trained super-resolution network and mapping network, panchromatic sharpening is performed on the data to be processed.

[0050] According to an embodiment of the present invention, after the GS transformation, the relationship between the high-resolution panchromatic image and the optimized first component is not a simple linear transformation. In traditional methods, the high-resolution panchromatic image is usually linearly mapped to the optimized first component. However, the embodiment of the present invention utilizes the non-linear expression ability of the deep neural network and uses a mapping network to map the high-resolution panchromatic image to the optimized first component, taking into account the complex mapping relationship between the two and improving the fusion effect.

[0051] Figure 2 Schematically shows an architecture diagram of a panchromatic sharpening method based on the GS transformation according to an embodiment of the present invention.

[0052] As Figure 2 shown, based on the architecture diagram of the panchromatic sharpening method based on the GS transformation, the panchromatic sharpening method provided in this embodiment may include operations S1-S9.

[0053] In operation S1, downsample the low-resolution multispectral images (LRMS).

[0054] Assume that the resolution difference between the high-resolution panchromatic images (HRPan) and LRMS is R. First, downsample LRMS by a factor of R to obtain the downsampled image D-LRMS.

[0055] In some embodiments, based on LRMS and D-LRMS, a first image pair may be generated to construct at least one set of training data pairs. The at least one set of training data pairs is used to train the super-resolution network.

[0056] In operation S2, construct a super-resolution network.

[0057] The input of the super-resolution network is D-LRMS, and the output is the prediction result F-LRMS of the network. The similarity between F-LRMS and LRMS can be calculated as the loss function of the super-resolution network.

[0058] In operation S3, input LRMS into the super-resolution network to obtain the super-resolution image SR-LRMS.

[0059] It can be understood that the resolution of SR-LRMS is the same as that of HRPan.

[0060] In operation S4, perform the GS (Gram-Schmidt) transformation.

[0061] Take HRPan as the first band and the bands of SR-LRMS as subsequent bands to construct a new multi-band image. Perform GS orthogonal transformation on this multi-band image to obtain the original first component GS1 (highly correlated with HRPan) and GS2 to GS n other components (containing multispectral information).

[0062] In operation S5, construct a mapping network.

[0063] The inputs of the mapping network are HRPan, SR-LRMS, and GS1. The output of the mapping network is the optimized first component GS1'. The optimized first component GS1' can better fuse the spatial details of HRPan and the spectral information of SR-LRMS.

[0064] It can be understood that the mapping network can combine the spatial information of HRPan with the original first component GS1 to generate the optimized first component GS1'.

[0065] In operation S6, perform GS inverse transformation.

[0066] Replace the original first component GS1 with the optimized first component GS1' generated by the mapping network, and then perform GS inverse transformation to obtain the fused multi-band image. Take the second band and subsequent bands of the fused multi-band image as HRMS (retaining the fused high-resolution multispectral information).

[0067] In operation S7, calculate the spatial loss function and the spectral loss function.

[0068] Based on HRPan and HRMS, construct a spatial loss function to ensure that HRMS retains the spatial details of HRPan, such as the gradient difference loss. Based on LRMS and HRMS, construct a spectral loss function to ensure spectral fidelity.

[0069] In operation S8, train the network.

[0070] According to the process of operation S1 - operation S7, train the super-resolution network and the mapping network until convergence.

[0071] In operation S9, apply the network.

[0072] Use the trained super-resolution network and mapping network to perform pan-sharpening on the data to be processed according to the process of operation S1 - operation S7. The data to be processed includes LRMS and HRPan.

[0073] Specifically, the low-resolution multispectral image (LRMS) is super-resolution reconstructed using the trained super-resolution network to obtain the super-resolution LRMS (SR-LRMS); the trained mapping network is used to map the high-resolution panchromatic image (HRPan), SR-LRMS, and the first component GS1 to generate GS1', where GS1 is obtained by performing a GS transform based on HRPan and SR-LRMS; based on GS1', the high-resolution multispectral image (HRMS) is obtained.

[0074] According to the embodiments of the present invention, training is performed in a supervised manner in the super-resolution network and in an unsupervised manner in the mapping network. By combining supervised and unsupervised methods and separately separating the supervised and unsupervised networks, the strategy of separating supervised and unsupervised training not only gives play to the advantages of combining supervised and unsupervised methods but also allows for separately adjusting the hyperparameters of supervised and unsupervised methods, thus simplifying the training strategy. Since the supervised and unsupervised networks are separated, the existing loss functions of supervised and unsupervised methods respectively can be used to design the loss function of this method, improving the network performance.

[0075] Figure 3 The framework diagram of a panchromatic sharpening device based on GS transform according to the embodiments of the present invention is schematically shown.

[0076] As Figure 3 shown, the device 300 includes a super-resolution module 310, a mapping module 320, a determination module 330, and a sharpening module 340.

[0077] The super-resolution module 310 is configured to perform super-resolution reconstruction on a low-resolution multispectral image using a super-resolution network to obtain a super-resolution image. The super-resolution module 310 may perform, for example, operation S110, which will not be elaborated here.

[0078] The mapping module 320 is configured to use a mapping network to map a high-resolution panchromatic image, a super-resolution image, and an original first component to generate an optimized first component, where the original first component is obtained based on the high-resolution panchromatic image and the super-resolution image. The mapping module 320 may perform, for example, operation S120, which will not be elaborated here.

[0079] The determination module 330 is configured to determine the trained super-resolution network and mapping network based on the optimized first component. The determination module 330 may perform, for example, operation S130, which will not be elaborated here.

[0080] The sharpening module 340 is configured to perform panchromatic sharpening on the data to be processed based on the trained super-resolution network and mapping network. The sharpening module 140 may perform, for example, operation S140, which will not be elaborated here.

[0081] According to an embodiment of the present invention, the original first component is obtained based on a high-resolution panchromatic image and a super-resolution image, including: performing a GS transform with the high-resolution panchromatic image as the first band and the super-resolution image as subsequent bands to obtain the original first component.

[0082] According to an embodiment of the present invention, based on the optimized first component, determining the trained super-resolution network and mapping network includes: performing an inverse GS transform by replacing the original first component with the optimized first component to obtain a fused multi-band image; taking the second band and subsequent bands of the fused multi-band image as a multi-spectral image with high spatial resolution; constructing a spatial loss function according to the high-resolution panchromatic image and the multi-spectral image with high spatial resolution, and constructing a spectral loss function according to the low-resolution multi-spectral image LRMS and the multi-spectral image with high spatial resolution; jointly training the super-resolution network and the mapping network through the spatial loss function and the spectral loss function to obtain the trained super-resolution network and mapping network.

[0083] According to an embodiment of the present invention, the apparatus 300 further includes: a construction module configured to construct at least one set of training data pairs, where the training data pairs include a multi-spectral image and a downsampled image, the downsampled image is obtained by downsampling the multi-spectral image, and the training data pairs are used to train the super-resolution network.

[0084] According to an embodiment of the present invention, the downsampled image is obtained by downsampling the multi-spectral image, including: performing downsampling on the low-resolution multi-spectral image by a preset multiple to obtain the downsampled image, and the preset multiple is the spatial resolution ratio of the high-resolution panchromatic image to the low-resolution multi-spectral image.

[0085] According to an embodiment of the present invention, the apparatus 300 further includes: a first input module configured to input the downsampled image into the super-resolution network and output a prediction result; a calculation module configured to calculate the similarity between the prediction result and the low-resolution multi-spectral image as the loss function of the super-resolution network; a second input module configured to input the low-resolution multi-spectral image into the super-resolution network to obtain a super-resolution image.

[0086] Any number of modules, sub - modules, units, and sub - units according to embodiments of the present invention, or at least part of the functions of any of them, can be implemented in one module. Any one or more of the modules, sub - modules, units, and sub - units according to embodiments of the present invention can be split into multiple modules for implementation. Any one or more of the modules, sub - modules, units, and sub - units according to embodiments of the present invention can be at least partially implemented as a hardware circuit, such as a field - programmable gate array (FPGA), programmable logic array (PLA), system - on - chip, system - on - substrate, system - on - package, application - specific integrated circuit (ASIC), or can be implemented by any other reasonable way of integrating or packaging circuits, such as hardware or firmware, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, one or more of the modules, sub - modules, units, and sub - units according to embodiments of the present invention can be at least partially implemented as a computer program module, which can execute corresponding functions when the computer program module is run.

[0087] For example, any number of the super - resolution module 310, mapping module 320, determination module 330, and sharpening module 340 can be combined and implemented in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to embodiments of the present invention, at least one of the super - resolution module 310, mapping module 320, determination module 330, and sharpening module 340 can be at least partially implemented as a hardware circuit, such as a field - programmable gate array (FPGA), programmable logic array (PLA), system - on - chip, system - on - substrate, system - on - package, application - specific integrated circuit (ASIC), or can be implemented by any other reasonable way of integrating or packaging circuits, such as hardware or firmware, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, at least one of the super - resolution module 310, mapping module 320, determination module 330, and sharpening module 340 can be at least partially implemented as a computer program module, which can execute corresponding functions when the computer program module is run.

[0088] According to embodiments of the present invention, the present invention also provides a readable storage medium and a computer program product.

[0089] According to embodiments of the present invention, a non - transitory computer - readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0090] According to an embodiment of the present invention, a computer program product includes a computer program which, when executed by a processor, implements the method as described above.

[0091] It should be noted that the display control device part in the embodiment of the present invention corresponds to the display control method part in the embodiment of the present invention. For the description of the display control system part, please refer to the display control method part specifically, and details will not be repeated here.

[0092] Figure 4 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present invention is schematically shown. Figure 4 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0093] As Figure 4 shown, the electronic device 400 according to an embodiment of the present invention includes a processor 401 which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage section 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), and so on. The processor 401 may also include on-board memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0094] In the RAM 403, various programs and data required for the operation of the electronic device 400 are stored. The processor 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. The processor 401 performs various operations of the method flow according to an embodiment of the present invention by executing the programs in the ROM 402 and / or the RAM 403. It should be noted that the programs may also be stored in one or more memories other than the ROM 402 and the RAM 403. The processor 401 may also perform various operations of the method flow according to an embodiment of the present invention by executing the programs stored in the one or more memories.

[0095] According to an embodiment of the present invention, the electronic device 400 may further include an input / output (I / O) interface 405, and the input / output (I / O) interface 405 is also connected to the bus 404. The system 400 may further include one or more of the following components connected to the input / output (I / O) interface 405: an input portion 406 including a keyboard, a mouse, etc.; an output portion 407 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage portion 408 including a hard disk, etc.; and a communication portion 409 including a network interface card such as a LAN card, a modem, etc. The communication portion 409 performs communication processing via a network such as the Internet. A driver 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the driver 410 as needed so that a computer program read therefrom can be installed into the storage portion 408 as needed.

[0096] According to an embodiment of the present invention, the method flow according to the embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product including a computer program carried on a computer-readable storage medium, and the computer program includes program codes for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication portion 409 and / or installed from the removable medium 411. When the computer program is executed by the processor 401, the above-described functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the above-described systems, devices, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0097] The present invention also provides a computer-readable storage medium, which may be included in the device / device / system described in the above embodiments; or may exist separately without being assembled into the device / device / system. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0098] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0099] For example, according to embodiments of the present invention, the computer-readable storage medium may include one or more memories other than the above-described ROM 402 and / or RAM 403 and / or ROM 402 and RAM 403.

[0100] Embodiments of the present invention also include a computer program product, which includes a computer program that contains program code for executing the method provided by embodiments of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by embodiments of the present invention.

[0101] When the computer program is executed by the processor 401, the above functions defined in the system / apparatus of embodiments of the present invention are executed. According to embodiments of the present invention, the above-described systems, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0102] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and be downloaded and installed through the communication part 409, and / or be installed from the removable medium 411. The program code contained in the computer program can be transmitted by any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0103] In accordance with embodiments of the present invention, program code for executing the computer programs provided by the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The programming languages include, but are not limited to, programming languages such as Java, C++, Python, the "C" language, or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combinations of blocks in the block diagram or flowchart, can be implemented using a dedicated hardware-based system for performing the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions. Those skilled in the art can understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0105] The above describes the embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although the embodiments are described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present invention, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present invention.

Claims

1. A panchromatic sharpening method, characterized in that, The method includes: Performing super-resolution reconstruction on a low-resolution multispectral image using a super-resolution network to obtain a super-resolution image; Using a mapping network to map a high-resolution panchromatic image, the super-resolution image, and an original first component to generate an optimized first component, where the original first component is obtained by performing a GS transform based on the high-resolution panchromatic image and the super-resolution image; Determining a trained super-resolution network and mapping network based on the optimized first component; Performing panchromatic sharpening on the data to be processed based on the trained super-resolution network and mapping network.

2. The full-color sharpening method according to claim 1, wherein The original first component being obtained based on the high-resolution panchromatic image and the super-resolution image includes: Taking the high-resolution panchromatic image as the first band and the super-resolution image as subsequent bands to perform a GS transform to obtain the original first component.

3. The full-color sharpening method according to claim 1, wherein The determining a trained super-resolution network and mapping network based on the optimized first component includes: Performing an inverse GS transform by replacing the original first component with the optimized first component to obtain a fused multi-band image; Taking the second band and subsequent bands of the fused multi-band image as a high-spatial-resolution multispectral image; Constructing a spatial loss function based on the high-resolution panchromatic image and the high-spatial-resolution multispectral image, and constructing a spectral loss function based on the low-resolution multispectral image LRMS and the high-spatial-resolution multispectral image; Jointly training the super-resolution network and the mapping network through the spatial loss function and the spectral loss function to obtain a trained super-resolution network and mapping network.

4. The full-color sharpening method according to claim 1, characterized in that The method further includes: Constructing at least one set of training data pairs, where each training data pair includes a multispectral image and a downsampled image, and the downsampled image is obtained by downsampling the multispectral image, and the training data pair is used to train the super-resolution network.

5. The panchromatic sharpening method according to claim 4, wherein The downsampled image being obtained by downsampling the multispectral image includes: Performing downsampling on the low-resolution multispectral image by a preset multiple to obtain the downsampled image, where the preset multiple is the spatial resolution ratio of the high-resolution panchromatic image to the low-resolution multispectral image.

6. The panchromatic sharpening method according to claim 4 or 5, characterized in that, The method further includes: Inputting the downsampled image into the super-resolution network and outputting a prediction result; Calculating the similarity between the prediction result and the low-resolution multispectral image as the loss function of the super-resolution network; Inputting the low-resolution multispectral image into the super-resolution network to obtain a super-resolution image.

7. A panchromatic sharpening device, characterized in that, The apparatus includes: A super-resolution module for performing super-resolution reconstruction on a low-resolution multispectral image using a super-resolution network to obtain a super-resolution image; A mapping module for using a mapping network to map a high-resolution panchromatic image, the super-resolution image, and an original first component to generate an optimized first component, where the original first component is obtained based on the high-resolution panchromatic image and the super-resolution image; A determining module for determining a trained super-resolution network and mapping network based on the optimized first component; A sharpening module for performing panchromatic sharpening on the data to be processed based on the trained super-resolution network and mapping network.

8. An electronic device, comprising: One or more processors; A memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instruction is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instruction is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

Images