Panchromatic sharpening method, apparatus, device, medium, and product

Abstract

The application provides a panchromatic sharpening method, device, equipment, medium and product. The method comprises the following steps: performing super-resolution reconstruction on a low-resolution multispectral image by using a super-resolution network to obtain a super-resolution image; performing mapping on a high-resolution panchromatic image, the super-resolution image and an original first component by using a mapping network to generate an optimized first component, wherein the original first component is obtained by performing GS transformation on the high-resolution panchromatic image and the super-resolution image; determining the trained super-resolution network and the mapping network based on the optimized first component; and performing panchromatic sharpening on to-be-processed data based on the trained super-resolution network and the mapping network.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote sensing data processing, in particular to a panchromatic sharpening method, device, equipment, medium and product. BACKGROUND

[0002] Panchromatic and multispectral fusion (pan-sharpening) of remote sensing images is a key technology aimed at improving the spatial and spectral information of images. High-resolution panchromatic images (HRPan) are usually single-band images with high spatial resolution, while low-resolution multispectral images (LRMS) provide rich spectral information but have low spatial resolution. Therefore, panchromatic sharpening is to fuse the two to obtain high-resolution multispectral images (HRMS). However, the current panchromatic sharpening methods have low performance and the resulting fusion results are relatively poor. SUMMARY

[0003] In view of the above problems, the present application provides a panchromatic sharpening method, device, equipment, medium and product, which at least solve one of the above problems.

[0004] According to a first aspect of the present application, a GS transform-based panchromatic sharpening method is provided, the method comprising:

[0005] using a super-resolution network to perform super-resolution reconstruction on a low-resolution multispectral image to obtain a super-resolution image;

[0006] using a mapping network to map a high-resolution panchromatic image, the super-resolution image and an original first component to generate an optimized first component, the original first component being obtained by GS transform based on the high-resolution panchromatic image and the super-resolution image;

[0007] based on the optimized first component, determining a trained super-resolution network and mapping network;

[0008] based on the trained super-resolution network and mapping network, performing panchromatic sharpening on to-be-processed data.

[0009] According to an embodiment of the present application, the original first component is obtained based on the high-resolution panchromatic image and the super-resolution image, comprising:

[0010] performing GS transform on the high-resolution panchromatic image as a first band and the super-resolution image as a subsequent band to obtain the original first component.

[0011] According to the embodiment of the present application, the determining the trained super-resolution network and the mapping network based on the optimized first component comprises:

[0012] The GS inverse transform is performed on the optimized first component to replace the original first component, and a fused multi-band image is obtained;

[0013] The second band and the subsequent bands of the fused multi-band image are taken as a high-spatial-resolution multi-spectral image;

[0014] A spatial loss function is constructed according to the high-resolution panchromatic image and the high-spatial-resolution multi-spectral image, and a spectral loss function is constructed according to the low-resolution multi-spectral image LRMS and the high-spatial-resolution multi-spectral image;

[0015] The super-resolution network and the mapping network are trained together through the spatial loss function and the spectral loss function, and the trained super-resolution network and the mapping network are obtained.

[0016] According to the embodiment of the present application, the method further comprises:

[0017] At least one set of training data pairs is constructed, the training data pair comprising a multi-spectral image and a down-sampled image, the down-sampled image being obtained by down-sampling the multi-spectral image, and the training data pair being used for training the super-resolution network.

[0018] According to the embodiment of the present application, the down-sampling of the multi-spectral image comprises:

[0019] The low-resolution multi-spectral image is down-sampled by a preset multiple to obtain the down-sampled image, and the preset multiple is the spatial resolution ratio of the high-resolution panchromatic image to the low-resolution multi-spectral image.

[0020] According to the embodiment of the present application, the method further comprises:

[0021] The down-sampled image is input into the super-resolution network, and a prediction result is output;

[0022] The similarity between the prediction result and the low-resolution multi-spectral image is calculated as a loss function of the super-resolution network;

[0023] The low-resolution multi-spectral image is input into the super-resolution network to obtain a super-resolution image.

[0024] According to a second aspect of the present application, a panchromatic sharpening device based on GS transform is provided, the device comprising:

[0025] The super-resolution module is configured to perform super-resolution reconstruction on the low-resolution multispectral image using a super-resolution network to obtain a super-resolution image.

[0026] The mapping module is configured to map the high-resolution panchromatic image, the super-resolution image and an original first component using a mapping network to generate an optimized first component, the original first component being obtained based on the high-resolution panchromatic image and the super-resolution image.

[0027] The determining module is configured to determine the super-resolution network and the mapping network that have completed training based on the optimized first component.

[0028] The sharpening module is configured to perform panchromatic sharpening on the to-be-processed data based on the super-resolution network and the mapping network that have completed training.

[0029] According to the panchromatic sharpening method, device, equipment, medium and product provided by the embodiments of the present application, after GS transformation, the relationship between the high-resolution panchromatic image and the optimized first component is not a simple linear transformation relationship, and in the traditional method, the high-resolution panchromatic image is usually linearly mapped into the optimized first component. However, the embodiments of the present application use the nonlinear expression capability of the deep neural network to map the high-resolution panchromatic image into the optimized first component using the mapping network, taking into account the complex mapping relationship between the two, and improving the fusion effect. BRIEF DESCRIPTION OF DRAWINGS

[0030] The above content and other purposes, features and advantages of the present application will be more apparent from the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0031] Figure 1 A flowchart of the GS transformation-based panchromatic sharpening method according to an embodiment of the present application is schematically shown;

[0032] Figure 2 An architecture diagram of the GS transformation-based panchromatic sharpening method according to an embodiment of the present application is schematically shown;

[0033] Figure 3 A structure diagram of the GS transformation-based panchromatic sharpening device according to an embodiment of the present application is schematically shown;

[0034] Figure 4 A block diagram of an electronic device suitable for implementing the GS transformation-based panchromatic sharpening method according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0035] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely illustrative of the present application and in no way limits the scope of the present application. In the following detailed description of the embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that one or more embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and functions have been omitted or simply referenced in order not to obscure an understanding of this concept of the present application.

[0036] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including", means the inclusion of but not limited to, that is, there are other items not listed.

[0037] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present specification, and should not be interpreted in an idealized or overly formal way.

[0038] In the case of using expressions similar to "at least one of A, B, and C, etc.", it is generally to be interpreted as including any one of A, B, and C, etc., or a combination thereof. For example, "a system having at least one of A, B, and C" should be interpreted as including a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C together, etc.

[0039] At present, the panchromatic sharpening methods are mainly divided into four categories: 1) component substitution methods (CS), 2) multiresolution analysis (MRA), 3) variational optimization (VO), 4) deep learning (DL) based fusion algorithms. Generally, the first three methods are considered as traditional methods. The traditional methods have low performance, and the obtained fusion results are relatively poor, and the traditional methods can usually only complete linear mapping. In the GS fusion method, both the HRPan to the GS first component are linear mapping, and in fact, it is not a simple linear mapping between the two. Therefore, the performance of the traditional component substitution method is relatively low.

[0040] In the deep learning-based method, the supervised method uses a simulation data set for training, which cannot fully reflect the situation of the real data set, thus having a scale generalization problem. The unsupervised method utilizes the real data set, but such a method needs to finely adjust the hyperparameters. The method combining the supervised and unsupervised methods uses the same network to train on the simulation data set and the real data set, and the training strategy is relatively complex.

[0041] Figure 1 A flowchart of a GS transform-based panchromatic sharpening method according to an embodiment of the present application is schematically shown.

[0042] As shown in Figure 1 , the GS transform-based panchromatic sharpening method includes operation S110-operation S140.

[0043] In operation S110, a super-resolution network is used to perform super-resolution reconstruction on a low-resolution multispectral image to obtain a super-resolution image.

[0044] The super-resolution network (SRNet) is used to improve the low-resolution multispectral image to a high resolution to generate a super-resolution image, so that the super-resolution image is aligned with the high-resolution panchromatic image in spatial resolution. The SRNet can adopt a lightweight super-resolution network, such as ESPCN, SRCNN, EDSR, etc.

[0045] In operation S120, a mapping network is used to map the high-resolution panchromatic image, the super-resolution image, and the original first component to generate an optimized first component.

[0046] The original first component is obtained by GS transform based on the high-resolution panchromatic image and the super-resolution image.

[0047] The mapping network can be a deep neural network, such as U-Net, ResNet, etc. The mapping network is used to nonlinearly map the high-resolution panchromatic image and the original first component to a more optimal component, i.e., the optimized first component.

[0048] In operation S130, based on the optimized first component, the super-resolution network and the mapping network trained are determined.

[0049] In operation S140, based on the super-resolution network and the mapping network trained, panchromatic sharpening is performed on the data to be processed.

[0050] According to the embodiment of the present application, after the GS transformation, the high-resolution panchromatic image and the optimized first component are not in a simple linear transformation relationship, and in the traditional method, the high-resolution panchromatic image is usually linearly mapped to the optimized first component. However, the embodiment of the present application uses the nonlinear expression capability of the deep neural network to use a mapping network to map the high-resolution panchromatic image to the optimized first component, taking into account the complex mapping relationship between the two, and improving the fusion effect.

[0051] Figure 2 An architecture diagram of the GS transformation-based panchromatic sharpening method according to the embodiment of the present application is schematically shown.

[0052] As Figure 2 shown, based on the architecture diagram of the GS transformation-based panchromatic sharpening method, the GS transformation-based panchromatic sharpening method provided by the embodiment can include operations S1-S9.

[0053] In operation S1, low-resolution multispectral images (LRMS) are down-sampled.

[0054] Suppose the resolution difference between the high-resolution panchromatic image (HRPan) and the LRMS is R. First, the LRMS is down-sampled R times to obtain a down-sampled image D-LRMS.

[0055] In some embodiments, based on the LRMS and the D-LRMS, a first image pair can be generated to construct at least one set of training data pairs. The at least one set of training data pairs is used to train the super-resolution network.

[0056] In operation S2, a super-resolution network is constructed.

[0057] The input of the super-resolution network is the D-LRMS, and the output is the prediction result F-LRMS of the network. The similarity between F-LRMS and LRMS can be calculated as the loss function of the super-resolution network.

[0058] In operation S3, the LRMS is input into the super-resolution network to obtain a super-resolution image SR-LRMS.

[0059] It can be understood that the resolution of the SR-LRMS is consistent with that of the HRPan.

[0060] In operation S4, GS (Gram-Schmidt) transformation is performed.

[0061] A new multi-band image is constructed with HRPan as the first band and the bands of SR-LRMS as the subsequent bands. GS orthogonal transform is performed on the multi-band image to obtain the original first component GS1 (highly related to HRPan) and GS2 to GS n Other components (containing multispectral information).

[0062] In operation S5, a mapping network is constructed.

[0063] The input of the mapping network is HRPan, SR-LRMS and GS1. The output of the mapping network is the optimized first component GS1'. The optimized first component GS1' can better fuse the spatial details of HRPan and the spectral information of SR-LRMS.

[0064] It can be understood that the mapping network can combine the spatial information of HRPan with the original first component GS1 to generate the optimized first component GS1'.

[0065] In operation S6, GS inverse transform is performed.

[0066] The original first component GS1 is replaced by the optimized first component GS1' generated by the mapping network, and then GS inverse transform is performed to obtain the fused multi-band image. The second band and the subsequent bands of the fused multi-band image are taken as HRMS (retaining the fused high-resolution multispectral information).

[0067] In operation S7, spatial loss function and spectral loss function are calculated.

[0068] According to HRPan and HRMS, a spatial loss function is constructed to ensure that HRMS retains the spatial details of HRPan, such as gradient difference loss. According to LRMS and HRMS, a spectral loss function is constructed to ensure spectral fidelity.

[0069] In operation S8, the network is trained.

[0070] According to the process of operation S1 to operation S7, the super-resolution network and the mapping network are trained until convergence.

[0071] In operation S9, the network is applied.

[0072] Using the trained super-resolution network and mapping network, the process of operation S1 to operation S7 is used to panchromatic sharpen the data to be processed. The data to be processed includes LRMS and HRPan.

[0073] Specifically, the LRMS is super-resolution reconstructed using the trained super-resolution network to obtain an SR-LRMS; the HRPan, the SR-LRMS and the GS1 are mapped using the trained mapping network to generate a GS1', which is obtained based on the HRPan and the SR-LRMS; and the HRMS is obtained based on the GS1'.

[0074] According to the embodiment of the present application, the super-resolution network is trained in a supervised manner, the mapping network is trained in an unsupervised manner, the supervised and unsupervised methods are combined, and the supervised and unsupervised networks are separately separated. In this way, the strategy of separating the supervised and unsupervised training not only takes advantage of the combination of the supervised and unsupervised methods, but also can adjust the super-parameters of the supervised and unsupervised methods separately, thereby simplifying the training strategy. Since the supervised and unsupervised networks are separated, the loss function of the method can be designed using the existing loss functions of the supervised and unsupervised methods, thereby improving the network performance.

[0075] Figure 3 A framework diagram of the GS transform-based panchromatic sharpening device according to the embodiment of the present application is schematically shown.

[0076] As shown in Figure 3 The device 300 includes a super-resolution module 310, a mapping module 320, a determination module 330 and a sharpening module 340.

[0077] The super-resolution module 310 is configured to perform super-resolution reconstruction on a low-resolution multispectral image using a super-resolution network to obtain a super-resolution image. The super-resolution module 310 can perform, for example, operation S110, which will not be described herein again.

[0078] The mapping module 320 is configured to map a high-resolution panchromatic image, a super-resolution image and an original first component using a mapping network to generate an optimized first component, the original first component being obtained based on the high-resolution panchromatic image and the super-resolution image. The mapping module 320 can perform, for example, operation S120, which will not be described herein again.

[0079] The determination module 330 is configured to determine the trained super-resolution network and the mapping network based on the optimized first component. The determination module 330 can perform, for example, operation S130, which will not be described herein again.

[0080] The sharpening module 340 is configured to perform panchromatic sharpening on the to-be-processed data based on the trained super-resolution network and the mapping network. The sharpening module 140 can perform, for example, operation S140, which will not be described herein again.

[0081] According to the embodiment of the present application, the original first component is obtained based on the high-resolution panchromatic image and the super-resolution image, including: performing GS transformation on the high-resolution panchromatic image as a first waveband and the super-resolution image as a subsequent waveband to obtain the original first component.

[0082] According to the embodiment of the present application, the super-resolution network and the mapping network trained are determined based on the optimized first component, including: performing GS inverse transformation on the original first component replaced by the optimized first component to obtain a fused multi-waveband image; taking the second waveband and the subsequent waveband of the fused multi-waveband image as a high-spatial-resolution multi-spectral image; constructing a spatial loss function according to the high-resolution panchromatic image and the high-spatial-resolution multi-spectral image, and constructing a spectral loss function according to the low-resolution multi-spectral image LRMS and the high-spatial-resolution multi-spectral image; training the super-resolution network and the mapping network through the spatial loss function and the spectral loss function to obtain the super-resolution network and the mapping network trained.

[0083] According to the embodiment of the present application, the device 300 further includes a construction module configured to construct at least one set of training data pairs, the training data pair including a multi-spectral image and a down-sampled image, the down-sampled image being obtained by down-sampling the multi-spectral image, and the training data pair being used to train the super-resolution network.

[0084] According to the embodiment of the present application, the down-sampled image obtained by down-sampling the multi-spectral image includes: performing down-sampling on the low-resolution multi-spectral image by a preset multiple to obtain the down-sampled image, and the preset multiple being a spatial resolution ratio of the high-resolution panchromatic image to the low-resolution multi-spectral image.

[0085] According to the embodiment of the present application, the device 300 further includes: a first input module configured to input the down-sampled image into the super-resolution network to output a prediction result; a calculation module configured to calculate a similarity between the prediction result and the low-resolution multi-spectral image as a loss function of the super-resolution network; and a second input module configured to input the low-resolution multi-spectral image into the super-resolution network to obtain a super-resolution image.

[0086] Any of the modules, sub-modules, units, sub-units according to the embodiments of the present application, or at least part of functions of any of them, can be implemented in one module. Any of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be split into multiple modules for implementation. Any of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be implemented at least in part as a hardware circuit, for example, a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware by integrating or packaging circuits, or in any one of software, hardware and firmware implementation ways or in a proper combination of any of them. Alternatively, one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be implemented at least in part as computer program modules, which can perform corresponding functions when executed.

[0087] For example, any of the super-resolution module 310, the mapping module 320, the determining module 330 and the sharpening module 340 can be combined in one module for implementation, or any of them can be split into multiple modules. Alternatively, at least part of functions of one or more of the modules can be combined with at least part of functions of other modules, and implemented in one module. According to the embodiments of the present application, at least one of the super-resolution module 310, the mapping module 320, the determining module 330 and the sharpening module 340 can be implemented at least in part as a hardware circuit, for example, a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware by integrating or packaging circuits, or in any one of software, hardware and firmware implementation ways or in a proper combination of any of them. Alternatively, at least one of the super-resolution module 310, the mapping module 320, the determining module 330 and the sharpening module 340 can be implemented at least in part as computer program modules, which can perform corresponding functions when executed.

[0088] According to the embodiments of the present application, the present application further provides a readable storage medium and a computer program product.

[0089] According to the embodiments of the present application, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to make a computer execute the method as above.

[0090] According to an embodiment of the present application, a computer program product comprises a computer program which, when executed by a processor, implements the method as described above.

[0091] It should be noted that the display control device part in the embodiments of the present application corresponds to the display control method part in the embodiments of the present application, and the description of the display control system part is specifically referred to the display control method part, which will not be repeated here.

[0092] Figure 4 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is schematically shown. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0093] As shown in Figure 4 The electronic device 400 according to an embodiment of the present application includes a processor 401 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded from a storage part 408 into a random access memory (RAM) 403. The processor 401 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 can also include an on-board memory for cache use. The processor 401 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present application.

[0094] In the RAM 403, various programs and data required for the operation of the electronic device 400 are stored. The processor 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. The processor 401 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 402 and / or the RAM 403. It should be noted that the programs can also be stored in one or more memories other than the ROM 402 and the RAM 403. The processor 401 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0095] According to an embodiment of the present application, the electronic device 400 can further include an input / output (I / O) interface 405 that is also connected to the bus 404. The system 400 can further include one or more of the following components connected to the input / output (I / O) interface 405: an input part 406 including a keyboard, a mouse, etc.; an output part 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 408 including a hard disk, etc.; and a communication part 409 including a network interface card such as a LAN card, a modem, etc. The communication part 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as necessary. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 410 as necessary, so that a computer program read therefrom is installed in the storage part 408 as necessary.

[0096] According to an embodiment of the present application, the method flow according to the embodiments of the present application can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the processor 401, the above-described functions defined in the system implementing the embodiments of the present application are performed. According to an embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0097] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which when executed, implement the method according to the embodiments of the present application.

[0098] According to an embodiment of the present application, the computer readable storage medium can be a non-transitory computer readable storage medium. For example, it can include, but is not limited to, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0099] For example, according to an embodiment of the present application, the computer readable storage medium can include one or more memories of ROM 402 and / or RAM 403 and / or other than ROM 402 and RAM 403 described above.

[0100] Embodiments of the present application also include a computer program product, which includes a computer program containing program codes for executing the method provided by the embodiments of the present application, and when the computer program product is run on an electronic device, the program codes are used to make the electronic device implement the method provided by the embodiments of the present application.

[0101] When the computer program is executed by the processor 401, the above-mentioned functions defined in the system / apparatus of the embodiments of the present application are executed. According to an embodiment of the present application, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0102] In one embodiment, the computer program can rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program can also be transmitted, distributed, downloaded and installed in the form of signals on network media, and be downloaded and installed through the communication part 409, and / or installed from the detachable medium 411. The program codes contained in the computer program can be transmitted by any appropriate network media, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0103] According to embodiments of the present application, program code for implementing the computer programs provided by embodiments of the present application can be written in any combination of one or more programming languages, and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The programming language includes, but is not limited to, such languages as Java, C++, python, "C" language, or the like. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0104] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0105] The embodiments of the present application described above are merely intended to illustrate the present application. These embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present application. Although the above describes each embodiment separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present application, and these substitutions and modifications should fall within the scope of the present application.

Claims

1. A method for full-color sharpening, characterized in that, The method includes: A super-resolution network is used to perform super-resolution reconstruction on a low-resolution multispectral image to obtain a super-resolution image. A mapping network is used to map the high-resolution panchromatic image, the super-resolution image, and the original first component to generate an optimized first component. The original first component is obtained by performing a GS transform on the high-resolution panchromatic image and the super-resolution image. Based on the optimized first component, the trained super-resolution network and mapping network are determined. Based on the trained super-resolution network and mapping network, full-color sharpening is performed on the data to be processed; The step of determining the trained super-resolution network and mapping network based on the optimized first component includes: replacing the original first component with the optimized first component and performing an inverse GS transform to obtain a fused multi-band image; taking the second and subsequent bands of the fused multi-band image as a high spatial resolution multispectral image; constructing a spatial loss function based on the high-resolution panchromatic image and the high spatial resolution multispectral image, and constructing a spectral loss function based on the low-resolution multispectral image LRMS and the high spatial resolution multispectral image; and training the super-resolution network and mapping network together using the spatial loss function and the spectral loss function to obtain the trained super-resolution network and mapping network.

2. The full-color sharpening method according to claim 1, characterized in that, The original first component, derived from the high-resolution panchromatic image and the super-resolution image, includes: The original first component is obtained by performing a GS transform on the high-resolution panchromatic image as the first band and the super-resolution image as the subsequent band.

3. The full-color sharpening method according to claim 1, characterized in that, The method further includes: Construct at least one training data pair, the training data pair including a multispectral image and a downsampled image, the downsampled image being obtained by downsampling the multispectral image, the training data pair being used to train the super-resolution network.

4. The full-color sharpening method according to claim 3, characterized in that, The downsampled image is obtained by downsampling the multispectral image, including: The low-resolution multispectral image is downsampled by a preset factor to obtain the downsampled image, where the preset factor is the spatial resolution ratio between the high-resolution panchromatic image and the low-resolution multispectral image.

5. The full-color sharpening method according to claim 3 or 4, characterized in that, The method further includes: The downsampled image is input into the super-resolution network, which outputs the prediction result. The similarity between the predicted result and the low-resolution multispectral image is calculated and used as the loss function of the super-resolution network; The low-resolution multispectral image is input into the super-resolution network to obtain the super-resolution image.

6. A full-color sharpening device, characterized in that, The device includes: The super-resolution module is used to perform super-resolution reconstruction of low-resolution multispectral images using a super-resolution network to obtain super-resolution images. The mapping module is used to map the high-resolution panchromatic image, the super-resolution image, and the original first component using a mapping network to generate an optimized first component, wherein the original first component is obtained based on the high-resolution panchromatic image and the super-resolution image; The determination module is used to determine the trained super-resolution network and mapping network based on the optimized first component; The sharpening module is used to perform full-color sharpening on the data to be processed based on the trained super-resolution network and mapping network. The step of determining the trained super-resolution network and mapping network based on the optimized first component includes: replacing the original first component with the optimized first component and performing an inverse GS transform to obtain a fused multi-band image; taking the second and subsequent bands of the fused multi-band image as a high spatial resolution multispectral image; constructing a spatial loss function based on the high-resolution panchromatic image and the high spatial resolution multispectral image, and constructing a spectral loss function based on the low-resolution multispectral image LRMS and the high spatial resolution multispectral image; and training the super-resolution network and mapping network together using the spatial loss function and the spectral loss function to obtain the trained super-resolution network and mapping network.

7. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 5.

Images