Hyperspectral and multispectral image fusion method and system based on deep dictionary learning

Abstract

The invention relates to the technical field of remote sensing image processing, and particularly discloses a hyperspectral and multispectral image fusion method and system based on deep dictionary learning, and the method comprises the steps: obtaining a target hyperspectral image and a target multispectral image of a to-be-fused region; fusing the target hyperspectral image and the target multispectral image based on an image fusion model comprising a plurality of hybrid linear fusion modules connected in sequence to obtain a target fused image of the to-be-fused region; according to the method, the problem of non-overlapping spectral response can be effectively relieved, the problems of fuzzy fusion results and spectral distortion are reduced, and the method has the characteristics of process interpretability and light weight.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a hyperspectral and multispectral image fusion method and system based on deep dictionary learning. Background Art

[0002] Hyperspectral images possess rich spectral information and can accurately capture the spectral characteristics of ground objects. However, due to limitations in imaging principles and sensor technology, their spatial resolution is typically low. Multispectral images, on the other hand, possess higher spatial resolution but relatively limited spectral information. Fusion of hyperspectral and multispectral images can combine the advantages of both, generating high-resolution hyperspectral images that combine rich spectral information with high spatial resolution. High-quality fusion results can reduce sensor constraints and enhance the performance of remote sensing applications, such as providing a superior data foundation for land cover classification, target detection and recognition, and environmental monitoring.

[0003] Early hyperspectral and multispectral fusion methods were primarily based on dictionary learning theory. This assumes that the low-spatial-resolution hyperspectral image and the desired high-spatial-resolution hyperspectral image are described by the same dictionary, and the result is obtained through sparse coding and dictionary learning. However, while enhancing spatial detail, this often results in the destruction of the original spectral information, leading to a certain degree of spectral distortion. Subsequently, tensor decomposition techniques were introduced to mitigate this spectral distortion. Tensor decomposition directly models the three-dimensional data of hyperspectral images, better capturing the correlations between data in different dimensions. While these traditional fusion methods have achieved certain performance, they often rely on manually designed models, often require precise model assumptions, and are computationally expensive.

[0004] The hyperspectral and multispectral fusion method based on deep learning is to learn the mapping relationship between the input low spatial resolution hyperspectral image and high spatial resolution multispectral image to the target high spatial resolution hyperspectral image by constructing a deep neural network model. Hyperspectral images have significant spectral redundancy. Not only do they show a high degree of inter-band correlation in the image space, but they are also prone to redundant embedded information in the feature space, which increases the difficulty of modeling. Although the current hyperspectral and multispectral fusion method based on deep learning has improved the fusion accuracy to a certain extent, it still faces practical challenges such as limited model generalization ability and high dependence on computing resources. With the development of sensor technology, hyperspectral and multispectral images have shown more complex nonlinear relationships, which are difficult for traditional models to effectively characterize, and thus easily lead to distortion of the fusion results.

[0005] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the Invention

[0006] To solve the above technical problems, the present invention provides a method and system for fusion of hyperspectral and multispectral images based on deep dictionary learning.

[0007] In a first aspect, the present invention provides a method for fusion of hyperspectral and multispectral images based on deep dictionary learning. The technical solution of the method is as follows:

[0008] Obtaining the target hyperspectral image and target multispectral image of the area to be fused;

[0009] Based on an image fusion model comprising a plurality of sequentially connected hybrid linear fusion modules, the target hyperspectral image and the target multispectral image are fused to obtain a target fused image of the area to be fused;

[0010] Each hybrid linear fusion module includes: a dictionary generation network, a dictionary update network, an abundance update network and a linear fusion network; the dictionary generation network is used to receive the input hyperspectral image of the area to be fused and obtain the initial spectral dictionary corresponding to the hyperspectral image; the dictionary update network is used to generate a target spectral dictionary based on the target multispectral image and the initial spectral dictionary; the abundance update network is used to obtain a target abundance matrix based on the initial spectral dictionary and the reshaped hyperspectral image; the linear fusion network is used to linearly fuse the target spectral dictionary with the fine-tuned target abundance matrix to obtain a fused image and use it as the hyperspectral image of the area to be fused received by the next hybrid linear fusion module;

[0011] The fused image output by the last hybrid linear fusion module is determined as the target fused image.

[0012] The beneficial effects of the hyperspectral and multispectral image fusion method based on deep dictionary learning of the present invention are as follows:

[0013] The method of the present invention can effectively alleviate the problem of non-overlapping spectral responses, reduce the ambiguity and spectral distortion of the fusion results, and has the characteristics of process interpretability and lightweight.

[0014] On the basis of the above scheme, the hyperspectral and multispectral image fusion method based on deep dictionary learning of the present invention can also be improved as follows.

[0015] In an optional manner, the dictionary generation network is specifically used to:

[0016] The hyperspectral image is subjected to feature extraction using a dimensionality reduction convolutional neural network to obtain a feature map which is input into a reshaping function to construct the initial spectral dictionary.

[0017] In an optional manner, the abundance update network is specifically used to:

[0018] Inputting the hyperspectral image into a reshaping function to obtain the reshaped hyperspectral image;

[0019] The target abundance matrix is ​​calculated by using the least squares method and combining the initial spectral dictionary with the reshaped hyperspectral image.

[0020] In an optional manner, the dictionary update network is specifically used to:

[0021] Splicing the target multispectral image with the initial spectral dictionary to obtain splicing features;

[0022] The concatenated features are input into the transformer structure and fine-tuned in combination with the initial spectral dictionary to generate the target spectral dictionary.

[0023] In an optional manner, the linear fusion network is specifically used to:

[0024] Based on a one-dimensional attention mechanism, fine-tuning the target abundance matrix to obtain the fine-tuned target abundance matrix;

[0025] The target spectrum dictionary is linearly fused with the fine-tuned target abundance matrix to obtain the fused image.

[0026] In an optional manner, the method further includes:

[0027] Acquire a high-resolution original hyperspectral image of the area to be fused, and perform downsampling processing on the original hyperspectral image to obtain a low-resolution hyperspectral image;

[0028] An interpolation process is performed on the low-resolution hyperspectral image to obtain a low-resolution target hyperspectral image.

[0029] In a second aspect, the present invention provides a hyperspectral and multispectral image fusion system based on deep dictionary learning. The technical solution of the system is as follows:

[0030] Including: acquisition unit and fusion unit;

[0031] The acquisition unit is used to: acquire a target hyperspectral image and a target multispectral image of the area to be fused;

[0032] The fusion unit is configured to fuse the target hyperspectral image and the target multispectral image based on an image fusion model comprising a plurality of sequentially connected hybrid linear fusion modules to obtain a target fused image of the area to be fused;

[0033] Each hybrid linear fusion module includes: a dictionary generation network, a dictionary update network, an abundance update network and a linear fusion network; the dictionary generation network is used to receive the input hyperspectral image of the area to be fused and obtain the initial spectral dictionary corresponding to the hyperspectral image; the dictionary update network is used to generate a target spectral dictionary based on the target multispectral image and the initial spectral dictionary; the abundance update network is used to obtain a target abundance matrix based on the initial spectral dictionary and the reshaped hyperspectral image; the linear fusion network is used to linearly fuse the target spectral dictionary with the fine-tuned target abundance matrix to obtain a fused image and use it as the hyperspectral image of the area to be fused received by the next hybrid linear fusion module;

[0034] The fused image output by the last hybrid linear fusion module is determined as the target fused image.

[0035] The beneficial effects of the hyperspectral and multispectral image fusion system based on deep dictionary learning of the present invention are as follows:

[0036] The system of the present invention can effectively alleviate the problem of non-overlapping spectral responses, reduce the ambiguity and spectral distortion of the fusion results, and has the characteristics of process interpretability and lightweight.

[0037] Based on the above solution, the hyperspectral and multispectral image fusion system based on deep dictionary learning of the present invention can also be improved as follows.

[0038] In an optional manner, the method further includes: a pre-processing unit; the pre-processing unit is configured to:

[0039] Acquire a high-resolution original hyperspectral image of the area to be fused, and perform downsampling processing on the original hyperspectral image to obtain a low-resolution hyperspectral image;

[0040] An interpolation process is performed on the low-resolution hyperspectral image to obtain a low-resolution target hyperspectral image.

[0041] In a third aspect, the technical solution of an electronic device of the present invention is as follows:

[0042] The invention comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the hyperspectral and multispectral image fusion method based on deep dictionary learning of the present invention are realized.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0044] The computer-readable storage medium stores instructions. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the hyperspectral and multispectral image fusion method based on deep dictionary learning of the present invention.

[0045] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0047] Figure 1 Schematic diagram of a flow chart of an embodiment of a hyperspectral and multispectral image fusion method based on deep dictionary learning of the present invention;

[0048] Figure 2 It is a schematic diagram of the overall principle;

[0049] Figure 3 Schematic diagram for determining the effect of the number of hybrid linear fusion modules;

[0050] Figure 4 Schematic diagram for comparison of experimental results;

[0051] Figure 5 Schematic diagram of the structure of an embodiment of a hyperspectral and multispectral image fusion system based on deep dictionary learning of the present invention;

[0052] Figure 6 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0053] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0054] Figure 1A flow chart of an embodiment of a method for fusion of hyperspectral and multispectral images based on deep dictionary learning provided by the present invention is shown. The method for fusion of hyperspectral and multispectral images based on deep dictionary learning can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement a method for fusion of hyperspectral and multispectral images based on deep dictionary learning by calling computer-readable instructions stored in a memory through a processor. Taking the server as an example, the CPU of the server in this embodiment is Intel Xeon E5-2665, the GPU is NVIDIAGTX2080Ti, the operating system is Ubuntu 18.04, and the compilation environment is PyTorch1.1.0, Python3.5, CUDA9.0 and CUDNN7.1. As Figure 1 As shown in Figure 2, the hyperspectral and multispectral image fusion method based on deep dictionary learning includes the following steps:

[0055] S1. Obtain a target hyperspectral image and a target multispectral image of the area to be fused.

[0056] The area to be fused is the area where image fusion is required in this embodiment. Hyperspectral images are collected using a hyperspectral sensor, and multispectral images are collected using a multispectral sensor. The target hyperspectral image is a low-resolution hyperspectral image after interpolation processing. The expression of the target hyperspectral image is: N represents the number of bands of the target hyperspectral image, H and W represent the height and width of the target hyperspectral image. The expression of the target multispectral image is: n represents the number of channels of the target multispectral image.

[0057] S2. Based on an image fusion model including a plurality of sequentially connected hybrid linear fusion modules, the target hyperspectral image and the target multispectral image are fused to obtain a target fused image of the area to be fused.

[0058] Among them, Figure 2As shown, each hybrid linear fusion module includes: a dictionary generation network, a dictionary update network, an abundance update network and a linear fusion network. The dictionary generation network is used to receive the input hyperspectral image of the area to be fused and obtain the initial spectral dictionary corresponding to the hyperspectral image; the dictionary update network is used to generate a target spectral dictionary based on the target multispectral image and the initial spectral dictionary; the abundance update network is used to obtain a target abundance matrix based on the initial spectral dictionary and the reshaped hyperspectral image; the linear fusion network is used to linearly fuse the target spectral dictionary with the fine-tuned target abundance matrix to obtain a fused image and use it as the hyperspectral image of the area to be fused received by the next hybrid linear fusion module;

[0059] The fused image output by the last hybrid linear fusion module is determined as the target fused image.

[0060] It should be noted that the hyperspectral image input to the dictionary generation network in the first hybrid linear fusion module of the image fusion model is the target hyperspectral image, and the fused image output by the last hybrid linear fusion module is the target fused image. The multispectral image input to the dictionary update network in each hybrid linear fusion module is the target multispectral image, and the target multispectral image is continuously used to guide the recovery of the hyperspectral image dictionary.

[0061] In an optional manner, the dictionary generation network is specifically used to:

[0062] The hyperspectral image is subjected to feature extraction using a dimensionality reduction convolutional neural network to obtain a feature map which is input into a reshaping function to construct the initial spectral dictionary.

[0063] Among them, the expression corresponding to the dictionary generation network is: D k represents the initial spectral dictionary corresponding to the k-th hybrid linear fusion module, represents the hyperspectral image of the area to be fused received in the kth hybrid linear fusion module; f fe (.) represents a dimensionality reduction convolutional neural network, which is used to encode the N-band hyperspectral image into an m-channel feature map; Reshape(.) represents a reshaping function, which is used to construct the feature map into the initial spectral dictionary D k , D k The size is B×m×HW, where B represents the batch size.

[0064] In an optional manner, the abundance update network is specifically used to:

[0065] The hyperspectral image is input into a reshaping function to obtain the reshaped hyperspectral image.

[0066] Among them, the expression of the reshaped hyperspectral image is: represents the reshaped hyperspectral image corresponding to the k-th hybrid linear fusion module, The dimensions are B×N×HW.

[0067] The target abundance matrix is ​​calculated by using the least squares method and combining the initial spectral dictionary with the reshaped hyperspectral image.

[0068] Among them, the expression for calculating the target abundance matrix is: λ k represents the target abundance matrix corresponding to the k-th hybrid linear fusion module, α k is a learnable hyperparameter, E is the identity matrix, represents the initial spectral dictionary D k The transposed matrix of .

[0069] In an optional manner, the dictionary update network is specifically used to:

[0070] The target multispectral image is spliced ​​with the initial spectral dictionary to obtain splicing features, the splicing features are input into the transformer structure, and fine-tuned in combination with the initial spectral dictionary to generate the target spectral dictionary.

[0071] Among them, the expression for generating the target spectrum dictionary is: represents the target spectrum dictionary corresponding to the k-th hybrid linear fusion module, concat(.) represents the concatenation layer, f Dref (.) indicates the transformer structure.

[0072] It should be noted that traditional deep learning-based hyperspectral and multispectral image fusion tasks usually perform image fusion at the pixel level or feature level, while the framework proposed in this embodiment converts hyperspectral images and multispectral images into a low-dimensional dictionary space before fusion.

[0073] In an optional manner, the linear fusion network is specifically used to:

[0074] Based on a one-dimensional attention mechanism, the target abundance matrix is ​​fine-tuned to obtain the fine-tuned target abundance matrix.

[0075] Among them, the expression for generating the fine-tuned target abundance matrix is: represents the fine-tuned target abundance matrix corresponding to the k-th hybrid linear fusion module, f λref (.) represents a one-dimensional attention mechanism.

[0076] The target spectrum dictionary is linearly fused with the fine-tuned target abundance matrix to obtain the fused image.

[0077] Among them, the expression for generating the fused image is: The value of k is a positive integer between 1 and d, where d represents the total number of hybrid linear fusion modules. The hyperspectral image of the area to be fused received by the k+1th hybrid linear fusion module When k is d, Determined That is the target fused image.

[0078] It should be noted that the alternation of each hybrid linear fusion module in the image fusion model can be equivalent to an optimization process for solving the cost function. This embodiment uses the mean absolute error between the fused image and the true value as the loss function, and the expression of the loss function is: It represents the fused image corresponding to the dth (last) hybrid linear fusion module, that is, the target fused image.

[0079] In addition, the value of d is determined according to the effect during the model training process. Figure 3 As shown, the default value of d is 5. Figure 3 The horizontal axis in represents the computational complexity index, which is used to measure the computational size of the model. The larger the GFLOPS (billion floating-point operations per second) value, the higher the computational complexity of the model. Figure 3 The vertical axis represents the image quality assessment metric, which measures the similarity between the reconstructed image and the original image. A higher PSNR (Peak Signal-to-Noise Ratio) value indicates better image quality and a closer reconstruction to the original image.

[0080] In an optional manner, the method further includes:

[0081] A high-resolution original hyperspectral image of the area to be fused is obtained, and the original hyperspectral image is downsampled to obtain a low-resolution hyperspectral image.

[0082] An interpolation process is performed on the low-resolution hyperspectral image to obtain a low-resolution target hyperspectral image.

[0083] Among them, the expression of the original hyperspectral image is I HR , the expression of low-resolution hyperspectral image is

[0084] It should be noted that if Figure 4 As shown, Figure 4 The first line in the figure is the hyperspectral image restored by deep learning method. Figure 4 The second row in is the error map between the restored image and the true high-resolution hyperspectral image. Among them, the last column is the image restored using the fusion method in this embodiment. The less content contained in the error map, the smaller the difference between the restored image and the original image, that is, the higher the reconstruction quality and the better the fidelity of the spectral and spatial information. The comparison method selected a representative fusion method that has performed well in recent years. Judging from the detail retention of the restored image and the cleanliness of the error map, this embodiment is significantly superior to the existing methods in both visual effects and error control, reflecting good fusion performance and method advantages.

[0085] The technical solution of this embodiment abandons the traditional high-coupling fusion mode in the feature and image space, and constructs a maximum a posteriori probability model guided by the generalized mixed information model. Guided by this solution process, information fusion is performed in a low-dimensional dictionary space, which can effectively alleviate the problem of non-overlapping spectral responses, reduce the ambiguity and spectral distortion of the fusion results, and has the characteristics of interpretability and lightweight process.

[0086] Figure 5 FIG. 2 shows a schematic structural diagram of an embodiment of a hyperspectral and multispectral image fusion system 200 based on deep dictionary learning provided by the present invention. Figure 5 As shown, the system 200 includes: an acquisition unit 210 and a fusion unit 220;

[0087] The acquisition unit 210 is used to: acquire a target hyperspectral image and a target multispectral image of the area to be fused;

[0088] The fusion unit 220 is configured to fuse the target hyperspectral image and the target multispectral image based on an image fusion model comprising a plurality of sequentially connected hybrid linear fusion modules to obtain a target fused image of the area to be fused;

[0089] Each hybrid linear fusion module includes: a dictionary generation network, a dictionary update network, an abundance update network and a linear fusion network; the dictionary generation network is used to receive the input hyperspectral image of the area to be fused and obtain the initial spectral dictionary corresponding to the hyperspectral image; the dictionary update network is used to generate a target spectral dictionary based on the target multispectral image and the initial spectral dictionary; the abundance update network is used to obtain a target abundance matrix based on the initial spectral dictionary and the reshaped hyperspectral image; the linear fusion network is used to linearly fuse the target spectral dictionary with the fine-tuned target abundance matrix to obtain a fused image and use it as the hyperspectral image of the area to be fused received by the next hybrid linear fusion module;

[0090] The fused image output by the last hybrid linear fusion module is determined as the target fused image.

[0091] In an optional manner, the dictionary generation network is specifically used to:

[0092] The hyperspectral image is subjected to feature extraction using a dimensionality reduction convolutional neural network to obtain a feature map which is input into a reshaping function to construct the initial spectral dictionary.

[0093] In an optional manner, the abundance update network is specifically used to:

[0094] Inputting the hyperspectral image into a reshaping function to obtain the reshaped hyperspectral image;

[0095] The target abundance matrix is ​​calculated by using the least squares method and combining the initial spectral dictionary with the reshaped hyperspectral image.

[0096] In an optional manner, the dictionary update network is specifically used to:

[0097] Splicing the target multispectral image with the initial spectral dictionary to obtain splicing features;

[0098] The concatenated features are input into the transformer structure and fine-tuned in combination with the initial spectral dictionary to generate the target spectral dictionary.

[0099] In an optional manner, the linear fusion network is specifically used to:

[0100] Based on a one-dimensional attention mechanism, fine-tuning the target abundance matrix to obtain the fine-tuned target abundance matrix;

[0101] The target spectrum dictionary is linearly fused with the fine-tuned target abundance matrix to obtain the fused image.

[0102] In an optional manner, the method further includes: a pre-processing unit; the pre-processing unit is configured to:

[0103] Acquire a high-resolution original hyperspectral image of the area to be fused, and perform downsampling processing on the original hyperspectral image to obtain a low-resolution hyperspectral image;

[0104] An interpolation process is performed on the low-resolution hyperspectral image to obtain a low-resolution target hyperspectral image.

[0105] It should be noted that the beneficial effects of the hyperspectral and multispectral image fusion system based on deep dictionary learning provided by the above embodiment are the same as the beneficial effects of the hyperspectral and multispectral image fusion method based on deep dictionary learning, and will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0106] Among them, the hyperspectral and multispectral image fusion system based on deep dictionary learning of the present invention can be a computer program (including program code) running on a computer device. For example, the hyperspectral and multispectral image fusion system based on deep dictionary learning of the present invention is an application software that can be used to execute the corresponding steps in the hyperspectral and multispectral image fusion method based on deep dictionary learning of the present invention.

[0107] In some embodiments, the hyperspectral and multispectral image fusion system based on deep dictionary learning of the present invention can be implemented by a combination of software and hardware. As an example, the hyperspectral and multispectral image fusion system based on deep dictionary learning of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the hyperspectral and multispectral image fusion method based on deep dictionary learning of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0108] The modules described in the embodiments of the present invention may be implemented in software or hardware, and the name of a module does not necessarily limit the module itself.

[0109] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned hyperspectral and multispectral image fusion methods based on deep dictionary learning is implemented. That is, an electronic device according to an embodiment of the present invention may include but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the hyperspectral and multispectral image fusion method based on deep dictionary learning shown in any embodiment of the present invention by calling the computer program.

[0110] In an alternative embodiment, an electronic device is provided, such as Figure 6 As shown, Figure 6 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0111] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0112] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6In the figure, only one thick line is used to represent the bus 4002, but this does not mean that there is only one bus or one type of bus.

[0113] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0114] The memory 4003 is used to store application code (computer program) for executing the solution of the present invention, and is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0115] Among them, the electronic device can also be a terminal device, and the terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart TV, and smart car-mounted device.

[0116] It should be noted that Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0117] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, which, when executed by a processor, implements any of the above-mentioned hyperspectral and multispectral image fusion methods based on deep dictionary learning.

[0118] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0119] In an exemplary embodiment, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned hyperspectral and multispectral image fusion method based on deep dictionary learning.

[0120] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0121] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0122] The computer-readable storage medium provided in the embodiments of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or component.

[0123] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0124] The above description is merely a preferred embodiment of the present invention and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the present invention.

[0125] It should be noted that the terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and to define a specific order or precedence. Where appropriate, the order used for similar objects may be interchanged, such that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0126] Those skilled in the art will appreciate that the present invention may be implemented as a system, method, or computer program product. Therefore, the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may be implemented in the form of a computer program product embodied in one or more computer-readable media containing computer-readable program code.

[0127] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A hyperspectral and multispectral image fusion method based on deep dictionary learning, characterized in that: include: Obtaining the target hyperspectral image and target multispectral image of the area to be fused; Based on an image fusion model comprising a plurality of sequentially connected hybrid linear fusion modules, the target hyperspectral image and the target multispectral image are fused to obtain a target fused image of the area to be fused; Each hybrid linear fusion module includes: a dictionary generation network, a dictionary update network, an abundance update network and a linear fusion network; the dictionary generation network is used to receive the input hyperspectral image of the area to be fused and obtain the initial spectral dictionary corresponding to the hyperspectral image; the dictionary update network is used to generate a target spectral dictionary based on the target multispectral image and the initial spectral dictionary; the abundance update network is used to obtain a target abundance matrix based on the initial spectral dictionary and the reshaped hyperspectral image; the linear fusion network is used to linearly fuse the target spectral dictionary with the fine-tuned target abundance matrix to obtain a fused image and use it as the hyperspectral image of the area to be fused received by the next hybrid linear fusion module; The fused image output by the last hybrid linear fusion module is determined as the target fused image.

2. The hyperspectral and multispectral image fusion method based on deep dictionary learning according to claim 1 is characterized in that: The dictionary generation network is specifically used for: The hyperspectral image is subjected to feature extraction using a dimensionality reduction convolutional neural network to obtain a feature map which is input into a reshaping function to construct the initial spectral dictionary.

3. The hyperspectral and multispectral image fusion method based on deep dictionary learning according to claim 2 is characterized in that: The abundance update network is specifically used for: Inputting the hyperspectral image into a reshaping function to obtain the reshaped hyperspectral image; The target abundance matrix is ​​calculated by using the least squares method and combining the initial spectral dictionary with the reshaped hyperspectral image.

4. The hyperspectral and multispectral image fusion method based on deep dictionary learning according to claim 3 is characterized in that: The dictionary update network is specifically used for: Splicing the target multispectral image with the initial spectral dictionary to obtain splicing features; The concatenated features are input into the transformer structure and fine-tuned in combination with the initial spectral dictionary to generate the target spectral dictionary.

5. The hyperspectral and multispectral image fusion method based on deep dictionary learning according to claim 4 is characterized in that: The linear fusion network is specifically used for: Based on a one-dimensional attention mechanism, fine-tuning the target abundance matrix to obtain the fine-tuned target abundance matrix; The target spectrum dictionary is linearly fused with the fine-tuned target abundance matrix to obtain the fused image.

6. The hyperspectral and multispectral image fusion method based on deep dictionary learning according to any one of claims 1 to 5, characterized in that: Also includes: Acquire a high-resolution original hyperspectral image of the area to be fused, and perform downsampling processing on the original hyperspectral image to obtain a low-resolution hyperspectral image; An interpolation process is performed on the low-resolution hyperspectral image to obtain a low-resolution target hyperspectral image.

7. A hyperspectral and multispectral image fusion system based on deep dictionary learning, characterized in that: include: Acquisition units and fusion units; The acquisition unit is used to: acquire a target hyperspectral image and a target multispectral image of the area to be fused; The fusion unit is configured to fuse the target hyperspectral image and the target multispectral image based on an image fusion model comprising a plurality of sequentially connected hybrid linear fusion modules to obtain a target fused image of the area to be fused; Each hybrid linear fusion module includes: a dictionary generation network, a dictionary update network, an abundance update network and a linear fusion network; the dictionary generation network is used to receive the input hyperspectral image of the area to be fused and obtain the initial spectral dictionary corresponding to the hyperspectral image; the dictionary update network is used to generate a target spectral dictionary based on the target multispectral image and the initial spectral dictionary; the abundance update network is used to obtain a target abundance matrix based on the initial spectral dictionary and the reshaped hyperspectral image; the linear fusion network is used to linearly fuse the target spectral dictionary with the fine-tuned target abundance matrix to obtain a fused image and use it as the hyperspectral image of the area to be fused received by the next hybrid linear fusion module; The fused image output by the last hybrid linear fusion module is determined as the target fused image.

8. The hyperspectral and multispectral image fusion system based on deep dictionary learning according to claim 7, characterized in that: Also includes: Pre-processing unit; The pre-processing unit is used for: Acquire a high-resolution original hyperspectral image of the area to be fused, and perform downsampling processing on the original hyperspectral image to obtain a low-resolution hyperspectral image; An interpolation process is performed on the low-resolution hyperspectral image to obtain a low-resolution target hyperspectral image.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, and the memory stores at least one computer program. The at least one computer program is loaded and executed by the processor so that the electronic device implements the hyperspectral and multispectral image fusion method based on deep dictionary learning as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor so that the computer-readable storage medium implements the hyperspectral and multispectral image fusion method based on deep dictionary learning according to any one of claims 1 to 6.

Images