Super-resolution reconstruction method and system based on multi-modal image feature fusion

Abstract

The invention provides a super-resolution reconstruction method and system based on multi-modal image feature fusion, and belongs to the technical field of remote sensing image processing, and the method comprises the steps: obtaining superficial layer features of a multispectral image and an infrared image, and superposing the superficial layer features to obtain a total superficial layer feature; respectively inputting shallow layer features of the multispectral image and the infrared image into a double-branch global residual network to obtain corresponding deep layer features, and superposing the deep layer features to obtain a total deep layer feature; inputting the total shallow-layer features and the total deep-layer features into a feature fusion network to obtain fusion features of the multispectral and infrared images; and inputting the fusion features into an image reconstruction network to obtain a super-resolution image. According to the invention, a high-quality super-resolution image can be obtained.

Description

Technical Field

[0001] The present application belongs to the technical field of remote sensing image processing, and more specifically, relates to a super-resolution reconstruction method and system based on multimodal image feature fusion. Background Art

[0002] Image super-resolution reconstruction is an important task in digital image processing and is widely used in medical images, remote sensing imaging, video surveillance, and image restoration. Its purpose is to improve the resolution of the image while preserving the image texture details as much as possible. In recent years, with the rapid development of deep learning technology, image super-resolution reconstruction algorithms based on deep learning have made significant breakthroughs. Image super-resolution reconstruction based on deep learning has become the mainstream method in this field. However, there are still some problems to be explored in this field. On the one hand, most of the existing image super-resolution reconstruction methods based on deep learning only reconstruct images of a single modality, with limited available information, and poor reconstruction effects for targets with similar texture information; on the other hand, the efficient extraction and full utilization of deep features are also important factors affecting the quality of image reconstruction. Therefore, the existing technology needs to be further improved and developed. Summary of the invention

[0003] In view of the defects of the prior art, the purpose of the present application is to provide a super-resolution reconstruction method and system based on multimodal image feature fusion, aiming to solve the problems of poor super-resolution reconstruction of target images with similar texture information and insufficient utilization of deep features.

[0004] To achieve the above objectives, in a first aspect, the present application provides a super-resolution reconstruction method based on multimodal image feature fusion, comprising the following steps:

[0005] Step 1: Enlarge the multispectral image and infrared image to the scale of the super-resolution image respectively;

[0006] Step 2: Input the scaled multispectral image and infrared image into the pseudo-Siamese feature extraction network to obtain the shallow features of the multispectral image and the shallow features of the infrared image respectively; wherein the shallow features are used to represent the texture structure information;

[0007] Step 3: Superimpose the shallow features of the multispectral image and the shallow features of the infrared image to obtain a total shallow feature;

[0008] Step 4: Input the shallow features of the multispectral image and the shallow features of the infrared image obtained in step 2 into the dual-branch global residual network respectively to obtain the deep features of the multispectral image and the deep features of the infrared image; wherein the deep features are used to characterize the abstract essence;

[0009] Step 5: Superimpose the deep features of the multispectral image and the deep features of the infrared image obtained in step 4 to obtain a total deep feature;

[0010] Step 6: Input the total shallow features and the total deep features into the feature fusion network to obtain the fusion features of the multispectral and infrared images; wherein the fusion features have both texture structure information and abstract essential information;

[0011] Step 7: Input the fusion features obtained in step 6 into the image reconstruction network to obtain a super-resolution image.

[0012] Further preferably, the dual-branch global residual network in step 4 is a dual-branch network with the same network structure but non-shared weights, and a single-branch network is composed of multiple residual blocks; each residual block has a jump connection.

[0013] Further preferably, the operation method of the residual block is: the shallow features of the input multispectral image or the shallow features of the infrared image are subjected to one convolution and batch normalization, then activated using the PreLU function, and after another convolution and batch normalization, the result is summed element by element with the input.

[0014] Further preferably, the fusion process of the feature fusion network in step six is:

[0015] R=F high -F low

[0016] F=C 3×3 [C 1×1 (R)]+F high

[0017] Among them, C 3×3 represents 3*3 convolution; C 1×1 represents 1*1 convolution; R represents information that exists in one source but not in the other; F higj Represents the overall deep features; F low Represents the overall shallow features.

[0018] Further preferably, the pseudo-Siamese feature extraction network is dual-branch and has independent convolutional streams with the same structure.

[0019] In a second aspect, the present application provides a multi-modal image feature fusion super-resolution reconstruction system, comprising:

[0020] A scale transformation module is used to enlarge the multispectral image and the infrared image to the scale of the super-resolution image respectively;

[0021] A shallow feature extraction module is used to input the scaled multispectral image and infrared image into the pseudo-Siamese feature extraction network to obtain shallow features of the multispectral image and shallow features of the infrared image respectively; wherein the shallow features are used to represent texture structure information;

[0022] A first feature superposition module is used to superimpose the shallow features of the multispectral image and the shallow features of the infrared image to obtain a total shallow feature;

[0023] A deep feature extraction module is used to input the shallow features of the multispectral image and the shallow features of the infrared image into a dual-branch global residual network to obtain the deep features of the multispectral image and the deep features of the infrared image; wherein the deep features are used to represent the abstract essence;

[0024] A second feature superposition module is used to superimpose the deep features of the multispectral image and the deep features of the infrared image to obtain a total deep feature;

[0025] The feature fusion module is used to fuse the overall shallow features and the overall deep features to obtain fused features that have both texture structure information and abstract essential information;

[0026] The reconstruction module is used to input the fused features into the image reconstruction network to obtain a super-resolution image.

[0027] Further preferably, the dual-branch global residual network in the deep feature extraction module is a dual-branch network with the same network structure but non-shared weights, and a single network is composed of multiple residual blocks; each residual block has a jump connection.

[0028] Further preferably, the residual block structure is, from input to output, a first convolutional layer, a first batch of normalized layers, an activation function, a second convolutional layer, and a second batch of normalized layers.

[0029] Further preferably, the fusion method of the feature fusion network in the feature fusion module is:

[0030] R=F jigh -F low

[0031] F=C 3×3 [C 1×1 (R)]+F high

[0032] Among them, C 3×3 represents 3*3 convolution; C 1×1 represents 1*1 convolution; R represents information that exists in one source but not in the other; F high Represents the overall deep features; F low Represents the overall shallow features.

[0033] Further preferably, the pseudo-Siamese feature extraction network in the shallow feature extraction module is dual-branched and has independent convolutional flows with the same structure.

[0034] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the method described in the first aspect or a further preferred implementation of the first aspect.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or a further preferred implementation of the first aspect.

[0036] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or a further preferred implementation of the first aspect.

[0037] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0038] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0039] This application provides a super-resolution reconstruction method based on multimodal image feature fusion, which solves the problem of poor reconstruction quality of target images with similar texture information by introducing infrared feature information; secondly, a dual-branch global residual network is used to efficiently extract deep features of the image. In addition, a feature fusion network is used to make full use of the shallow and deep features of the image, so that the quality of the super-resolution image obtained is better than other reconstruction methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of a super-resolution reconstruction method based on multimodal image feature fusion provided in an embodiment of the present application;

[0041] Figure 2 is a schematic diagram of a pseudo-Siamese feature extraction network provided in an embodiment of the present application;

[0042] Figure 3 is a schematic diagram of a global residual network provided in an embodiment of the present application;

[0043] Figure 4 This is a schematic diagram of a feature fusion network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0046] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more than two.

[0047] The technical solutions provided in the embodiments of the present application are introduced below.

[0048] In a first aspect, the present application provides a super-resolution reconstruction method based on multimodal image feature fusion, comprising the following steps:

[0049] Step 1: Before inputting the LR (Low Resolution) image into the network, the multispectral image and the infrared image are enlarged to the SR (Super Resolution) scale respectively;

[0050] Step 2: Input the scaled multispectral image and infrared image into the pseudo-Siamese feature extraction network to obtain the shallow features of the multispectral image and the shallow features of the infrared image respectively;

[0051] Step 3: Superimpose the shallow features of the multispectral image and the shallow features of the infrared image obtained in step 2 to obtain a total shallow feature;

[0052] Step 4: Input the shallow features of the multispectral image and the shallow features of the infrared image obtained in step 2 into the dual-branch global residual network respectively to obtain the deep features of the multispectral image and the deep features of the infrared image;

[0053] Step 5: Superimpose the deep features of the multispectral image and the deep features of the infrared image obtained in step 4 to obtain a total deep feature;

[0054] Step 6: Input the total shallow features and the total deep features obtained in step 3 and step 5 respectively into the feature fusion network to obtain the fusion features of the multispectral and infrared images;

[0055] Step 7: Input the fusion features obtained in step 6 into the image reconstruction network to obtain an SR image.

[0056] Further preferably, in step 1, the low-resolution multispectral image and infrared image are enlarged to the SR scale using a bicubic interpolation algorithm.

[0057] Further preferably, in step 2, the pseudo-Siamese feature extraction network is a double-branch network with the same network structure but non-shared weights.

[0058] Further preferably, in step 2, the shallow features of the multispectral image extracted by the pseudo-Siamese feature extraction network are The shallow features of the infrared image extracted by the pseudo-Siamese feature extraction network are

[0059]

[0060] Further preferably, in step 4, the dual-branch global residual network is a dual-branch network with the same network structure but non-shared weights, and a single-branch network is composed of multiple residual blocks.

[0061] Further preferably, in step 4, the deep features of the multispectral image extracted by the dual-branch global residual network are The deep features of infrared images extracted by the global residual network are

[0062] In a second aspect, the present application provides a super-resolution reconstruction system based on multimodal image feature fusion, comprising:

[0063] The shallow feature extraction module is used to extract the shallow features of the multispectral image and infrared image that represent the texture structure information, and finally obtain the fused shallow features;

[0064] The deep feature extraction module is used to extract the deep features of multispectral images and infrared images that represent the abstract essence, and finally obtain the fused deep features;

[0065] A feature fusion module is used to fuse the extracted fused shallow features and fused deep features to obtain fused features with both texture structure information and abstract essential information;

[0066] The reconstruction module is used to input the fused features into the image reconstruction network to obtain a super-resolution image.

[0067] It should be pointed out here that the convolution operations of the feature extraction module, feature fusion module and reconstruction module are all 3D convolutions.

[0068] In order to achieve the above-mentioned objectives, the present application also provides an intelligent terminal, which includes a memory, a processor, and a super-resolution reconstruction program based on multimodal image feature fusion stored in the above-mentioned memory and executable on the above-mentioned processor. When the above-mentioned super-resolution reconstruction program based on multimodal image feature fusion is executed by the above-mentioned processor, any step of the above-mentioned super-resolution reconstruction method based on multimodal image feature fusion is implemented.

[0069] In order to achieve the above-mentioned objectives, the present application also provides a computer-readable storage medium, on which is stored a super-resolution reconstruction method program based on multimodal image feature fusion. When the super-resolution reconstruction method program based on multimodal image feature fusion is executed by a processor, any step of the super-resolution reconstruction method based on multimodal image feature fusion is implemented.

[0070] Example

[0071] like Figure 1 As shown, the present application provides a super-resolution reconstruction method based on multimodal image feature fusion, comprising the following steps:

[0072] Step 1: Data preparation: multispectral data Infrared data

[0073] Step 2: Use the bicubic interpolation algorithm to enlarge the multispectral image and infrared image to the SR scale, for example

[0074] Step 3: Input the enlarged multispectral image and infrared image obtained in step 2 into the pseudo-Siamese feature extraction network at the same time to obtain the shallow features of the multispectral image and shallow features of infrared images

[0075] Step 4: Input the shallow features of the multispectral image and the shallow features of the infrared image obtained in the third step into the dual-branch global residual network to obtain the deep features of the multispectral image. and deep features of infrared images

[0076] Step 5: The shallow features of the multispectral image obtained in step 3 and shallow features of infrared images Superposition is performed to obtain the fused shallow feature F low , the deep features of the multispectral image obtained in the fourth step and deep features of infrared images Superposition is performed to obtain the fused deep feature F hihh ;

[0077] Step 6: Fusion shallow feature F obtained in step 5 low And fusion deep features F high Input into the feature fusion network to obtain the final fused depth residual feature;

[0078] Step 7: Input the fused depth residual features obtained in the sixth step into the image reconstruction network to obtain the super-resolution image SR.

[0079] Specifically, Figure 2 As shown, in the third step, a pseudo-Siamese feature extraction network is involved. Pseudo-Siamese is a dual-branch independent convolutional flow with the same structure. Since multispectral images and infrared images have different imaging characteristics, they are respectively input into a feature extraction network with the same structure, which can avoid interference between the features of multispectral images and infrared images.

[0080] It should be noted that the pseudo-Siamese feature extraction network involved in the third step is a network. Figure 2 In the figure, two pseudo-Siamese feature extraction networks are used to represent it;

[0081] like Figure 3 As shown, in the fourth step, a dual-branch global residual network is involved, which is composed of a dual-branch network with the same network structure but unshared weights. A single branch is composed of multiple residual blocks, and each residual block has a skip connection, that is, a shortcut. In each residual block, the input data first undergoes a convolution (Convolution, Conv) and batch normalization (Batch Normalization, BN), and then uses the PreLU function for activation. After another convolution and batch normalization, the result is summed element-wise with the input, that is, the mapping of y=f(x)+x is realized.

[0082] It should be noted that the role of the global residual network shortcut in the fourth step is to increase the number of network layers. When a layer in the deep network has a gradient vanishing problem, the gradient can be directly "skipped" through the shortcut and directly propagated to a shallower layer, which effectively alleviates the gradient vanishing problem and enables a deeper network to improve the overall performance of the model.

[0083] like Figure 4 As shown, the sixth step involves a feature fusion network, the input of which is the fused shallow feature F obtained in the fifth step. low And fusion deep features Fhigh , this fusion process can be described as:

[0084] R=F high -F low

[0085] F=C 3×3 [C 1×1 (R)]+F high

[0086] Among them, C represents 3*3 and 1*1 convolution; R represents the information that exists in one source but not in the other source; F represents the fused deep residual feature obtained after fusion; this fusion module enables the network to bypass the same parts of different source features and pay more attention to the special parts of different feature sources; this module can improve the network's discrimination ability and perform residual learning while fusing features from different sources, which is conducive to improving feature learning ability compared with simple addition and concatenation operations.

[0087] In summary, compared with the prior art, the present application has the following advantages:

[0088] This application provides a super-resolution reconstruction method based on multimodal image feature fusion, which solves the problem of poor reconstruction quality of target images with similar texture information by introducing infrared feature information; secondly, a dual-branch global residual network is used to efficiently extract deep features of the image. In addition, a feature fusion network is used to make full use of the shallow and deep features of the image, so that the quality of the super-resolution image obtained is better than other reconstruction methods.

[0089] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0090] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device. The device may include: at least one memory for storing programs and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the above embodiment.

[0091] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0092] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0093] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0094] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0095] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0096] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0097] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A super-resolution reconstruction method for multimodal image feature fusion, characterized in that: The following steps are involved: Step 1: Enlarge the multispectral image and infrared image to the scale of the super-resolution image respectively; Step 2: Input the scaled multispectral image and infrared image into the pseudo-Siamese feature extraction network to obtain the shallow features of the multispectral image and the shallow features of the infrared image respectively; wherein the shallow features are used to represent the texture structure information; Step 3: Superimpose the shallow features of the multispectral image and the shallow features of the infrared image to obtain a total shallow feature; Step 4: Input the shallow features of the multispectral image and the shallow features of the infrared image obtained in step 2 into the dual-branch global residual network respectively to obtain the deep features of the multispectral image and the deep features of the infrared image; wherein the deep features are used to characterize the abstract essence; Step 5: Superimpose the deep features of the multispectral image and the deep features of the infrared image to obtain a total deep feature; Step 6: Input the total shallow features and the total deep features into the feature fusion network to obtain the fusion features of the multispectral and infrared images; wherein the fusion features have both texture structure information and abstract essential information; Step 7: Input the fusion features obtained in step 6 into the image reconstruction network to obtain a super-resolution image.

2. The super-resolution reconstruction method according to claim 1, characterized in that: The dual-branch global residual network in step 4 is a dual-branch network with the same network structure but unshared weights. A single branch network consists of multiple residual blocks; each residual block has a skip connection.

3. The super-resolution reconstruction method according to claim 2, characterized in that: The operation method of the residual block is: the shallow features of the input multispectral image or the shallow features of the infrared image are convolved and batch normalized once, then activated using the PreLU function, and after another convolution and batch normalization, the result is summed element by element with the input.

4. The super-resolution reconstruction method according to any one of claims 1 to 3, characterized in that: The fusion method of the feature fusion network in step six is: R=F high -F low F=C 3×3 [C 1×1 (R)]+F high Among them, C 3×3 represents 3*3 convolution; C 1×1 represents 1*1 convolution; R represents information that exists in one source but not in the other; F high Represents the overall deep features; F l0w Represents the overall shallow features.

5. The super-resolution reconstruction method according to claim 1, characterized in that: The pseudo-Siamese feature extraction network is dual-branch and has independent convolutional streams with the same structure.

6. A super-resolution reconstruction system for multimodal image feature fusion, characterized in that: include: A scale transformation module is used to enlarge the multispectral image and the infrared image to the scale of the super-resolution image respectively; A shallow feature extraction module is used to input the scaled multispectral image and infrared image into the pseudo-Siamese feature extraction network to obtain shallow features of the multispectral image and shallow features of the infrared image respectively; wherein the shallow features are used to represent texture structure information; A first feature superposition module is used to superimpose the shallow features of the multispectral image and the shallow features of the infrared image to obtain a total shallow feature; A deep feature extraction module is used to input the shallow features of the multispectral image and the shallow features of the infrared image into a dual-branch global residual network to obtain the deep features of the multispectral image and the deep features of the infrared image; wherein the deep features are used to represent the abstract essence; A second feature superposition module is used to superimpose the deep features of the multispectral image and the deep features of the infrared image to obtain a total deep feature; The feature fusion module is used to fuse the overall shallow features and the overall deep features to obtain fused features that have both texture structure information and abstract essential information; The reconstruction module is used to input the fused features into the image reconstruction network to obtain a super-resolution image.

7. The super-resolution reconstruction system according to claim 6, characterized in that: The dual-branch global residual network in the deep feature extraction module is a dual-branch network with the same network structure but unshared weights. A single network consists of multiple residual blocks; each residual block has a skip connection.

8. The super-resolution reconstruction system according to claim 7, characterized in that: The residual block structure from input to output is the first convolutional layer, the first batch of normalization layers, the activation function, the second convolutional layer and the second batch of normalization layers.

9. The super-resolution reconstruction system according to any one of claims 6 to 8, characterized in that: The fusion method of the feature fusion network in the feature fusion module is: R=F high -F low F=C 3×3 [C 1×1 (R)]+F high Among them, C 3×3 represents 3*3 convolution; C 1×1 represents 1*1 convolution; R represents information that exists in one source but not in the other; F high Represents the overall deep features; F low Represents the overall shallow features.

10. The super-resolution reconstruction system according to claim 6, characterized in that: The pseudo-Siamese feature extraction network in the shallow feature extraction module is dual-branched and has independent convolutional streams with the same structure.

Images