A super-resolution reconstruction method and system based on multi-modal image feature fusion

Abstract

The application provides a kind of super-resolution reconstruction method and system based on multimodal image feature fusion, belong to remote sensing image processing technical field, method is: obtaining the shallow feature of multispectral image and infrared image, and superimposes total shallow feature;The shallow feature of multispectral image and infrared image is respectively input into double-branch global residual network, corresponding deep feature is obtained, and total deep feature is obtained by superimposition;Total shallow feature and total deep feature are input into feature fusion network, and the fusion feature of multispectral and infrared image is obtained;The fusion feature is input into image reconstruction network, and the super-resolution image is obtained.The application can obtain high-quality super-resolution image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing, and more particularly relates to a super-resolution reconstruction method and system based on multi-modal image feature fusion. BACKGROUND

[0002] Image super-resolution reconstruction, as an important task in digital image processing, has wide applications in medical images, remote sensing imaging, video monitoring and image restoration, etc. The purpose is to improve the resolution of the image while preserving the texture details as much as possible. In recent years, with the rapid development of deep learning technology, the image super-resolution reconstruction algorithm based on deep learning has made a major breakthrough. The image super-resolution reconstruction based on deep learning has become the mainstream method in this field. However, there are still some problems to be explored in this field. On the one hand, most of the existing image super-resolution reconstruction methods based on deep learning are only for single modal image reconstruction, which has limited available information and poor reconstruction effect for similar texture information targets. On the other hand, efficient extraction and full use of deep features are also important factors affecting the quality of image reconstruction. Therefore, the existing technology still needs to be further improved and developed. SUMMARY

[0003] In view of the defects of the prior art, the purpose of the present application is to provide a super-resolution reconstruction method and system based on multi-modal image feature fusion, which aims to solve the problems of poor super-resolution reconstruction effect of target images with similar texture information and insufficient use of deep features.

[0004] To achieve the above-mentioned purpose, in a first aspect, the present application provides a super-resolution reconstruction method based on multi-modal image feature fusion, comprising the following steps:

[0005] Step one: magnify the multispectral image and the infrared image to the scale of the super-resolution image respectively;

[0006] Step two: input the multispectral image and the infrared image after the scale transformation into a pseudo-siamese feature extraction network to obtain the shallow features of the multispectral image and the shallow features of the infrared image respectively; wherein the shallow features are used to represent the texture structure information;

[0007] Step three: superimpose the shallow features of the multispectral image and the shallow features of the infrared image to obtain a total shallow feature;

[0008] Step four: input the shallow features of the multispectral image and the shallow features of the infrared image obtained in step two into a double-branch global residual network respectively to obtain the deep features of the multispectral image and the deep features of the infrared image; wherein the deep features are used to represent the abstract essence;

[0009] Step five: superimpose the deep features of the multispectral image and the deep features of the infrared image obtained in step four to obtain a total deep feature;

[0010] Step six: input the total shallow feature and the total deep feature into a feature fusion network to obtain a fusion feature of the multispectral and infrared images; wherein the fusion feature has both texture structure information and abstract essential information;

[0011] Step seven: input the fusion feature obtained in step six into an image reconstruction network to obtain a super-resolution image.

[0012] Further preferably, the dual-branch global residual network in step four is a dual-branch network with the same network structure but not sharing weights, and each single-branch network is composed of multiple residual blocks; each residual block has a skip connection.

[0013] Further preferably, the operation method of the residual block is: the input shallow feature of the multispectral image or the input shallow feature of the infrared image is subjected to one convolution and batch normalization, then activated by a PreLU function, and then subjected to one convolution and batch normalization again, and the result is summed with the input element by element.

[0014] Further preferably, the fusion process of the feature fusion network in step six is:

[0015] R=F high -F low

[0016] F=C 3×3 [C 1×1 (R)]+F high

[0017] Wherein, C 3×3 represents a 3*3 convolution; C 1×1 represents a 1*1 convolution; R represents information existing in one source but not in the other; F higj represents the total deep feature; F low represents the total shallow feature.

[0018] Further preferably, the pseudo-Siamese feature extraction network is a dual-branch independent convolution stream with the same structure.

[0019] In a second aspect, the present application provides a multi-modal image feature fusion super-resolution reconstruction system, comprising:

[0020] A scale transformation module for respectively enlarging the multispectral image and the infrared image to the scale of the super-resolution image;

[0021] The shallow feature extraction module is configured to input the multi-spectral image and the infrared image after the scale transformation into a pseudo-siamese feature extraction network to obtain a shallow feature of the multi-spectral image and a shallow feature of the infrared image respectively; wherein the shallow features are used to represent texture structure information;

[0022] The first feature superposition module is configured to superimpose the shallow feature of the multi-spectral image and the shallow feature of the infrared image to obtain a total shallow feature;

[0023] The deep feature extraction module is configured to input the shallow feature of the multi-spectral image and the shallow feature of the infrared image into a double-branch global residual network respectively to obtain a deep feature of the multi-spectral image and a deep feature of the infrared image; wherein the deep features are used to represent abstract essence;

[0024] The second feature superposition module is configured to superimpose the deep feature of the multi-spectral image and the deep feature of the infrared image to obtain a total deep feature;

[0025] The feature fusion module is configured to fuse the total shallow feature and the total deep feature to obtain a fusion feature having both texture structure information and abstract essence information;

[0026] The reconstruction module is configured to input the fusion feature into an image reconstruction network to obtain a super-resolution image.

[0027] Further preferably, the double-branch global residual network in the deep feature extraction module is a double-branch network with the same network structure but not sharing weights, and each single-branch network is composed of multiple residual blocks; each residual block has a skip connection.

[0028] Further preferably, the residual block structure sequentially comprises a first convolutional layer, a first batch normalization layer, an activation function, a second convolutional layer and a second batch normalization layer from input to output.

[0029] Further preferably, the fusion method of the feature fusion network in the feature fusion module is as follows:

[0030] R=F jigh -F low

[0031] F=C 3×3 [C 1×1 (R)]+F high

[0032] wherein C 3×3 represents a 3*3 convolution; C 1×1 represents a 1*1 convolution; R represents information existing in one source but not in the other source; F high represents the total deep feature; F low represents the total shallow feature.

[0033] Further preferably, the pseudo-Siamese feature extraction network in the shallow feature extraction module is a double-branch independent convolutional flow with the same structure.

[0034] In a third aspect, the present application provides an electronic device, comprising: at least one memory for storing a program; at least one processor for executing the program stored in the memory, when the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or the further preferred implementation manner of the first aspect.

[0035] In a fourth aspect, the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program runs on the processor, the processor executes the method described in the first aspect or the further preferred implementation manner of the first aspect.

[0036] In a fifth aspect, the present application provides a computer program product, when the computer program product runs on the processor, the processor executes the method described in the first aspect or the further preferred implementation manner of the first aspect.

[0037] It can be understood that the beneficial effects of the above-mentioned second aspect to fifth aspect can be referred to the related description in the first aspect, which will not be repeated here.

[0038] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0039] The present application provides a super-resolution reconstruction method based on multi-modal image feature fusion, which solves the problem of poor reconstruction quality of target image with similar texture information by introducing infrared feature information. Secondly, a double-branch global residual network is used to efficiently extract deep features of the image. In addition, a feature fusion network is used to fully utilize the shallow features and deep features of the image, so that the quality of the obtained super-resolution image is better than that of other reconstruction methods. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a super-resolution reconstruction method based on multi-modal image feature fusion provided by the embodiment of the present application;

[0041] Figure 2 is a pseudo-Siamese feature extraction network schematic diagram provided by the embodiment of the present application;

[0042] Figure 3 is a global residual network schematic diagram provided by the embodiment of the present application;

[0043] Figure 4 is a feature fusion network schematic diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application and not to limit the present application.

[0045] In the embodiments of the present application, the words such as "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of the words such as "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0046] In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0047] The technical solutions provided in the embodiments of the present application will be introduced below.

[0048] In a first aspect, the present application provides a super-resolution reconstruction method based on multi-modal image feature fusion, comprising the following steps:

[0049] Step one: before inputting the LR (Low Resolution, low resolution) image into the network, the multispectral image and the infrared image are respectively enlarged to the SR (Super Resolution, super resolution) scale;

[0050] Step two: input the multispectral image and the infrared image after the transformation scale into the pseudo-siamese feature extraction network, to obtain the shallow features of the multispectral image and the shallow features of the infrared image, respectively;

[0051] Step three: superimpose the shallow features of the multispectral image and the shallow features of the infrared image obtained in step two to obtain a total shallow feature;

[0052] Step four: input the shallow features of the multispectral image and the shallow features of the infrared image obtained in step two into the double-branch global residual network, respectively, to obtain the deep features of the multispectral image and the deep features of the infrared image;

[0053] Step five: superimpose the deep features of the multispectral image and the deep features of the infrared image obtained in step four to obtain a total deep feature;

[0054] Step six: input the total shallow feature and the total deep feature obtained in steps three and five, respectively, into the feature fusion network to obtain the fusion features of the multispectral and infrared images;

[0055] Step seven: input the fusion feature obtained in step six into the image reconstruction network to obtain an SR image.

[0056] Further preferably, in step one, the low-resolution multispectral image and the infrared image are enlarged to the SR scale using a bicubic interpolation algorithm.

[0057] Further preferably, in step two, the pseudo-Siamese feature extraction network is a double-branch network with the same network structure but without shared weights.

[0058] Further preferably, in step two, the shallow features of the multispectral image extracted by the pseudo-Siamese feature extraction network are the shallow features of the infrared image extracted by the pseudo-Siamese feature extraction network are

[0059] Further preferably, in step four, the double-branch global residual network is a double-branch network with the same network structure but without shared weights, and each branch network is composed of multiple residual blocks.

[0060] Further preferably, in step four, the deep features of the multispectral image extracted by the double-branch global residual network are the deep features of the infrared image extracted by the global residual network are

[0061] In a second aspect, the present application provides a super-resolution reconstruction system based on multi-modal image feature fusion, comprising:

[0062] a shallow feature extraction module for extracting shallow features of a multispectral image and an infrared image representing texture structure information, and finally obtaining fused shallow features;

[0063] a deep feature extraction module for extracting deep features of the multispectral image and the infrared image representing abstract essence, and finally obtaining fused deep features;

[0064] a feature fusion module for fusing the extracted fused shallow features and the fused deep features to obtain fusion features with both texture structure information and abstract essence information;

[0065] a reconstruction module for inputting the fusion features into an image reconstruction network to obtain a super-resolution image.

[0066] It should be noted that the convolution operations of the feature extraction module, the feature fusion module, and the reconstruction module are all 3D convolutions.

[0067] To achieve the above object, the present application further provides a kind of intelligent terminal, and the intelligent terminal includes memory, processor and the storage based on the super-resolution reconstruction program of multi-modal image feature fusion of being stored on above-mentioned memory and can be run on above-mentioned processor, above-mentioned super-resolution reconstruction program based on the feature fusion of multi-modal image is implemented when the steps of any one above-mentioned super-resolution reconstruction method based on the feature fusion of multi-modal image by above-mentioned processor executes.

[0068] To achieve the above object, the present application further provides a kind of computer readable storage medium, and the super-resolution reconstruction method program based on the feature fusion of multi-modal image is stored on above-mentioned computer readable storage medium, above-mentioned super-resolution reconstruction method program based on the feature fusion of multi-modal image is implemented when the steps of any one above-mentioned super-resolution reconstruction method based on the feature fusion of multi-modal image by processor executes.

[0069] Embodiment

[0070] As Figure 1 shown, the present application provides a kind of super-resolution reconstruction method based on the feature fusion of multi-modal image, including the following steps:

[0071] First step: data preparation: multispectral data Infrared data

[0072] Second step: above-mentioned multispectral image and infrared image are respectively enlarged to SR scale using bicubic interpolation algorithm, for example

[0073] Third step: the multispectral image and infrared image after size obtained in the second step are simultaneously input to pseudo-siam feature extraction network, and multispectral image shallow feature And infrared image shallow feature

[0074] Fourth step: multispectral image shallow feature and infrared image shallow feature obtained in the third step are respectively input to double-branch global residual network, and the deep feature of multispectral image And the deep feature of infrared image

[0075] Fifth step: multispectral image shallow feature And infrared image shallow feature Fusion shallow feature F is obtained by superimposing the third step low , the deep feature of multispectral image And the deep feature of infrared image Fusion deep feature F is obtained by superimposing the fourth step hihh ;

[0076] Step 6: input the fusion shallow features F low and the fusion deep features F high into the feature fusion network to obtain the final fusion deep residual features;

[0077] Step 7: input the fusion deep residual features obtained in Step 6 into the image reconstruction network to obtain the super-resolution image SR.

[0078] Specifically, as shown in Figure 2 , in Step 3, a pseudo-Siamese feature extraction network is involved, which is a double-branch independent convolutional stream with the same structure; since the multispectral image and the infrared image have different imaging characteristics, they are input into the feature extraction networks with the same structure respectively, which can avoid the interference between the features of the multispectral image and the infrared image.

[0079] It should be noted that the pseudo-Siamese feature extraction network involved in Step 3 is a network, and in order to explain in detail in Figure 2 , it is represented by two pseudo-Siamese feature extraction networks.

[0080] As shown in Figure 3 , in Step 4, a double-branch global residual network is involved, which is composed of two branch networks with the same structure but not sharing weights, and each branch is composed of multiple residual blocks, each of which has a skip connection, i.e., a shortcut; in each residual block, the input data is first subjected to convolution (Conv) and batch normalization (BN), then activated using the PreLU function, and then subjected to convolution and batch normalization again, and the results are summed element by element with the input, i.e., the mapping y = f(x) + x is realized.

[0081] It should be noted that in Step 4, the shortcut of the global residual network is to increase the number of network layers, and when there is a gradient vanishing problem in a certain layer of the deep network, the gradient can directly "jump over" that layer through the shortcut and propagate to a shallower layer, which effectively alleviates the gradient vanishing problem and allows a deeper network to be implemented to improve the overall performance of the model.

[0082] As shown in Figure 4 , in Step 6, a feature fusion network is involved, and the input of the network is the fusion shallow features F low and the fusion deep features Fhigh This fusion process can be described as:

[0083] R=F high -F low

[0084] F=C 3×3 [C 1×1 (R)]+F high

[0085] Wherein, C represents 3*3 and 1*1 convolution; R represents information existing in one source and not existing in another source; F represents the fusion depth residual feature obtained after fusion; The fusion module can make the network bypass the same part in different source features and pay more attention to the special part in different feature sources; The module can improve the discrimination ability of the network, and residual learning is performed while fusing different source features, compared with simple addition operation and splicing operation, which is beneficial to improve the feature learning ability.

[0086] In summary, compared with the prior art, the present application has the following advantages:

[0087] The present application provides a super-resolution reconstruction method based on multi-modal image feature fusion, which solves the problem of poor reconstruction quality of target images with similar texture information by introducing infrared feature information; Secondly, a double-branch global residual network is used to efficiently extract deep features of the image. In addition, the feature fusion network fully utilizes the shallow features and deep features of the image, so that the quality of the obtained super-resolution image is better than that of other reconstruction methods.

[0088] It should be understood that the above device is used to execute the method in the above embodiment, and the corresponding program module in the device, the implementation principle and technical effect are similar to the description in the above method, and the working process of the device can refer to the corresponding process in the above method, which will not be described here.

[0089] Based on the method in the above embodiment, the present application embodiment provides an electronic device. The device can include at least one memory for storing programs and at least one processor for executing programs stored in the memory. Wherein, when the program stored in the memory is executed, the processor is used to execute the method described in the above embodiment.

[0090] Based on the method in the above embodiment, the present application embodiment provides a computer readable storage medium, and the computer readable storage medium stores a computer program, when the computer program runs on the processor, so that the processor executes the method in the above embodiment.

[0091] Based on the method in the above embodiment, the present application embodiment provides a computer program product, when the computer program product runs on the processor, so that the processor executes the method in the above embodiment.

[0092] It can be appreciated that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0093] The method steps in the embodiments of the present application can be implemented in the form of hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0094] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0095] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application.

[0096] Those skilled in the art easily understand that the above only describes the preferred embodiments of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A multi-modal image feature fusion super-resolution reconstruction method, characterized in that, The method comprises the following steps: Step 1: magnify the multispectral image and the infrared image to the scale of the super-resolution image respectively; Step 2: input the multispectral image and the infrared image after scale transformation into the pseudo-Siamese feature extraction network to obtain the shallow features of the multispectral image and the shallow features of the infrared image respectively; wherein the shallow features are used to represent the texture structure information; Step 3: superimpose the shallow features of the multispectral image and the shallow features of the infrared image to obtain a total shallow feature; Step 4: input the shallow features of the multispectral image and the shallow features of the infrared image obtained in step 2 into the double-branch global residual network respectively to obtain the deep features of the multispectral image and the deep features of the infrared image; wherein the deep features are used to represent the abstract essence; Step 5: superimpose the deep features of the multispectral image and the deep features of the infrared image to obtain a total deep feature; Step 6: input the total shallow feature and the total deep feature into the feature fusion network to obtain the fusion features of the multispectral and infrared images; wherein the fusion features have both the texture structure information and the abstract essence information; Step 7: input the fusion features obtained in step 6 into the image reconstruction network to obtain the super-resolution image.

2. The super-resolution reconstruction method of claim 1, wherein, The double-branch global residual network in step 4 is a double-branch network with the same network structure but not sharing the weights, and each single-branch network is composed of multiple residual blocks; each residual block has a skip connection.

3. The super-resolution reconstruction method of claim 2, wherein, The operation method of the residual block is as follows: the input shallow features of the multispectral image or the input shallow features of the infrared image are subjected to one convolution and batch normalization, then activated by a PreLU function, and then subjected to one convolution and batch normalization again, and the result is summed with the input element by element.

4. The super-resolution reconstruction method according to any one of claims 1 to 3, characterized in that, The fusion method of the feature fusion network in step 6 is as follows: R = F high - F low F = C 3×3 [C 1×1 (R)]+F high where C 3×3 represents a 3*3 convolution; C 1×1 represents a 1*1 convolution; R represents information present in one source but not the other; F high represents total deep features; F l0w represents total shallow features.

5. The super-resolution reconstruction method of claim 1, wherein, The pseudo-Siamese feature extraction network is a double-branch independent convolutional flow with the same structure.

6. A multi-modal image feature fusion super-resolution reconstruction system, characterized in that, The method comprises the following steps: a scale transformation module, used for magnifying the multispectral image and the infrared image to the scale of the super-resolution image respectively; a shallow feature extraction module, used for inputting the multispectral image and the infrared image after scale transformation into the pseudo-Siamese feature extraction network to obtain the shallow features of the multispectral image and the shallow features of the infrared image respectively; wherein the shallow features are used to represent the texture structure information; a first feature superimposition module, used for superimposing the shallow features of the multispectral image and the shallow features of the infrared image to obtain a total shallow feature; a deep feature extraction module, used for inputting the shallow features of the multispectral image and the shallow features of the infrared image into the double-branch global residual network respectively to obtain the deep features of the multispectral image and the deep features of the infrared image; wherein the deep features are used to represent the abstract essence; a second feature superimposition module, used for superimposing the deep features of the multispectral image and the deep features of the infrared image to obtain a total deep feature; a feature fusion module, used for fusing the total shallow feature and the total deep feature to obtain the fusion features having both the texture structure information and the abstract essence information; a reconstruction module, used for inputting the fusion features into the image reconstruction network to obtain the super-resolution image.

7. The super-resolution reconstruction system of claim 6, wherein, The double-branch global residual network in the deep feature extraction module is a double-branch network with the same network structure but not sharing weights, and the single-branch network is composed of multiple residual blocks.

8. The super-resolution reconstruction system of claim 7, wherein, The residual block structure sequentially comprises a first convolutional layer, a first batch normalization layer, an activation function, a second convolutional layer and a second batch normalization layer from input to output.

9. The super-resolution reconstruction system of any of claims 6 to 8, wherein, The fusion method of the feature fusion network in the feature fusion module is: R = F high - F low F = C 3×3 [C 1×1 (R)]+F high where C 3×3 represents a 3*3 convolution; C 1×1 represents a 1*1 convolution; R represents information present in one source but not the other; F high represents total deep features; F low represents total shallow features.

10. The super-resolution reconstruction system of claim 6, wherein, The pseudo-Siamese feature extraction network in the shallow feature extraction module is a double-branch independent convolutional stream with the same structure.

Images