Multi-super-resolution result reconstruction method based on mutual guidance of infrared light and visible light

Abstract

The invention discloses an infrared and visible light mutual guidance multi-super-resolution result reconstruction method, which belongs to the technical field of image processing, and comprises the following steps: carrying out feature extraction on a low-resolution first infrared image and a high-resolution first visible light image to obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector; mutually guiding and refining the two high-dimensional feature vectors to obtain a first refining infrared feature vector and a first refining visible light feature vector; guiding the first refining visible light feature vector to generate a visible light infrared image; single infrared super-resolution reconstruction is carried out on the first refining infrared feature vector, and mixed branch super-resolution reconstruction is carried out on the first refining infrared feature vector; and carrying out weighted fusion on the basis of the visible light infrared image, the single infrared super-resolution reconstruction result and the mixed branch super-resolution reconstruction result to obtain a multi-branch fusion super-resolution reconstruction result. According to the invention, the diversity of model super-resolution reconstruction results is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a multi-super-resolution reconstruction result method guided by infrared and visible light. Background Art

[0002] With the rapid development of remote sensing, medical imaging, monitoring systems and other fields, the requirements for image resolution are getting higher and higher. Super-resolution technology (SR) has emerged to improve image clarity and enhance image details through software algorithms.

[0003] In the existing infrared super-resolution reconstruction technology, most mainstream methods use single infrared modality information for feature extraction, self-attention mechanism and other technologies for super-resolution reconstruction, and also retain a lot of detail information. However, for weak texture areas, these super-resolution methods that only use a single modality usually blur these textures, resulting in serious information loss.

[0004] For the fusion magnification method of infrared and visible light modalities, they can effectively solve the problem that weak texture areas cannot be restored. For example, the Chinese patent document with the publication number CN109697695B proposes a visible light image-guided ultra-low resolution thermal infrared image interpolation algorithm. Given an input image I and a guide image G, the output image O can be obtained through multiple groups of iterative optimization. The technical solution of this invention adopts a layered multi-group iterative interpolation method to obtain the result image, gradually interpolating from the coarse layer to the fine layer to finally obtain a high-resolution thermal infrared image; three double smoothing interpolation and secondary guided interpolation are used in each layer, and the result image is used as the sparse input of the next fine layer; iterative interpolation to the finest layer in the above manner can finally obtain a clear high-resolution thermal infrared image.

[0005] This algorithm uses high-resolution visible light images to extract the surface distribution of scene objects based on the law of correlation between the surface distribution of scene objects and the surface distribution of scene objects, and guides the interpolation of ultra-low resolution thermal infrared sensor data to obtain the temperature distribution of the real scene. However, this interpolation method will cause serious aliasing problems and is very abrupt in the transmission of features. It only superimposes high-frequency or low-frequency information, which will cause serious modal conflict problems and hinder the interpretation of temperature information.

[0006] Therefore, a multi-super-resolution reconstruction method with mutual guidance of infrared and visible light is needed. Without introducing additional modal conflict information, the two modal information of infrared and visible light can be integrated to generate clearer super-resolution reconstruction results and improve the diversity of super-resolution reconstruction results. Summary of the invention

[0007] In order to solve the above problems, the present invention provides a multi-super-resolution reconstruction result method with mutual guidance of infrared and visible light to solve the problems in the prior art.

[0008] In order to achieve the above-mentioned invention object, the present invention proposes a multi-super-resolution reconstruction method with infrared and visible light guiding each other, comprising:

[0009] S1: Acquire a low-resolution first infrared image and a high-resolution first visible light image, perform feature extraction on the first infrared image and the first visible light image, and obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector respectively;

[0010] S2: Mutually guiding and refining the infrared high-dimensional feature vector and the visible light high-dimensional feature vector to obtain a first refined infrared feature vector and a first refined visible light feature vector;

[0011] S3: Based on the first infrared image, the first refined visible light feature vector is guided to generate a visible light infrared image, the first refined infrared feature vector is subjected to single infrared super-resolution reconstruction to obtain a first reconstruction result, and the first refined visible light feature vector and the first refined infrared feature vector are subjected to mixed branch super-resolution reconstruction to obtain a second reconstruction result;

[0012] S4: Perform weighted fusion based on the visible light infrared image, the first reconstruction result and the second reconstruction result to obtain a multi-branch fusion super-resolution reconstruction result.

[0013] Furthermore, extracting features from the first infrared image and the first visible light image comprises the following steps:

[0014] Perform two convolution operations on the first infrared image and the first visible light image respectively to obtain corresponding preliminary features, and input the preliminary features after convolution into the ResNet network respectively to obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector.

[0015] Furthermore, guiding the infrared high-dimensional feature vector and the visible light high-dimensional feature vector mutually comprises the following steps:

[0016] The infrared high-dimensional feature vector and the visible light high-dimensional feature vector are feature spliced ​​and input into the UNet network to obtain corresponding infrared attention weights and visible light attention weights; based on the infrared attention weights and the visible light attention weights, the visible light high-dimensional feature vector and the infrared high-dimensional feature vector are guided respectively to obtain guided infrared feature vectors and visible light feature vectors.

[0017] Furthermore, refining the visible light high-dimensional feature vector comprises the following steps:

[0018] The guided infrared feature vector and the visible light high-dimensional feature vector are feature concatenated and input into the ResNet network for learning to obtain a first residual, and the sum of the first residual and the visible light high-dimensional feature vector is used as the refined first visible light feature vector.

[0019] Furthermore, refining the infrared high-dimensional feature vector includes the following steps:

[0020] The guided visible light feature vector and the infrared high-dimensional feature vector are feature concatenated and input into the ResNet network for learning to obtain a second residual, and the sum of the second residual and the infrared high-dimensional feature vector is used as the refined first infrared feature vector.

[0021] Obtaining a first refined infrared feature vector and a first refined visible light feature vector comprises the following steps:

[0022] The refined first visible light feature vector and the refined first infrared feature vector are again subjected to mutual guidance and refining operations to obtain a first refined infrared feature vector and a first refined visible light feature vector, wherein the first refined infrared feature vector and the first refined visible light feature vector are feature vectors after fully integrating infrared features and visible light features.

[0023] Furthermore, generating a visible light infrared image includes the following steps:

[0024] The first refined visible light feature vector is input into the ResNet network for residual extraction to obtain a third residual, a convolution operation is performed on the third residual, and a secondary convolution operation is performed on the first infrared image after the upsampling operation and the convolved third residual to obtain a visible light infrared image after feature integration.

[0025] Furthermore, the single infrared super-resolution reconstruction includes the following steps:

[0026] The first refined infrared feature vector is input into the ResNet network for residual extraction to obtain a fourth residual, a convolution operation is performed on the fourth residual, a secondary convolution operation is performed on the first infrared image after the upsampling operation and the convolved fourth residual to obtain a single infrared super-resolution reconstruction result, and the single infrared super-resolution reconstruction result is defined as a first reconstruction result.

[0027] Furthermore, the hybrid branch super-resolution reconstruction includes the following steps:

[0028] The first refined visible light feature vector and the first refined infrared feature vector are feature concatenated and input into the UNet network to learn corresponding infrared attention weights, the first refined visible light feature vector is guided again based on the infrared attention weights to obtain a guided first visible light feature vector, the guided first visible light feature vector and the first refined infrared feature vector are feature concatenated and input into the ResNet network to learn to obtain a fifth residual required for hybrid branch super-resolution reconstruction, a convolution operation is performed on the fifth residual, a secondary convolution operation is performed on the first infrared image that has undergone an upsampling operation and the fifth residual after convolution to obtain a hybrid branch super-resolution reconstruction result, and the hybrid branch super-resolution reconstruction result is defined as a second reconstruction result.

[0029] Furthermore, obtaining the multi-branch fusion super-resolution reconstruction result includes the following steps:

[0030] Corresponding weights are assigned to the visible light infrared image, the first reconstruction result, and the second reconstruction result, and weighted fusion is performed to obtain a multi-branch fusion super-resolution reconstruction result.

[0031] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0032] The present invention uses infrared and visible light modal information to guide each other for super-resolution reconstruction, and can simultaneously obtain visible light-like infrared results, visible light-guided infrared super-resolution reconstruction results, and single infrared super-resolution reconstruction results from one network structure. It can also simply use one branch to obtain results that are better than super-resolution reconstruction using only single modal information, and can perform weighted fusion on the results of the three branches to obtain super-resolution reconstruction results of various styles for users to choose from. The algorithm in the present invention has the characteristics of strong robustness, practicality, universality, etc., and can be well applied to infrared and visible light dual-light systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of the steps of a multi-super-resolution reconstruction method of infrared and visible light guided by each other according to the present invention;

[0034] Figure 2 It is the overall architecture diagram of the present invention;

[0035] Figure 3 is a first network structure diagram of the present invention;

[0036] Figure 4 A network architecture diagram of a mutual guidance module in a hybrid branch of the present invention;

[0037] Figure 5 This is a diagram of the visible light feature refining network architecture of the present invention;

[0038] Figure 6 This is a diagram of the infrared feature refining network architecture of the present invention;

[0039] Figure 7 This is a diagram of the hybrid branch super-resolution reconstruction network architecture of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0042] like Figure 1 As shown, a multi-super-resolution reconstruction method with mutual guidance of infrared and visible light includes:

[0043] S1: Acquire a low-resolution first infrared image and a high-resolution first visible light image, perform feature extraction on the first infrared image and the first visible light image, and obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector, respectively.

[0044] Specifically, the first infrared image and the first visible light image are images that have been registered in advance, ensuring the geometric consistency between the images for better subsequent feature extraction. The specific registration method is, for example, a deep learning method or a registration parameter calibration method. Feature extraction can use a variety of deep learning algorithms for feature extraction, such as ResNet, DenseNet, Transformer and other deep learning algorithms. In the present invention, we use ResNet for feature extraction, as shown in the following example. Figure 2 As shown, it is the overall architecture diagram, which divides the overall architecture diagram into three branches: visible light branch, hybrid branch and infrared branch, which are explained separately.

[0045] Step S2: The infrared high-dimensional feature vector and the visible light high-dimensional feature vector are mutually guided and refined to obtain a first refined infrared feature vector and a first refined visible light feature vector.

[0046] Specifically, the present invention generates attention weights corresponding to infrared high-dimensional feature vectors and visible light high-dimensional feature vectors based on the UNet algorithm to perform the first mutual guidance. Through mutual guidance, the features of the two modalities can be more effectively integrated to generate feature vectors that are more suitable for specific tasks. The guided feature vectors are feature spliced ​​to obtain infrared feature vectors and visible light feature vectors. The extraction of the two feature vectors provides a basis for subsequent feature refinement of visible light branches and infrared branches.

[0047] The guided infrared feature vector is used to perform the first refinement on the visible light high-dimensional feature vector. During the refining process, special attention can be paid to the outstanding visible light high-dimensional feature vectors, and these features can be strengthened to make them more suitable for the subsequent use of visible light infrared images. Similarly, the guided visible light feature vector is used to perform the first refinement on the infrared high-dimensional feature vector. Infrared images often have low resolution. Using high-resolution visible light feature vectors can provide more details and help improve the resolution of infrared images, making them more suitable for the subsequent infrared super-resolution reconstruction module.

[0048] In actual use, multiple iterative operations are required to generate more effective mutual guidance information between branches. Therefore, multiple mutual guidance and refinement operations are performed on the first visible light feature vector and the first infrared feature vector to obtain the first refined infrared feature vector and the first refined visible light feature vector.

[0049] S3: Based on the first infrared image, the first refined visible light feature vector is guided to generate a visible light infrared image, a single infrared super-resolution reconstruction is performed on the first refined infrared feature vector to obtain a first reconstruction result, and a mixed branch super-resolution reconstruction is performed on the first refined visible light feature vector and the first refined infrared feature vector to obtain a second reconstruction result.

[0050] Specifically, under the guidance of the infrared features in the first infrared image, the first refined visible light feature vector generates a synthetic image of a visible light-like infrared image, which combines the high resolution and rich color information of the visible light image but still retains some characteristics of the infrared image. This guidance can help generate higher quality high-resolution images and reduce blur and artifacts in the reconstruction process.

[0051] The first reconstruction result is an image obtained by processing the first refined infrared feature vector through the single infrared super-resolution reconstruction technology. The first refined infrared feature vector is a high-dimensional feature representation that has been refined n times. These feature vectors are extracted and optimized from the original infrared image. These refined feature vectors are used as input and the super-resolution reconstruction algorithm is applied to increase the resolution of the image. The generated first reconstruction result is an image with a higher pixel density than the original input first infrared image, thereby providing more details and information.

[0052] The second reconstruction result is an image output from a hybrid branch super-resolution reconstruction process using the first refined visible light feature vector and the first refined infrared feature vector. The hybrid branch super-resolution reconstruction utilizes features of the two modalities, and the generated image may be superior to the reconstruction result of a single modality in terms of detail and clarity.

[0053] S4: Perform weighted fusion based on the visible light infrared image, the first reconstruction result and the second reconstruction result to obtain a multi-branch fusion super-resolution reconstruction result.

[0054] Specifically, through the above steps, we can obtain visible light-like infrared images, single infrared super-resolution reconstruction results (first reconstruction results) and hybrid branch super-resolution reconstruction results (second reconstruction results). The results of the three branches have their own unique characteristics. The visible light branch has its own visible light high-resolution characteristics. Using infrared information for guidance can generate infrared-like images with clear high-resolution visible light features; the infrared branch is completely loyal to its own infrared modal information and does not transfer visible light features. In the process of mutual feature correction, the generated high-resolution infrared information is made more accurate; the hybrid branch introduces additional high-resolution visible light guidance information on the basis of being as loyal as possible to the infrared modal information, so as to generate clearer and more faithful infrared super-resolution results without introducing additional modal conflict information; it can be combined according to different weight alignments to generate a fused super-resolution reconstruction result.

[0055] As a preferred technical solution of the present invention, extracting features from the first infrared image and the first visible light image includes the following steps:

[0056] The first infrared image and the first visible light image are convolved twice to obtain corresponding preliminary features, and the preliminary features after convolution are input into the ResNet network to obtain infrared high-dimensional feature vectors and visible light high-dimensional feature vectors.

[0057] Specifically, the ResNet residual neural network is set as the first network structure, and the first infrared image and the first visible light image are respectively extracted based on the first formula and the second formula. The first formula is: ir =ResNet(Conv(Conv(I ir ))), the second formula is: F vis =ResNet(Conv(Conv(I vis ))), where I ir and I vis are the first infrared image and the first visible light image, respectively, ir and F visare the extracted infrared high-dimensional feature vector and visible light high-dimensional feature vector respectively, and Conv is the convolution operation, such as Figure 3 As shown in the figure, it is a first network structure diagram, which includes 3 groups of continuous residual modules, wherein each group of residual modules includes a 3*3 convolution layer, ReLU (activation function), continuous Conv 3*3 and ReLU. After a series of convolution operations and activation function processing, the features in images of different resolutions are integrated, so that the low-resolution infrared image can learn richer details and structural information from the high-resolution visible light image, thereby finally obtaining infrared high-dimensional feature vectors and visible light high-dimensional feature vectors.

[0058] The mutual guidance of infrared high-dimensional feature vectors and visible light high-dimensional feature vectors includes the following steps:

[0059] The infrared high-dimensional feature vector and the visible light high-dimensional feature vector are feature concatenated and input into the UNet network to obtain the corresponding infrared attention weights and visible light attention weights. Based on the infrared attention weights and the visible light attention weights, the visible light high-dimensional feature vector and the infrared high-dimensional feature vector are guided respectively to obtain the guided infrared feature vector and visible light feature vector.

[0060] Specifically, the infrared high-dimensional feature vector and the visible light high-dimensional feature vector are concatenated through the third and fourth formulas, and then they are respectively input into the 5-layer UNet network to generate the corresponding infrared attention weights and visible light attention weights The third formula is: The fourth formula is: Among them, UNet is a convolutional neural network, (·,·) is a splicing operation, and other attention mechanisms can also achieve the same purpose. The infrared feature vector after guidance is generated based on the fifth and sixth formulas. and the visible light feature vector The fifth formula is: The sixth formula is: Among them, ⊙ is the point multiplication operator. This guidance is the first mutual guidance. The objects of guidance are infrared high-dimensional feature vectors and visible light high-dimensional feature vectors. In the process of mutual guidance, the generated attention weights help to identify and emphasize key features in different modalities, so that infrared high-dimensional feature vectors and visible light high-dimensional feature vectors can be more effectively combined during feature fusion.

[0061] Refining the high-dimensional feature vector of visible light includes the following steps:

[0062] The guided infrared feature vector and the visible light high-dimensional feature vector are concatenated and input into the ResNet network for learning to obtain the first residual. The sum of the first residual and the visible light high-dimensional feature vector is taken as the refined first visible light feature vector.

[0063] Specifically, Figure 5 As shown in the figure, it is the visible light feature refinement network architecture diagram. First, the visible light high-dimensional feature vector F vis and the infrared feature vector after guidance Get splicing Then input it into ResNet to learn and get the first residual. The first residual refers to the difference between the concatenated feature vector and the original visible light high-dimensional feature vector F vis The difference between the first residual and the original visible light high-dimensional feature vector is then obtained by adding the first residual to the original visible light high-dimensional feature vector to obtain the first refined visible light high-dimensional feature vector, that is, the first visible light feature vector The refined first visible light feature vector It can be calculated based on the seventh formula: The seventh formula is:

[0064] Refining the infrared high-dimensional feature vector includes the following steps:

[0065] The guided visible light feature vector and the infrared high-dimensional feature vector are concatenated and input into the ResNet network for learning to obtain the second residual. The sum of the second residual and the infrared high-dimensional feature vector is used as the refined first infrared feature vector.

[0066] Specifically, Figure 6 As shown in the figure, it is the infrared feature refinement network architecture diagram, which transforms the infrared high-dimensional feature vector F ir and the visible light feature vector after the first guidance Get splicing Then input it into ResNet to learn and get the second residual, which is the concatenated feature vector Compared with the original infrared high-dimensional feature vector F ir The difference between the second residual and the original infrared high-dimensional feature vector is obtained by adding the second residual to the original infrared high-dimensional feature vector to obtain the first refined infrared high-dimensional feature vector, that is, the first infrared feature vector The refined first infrared feature vector It can be calculated based on the eighth formula: The eighth formula is:

[0067] Obtaining a first refined infrared feature vector and a first refined visible light feature vector comprises the following steps:

[0068] The refined first visible light feature vector and the refined first infrared feature vector are again subjected to mutual guidance and refining operations to obtain a first refined infrared feature vector and a first refined visible light feature vector. The first refined infrared feature vector and the first refined visible light feature vector are feature vectors after fully integrating infrared features and visible light features.

[0069] Specifically, this embodiment is described by taking three mutual guidance and refining operations as an example, then n=3, 1≤m<3.

[0070] The above steps for the first iteration have been explained and will not be repeated here.

[0071] The second iteration process is as follows: at this time, m = 1, and the first visible light feature vector after the first refinement is obtained by using the same 5-layer UNet network as the third formula. and the first infrared feature vector after the first refinement Conduct a second mutual guidance to obtain the corresponding infrared attention weight and visible light attention weights The attention weight generation process is as follows: The ninth formula is: The tenth formula is: Based on the eleventh formula: Generate the visible light feature vector after the second guidance Based on the twelfth formula: Generate infrared feature vector after second guidance Based on the thirteenth formula: Generate the first infrared feature vector after the second refinement Based on the fourteenth formula: Generate the first visible light feature vector after the second refinement

[0072] The third iteration process is as follows: At this time, m = 2, and the first visible light feature vector after the second refinement is and the first infrared feature vector after the second refinement Conduct the third mutual guidance to obtain the corresponding infrared attention weight and visible light attention weights The attention weight generation process is as follows: The ninth formula is: The tenth formula is: Based on the eleventh formula: Generate the visible light feature vector after the third guidance Based on the twelfth formula: Generate infrared feature vector after the third guidance Based on the thirteenth formula: Generate the first infrared feature vector after the third refinement Based on the fourteenth formula: Generate the first visible light feature vector after the third refinement

[0073] The first infrared feature vector obtained after three iterative operations is defined as the first refined infrared feature vector, and the first visible light feature vector obtained after three iterative operations is defined as the first refined visible light feature vector. Multiple iterations enable the model to learn and extract infrared features and visible light features more deeply. Each iteration may capture more subtle or different levels of feature information. Through the mutual guidance of multiple iterations, the complementary information between visible light and infrared images can be better utilized, which can significantly improve the quality of feature vectors, enhance the performance of the model, and provide more reliable input for tasks such as super-resolution reconstruction.

[0074] Generating a visible-light infrared image includes the following steps:

[0075] The first refined visible light feature vector is input into the ResNet network for residual extraction to obtain the third residual, a convolution operation is performed on the third residual, and a secondary convolution operation is performed on the first infrared image after the upsampling operation and the convolved third residual to obtain a visible light infrared image after feature integration.

[0076] Specifically, the first refined visible light feature vector after the third refinement Input to ResNet for residual extraction to obtain the third residual R vis , the residual of the first refined visible light feature vector is extracted based on the fifteenth formula, and the fifteenth formula is: The third residual reflects the difference between the first refined visible light feature vector and the first visible light vector. Based on the sixteenth formula, the visible light infrared image S is generated. vis , the sixteenth formula is: S vis =Conv(up(I ir )+Conv(R vis ), where up is an upsampling operation, for the first infrared image I ir An upsampling operation is performed. Upsampling can improve the resolution of the first infrared image to match the high-resolution image. The upsampling operation can be implemented using nearest neighbor interpolation, linear interpolation, bilinear interpolation, and cubic interpolation. The third residual is learned through convolutional layer processing. The third residual captures the missing or adjusted parts of the visible light image. Additional convolutional layers are applied to convert the visible light features so that it has characteristics similar to infrared images, and finally a visible light-like infrared image is generated.

[0077] Single infrared super-resolution reconstruction includes the following steps:

[0078] The first refined infrared feature vector is input into the ResNet network for residual extraction to obtain a fourth residual, a convolution operation is performed on the fourth residual, a secondary convolution operation is performed on the first infrared image after the upsampling operation and the convolved fourth residual to obtain a single infrared super-resolution reconstruction result, and the single infrared super-resolution reconstruction result is defined as the first reconstruction result.

[0079] Specifically, similar to the step of generating a visible light infrared image, the first refined infrared feature vector after the third refinement is Input to ResNet for residual extraction to obtain the fourth residual R ir , the residual extraction of the first refined infrared feature vector is performed based on the seventeenth formula, and the seventeenth formula is: The fourth residual captures the difference between the low-resolution infrared image and the first refined infrared feature vector, and the first infrared image I is obtained based on the eighteenth formula ir Perform upsampling operation and generate the first reconstruction result S ir , the first reconstruction result is the single infrared super-resolution reconstruction result, and the eighteenth formula is: S ir =Conv(up(I ir )+Conv(R ir )), for the first infrared image I ir An upsampling operation is performed to improve the resolution of the image, and the fourth residual is learned through convolution layer processing. Then, the residual after convolution processing and the first infrared image after upsampling processing are convolved again. The second convolution operation fuses the infrared features obtained by two different methods to obtain a single infrared super-resolution reconstruction result.

[0080] The hybrid branch super-resolution reconstruction includes the following steps:

[0081] The first refined visible light feature vector and the first refined infrared feature vector are feature concatenated and input into the UNet network to learn the corresponding infrared attention weights; the first refined visible light feature vector is guided again based on the infrared attention weights to obtain the guided first visible light feature vector; the guided first visible light feature vector and the first refined infrared feature vector are feature concatenated and input into the ResNet network to learn to obtain the fifth residual required for hybrid branch super-resolution reconstruction; a convolution operation is performed on the fifth residual; a secondary convolution operation is performed on the upsampled first infrared image and the convolved fifth residual to obtain a hybrid branch super-resolution reconstruction result; and the hybrid branch super-resolution reconstruction result is defined as the second reconstruction result.

[0082] Specifically, Figure 7 As shown in the figure, it is a hybrid branch super-resolution reconstruction network architecture diagram. When n=3, based on the first refined visible light feature vector after the third refinement and the first refined infrared feature vector Perform feature splicing based on the nineteenth formula: Generate IR attention weights after the fourth instruction Based on the 20th formula: For the first refined visible light feature vector Perform the fourth instruction and obtain the first visible light feature vector after the fourth instruction Based on the twenty-first formula: The first visible light feature vector after the fourth instruction and the first refined infrared feature vector Perform feature splicing to generate the residual R required for constructing hybrid branch super-resolution reconstruction mix , and then based on the 22nd formula: S mix =Conv(up(I ir )+Conv(R mix )) to generate a hybrid branch super-resolution reconstruction result. This process ensures that the super-resolution reconstruction result does not carry redundant modal conflict information, can better use the common visible light features to restore weak texture areas, and provide effective high-resolution guidance information for the super-resolution of low-resolution infrared images.

[0083] Obtaining multi-branch fusion super-resolution reconstruction results includes the following steps:

[0084] Corresponding weights are assigned to the visible light infrared image, the first reconstruction result and the second reconstruction result, and weighted fusion is performed to obtain a multi-branch fusion super-resolution reconstruction result.

[0085] Specifically, based on the twenty-third formula, the visible light infrared image, the first reconstruction result and the second reconstruction result are weightedly fused to obtain a multi-branch fusion super-resolution reconstruction result out. The twenty-third formula is: out = a1S ir +a2S vis +a3S mix , where a1>0, a2>0, a3>0, and a1+a2+a3=1, a1, a2 and a3 are the first weight, the second weight and the third weight respectively, the first weight is the weight of the single infrared super-resolution reconstruction result of the infrared branch, the second weight is the weight of the visible light class infrared image of the visible light branch, and the third weight is the weight of the mixed branch result.

[0086] At the same time, during the model training process, the present invention can also randomly set the input first infrared image and the first visible light image to 0 respectively to simulate the situation where a single modal information is lost. After the model is fitted, the three-branch network has a robust effect of modal loss, and even if one modal information input is missing, an effective super-resolution reconstruction result can be obtained. For example, when only visible light infrared image results need to be generated, the present invention can use only the visible light branch, set the infrared image to 0 forever, and only input the visible light single modal image. The three-branch model in the present invention can also generate excellent high-resolution infrared images; when only single infrared super-resolution reconstruction results need to be generated, the present invention can use only the infrared branch, set the visible light image to 0 forever, and only input the infrared single modal image. The three-branch model in the present invention can produce results that are better than single infrared super-resolution reconstruction.

[0087] Compared with the prior art, the present invention has at least the following beneficial effects:

[0088] The present invention uses infrared and visible light modal information to guide each other for super-resolution reconstruction, and can simultaneously obtain visible light-like infrared results, visible light-guided infrared super-resolution reconstruction results, and single infrared super-resolution reconstruction results from one network structure. It can also simply use one branch to obtain results that are better than super-resolution reconstruction using only single modal information, and can perform weighted fusion on the results of the three branches to obtain super-resolution reconstruction results of various styles for users to choose from. The algorithm in the present invention has the characteristics of strong robustness, practicality, universality, etc., and can be well applied to infrared and visible light dual-light systems.

[0089] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0090] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0091] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0093] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-super-resolution reconstruction method with mutual guidance of infrared and visible light, It is characterized in that The method comprises the following steps: S1: Acquire a low-resolution first infrared image and a high-resolution first visible light image, perform feature extraction on the first infrared image and the first visible light image, and obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector respectively; S2: Mutually guiding and refining the infrared high-dimensional feature vector and the visible light high-dimensional feature vector to obtain a first refined infrared feature vector and a first refined visible light feature vector; S3: Based on the first infrared image, the first refined visible light feature vector is guided to generate a visible light infrared image, the first refined infrared feature vector is subjected to single infrared super-resolution reconstruction to obtain a first reconstruction result, and the first refined visible light feature vector and the first refined infrared feature vector are subjected to mixed branch super-resolution reconstruction to obtain a second reconstruction result; S4: Perform weighted fusion based on the visible light infrared image, the first reconstruction result and the second reconstruction result to obtain a multi-branch fusion super-resolution reconstruction result.

2. The method according to claim 1, It is characterized in that Extracting features from the first infrared image and the first visible light image comprises the following steps: Perform two convolution operations on the first infrared image and the first visible light image respectively to obtain corresponding preliminary features, and input the preliminary features after convolution into the ResNet network respectively to obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector.

3. The method according to claim 2, It is characterized in that The mutual guidance of the infrared high-dimensional feature vector and the visible light high-dimensional feature vector comprises the following steps: The infrared high-dimensional feature vector and the visible light high-dimensional feature vector are feature spliced ​​and input into the UNet network to obtain corresponding infrared attention weights and visible light attention weights; based on the infrared attention weights and the visible light attention weights, the visible light high-dimensional feature vector and the infrared high-dimensional feature vector are guided respectively to obtain guided infrared feature vectors and visible light feature vectors.

4. The method according to claim 3, It is characterized in that Refining the visible light high-dimensional feature vector comprises the following steps: The guided infrared feature vector and the visible light high-dimensional feature vector are feature concatenated and input into the ResNet network for learning to obtain a first residual, and the sum of the first residual and the visible light high-dimensional feature vector is used as the refined first visible light feature vector.

5. The method according to claim 4, It is characterized in that Refining the infrared high-dimensional feature vector includes the following steps: The guided visible light feature vector and the infrared high-dimensional feature vector are feature concatenated and input into the ResNet network for learning to obtain a second residual, and the sum of the second residual and the infrared high-dimensional feature vector is used as the refined first infrared feature vector.

6. The method according to claim 5, It is characterized in that Obtaining a first refined infrared feature vector and a first refined visible light feature vector comprises the following steps: The refined first visible light feature vector and the refined first infrared feature vector are again subjected to mutual guidance and refining operations to obtain a first refined infrared feature vector and a first refined visible light feature vector, wherein the first refined infrared feature vector and the first refined visible light feature vector are feature vectors after fully integrating infrared features and visible light features.

7. The method according to claim 6, It is characterized in that Generating a visible-light infrared image includes the following steps: The first refined visible light feature vector is input into the ResNet network for residual extraction to obtain a third residual, a convolution operation is performed on the third residual, and a secondary convolution operation is performed on the first infrared image after the upsampling operation and the convolved third residual to obtain a visible light infrared image after feature integration.

8. The method according to claim 6, It is characterized in that Single infrared super-resolution reconstruction includes the following steps: The first refined infrared feature vector is input into the ResNet network for residual extraction to obtain a fourth residual, a convolution operation is performed on the fourth residual, a secondary convolution operation is performed on the first infrared image after the upsampling operation and the convolved fourth residual to obtain a single infrared super-resolution reconstruction result, and the single infrared super-resolution reconstruction result is defined as a first reconstruction result.

9. The method according to claim 8, It is characterized in that The hybrid branch super-resolution reconstruction includes the following steps: The first refined visible light feature vector and the first refined infrared feature vector are feature concatenated and input into the UNet network to learn corresponding infrared attention weights, the first refined visible light feature vector is guided again based on the infrared attention weights to obtain a guided first visible light feature vector, the guided first visible light feature vector and the first refined infrared feature vector are feature concatenated and input into the ResNet network to learn to obtain a fifth residual required for hybrid branch super-resolution reconstruction, a convolution operation is performed on the fifth residual, a secondary convolution operation is performed on the first infrared image that has undergone an upsampling operation and the fifth residual after convolution to obtain a hybrid branch super-resolution reconstruction result, and the hybrid branch super-resolution reconstruction result is defined as a second reconstruction result.

10. The method according to claim 9, It is characterized in that Obtaining multi-branch fusion super-resolution reconstruction results includes the following steps: Corresponding weights are assigned to the visible light infrared image, the first reconstruction result, and the second reconstruction result, and weighted fusion is performed to obtain a multi-branch fusion super-resolution reconstruction result.

Images