Multi-super-resolution reconstruction result method with mutual guidance between infrared light and visible light

Abstract

The present invention belongs to the technical field of image processing. Disclosed is a multi-super-resolution reconstruction result method with mutual guidance between infrared light and visible light. The method comprises: performing feature extraction on a low-resolution first infrared image and a high-resolution first visible light image to obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector; performing mutual guidance and refining on the two high-dimensional feature vectors to obtain a first refined infrared feature vector and a first refined visible light feature vector; guiding the first refined visible light feature vector to generate a visible light infrared-like image; performing single-infrared super-resolution reconstruction on the first refined infrared feature vector, and performing hybrid branch super-resolution reconstruction on same and the first refined visible light feature vector; and, on the basis of the visible light infrared-like image, the single-infrared super-resolution reconstruction result and the hybrid branch super-resolution reconstruction result, performing weighted fusion to obtain a multi-branch-fused super-resolution reconstruction result. The present invention improves the diversity of super-resolution reconstruction results of models.

Description

Infrared and visible light mutual guidance multi-super resolution reconstruction result method TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an infrared and visible light mutual guidance multi-super resolution reconstruction result method. BACKGROUND

[0002] With the rapid development of remote sensing detection, medical imaging, monitoring system and other fields, the requirement for image resolution is getting higher and higher. Super-resolution technology (SR) emerges as the times require, aiming to improve the definition of images and enhance image details through software algorithms.

[0003] In the existing infrared super-resolution reconstruction technology, the mainstream method mostly uses single infrared modal information for feature extraction, self-attention mechanism and other technologies for super-resolution reconstruction, and also retains a lot of detailed information. However, for weak texture areas, these super-resolution methods using only single modal usually blur these textures, resulting in serious loss of information.

[0004] For the fusion and amplification method of infrared and visible light two modalities, they can effectively solve the problem that weak texture areas cannot be recovered. For example, the Chinese patent document with publication number CN109697695B proposes a visible light image guided super-low resolution thermal infrared image interpolation algorithm. An input image I and a guide image G are given, and an output image O can be obtained through multiple iterations. The invention adopts a hierarchical multi-group iterative interpolation method to obtain a result image, which is gradually interpolated from a coarse layer to a fine layer to finally obtain a high-resolution thermal infrared image. In each layer, one three double smooth interpolation and twice guided interpolation are used to obtain a result image as the sparse input of the next fine layer. Through the above iterative interpolation to the most fine layer, a clear high-resolution thermal infrared image can be finally obtained.

[0005] The algorithm uses the high-resolution visible light image to extract the scene object surface distribution map to guide the super-low resolution thermal infrared sensor data interpolation according to the law that the scene surface temperature distribution is related to the scene object surface distribution, so that the temperature distribution of the real scene can be obtained. However, this interpolation method will cause serious jagged problems, and the feature transmission is very harsh. Only through the superposition of high-frequency or low-frequency information, serious modal conflict problems will be caused, which will cause obstacles in interpreting temperature information.

[0006] Therefore, an infrared and visible light mutual guidance multi-super resolution reconstruction result method is needed, which can fuse infrared and visible light two modal information without introducing additional modal conflict information, generate clearer super-resolution reconstruction results, and improve the diversity of super-resolution reconstruction results. SUMMARY

[0007] To solve the above problems, the present application provides an infrared and visible light mutual guidance multi-super resolution reconstruction result method to solve the problems in the prior art.

[0008] To achieve the above-mentioned purposes of the application, the present application provides an infrared and visible light mutual guidance multi-super resolution reconstruction result method, comprising:

[0009] S1: obtaining a low-resolution first infrared image and a high-resolution first visible light image, and performing feature extraction on the first infrared image and the first visible light image to obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector, respectively;

[0010] S2: mutually guiding and refining the infrared high-dimensional feature vector and the visible light high-dimensional feature vector to obtain a first refined infrared feature vector and a first refined visible light feature vector;

[0011] S3: guiding the first refined visible light feature vector based on the first infrared image to generate a visible light infrared-like image, performing single infrared super resolution reconstruction on the first refined infrared feature vector to obtain a first reconstruction result, and performing hybrid branch super resolution reconstruction on the first refined visible light feature vector and the first refined infrared feature vector to obtain a second reconstruction result;

[0012] S4: performing weighted fusion based on the visible light infrared-like image, the first reconstruction result and the second reconstruction result to obtain a multi-branch fusion super resolution reconstruction result.

[0013] Further, the feature extraction on the first infrared image and the first visible light image comprises the following steps:

[0014] Performing two convolution operations on the first infrared image and the first visible light image, respectively, to obtain corresponding preliminary features, and inputting the convolved preliminary features into a ResNet network to obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector.

[0015] Further, the mutual guidance of the infrared high-dimensional feature vector and the visible light high-dimensional feature vector comprises the following steps:

[0016] Performing feature splicing on the infrared high-dimensional feature vector and the visible light high-dimensional feature vector and inputting them into a UNet network to obtain corresponding infrared attention weights and visible light attention weights, and guiding the visible light high-dimensional feature vector and the infrared high-dimensional feature vector based on the infrared attention weights and the visible light attention weights to obtain guided infrared feature vectors and visible light feature vectors.

[0017] Further, the refining of the visible light high-dimensional feature vector comprises the following steps:

[0018] The guided infrared feature vector and the visible light high-dimensional feature vector are spliced and input into the ResNet network for learning to obtain a first residual error, and the sum of the first residual error and the visible light high-dimensional feature vector is taken as a first refined visible light feature vector.

[0019] Further, the refining of the infrared high-dimensional feature vector comprises the following steps:

[0020] The guided visible light feature vector and the infrared high-dimensional feature vector are spliced and input into the ResNet network for learning to obtain a second residual error, and the sum of the second residual error and the infrared high-dimensional feature vector is taken as a first refined infrared feature vector.

[0021] The first refined infrared feature vector and the first refined visible light feature vector comprise the following steps:

[0022] The first refined visible light feature vector and the first refined infrared feature vector are guided and refined again to obtain a first refined infrared feature vector and a first refined visible light feature vector, and the first refined infrared feature vector and the first refined visible light feature vector are feature vectors after sufficient fusion of infrared features and visible light features.

[0023] Further, the generation of the visible light infrared-like image comprises the following steps:

[0024] The first refined visible light feature vector is input into the ResNet network to extract a third residual error, the third residual error is convolved, the first infrared image after up-sampling is twice convolved with the convolved third residual error, and a visible light infrared-like image after feature integration is obtained.

[0025] Further, the single infrared super-resolution reconstruction comprises the following steps:

[0026] The first refined infrared feature vector is input into the ResNet network to extract a fourth residual error, the fourth residual error is convolved, the first infrared image after up-sampling is twice convolved with the convolved fourth residual error, and a single infrared super-resolution reconstruction result is obtained, and the single infrared super-resolution reconstruction result is defined as a first reconstruction result.

[0027] Further, the mixed branch super-resolution reconstruction comprises the following steps:

[0028] The first refined visible light feature vector and the first refined infrared feature vector are feature spliced and input into the UNet network to learn a corresponding infrared attention weight, the first refined visible light feature vector is guided again based on the infrared attention weight, a guided first visible light feature vector is obtained, the guided first visible light feature vector and the first refined infrared feature vector are feature spliced and input into the ResNet network to learn a fifth residual required for mixed branch super-resolution reconstruction, the fifth residual is subjected to convolution operation, the first infrared image subjected to up-sampling operation is subjected to secondary convolution operation with the fifth residual subjected to convolution, and a mixed branch super-resolution reconstruction result is obtained, and the mixed branch super-resolution reconstruction result is defined as a second reconstruction result.

[0029] Further, obtaining the multi-branch fusion super-resolution reconstruction result comprises the following steps:

[0030] The visible light infrared image, the first reconstruction result and the second reconstruction result are assigned corresponding weights and weighted fused to obtain a multi-branch fusion super-resolution reconstruction result.

[0031] Compared with the prior art, the present application has at least the following advantages:

[0032] The present application uses infrared and visible light modal information to guide each other for super-resolution reconstruction, can obtain visible light infrared results, visible light guided infrared super-resolution reconstruction results and single infrared super-resolution reconstruction results from one network structure, can simply use one branch to obtain results better than super-resolution reconstruction using only single modal information, and can weight fuse three branch results to obtain super-resolution reconstruction results of various styles for user selection and use, the algorithm in the present application has strong robustness, practicality, universality and other characteristics, and can be well applied in infrared and visible light dual optical systems. BRIEF DESCRIPTION OF DRAWINGS

[0033] Fig. 1 is a step flow chart of a multi-super-resolution reconstruction result method of infrared and visible light mutual guidance according to the present application;

[0034] Fig. 2 is a general architecture diagram of the present application;

[0035] Fig. 3 is a first network structure diagram of the present application;

[0036] Fig. 4 is a mutual guidance module network architecture diagram in the mixed branch of the present application;

[0037] Fig. 5 is a visible light feature refinement network architecture diagram of the present application;

[0038] Fig. 6 is an infrared feature refinement network architecture diagram of the present application;

[0039] FIG. 7 is a mixed branch super-resolution reconstruction network architecture diagram of the present application. DETAILED DESCRIPTION

[0040] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0041] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various elements, but unless specifically stated, these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of the present application, the first xx script can be referred to as the second xx script, and similarly, the second xx script can be referred to as the first xx script.

[0042] As shown in FIG. 1, an infrared and visible light mutual guidance multi-super-resolution reconstruction result method includes:

[0043] S1: Obtain a low-resolution first infrared image and a high-resolution first visible light image, and perform feature extraction on the first infrared image and the first visible light image to obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector, respectively.

[0044] Specifically, the first infrared image and the first visible light image are images that have been registered in advance, ensuring geometric consistency between the images for better subsequent feature extraction. The specific registration method can be, for example, a method using deep learning or a method using registration parameter calibration. Feature extraction can use various deep learning algorithms for feature extraction, such as ResNet, DenseNet, Transformer, etc. In the present application, we use ResNet for feature extraction, as shown in FIG. 2, which is a general architecture diagram. The general architecture diagram is divided into three branches: a visible light branch, a mixed branch, and an infrared branch, which are described separately.

[0045] Step S2: Interactively guide and refine the infrared high-dimensional feature vector and the visible light high-dimensional feature vector to obtain a first refined infrared feature vector and a first refined visible light feature vector.

[0046] Specifically, the present application generates attention weights corresponding to infrared high-dimensional feature vectors and visible light high-dimensional feature vectors based on the UNet algorithm for the first mutual guidance. Through mutual guidance, the features of the two modalities can be more effectively fused to generate feature vectors more suitable for specific tasks. The guided feature vectors are then spliced to obtain infrared feature vectors and visible light feature vectors, which provide a basis for subsequent feature refinement of the visible light branch and the infrared branch by extracting the two types of feature vectors.

[0047] The guided infrared feature vectors are used to refine the visible light high-dimensional feature vectors for the first time. During the refinement process, particular attention can be paid to the outstanding visible light high-dimensional feature vectors, and these features can be strengthened to make them more suitable for subsequent use of visible light infrared image. Similarly, the guided visible light feature vectors are used to refine the infrared high-dimensional feature vectors for the first time. Infrared images often have low resolution, and the use of high-resolution visible light feature vectors can provide more details to help improve the resolution of infrared images and make them more suitable for subsequent use of infrared super-resolution reconstruction modules.

[0048] However, in actual use, multiple iterations are required to produce more effective mutual guidance information between branches. Therefore, multiple mutual guidance and refinement operations are performed on the first visible light feature vectors and the first infrared feature vectors to obtain first refined infrared feature vectors and first refined visible light feature vectors.

[0049] S3: guiding the first refined visible light feature vectors based on the first infrared image to generate a visible light infrared image, performing single infrared super-resolution reconstruction on the first refined infrared feature vectors to obtain a first reconstruction result, and performing hybrid branch super-resolution reconstruction on the first refined visible light feature vectors and the first refined infrared feature vectors to obtain a second reconstruction result.

[0050] Specifically, under the guidance of infrared features in the first infrared image, the first refined visible light feature vectors generate a synthetic image of the visible light infrared image. The visible light infrared image combines the high resolution and rich color information of the visible light image, but still retains some characteristics of the infrared image. This guidance can help generate higher-quality high-resolution images and reduce blurring and artifacts during the reconstruction process.

[0051] The first reconstruction result is an image obtained by processing the first refined infrared feature vectors through single infrared super-resolution reconstruction technology. The first refined infrared feature vectors are high-dimensional feature representations after n refinements. These feature vectors are extracted and optimized from the original infrared image. By using these refined feature vectors as input, a super-resolution reconstruction algorithm is applied to increase the resolution of the image. The generated first reconstruction result is an image with higher pixel density than the original input first infrared image, thereby providing more details and information.

[0052] The second reconstruction result is an image output by a mixed branch super-resolution reconstruction process using the first refined visible light feature vector and the first refined infrared feature vector, the mixed branch super-resolution reconstruction utilizing features of the two modalities, and the generated image can be superior to the single-modal reconstruction result in details and clarity.

[0053] S4: performing weighted fusion based on the visible light infrared image, the first reconstruction result and the second reconstruction result to obtain a multi-branch fusion super-resolution reconstruction result.

[0054] Specifically, through the above steps, we can obtain a visible light infrared image, a single infrared super-resolution reconstruction result (the first reconstruction result) and a mixed branch super-resolution reconstruction result (the second reconstruction result) respectively, and the results of the three branches have their own unique characteristics. The visible light branch has its own high-resolution visible light characteristics, and the use of infrared information can generate a clear high-resolution visible light feature infrared image. The infrared branch is completely loyal to its infrared modal information, and does not transfer visible light features. In the process of mutual correction of features, the generated high-resolution infrared information is more accurate. The mixed branch introduces additional high-resolution visible light guiding information on the basis of being loyal to the infrared modal information, so as to generate a clearer and more accurate infrared super-resolution result without introducing additional modal conflict information. The fusion super-resolution reconstruction result can be generated by combining the results according to different weights.

[0055] As a preferred technical solution of the present application, the feature extraction on the first infrared image and the first visible light image includes the following steps:

[0056] The first infrared image and the first visible light image are respectively subjected to twice convolution operations to obtain corresponding preliminary features, and the preliminary features after the convolution are respectively input into a ResNet network to obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector.

[0057] Specifically, the ResNet residual neural network is set as a first network structure, and the feature extraction on the first infrared image and the first visible light image is performed based on a first formula and a second formula, the first formula being: F ir = ResNet(Conv(Conv(I ir ))), and the second formula being: F vis = ResNet(Conv(Conv(I vis ))), wherein I ir and I vis are the first infrared image and the first visible light image respectively, F ir and F visrespectively, are extracted infrared high-dimensional feature vectors and visible light high-dimensional feature vectors, and Conv is a convolution operation, as shown in FIG. 3, which is a first network structure diagram, including three groups of continuous residual modules, wherein each group of residual modules includes a 3*3 convolution layer, a ReLU (activation function), a continuous Conv 3*3 and a ReLU, after processing by a series of convolution operations and activation functions, the features in different resolution images are integrated, so that the low-resolution infrared image can learn more rich details and structural information from the high-resolution visible light image, thereby finally obtaining the infrared high-dimensional feature vector and the visible light high-dimensional feature vector.

[0058] The mutual guidance of the infrared high-dimensional feature vector and the visible light high-dimensional feature vector includes the following steps:

[0059] The infrared high-dimensional feature vector and the visible light high-dimensional feature vector are feature spliced and input into a UNet network to obtain corresponding infrared attention weights and visible light attention weights, based on the infrared attention weights and the visible light attention weights, the visible light high-dimensional feature vector and the infrared high-dimensional feature vector are guided respectively to obtain guided infrared feature vectors and visible light feature vectors.

[0060] Specifically, the infrared high-dimensional feature vector and the visible light high-dimensional feature vector are feature spliced by the third formula and the fourth formula, and then input into a 5-layer UNet network to generate corresponding infrared attention weights and visible light attention weights The third formula is: The fourth formula is: Wherein, UNet is a convolutional neural network, (·, ·) is a splicing operation, other attention mechanisms can also achieve the same purpose, based on the fifth formula and the sixth formula, the guided infrared feature vector and the visible light feature vector The fifth formula is: The sixth formula is: Wherein, is a dot product operator, this guidance is the first mutual guidance, the objects of the guidance are the infrared high-dimensional feature vector and the visible light high-dimensional feature vector, and in the process of mutual guidance, the generated attention weights are helpful to identify and emphasize the key features in different modalities, so that the infrared high-dimensional feature vector and the visible light high-dimensional feature vector can be more effectively combined when the features are fused.

[0061] The refining of the visible light high-dimensional feature vector includes the following steps:

[0062] The guided infrared feature vector and the visible light high-dimensional feature vector are spliced and input into the ResNet network for learning to obtain a first residual, and a sum of the first residual and the visible light high-dimensional feature vector is taken as a refined first visible light feature vector.

[0063] Specifically, as shown in FIG. 5, which is a visible light feature refinement network architecture diagram, a visible light high-dimensional feature vector F vis and a guided infrared feature vector are spliced to obtain which is then input into the ResNet for learning to obtain a first residual, the first residual being a difference between the spliced feature vector and an original visible light high-dimensional feature vector F vis , and the first residual is added to the original visible light high-dimensional feature vector to obtain a first refined visible light high-dimensional feature vector, i.e., a first visible light feature vector The refined first visible light feature vector can be calculated based on a seventh formula: the seventh formula is

[0064] Refining the infrared high-dimensional feature vector includes the following steps:

[0065] The guided visible light feature vector and the infrared high-dimensional feature vector are spliced and input into the ResNet network for learning to obtain a second residual, and a sum of the second residual and the infrared high-dimensional feature vector is taken as a refined first infrared feature vector.

[0066] Specifically, as shown in FIG. 6, which is an infrared feature refinement network architecture diagram, an infrared high-dimensional feature vector F ir and a first guided visible light feature vector are spliced to obtain which is then input into the ResNet for learning to obtain a second residual, the second residual being a difference between the spliced feature vector and an original infrared high-dimensional feature vector F ir , and the second residual is added to the original infrared high-dimensional feature vector to obtain a first refined infrared high-dimensional feature vector, i.e., a first infrared feature vector The refined first infrared feature vector can be calculated based on an eighth formula: the eighth formula is

[0067] Obtaining the first refined infrared feature vector and the first refined visible light feature vector includes the following steps:

[0068] The first refined visible light feature vector and the first refined infrared feature vector are subjected to mutual guidance operation and refinement operation again to obtain a first refined infrared feature vector and a first refined visible light feature vector, and the first refined infrared feature vector and the first refined visible light feature vector are feature vectors after sufficient fusion of infrared features and visible light features.

[0069] Specifically, the embodiment is described by taking three mutual guidance and refinement operations as an example, and n=3 and 1≤m<3.

[0070] The first iteration of the above steps has been described and will not be repeated.

[0071] The second iteration process is as follows, and m=1 at this time. The first refined first visible light feature vector and the first refined first infrared feature vector are subjected to second mutual guidance to obtain corresponding infrared attention weights and visible light attention weights The attention weight generation process is as follows: the ninth formula is: The tenth formula is: Based on the eleventh formula: The second guided visible light feature vector is generated Based on the twelfth formula: The second guided infrared feature vector is generated Based on the thirteenth formula: The second refined first infrared feature vector is generated Based on the fourteenth formula: The second refined first visible light feature vector is generated

[0072] The third iteration process is as follows: at this time, m=2, the second refined first visible light feature vector and the second refined first infrared feature vector are subjected to third mutual guidance to obtain corresponding infrared attention weights and visible light attention weights The attention weight generation process is as follows: the ninth formula is: The tenth formula is: Based on the eleventh formula: The third guided visible light feature vector is generated Based on the twelfth formula: The third guided infrared feature vector is generated Based on the thirteenth formula: generate the first infrared feature vector after the third refining based on the fourteenth formula: generate the first visible light feature vector after the third refining

[0073] The first infrared feature vector obtained after 3 iterations is defined as the first refined infrared feature vector, and the first visible light feature vector obtained after 3 iterations is defined as the first refined visible light feature vector. Multiple iterations enable the model to learn and extract infrared features and visible light features more deeply. Each iteration can capture more subtle or different levels of feature information. Through multiple iterations of mutual guidance, the complementary information between visible light and infrared images can be better utilized, which can significantly improve the quality of the feature vector and enhance the performance of the model, and provide more reliable input for tasks such as super-resolution reconstruction.

[0074] Generating a visible light infrared image includes the following steps:

[0075] The first refined visible light feature vector is input into the ResNet network to extract the residual and obtain the third residual. The first infrared image after up-sampling operation is convolved with the third residual after convolution to obtain the visible light infrared image after feature integration.

[0076] Specifically, the first refined visible light feature vector after the third refining is input into the ResNet to extract the residual and obtain the third residual R vis based on the fifteenth formula for residual extraction of the first refined visible light feature vector, and the fifteenth formula is: The third residual reflects the difference between the first refined visible light feature vector and the first visible light vector, and the visible light infrared image S vis is generated based on the sixteenth formula, and the sixteenth formula is: S vis =Conv(up(I ir )+Conv(R vis )), wherein up is an up-sampling operation, and the first infrared image I ir is up-sampled. Up-sampling can improve the resolution of the first infrared image to match high-resolution images. Up-sampling operations can be implemented using methods such as nearest-neighbor interpolation, linear interpolation, bilinear interpolation, and cubic interpolation. The third residual is learned through a convolution layer, which captures the missing or adjusted parts of the visible light image. Additional convolution layers are applied to convert the visible light features to have characteristics similar to infrared images, and finally generate a visible light infrared image.

[0077] Single infrared super-resolution reconstruction includes the following steps:

[0078] The first refined infrared feature vector is input into the ResNet network for residual extraction to obtain a fourth residual, a convolution operation is performed on the fourth residual, a second convolution operation is performed on the first infrared image after the up-sampling operation and the fourth residual after the convolution, a single infrared super-resolution reconstruction result is obtained, and the single infrared super-resolution reconstruction result is defined as a first reconstruction result.

[0079] Specifically, similar to the step of generating the visible light type infrared image, the thirdly refined first refined infrared feature vector is input into the ResNet for residual extraction to obtain a fourth residual R ir , and the first refined infrared feature vector is subjected to residual extraction based on a seventeenth formula: The fourth residual captures the difference between the low-resolution infrared image and the first refined infrared feature vector, and the first infrared image I ir is subjected to an up-sampling operation and a first reconstruction result S ir is generated, the first reconstruction result is a single infrared super-resolution reconstruction result, and an eighteenth formula is: ir S ir = Conv(up(I ir )+Conv(R ir )), the first infrared image I mix is subjected to an up-sampling operation to improve the resolution of the image, the fourth residual is learned through a convolution layer, and a second convolution operation is performed on the residual after the convolution and the first infrared image after the up-sampling operation, the second convolution operation fuses the infrared features obtained by the two different ways to obtain the single infrared super-resolution reconstruction result.

[0080] The hybrid branch super-resolution reconstruction includes the following steps:

[0081] The first refined visible light feature vector and the first refined infrared feature vector are subjected to feature splicing and input into the UNet network to learn the corresponding infrared attention weight, the first refined visible light feature vector is guided again based on the infrared attention weight to obtain a guided first visible light feature vector, the guided first visible light feature vector and the first refined infrared feature vector are subjected to feature splicing and input into the ResNet network to learn a fifth residual required for the hybrid branch super-resolution reconstruction, a convolution operation is performed on the fifth residual, a second convolution operation is performed on the first infrared image after the up-sampling operation and the fifth residual after the convolution, a hybrid branch super-resolution reconstruction result is obtained, and the hybrid branch super-resolution reconstruction result is defined as a second reconstruction result.

[0082] Specifically, as shown in FIG. 7, it is a hybrid branch super-resolution reconstruction network architecture diagram, when n = 3, the thirdly refined first refined visible light feature vector and the first refined infrared feature vector The feature splicing is performed based on the nineteenth formula: The fourth guided infrared attention weight is generated based on the twentieth formula: to the first refined visible light feature vector The fourth guidance is performed to obtain the first visible light feature vector after the fourth guidance based on the twenty-first formula: to the first visible light feature vector after the fourth guidance and the first refined infrared feature vector The feature splicing is performed to generate the residual R required for constructing the hybrid branch super-resolution reconstruction mix , and based on the twenty-second formula: S mix = Conv (up (I ir ) + Conv (R mix )), the hybrid branch super-resolution reconstruction result is generated. This process ensures that the super-resolution reconstruction result does not contain redundant modal conflict information, and can better use the common visible light features to restore weak texture areas, and provide effective high-resolution guidance information for the super-resolution of low-resolution infrared images.

[0083] The multi-branch fusion super-resolution reconstruction result includes the following steps:

[0084] The visible light infrared image, the first reconstruction result and the second reconstruction result are assigned corresponding weights and weighted fused to obtain the multi-branch fusion super-resolution reconstruction result.

[0085] Specifically, the visible light infrared image, the first reconstruction result and the second reconstruction result are weighted fused based on the twenty-third formula to obtain the multi-branch fusion super-resolution reconstruction result out, and the twenty-third formula is: out = a1S ir +a2S vis +a3S mix , wherein a1>0, a2>0, a3>0, and a1+a2+a3=1, a1, a2 and a3 are respectively the first weight, the second weight and the third weight, the first weight is the single infrared super-resolution reconstruction result weight of the infrared branch, the second weight is the visible light infrared image weight of the visible light branch, and the third weight is the weight of the hybrid branch result.

[0086] Meanwhile, in the process of model training, the first infrared image and the first visible light image inputted can be respectively set to 0 at random to simulate the case of single modal information loss, and after model fitting, the three branch networks have the effect of modal loss robustness, and even if one modal information is missing, effective super-resolution reconstruction results can be obtained. For example, when only visible light infrared image results are needed, the visible light branch can be used, the infrared image is always set to 0, and only the visible light single modal image is inputted, and the three branch model in the application can also generate excellent high-resolution infrared image; when only single infrared super-resolution reconstruction results are needed, the infrared branch can be used, the visible light image is always set to 0, and only the infrared single modal image is inputted, and the three branch model in the application can generate results superior to single infrared super-resolution reconstruction.

[0087] Compared with the prior art, the application has at least the following advantages:

[0088] The application uses infrared and visible light two modal information to guide each other for super-resolution reconstruction, and can obtain visible light infrared results, visible light guided infrared super-resolution reconstruction results and single infrared super-resolution reconstruction results from one network structure, can also simply use one branch to obtain results superior to super-resolution reconstruction using only single modal information, and can weight and fuse three branch results to obtain various styles of super-resolution reconstruction results for user selection and use, the algorithm in the application has strong robustness, practicality and universality, and can be well applied in infrared and visible light dual light systems.

[0089] It should be understood that although each step in the flowchart of each embodiment of the application is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps has no strict sequence limitation, and these steps can be executed in other orders. Moreover, at least part of the steps in each embodiment can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0090] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a non-volatile computer readable storage medium, and the program can include the processes of the above-mentioned embodiment methods when executed. Any reference to memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0091] Any combination of the technical features of the above-mentioned embodiments can be combined, and for the sake of brevity, not all possible combinations of the technical features in the above-mentioned embodiments are described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0092] The above-mentioned embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as limiting the scope of the present application. It should be noted that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

[0093] The above-mentioned is only the preferred embodiment of the present application, and does not limit the present application, any modification, equivalent replacement, and improvement within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method of infrared and visible light inter-guided multi-super-resolution reconstruction results, characterized in that, The method comprises the following steps: S1: obtaining a first infrared image with low resolution and a first visible light image with high resolution, performing feature extraction on the first infrared image and the first visible light image to obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector respectively; S2: performing mutual guidance and refining on the infrared high-dimensional feature vector and the visible light high-dimensional feature vector to obtain a first refined infrared feature vector and a first refined visible light feature vector; S3: guiding the first refined visible light feature vector based on the first infrared image to generate a visible light infrared-like image, performing single-infrared super-resolution reconstruction on the first refined infrared feature vector to obtain a first reconstruction result, and performing mixed branch super-resolution reconstruction on the first refined visible light feature vector and the first refined infrared feature vector to obtain a second reconstruction result; S4: performing weighted fusion based on the visible light infrared-like image, the first reconstruction result and the second reconstruction result to obtain a multi-branch fusion super-resolution reconstruction result.

2. The method of claim 1, wherein, The feature extraction on the first infrared image and the first visible light image comprises the following steps: Performing two convolution operations on the first infrared image and the first visible light image respectively to obtain corresponding preliminary features, and inputting the convolutional preliminary features into a ResNet network to obtain an infrared high-dimensional feature vector and a visible light high-dimensional feature vector.

3. The method of claim 2, wherein, The mutual guidance on the infrared high-dimensional feature vector and the visible light high-dimensional feature vector comprises the following steps: Performing feature splicing on the infrared high-dimensional feature vector and the visible light high-dimensional feature vector and inputting the same into a UNet network to obtain corresponding infrared attention weights and visible light attention weights, and guiding the visible light high-dimensional feature vector and the infrared high-dimensional feature vector based on the infrared attention weights and the visible light attention weights to obtain guided infrared feature vectors and visible light feature vectors.

4. The method of claim 3, wherein, The refining on the visible light high-dimensional feature vector comprises the following steps: Performing feature splicing on the guided infrared feature vectors and the visible light high-dimensional feature vector and inputting the same into the ResNet network to obtain a first residual error, and taking the sum of the first residual error and the visible light high-dimensional feature vector as a refined first visible light feature vector.

5. The method of claim 4, wherein, The refining on the infrared high-dimensional feature vector comprises the following steps: Performing feature splicing on the guided visible light feature vectors and the infrared high-dimensional feature vector and inputting the same into the ResNet network to obtain a second residual error, and taking the sum of the second residual error and the infrared high-dimensional feature vector as a refined first infrared feature vector.

6. The method of claim 5, wherein, The first refined infrared feature vector and the first refined visible light feature vector comprise the following steps: Performing mutual guidance operation and refining operation again on the refined first visible light feature vector and the refined first infrared feature vector to obtain a first refined infrared feature vector and a first refined visible light feature vector, and the first refined infrared feature vector and the first refined visible light feature vector are feature vectors after sufficient fusion of infrared features and visible light features.

7. The method of claim 6, wherein, The generating visible light-like infrared image comprises the following steps: The first refined visible light feature vector is input into the ResNet network to obtain a third residual error, and the third residual error is subjected to convolution operation, and the first infrared image subjected to up-sampling operation is subjected to secondary convolution operation with the third residual error subjected to convolution operation, to obtain a visible light-like infrared image after feature integration.

8. The method of claim 6, wherein, The single infrared super-resolution reconstruction comprises the following steps: The first refined infrared feature vector is input into the ResNet network to obtain a fourth residual error, and the fourth residual error is subjected to convolution operation, and the first infrared image subjected to up-sampling operation is subjected to secondary convolution operation with the fourth residual error subjected to convolution operation, to obtain a single infrared super-resolution reconstruction result, and the single infrared super-resolution reconstruction result is defined as a first reconstruction result.

9. The method of claim 8, wherein, The mixed branch super-resolution reconstruction comprises the following steps: The first refined visible light feature vector and the first refined infrared feature vector are subjected to feature splicing and input into the UNet network to learn corresponding infrared attention weights, the first refined visible light feature vector is guided again based on the infrared attention weights to obtain a guided first visible light feature vector, the guided first visible light feature vector and the first refined infrared feature vector are subjected to feature splicing and input into the ResNet network to learn a fifth residual error required for mixed branch super-resolution reconstruction, the fifth residual error is subjected to convolution operation, and the first infrared image subjected to up-sampling operation is subjected to secondary convolution operation with the fifth residual error subjected to convolution operation, to obtain a mixed branch super-resolution reconstruction result, and the mixed branch super-resolution reconstruction result is defined as a second reconstruction result.

10. The method of claim 9, wherein, The obtaining a multi-branch fusion super-resolution reconstruction result comprises the following steps: The visible light-like infrared image, the first reconstruction result and the second reconstruction result are subjected to corresponding weight assignment and weighted fusion to obtain a multi-branch fusion super-resolution reconstruction result.

Images