Super-resolution reconstruction method and system for transmitting visible light characteristics to infrared light

Abstract

The invention belongs to the technical field of super-resolution reconstruction, and particularly relates to a super-resolution reconstruction method and system for transmitting visible light characteristics to infrared rays, and the method comprises the steps: S1, carrying out the registration processing of an infrared image and a visible light image through an image registration module, so as to generate a one-to-one correspondence relation between the infrared image and the visible light image in spatial positions; s2, converting the registered visible light image and infrared image into two high-dimensional feature vectors by using a feature extraction module; s3, performing feature fusion processing on the high-dimensional feature vector of the registered visible light image and the high-dimensional feature vector of the infrared image through a feature transmission module; and S4, using a super-resolution reconstruction module to extract the residual error of the high-dimensional mixed feature vector after the visible light features are transmitted, and performing super-resolution reconstruction processing based on the extracted residual error. According to the method, a weak texture region is well recovered in a super-division reconstruction result, and visible light texture feature details are increased.

Description

Technical Field

[0001] The present invention belongs to the technical field of super-resolution reconstruction, and in particular relates to a super-resolution reconstruction method and system for transmitting visible light features to infrared. Background Art

[0002] In the existing low-resolution infrared super-resolution methods, the mainstream method only uses low-resolution infrared images for super-resolution reconstruction. Its network structure is simple, and there is no need to consider the image registration problem of dual-light equipment. It can solve the jagged and blurring problems caused by the magnification of traditional image methods. However, these methods usually lose part of the image detail information during the reconstruction process, and cannot have a good super-resolution reconstruction result for the weak texture area. In addition, there is an image fusion function in the current infrared and visible light dual-light equipment, which fuses the infrared image with the high-frequency or low-frequency image of visible light, which can effectively increase the texture information of the infrared image. However, it is not a super-resolution reconstruction method, and there will still be jagged and blurring caused by magnification, and the visible light feature transmission is relatively stiff, which is completely inconsistent with the modal information of infrared, and will cause reading obstacles when interpreting infrared thermal information. Therefore, the present invention provides a super-resolution reconstruction method and system for transmitting visible light features to infrared. Summary of the invention

[0003] The present invention performs registration processing on infrared images and visible light images through an image registration module, uses a feature extraction module to convert the registered visible light image and infrared image into two high-dimensional feature vectors respectively, relies on a feature transfer module to perform feature fusion processing on the high-dimensional feature vector of the registered visible light image and the high-dimensional feature vector of the infrared image, extracts the residual of the high-dimensional mixed feature vector after the visible light feature transfer according to a super-resolution reconstruction module, and performs super-resolution reconstruction processing based on the extracted residual. The purpose of the present invention is to better restore weak texture areas and increase visible light texture feature details in super-resolution reconstruction results, while there is no obvious modal conflict problem.

[0004] In order to achieve the above-mentioned purpose of the invention, the present invention provides a super-resolution reconstruction method for transferring visible light features to infrared as described below, which mainly includes the following steps:

[0005] S1, performing registration processing on the infrared image and the visible light image to generate a one-to-one correspondence relationship between the infrared image and the visible light image in spatial position;

[0006] S2, respectively converting the registered visible light image and infrared image into two high-dimensional feature vectors, where the two high-dimensional feature vectors respectively represent the local and global features contained in the registered visible light image and infrared image;

[0007] S3, performing feature fusion processing on the high-dimensional feature vector of the registered visible light image and the high-dimensional feature vector of the infrared image to obtain a high-dimensional mixed feature vector after visible light feature transfer;

[0008] S4. Extract the residual of the high-dimensional mixed feature vector after the visible light feature is transferred, and perform super-resolution reconstruction processing based on the extracted residual.

[0009] As a preferred technical solution of the present invention, in S1, the image registration module uses the following calculation formula to perform registration processing on the visible light image:

[0010]

[0011] Among them, x and y represent the pixel positions of the input visible light image, They represent the pixel positions of the registered visible light image respectively, a, b, c, d are the rotation and scaling parameters used in the affine transformation, and e, f are the translation parameters used in the affine transformation.

[0012] As a preferred technical solution of the present invention, the dual-light device obtains corresponding alignment parameters by calibrating at different distances. During the calibration stage, a fixed target device is photographed at intervals of 1 m within a range of 0.5 m to 10 m, and corresponding targets are selected in the dual-light image based on the position of the fixed target device. The affine alignment parameters (a, b, c, d, e, f) corresponding to different distances are calculated, and the average of different rotation and scaling parameters a, b, c, and d are taken as the rotation and scaling parameters of the dual-light device. Interpolation fitting is performed to different distances for different translation parameters e and f.

[0013] As a preferred technical solution of the present invention, in S2, the feature extraction module uses the network structure ResNet to extract features, including the following steps:

[0014] S21, the feature extraction module uses the following calculation formula to calculate the registered visible light image s v and infrared images i Perform shallow feature extraction:

[0015]

[0016] in, To extract the shallow features of visible light and infrared light, Conv is a convolution operation;

[0017] S22, the feature extraction module inputs the extracted shallow features into the network structure ResNet to obtain deep high-dimensional feature vectors:

[0018]

[0019] in, They are respectively the high-dimensional feature vectors extracted from the deep visible light layer and the deep infrared layer. The network structure ResNet contains 3 groups of continuous residual modules.

[0020] As a preferred technical solution of the present invention, S3 comprises the following steps:

[0021] S31, using the following calculation formula to calculate the mixed feature vector after fusing the high-dimensional feature vector of the visible light deep layer and the high-dimensional feature vector of the infrared deep layer:

[0022]

[0023] Among them, F me , F ma They are the mixed feature vectors obtained by fusion through the average method and the mixed feature vectors obtained by fusion through the maximum method, max and mean are the maximum and minimum operators respectively;

[0024] S32, through the following calculation formula, for F me , F ma Perform convolution:

[0025]

[0026] in, For F me , F ma The feature vector obtained by convolution processing;

[0027] S33, for Perform concatenation and continue to perform two layers of convolution on the concatenated vector to obtain the output result:

[0028]

[0029] in, is the high-dimensional mixed feature vector after the visible light feature is transferred, and M is the function for concatenating two vectors.

[0030] As a preferred technical solution of the present invention, the feature extraction network used by the super-resolution reconstruction module to extract the residual of the high-dimensional mixed feature vector after the visible light feature is transferred includes ResNet, DenseNet, and Transformer.

[0031] As a preferred technical solution of the present invention, S4 comprises the following steps:

[0032] S41. Input the high-dimensional mixed feature vector after the visible light feature is transferred into ResNet for residual extraction:

[0033]

[0034] Among them, R0 is the extracted residual, and the network structure ResNet contains 3 groups of continuous residual modules;

[0035] S42, calculate the result of super-resolution reconstruction by the following calculation formula:

[0036]

[0037] in, is the final super-resolution reconstruction result, and up is the upsampling processing function, including the use of nearest neighbor interpolation, linear interpolation, bilinear interpolation, and cubic interpolation.

[0038] The present invention also provides a super-resolution reconstruction system for transmitting visible light features to infrared, which mainly includes the following modules:

[0039] An image registration module is used to perform registration processing on the infrared image and the visible light image before entering the feature transfer module to generate a one-to-one correspondence relationship between the infrared image and the visible light image in spatial position;

[0040] A feature extraction module is used to convert the registered visible light image and infrared image into two high-dimensional feature vectors, respectively, and the two high-dimensional feature vectors represent the local and global features contained in the registered visible light image and infrared image respectively;

[0041] A feature transfer module is used to perform feature fusion processing on the high-dimensional feature vector of the registered visible light image and the high-dimensional feature vector of the infrared image to obtain a high-dimensional mixed feature vector after visible light feature transfer;

[0042] The super-resolution reconstruction module is used to extract the residual of the high-dimensional mixed feature vector after the visible light feature is transferred, and perform super-resolution reconstruction based on the extracted residual.

[0043] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0044] In the present invention, firstly, before entering the feature transfer module, the image registration module performs registration processing on the infrared image and the visible light image to generate a one-to-one correspondence relationship between the infrared image and the visible light image in spatial position; secondly, the feature extraction module is used to convert the registered visible light image and the infrared image into two high-dimensional feature vectors, and the two high-dimensional feature vectors represent the local and global features contained in the registered visible light image and the infrared image respectively; thirdly, the feature transfer module performs feature fusion processing on the high-dimensional feature vector of the registered visible light image and the high-dimensional feature vector of the infrared image to obtain the high-dimensional mixed feature vector after the visible light feature transfer; finally, the super-resolution reconstruction module is used to extract the residual of the high-dimensional mixed feature vector after the visible light feature transfer, and the super-resolution reconstruction processing is performed based on the extracted residual. The present invention can better restore weak texture areas and increase visible light texture feature details in the super-resolution reconstruction results, and there is no obvious modal conflict problem at the same time. It has strong robustness, practicality, universality and other characteristics, and can be well applied in infrared and visible light dual-light fusion systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A first flow chart of a super-resolution reconstruction method for transferring visible light features to infrared according to the present invention;

[0046] Figure 2 A second flow chart of a super-resolution reconstruction method for transferring visible light features to infrared according to the present invention;

[0047] Figure 3 Schematic diagram of the network structure ResNet in the present invention;

[0048] Figure 4 This is a structural diagram of a super-resolution reconstruction system for transferring visible light features to infrared according to the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0050] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0051] The inventors found that in the existing low-resolution infrared super-resolution methods, the mainstream method only uses low-resolution infrared images for super-resolution reconstruction. Its network structure is simple, and there is no need to consider the image registration problem of the dual-light device. It can solve the jagged and blurring problems caused by the magnification of traditional image methods. However, these methods usually lose part of the image detail information during the reconstruction process, and cannot have a good super-resolution reconstruction result for areas with weak textures. In addition, the current infrared and visible light dual-light devices have an image fusion function, which fuses the infrared image with the high-frequency or low-frequency image of the visible light, which can increase the texture information of the infrared image. However, it is not a super-resolution reconstruction method, and there will still be jagged and blurring caused by magnification, and the visible light feature transmission is relatively stiff, which is completely inconsistent with the infrared modal information, and will cause interpretation obstacles when interpreting infrared thermal information.

[0052] In order to solve the above technical problems, the present invention provides Figure 1 A super-resolution reconstruction method for transferring visible light features to infrared is shown, which is mainly implemented by performing the following steps:

[0053] S1. The image registration module performs registration processing on the infrared image and the visible light image to generate a one-to-one correspondence relationship between the infrared image and the visible light image in spatial position.

[0054] Furthermore, in S1, the image registration module performs registration processing on the visible light image using the following calculation formula:

[0055]

[0056] Among them, x and y represent the pixel positions of the input visible light image, They represent the pixel positions of the registered visible light image respectively, a, b, c, d are the rotation and scaling parameters used in the affine transformation, and e, f are the translation parameters used in the affine transformation.

[0057] Furthermore, the bi-optical device obtains corresponding alignment parameters by calibrating at different distances. During the calibration stage, a fixed target device is photographed at intervals of 1 m in the range of 0.5 m to 10 m, and corresponding targets are selected in the bi-optical image based on the position of the fixed target device. The affine alignment parameters (a, b, c, d, e, f) corresponding to different distances are calculated, and the average of different rotation and scaling parameters a, b, c, d is taken as the rotation and scaling parameters of the bi-optical device, and interpolation fitting is performed to different distances for different translation parameters e and f.

[0058] Specifically, before entering the feature transfer module, the infrared image and the visible light image need to be registered to complete the one-to-one correspondence between the infrared and visible light images in space. Therefore, the order of placing the image registration module can be changed, as long as it is completed before entering the feature transfer module. The first case is to complete it at the beginning of the image, such as Figure 1 As shown in the second case, it is completed after feature extraction, such as Figure 2 shown.

[0059] The first case has been introduced in detail above. Here we will briefly introduce the second case. After feature extraction, high-dimensional infrared and visible light feature vectors are obtained respectively. For the registration between feature vectors, it is mainly performed through a deep learning network to achieve one-to-one alignment of variable feature vectors in spatial position. Since a variety of network structures in the existing technology can be used to complete the registration function, it will not be described in detail here.

[0060] S2. Use a feature extraction module to convert the registered visible light image and infrared image into two high-dimensional feature vectors, respectively. The two high-dimensional feature vectors represent the local and global features contained in the registered visible light image and infrared image, respectively.

[0061] Furthermore, in S2, the network structures used by the feature extraction module for feature extraction include ResNet, DenseNet, and Transformer.

[0062] Furthermore, in S2, the feature extraction module uses the network structure ResNet to extract features, including the following steps:

[0063] S21, the feature extraction module uses the following calculation formula to calculate the registered visible light image s v and infrared images i Perform shallow feature extraction:

[0064]

[0065] in, To extract the shallow features of visible light and infrared light, Conv is a convolution operation;

[0066] S22, the feature extraction module inputs the extracted shallow features into the network structure ResNet to obtain deep high-dimensional feature vectors:

[0067]

[0068] in, They are respectively the high-dimensional feature vectors extracted from the deep visible light layer and the deep infrared layer. The network structure ResNet contains 3 groups of continuous residual modules.

[0069] Specifically, the above describes two placement orders of the registration module. In the two placement orders, the other modules are the same. Here we continue to describe the order of placing the registration module at the beginning of the image. The function of the feature extraction module is to convert the input registered visible light image and infrared image into high-dimensional feature vectors respectively. The high-dimensional feature vector is a vector that combines feature data of multiple dimensions, which can fully express the local and global features contained in the image itself. The feature extraction module can use a variety of network structures for feature extraction, such as ResNet, DenseNet, Transformer and other methods. Here, the network structure ResNet is used to illustrate the feature extraction module. The network structure ResNet is as follows: Figure 3 As shown, it contains 3 groups of consecutive residual modules.

[0070] S3. Performing feature fusion processing on the high-dimensional feature vector of the registered visible light image and the high-dimensional feature vector of the infrared image through the feature transfer module to obtain a high-dimensional mixed feature vector after visible light feature transfer.

[0071] Furthermore, S3 includes the following steps:

[0072] S31, using the following calculation formula to calculate the mixed feature vector after fusing the high-dimensional feature vector of the visible light deep layer and the high-dimensional feature vector of the infrared deep layer:

[0073]

[0074] Among them, F me , F ma They are the mixed feature vectors obtained by fusion through the average method and the mixed feature vectors obtained by fusion through the maximum method, max and mean are the maximum and minimum operators respectively;

[0075] S32, through the following calculation formula, for F me , F ma Perform convolution:

[0076]

[0077] in, For F me , F ma The feature vector obtained by convolution processing;

[0078] S33, for Perform concatenation and continue to perform two layers of convolution on the concatenated vector to obtain the output result:

[0079]

[0080] in, is the high-dimensional mixed feature vector after the visible light feature is transferred, and M is the function for concatenating two vectors.

[0081] Specifically, through the feature extraction module, a high-dimensional feature vector that can express local and global features is extracted, and infrared and visible light feature fusion is performed here to effectively transfer the visible light features to the infrared super-resolution results, so that the extra visible light features appear in infrared mode, rather than the traditional superposition of high-frequency or low-frequency results. Here, the feature transfer method can be diversified, and can be achieved by splicing, maximizing and averaging the infrared and visible light feature vectors, as well as any combination of the three. The present invention only explains the combination of averaging and maximizing.

[0082] S4. Use the super-resolution reconstruction module to extract the residual of the high-dimensional mixed feature vector after the visible light feature is transferred, and perform super-resolution reconstruction processing based on the extracted residual.

[0083] Furthermore, the feature extraction networks used by the super-resolution reconstruction module to extract the residual of the high-dimensional mixed feature vector after the visible light feature transfer include ResNet, DenseNet, and Transformer.

[0084] Further, S4 includes the following steps:

[0085] S41. Input the high-dimensional mixed feature vector after the visible light feature is transferred into ResNet for residual extraction:

[0086]

[0087] Among them, R0 is the extracted residual, and the network structure ResNet contains 3 groups of continuous residual modules;

[0088] S42, calculate the result of super-resolution reconstruction by the following calculation formula:

[0089]

[0090] in, is the final super-resolution reconstruction result, and up is the upsampling processing function, including the use of nearest neighbor interpolation, linear interpolation, bilinear interpolation, and cubic interpolation.

[0091] Specifically, the high-dimensional mixed features after the visible light feature transfer are obtained through the above steps. Next, the residual required for constructing the super-resolution result extracted from the high-dimensional mixed feature vector is used for super-resolution reconstruction. The residual reflects the difference between the high-dimensional mixed feature vector and the image. The use of the residual for super-resolution reconstruction can more accurately restore the details and texture information of the image. The extraction of the residual can be achieved through a large number of feature extraction networks, such as ResNet, DenseNet, Transformer, etc. The present invention illustrates the use of the network structure ResNet to extract the residual.

[0092] References Figure 4 As shown, the present invention also provides a super-resolution reconstruction system for transferring visible light features to infrared, including an image registration module, a feature extraction module, a feature transfer module, and a super-resolution reconstruction module, which is used to implement a super-resolution reconstruction method for transferring visible light features to infrared as described above.

[0093] The functions of each module are as follows:

[0094] An image registration module is used to perform registration processing on the infrared image and the visible light image before entering the feature transfer module to generate a one-to-one correspondence relationship between the infrared image and the visible light image in spatial position;

[0095] A feature extraction module is used to convert the registered visible light image and infrared image into two high-dimensional feature vectors, respectively, and the two high-dimensional feature vectors represent the local and global features contained in the registered visible light image and infrared image respectively;

[0096] A feature transfer module is used to perform feature fusion processing on the high-dimensional feature vector of the registered visible light image and the high-dimensional feature vector of the infrared image to obtain a high-dimensional mixed feature vector after visible light feature transfer;

[0097] The super-resolution reconstruction module is used to extract the residual of the high-dimensional mixed feature vector after the visible light feature is transferred, and perform super-resolution reconstruction based on the extracted residual.

[0098] In summary, the technical problems existing in the prior art are as follows: 1) only using low-resolution infrared images for super-resolution reconstruction will result in loss of some image detail information in the reconstruction process, and will not provide good super-resolution reconstruction results for weak texture areas; 2) the image fusion function in the dual-light device of infrared and visible light will result in jagged and blurring phenomena caused by magnification, and the transmission of visible light features is relatively stiff, which is completely inconsistent with the modal information of infrared, and will cause interpretation obstacles when interpreting infrared thermal information. The present invention uses visible light information in the dual-light device of visible light and infrared to perform super-resolution reconstruction on low-resolution infrared images: in areas with common features, high-resolution visible light information is used to guide super-resolution reconstruction of infrared images; in areas where visible light has features but infrared has no features, visible light information is transmitted to infrared as infrared modal information for super-resolution reconstruction; in areas where infrared has features but visible light has no features, only infrared information itself is used for super-resolution reconstruction. Based on the above super-resolution reconstruction strategy, the present invention can better restore weak texture areas and increase visible light texture feature details in the super-resolution reconstruction results. At the same time, there is no obvious modal conflict problem. It also has strong robustness, practicality, universality and other characteristics, and can be better applied in infrared and visible light dual-light fusion systems.

[0099] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0100] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0101] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0103] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A super-resolution reconstruction method for transferring visible light features to infrared, characterized in that: The steps include: S1, performing registration processing on the infrared image and the visible light image to generate a one-to-one correspondence relationship between the infrared image and the visible light image in spatial position; S2, respectively converting the registered visible light image and infrared image into two high-dimensional feature vectors, where the two high-dimensional feature vectors respectively represent the local and global features contained in the registered visible light image and infrared image; S3, performing feature fusion processing on the high-dimensional feature vector of the registered visible light image and the high-dimensional feature vector of the infrared image to obtain a high-dimensional mixed feature vector after visible light feature transfer; S4. Extract the residual of the high-dimensional mixed feature vector after the visible light feature is transferred, and perform super-resolution reconstruction processing based on the extracted residual.

2. The method according to claim 1, characterized in that: In S1, the image registration module performs registration processing on the visible light image using the following calculation formula: Among them, x and y represent the pixel positions of the input visible light image, They represent the pixel positions of the registered visible light image respectively, a, b, c, d are the rotation and scaling parameters used in the affine transformation, and e, f are the translation parameters used in the affine transformation.

3. The method according to claim 2, characterized in that The bi-optical device obtains the corresponding registration parameters by calibrating at different distances. During the calibration stage, a fixed target device is photographed at intervals of 1m in the range of 0.5m to 10m. Based on the position of the fixed target device, one-to-one corresponding targets are selected in the bi-optical image, and the affine registration parameters (a, b, c, d, e, f) corresponding to different distances are calculated. The average of different rotation and scaling parameters a, b, c, and d are taken as the rotation and scaling parameters of the bi-optical device, and interpolation fitting is performed to different distances for different translation parameters e and f.

4. The method according to claim 1, characterized in that: In S2, the feature extraction module uses the network structure ResNet to extract features, including the following steps: S21, the feature extraction module uses the following calculation formula to calculate the registered visible light image s v and infrared images i Perform shallow feature extraction: in, To extract the shallow features of visible light and infrared light, Conv is a convolution operation; S22, the feature extraction module inputs the extracted shallow features into the network structure ResNet to obtain deep high-dimensional feature vectors: in, They are respectively the high-dimensional feature vectors extracted from the deep visible light layer and the deep infrared layer. The network structure ResNet contains 3 groups of continuous residual modules.

5. The method according to claim 1, characterized in that The S3 comprises the following steps: S31, using the following calculation formula to calculate the mixed feature vector after fusing the high-dimensional feature vector of the visible light deep layer and the high-dimensional feature vector of the infrared deep layer: Among them, F me , F ma They are the mixed feature vectors obtained by fusion through the average method and the mixed feature vectors obtained by fusion through the maximum method, max and mean are the maximum and minimum operators respectively; S32, through the following calculation formula, for F me , F ma Perform convolution: in, For F me , F ma The feature vector obtained by convolution processing; S33, for Perform concatenation and continue to perform two layers of convolution on the concatenated vector to obtain the output result: in, is the high-dimensional mixed feature vector after the visible light feature is transferred, and M is the function for concatenating two vectors.

6. The method according to claim 1, characterized in that The feature extraction networks used by the super-resolution reconstruction module to extract the residual high-dimensional mixed feature vector after the visible light feature is transferred include ResNet, DenseNet, and Transformer.

7. The method according to claim 6, characterized in that The S4 comprises the following steps: S41. Input the high-dimensional mixed feature vector after the visible light feature is transferred into ResNet for residual extraction: Among them, R0 is the extracted residual, and the network structure ResNet contains 3 groups of continuous residual modules; S42, calculate the result of super-resolution reconstruction by the following calculation formula: in, is the final super-resolution reconstruction result, and up is the upsampling processing function, including the use of nearest neighbor interpolation, linear interpolation, bilinear interpolation, and cubic interpolation.

8. A super-resolution reconstruction system for transferring visible light features to infrared, used to implement the method according to any one of claims 1 to 7, characterized in that: Includes the following modules: An image registration module is used to perform registration processing on the infrared image and the visible light image before entering the feature transfer module to generate a one-to-one correspondence relationship between the infrared image and the visible light image in spatial position; A feature extraction module is used to convert the registered visible light image and infrared image into two high-dimensional feature vectors, respectively, and the two high-dimensional feature vectors represent the local and global features contained in the registered visible light image and infrared image respectively; A feature transfer module is used to perform feature fusion processing on the high-dimensional feature vector of the registered visible light image and the high-dimensional feature vector of the infrared image to obtain a high-dimensional mixed feature vector after visible light feature transfer; The super-resolution reconstruction module is used to extract the residual of the high-dimensional mixed feature vector after the visible light feature is transferred, and perform super-resolution reconstruction based on the extracted residual.

Images