A cross-modal super-resolution reconstruction method, terminal device and readable storage medium

Abstract

The present invention belongs to the field of infrared image super-resolution technology in computer vision, and is particularly a cross-modal super-resolution reconstruction method, comprising the following steps: Step 1, data set preprocessing: preparing infrared and visible light image pairs of the same scene, converting the visible light image into a grayscale image, and enhancing the infrared data set using DDE to improve the contrast; then performing data augmentation on all images, and finally coarsely aligning the three images to facilitate the training of subsequent feature texture extraction tasks. The present invention constructs a method for cross-modal supplementation of infrared image texture information by a contrastive learning-based method. This method does not rely on artificial design, and introduces a grayscale image as a connection point between infrared and visible light images, effectively reducing the influence of the mismatch between visible light and infrared image textures caused by color factors.

Description

Technical Field

[0001] The present invention relates to the field of infrared image super-resolution technology in computer vision, and specifically to a cross-modal super-resolution reconstruction method, terminal equipment, and readable storage medium. Background Art

[0002] Cross-modal super-resolution reconstruction uses additional visible light and grayscale images to supplement the high-frequency details missing from the input low-resolution infrared image. Visible light and grayscale images often have similar content to infrared images, providing high-resolution images with realistic details. In recent years, most related research has employed convolutional neural network frameworks. However, because infrared images lose over 93% of their high-frequency information in 4× super-resolution tasks, relying solely on the high-frequency information in the image itself is difficult to achieve superior results.

[0003] The Chinese patent publication number is "CN116049469A," titled "Multi-matching Search and Super-resolution Reconstruction Method Based on Reference Images." This method uses an encoding module to extract features from a low-resolution image, a reference image, and a bicubic interpolation-downsampled reference image. A multi-matching search and extraction module then performs a match search on the bicubic interpolation-downsampled reference image for multiple image blocks similar to the low-resolution image. The corresponding image blocks are extracted from the high-resolution reference image feature map and weighted summed to obtain a new reference image. Finally, a high-resolution image is reconstructed by multi-scale fusion of the low-resolution image and the new reference image. This method has poor texture detail recovery capabilities, and when applied to infrared images, color information can lead to the destruction of the original infrared image information. Therefore, we propose a cross-modal super-resolution reconstruction method, terminal device, and readable storage medium to address these issues. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a cross-modal super-resolution reconstruction method, terminal device and readable storage medium, which uses visible light and grayscale images matched with infrared image features as reference images for super-resolution reconstruction of the deep network, inputs the visible light, grayscale reference images and low-resolution infrared images into the network, and finally obtains a high-resolution infrared image, solving the problems raised in the above-mentioned background technology.

[0006] (2) Technical solution

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] A cross-modal super-resolution reconstruction method comprises the following steps:

[0009] Step 1, dataset preprocessing: Prepare infrared and visible light image pairs of the same scene, convert the visible light images into grayscale images, and enhance the infrared dataset using DDE to improve contrast; then perform data augmentation on all images, and finally perform coarse registration on the three images to facilitate training of subsequent feature texture extraction tasks;

[0010] Step 2: Construct a cross-modal super-resolution reconstruction network: The cross-modal super-resolution reconstruction network consists of an encoding module, a dual-branch feature extraction module, and a decoding module; the dual-branch feature extraction module includes a visible light feature extraction module and a grayscale feature extraction module;

[0011] Step 3: Prepare a dataset for dual-branch feature weight pre-training: This step is basically the same as step 1, but the dataset images need to be cut into image block pairs of uniform specifications;

[0012] Step 4: Pre-training of dual-branch feature weights: The pre-training of dual-branch feature weights consists of three stages: coarse feature matching, fine feature matching using dual contrast learning, and feature import. The texture similarity weights between the infrared image and the reference image are imported into the cross-modal super-resolution reconstruction network through contrast learning.

[0013] Step 5: Train the cross-modal super-resolution reconstruction network parameters: The weights of the two-branch feature extraction module are normalized and converted into a ranking loss to obtain the weight ratio of the two branch features. Finally, texture information is added to the infrared image through weighted loss. The super-resolution uses L1 loss, perceptual loss, and adversarial loss to optimize the network parameters.

[0014] Step 6, save the model: solidify the final model parameters. When infrared image super-resolution reconstruction is required, directly input the image into the network to obtain the final super-resolution image.

[0015] Furthermore, the data set in step 1 uses the FLIR ADAS data set; some image pairs are selected, the paired feature points are manually marked, and after calculating the corrected coordinate map, all the data set feature points are automatically paired to obtain a coarse registration image.

[0016] Furthermore, the data augmentation method in step 1 includes image flipping, translation, rotation, and scaling, and the center of all transformations is the image center; similarly, the data augmentation method in step 3 adds contrast transformation and random Gaussian blur on the basis of step 1, and the center of all transformations is still the image center.

[0017] Furthermore, in step 2, the encoding module is used to extract features of the high-resolution visible light downsampled image, the grayscale downsampled image converted from the visible light image, and the low-resolution infrared image; the dual-branch feature extraction module extracts similar features between the infrared image and the visible light image and similar features between the infrared image and the grayscale image, respectively, and finally introduces the two sets of texture features and weights into the low-resolution infrared features; finally, the decoding module reconstructs the high-resolution infrared image from the infrared image with supplemented texture information.

[0018] Furthermore, in the rough feature matching stage in step 4, the cosine similarity is calculated for each infrared image block and each visible light and grayscale image block, so that the image blocks are preliminarily paired; in the double contrast learning feature fine matching stage, the textures of the infrared image and the reference image that are similar are used as positive samples, and the dissimilar blocks are used as negative samples for training through contrast loss; finally, the texture information and training weights are saved, and in the feature import stage, the two training weights are respectively imported into the corresponding branches of the dual-branch feature extraction module.

[0019] Furthermore, the pre-training of the dual-branch feature weights in step 4 uses contrastive learning, proposes a contrastive loss, and retains the training weights to be imported into the corresponding feature extraction module.

[0020] Furthermore, in step 5, the dual-branch feature extraction module uses a new sorting loss, the sum of the weights of the two branches is 1, and the range of each branch is between 0 and 1.

[0021] Furthermore, the L1 loss, perceptual loss, and adversarial loss are used in the super-resolution in step 5.

[0022] A cross-modal super-resolution reconstruction terminal device comprises a processor, a memory, a communication port, and a computer program stored in the memory and executed by the processor.

[0023] A computer-readable storage medium for cross-modal super-resolution reconstruction, wherein the computer-readable medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled.

[0024] (3) Beneficial effects

[0025] Compared with the prior art, the present invention provides a cross-modal super-resolution reconstruction method, terminal device, and readable storage medium, which have the following beneficial effects:

[0026] The present invention constructs a cross-modal method for supplementing infrared image texture information based on contrastive learning. This method does not rely on manual design and introduces grayscale images as connection points between infrared and visible light images, effectively reducing the impact of texture mismatch between visible light and infrared images caused by color factors.

[0027] The present invention uses very few manually annotated feature points in the training of the cross-modal feature extraction module, which can be automatically applied to the global image, reducing the dependence on manually annotated datasets.

[0028] The present invention reduces the computational parameters of the network by proposing a method of pre-training feature extraction, and can perform secondary training on the pre-training weights at any time to increase its feature extraction capability and robustness.

[0029] The present invention proposes a novel weighted loss function for dual-branch contrast learning and uses learnable self-supervisory branch weights to enhance the ability of the reference image to supplement the texture information of the infrared image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of a cross-modal super-resolution reconstruction method of the present invention;

[0031] Figure 2 This is a network structure diagram of a cross-modal super-resolution reconstruction method of the present invention;

[0032] Figure 3 This is a network structure diagram of the feature extraction module of the present invention;

[0033] Figure 4 This is the structure diagram of the self-attention module of the present invention;

[0034] Figure 5 This is the shallow feature extraction network structure diagram of the present invention;

[0035] Figure 6 This is a schematic diagram of the dual-branch feature weight pre-training of the present invention;

[0036] Figure 7 This is a diagram of the terminal device of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] Example

[0039] like Figure 1 、 2 As shown in FIG6 , a cross-modal super-resolution reconstruction method proposed in one embodiment of the present invention is:

[0040] Step 1, data set preprocessing: The data set used for feature fusion super-resolution reconstruction network uses FLIR ADAS data set and some self-collected infrared and visible light image pairs. The data set includes 8862 infrared and visible light image pairs with a resolution of 512×640. The visible light images are copied and the colors are removed and converted into grayscale data sets. 300 image pairs are selected as training data sets, and the infrared images are enhanced using DDE. 20 image pairs are selected and the control point in the MATLAB toolbox is used. The select tool manually selects paired feature points in these infrared, visible light, and grayscale images, merges the feature point data of 20 pairs of images, and then uses the infrared image as a reference to calculate the correction coordinate map using the nonlinear transformation method polynomial. The mapping matrix is ​​used to coarsely align all image pairs with the correction coordinate map. The image is cut into 256×256 image blocks, and a total of 64,000 image blocks are obtained. These image blocks are subjected to data augmentation, including image flipping, translation, rotation, and scaling. The center of all transformations is the image center, and the probability of each transformation is 0.5. Random transformations are performed again during each training to improve the generalization ability of the training.

[0041] Step 2: Construct a cross-modal super-resolution reconstruction network: Figure 2 As shown in FIG, the low-resolution infrared image is upsampled by bicubic interpolation to the expected super-resolution ratio. The visible light and grayscale image blocks are also downsampled to the low-resolution infrared image size and then upsampled by bicubic interpolation to the expected super-resolution ratio. The upsampled infrared image block, the visible light downsampled and upsampled image block, and the high-resolution visible light image block are input into the visible light feature extraction module. The feature extraction module is shown in FIG. Figure 3 As shown, it consists of a standard convolution layer 1 and a standard convolution layer 2 with a residual structure, a standard convolution layer 3 with a residual structure, multiple self-attention modules and a standard convolution layer 4. The self-attention module consists of a batch normalization layer 1 with a residual structure, a self-attention layer, a batch normalization layer 2 with a residual structure, and a multi-head perception mechanism. The grayscale feature extraction module structure is the same as the visible light feature extraction module. The fusion ratios α and β of the grayscale feature extraction module and the visible light feature extraction module are set to 0.5 respectively. The shallow feature extraction module is as shown in FIG. Figure 4 As shown in the figure, it is composed of standard convolution five, standard convolution six, and standard convolution seven. The kernel size of the standard convolution here is 3×3, the step size is 1, the padding is 1, and there is no bias parameter; the feature fusion module recommends selecting the TTSR Transformer network structure to more accurately fuse texture features; both the grayscale feature extraction module and the visible light feature extraction module need to be pre-trained.

[0042] Step 3: Prepare a pre-training dataset for dual-branch feature weights: Use the FLIRADAS dataset for pre-training dual-branch feature weights. Select 4,000 different image pairs from step 1 and perform the same operation as step 1. Cut the images into 256×256 image blocks, and obtain a total of 48,000 image blocks for subsequent network training.

[0043] Step 4: Construct a dual-branch feature weight pre-training network: calculate the cosine similarity between each low-resolution infrared image block and the reference image block to find the reference image block that best matches it and complete the rough matching of features; intercept multiple dx×dy blocks on the infrared image block, and intercept the reference image blocks at the corresponding positions in the similar visible light and grayscale image blocks to reduce the computational complexity of the subsequent double contrast learning fine matching task; double contrast learning feature fine matching is as follows Figure 4 As shown in the figure, in the fine matching stage, the 3×3 texture details in the infrared low-resolution image block and the corresponding multiple reference image blocks are matched and compared and learned respectively to calculate their correlation. The textures of the infrared image and the reference image that are similar are used as positive samples, and the dissimilar blocks are used as negative samples for training. Finally, the texture information and training weights are saved, and the contrast loss is shown as follows:

[0044]

[0045] Where S is the indicator function, O r , O v , O g Represent the feature vectors obtained from infrared, visible light and grayscale datasets, respectively, sim (x1, x2) is a similarity measurement function. The present invention uses cosine similarity measurement, which can be expressed as:

[0046]

[0047] By reverse gradient conduction, the loss function of the network is minimized to obtain a better weight parameter ω for infrared and visible light r，v and the weight parameter ω of infrared grayscale r，g ; Finally, in the feature import stage, the pre-trained texture information and training weights are imported into the corresponding branches in the dual-branch feature extraction module.

[0048] Step 5: Train the cross-modal network parameters: set the number of training times to 200, the number of images input to the network each time to 64-512, the upper limit of the number of images input to the network each time is mainly determined by the storage capacity of the computer display memory, and the learning rate of the training process is set to 0.1; the overall loss threshold of super-resolution is set to 0.01, and after three consecutive rounds of operations less than this threshold, the network training is considered to be basically completed; the infrared and visible light datasets are combined (B, H and W are the batch size, height and width of the input image set, respectively, and N is the total number of batches) input into the visible light feature extraction network; similarly, the infrared and grayscale datasets are The input is fed into the grayscale feature extraction network. The difference is that the grayscale image needs to be input three times as much to ensure the consistency of the three color channels of the infrared image. The sorting loss of the dual-branch feature extraction module can be expressed as follows:

[0049]

[0050] α=0.5+L por

[0051] β=0.5-L por

[0052] α and β are the weighted proportions of the infrared visible light branch and the infrared grayscale branch respectively;

[0053] L w =αω r，v +βω r，g

[0054] L w A weighted loss is used for dual-branch feature extraction. The ratio of the texture details of the low-resolution infrared image to the visible light and grayscale reference images is adjusted according to the weighted loss. In the feature fusion and feature mapping parts, L1 loss, perceptual loss, adversarial loss, and overall loss are used as follows:

[0055] L reC =||I HR -I SR ||1

[0056]

[0057]

[0058] L(θ)=λ rec L rec +λ per L per +λ adv L adv

[0059] The weight coefficient λ of the loss function rec ,λ per ,λ adv Set to 1, 1, 0.005 respectively, where I SR , I HR are low-resolution images and high-resolution images in the dataset respectively; by minimizing the loss function generated by the network through reverse gradient conduction, better weight parameters can be obtained.

[0060] Step 6, Save the model: After the network training is completed, all parameters in the network need to be saved. Then, input a low-resolution infrared image into the network to obtain a super-resolved infrared image. Inputting an image that is too large will cause the graphics card memory limit to be exceeded, so it is recommended that the input image be between 64×64 and 1024×1024.

[0061] Among them, the implementation of convolution, activation function, splicing operation, batch normalization, multiple perceptrons, self-attention layer and residual network are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0062] This paper constructs a cross-modal super-resolution reconstruction method that enhances the texture information of infrared images using a reference image and then performs super-resolution reconstruction. This method introduces a grayscale image as a link between the visible light and infrared images, reducing the influence of color information on texture fusion. Under the same conditions, the feasibility and superiority of this method were further verified by calculating image correlation indices obtained using existing methods. A comparison of the correlation indices between the existing technology and the proposed method is shown in Table 1:

[0063] Table 1 Comparison of relevant indicators of the prior art and the method proposed in this invention

[0064]

[0065] Experimental data show that the method proposed in this invention is superior to other super-resolution reconstruction methods and has higher peak signal-to-noise ratio and structural similarity than existing methods.

[0066] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0067] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. Figure 7 As shown, it should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A cross-modal super-resolution reconstruction method, characterized in that: The following steps are included: Step 1, data set preprocessing: prepare infrared and visible light image pairs of the same scene, convert the visible light image into a grayscale image, and use DDE to enhance the infrared data set to improve the contrast; then perform data augmentation on all images, and finally perform rough registration on the three images to facilitate the training of subsequent feature texture extraction tasks; Step 2: construct a cross-modal super-resolution reconstruction network: the cross-modal super-resolution reconstruction network consists of an encoding module, a dual-branch feature extraction module, and a decoding module; the dual-branch feature extraction module includes a visible light feature extraction module and a grayscale feature extraction module; In step 2, the encoding module is used to extract features of the high-resolution visible light downsampled image, the grayscale downsampled image converted from the visible light image, and the low-resolution infrared image; the dual-branch feature extraction module extracts similar features between the infrared image and the visible light image and similar features between the infrared image and the grayscale image, and finally introduces the two sets of texture features and weights into the low-resolution infrared features; Finally, the infrared image with supplementary texture information is reconstructed into a high-resolution infrared image through a decoding module; Step 3, prepare a data set for dual-branch feature weight pre-training: This step is basically the same as step 1, but the data set images need to be cut into image block pairs of uniform specifications; Step 4, pre-training of dual-branch feature weights: The pre-training of dual-branch feature weights includes three stages, namely, feature coarse matching stage, double contrast learning feature fine matching stage, and feature import stage; the texture similarity weights of the infrared image and the reference image are imported into the cross-modal super-resolution reconstruction network through contrast learning; In the feature rough matching stage in step 4, the cosine similarity is calculated for each infrared image block and each visible light and grayscale image block, so that the image blocks are preliminarily matched; in the double contrast learning feature fine matching stage, the textures similar to the infrared image and the reference image are used as positive samples through contrast loss, and the dissimilar blocks are used as negative samples for training; finally, the texture information and training weights are saved, and the two training weights are respectively imported into the corresponding branches of the dual-branch feature extraction module in the feature import stage; Step 5: Train the parameters of the cross-modal super-resolution reconstruction network: The weights of the dual-branch feature extraction module are normalized and converted into a sorting loss to obtain the weight ratio of the two branch features. Finally, the texture information is added to the infrared image through the weighted loss. The super-resolution uses L1 loss, perceptual loss, and adversarial loss to optimize the network parameters. Step 6, save the model: solidify the final model parameters, and when the infrared image super-resolution reconstruction operation is needed, directly input the image into the network to obtain the final super-resolution image.

2. The cross-modal super-resolution reconstruction method according to claim 1, characterized in that: In step 1, the data set uses the FLIRADAS data set; some image pairs are selected, the paired feature points are manually marked, and after the correction coordinate map is calculated, the feature points of all data sets are automatically paired to obtain a coarse registration image.

3. The cross-modal super-resolution reconstruction method according to claim 1, characterized in that: The data augmentation method in step 1 includes image flipping, translation, rotation, and scaling, and the center of all transformations is the center of the image; similarly, the data augmentation method in step 3 adds contrast transformation and random Gaussian blur on the basis of step 1, and the center of all transformations is still the center of the image.

4. The cross-modal super-resolution reconstruction method according to claim 1, characterized in that: The pre-training of the dual-branch feature weights in step 4 uses contrastive learning, proposes a contrastive loss, and retains the training weights and imports them into the corresponding feature extraction module.

5. The cross-modal super-resolution reconstruction method according to claim 1, characterized in that: In step 5, the dual-branch feature extraction module uses a new sorting loss, the sum of the weights of the two branches is 1, and the range of each branch is between 0 and 1.

6. A cross-modal super-resolution reconstruction terminal device, characterized in that: The invention comprises a processor, a memory, a communication port and a computer program stored in the memory and executed by the processor, wherein when the computer program is executed, the cross-modal super-resolution reconstruction method as claimed in any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium for cross-modal super-resolution reconstruction, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the cross-modal super-resolution reconstruction method according to any one of claims 1 to 5.

Images