Method and system for matching infrared image and visible light image

Abstract

The present application relates to a kind of infrared image and visible light image fusion matching method and system, the method includes the following steps: judging whether visible light image is obtained under strong light illumination or under weak light illumination acquisition;If visible light image is obtained under strong light illumination, then visible light image and infrared image are directly fused, and fusion image is obtained;If visible light image is obtained under weak light illumination, then visible light image generates infrared image, then the generated infrared image and the infrared image collected are fused under infrared image, and weighted infrared image is generated, and weighted infrared image is spliced with visible light image again, and the final fusion image is generated.The present application can make up the problem of brightness drop caused by visible light image and infrared image fusion under normal illumination, and can also realize the problem of insufficient details of fusion image under dark illumination.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image fusion, and particularly relates to an infrared image and visible light image fusion matching method and system. BACKGROUND

[0002] With the increasing demand for high-quality monitoring images, whether in remote sensing analysis or in the daily monitoring process, a high-quality infrared spectrum and visible light fusion system and the corresponding method research have important practical value. In particular, the use of infrared thermal radiation information and visible light texture details to achieve a comprehensive image with rich texture details and thermal radiation information has become an important content of related field research.

[0003] Traditional infrared and visible light fusion mostly uses scale transformation and component analysis of two types of images to perform time domain image fusion, and modern image fusion mostly uses weighted average or maximum selection strategy of convolutional neural network to realize organic fusion of two types of images.

[0004] However, in the above-mentioned manner, scale transformation can accurately represent the spatial structure of an image, but there are many problems such as noise, artifacts and detail loss that are difficult to effectively solve. The algorithm based on convolutional neural network lacks significant information of a target region of interest, resulting in problems such as unsatisfactory fusion effect. SUMMARY

[0005] The present application aims to improve the problem of unsatisfactory fusion effect in the prior art, and provides an infrared image and visible light image fusion matching method and system.

[0006] In order to achieve the above-mentioned application purpose, the embodiments of the present application provide the following technical solutions:

[0007] On one hand, the embodiments of the present application provide an infrared image and visible light image fusion matching method, the infrared image and the visible light image are collected in the same environment, and the method comprises the following steps:

[0008] It is judged whether the visible light image is collected under strong light intensity or weak light intensity;

[0009] If the visible light image is collected under strong light intensity, the visible light image is directly fused with the collected infrared image to obtain a fusion image;

[0010] If the visible light image is collected under weak light intensity, the visible light image Iv is first generated into an infrared image, then the generated infrared image is fused with the collected infrared image to generate a weighted infrared image, and the weighted infrared image is spliced with the visible light image to generate a final fusion image.

[0011] In another aspect, the embodiment of the present application also provides a system for infrared image and visible light image fusion matching, wherein the infrared image and the visible light image are collected in the same environment, and the system comprises:

[0012] An illumination judgment module is configured to judge whether the visible light image is collected under strong light illumination or weak light illumination.

[0013] A fusion module comprises an integrated fusion model, wherein the integrated fusion model is configured to directly fuse the visible light image and the infrared image to obtain a fusion image when the visible light image is collected under strong light illumination, and generate an infrared image from the visible light image when the visible light image is collected under weak light illumination, then perform infrared image fusion between the generated infrared image and the collected infrared image to generate a weighted infrared image, and finally perform splicing between the weighted infrared image and the visible light image to generate a final fusion image.

[0014] Compared with the prior art, the method of the present application has the following technical advantages:

[0015] The present application can compensate for the brightness drop caused by the fusion of the visible light image and the infrared image under normal illumination conditions, and can also solve the problem of insufficient details of the fusion image under dark illumination conditions.

[0016] The other advantages of the present application will be described in the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0018] Figure 1 A flow chart of a method for infrared image and visible light image fusion matching in an embodiment.

[0019] Figure 2 A structure diagram of an integrated fusion model in an embodiment.

[0020] Figure 3 A structure diagram of a generator G1 in an embodiment.

[0021] Figure 4 A structure diagram of a generator G2 in an embodiment.

[0022] Figure 5 A structure diagram of a discriminator in an embodiment.

[0023] Figure 6 Fig. 6 is a comparison chart of the effects of several network models in the experimental examples.

[0024] Figure 7a 、 Figure 7b Fig. 6 is a comparison chart of the effects of several network models in the experimental examples. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be apparently and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0026] Please refer to Figure 1 The present embodiment provides a method for matching infrared image and visible light image fusion, wherein the infrared image and the visible light image are different resolution images obtained in the same scene. The method comprises the following steps:

[0027] S10, judging whether the visible light image is collected in strong light intensity or in weak light intensity, if the visible light image is collected in strong light intensity, entering step S20, otherwise entering step S30.

[0028] It should be noted here that the strong light intensity (bright environment) and the weak light intensity (dark environment) are only two relative concepts, rather than absolute concepts, which represent images collected in two different light environments respectively.

[0029] In this embodiment, whether the visible light image is collected under strong light illumination or weak light illumination is determined mainly by using the brightness statistics of the visible light image Iv. Generally, the image collected by the camera is RGB three colors, and the brightness statistics of each pixel is calculated by V(i,j)=max(R,G,B), i,j is the coordinate position, V(i,j) is the brightness of the pixel point with the coordinate position (i,j). Here, the maximum value method is used to calculate the brightness of each pixel, which can ensure that each pixel point is the maximum value of R, G, and B, and ensure that the details of all colors are guaranteed. The brightness of the whole visible light image is obtained by weighted average of all pixel points. For example, as an example of an implementable manner, assuming that a visible light image has four pixel points, and the R, G, and B components are (255, 0, 0), (255, 0, 0), (0, 255, 0), and (0, 0, 255), then the brightness of the whole visible light image is (255+255+255+255) / 4=255. Set the judgment threshold value as (255+1) / 2=128, then compare the brightness 255 of the whole visible light image with the judgment threshold value 128, and the brightness of the visible light image is greater than the judgment threshold value, so it is judged that the visible light image is collected under strong light illumination.

[0030] S20, directly fuse the visible light image and the infrared image to obtain a fused image If.

[0031] Here, directly fusing the visible light image and the infrared image means combining the edges of the infrared image and the features of the visible light image together, enhancing the image details, and reconstructing the model. In this embodiment, the multi-scale features of the deep feature extraction network are first used to obtain the feature maps of the infrared image Ir and the visible light image Iv, and then the feature up-sampling reconstruction process is used to fuse the two feature maps.

[0032] S30, using the multi-scale feature extraction and reconstruction architecture, generating the infrared image Iv’ from the visible light image Iv, and then performing infrared image fusion between the infrared image Iv’ and the infrared image Ir to generate a weighted infrared image Iv’r, and then performing splicing between the weighted infrared image Iv’r and the visible light image Iv to generate a final fused image If.

[0033] In this step, in order to facilitate description, the originally obtained infrared image is denoted as Ir, the originally obtained visible light image is denoted as Iv, the visible light image Iv is denoted as Iv’ after using the multi-scale feature extraction and reconstruction architecture, the weighted infrared image Iv’r is denoted as Iv’r after performing weighted fusion between the infrared image Iv’ and the infrared image Ir, and the finally generated fused image is denoted as If.

[0034] As Figure 2As shown, the integrated fusion model is used, for the visible light image Iv obtained under strong light intensity, the visible light image Iv is directly fused with the infrared image Ir through the generator G2 to generate the fused image If; for the visible light image Iv obtained under weak light intensity, the visible light image Iv is first used to generate the infrared image Iv' by the generator G1, and then the infrared image Iv' is fused with the infrared image Ir under the infrared image to generate the weighted infrared image Iv'r, and finally the weighted infrared image Iv'r is spliced with the visible light image Iv again through the generator G2 to generate the final fused image If. In the fusion process, in order to guarantee the quality of the fused image, the discriminator D1 and the discriminator D2 are also introduced, the discriminator D1 performs quality discrimination on the generated infrared image Iv', and the discriminator D2 performs quality discrimination on the generated fused image If.

[0035] In this embodiment, the structure of the generator G1 is as shown in Figure 3 The generation process of the infrared image Iv' is as follows: the generator G1 is composed of an encoder and a decoder, first, the multi-scale features of the deep feature extraction network are used to obtain the feature maps of the infrared image Ir and the visible light image Iv, and then the feature up-sampling reconstruction process is used to fuse the images. The entire network adopts a completely symmetrical structure. The encoder contains 7 convolutional blocks for down-sampling, except for the first convolutional block, the last 6 convolutional blocks contain a convolutional layer, a LeakyRelu activation function and a batch normalization (BatchNorm), and the convolutional layer uses a 4x4 convolutional kernel with a step q of 2. The decoder also contains 7 convolutional blocks, the first 6 convolutional blocks contain a convolutional layer, a Relu activation function and a batch normalization (BatchNorm), and the last convolutional block contains a convolutional layer and a tanh activation function. In addition, the stack operation is used in the jump connection part, and the same color represents that the width and height of the feature maps are the same.

[0036] First, a large number of visible light images and corresponding (same environment) infrared images are used to train the generator G1, and when the image is stable, the corresponding infrared image can be obtained by inputting the visible light image.

[0037] There are many image fusion methods, such as maximum and minimum value method, pixel weighted average method, scale-based fusion method, and wavelet transform image fusion method. In this embodiment, for the generated infrared image Iv' and the collected infrared image Ir, the pixel weighted fusion method is mainly used to generate the weighted infrared image Iv'r, that is, according to the gray information of the two images, the corresponding weight is set, and the gray values of the two images are respectively assigned to the same pixel point, and the gray value of the fused image is the weighted sum of the gray values of the two images.

[0038] In this embodiment, the structures of the generator G2 and the generator G3 are the same, as shown inFigure 4 As shown, the generation process of the final fusion image If is as follows: in the feature sampling and fusion process, the image weight under different scenes is set, the fusion features are calculated, and the calculation adopts a five-level convolution structure, wherein: the first and third layers contain an inverse convolution, a LeakyRelu activation function and a batch normalization (BatchNorm). The second and fourth layers contain a convolution, a LeakyRelu activation function and a batch normalization (BatchNorm). The first two layers of the network use a 5x5 convolution kernel, and the third and fourth layers use a 3x3 convolution kernel. In order to ensure that there is no color deviation in the generation process of the color image, the entire generator uses 3 filters in the last layer to output a 3-channel color image.

[0039] It can be seen that the visible light image has an image feature that will cause the brightness to decrease when generating a fusion image with an infrared image. In this method, the same generator G2 is used for visible light images under strong light intensity as for visible light images under weak light intensity. The generator G2 itself improves the brightness of the image, so it can make up for the brightness decrease caused by the fusion of visible light images and infrared images under normal light conditions, and the effect is better than that of traditional weighted methods and wavelet methods, and the brightness will be improved. For example, Figure 7a and Figure 7b As shown, Figure 7a is the result processed by the PMCI model, Figure 7b is the result processed by the generator G2 of the present application, and it can be seen that the present application can significantly improve the brightness (and the color is also better in fact).

[0040] In order to ensure that the generated image can better extract the features of the source image, the entire scheme adopts a feature discrimination structure to complete the judgment of the quality of the synthesized image. As shown, Figure 5 each discriminator (D1 and D2) structure is composed of four convolution layers and a fully connected layer. LeakyRelu activation function, 3x3 convolution kernel and step length of 2 are used in the first four convolution layers, and a fully connected layer is finally used to integrate and classify the features after the first four convolution layers.

[0041] A large number of visible light images and corresponding infrared images and corresponding high-quality generated images (the high-quality generated images here refer to images that meet the standards set by the user, which can be different based on different user-set standards) are used to train the discriminator model. In the training process, features are extracted from the visible light images and the infrared images, the visible light images are generated, and then compared with the high-quality generated images, and returned to iteration to generate reasonable network parameters. After the training is completed, the trained discriminator is used to judge the visible light images, and when the judgment result is that the generated image does not meet the ideal effect, the generator (G1 or G2) is returned for reiteration until the output result meets the judgment standard of the discriminator.

[0042] In the discriminator use, in view of the training instability, slow speed and other problems that may occur in the generator, a double time scale update rule (i.e. different learning rates are set for the generator and discriminator network respectively) and a spectral normalization algorithm (i.e. the spectral norm of the weight matrix of each layer of the GAN network discriminator is constrained to constrain the Lipschitz constant of the discriminator, and the stability in the GAN training process is enhanced) are introduced into the network, so that the generation and discrimination are updated at an equal rate of 1:1, and the singular value decomposition of each layer of the neural network parameter is carried out, and then the largest singular value is limited to 1, to meet the Lipschitz continuous condition.

[0043] Finally, the fusion image If fusion quality is evaluated by subjective and objective evaluation indexes, the subjective evaluation is whether the visual effects such as color, brightness, fidelity and the like of the fusion image give people a satisfactory feeling. The objective evaluation is to quantitatively analyze the fusion image If by calculating the related index information of the image through a specific formula, including information entropy (Information Entropy, IE), standard deviation (Standard Deviation, SD), average gradient (Average Gradient, AG), spatial frequency (Spatial Frequencies, SF), structural similarity (Structural Similarity, SSIM), correlation coefficient (Correlation Coefficient, CC) and edge information transmission Q AB / F Seven indexes are used for objective evaluation, as shown in Table 1.

[0044] Table 1 Objective evaluation index

[0045]

[0046]

[0047] In order to verify the advantages of the method of the application, the method of the application and other five image fusion methods DeepFuse, FusionGan, IFCNN, MEF-GAN and PMCI are compared and analyzed in each stage model through the subjective and objective evaluation indexes and running time.

[0048] From the subjective indicators, for the details of the dark image (visible light image obtained under low light intensity), the information loss of the methods of FusionGAN, MEF-GAN and PMCI is serious, and the fusion results are unnatural, and the image outline of MEF-GAN is too sharp. In contrast, the results of the present application show clear targets and details closer to the visible light image. Compared with FusionGAN, due to the introduction of different network architectures, loss functions and training strategies, the final fusion image contains more natural details. For the contrast of the bright image (visible light image obtained under high light intensity), the contrast of the methods of DeepFuse, IFCNN and MEF-GAN is not high, the fusion results do not highlight the target, and the fusion results of PMCI are unnatural. The results of the present application not only show high-contrast small targets, but also retain complex texture backgrounds, and the fusion image looks very natural. In addition, the fusion image If generated by the present application is an RGB color image, which is beneficial for target detection in complex environments such as fire.

[0049] Select 20 images for quantitative comparison of 6 fusion evaluation indicators, and the results are shown in Table 1. Figure 6 As can be seen, the pre-trained model indicators have the optimal average values of EN, SD, SF and SSIM. The optimal average value is obtained on EN, which means that the information amount of the fusion image is rich and the quality is good; the optimal average value is obtained on SD, which is sufficient to prove that the generated results have high contrast between the target and the background; the optimal average value is obtained on SF, which means that more detailed information is obtained from the fusion image, and the fusion image is clearer; and the optimal average value is obtained on SSIM, which means that the fusion image retains the information of the visible light image and the infrared image. These results show that the image generated by the integrated fusion model in the present application has the largest information amount, the highest contrast, the most abundant edge and texture details, and the greatest similarity to the source image.

[0050] Finally, the running time of different network models is compared, as shown in Table 2. The integrated fusion model of the present application has little difference from DeepFuse on STD, but the Mean value of the model after transfer learning is improved by 53.3% compared with other methods. Therefore, overall, the running efficiency of the present application is the highest.

[0051] Table 2 Running time of different models

[0052] Method DeepFuse FusionGAN FusionDN IFCNN MEF-GAN PMCI Our Mean 0.6898 3.2632 18.4564 2.9738 4.5387 1.5457 0.3221 STD 0.0250 0.2642 0.5497 0.2437 0.0542 0.0358 0.0305

[0053] By the above method of the present application, the problem of brightness reduction caused by the fusion of visible light images and infrared images under normal lighting conditions can be compensated, and the problem of insufficient details of the fusion image under dark illumination conditions can also be solved.

[0054] Based on the same inventive concept, the embodiment of the present application also provides a system for infrared image and visible light image fusion matching, comprising:

[0055] An illumination judgment module is configured to judge whether the visible light image is collected under strong light illumination or weak light illumination.

[0056] A fusion module comprises an integrated fusion model, which is configured to directly fuse the visible light image and the infrared image to obtain a fusion image when the visible light image is collected under strong light illumination, and generate an infrared image from the visible light image, then fuse the generated infrared image and the collected infrared image to generate a weighted infrared image, and finally splice the weighted infrared image and the visible light image to generate a final fusion image when the visible light image is collected under weak light illumination.

[0057] Specifically, the integrated fusion model comprises a first generator G1, a second generator G2, a third generator G3, a first discriminator D1 and a second discriminator D2, the second generator G2 and the third generator G3 have the same structure, the first generator G1 is configured to generate an infrared image from a visible light image, and the first discriminator D1 is configured to perform quality discrimination on the generated infrared image; the second generator G2 is configured to fuse the generated infrared image and the collected infrared image to generate a weighted infrared image, and splice the weighted infrared image and the visible light image to generate a final fusion image; the third generator G3 is configured to directly fuse the visible light image and the infrared image to obtain a fusion image; and the second discriminator D2 is configured to perform quality discrimination on the generated fusion image.

[0058] Other details can be found in the foregoing method embodiments, and will not be described here.

[0059] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for fusing and matching infrared images and visible light images, wherein the infrared images and visible light images are acquired in the same environment, characterized in that... Including the following steps: Determine whether a visible light image was acquired under strong light or weak light conditions; If the visible light image is acquired under strong light intensity, the visible light image is directly fused with the acquired infrared image to obtain a fused image; If the visible light image is acquired under low light conditions, the visible light image is first converted into an infrared image. Then, the generated infrared image is fused with the acquired infrared image to generate a weighted infrared image. The weighted infrared image is then stitched with the visible light image to generate the final fused image. The steps for generating an infrared image from a visible light image Iv include: first, using multi-scale features from a deep feature extraction network to obtain feature maps of the acquired infrared image Ir and the visible light image Iv respectively; then, using a feature upsampling reconstruction process to fuse the two feature maps to generate the infrared image Iv'.

2. The method for fusing and matching infrared images and visible light images according to claim 1, characterized in that, The step of determining whether a visible light image was acquired under strong light intensity or weak light intensity includes: calculating the brightness of each pixel using the maximum value method, then using the average brightness of all pixels as the brightness of the entire visible light image, and determining whether the brightness of the entire visible light image is greater than a set threshold. If it is, the visible light image is determined to have been acquired under strong light intensity; otherwise, it is determined to have been acquired under weak light intensity.

3. The method for fusing and matching infrared images and visible light images according to claim 1, characterized in that, The steps for directly fusing visible light images with acquired infrared images include: first, using multi-scale features from a deep feature extraction network to obtain feature maps of the acquired infrared and visible light images respectively; and then using a feature upsampling reconstruction process to fuse the two feature maps.

4. The method for fusing and matching infrared images and visible light images according to claim 1, characterized in that, The step of fusing the generated infrared image with the acquired infrared image to generate a weighted infrared image includes: setting corresponding weights based on the grayscale information of the generated infrared image and the acquired infrared image; assigning weights to the grayscale values ​​of the generated infrared image and the acquired infrared image at the same pixel point; and the grayscale of the weighted infrared image is the weighted sum of the grayscale values ​​of the generated infrared image and the acquired infrared image.

5. The method for fusing and matching infrared images and visible light images according to claim 1, characterized in that, It also includes the step of quality assessment of the final fused image.

6. A system for fusing and matching infrared and visible light images, wherein the infrared and visible light images are acquired in the same environment, characterized in that... include: The illuminance determination module is used to determine whether a visible light image was acquired under strong illuminance or weak illuminance. The fusion module includes an integrated fusion model, which is used to directly fuse a visible light image with an infrared image when the visible light image is acquired under strong illuminance to obtain a fused image; and when the visible light image is acquired under weak illuminance, it generates an infrared image from the visible light image, then fuses the generated infrared image with the acquired infrared image to generate a weighted infrared image, and then stitches the weighted infrared image with the visible light image to generate the final fused image. The integrated fusion model includes a first generator, which is used to generate an infrared image from a visible light image. The first generator first uses multi-scale features of a deep feature extraction network to obtain feature maps of the infrared image Ir and the visible light image Iv, and then uses a feature upsampling reconstruction process to fuse the images to generate the infrared image Iv'.

7. The system for fusing and matching infrared and visible light images according to claim 6, characterized in that, The integrated fusion model further includes a second generator, a third generator, a first discriminator, and a second discriminator. The first discriminator is used to perform quality discrimination on the generated infrared image. The second generator is used to fuse the generated infrared image with the acquired infrared image to generate a weighted infrared image, and then stitch the weighted infrared image with the visible light image to generate the final fused image. The third generator is used to directly fuse the visible light image and the infrared image to obtain the fused image. The second discriminator is used to perform quality discrimination on the generated fused image.

8. The system for fusing and matching infrared and visible light images according to claim 7, characterized in that, The first generator consists of an encoder and a decoder. The encoder contains 7 convolutional blocks for downsampling. Except for the first convolutional block, the next 6 convolutional blocks contain a convolutional layer, a LeakyReLU activation function, and a batch normalization. The convolutional layers use 4×4 convolutional kernels with a stride q of 2. The decoder contains 7 convolutional blocks. The first 6 convolutional blocks contain a convolutional layer, a ReLU activation function, and a batch normalization. The last convolutional block contains a convolutional layer and a tanh activation function.

9. The system for fusing and matching infrared and visible light images according to claim 7, characterized in that, The second generator consists of five convolutional layers. The first and third layers contain a deconvolution, a LeakyReLU activation function, and a batch normalization. The second and fourth layers contain a convolution, a LeakyReLU activation function, and a batch normalization. The first and second layers use 5×5 convolutional kernels, while the third and fourth layers use 3×3 convolutional kernels.

Images