Visible light and thermal imaging image fusion method based on adaptive weight allocation

Abstract

The present invention provides a method for fusing visible light and thermal imaging images based on adaptive weight allocation, and relates to the technical field of dual-light night vision device image processing. The method fuses images of a partial region of visible light and corresponding regions of a thermal imaging zoom image according to the similarity. The present invention is based on an in-depth analysis and utilization of the image's own feature information, fully utilizing the complementarity of visible light images and thermal imaging images in terms of texture, brightness, contrast, etc., and realizes efficient fusion of the two images by adaptively adjusting the fusion weights of different regions.

Description

Technical Field

[0001] The present invention relates to the technical field of dual-light night vision device image fusion, and specifically provides a dual-light fusion method of visible light and thermal imaging images based on multi-scale features. Background Art

[0002] Visible light images and thermal images provide different visual information. Visible light images have rich color, texture and detail information, while thermal images can reflect the thermal radiation distribution of objects. They have significant advantages for certain specific applications, such as target tracking, night monitoring, and when targets and backgrounds are difficult to distinguish in visible light scenes. Therefore, fusing visible light images and thermal images can make full use of the advantages of both and improve the information content and clarity of the images. However, due to the different imaging principles of visible light images and thermal images, there are large differences in their image features, resolution, field of view, brightness, etc. Direct fusion will produce large distortion and deviation.

[0003] Image fusion is the process of comprehensively processing multi-source image information to obtain more comprehensive and accurate information. In the fusion of visible light and thermal images, due to the different imaging principles of the two, there are often differences in brightness, contrast, texture, etc., resulting in unsatisfactory fusion results. Therefore, how to effectively fuse visible light and thermal images has become a problem that needs to be solved in the current image processing field. Summary of the invention

[0004] The technical task of the present invention is to provide a dual-light fusion method of visible light and thermal imaging images based on multi-scale features in order to address the above-mentioned problems.

[0005] A method for fusing visible light and thermal imaging images based on adaptive weight allocation is provided. The method is based on a partial area image of visible light. Zooming images with thermal imaging The corresponding regions are fused based on their similarity.

[0006] Furthermore, the implementation steps of the method are as follows:

[0007] (1) Pull the stream to obtain the current frame visible light image V and the current frame thermal imaging image I;

[0008] (2) Obtain the corresponding areas of the visible light image V and the thermal imaging image I through the camera parameters,

[0009] Preprocess the visible light image V and the thermal image I.

[0010] Get a partial area image of visible light aligned at the pixel level (The field of view of visible light images is wider than that of thermal imaging) and thermal imaging zoom images ;

[0011] (3) Comparison of images of some regions of visible light Zooming images with thermal imaging Similarity of corresponding regions;

[0012] (4) Fusion is performed based on the degree of similarity to obtain a fused image.

[0013] Furthermore, in step (4), if the similarity is large, the image of the partial area of ​​the visible light Zooming images with thermal imaging The information in this area is relatively consistent, and the same parameters (such as average weight) are used for fusion;

[0014] If the similarity is small, select texture, gradient information complexity and Gaussian weight to create parameter fusion;

[0015] Or, choose to fuse the highlighted areas of the thermal image with the main weight.

[0016] Furthermore, the camera parameters are: resolution, focal length, pixel size, and baseline.

[0017] The preprocessing method includes denoising and size adjustment.

[0018] Furthermore, the partial area image of the visible light Zooming images with thermal imaging The similarity of the corresponding areas is measured by using a 16*16 sliding window in the visible light area image. Zooming images with thermal imaging Traverse up to compare.

[0019] Furthermore, the similarity calculation method includes: calculating the grayscale histogram difference or structural similarity index between regions.

[0020] Furthermore, when the corresponding regions of the two images are similar, the images of the visible light regions Zooming images with thermal imaging The fusion formula is as follows:

[0021] (1)

[0022] in, ,in is the coordinate of the pixel on the image.

[0023] Furthermore, when the similarity is small, the fusion weight determination process is as follows:

[0024] (5.1) Analysis of partial area images of visible light Zooming images with thermal imaging Texture information of the corresponding area;

[0025] (5.2) After determining which image has richer texture information, an adaptive weight allocation strategy is used for image fusion: image regions with more texture information are fused using larger weights, and image regions with less texture information are fused using smaller weights.

[0026] Furthermore, the method uses Gaussian weight parameters for image fusion, and the specific process is as follows:

[0027] First, the texture complexity and edge information of all corresponding areas in the 3×3 sliding window are calculated, and then the fusion parameters are determined based on this information;

[0028] The formula for determining the fusion parameters is expressed in the form of a Gaussian function;

[0029] The Gaussian function is a continuous and differentiable function, whose shape is determined by the mean and standard deviation;

[0030] In image fusion, the mean of the Gaussian function is set as the center point of the sliding window, and the standard deviation is adjusted according to actual needs;

[0031] A Gaussian function is used to assign a weight to each pixel that is related to its distance from the center point, thus achieving a smooth and continuous fusion effect.

[0032] The Gaussian weight parameter can automatically adjust the weight size according to the distance between the pixel and the center point, thereby achieving smooth transition and continuous fusion;

[0033] The specific form of the Gaussian function is as follows:

[0034]

[0035] in, Indicates at location Gaussian weight, A is the amplitude, which is used to control the size of the weight, and the value is 1; is the center point of the Gaussian function and the center point of the 3*3 sliding window. is the standard deviation, and the value is 2;

[0036] The final fusion weight parameter is obtained by calculating the Gaussian weight of each pixel and multiplying it with the corresponding texture complexity or edge information (gradient information). The definition is as follows:

[0037]

[0038] Then substitute (3) into (1) and fuse them together, where n is the number of pixels in the sliding window, is the coordinate within the range of Gaussian weight, They are the gradient information in the horizontal and vertical directions respectively.

[0039] These parameters are used to guide the fusion process of visible light images and thermal images. This processing can not only eliminate the discontinuity of the fusion boundary, but also enhance the texture information to achieve high-quality and high-definition fusion results.

[0040] Furthermore, for thermal imaging zoom images The corresponding area is in the highlighted area, while the image of the visible light area When the corresponding area of ​​the image is dark and has no detail information, the method uses the image of the highlight area as the main fusion information and sets a larger fusion weight. The calculation formula is as follows:

[0041]

[0042] A parameter that is a factor of 0.98 of the normalized brightness;

[0043] The highlight area is determined by the brightness threshold, and the contrast difference between the highlight area and the dark area is calculated to ensure that the highlight information is effectively retained in the fusion result.

[0044] Compared with the prior art, the dual-light fusion method of visible light and thermal imaging images based on multi-scale features of the present invention has the following outstanding beneficial effects:

[0045] The method of the present invention is based on the in-depth analysis and utilization of the characteristic information of the image itself, fully utilizes the complementarity of visible light image and thermal image in terms of texture, brightness, contrast, etc., and realizes the efficient fusion of the two images by adaptively adjusting the fusion weights of different areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart for realizing the method of the present invention;

[0047] Figure 2 It is the original visible light image collected at night;

[0048] Figure 3 It is the original thermal imaging image collected at night;

[0049] Figure 4 is the denoised image of thermal imaging;

[0050] Figure 5 It is the transformation area of ​​the denoised image of thermal imaging relative to the original visible light image;

[0051] Figure 6 It is a comparison of the tree trunk area of ​​the visible light image, thermal imaging image, and fused image with high similarity;

[0052] Figure 7 This is a comparison of the pine tree area in visible light, the pine tree area in thermal imaging, and the fused pine tree area;

[0053] Figure 8 It is the data of the pine tree area in the Blue color space of the visible light image;

[0054] Fig. 9 It is the data corresponding to the Blue color space of the thermal imaging image;

[0055] Fig.10 is the calculated fusion weight data;

[0056] Fig.11 It corresponds to Figure 8 and Fig. 9 use Fig.10 The data after weight data fusion;

[0057] Fig.12 It is a comparison chart of the pedestrian area of ​​visible light, the pedestrian area of ​​thermal imaging, and the pedestrian area after fusion;

[0058] Fig.13 is the final fused result image. DETAILED DESCRIPTION

[0059] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0060] like Figure 1 As shown, the steps of image fusion are as follows:

[0061] (1) Pull the stream to obtain the current frame visible light image V, such as Figure 2 As shown, the current frame thermal imaging image I, such as Figure 3 As shown;

[0062] (2) The camera parameters of different products are also different. The corresponding parameter adjustments are also required for different observation distances. The following table gives the corresponding parameters for a 50m observation distance:

[0063]

[0064] The corresponding areas of the visible light image V and the thermal imaging image I are obtained through the camera parameters, and the visible light image and the thermal imaging image are preprocessed, including denoising, size adjustment, etc. Figure 4 As shown, the part of the image area where visible light is (The field of view of visible light images is wider than that of thermal images) Zoomed image of thermal images Align at the pixel level, such as Figure 5 As shown;

[0065] (3) Use a 16*16 sliding window to map the visible light image With thermal imaging Traverse up and compare the similarity of corresponding regions. The similarity can be obtained by, but not limited to, calculating the grayscale histogram difference between regions, the structural similarity index, and other methods; Example 1

[0066] like Figure 6 The trunk area shown in the figure is the same as the trunk area shown in the figure. When the similarity of the corresponding areas of the two images is large, it is considered that the information of the two images in this area is relatively consistent. In the visible light image and the thermal image, the edge information is obvious and similar, but the texture information is not very obvious. Formula (1) is used for fusion. , using the same parameters (such as average weight) for fusion, the formula is as follows:

[0067]

[0068] Here ,in Figure 6 In the tree trunk area of ​​the visible light image on the middle left, the value of the first pixel in the blue color space is 55, and the value of the first pixel in the blue color space of the middle image is 123. Using formula (1), the value of the first pixel in the blue color space of the fused image is R=0.5*55+0.5*123=89, as shown in the right figure. The other pixel values ​​are processed in turn. Example 2

[0069] When the similarity is small, the following scheme is used to determine the fusion weight:

[0070] When fusing regions with complex texture and gradient information, the fusion weights are calculated using formulas (2) and (3). The fused image is Figure 7 As shown;

[0071] Here, taking the pixels of the 3*3 area at the edge of the pine tree (i.e., 9*9 pixels) as an example, where n=3 in formula (3), the data of the pine tree area in the blue color space of the visible light image is selected as follows: Figure 8 As shown;

[0072] The data corresponding to the Blue color space of the thermal imaging image is as follows Fig. 9 As shown;

[0073] The fusion weight data calculated according to formula (2) and (3) is as follows: Fig.10 As shown;

[0074] Substituting into formula (1), the fused data is as follows Fig.11 As shown;

[0075] Example 3

[0076] For the highlighted areas of the thermal image, the highlighted pixels of the thermal image account for the main fusion weight during fusion, for example Fig.12 In the pedestrian target area, a coefficient of 0.98 times the normalized brightness is used as the fusion weight. The fused image is as follows: Fig.12 As shown in the rightmost figure;

[0077] Among them, for example Fig.12 The value of the first pixel in the Blue color space of the left image is 92, and the value of the first pixel in the Blue color space of the middle image is 63. Using formula (1), the value of the first pixel in the Blue color space of the fused image is R=(1-0.98*63 / 255)*92+(0.98*63 / 255)*63=84.97 (the integer 85 when the image is saved). The other pixel values ​​are processed in turn.

[0078] After the calculations in the above embodiments, the final fusion result is as follows: Fig.13 shown.

[0079] The embodiments described above are only preferred specific implementations of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for fusing visible light and thermal imaging images based on adaptive weight allocation, characterized in that: The method is based on the partial area image V of visible light P Scaling images with thermal imaging I s The corresponding regions are fused according to their similarity; The implementation steps of the method are as follows: (1) Pull the stream to obtain the current frame visible light image V and the current frame thermal imaging image I; (2) Obtain the corresponding areas of the visible light image V and the thermal imaging image I through camera parameters, Preprocess the visible light image V and the thermal image I. Get the partial area image V of the visible light aligned at the pixel level P Scaling images with thermal imaging I s ; (3) Comparison of partial region images V of visible light P Scaling images with thermal imaging I s Similarity of corresponding regions; (4) Fusion is performed based on the similarity to obtain a fused image; In the step (4), if the similarity is large, the partial area image V of the visible light P Scaling images with thermal imaging I s The information in this area is relatively consistent, and the same parameters are used for fusion; If the similarity is small, select texture, gradient information complexity and Gaussian weight to create parameter fusion; Or, select the highlighted area of ​​the thermal image to account for the main weight fusion; Among them, the specific process of selecting texture, gradient information complexity and using Gaussian weight parameters for image fusion is as follows: First, the texture complexity and edge information of all corresponding areas in the 3×3 sliding window are calculated, and then the fusion parameters are determined based on this information; The formula for determining the fusion parameters is expressed in the form of a Gaussian function; The Gaussian function is a continuous and differentiable function, whose shape is determined by the mean and standard deviation; In image fusion, the mean of the Gaussian function is set as the center point of the sliding window, and the standard deviation is adjusted according to actual needs; A Gaussian function is used to assign a weight to each pixel that is related to its distance from the center point, thus achieving a smooth and continuous fusion effect. The specific form of the Gaussian function is as follows: Among them, G(x,y) represents the Gaussian weight at the position (x,y), A is the amplitude, which is used to control the size of the weight, and the value is 1; (x c ,y c ) is the center point of the Gaussian function, is the center point of the 3*3 sliding window, σ x ,σ y is the standard deviation, and the value is 2; By calculating the Gaussian weight of each pixel and multiplying it with the corresponding texture complexity or edge information, the final fusion weight parameter α is defined as follows: Then substitute (3) into (1) to merge, where n is the number of pixels in the sliding window, i, j are the coordinates within the range of Gaussian weight, G x , G y They are the gradient information in the horizontal and vertical directions respectively.

2. The method for fusing visible light and thermal imaging images based on adaptive weight allocation according to claim 1, characterized in that: The camera parameters are: resolution, focal length, pixel size, baseline; The preprocessing method includes denoising and size adjustment.

3. The method for fusing visible light and thermal imaging images based on adaptive weight allocation according to claim 1, characterized in that: The partial region image V of the visible light P Scaling images with thermal imaging I s The similarity of the corresponding regions is measured by using a 16*16 sliding window in the visible light region image V P Scaling images with thermal imaging I s Traverse up to compare.

4. The method for fusing visible light and thermal imaging images based on adaptive weight allocation according to claim 1, characterized in that: The similarity calculation method includes: calculating the gray histogram difference or structural similarity index between regions.

5. The method for fusing visible light and thermal imaging images based on adaptive weight allocation according to claim 1, characterized in that: When the similarity of the corresponding areas of the two images is large, the image V of the visible light area P Scaling images with thermal imaging I s The fusion formula is as follows: R(x,y)=αI s (x,y)+(1-α)V P (x,y) (1) Among them, α=0.5, and (x, y) is the coordinate of the pixel point on the image.

6. The method for fusing visible light and thermal imaging images based on adaptive weight allocation according to claim 1, characterized in that: When the similarity is small, the fusion weight determination process is as follows: (5.1) Analyze the partial region image V of the visible light P Scaling images with thermal imaging I s Texture information of the corresponding area; (5.2) After determining which image has richer texture information, an adaptive weight allocation strategy is used for image fusion: image regions with more texture information are fused using larger weights, and image regions with less texture information are fused using smaller weights.

7. The method for fusing visible light and thermal imaging images based on adaptive weight allocation according to claim 1, characterized in that: For thermal imaging zoom image I s The corresponding area is in the highlighted area, while the image V P When the corresponding area of ​​the image is dark and has no detail information, the method uses the image of the highlight area as the main fusion information and sets a larger fusion weight. The calculation formula is as follows: R(x,y)=αI s (x,y)+(1-α)V P (x,y) α is a parameter that is 0.98 times the normalized brightness; The highlight area is determined by the brightness threshold, and the contrast difference between the highlight area and the dark area is calculated to ensure that the highlight information is effectively retained in the fusion result.

Images