An adaptive image fusion method and system based on saliency preservation

Abstract

The application discloses a kind of based on saliency keeping adaptive image fusion method and system, it is related to image processing technical field, method includes: the threshold division of the gray histogram of visible light image, the pixel number proportion of three pixel regions is calculated;Based on the LC algorithm of histogram, the target saliency of infrared image is extracted, and the saliency image is obtained;Visible light image and infrared image are input to the basic fusion model of deep learning to generate basic fusion image;The adaptive fusion of saliency image and basic fusion image is carried out using the gray histogram of visible light image and the pixel number proportion of three pixel regions.The application can improve the quality of fusion image under different illumination environments, and the robustness of subsequent advanced visual detection task;Using the saliency detection algorithm combined with histogram, the target saliency of infrared image is extracted, and adaptive re-fusion is carried out with the basic fusion image, which effectively enhances the target saliency feature of fusion image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an adaptive image fusion method and system based on saliency preservation. Background Technology

[0002] With the rapid development of sensing technology, infrared imaging technology has been widely used in industry, military, and civilian applications. Visible light has low sensitivity to illumination and low penetrating power, making it unsuitable for visual tasks in nighttime working environments; while infrared light has strong penetrating power, its images contain less textural feature information. Therefore, fusing visible and infrared light to generate fused images rich in feature information is of great significance for improving the performance of visual tasks.

[0003] Existing dual-light fusion algorithms suffer from problems such as loss of texture details, incomplete preservation of salient targets and contours, leading to insufficient target saliency and missed detections in subsequent computer vision tasks. Furthermore, most fusion methods use consistent fusion weights and do not consider the effectiveness of visible light images under different lighting conditions, resulting in some fused images becoming noisy images, which is not conducive to the perception and understanding of scene information. Summary of the Invention

[0004] The present invention provides an adaptive image fusion method and system based on saliency preservation, which can alleviate the above-mentioned problems.

[0005] To alleviate the above problems, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides an adaptive image fusion method based on saliency preservation, comprising the following steps:

[0007] S1. Acquire infrared and visible light images, preprocess the infrared images, align the infrared and visible light images in time and space, and perform unified normalization on the infrared and visible light images.

[0008] S2. Obtain the grayscale histogram of the visible light image. Utilize the characteristic that the pixel range of the grayscale histogram can represent the brightness of the image. Divide the grayscale histogram into three pixel regions based on the segmentation threshold. Calculate the pixel count ratio of the three pixel regions (S1, S2, S3) according to the segmentation threshold.

[0009] S3. Using the histogram-based LC algorithm, extract the target saliency of the preprocessed infrared image in step S1 to obtain a salient image;

[0010] S4. Input the visible light image and the preprocessed infrared image from step S1 into the basic fusion model of deep learning to generate a basic fusion image. The basic fusion model includes an encoder and a decoder. The encoder is used to extract the feature information of the input infrared image and the visible light image, and the decoder is used to fuse the extracted feature information of the infrared image and the visible light image.

[0011] S5. Adaptive fusion of salient image and basic fusion image is performed using the grayscale histogram of visible light image and the pixel quantity ratio of three pixel regions (S1, S2, S3).

[0012] In a preferred embodiment of the present invention, the method for preprocessing the infrared image in step S1 includes denoising the infrared image using a median filter operator and increasing the image contrast of the infrared image.

[0013] In a preferred embodiment of the present invention, in step S1, the grayscale of the noise-filtered infrared image is mapped to a new range by a nonlinear grayscale transformation operator, thereby increasing the image contrast.

[0014] In a preferred embodiment of the present invention, the formula for calculating the pixel count ratio of the three pixel regions in step S2 is as follows:

[0015]

[0016] Where, n i The value represents the number of pixels when the pixel value is i, N represents the total number of pixels in the grayscale histogram, and m1 and m2 represent the segmentation thresholds for the two pixel intervals.

[0017] In a preferred embodiment of the present invention, step S3 includes the following steps:

[0018] S31. Normalize the pixel feature values ​​of the preprocessed infrared image to [0, 255];

[0019] S32. Based on the grayscale histogram of the visible light image, classify and statistically analyze the normalized pixel feature values ​​to obtain the frequency f of the feature values. n , where n represents the normalized feature value of a pixel;

[0020] S33. Calculate the Euclidean distance d between any two normalized pixel feature values ​​p and n.

[0021] d = D(p,n), p ≠ n;

[0022] S34. Based on the Euclidean distance d and the frequency of occurrence of eigenvalues ​​f nCalculate the target saliency S(p) of the preprocessed infrared image.

[0023]

[0024] S35. Normalize the target saliency S(p) to [0,255] to obtain the saliency image.

[0025] In a preferred embodiment of the present invention, in step S4,

[0026] The input infrared image features include semantic structural features; the input visible light image features include contour features and texture features; the encoder includes two paths: one path consists of convolutional layers, batch normalization layers, and activation functions connected in series to obtain the input visible light image features, and the other path consists of residual modules and convolutional layers connected in series to obtain the input infrared image features.

[0027] The decoder consists of a cascaded feature fusion layer and a convolutional layer, used to decode the feature information of the acquired infrared and visible light images to obtain the base fused image.

[0028] In a preferred embodiment of the present invention, in step S5...

[0029] The fusion weights are calculated using the following formula:

[0030]

[0031] Where 1>b1>b2>b3>0, and max(S1,S2,S3) represents the calculation of the maximum value of (S1,S2,S3);

[0032] The adaptive fusion strategy is as follows:

[0033] F = aI1 + bI2, a + b = 1

[0034] Where F represents the final fused image, I1 represents the base fused image features, a represents the fusion weight of I1, I2 represents the saliency features, and b represents the fusion weight of I2.

[0035] Secondly, the present invention provides a system comprising a dual-light camera capable of acquiring infrared and visible light images, and a terminal processor connected to the dual-light camera via a network, wherein the terminal processor is programmed to execute the aforementioned adaptive image fusion method based on saliency preservation when the program is executed.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] 1) An adaptive fusion strategy based on illumination changes is proposed. The gray-level histogram of the visible light image is used to perform thresholding to calculate the brightness of the image. The fusion weight is set according to the brightness distribution. The fusion weight is adaptively adjusted according to the changes in the brightness distribution characteristics of the image to improve the quality of the fused image under different illumination environments and the robustness of subsequent advanced visual detection tasks.

[0038] 2) By using the LC saliency detection algorithm combined with histograms, the target saliency of the infrared image is extracted and adaptively re-fused with the base fused image, which effectively enhances the target saliency features of the fused image.

[0039] 3) The fusion is performed in two steps. First, the encoder and decoder are used to obtain feature information and perform basic fusion to ensure that the semantic features of the fused image are not destroyed. Then, an adaptive fusion strategy is used to adaptively fuse the target saliency of the basic fused image with the infrared image extracted by the LC saliency detection algorithm to enhance the contour and detail features of the target.

[0040] 4) Since the three paths of LC saliency detection algorithm, acquisition of basic fused image, and threshold division calculation of adaptive fusion weight using gray-level histogram of visible light image are parallel, the LC saliency detection algorithm and threshold division calculation of adaptive fusion weight using gray-level histogram of visible light image can be directly used for saliency adaptive fusion of basic fused image obtained by other fusion algorithms to improve the target saliency of basic fused image while ensuring the quality of fused image.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, embodiments of the present invention are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of grayscale histogram threshold division in this invention;

[0044] Figure 2 This is a schematic diagram of the algorithm flow of the present invention;

[0045] Figure 3 This is an example of grayscale histograms of visible light images with different brightness in this invention;

[0046] Figure 4 This is a diagram of the saliency-preserving fusion model algorithm structure in this invention;

[0047] Figure 5 This is a diagram illustrating the algorithm. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0049] Please refer to Figure 2 This invention provides an adaptive image fusion method based on saliency preservation, comprising the following steps:

[0050] S1. Acquire infrared and visible light images. Apply a median filter to the infrared data to remove noise. Simultaneously, apply a nonlinear grayscale transformation operator to the infrared image data to map the grayscale of the image to a new range, increasing image contrast. Then, align the infrared and visible light images in time and space. Finally, to accelerate the model convergence speed, normalize the infrared and visible light images.

[0051] S2: Using the OpenCV library in Python, first convert the visible light image into a grayscale image with pixel values ​​ranging from 0 to 255. Then, use the library function `histogram()` to calculate the number of each pixel value. Finally, obtain the grayscale histogram of the visible light image with the number of pixels as the ordinate and the pixel values ​​as the abscissa. Utilizing the characteristic that the pixel range of the grayscale histogram can represent the brightness of the image, thresholding is performed on the grayscale histogram based on a segmentation threshold, such as... Figure 1 As shown, three pixel regions are obtained, and the pixel count ratios (S1, S2, S3) of the three pixel regions are calculated based on the segmentation threshold.

[0052] The formula for calculating the percentage of pixels in the three pixel regions is:

[0053]

[0054] Where, n i S represents the number of pixels when the pixel value is i, N represents the total number of pixels in the grayscale histogram, and m1 and m2 represent the pixel segmentation thresholds for dividing the pixel intervals into weak and bright light, respectively. The grayscale histograms and corresponding S values ​​of visible light images with different brightness are shown below. Figure 3 As shown.

[0055] S3. Based on the histogram-based LC algorithm, extract the target saliency of the preprocessed infrared image in step S1 to obtain a salient image. This specifically includes the following steps:

[0056] S31. Normalize the pixel feature values ​​of the preprocessed infrared image to [0, 255];

[0057] S32. Based on the grayscale histogram of the visible light image, classify and statistically analyze the normalized pixel feature values ​​to obtain the frequency f of the feature values. n , where n represents the normalized feature value of a pixel;

[0058] S33. Calculate the Euclidean distance d between any two normalized pixel feature values ​​p and n.

[0059] d = D(p,n), p ≠ n;

[0060] S34. Based on the Euclidean distance d and the frequency of occurrence of eigenvalues ​​f n Calculate the target saliency S(p) of the preprocessed infrared image.

[0061]

[0062] S35. Normalize the target saliency S(p) to [0,255] to obtain the saliency image.

[0063] S4. Input the visible light image and the preprocessed infrared image from step S1 into the deep learning base fusion model to generate a base fusion image. The base fusion model includes an encoder and a decoder.

[0064] The encoder consists of two paths. One path comprises a concatenated convolutional layer, a batch normalization layer, and an activation function, primarily used to acquire contour features, texture, and other feature information from the visible light image. The other branch, a deeper branch consisting of a residual module and a concatenated convolutional layer, is mainly used to acquire semantic structural features from the infrared image. The decoder mainly consists of a feature fusion layer and a concatenated convolutional layer, primarily used to decode the feature information from the acquired infrared and visible light images to obtain a basic fused image.

[0065] S5. Adaptive fusion of salient image and basic fusion image is performed using the grayscale histogram of visible light image and the pixel quantity ratio of three pixel regions (S1, S2, S3).

[0066] The fusion weight is calculated as follows:

[0067]

[0068] Where 1>b1>b2>b3>0, and max(S1,S2,S3) represents the calculation of the maximum value of (S1,S2,S3);

[0069] The adaptive fusion strategy is as follows:

[0070] F = aI1 + bI2, a + b = 1

[0071] Where F represents the final fused image, I1 represents the base fused image features, a represents the fusion weight of I1, I2 represents the saliency features, and b represents the fusion weight of I2.

[0072] The saliency-preserving adaptive image fusion method described in this invention fuses infrared and visible light images using an ncoder and decoder to generate a base fused image. Simultaneously, the LC algorithm based on histograms extracts the target saliency from the infrared image. Using the OpenCV library in Python, the visible light image is first converted into a grayscale image with pixel values ​​ranging from 0 to 255. Then, the histogram() function is used to calculate the number of each pixel value. Finally, a grayscale histogram of the visible light image is obtained with the number of pixels as the ordinate and the pixel values ​​as the abscissa. Next, the grayscale histogram is thresholded, and fusion weights are adaptively set according to the thresholds. Finally, the base fused image and the extracted saliency features are fused using the adaptive fusion weights, which improves both the quality and target saliency of the fused image, thereby enhancing the performance of the visual task. The overall algorithm structure is shown in the diagram below. Figure 4 As shown.

[0073] To achieve saliency-preserving adaptive image fusion, a system can be employed, for example, where a dual-light camera is mounted on a rooftop, overlooking the monitored area. The dual-light camera is connected to a terminal processor via a network, transmitting the captured video stream to the terminal processor. The algorithm is then configured in the terminal processor, placed before the relevant visual task model, such as... Figure 5 As shown, this can enhance the performance of related visual task models.

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An adaptive image fusion method based on saliency preservation, characterized in that, Includes the following steps: S1. Acquire infrared and visible light images, preprocess the infrared images, align the infrared and visible light images in time and space, and perform unified normalization on the infrared and visible light images. S2. Obtain the grayscale histogram of the visible light image. Utilize the characteristic that the pixel range of the grayscale histogram can represent the brightness of the image. Divide the grayscale histogram into three pixel regions based on the segmentation threshold. Calculate the pixel count ratio of the three pixel regions (S1, S2, S3) according to the segmentation threshold. S3. Using the histogram-based LC algorithm, extract the target saliency of the preprocessed infrared image in step S1 to obtain a salient image; S4. Input the visible light image and the preprocessed infrared image from step S1 into the deep learning base fusion model to generate a base fusion image. The base fusion model includes an encoder and a decoder. The encoder is used to extract feature information from the input infrared and visible light images. The feature information of the input infrared image includes semantic structure features; the feature information of the input visible light image includes contour features and texture features. The encoder includes two paths: one path consists of a concatenated convolutional layer, a batch normalization layer, and an activation function to obtain the feature information of the input visible light image; the other path consists of a concatenated residual module and a concatenated convolutional layer to obtain the feature information of the input infrared image. The decoder includes a concatenated feature fusion layer and a convolutional layer to decode the obtained feature information of the infrared and visible light images to obtain the base fusion image. S5. Using the grayscale histogram of the visible light image and the pixel count ratios of the three pixel regions (S1, S2, S3), adaptive fusion of the salient image and the base fusion image is performed. The fusion weights are calculated using the following formula: Where 1>b1>b2>b3>0, and max(S1,S2,S3) represents the calculation of the maximum value of (S1,S2,S3); The adaptive fusion strategy is as follows: F = aI1 + bI2, a + b = 1 Where F represents the final fused image, I1 represents the base fused image features, a represents the fusion weight of I1, I2 represents the saliency features, and b represents the fusion weight of I2.

2. The adaptive image fusion method based on saliency preservation according to claim 1, characterized in that, In step S1, the method for preprocessing the infrared image includes noise filtering of the infrared image using a median filter operator and increasing the image contrast of the infrared image.

3. The adaptive image fusion method based on saliency preservation according to claim 1, characterized in that, In step S1, the grayscale of the noise-filtered infrared image is mapped to a new range using a nonlinear grayscale transformation operator, thereby increasing the image contrast.

4. The adaptive image fusion method based on saliency preservation according to claim 1, characterized in that, In step S2, the method for obtaining the grayscale histogram of the visible light image includes: using the OpenCV library in the Python environment, first converting the visible light image into a grayscale image with pixel values ​​of 0-255, then using the library function histogram() to calculate the number of each pixel value, and finally obtaining the grayscale histogram of the visible light image with the number as the vertical axis and the pixel value as the horizontal axis.

5. The adaptive image fusion method based on saliency preservation according to claim 1, characterized in that, In step S2, the formula for calculating the percentage of pixels in the three pixel regions is: Where, n i The value represents the number of pixels when the pixel value is i, N represents the total number of pixels in the grayscale histogram, and m1 and m2 represent the segmentation thresholds for the two pixel intervals.

6. The adaptive image fusion method based on saliency preservation according to claim 1, characterized in that, Step S3 includes the following steps: S31. Normalize the pixel feature values ​​of the preprocessed infrared image to [0, 255]; S32. Based on the grayscale histogram of the visible light image, classify and statistically analyze the normalized pixel feature values ​​to obtain the frequency f of the feature values. n , where n represents the normalized feature value of a pixel; S33. Calculate the Euclidean distance d between any two normalized pixel feature values ​​p and n. d = D(p,n), p ≠ n; S34. Based on the Euclidean distance d and the frequency of occurrence of eigenvalues ​​f n Calculate the target saliency S(p) of the preprocessed infrared image. S35. Normalize the target saliency S(p) to [0,255] to obtain the saliency image.

7. An adaptive image fusion system based on saliency preservation, characterized in that, It includes a dual-light camera capable of acquiring infrared and visible light images, and a terminal processor connected to the dual-light camera via a network. The terminal processor has a program written on it, which, when executed, can perform the adaptive image fusion method based on saliency preservation as described in any one of claims 1 to 6.

Images