Multispectral image fusion method, device, equipment and storage medium

By determining the environmental state based on the white light image in multi-spectral image fusion, NSST transformation and weighted fusion technology are used to solve the problem of difficulty in infrared and white light image fusion, achieving all-weather high-quality image fusion, improving image clarity and image quality .

CN118674640BActive Publication Date: 2024-11-15JIANGSU NORTH LAKE OPTOELECTRONICS CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411151384.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-15
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

The prior art is difficult to effectively fuse infrared images and white light images, especially in different environments (day and night) image information complementarity is limited, and the differences in physical characteristics of multi-spectral images lead to difficulty in fusion.

Method used

The environment state is determined by determining the brightness of the pixels in the white light image, switching to obtain infrared and visible light images, and using non-downsampled shear wave NSST transformation for wavelet decomposition, combining weighted least squares filtering and structural similarity analysis to perform weighting of high-frequency and low-frequency images Fusion, and finally generate the target fusion image through inverse NSST transformation.

Benefits of technology

It realizes complementary image information all-weather, improves image clarity and image quality, makes up for the problems of low definition and single image quality of images captured by a single camera, and maximizes the preservation of high-frequency beneficial signals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118674640B_ABST
    Figure CN118674640B_ABST
Patent Text Reader

Abstract

The present application discloses a multispectral image fusion method, device, equipment and storage medium, which relates to the field of image fusion. The current environmental state is determined according to the pixel brightness in the white light image, and two images to be fused are obtained in the corresponding state mode; the images to be fused are decomposed by NSST transformation respectively to obtain high-frequency and low-frequency images; the infrared low-frequency image is filtered to obtain an infrared saliency map; the infrared saliency map is weightedly fused with the low-frequency image of the visible light image to obtain a fused low-frequency image; the infrared high-frequency image and the visible light high-frequency image are subjected to structural similarity analysis, and weighted fusion is performed according to the structural similarity of the high-frequency image in each scale direction to obtain a fused high-frequency image; the fused high-frequency image and the fused low-frequency image are subjected to inverse NSST transformation to obtain a target fused image. The scheme can switch between different scenes and obtain spectral images, and use a fusion algorithm of multi-scale analysis to broaden the application scenarios and improve the clarity of the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image fusion, and in particular to a multispectral image fusion method, device, equipment and storage medium. Background Art

[0002] As multispectral images become more and more popular in industrial and military applications, their application scenarios are gradually integrated into life. However, due to the diversity of imaging mechanisms, spectral images of different bands have different image characteristics in different scenarios. For example, infrared images at night obtain richer image feature information than ordinary white light images. In order to tap the technical potential of more spectral images and realize the complementarity of multispectral image information, the commonly used technical means is image fusion.

[0003] Infrared images directly reflect the temperature distribution on the surface of an object, and provide clearer high-contrast images at night or in extremely dim light environments. White light images can obtain rich visible spectrum image data in daytime environments. Low-light CMOS can be used in night environments. Compared with infrared images, it has greater advantages in dealing with urban light pollution and strong light reflection. Multispectral images have different physical properties in different usage scenarios and climate environments. There are great difficulties in how to use multi-scale analysis algorithms to achieve multispectral image fusion between infrared images and white light images. Summary of the invention

[0004] The embodiments of the present application provide a multispectral image fusion method, device, equipment and storage medium to solve the problem that ordinary white light and infrared images have less image information and limited application scenarios.

[0005] On the one hand, the present application provides a multispectral image fusion method, the method comprising:

[0006] Determine the current environmental state according to the pixel brightness in the white light image, and switch to the corresponding state mode to obtain the first image to be fused and the second image to be fused; wherein the first image to be fused is an infrared image, and the second image to be fused is a visible light image;

[0007] The two images to be fused are decomposed by wavelet through non-subsampled shearlet NSST transform to obtain high-frequency image and low-frequency image of the target layer;

[0008] The low-frequency image corresponding to the infrared image is filtered using a weighted least squares filter to obtain an infrared saliency map; the infrared saliency map is weightedly fused with the low-frequency image of the visible light image to obtain a fused low-frequency image;

[0009] Performing structural similarity analysis on the infrared high-frequency image of the infrared image and the high-frequency image of visible light, and performing weighted fusion according to the structural similarities of the high-frequency images in various scale directions to obtain a fused high-frequency image;

[0010] The fused high-frequency image and the fused low-frequency image are subjected to inverse NSST transformation to obtain a target fused image.

[0011] Specifically, determining the current environmental state according to the pixel brightness in the white light image includes:

[0012] Extracting pixel information from the white light image and calculating the brightness mean of the image based on the HSV color space;

[0013] When the average brightness of the image is lower than the brightness threshold, it indicates that the current environment is dark, and the system switches to dark mode and acquires infrared images and low-light images.

[0014] When the average brightness of the image is not lower than the brightness threshold, it indicates that the current environment state is daytime, and the system switches to daytime mode and acquires infrared images and white light images.

[0015] Specifically, the wavelet decomposition is performed by non-subsampled shearlet NSST transform to obtain a high-frequency image and a low-frequency image of a target layer, including:

[0016] Perform k-times pyramid NSP decomposition on the two images to be fused respectively to obtain a low-frequency background layer and k high-frequency detail layers;

[0017] The low-frequency background layer and the high-frequency detail layer are respectively transformed into a trapezoidal support interval by geometric affine transformation, and the support base slope of the trapezoidal support interval is gradually changed to obtain high-frequency sub-bands and low-frequency sub-bands in different directions;

[0018] A low-frequency image is generated based on the low-frequency subbands of the low-frequency background layer, and a high-frequency image is generated based on the high-frequency subbands of the k high-frequency detail layers.

[0019] Specifically, filtering the infrared low-frequency image of the infrared image using a weighted least squares filter and obtaining an infrared saliency map includes:

[0020] The infrared low-frequency image I is smoothed and filtered according to the following formula to obtain a smoothed filtered image U:

[0021]

[0022] The subscript p indicates the pixel position. is the regularization parameter, , is the smoothing weight coefficient; and Respectively represent the pixels at point p before and after filtering;

[0023] The smoothed filtered image U is processed in grayscale to obtain a grayscale image ;

[0024] Use weighted least squares filter to filter the grayscale image Filter the original infrared low-frequency image I to obtain the infrared saliency map ; means as follows:

[0025]

[0026] Among them Indicates in ( ) point, Infrared low frequency image I exist( ) is the mean value of pixels at the point.

[0027] Specifically, after obtaining the infrared saliency map, the method further includes:

[0028] Infrared saliency map After normalization, the normalized infrared saliency map is obtained, which is expressed as follows:

[0029]

[0030] Among them for The minimum gray value in ; for The maximum gray value in ;

[0031] The step of weightedly fusing the infrared saliency map with the low-frequency image of the visible light image comprises:

[0032] Use the following low frequency fusion rule formula:

[0033]

[0034] Among them represents the fusion of low-frequency images, and are the infrared and visible light weight coefficients, respectively. and Represent the pixels of the infrared saliency map and the visible light low-frequency map at point (x, y) respectively.

[0035] Specifically, performing structural similarity analysis on the infrared high-frequency image of the infrared image and the high-frequency image of the visible light includes:

[0036] The structural similarity of the high-frequency part in the first direction of the k-th scale high-frequency image of infrared light and visible light is calculated as follows:

[0037]

[0038] Among them is the structural similarity of the two high-frequency images in each scale and direction, and is the local mean of infrared and visible light high-frequency images, and is the standard deviation of infrared and visible high-frequency images, is the cross-covariance of infrared and visible high-frequency images; and is a constant.

[0039] Specifically, performing weighted fusion according to the structural similarity of the high-frequency images in each scale direction to obtain a fused high-frequency image includes:

[0040] Calculate the regional energy of high-frequency sub-bands in infrared and visible high-frequency images , which is expressed as follows:

[0041]

[0042] in, represents the regional energy at (x, y), M and N are the height and width of the regional window, is the window mask matrix, Represents different components in the high frequency band;

[0043] when When it is less than the threshold α, the fusion rule adopts the method of taking the larger regional energy, which is expressed as:

[0044]

[0045] when When it is not less than the threshold α, the fusion rule adopts the weighted average method, which is expressed as:

[0046]

[0047]

[0048]

[0049] in, is the matching threshold, and is the weight coefficient, and Represents the infrared and visible light images in The regional energy value at To fuse high frequency images.

[0050] On the other hand, the present application provides a multispectral image fusion device, the device comprising:

[0051] An image acquisition module is used to determine the current environmental state according to the pixel brightness in the white light image, and switch to the corresponding state mode to acquire the first image to be fused and the second image to be fused; wherein the first image to be fused is an infrared image, and the second image to be fused is a visible light image;

[0052] The image decomposition module is used to perform wavelet decomposition on the two images to be fused through non-subsampled shearlet NSST transform to obtain high-frequency image and low-frequency image of the target layer;

[0053] An image fusion module is used to filter the low-frequency image corresponding to the infrared image using a weighted least squares filter and obtain an infrared saliency map; and to perform weighted fusion of the infrared saliency map with the low-frequency image of the visible light image to obtain a fused low-frequency image;

[0054] Performing structural similarity analysis on the infrared high-frequency image of the infrared image and the high-frequency image of visible light, and performing weighted fusion according to the structural similarities of the high-frequency images in various scale directions to obtain a fused high-frequency image;

[0055] The target acquisition module is used to perform inverse NSST transformation on the fused high-frequency image and the fused low-frequency image to obtain a target fused image.

[0056] On the other hand, the present application provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the multispectral image fusion method described in the above aspect.

[0057] On the other hand, the present application provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the multispectral image fusion method described in the above aspect.

[0058] The beneficial effects of the technical solution provided by the embodiment of the present application include at least: judging the environmental state according to the captured white light image, and then adaptively switching the camera device according to the state to obtain the image to be fused, so that the system can obtain suitable images for fusion and complementation around the clock. As for image fusion, a multi-scale analysis algorithm is used to decompose the image to be fused in multiple scales. After decomposition, the steps of high-frequency fusion and low-frequency fusion are performed. The low frequency is filtered by the least squares method and the infrared significant feature map is used for fusion. The high frequency uses structural similarity calculation to evaluate each sub-band and then weighted fusion to retain the high-frequency beneficial signal to the maximum extent. Finally, the high-frequency and low-frequency fused images are transformed by inverse NSST to obtain a target fused image with higher image quality. Compared with traditional image fusion, it can remove noise and retain more useful information, and the image clarity formed is higher, which makes up for the problem of low image clarity and single image quality of the image taken by a single camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of a multispectral image fusion method provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram showing a possible mode switching and image fusion;

[0061] Figure 3 An algorithm flow chart of a multispectral image fusion method is shown;

[0062] Figure 4 It is a schematic diagram of using HSV color space to judge the state of the environment;

[0063] Figure 5 It is a schematic diagram of the trapezoidal support interval of NSST decomposition;

[0064] Figure 6 It is a schematic diagram of the NSST image decomposition process;

[0065] Figure 7 This is a structural block diagram of a multispectral image fusion device provided in this application. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0067] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0068] Figure 1 : is a flow chart of the multispectral image fusion method provided in the embodiment of the present application, which specifically includes the following steps:

[0069] S1, determining the current environmental state according to the pixel brightness in the white light image, and switching to the corresponding state mode to obtain the first image to be fused and the second image to be fused;

[0070] The system used in the embodiment of the present application includes at least three image acquisition devices, namely a device for acquiring white light images, a device for acquiring low-light images, and a device for acquiring infrared images. In some multi-device systems, an infrared camera can be used to capture infrared images, a low-light micro-CMOS can be used to capture low-light images, and a color white light CCD camera can be used to capture white light images.

[0071] The image fusion in this scheme is the first image to be fused and the second image to be fused, and the two images to be fused are determined according to the environmental state, that is, the daytime and nighttime state modes. The ordinary white light image acquired in the daytime state has more spectral information, but in the night, the white light image cannot obtain enough useful image information. At this time, it is necessary to switch the mode and use the corresponding camera equipment to obtain the image for compensation fusion.

[0072] Figure 2 A schematic diagram of a possible mode switching and image fusion is shown. First, a (color) white light image is obtained, and the current environment brightness is determined according to the picture pixel brightness to judge the environment state. When the detection determines that it is a daytime state, the system is switched to the day mode, and then the first and second images to be fused are obtained through the corresponding equipment. At this time, the first image to be fused is an infrared image, and the second image to be fused is a (color) white light image. When the detection determines that it is a night state, the system is switched to the night mode, and then the first and second images to be fused are obtained through the corresponding equipment. At this time, the first image to be fused is an infrared image, and the second image to be fused is a low-light image. The low-light image and the (color) white light image here are both images acquired in the visible light band.

[0073] In some all-weather image capture systems, the white light images acquired during the day can clearly display the captured images, and there is no need to use an infrared camera to capture infrared images, so there is no need to fuse the images. Only in the dark will the mode be switched additionally, that is, the white light images are used to determine the environment, and then the infrared camera and low-light CMOS camera are enabled to respectively acquire infrared images and low-light images, and the two images are fused.

[0074] S2, the two images to be fused are decomposed by wavelet through NSST transformation to obtain the high-frequency image and low-frequency image of the target layer;

[0075] For the two selected images to be fused, the traditional image fusion cannot maximize the use of available information, but will introduce some noise, resulting in worse image quality. This embodiment uses the latest multi-scale analysis algorithm, namely the non-subsampled shearlet transform technology, for multi-spectral image fusion. The two types of images are decomposed separately, and the original spectrum image is split into high-frequency image and low-frequency image according to the high-frequency component and the low-frequency component, that is, the infrared image is decomposed into infrared high-frequency image and infrared low-frequency image, and the visible light image is decomposed into visible light high-frequency image and visible light low-frequency image. Superposition and fusion according to spectral components can perform targeted filtering and feature fusion, effectively suppress measurement noise, and thus improve image quality.

[0076] S3, filtering the low-frequency image corresponding to the infrared image using a weighted least squares filter, and obtaining an infrared saliency map; weighted fusion of the infrared saliency map and the low-frequency image of the visible light image to obtain a fused low-frequency image;

[0077] This step mainly involves classifying and fusing the two decomposed high- and low-frequency images, namely, fusing the infrared high-frequency image with the visible light high-frequency image, and fusing the infrared low-frequency image with the visible light low-frequency image. Considering the low-frequency characteristics, the low-frequency subband is an approximate component of the source image, which contains a large amount of available information. Therefore, the fusion result of the low-frequency image plays a decisive role in the fusion effect of the entire image. In order to fully highlight the infrared target features and retain the detailed information of the visible light image, this scheme uses the saliency detection algorithm to extract the saliency map of the infrared image to guide the fusion of the low-frequency coefficients.

[0078] In order to filter out noise and maintain the edge characteristics of the image, the least squares large filter is designed and filtered based on the filtering principle of the weighted least squares method to obtain a smoother low-frequency image. Then, the saliency map of the infrared image is extracted based on the filtered image using the saliency detection algorithm, and the infrared saliency map is weightedly fused with the low-frequency image of the visible light image.

[0079] S4, performing structural similarity analysis on the infrared high-frequency image of the infrared image and the high-frequency image of the visible light, performing weighted fusion according to the structural similarity of the high-frequency images in various scale directions, and obtaining a fused high-frequency image;

[0080] For high-frequency images, considering the structural similarity of the high-frequency signals of the images, it is necessary to perform similarity analysis on the images. This process mainly performs similarity analysis on the high-frequency sub-bands after NSST decomposition, and selectively performs weighted fusion of the high-frequency signals according to the similarity of the high-frequency sub-bands to retain the high-frequency beneficial signal sub-bands to the maximum extent, thus obtaining a high-quality fused high-frequency image.

[0081] It is particularly noted that the high-frequency fusion of step S4 and the low-frequency fusion of step S3 do not strictly distinguish the order of steps and can be performed synchronously or asynchronously.

[0082] S5, performing inverse NSST transformation on the fused high-frequency image and the fused low-frequency image to obtain a target fused image.

[0083] Figure 3 The algorithm flow chart of the multispectral image fusion method is shown. The low-frequency and high-frequency components are fused after weighted average setting coefficients. The difference is that the infrared low-frequency component needs to be filtered and the saliency map is extracted, while the visible light low-frequency component is directly fused with the infrared saliency map. For the high-frequency and low-frequency fused images, the high and low frequency components are finally transformed by inverse NSST to obtain the target fused image.

[0084] In summary, the present application judges the state of the environment based on the captured white light image, and then adaptively switches the camera device according to the state to obtain the image to be fused, so that the system can obtain suitable images for fusion and complementation around the clock. As for image fusion, a multi-scale analysis algorithm is used to decompose the image to be fused at multiple scales. After decomposition, the steps of high-frequency fusion and low-frequency fusion are performed. The low frequency is filtered by the least squares method and the infrared significant feature map is used for fusion. The high frequency uses structural similarity calculation to evaluate each sub-band and then weighted fusion to maximize the retention of high-frequency beneficial signals. Finally, the high-frequency and low-frequency fused images are transformed by inverse NSST to obtain a target fused image with higher image quality. Compared with traditional image fusion, noise can be removed and more useful information can be retained, and the image clarity formed is higher, which makes up for the problem of low image clarity and single image quality taken by a single camera.

[0085] In some embodiments, a light sensor can be introduced into the system to determine the ambient brightness based on the sensor detection value, so as to switch the lens module and shooting mode. When the ambient state is determined by using a white light image, it can be determined in the HSV color space.

[0086] Figure 4 It is a schematic diagram of using the HSV color space to judge the environmental state. First, the pixel information in the white light image is extracted, and the average brightness of the image is calculated based on the HSV color space. When the average brightness of the image is lower than the brightness threshold, it indicates that the current environmental state is night, and the mode is switched to night mode and infrared images and low-light images are acquired. When the average brightness of the image is not lower than the brightness threshold, it indicates that the current environmental state is daytime, and the mode is switched to daytime mode and infrared images and white light images are acquired. In the embodiment of the present application, the selected brightness threshold is 130, that is, when the value exceeds 130, it is switched to daytime shooting mode, and when it is lower than 130, it is switched to night shooting mode, and then according to the value Figure 2The corresponding camera module is started in the mode to acquire images and perform image fusion.

[0087] Regarding the NSST decomposition step in the above step S2, the core principle is to decompose the original image into high- and low-frequency detail information of the target scale. The NSST image decomposition transformation is mainly divided into two steps: pyramid decomposition (NSP) and shearlet local directional filtering (SF). The specific process includes the following:

[0088] A, perform k-times pyramid NSP decomposition on the two images to be fused to obtain a low-frequency background layer and k high-frequency detail layers;

[0089] In this step, the original image (image to be fused) is decomposed by NSP k times to obtain a low-frequency background layer and k high-frequency detail layers. The size of each frequency band is consistent with that of the source image.

[0090] B. The low-frequency background layer and the high-frequency detail layer are transformed into a trapezoidal support interval through geometric affine transformation, and the support base slope of the trapezoidal support interval is changed successively to obtain high-frequency sub-bands and low-frequency sub-bands in different directions;

[0091] Figure 5 is a schematic diagram of the trapezoidal support interval of NSST decomposition. The directional decomposition of NSST is realized by using shearlet local directional filtering. Through geometric affine transformation, we get Figure 5 By changing the support base slope, we can get support base intervals in different directions. Compared with the square support interval of wavelet transform, direction extraction in the trapezoidal support interval can better approximate the curve.

[0092] The theory of non-subsampled shearlet transform (NSST) is derived from Shearlet transform. The traditional Shearlet transform performs directional filtering by windowing and translating in the pseudo-Polar coordinate system, which requires downsampling of the data. The transformation is translation-variable and prone to drift. NSST avoids the downsampling operation by mapping the shearing filter from the pseudo-Polar coordinate system to the Cartesian coordinate system and then filtering the image. Assuming that the scale is j, the decomposition process can be seen in Figure 6 As shown, the specifics can be summarized as follows:

[0093] (1) Using NSP to image Perform multi-scale decomposition to obtain the low-frequency background layer and high frequency details ;

[0094] (2) For the high frequency coefficients , shearlet filtering is used to extract information in different directions;

[0095] ①Construct the filter W in the pseudo-Polar coordinate system;

[0096] ② Transform the filter W from pseudo-Polar coordinates to Cartesian coordinates to obtain the filter W';

[0097] ③Calculate the high frequency part Fourier transform of and multiply with the window filter;

[0098] ④ Performing inverse Fourier transform can obtain high-frequency coefficients in l directions;

[0099] ⑤Continue to change the slope direction of the filter, and use the same method to extract high-frequency coefficients in all directions;

[0100] C, a low-frequency image is generated based on the low-frequency sub-bands of the low-frequency background layer, and a high-frequency image is generated based on the high-frequency sub-bands of the k high-frequency detail layers.

[0101] The classic FT algorithm uses the Gaussian difference bandpass filtering method, the Gaussian kernel size is selected as 5×5, and the frequency range is (0, / 2.75], the formula for obtaining the infrared image saliency map is defined as:

[0102]

[0103] In the formula, is the average value of the original image (infrared low-frequency image). is the original image I after Gaussian blur ( ) to eliminate fine texture details and noise. This means that the sum of the squares of the differences between the pixel values ​​at each point and the mean after Gaussian blur filtering at each color channel in the input original image I is obtained. However, the above-mentioned classic FT algorithm is prone to cause problems such as image edge blur and loss of detail information when performing Gaussian blur filtering. In order to improve the quality of the saliency map, this embodiment uses a weighted least squares filter with strong edge preservation and detail enhancement capabilities to replace the Gaussian blur filter in the classic FT algorithm to form an improved FT algorithm to obtain the saliency map of the infrared image. For this purpose, this application is aimed at Figure 3 The low- and medium-frequency fusion step obtains the infrared saliency map, and the steps include the following:

[0104] A, smooth the infrared low-frequency image I according to the following formula to obtain the smoothed filtered image U:

[0105]

[0106] The subscript p indicates the pixel position. is the regularization parameter, , is the smoothing weight coefficient; and Respectively represent the pixels at point p before and after filtering; The role of smoothing the filtered image U is to make the input image original image I More similar.

[0107] B, grayscale the smoothed filtered image U to obtain a grayscale image ;

[0108] C, use weighted least squares filter to filter the grayscale image and the original infrared low-frequency image I Filter to obtain the infrared saliency map ; means as follows:

[0109]

[0110] Among them Indicates in ( ) point, Infrared low frequency image I exist( ) is the mean value of pixels at the point.

[0111] In order to make the image fusion process smoother and more natural, after obtaining the infrared saliency map After that, it can also be normalized to obtain the normalized infrared saliency map , which is expressed as follows:

[0112]

[0113] Among them for The minimum gray value in ; for The maximum gray value in .

[0114] D. Weighted fusion of the infrared saliency map and the low-frequency image of the visible light image using the following low-frequency fusion rule formula:

[0115]

[0116] Among them represents the fusion of low-frequency images, and are the infrared and visible light weight coefficients, respectively. and Respectively represent the infrared saliency map and the visible light low frequency map in ( x,y ) pixels. In particular, the coefficient and The relationship is expressed as follows:

[0117]

[0118] In some embodiments, the high-frequency subband represents the high-frequency component of the source image. The common rule of taking the largest absolute value easily causes the fused image to be mixed with a lot of invalid information. In order to make the fused image more in line with the visual characteristics of the human eye, the present application uses the rule of structural similarity combined with regional energy to fuse the coefficients of the high-frequency part. The high-frequency fusion process in step S4 specifically includes the following steps:

[0119] A, calculate the structural similarity of the high-frequency part in the first direction of the k-th scale high-frequency image of infrared light and visible light, which is expressed as follows:

[0120]

[0121] Among them is the structural similarity of the two high-frequency images in each scale and direction, and is the local mean of infrared and visible light high-frequency images, and is the standard deviation of infrared and visible high-frequency images, is the cross-covariance of infrared and visible high-frequency images; and is a constant to prevent the ratio of the formula from being infinite.

[0122] B. Calculate the regional energy of high-frequency sub-bands in infrared and visible light high-frequency images , which is expressed as follows:

[0123]

[0124] in, Indicates the energy size of the high-frequency region at (x, y), M and N are the height and width of the region window, indicating that the region window size is (2M+1)×(2N+1), is the window mask matrix. The closer to the center point, the larger its weight is. Represents different components in the high frequency band.

[0125] C, when When it is less than the threshold α, the fusion rule adopts the method of taking the larger regional energy, which is expressed as:

[0126]

[0127] D, when When it is not less than the threshold α, the fusion rule adopts the weighted average method, which is expressed as:

[0128]

[0129]

[0130]

[0131] in, is the matching threshold, and is the weight coefficient, and Represents the infrared and visible light images in The regional energy value at To fuse high frequency images.

[0132] After the above steps, the fusion process of high-frequency and low-frequency images can be completed. After that, the two images are inversely transformed by NSST to obtain the target fused image.

[0133] In summary, the beneficial effects of the multispectral fusion algorithm provided by this application are:

[0134] It combines ordinary visible light images, infrared images and low-light images, and has the ability to identify day and night. It switches the video source of image fusion at different time periods during the day and night, fully considering the different physical characteristics of multi-spectral images to achieve the best fusion effect;

[0135] The latest multi-scale analysis algorithm, non-subsampled shearlet transform (NSST), is used for multispectral image fusion;

[0136] Different fusion algorithms are used for high and low frequency components. The improved weighted least squares filter is used to extract the infrared saliency map, which is used to guide the low-frequency fusion with other images. In terms of high-frequency fusion, considering the structural similarity of the high-frequency signals of the image, the regional energy maximum method and weighted average method are used to realize the fusion of high-frequency signals, so as to retain the high-frequency beneficial signals to the maximum extent.

[0137] Figure 7 This is a structural block diagram of a multispectral image fusion device provided by the present application, the device comprising:

[0138] The image acquisition module 701 is used to determine the current environment state according to the pixel brightness in the white light image, and switch to the corresponding state mode to acquire the first image to be fused and the second image to be fused; wherein the first image to be fused is an infrared image, and the second image to be fused is a visible light image;

[0139] An image decomposition module 702 is used to perform wavelet decomposition on the two images to be fused by non-subsampled shearlet NSST transform to obtain a high-frequency image and a low-frequency image of a target layer;

[0140] The image fusion module 703 is used to filter the low-frequency image corresponding to the infrared image using a weighted least squares filter to obtain an infrared saliency map; and to perform weighted fusion of the infrared saliency map with the low-frequency image of the visible light image to obtain a fused low-frequency image;

[0141] Performing structural similarity analysis on the infrared high-frequency image of the infrared image and the high-frequency image of visible light, and performing weighted fusion according to the structural similarities of the high-frequency images in various scale directions to obtain a fused high-frequency image;

[0142] The target acquisition module 704 is used to perform an inverse NSST transformation on the fused high-frequency image and the fused low-frequency image to obtain a target fused image.

[0143] Furthermore, the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the multispectral image fusion method described in any of the above embodiments.

[0144] Furthermore, the present application also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the multispectral image fusion method described in any of the above embodiments.

[0145] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A multispectral image fusion method, characterized in that: The method comprises: Determine the current environmental state according to the pixel brightness in the white light image, and switch to the corresponding state mode to obtain the first image to be fused and the second image to be fused; wherein the first image to be fused is an infrared image, and the second image to be fused is a visible light image; The two images to be fused are decomposed by wavelet through non-subsampled shearlet NSST transform to obtain high-frequency image and low-frequency image of the target layer; The low-frequency image corresponding to the infrared image is filtered using a weighted least squares filter to obtain an infrared saliency map; the infrared saliency map and the low-frequency image of the visible light image are weighted fused according to a low-frequency fusion rule to obtain a fused low-frequency image; the formula of the low-frequency fusion rule is as follows: Among them represents the fusion of low-frequency images, and are the infrared and visible light weight coefficients, respectively. and Respectively represent the infrared saliency map and the visible light low-frequency image in ( x,y ) pixels; The infrared high-frequency image of the infrared image and the high-frequency image of the visible light are subjected to structural similarity analysis, and weighted fusion is performed according to the structural similarity of the high-frequency images in various scale directions to obtain a fused high-frequency image; specifically, the regional energy of the high-frequency sub-bands in the infrared and visible light high-frequency images is calculated. , which is expressed as follows: in, Indicates in ( x,y ), M and N are the regional window height and width, is the window mask matrix, k is the high-frequency detail layer, To represent different components in the high frequency band; Indicated in The pixel value at ; when When it is less than the threshold α, the fusion rule adopts the method of taking the larger regional energy, which is expressed as: when When it is not less than the threshold α, the fusion rule adopts the weighted average method, which is expressed as: in Indicates the structural similarity of two high-frequency images in each scale and direction. is the matching threshold, and is the weight coefficient, and Represents the infrared and visible light images in The regional energy value at To fuse high-frequency images; The fused high-frequency image and the fused low-frequency image are subjected to inverse NSST transformation to obtain a target fused image.

2. The method according to claim 1, characterized in that Determining the current environmental state according to the pixel brightness in the white light image includes: Extracting pixel information from the white light image and calculating the brightness mean of the image based on the HSV color space; When the average brightness of the image is lower than the brightness threshold, it indicates that the current environment is dark, and the system switches to dark mode and acquires infrared images and low-light images. When the average brightness of the image is not lower than the brightness threshold, it indicates that the current environment state is daytime, and the system switches to daytime mode and acquires infrared images and white light images.

3. The method according to claim 1, characterized in that The wavelet decomposition is performed by non-subsampled shearlet NSST transform to obtain a high-frequency image and a low-frequency image of a target layer, including: Perform k-times pyramid NSP decomposition on the two images to be fused respectively to obtain a low-frequency background layer and k high-frequency detail layers; The low-frequency background layer and the high-frequency detail layer are respectively transformed into a trapezoidal support interval by geometric affine transformation, and the support base slope of the trapezoidal support interval is gradually changed to obtain high-frequency sub-bands and low-frequency sub-bands in different directions; A low-frequency image is generated based on the low-frequency subbands of the low-frequency background layer, and a high-frequency image is generated based on the high-frequency subbands of the k high-frequency detail layers.

4. The method according to claim 2, characterized in that: The step of filtering the infrared low-frequency image of the infrared image using a weighted least squares filter and obtaining an infrared saliency map comprises: The infrared low frequency image I According to the following formula, smooth filtering is performed to obtain the smoothed filtered image U: The subscript p indicates the pixel position. is the regularization parameter, , is the smoothing weight coefficient; and Respectively represent the pixels at point p before and after filtering; The smoothed filtered image U is processed in grayscale to obtain a grayscale image ; Use weighted least squares filter to filter the grayscale image and the original infrared low-frequency image I Filter to obtain the infrared saliency map ; means as follows: Among them Indicates in ( ) point, Infrared low frequency image I exist( ) is the mean value of pixels at the point.

5. The method according to claim 3, characterized in that: After acquiring the infrared saliency map, the method further includes: Infrared saliency map After normalization, the normalized infrared saliency map is obtained, which is expressed as follows: Among them for The minimum gray value in ; for The maximum gray value in .

6. The method according to claim 5, characterized in that The step of performing structural similarity analysis on the infrared high-frequency image of the infrared image and the high-frequency image of the visible light includes: The structural similarity of the high-frequency part in the first direction of the k-th scale high-frequency image of infrared light and visible light is calculated as follows: Among them is the structural similarity of the two high-frequency images in each scale and direction, and is the local mean of infrared and visible light high-frequency images, and is the standard deviation of infrared and visible high-frequency images, is the cross-covariance of infrared and visible high-frequency images; and is a constant.

7. A multispectral image fusion device, characterized in that: The device comprises: An image acquisition module is used to determine the current environmental state according to the pixel brightness in the white light image, and switch to the corresponding state mode to acquire the first image to be fused and the second image to be fused; wherein the first image to be fused is an infrared image, and the second image to be fused is a visible light image; The image decomposition module is used to perform wavelet decomposition on the two images to be fused through non-subsampled shearlet NSST transform to obtain high-frequency image and low-frequency image of the target layer; The image fusion module is used to filter the low-frequency image corresponding to the infrared image using a weighted least squares filter and obtain an infrared saliency map; the infrared saliency map and the low-frequency image of the visible light image are weightedly fused according to a low-frequency fusion rule to obtain a fused low-frequency image; the formula of the low-frequency fusion rule is as follows: Among them represents the fusion of low-frequency images, and are the infrared and visible light weight coefficients, respectively. and Respectively represent the infrared saliency map and the visible light low-frequency image in ( x,y ) pixels; The infrared high-frequency image of the infrared image and the high-frequency image of the visible light are subjected to structural similarity analysis, and weighted fusion is performed according to the structural similarity of the high-frequency images in various scale directions to obtain a fused high-frequency image; specifically, the regional energy of the high-frequency sub-bands in the infrared and visible light high-frequency images is calculated. , which is expressed as follows: in, Indicates in ( x,y ), M and N are the regional window height and width, is the window mask matrix, k is the high-frequency detail layer, To represent different components in the high frequency band; Indicated in The pixel value at ; when When it is less than the threshold α, the fusion rule adopts the method of taking the larger regional energy, which is expressed as: when When it is not less than the threshold α, the fusion rule adopts the weighted average method, which is expressed as: in Indicates the structural similarity of two high-frequency images in each scale and direction. is the matching threshold, and is the weight coefficient, and Represents the infrared and visible light images in The regional energy value at To fuse high-frequency images; The target acquisition module is used to perform inverse NSST transformation on the fused high-frequency image and the fused low-frequency image to obtain a target fused image.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the multispectral image fusion method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the multispectral image fusion method as described in any one of claims 1 to 6.

Citation Information

Cited By

  • Single-chip spatio-temporal synchronous hdr domain dual-band image adaptive fusion device and method

    CN122928645A