Image fusion method and system based on frequency band coefficient decomposition

Abstract

The present invention discloses an image fusion method and system based on frequency band coefficient decomposition. The innovation of the image fusion method based on frequency band coefficient decomposition of the present invention and the previous image fusion methods mainly lies in the sub-band coefficient fusion rule, which includes that the low-frequency part first adopts the fusion rule based on local directional energy and matching, and then adopts the fusion rule based on image significance to fuse the low-frequency sub-band coefficients; the high-frequency part adopts the fusion rule based on local directional contrast in the highest decomposition layer, and the fusion rule based on local energy matching in other decomposition layers to fuse the high-frequency bandpass directional sub-band coefficients. The image fusion system based on frequency band coefficient decomposition of the present invention has a simple structure and is easy to install. The final fused image information is relatively accurate and is not affected by meteorological conditions. It not only retains the significant target features of infrared images, but also retains the clear background detail information of visible light images.

Description

Technical Field

[0001] The present invention relates to the field of vehicle driving road condition image information acquisition and processing, and in particular to an image fusion method and system based on frequency band coefficient decomposition. Background Art

[0002] As an important branch of multi-source sensor information fusion technology, image fusion technology first appeared in the late 1970s. In 1979, Daily et al. fused radar images with Landsat-MSS images; in 1985, Bonn et al. fused multispectral remote sensing images with high-resolution images from SPOT satellites. However, these fusions are based on a hierarchical spatial domain, directly selecting or weighted averaging the pixel values ​​of the image, resulting in low signal-to-noise ratio and unclear features.

[0003] Subsequently, multiscale analysis tools were introduced into the field of image fusion. In 1984, Burt et al. first proposed a single-pixel image fusion method based on Laplace pyramid decomposition; in 1985, Li et al. proposed an image fusion method based on contrast pyramid to fuse infrared images and SAR images. In the 1990s, wavelet transform was applied to the field of image fusion, and further, Ranchin et al. proposed an image fusion method based on discrete wavelet transform. Wavelet transform is effective in capturing zero-dimensional singular points, but it is inefficient when processing two-dimensional images and has limitations in direction. To this end, researchers proposed multiscale transform tools such as Contourlet and Shearlet to achieve two-dimensional transformation of images. In 2002, Do et al. proposed Contourlet transform, and in 2006, ALCunha et al. proposed non-subsampled Contourlet transform, which has the advantages of multi-scale, directional flexibility and translation invariance. In 2005, Cunha et al. proposed Shearlet transform, and in 2008, Easley et al. proposed non-subsampled Shearlet transform, which has the advantages of non-subsampled Contourlet transform and higher computational efficiency. In 2010, Yang et al. introduced sparse representation into the field of image fusion. Subsequently, Liu et al. proposed an image fusion method combining multi-scale decomposition and sparse representation models. In recent years, deep neural networks have been introduced into the field of image fusion. In 2017, Liu Yu et al. first applied convolutional neural networks to multi-focus image fusion; Ma et al. used adversarial neural networks to fuse infrared and visible light images; in 2020, Zhang et al. proposed an image fusion model based on CNN.

[0004] The research on infrared and visible light image fusion began in the 1990s. It was first applied in the military field and installed on fighter jets to enhance the aircraft's all-weather combat capability. After 2000, infrared and visible light image fusion was applied to the security field. OCTCE in the UK and Lehigh University in the United States developed infrared and visible light fusion systems. Subsequently, infrared solutions in the United States developed commercial infrared and visible light image fusion cameras, and infrared and visible light image fusion began to enter the civilian field.

[0005] At present, there are many algorithms for infrared and visible light image fusion, such as multi-scale transformation, sparse representation, deep neural network, etc. Among them, the method based on sparse representation cannot extract information at multiple scales, which is easy to cause information loss; the method based on deep neural network can process a large amount of information, but the calculation amount is large and the calculation speed is restricted; the multi-scale transformation has a simple structure and high calculation efficiency. Among them, the non-subsampled Shearlet transform has multi-scale, directional flexibility, translation invariance and high calculation speed, and is widely used in life. However, this method still has problems such as incomplete image information and poor fusion effect of fused images in clarity and resolution. Therefore, the present invention proposes an image fusion method and system based on frequency band coefficient decomposition, which is installed on the car to output a fused image with a higher quality level, which is used to improve the visual observation level of the driver and avoid traffic accidents. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide an image fusion method and system based on frequency band coefficient decomposition, so as to solve the defects of the existing single sensor that obtains insufficient information and has low accuracy, and the existing multi-source sensor fusion effect is not significant and there is serious information loss.

[0007] The present invention solves the technical problem by adopting the following technical solutions:

[0008] The image fusion system based on frequency band coefficient decomposition provided by the present invention comprises two modules, wherein the first module is an image acquisition module, which is provided with an infrared thermal imager and a visible light camera; the second module is an image fusion module, which is provided with a vehicle-mounted computer; the two modules are connected via a network cable.

[0009] The infrared thermal imager collects infrared images ir, and the visible light camera collects visible light images vis. The collected images are transmitted to the on-board computer via a network cable, and image fusion based on frequency band coefficient decomposition is completed in the on-board computer to display the fused image F.

[0010] The image fusion system based on frequency band coefficient decomposition provided by the present invention updates the fused image F in real time according to a set time interval of 10-15 ms.

[0011] The image fusion method based on frequency band coefficient decomposition provided by the present invention is specifically as follows: first, the infrared image ir and the visible light image vis are decomposed by NSST frequency band coefficients to obtain low-frequency sub-bands and high-frequency band pass direction sub-bands, wherein: the low-frequency sub-bands first adopt a fusion rule based on local directional energy and matching, and then adopt a fusion rule based on image significance to fuse the low-frequency sub-band coefficients; the high-frequency band pass direction sub-bands adopt a fusion rule based on local directional contrast at the highest decomposition layer, and other decomposition layers use a fusion rule based on local energy matching to fuse the high-frequency band pass direction sub-band coefficients. Then, the above fused coefficients are inversely transformed to obtain a fused image F.

[0012] In the above method, the low-frequency sub-band coefficients can be firstly fused based on local directional energy and matching, and then fused based on image saliency. The calculation formula of the low-frequency sub-band coefficients of the obtained fused image F at the pixel point (m, n) is:

[0013]

[0014] Where: C F (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the fused image F, C ir (m,n) is the low-frequency subband coefficient at the infrared image pixel (m,n), C vis (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the visible light image, G ir (m,n) is the local directional energy sum at the infrared image pixel point (m,n), G vis (m,n) is the local directional energy sum at the pixel point (m,n) of the visible light image, M G (m,n) is the local directional energy and matching degree at the pixel point (m,n), W X (m,n) is the fusion weighting coefficient of the low-frequency subband coefficient at the pixel point (m,n) of the source image X, γ is the local directional energy and matching threshold, and γ=0.7 is taken, X=ir or vis, ir is the infrared image, and vis is the visible light image.

[0015] In the above method, when M G When (m,n)≤γ, a fusion rule based on local directional energy and matching can be adopted for the low-frequency subband coefficients, which specifically includes the following steps:

[0016] Step 1. Calculate the local directional energy of the source image pixel point X (m, n) in the horizontal H, vertical V and diagonal D directions respectively. The calculation formula is:

[0017]

[0018] Where: I X (m,n) is the gray value of the pixel (m,n) of the source image X. Similarly, I X (m,n+z),I X (m+z,n),I X (m+z,n+z) is the grayscale value of the corresponding image pixel, and the value of z is -1, 1, -2, 2. is the local directional energy of the source image pixel point X (m, n) in the horizontal H, vertical V and diagonal D directions, X = ir or vis, ir is the infrared image, vis is the visible light image;

[0019] Step 2, calculate the local directional energy and the calculation formula is:

[0020]

[0021] Where: the selected local area is recorded as A, the size is a×a, w(m,n) is the weighting matrix corresponding to the coefficient matrix, is the local directional energy of the source image pixel X (m, n) in the horizontal H, vertical V and diagonal D directions, AG X (m,n) is the local directional energy sum of the pixel point (m,n) of the source image X, X = ir or vis, ir is the infrared image, vis is the visible light image;

[0022] Step 3. Calculate the cross-local directional energy in the horizontal H, vertical V and diagonal D directions of the pixel point (m, n) of the infrared and visible light images. The calculation formula is:

[0023]

[0024] Where: I ir (m,n) is the gray value of the pixel (m,n) of the infrared image ir, I vis (m,n) is the gray value of the pixel (m,n) of the visible light image vis. Similarly, I ir (m,n+z),I vis (m,n+z),I ir (m+z,n),I vis (m+z,n),I ir (m+z,n+z),I vis (m+z,n+z) is the grayscale value of the pixel of the corresponding image, the value of z is -1, 1, -2, 2, G H (m,n),G V (m,n),G D (m,n) is the cross local directional energy in the horizontal H, vertical V and diagonal D directions of the pixel point (m,n) of the infrared and visible light images;

[0025] Step 4: According to the above steps, the low-frequency sub-band coefficient C X (m,n) is selected. G When (m,n)≤γ, the local directional energy and the large image subband coefficient are selected as the final value C F (m,n), the calculation formula is:

[0026]

[0027] Where: C F (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the fused image F, C ir (m,n) is the low-frequency subband coefficient at the infrared image pixel (m,n), C vis (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the visible light image, G ir (m,n) is the local directional energy sum at the infrared image pixel point (m,n), G vis (m,n) is the local directional energy sum at the pixel point (m,n) of the visible light image, M G (m,n) is the local directional energy and matching degree at the pixel point (m,n), γ is the local directional energy and matching degree threshold, and γ=0.7.

[0028] In the above method, the cross local directional energy and AG(m,n) of the cross infrared and visible light images can be calculated according to formula (3), and then the local directional energy and matching degree of the pixel point (m,n) can be calculated. The calculation formula is:

[0029]

[0030] Where: M G (m,n) is the local directional energy and matching degree of pixel point (m,n), AG(m,n) is the cross local directional energy and of pixel point (m,n) of infrared and visible light images, AG ir (m,n) is the local directional energy sum of the infrared image pixel point (m,n), AG vis (m,n) is the local directional energy sum of the pixel point (m,n) of the visible light image.

[0031] In the above method, when M G When (m,n)>γ, the remaining low-frequency sub-band coefficients can be fused according to the image significance, which specifically includes the following steps:

[0032] Step 1: Use the Frequency Tuned algorithm to perform saliency detection on the pixel point (m, n) in the source image X, and then calculate the saliency value S at the pixel point (m, n)X (m,n), the calculation formula is:

[0033]

[0034] Where: is the average gray value of the source image X, The grayscale value of the source image X at the pixel (m, n) is calculated by the Frequency Tuned algorithm, X = ir or vis, ir is the infrared image, vis is the visible light image;

[0035] Step 2: For the significance value S X (m,n) is normalized and then S-curve contrast stretching transformation is performed. The calculation formula is:

[0036]

[0037] Where: min S X and maxS X is the saliency map S of the source image X X The minimum and maximum grayscale values, is the saliency value S of the pixel (m,n) X The normalized calculated value of (m,n), is the normalized value of the pixel (m,n) The stretch transformation calculation value of X = ir or vis, ir is the infrared image, vis is the visible light image;

[0038] Step 3. Transform the value according to the stretching The size of the source image X is selected to obtain the weight coefficient W for the fusion of the low-frequency sub-band coefficients at the pixel point (m, n) of the source image X. X (m,n), the calculation formula is:

[0039]

[0040] Where: W ir (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the infrared image pixel point (m,n), W vis (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the pixel point (m,n) of the visible light image, is the calculated value of the significant stretch transformation of the infrared image pixel (m,n), is the calculated value of the significant stretch transformation of the visible light image pixel (m,n);

[0041] Step 4: Perform weighted fusion on the low-frequency sub-band coefficients to obtain the low-frequency sub-band coefficient C of the fused image F at the pixel point (m, n) F(m,n), the calculation formula is:

[0042] C F (m,n)=W ir (m,n)×C ir (m,n)+W vis (m,n)×C vis (m,n)M G (m,n)>γ (10)

[0043] Where: C F (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the fused image F, C ir (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the infrared image, C vis (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the visible light image, W ir (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the pixel point (m,n) of the infrared image, W vis (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the pixel point (m,n) of the visible light image, M G (m,n) is the local directional energy and matching degree at the pixel point (m,n), γ is the local directional energy and matching degree threshold, and γ=0.7.

[0044] In the above method, the highest decomposition layer uses a fusion rule based on local directional contrast, and other decomposition layers use a fusion rule based on local energy matching, which specifically includes the following steps:

[0045] Step 1: Calculate the local directional contrast at the highest decomposition layer, i direction, and pixel point (m, n) of the source image X The formula is:

[0046]

[0047] In the formula: the selected local area is A, the size is a×a, is the high-frequency bandpass direction subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the source image X, is the average value of the low-frequency subband coefficient of the source image X in the local area at the pixel point (m, n), C X (m,n) is the low-frequency subband coefficient of the source image X pixel (m,n), is the local directional contrast at the highest decomposition layer, i direction, pixel point (m, n) of the source image X, X = ir or vis, ir is the infrared image, vis is the visible light image;

[0048] Step 2: According to the local directional contrast, select the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, and pixel point (m, n) of the fused image F. The formula is:

[0049]

[0050] Where: is the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the fused image, is the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the infrared image, is the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the visible light image, is the local directional contrast at the highest decomposition layer, i direction, pixel point (m, n) of the infrared image, is the local directional contrast at the highest decomposition layer, i direction, pixel point (m, n) of the visible light image;

[0051] Step 3: Calculate the local energy of the local area at other decomposition layers, j scale, i direction, and pixel point (m, n) of the source image X Further determine the local energy matching degree M E (m,n), the calculation formula is:

[0052]

[0053] Where: the selected local area is A, the size is a×a, w(x,y) is the weighting matrix corresponding to the sub-band coefficient matrix in the high-frequency bandpass direction, I X (m,n) is the gray value of the pixel (m,n) of the source image X. Similarly, I X (m±x,n±y) is the gray value of the pixel of the corresponding image, The local energy of the local area at the other decomposition layers of the source image X, j scale, i direction, pixel point (m, n), M E (m,n) is the local energy matching degree of pixel point (m,n), X = ir or vis, ir is infrared image, vis is visible light image;

[0054] Step 4: Calculate the weighted coefficient W of the high-frequency bandpass direction sub-band coefficient according to the local energy matching degree min (m,n),W max (m,n), the calculation formula is:

[0055]

[0056] Where: λ is the local energy matching threshold, λ=0.7, W min (m,n),W max (m,n) is the weighting coefficient of the high-frequency bandpass subband coefficient of the pixel point (m,n), M E (m,n) is the local energy matching degree of pixel point (m,n);

[0057] Step 5. Obtain the high-frequency bandpass direction sub-band coefficients of the fused image F at other decomposition layers, j scales, i directions, and pixel points (m, n). The calculation formula is:

[0058]

[0059] Where: is the high-frequency bandpass direction subband coefficient at other decomposition layers, i direction, and pixel point (m, n) of the fused image, is the high-frequency bandpass direction subband coefficient at other decomposition layers, i direction, and pixel point (m, n) of the infrared image, is the high-frequency bandpass direction subband coefficient at other decomposition layers, i direction, pixel point (m, n) of the visible light image, W min (m,n),W max (m,n) is the weighting coefficient of the high-frequency bandpass subband coefficient of the pixel point (m,n), M E (m,n) is the local energy matching degree of pixel point (m,n), λ is the local energy matching degree threshold, and λ=0.7, is the local energy of the local area at the other decomposition layers, j scale, i direction, pixel point (m, n) of the infrared image, is the local energy of the local area at the other decomposition layers, j scale, i direction, and pixel point (m, n) of the visible light image.

[0060] The image fusion method and system based on frequency band coefficient decomposition provided by the present invention are used to improve the visual observation level of drivers. In this process, the influence of complex weather conditions such as night and rainy days on the driver's visual perception can be effectively reduced. The fused image has high clarity and resolution, thus avoiding the occurrence of traffic accidents.

[0061] Compared with the existing technology, the present invention has the following main advantages:

[0062] (1) Simple structure and easy to install.

[0063] (2) An image fusion method based on frequency band coefficient decomposition is proposed. The low-frequency sub-band coefficients adopt the fusion rule of directional energy and-significance, and the high-frequency band pass directional sub-band coefficients adopt the fusion rule of directional contrast-energy matching. The information loss of infrared images and visible light images is relatively small.

[0064] (3) The fused image has high clarity and resolution, which is suitable for human observation and helps improve the driver's recognition performance.

[0065] (4) The fused images have better visual effects under complex weather conditions such as night and rainy days, greatly improving the safety of driving under complex weather conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is the overall structural block diagram of the system of the present invention.

[0067] Figure 2 It is a block diagram of a single workflow of the system of the present invention.

[0068] Figure 3 It is a flowchart of the image fusion method of the present invention. DETAILED DESCRIPTION

[0069] The present invention relates to an image fusion method and system based on frequency band coefficient decomposition. The purpose is to provide a novel image fusion method and system that is easy to install and has a high accuracy. The steps of realizing the image fusion system are: installing an infrared thermal imager, a visible light camera and an on-board computer, initializing the system, the infrared thermal imager and the visible light camera synchronously collecting infrared images and visible light images in real time, the on-board computer reading images, pre-processing images, decomposing and fusing images based on frequency band coefficients, and outputting fused images. The present invention can effectively reduce the impact of complex meteorological conditions such as nighttime and rainy days on the visual perception of drivers, and the fused image has high clarity and resolution, and has high practical value.

[0070] The present invention will be further described below in conjunction with the embodiments and drawings, but the present invention is not limited thereto.

[0071] The image fusion system based on frequency band coefficient decomposition provided by the present invention is as follows: Figure 1 As shown, it includes two modules. The first module is an image acquisition module, which is equipped with an infrared thermal imager and a visible light camera; the second module is an image fusion module, which is equipped with an on-board computer. The two modules are connected by a network cable.

[0072] The infrared thermal imager and visible light camera can be selected from the DS-2TD5566T-15 / V2 dual-spectrum MINI pan-tilt camera, with a power supply voltage of 24V DC, an original resolution of 640*512, and support for mirroring and video output. Or, it can be selected according to the actual situation.

[0073] The vehicle computer can be a DLT-V72 FACELIFT vehicle terminal, with a power supply voltage of 12V DC, supporting WLAN wireless network connection, and supporting Windows operating system. Or, it can be selected according to actual conditions.

[0074] The vehicle-mounted computer and the PTZ camera are powered directly or indirectly by a vehicle-mounted power supply.

[0075] The PTZ camera is installed on the roof, and the onboard computer is installed under the dashboard of the car driver's seat and connected to the onboard power supply for power supply. The PTZ camera is connected to the onboard computer using a network cable, and the onboard computer is connected to the WAN wireless local area network to perform function configuration and parameter setting on the PTZ camera, thereby completing the activation and configuration of the PTZ camera.

[0076] The image fusion system based on frequency band coefficient decomposition provided by the present invention has a specific workflow as follows: Figure 2 As shown:

[0077] Step 1. Initialize the system.

[0078] Step 2: The PTZ camera starts working, collects images in real time, and transmits image data to the vehicle computer via the network cable.

[0079] Step 3: The on-board computer receives infrared images and visible light images, and performs grayscale conversion, noise reduction, and registration on the images.

[0080] Step 4: The on-board computer reads the processed infrared image and visible light image.

[0081] Step 5, the on-board computer fuses the read infrared image and visible light image. That is, the image is decomposed by NSST frequency band coefficients to obtain the low-frequency sub-band and high-frequency bandpass directional sub-band of the corresponding image. The low-frequency part first uses the fusion rule based on local directional energy and matching, and then uses the fusion rule based on image significance to fuse the low-frequency sub-band coefficients. The high-frequency part uses the fusion rule based on local directional contrast at the highest decomposition layer and the fusion rule based on local energy matching at other decomposition layers to fuse the high-frequency bandpass sub-band coefficients. Finally, the fused sub-band coefficients are inversely transformed to obtain a fused image.

[0082] Step 6: The on-board computer outputs the fused image and displays it on the monitor.

[0083] Step 7. Single system operation is completed.

[0084] The image fusion system of the present invention works cyclically according to the above steps, displays the fused image according to the 10-15ms interval time set by the user, and updates in real time.

[0085] The image fusion method based on frequency band coefficient decomposition provided by the present invention is as follows: Figure 3 As shown, the following steps are included:

[0086] Step 1, read the infrared image ir and the visible light image vis.

[0087] Step 2: Perform NSST frequency band coefficient decomposition on the read infrared image ir and visible light image vis to obtain the low-frequency sub-band coefficient C X (m, n) and high-frequency bandpass direction sub-band coefficients, and further divide the high-frequency bandpass direction sub-band coefficients into the highest decomposition layer high-frequency bandpass direction sub-band coefficients and other decomposition layer high frequency band pass direction subband coefficients (X=ir or vis, ir is infrared image, vis is visible light image, i is direction series, j is scale series).

[0088] Step 3: For the low-frequency sub-band coefficients, the fusion rule based on local directional energy and matching is first adopted, and then the fusion rule based on image saliency is adopted to obtain the low-frequency sub-band coefficients of the fused image F. The overall calculation formula is:

[0089]

[0090] Where: C F (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the fused image F, C ir (m,n) is the low-frequency subband coefficient at the infrared image pixel (m,n), C vis (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the visible light image, G ir (m,n) is the local directional energy sum at the infrared image pixel point (m,n), G vis (m,n) is the local directional energy sum at the pixel point (m,n) of the visible light image, W X (m,n) is the fusion weighting coefficient of the low-frequency subband coefficient at the pixel point (m,n) of the source image X, M G (m,n) is the local directional energy and matching degree at the pixel point (m,n), γ is the local directional energy and matching degree threshold, γ=0.7, X=ir or vis, ir is the infrared image, vis is the visible light image;

[0091] Step 4: First, the fusion rule based on local directional energy and matching is adopted for the low-frequency sub-band coefficients. The specific process is as follows:

[0092] Step 4-1. Take a local area A at a pixel point (m, n) in the low-frequency subband of the source image X, with a size of a×a. Define the local directional energy as the cumulative sum of the squares of the difference between the grayscale value in a fixed direction in the area and the selected grayscale value, and calculate the local directional energy of the pixel point (m, n) in the horizontal, vertical and diagonal directions in the local area respectively. The calculation formula is as follows:

[0093]

[0094] Where: I X (m,n) is the gray value of the pixel (m,n) of the source image X. Similarly, I X (m,n+z),I X (m+z,n),I X (m+z,n+z) is the gray value of the corresponding pixel, and the value of z is -1, 1, -2, 2. is the local directional energy of the source image pixel point X (m, n) in the horizontal H, vertical V and diagonal D directions, X = ir or vis, ir is the infrared image, vis is the visible light image;

[0095] Step 4-2, the local directional energy sum of the pixel point (m, n) of the source image X is equal to the sum of the directional energies of the window area. According to the above calculation results, the local directional energy sum of the pixel point (m, n) in the local area is calculated. The calculation formula is as follows:

[0096]

[0097] Where: is the local directional energy of the source image pixel X (m, n) in the horizontal H, vertical V and diagonal D directions, AG X (m,n) is the local directional energy sum of the pixel point (m,n) of the source image X. The selected local area is recorded as A, and the size is a×a. In order to make the calculation result more accurate, the local directional energy sum is removed from the size of the local area to avoid the interference of the local area size on the calculation result. In addition, in order to reduce the influence of image noise on the calculation result, it is necessary to perform weighted processing on the local directional energy sum. Let w(m,n) be the weighted coefficient matrix corresponding to the subband coefficient matrix, X=ir or vis, ir is the infrared image, and vis is the visible light image;

[0098] Step 4-3, calculate the cross local directional energy, define the cross local directional energy as equal to the gray value in a certain direction in the local area A where the pixel point (m, n) in the low-frequency subband of the infrared image ir and the visible light image vis is located and the selected gray value I XThe difference product of (m,n) is used to calculate the cross local directional energy G of the pixel point (m,n) in the horizontal, vertical and diagonal directions in the neighborhood H (m,n),G V (m,n),G D (m,n), the calculation formula is as follows:

[0099]

[0100] Where: I ir (m,n) is the gray value of the pixel (m,n) of the infrared image ir, I vis (m,n) is the gray value of the pixel (m,n) of the visible light image vis. Similarly, I ir (m,n+z),I vis (m,n+z),I ir (m+z,n),I vis (m+z,n),I ir (m+z,n+z),I vis (m+z,n+z) is the grayscale value of the pixel of the corresponding image, the value of z is -1, 1, -2, 2, G H (m,n),G V (m,n),G D (m,n) is the cross local directional energy in the horizontal H, vertical V and diagonal D directions of the pixel point (m,n) of the infrared and visible light images;

[0101] Furthermore, the cross-local direction energy and AG(m,n) in the local area where the pixel point (m,n) is located are calculated according to formula (3).

[0102] Step 4-4, define the local directional energy and matching degree M in the neighborhood of the pixel point (m, n) G (m,n), which is approximately the similarity of the sum of the two local directional energies, calculates the local directional energy and matching degree of the image, and the calculation formula is as follows:

[0103]

[0104] Where: M G (m,n) is the local directional energy and matching degree of pixel point (m,n), AG(m,n) is the cross local directional energy and of pixel point (m,n) in infrared and visible light images. ir (m,n) is the local directional energy sum of the infrared image pixel point (m,n), AG vis (m,n) is the local directional energy sum of the pixel point (m,n) of the visible light image;

[0105] Step 3-5, combine the above steps to select the low-frequency sub-band coefficients of the infrared image ir and the visible light image vis. G The value range of (m,n) is [0,1], and a local direction energy and matching threshold γ are given, and γ=0.7. Compare γ and M G (m,n) size, if M G When (m,n)≤γ, the low-frequency subband coefficient of the fused image F is selected according to the size of the local directional energy sum of the infrared image ir and the visible light image vis, that is, the low-frequency subband coefficient of the image with the largest local directional energy sum is selected as the final value to obtain the low-frequency subband coefficient C of the fused image F. F (m,n), the calculation formula is as follows:

[0106]

[0107] Where: C F (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the fused image F, C ir (m,n) is the low-frequency subband coefficient at the infrared image pixel (m,n), C vis (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the visible light image, G ir (m,n) is the local directional energy sum at the infrared image pixel point (m,n), G vis (m,n) is the local directional energy sum at the pixel point (m,n) of the visible light image, M G (m,n) is the local directional energy and matching degree at the pixel point (m,n), γ is the local directional energy and matching degree threshold, and γ=0.7;

[0108] Step 5, if M G When (m,n)>γ, the remaining low-frequency sub-band coefficients are fused according to the image saliency rule. The specific process is as follows:

[0109] Step 5-1, use the Frequency Tuned algorithm to perform saliency detection on the infrared image ir and the visible light image vis respectively. Use the Frequency Tuned algorithm to perform saliency detection on the pixel point (m, n) in the source image X, and then calculate the saliency value S at the pixel point (m, n) X (m,n), the calculation formula is:

[0110]

[0111] Where: is the average gray value of the source image X, The grayscale value of the source image X at the pixel (m, n) is calculated by the Frequency Tuned algorithm, X = ir or vis, ir is the infrared image, vis is the visible light image;

[0112] Step 5-2: For the significance value S X (m,n) are respectively normalized and transformed by S-curve contrast stretching, and the calculation formula is:

[0113]

[0114] Where: min S X and maxS X is the saliency map S of the source image X X The minimum and maximum grayscale values, is the saliency value S of the pixel (m,n) of the source image X X The normalized calculated value of (m,n), is the normalized value of the source image X pixel (m,n) The stretch transformation calculation value of X = ir or vis, ir is the infrared image, vis is the visible light image;

[0115] Step 5-3, according to the stretching transformation value The size of the source image X is selected to obtain the weight coefficient W for the fusion of the low-frequency sub-band coefficients at the pixel point (m, n) of the source image X. X (m,n), that is, select For images with large values, take the The value is used as the weighting coefficient W of the low-frequency subband coefficient X (m,n), the weighting coefficient of the low-frequency subband coefficient of the other image is 1-W X (m,n), the calculation formula is as follows:

[0116]

[0117] Where: W ir (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the pixel point (m,n) of the infrared image, W vis (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the pixel point (m,n) of the visible light image. is the calculated value of the significant stretch transformation of the pixel (m,n) of the infrared image, is the calculated value of the significant stretch transformation of the pixel (m,n) of the visible light image;

[0118] Step5-4, Combining the above steps, when M GWhen (m,n)>γ, the low-frequency sub-band coefficients of the infrared image ir and the visible light image vis are fused according to the image saliency rule, that is, the low-frequency sub-band coefficients are weightedly fused to obtain the low-frequency sub-band coefficient C of the fused image F at the pixel point (m,n). F (m,n), the calculation formula is as follows:

[0119] C F (m,n)=W ir (m,n)×C ir (m,n)+W vis (m,n)×C vis (m,n)M G (m,n)>γ (10)

[0120] Where: C F (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the fused image F, C ir (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the infrared image, C vis (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the visible light image, W ir (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the pixel point (m,n) of the infrared image, W vis (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the pixel point (m,n) of the visible light image, M G (m,n) is the local directional energy and matching degree at the pixel point (m,n), γ is the local directional energy and matching degree threshold, and γ=0.7;

[0121] Step 6: For the high-frequency bandpass direction sub-band coefficients, the high-frequency bandpass direction sub-band coefficients of the highest decomposition layer Using the fusion rule based on local directional contrast, the high-frequency bandpass directional sub-band coefficients of other decomposition layers The fusion rule based on local energy matching is used to obtain the high-frequency bandpass direction sub-band coefficients of the fused image F. The following steps are involved:

[0122] Step 7, first of all, the high-frequency bandpass direction sub-band coefficients of the highest decomposition layer The fusion rule based on local directional contrast is adopted. The specific process is as follows:

[0123] Step 7-1, define the local directional contrast of the highest decomposition layer as the absolute value of the high-frequency bandpass directional subband coefficient in that direction divided by the mean of the low-frequency subband coefficient. Take a local area at a pixel point (m, n) on the subband image of the highest decomposition layer, and calculate the local directional contrast at the highest decomposition layer, direction i, and pixel point (m, n) of the source image X The calculation formula is as follows:

[0124]

[0125] In the formula: the selected local area is A, the size is a×a, is the high-frequency bandpass direction subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the source image X, is the average value of the low-frequency subband coefficient of the source image X in the local area at the pixel point (m,n), is the local directional contrast at the highest decomposition layer of the source image X, direction i, pixel point (m, n), C X (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the source image X, X = ir or vis, ir is the infrared image, vis is the visible light image;

[0126] Step 7-2: Combine the above steps to compare the local directional contrast at the pixel point (m, n) of the highest decomposition layer The high-frequency bandpass directional sub-band coefficient is selected, that is, the high-frequency bandpass directional sub-band coefficient of the image with large local directional contrast value is selected as the final value, and the high-frequency bandpass directional sub-band coefficient of the highest decomposition layer is calculated. The calculation formula is as follows:

[0127]

[0128] Where: is the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the fused image, is the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the infrared image, is the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the visible light image, is the local directional contrast at the highest decomposition layer, i direction, pixel point (m, n) of the infrared image, is the local directional contrast at the highest decomposition layer, i direction, pixel point (m, n) of the visible light image;

[0129] Step 8: Sub-band coefficients of high-frequency passbands in other decomposition layers The fusion rule based on local energy matching is adopted. The specific process is as follows:

[0130] Step 8-1. Take a local area A at a pixel point (m, n) in the high-frequency bandpass direction subband, with a size of a×a, and calculate the local energy of the local area A at other decomposition layers, j scales, i directions, and pixel points (m, n) of the source image X. Furthermore, the local energy matching degree is defined to characterize the similarity of the local energy of the two images, and the local energy matching degree M at the pixel point (m, n) is calculated. E (m,n), the calculation formula is:

[0131]

[0132] Where: I X (m,n) is the gray value of the pixel (m,n) of the source image X. Similarly, I X (m±x,n±y) is the gray value of the pixel of the corresponding image, The local energy of the local area A at the other decomposition layers, j scale, i direction, pixel point (m, n) of the source image X, M E (m,n) is the local energy matching degree at the pixel point (m,n). In order to reduce the influence of image noise on the calculation results, it is necessary to perform weighted processing on the local directional energy and, let w(x,y) be the weighted matrix corresponding to the high-frequency bandpass directional subband coefficient matrix, X = ir or vis, ir is the infrared image, vis is the visible light image;

[0133] Step 8-2: Determine the weighting coefficient W of the high-frequency band pass sub-band coefficient of the infrared image ir and the visible light image vis according to the local energy matching degree of Step 4-3. min (m,n),W max (m,n), the calculation formula is as follows:

[0134]

[0135] Where: λ is the local energy matching threshold, λ=0.7, W min (m,n),W max (m,n) is the weighting coefficient of the high-frequency bandpass subband coefficient at the pixel point (m,n), M E (m,n) is the local energy matching degree at pixel point (m,n);

[0136] Step 8-3, combining the above steps, the high-frequency bandpass direction sub-band coefficients of other decomposition layers of the infrared image ir and the visible light image vis are fused. E When (m,n)≤λ, the high-frequency bandpass direction subband coefficient is selected according to the size of the local energy of the image, that is, the high-frequency bandpass direction subband coefficient of the image with large local energy is selected as the result; when M E When (m,n)>λ, the weighted proportion coefficient of the high-frequency bandpass direction sub-band coefficient is determined according to the local energy matching degree of the image, and the weighted fusion of the corresponding high-frequency bandpass direction sub-band coefficient is performed to calculate the high-frequency bandpass direction sub-band coefficient of the fused image F at other decomposition layers, j scales, i directions, and pixel points (m,n). The calculation formula is:

[0137]

[0138] Where: is the high-frequency bandpass direction subband coefficient at other decomposition layers, i direction, and pixel point (m, n) of the fused image, is the high-frequency bandpass direction subband coefficient at other decomposition layers, i direction, and pixel point (m, n) of the infrared image, is the high-frequency bandpass direction subband coefficient at other decomposition layers, i direction, pixel point (m, n) of the visible light image, W min (m,n),W max (m,n) is the weighting coefficient of the high-frequency bandpass sub-band coefficient at the pixel point (m,n), is the local energy of the local area A at the other decomposition layers, j scale, i direction, pixel point (m, n) of the infrared image, is the local energy of the local area A at the other decomposition layers, j scale, i direction, pixel point (m, n) of the visible light image, M E (m,n) is the local energy matching degree at the pixel point (m,n), λ is the local energy matching degree threshold, and λ=0.7;

[0139] Step 9: Obtain the low-frequency subband coefficient C according to the above steps F (m,n) and high-frequency bandpass direction subband coefficients Perform inverse coefficient transformation to obtain the fused image F.

[0140] The image fusion method and system based on frequency band coefficient decomposition provided by the present invention have the following characteristics:

[0141] 1. An image fusion method based on frequency band coefficient decomposition is proposed. The low-frequency sub-band coefficients adopt the fusion rule of directional energy and-significance, and the high-frequency band-pass directional sub-band coefficients adopt the fusion rule of directional contrast-energy matching.

[0142] 2. A low-frequency sub-band coefficient fusion method is proposed, which first adopts a fusion rule based on local directional energy and matching, and then adopts a fusion rule based on image saliency to fuse the low-frequency sub-band coefficients.

[0143] 3. A high-frequency bandpass directional sub-band coefficient fusion method is proposed, that is, the highest decomposition layer uses a fusion rule based on local directional contrast, and other decomposition layers use a fusion rule based on local energy matching to fuse high-frequency bandpass directional sub-band coefficients.

[0144] 4. The image fusion system based on frequency band coefficient decomposition of the present invention has a simple structure and is easy to install. The final fused image information is relatively accurate and is not affected by meteorological conditions. It retains both the significant target features of the infrared image and the clear background detail information of the visible light image.

[0145] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. An image fusion method based on frequency band coefficient decomposition, characterized in that: Firstly, the infrared image ir and the visible light image vis are decomposed by NSST frequency band coefficients to obtain low-frequency sub-bands and high-frequency band pass directional sub-bands, wherein: the low-frequency sub-band first adopts the fusion rule based on local directional energy and matching, and then adopts the fusion rule based on image saliency to fuse the low-frequency sub-band coefficients; the high-frequency band pass directional sub-band adopts the fusion rule based on local directional contrast at the highest decomposition layer, and the other decomposition layers use the fusion rule based on local energy matching to fuse the high-frequency band pass directional sub-band coefficients. Then, the above fused coefficients are inversely transformed to obtain a fused image F; The method uses the following steps: When the fusion rule based on local directional energy and matching is first used for the low-frequency sub-band coefficients, and then the fusion rule based on image saliency is used, the calculation formula of the low-frequency sub-band coefficients of the obtained fused image F at the pixel point (m, n) is: Where: C F (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the fused image F, C ir (m,n) is the low-frequency subband coefficient at the infrared image pixel (m,n), C vis (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the visible light image, G ir (m,n) is the local directional energy sum at the infrared image pixel point (m,n), G vis (m,n) is the local directional energy sum at the pixel point (m,n) of the visible light image, M G (m,n) is the local directional energy and matching degree at the pixel point (m,n), W X (m,n) is the fusion weighting coefficient of the low-frequency subband coefficient at the pixel point (m,n) of the source image X, γ is the local directional energy and matching threshold, γ=0.7, X=ir or vis, ir is the infrared image, vis is the visible light image; The highest decomposition layer uses a fusion rule based on local directional contrast, and other decomposition layers use a fusion rule based on local energy matching, which specifically includes the following steps: Step 1: Calculate the local directional contrast at the highest decomposition layer, i direction, and pixel point (m, n) of the source image X Step 2: According to the local directional contrast, select the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, and pixel point (m, n) of the fused image F. Step 3: Calculate the local energy of the local area A at the j scale, i direction, and pixel point (m, n) in other decomposition layers of the source image X Further determine the local energy matching degree M E (m,n), Step 4: Calculate the weighted coefficient W of the high-frequency bandpass direction sub-band coefficient according to the local energy matching degree min (m,n),W max (m,n), Step 5. Obtain the high-frequency bandpass direction sub-band coefficients of the fused image F at other decomposition layers, j scales, i directions, and pixel points (m, n). The calculation formula is: Where: is the high-frequency bandpass direction subband coefficient at other decomposition layers, i direction, and pixel point (m, n) of the fused image, is the high-frequency bandpass direction subband coefficient at other decomposition layers, i direction, and pixel point (m, n) of the infrared image, is the high-frequency bandpass direction subband coefficient at other decomposition layers, i direction, pixel point (m, n) of the visible light image, W min (m,n),W max (m,n) is the weighting coefficient of the high-frequency bandpass subband coefficient of the pixel point (m,n), M E (m,n) is the local energy matching degree of pixel point (m,n), λ is the local energy matching degree threshold, and λ=0.7, is the local energy of the local area A at the pixel point (m, n) at the j scale and i direction in other decomposition layers of the infrared image, is the local energy of the local area A at the pixel point (m, n) at the j scale and i direction in other decomposition layers of the visible light image.

2. The image fusion method based on frequency band coefficient decomposition according to claim 1, characterized in that: When M G When (m,n)≤γ, the fusion rule based on local directional energy and matching is adopted for the low-frequency subband coefficients, which specifically includes the following steps: Step 1. Calculate the local directional energy of the source image pixel point X (m, n) in the horizontal H, vertical V and diagonal D directions respectively. The calculation formula is: Where: I X (m,n) is the gray value of the pixel (m,n) of the source image X. Similarly, I X (m,n+z),I X (m+z,n),I X (m+z,n+z) is the grayscale value of the corresponding image pixel, and the value of z is -1, 1, -2, 2. is the local directional energy of the source image pixel point X (m, n) in the horizontal H, vertical V and diagonal D directions, X = ir or vis, ir is the infrared image, vis is the visible light image, Step 2, calculate the local directional energy and the calculation formula is: Where: the selected local area is recorded as A, the size is a×a, w(m,n) is the weighting matrix corresponding to the coefficient matrix, is the local directional energy of the source image pixel X (m, n) in the horizontal H, vertical V and diagonal D directions, AG X (m,n) is the local directional energy sum of the source image pixel point (m,n), X = ir or vis, ir is the infrared image, vis is the visible light image, Step 3. Calculate the cross-local directional energy in the horizontal H, vertical V and diagonal D directions of the pixel point (m, n) of the infrared and visible light images. The calculation formula is: Where: I ir (m,n) is the gray value of the pixel (m,n) of the infrared image ir, I vis (m,n) is the gray value of the pixel (m,n) of the visible light image vis. Similarly, I ir (m,n+z),I vis (m,n+z),I ir (m+z,n),I vis (m+z,n),I ir (m+z,n+z),I vis (m+z,n+z) is the grayscale value of the pixel of the corresponding image, the value of z is -1, 1, -2, 2, G H (m,n),G V (m,n),G D (m,n) is the cross local directional energy in the horizontal H, vertical V and diagonal D directions of the pixel point (m,n) of the infrared and visible light images, Step 4: According to the above steps, the low-frequency sub-band coefficient C X (m,n) is selected, when M G When (m,n)≤γ, the local directional energy and the large image subband coefficient are selected as the final value C F (m,n), the calculation formula is: Where: C F (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the fused image F, C ir (m,n) is the low-frequency subband coefficient at the infrared image pixel (m,n), C vis (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the visible light image, G ir (m,n) is the local directional energy sum at the infrared image pixel point (m,n), G vis (m,n) is the local directional energy sum at the pixel point (m,n) of the visible light image, M G (m,n) is the local directional energy and matching degree at the pixel point (m,n), γ is the local directional energy and matching degree threshold, and γ=0.7; According to formula (3), the cross local directional energy and AG(m,n) of the cross infrared and visible light images are calculated, and then the local directional energy and matching degree of the pixel point (m,n) are calculated. The calculation formula is: Where: M G (m,n) is the local directional energy and matching degree of pixel point (m,n), AG(m,n) is the cross local directional energy and of pixel point (m,n) of infrared and visible light images, AG ir (m,n) is the local directional energy sum of the infrared image pixel point (m,n), AG vis (m,n) is the local directional energy sum of the pixel point (m,n) of the visible light image; When M G When (m,n)>γ, the remaining low-frequency sub-band coefficients are fused according to the image significance, which includes the following steps: Step 1: Use the Frequency Tuned algorithm to perform saliency detection on the pixel point (m, n) in the source image X, and then calculate the saliency value S at the pixel point (m, n) X (m,n), the calculation formula is: in, is the average gray value of the source image X, The grayscale value of the source image X at the pixel (m, n) is calculated by the Frequency Tuned algorithm, X = ir or vis, ir is the infrared image, vis is the visible light image; Step 2: For the significance value S X (m,n) is normalized and then S-curve contrast stretching transformation is performed. The calculation formula is: Where, min S X and maxS X is the saliency map S of the source image X X The minimum and maximum grayscale values, is the saliency value S of the pixel (m,n) X The normalized calculated value of (m,n), is the normalized value of the pixel (m,n) The calculated value of the stretch transformation, X = ir or vis, ir is the infrared image, vis is the visible light image; Step 3. Transform the value according to the stretching The size of the source image X is selected to obtain the weight coefficient W for the fusion of the low-frequency sub-band coefficients at the pixel point (m, n) of the source image X. X (m,n), the calculation formula is: Where: W ir (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the infrared image pixel point (m,n), W vis (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the pixel point (m,n) of the visible light image, is the calculated value of the significant stretch transformation of the infrared image pixel (m,n), is the calculated value of the significant stretch transformation of the visible light image pixel (m,n); Step 4: Perform weighted fusion on the low-frequency sub-band coefficients to obtain the low-frequency sub-band coefficient C of the fused image F at the pixel point (m, n) F (m,n), the calculation formula is: C F (m,n)=W ir (m,n)×C ir (m,n)+W vis (m,n)×C vis (m,n) M G (m,n)>γ (10) Where: C F (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the fused image F, C ir (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the infrared image, C vis (m,n) is the low-frequency subband coefficient at the pixel point (m,n) of the visible light image, W ir (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the pixel point (m,n) of the infrared image, W vis (m,n) is the weight coefficient of the low-frequency sub-band coefficient fusion at the pixel point (m,n) of the visible light image, M G (m,n) is the local directional energy and matching degree at the pixel point (m,n), γ is the local directional energy and matching degree threshold, and γ=0.

7.

3. The image fusion method based on frequency band coefficient decomposition according to claim 1, characterized in that: The highest decomposition layer uses a fusion rule based on local directional contrast, and other decomposition layers use a fusion rule based on local energy matching. The formulas used in Step 1 to Step 4 are: Step 1: Calculate the local directional contrast at the highest decomposition layer, i direction, and pixel point (m, n) of the source image X The formula is: In the formula: the selected local area is A, the size is a×a, is the high-frequency bandpass direction subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the source image X, is the average value of the low-frequency subband coefficient of the source image X in the local area at the pixel point (m, n), C X (m,n) is the low-frequency subband coefficient of the source image X pixel (m,n), is the local directional contrast of the highest decomposition layer of the source image X, direction i, and pixel point (m, n), X = ir or vis, ir is the infrared image, vis is the visible light image, Step 2: According to the local directional contrast, select the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, and pixel point (m, n) of the fused image F. The formula is: Where: is the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the fused image, is the high-frequency bandpass directional subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the infrared image, is the high-frequency bandpass direction subband coefficient at the highest decomposition layer, i direction, pixel point (m, n) of the visible light image, is the local directional contrast at the highest decomposition layer, i direction, pixel point (m, n) of the infrared image, is the local directional contrast at the highest decomposition layer, i direction, pixel point (m, n) of the visible light image, Step 3: Calculate the local energy of the local area A at the j scale, i direction, and pixel point (m, n) in other decomposition layers of the source image X Further determine the local energy matching degree M E (m,n), the calculation formula is: Where: w(x,y) is the weighting matrix corresponding to the sub-band coefficient matrix in the high-frequency bandpass direction, I X (m,n) is the gray value of the pixel (m,n) of the source image X. Similarly, I X (m±x,n±y) is the gray value of the pixel of the corresponding image, The local energy of the local area A at the pixel point (m, n) at the j scale and i direction in other decomposition layers of the source image X, M E (m,n) is the local energy matching degree of pixel point (m,n), X = ir or vis, ir is infrared image, vis is visible light image, Step 4: Calculate the weighted coefficient W of the high-frequency bandpass direction sub-band coefficient according to the local energy matching degree min (m,n),W max (m,n), the calculation formula is: Where: λ is the local energy matching threshold, λ=0.7, W min (m,n),W max (m,n) is the weighting coefficient of the high-frequency bandpass subband coefficient of the prime point (m,n), M E (m,n) is the local energy matching degree of pixel point (m,n).

4. The image fusion method based on frequency band coefficient decomposition according to any one of claims 1 to 3, characterized in that: It is used to improve the driver's visual observation level. In this process, it can effectively reduce the impact of complex weather conditions at night and in rainy days on the driver's visual perception, and integrate high image clarity and resolution to avoid traffic accidents.

5. An image fusion system based on frequency band coefficient decomposition, characterized in that: The image fusion method based on frequency band coefficient decomposition as described in claim 1, 2, 3 or 4 includes two modules, wherein the first module is an image acquisition module, which is provided with an infrared thermal imager and a visible light camera; the second module is an image fusion module, which is provided with a vehicle-mounted computer; the two modules are connected by a network cable.

6. The image fusion system based on frequency band coefficient decomposition according to claim 5, characterized in that: The infrared thermal imager collects infrared images ir, and the visible light camera collects visible light images vis. The collected images are transmitted to the on-board computer via a network cable, and image fusion based on frequency band coefficient decomposition is completed in the on-board computer to display the fused image F.

7. The image fusion system based on frequency band coefficient decomposition according to claim 6, characterized in that: The system updates the fused image F in real time according to the set 10-15ms time interval.

Images