A near-infrared and visible light image fusion method and system

Abstract

The application discloses a near-infrared and visible light image fusion method and system, and the method comprises the following steps: acquiring a near-infrared image and a visible light image, constructing a regularization term, wherein the regularization term comprises a data fidelity term, a multi-order hyper-Laplacian prior term and a noise mapping term, constructing a fusion model for near-infrared image and visible light image fusion based on the regularization term, solving the optimal value of the fusion model, and obtaining an optimal fusion image. The method solves the technical problems of detail loss and inconsistent structure in the existing near-infrared and visible light image fusion method, realizes fine extraction of image details in a foggy / low-light scene, can reveal details in a heavy foggy / nose degradation area in a low-light / foggy scene, and can effectively avoid color distortion in the fusion image.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the field of image processing technology, in particular to a near-infrared and visible light image fusion method and system. BACKGROUND

[0002] Near-infrared and visible light image fusion is an important image enhancement technique, the purpose of which is to generate a high-quality fused image with rich texture by fusing near-infrared and visible light images under extreme environments. Near-infrared and visible light images have different characteristics in terms of color and visibility. The image captured by a visible light band sensor has the highest sensitivity to radiation with a wavelength in the range of (400-700 nm), but the visibility and scene contrast of the visible light band image are low under special conditions such as fog, haze or low light. In contrast, the near-infrared (NIR) band (650-1650 nm) radiation can capture images with strong visibility and contrast, which contain complementary details of the scene that can be missed in the visible light image.

[0003] The fusion method of near-infrared and visible light images is mainly divided into three categories: blind method, variational-based method and data-driven method. The blind method has fast execution time, but the fused image produced by it is prone to visual artifacts and detail loss. The variational-based method can produce satisfactory fusion results, but its performance mainly depends on the selection of prior knowledge. The data-driven method needs a training data set with basic true values. However, in extremely harsh environments, it becomes more difficult to collect a large number of clear images. In summary, most of the existing near-infrared and visible light image fusion methods have the problems of detail loss and structural inconsistency under heavy fog / haze / noise conditions. In order to solve these problems, the present application proposes a new near-infrared and visible light image fusion method based on multi-order super-Laplacian prior, which applies the l 1 / 2 norm to the multi-order gradient difference of near-infrared and visible light images, an effective multi-order super-Laplacian prior is designed to alleviate the loss of details caused by fog / haze / noise degradation. In addition, considering the incompatibility of denoising and fusion tasks in low-light scenes, a noise map is designed to eliminate inconsistent structures. Finally, a unified variational framework is established based on the multi-order super-Laplacian prior and the noise map for near-infrared and visible light image / video fusion. This framework can preserve the basic structure and improve the visual fidelity of the fused image / video in harsh scenes.

[0004] The patent with publication number CN 111429389 A discloses a visible light and near-infrared image fusion method that preserves spectral characteristics. The invention proposes a reflection weight model based on the differences in visible light and near-infrared spectral reflectance characteristics. This model takes into account the relationship between object apparent color and visible light spectral reflectance characteristics, achieving a natural and realistic color in the fused image. Secondly, texture layers and contour layers are obtained layer by layer from guided filtering and Gaussian low-pass filtering. The fusion weights of the texture layers and contour layers are calculated, and the visible light and near-infrared images are fused layer by layer to obtain a clear and natural color fusion image.

[0005] However, this method may cause serious structural inconsistency problems during the fusion process. This method estimates the final fused image to improve details by calculating the reflection weight model and the fusion weight. However, this method only considers improving the visual visibility of foggy scenes and does not consider the inconsistent structure caused by noise.

[0006] The patent with publication number CN 113160286 A discloses a near-infrared and visible light image fusion method based on a convolutional neural network. The invention proposes a near-infrared and visible light image fusion method based on a convolutional neural network. A hybrid loss function composed of pixel loss, structure loss, and edge loss is designed, and the training set is input to train the fusion network. Finally, the near-infrared and visible light image pairs to be fused are input to obtain the final fusion image.

[0007] However, this method still has poor image fusion effect under heavy haze / noise degradation, and cannot effectively extract the detail features of the two types of degradation. This method has poor robustness in extreme environments where true values cannot be obtained, as it becomes more difficult to collect a large number of clear images in extremely harsh environments. SUMMARY

[0008] The near-infrared and visible light image fusion method and system provided by the present invention solve the technical problems of detail loss and structural inconsistency in existing near-infrared and visible light image fusion methods.

[0009] To solve the above technical problems, the near-infrared and visible light image fusion method proposed by the present invention includes:

[0010] Obtain a near-infrared image and a visible light image.

[0011] Construct a regularization term, which includes a data fidelity term, a multi-order hyper-Laplacian prior term, and a noise mapping term.

[0012] Based on the regularization term, construct a fusion model for near-infrared image and visible light image fusion.

[0013] Solve the optimal value of the fusion model to obtain an optimal fusion image.

[0014] Further, the specific formula of the multi-order super Laplace prior term in the regularization term is as follows:

[0015]

[0016] Wherein, f MHLP represents the multi-order super Laplace prior term, X is an optimal fusion image to be solved, P is a near-infrared image, and ∇1X and ∇2X represent first-order gradients of X in X direction and Y direction, respectively, ∇1P and ∇2P represent first-order gradients of P in X direction and Y direction, respectively, ΔX and ΔP represent second-order gradients of X and P, respectively, α1, α2 and γ are penalty parameters, and ||·||1 1 / 2 represents the l1 norm. 1 / 2

[0017] Further, the specific formula of the noise mapping term is as follows:

[0018]

[0019] Wherein, f TV represents the noise mapping term, N represents an estimated noise mapping image, β1, β2, β3 and δ are regularization parameters, ∇1X, ∇2X and ∇3X represent first-order gradients of the optimal fusion image X to be solved in X direction, Y direction and Z direction, respectively, ||·||1 represents the l1 norm, and ||·||2 F represents the l2 norm. F

[0020] Further, the specific formula of the fusion model is as follows:

[0021]

[0022] Wherein, Y is a degraded image collected in a harsh environment.

[0023] Further, the optimal value of the fusion model is solved to obtain an optimal fusion image:

[0024] The optimization problem of solving the optimal value of the fusion model is changed into a Lagrange extended function by using a Lagrange multiplier optimization method.

[0025] The Lagrange extended function is decomposed into five sub-problems by using an alternating minimum iteration framework, which are sub-problems of W i , M, N, H j and X, and the specific formulas are as follows:

[0026]

[0027] Wherein, W i = ∇ i X- ∇ i ​​P, M = ΔX - ΔP, H j =▽ j X, A1 is the Lagrange multiplier multiplier of W1 in the X direction, A2 is the Lagrange multiplier multiplier of W2 in the Y direction, B is the Lagrange multiplier multiplier of M, C1 is the Lagrange multiplier multiplier of H1 in the X direction, C2 is the Lagrange multiplier multiplier of H2 in the Y direction, C3 is the Lagrange multiplier multiplier of H3 in the Z direction, ψ and η are penalty parameters, i = {1, 2}, j = {1, 2, 3}.

[0028] The five sub-problems are iteratively solved, and after the (k+1)th iteration, W i , M, N, H j and X are obtained, wherein the iteration termination condition is specifically:

[0029]

[0030] X k and X k+1 respectively represent the solutions of X after the kth and (k+1)th iteration.

[0031] According to the solution of X obtained after the (k+1)th iteration, the optimal fused image is obtained.

[0032] Further, the near-infrared image is specifically a near-infrared frame contained in a near-infrared video, and the visible light image is specifically a visible light frame contained in a visible light video.

[0033] Further, the specific formula of the fusion model is:

[0034]

[0035] X = {X1, X2,..., X n}, Y = {Y1, Y2,..., Y n}, P = {P1, P2,..., P n} and N = {N1, N2,..., N n} respectively represent the estimated fusion result sequence, the input visible light frame sequence, the near-infrared frame sequence and the estimated noise map sequence, and t = {1,..., n}, n represents the total number of input frames, X t , Y t , P t , N t respectively represent the fusion result, the input visible light frame, the near-infrared frame and the estimated noise map of any one frame in the input video stream, △X t , △F t,o (X o ) and △P t respectively represent the gradients of Xt , F t,o (X o ) and P t The first-order gradient sums of X direction and Y direction, that is, ∇X t = ∇1X t + ∇2X t , ∇F t,o (X o ) = ∇1F t,o (X o ) + ∇2F t,o (X o ), ∇P t = ∇1P t + ∇2P t , F t,o (X o ) represents the position correspondence between X t and X o .

[0036] The near-infrared and visible light image fusion system provided by the application comprises:

[0037] The memory, the processor and the computer program stored in the memory and executable on the processor, and the processor implements the steps of the near-infrared and visible light image fusion method provided by the application when executing the computer program.

[0038] The near-infrared and visible light image fusion method and system provided by the application solve the technical problems of detail loss and structure inconsistency in the existing near-infrared and visible light image fusion method by obtaining a near-infrared image and a visible light image, constructing a regularization term, the regularization term comprising a data fidelity term, a multi-order super Laplace prior term and a noise mapping term, constructing a fusion model for near-infrared image and visible light image fusion based on the regularization term, and solving the optimal value of the fusion model to obtain an optimal fusion image, realize fine extraction of image details in a foggy / low-light scene, not only can reveal details in a heavy foggy / low-light / noise degraded area in a low-light / foggy scene, but also can effectively avoid color distortion in the fusion image.

[0039] The beneficial effects of the application specifically include:

[0040] (1) The application proposes a simple and effective multi-order super Laplace prior term to maintain image structure and details in a foggy / low-light scene. The prior term is a combination of the l 1 / 2 norm of the first-order and second-order gradient differences of near-infrared and visible light images. The former can extract fine large-scale detail information of the image, and the latter can compensate for the scene details ignored by the former by capturing relatively small-scale high-order edges in the gradient domain.

[0041] (2) Considering the incompatibility of denoising and fusion tasks in low-light scenes, the application designs a noise mapping term to separate noise information from the image to prevent interference between denoising and fusion tasks. Specifically, the noise mapping term regularizes the fused image to estimate the noise map in the image, and then uses anisotropic total variation to smooth the noise map to ensure that the noise is effectively suppressed during the fusion process, and finally realizes the fusion task of near-infrared and visible light images in low-light environments.

[0042] (3) Based on the data fidelity term, the multi-order hyper-Laplacian prior term and the noise mapping term, the application establishes a unified near-infrared and visible light image fusion framework. The framework decomposes the complex minimization problem into five simple sub-problems, and iteratively solves the five sub-problems by alternating minimization to ensure the convergence of the entire framework. In addition, the noise mapping term is set to zero and non-zero terms are applied to image restoration in foggy and low-light scenes.

[0043] (4) The overall framework proposed by the application is not only applicable to images, but can also be extended to videos. Specifically, the extracted video features should be along the inter-frame continuous motion trajectory to facilitate the inference of the lost information in the degradation process. Therefore, a time coherence term is introduced in the established framework to extract detailed features from adjacent frames to enhance the structural consistency between frames and realize the near-infrared and visible light video fusion task. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The near-infrared and visible light image fusion method flowchart of the second embodiment of the application;

[0045] Figure 2 The near-infrared and visible light image fusion result schematic diagram in the foggy scene of the second embodiment of the application;

[0046] Figure 3 The near-infrared and visible light image fusion result schematic diagram in the low-light scene of the second embodiment of the application;

[0047] Figure 4 The structure block diagram of the near-infrared and visible light image fusion system of the embodiment of the application.

[0048] REFERENCE NUMERALS:

[0049] 10, memory; 20, processor. DETAILED DESCRIPTION

[0050] In order to facilitate the understanding of the application, the application will be described more fully and in detail below with reference to the accompanying drawings and preferred embodiments, but the scope of protection of the application is not limited to the following specific embodiments.

[0051] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0052] Example 1

[0053] The near-infrared and visible light image fusion method provided in Embodiment 1 of the present invention includes:

[0054] Step S101: Acquire near-infrared and visible light images.

[0055] Step S102: Construct regularization terms, which include data fidelity terms, multi-order hyperLaplace prior terms, and noise mapping terms.

[0056] Step S103: Based on the regularization term, construct a fusion model for fusing near-infrared images and visible light images.

[0057] Step S104: Solve for the optimal value of the fusion model to obtain the optimal fused image.

[0058] The near-infrared and visible light image fusion method provided in this invention acquires near-infrared and visible light images, constructs a regularization term, which includes a data fidelity term, a multi-order super-Laplacian prior term, and a noise mapping term. Based on the regularization term, a fusion model for fusing near-infrared and visible light images is constructed, and the optimal value of the fusion model is solved to obtain the optimal fused image. This method solves the technical problems of detail loss and structural inconsistency in existing near-infrared and visible light image fusion methods, and achieves fine extraction of image details in foggy / low-light scenes. It can not only reveal details in areas degraded by heavy fog / noise in low-light / foggy scenes, but also effectively avoid color distortion in the fused image.

[0059] Example 2

[0060] like Figure 1 As shown, this embodiment of the invention proposes a near-infrared and visible light image fusion method based on multi-order super-Laplacian priors, including the following steps:

[0061] S1: Construct regularization terms with different functions, including: data fidelity terms, multi-order hyperLaplace prior terms, and noise mapping terms.

[0062] S2: Based on the regularization term established in step (1), a variational framework for near-infrared and visible light image fusion is designed. Multi-order hyperLaplacian priors are applied to the proposed framework to capture details of fine scenes in harsh environments.

[0063] S3: Time consistency is added as a regularization term to the variational framework established in step (2), which is applied to the near-infrared and visible light video fusion task.

[0064] The specific implementation scheme is as follows:

[0065] S1: Establish different function regularization terms:

[0066] Mathematically, the visible light degradation problem can be expressed by a linear model as follows:

[0067]

[0068] Wherein is the degraded image collected in a harsh environment, represents the original high-quality image, and N represents the additional random noise, represents a spatial linear degradation operator, which refers to low light / haze degradation. For different settings, formula (1) can represent different visible light image restoration problems. When the input image is degraded by low light, it becomes a near-infrared and visible light image fusion problem to remove low light image noise; when the input image is degraded by haze, it becomes a near-infrared and visible light image fusion problem to restore haze details. Note that N=0 at this time. The framework of near-infrared and visible light fusion is as follows:

[0069]

[0070] Wherein f Data (X,Y) is a data fidelity term to enforce consistency between X and Y. f MHLP (X,P) is a detail injection term that uses the exact l 1 / 2 norm to enforce the sparsity of the multi-order gradient difference. Here P is the input near-infrared image. f TV (X,N) is a noise mapping term that is used to enhance the sparsity of the fused image and remove the noise of the low light image. The specific details are as follows:

[0071] Data fidelity term: According to equation (1), the data fidelity term can be set as:

[0072]

[0073] Where Y is the degraded image collected in a harsh environment, X is the original high-quality visible light image, and N is the estimated noise mapping image. Here is the Frobenius norm. Note that if the input visible light image is an image degradation caused by noise, N=0.

[0074] Detail injection term: there are rich image structures in the first-order and second-order gradient differences of visible light and near-infrared, including significant edges and fine-scale details. The first-order gradient difference can extract fine details, and the second-order gradient difference can compensate for the loss of details caused by the first-order gradient difference. Therefore, the embodiment of the present application attempts to model the first-order and second-order gradient differences of the detail extraction with the super-Laplacian prior, to obtain complete structures and fine-scale details. The multi-order super-Laplacian prior f(X, P) is defined as:

[0075]

[0076] Wherein, f MHLP represents the multi-order super-Laplacian prior term, X is the optimal fusion image to be solved, P is the near-infrared image, and ∇1X and ∇2X represent the first-order gradients of X in the X direction and the Y direction, respectively, ∇1P and ∇2P represent the first-order gradients of P in the X direction and the Y direction, respectively, ΔX and ΔP represent the second-order gradients of X and P, respectively, and α1, α2 and γ are penalty parameters, and ||·||1 1 / 2 represents the l 1 / 2 1 norm.

[0077] Noise mapping term: there is noise interference in the low-light environment, which will directly affect the visual effect of the image fusion result. The most intuitive solution is to use an advanced denoising algorithm to preprocess the visible image, and then combine the near-infrared and visible light images through the fusion method. However, separating the image denoising and image fusion processing often causes incompatibility problems, resulting in poor image fusion effect. In order to prevent the mutual interference of denoising and fusion tasks, the embodiment of the present application uses total variation regularization (TV) to regularize the fusion image, separates the noise information from the image, and thus ensures the suppression of noise intensity in the fusion process. This regularization can maintain the piecewise constant and sparsity of the fusion result. The anisotropic TV regularization is as follows:

[0078]

[0079] Where (i, j, k) represents the index of the pixel, and X is a tensor. Therefore, the noise mapping term is defined as follows:

[0080]

[0081] Wherein, f TV represents the noise mapping term, N represents the estimated noise mapping image, β1, β2, β3 and δ are regularization parameters, ∇1X, ∇2X and ∇3X represent the first-order gradients of the optimal fusion image X to be solved in the X direction, the Y direction and the Z direction, respectively, ||·||1 represents the l1 norm, and ||·||2 F represents the l F 2 norm.

[0082] S2: Unified variational framework for near-infrared and visible image fusion:

[0083] Combining the above, the final model can be expressed as:

[0084]

[0085] By introducing three auxiliary variables W i , M, H j , the optimization problem of (7) is changed into a Lagrange extended function by the Lagrange multiplier optimization method:

[0086]

[0087] where W i =▽ i X-▽ i P, M=ΔX-ΔP, H j =▽ j X, A1 is the Lagrange multiplier multiplier of W1 in the X direction, A2 is the Lagrange multiplier multiplier of W2 in the Y direction, B is the Lagrange multiplier multiplier of M, C1 is the Lagrange multiplier multiplier of H1 in the X direction, C2 is the Lagrange multiplier multiplier of H2 in the Y direction, C3 is the Lagrange multiplier multiplier of H3 in the Z direction, ψ and η are penalty parameters, i={1,2}, j={1,2,3}. Since the solution of problem (8) is very time-consuming and difficult, problem (8) is decomposed into five sub-problems of W i , M, N, H j and X using an alternating minimum iteration framework. The specific details are as follows:

[0088]

[0089] The five sub-problems in formula (9) are iteratively solved, and the solutions of W i , M, N, H j and X after the (k+1)th iteration are obtained:

[0090]

[0091] where is a half-threshold iterative solution algorithm for solving l 1 / 2 norm, where is the local minimum value of y. Soft(a,b):=sign(a)·max(|a|-b,0) is a soft threshold convergence formula for solving l1 norm. The superscript T is the transpose of the matrix. f(·) and f -1 (·) represent fast Fourier transform and its inverse transform, respectively. The multiplier multipliers A, B and C in formula (10) are updated as follows each iteration:

[0092]

[0093] A single solution to equation (9) represents one iteration of the alternating minimization framework, which decomposes the complex minimization problem (8) into five simpler subproblems. The convergence theorem assumes that each variable is convex, guaranteeing the convergence of the entire algorithm. The relative change (RelCha) and the number of iterations are used as the termination conditions of the algorithm. RelCha is defined as:

[0094]

[0095] The iteration terminates when RelCha < 0.01.

[0096] S3: Near-infrared and visible light video fusion task:

[0097] This invention extends the proposed fusion framework to visible and near-infrared video fusion. Compared to near-infrared and visible image pair fusion, near-infrared and visible video fusion has another key factor: temporal coherence. Specifically, the extracted features should follow continuous motion trajectories between frames to facilitate the inference of information lost during degradation. Based on these considerations, the optimization function for video fusion has also undergone the following changes: a temporal coherence term is introduced into formula (8) to extract detailed features from adjacent frames of the video, enhancing the structural consistency between frames. As follows:

[0098]

[0099] Where X = {X1, X2, ..., X} n}, Y = {Y1, Y2, ..., Y} n}, P = {P1, P2, ..., P} n} and N = {N1, N2, ..., N n Let} represent the estimated fusion result sequence, the input visible light frame sequence, the near-infrared frame sequence, and the estimated noise map sequence, respectively, and t={1,...,n}, where n represents the total number of input frames, X t Y t P t N t Represents the fusion result of any frame in the input video stream, the input visible light frame, the near-infrared frame, and the estimated noise map, respectively. ▽X t 、▽F t,o (X o ) and ▽P t They represent X respectively t F t,o (X o ) and P tFirst order gradient of X and Y, i.e. ∇X t = ∇1X t + ∇2X t , ∇F t,o (X o ) = ∇1F t,o (X o ) + ∇2F t,o (X o ), ∇P t = ∇1P t + ∇2P t , F t,o (X o ) represents the position correspondence between X t and X o . Note that n equals 1 when processing image pairs.

[0100] By introducing four auxiliary variables W i , M, H j , V, the optimization problem of (13) is changed into a Lagrangian extended function by Lagrange multiplier optimization method:

[0101]

[0102] The solution process of formula (14) is similar to formula (8). Here, because an auxiliary variable V is added, the problem (14) is decomposed into six sub-problems using an alternating minimum iteration framework, which are W i , M, N, H j , V and X t sub-problems. Among the six sub-problems, the solution process of W i , M, N, H j sub-problems is exactly the same as the four sub-problems in formula (9). The solution process of V and X t is as follows:

[0103]

[0104] The five sub-problems in formula (15) are solved iteratively. After the (k+1)th iteration, the solutions of V and X t are obtained:

[0105]

[0106] The update of the multiplier multipliers A, B, C and D in formula (16) in each iteration is as follows:

[0107]

[0108] The embodiment of the present application takes the near-infrared and visible light images captured in the low light / foggy scene as the research object, and proposes a new near-infrared and visible light image fusion method based on multi-order super Laplace prior, so as to realize fine extraction of image details in the foggy / low light scene. Considering the structural inconsistency caused by the incompatibility of the denoising and fusion tasks, a noise map is designed to prevent the structure or details in the fused image from being smoothed. Then, a unified near-infrared and visible light image fusion framework is established, and the final fused image is estimated by solving iteratively. In addition, a time coherence term is introduced into the established framework to extract the detail features from the adjacent frames, so as to enhance the structural consistency between the frames and realize the near-infrared and visible light video fusion task.

[0109] With reference to Figure 2 and Figure 3 , Figure 2 and Figure 3 are the near-infrared and visible light image fusion results obtained by the second embodiment of the present application in the foggy and low light scenes, respectively. As can be seen, in the low light and foggy scenes, the method proposed by the second embodiment of the present application has good effect in the near-infrared and visible light image fusion. The fused image obtained by the embodiment of the present application can effectively retain the details of the image and filter out the noise in the low light environment.

[0110] With reference to Figure 4 , the near-infrared and visible light image fusion system proposed by the embodiment of the present application comprises a memory 10, a processor 20, and a computer program stored in the memory 10 and executable on the processor 20, wherein the processor 20 implements the steps of the near-infrared and visible light image fusion method proposed by the embodiment when executing the computer program.

[0111] The specific working process and working principle of the near-infrared and visible light image fusion system of the embodiment can refer to the working process and working principle of the near-infrared and visible light image fusion method of the embodiment.

[0112] The above is only the preferred embodiment of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A near-infrared and visible light image fusion method characterized by, The method comprises: acquiring a near-infrared image and a visible light image; constructing a regularization term, the regularization term comprising a data fidelity term, a multi-order hyper-Laplacian prior term and a noise mapping term, wherein the specific formula of the multi-order hyper-Laplacian prior term in the regularization term is: wherein f MHLP represents a multi-order hyper-Laplacian prior term, X is an optimal fused image to be solved, P is a near-infrared image, and ∇1X and ∇2X represent first-order gradients of X in X and Y directions, respectively, ∇1P and ∇2P represent first-order gradients of P in X and Y directions, respectively, ΔX and ΔP represent second-order gradients of X and P, respectively, and α1, α2 and γ are penalty parameters, ||·|| is an l 1 / 2 norm, and a specific formula of the noise mapping term is: 1 / 2 ​ wherein f TV represents the noise mapping term, N represents the estimated noise map, β1, β2, β3 and δ are regularization parameters, ▽1X, ▽2X and ▽3X represent the first-order gradients of the to-be-solved optimal fused image X in the X direction, the Y direction and the Z direction, respectively, ||·||1 represents the l1 norm, ||·||2 represents the l2 norm, and ||·||∞ represents the l∞ norm. F represents the l F norm; based on the regularization term, constructing a fusion model for near-infrared image and visible light image fusion, wherein the specific formula of the fusion model is: wherein Y is a degraded image collected in a harsh environment; solving the optimal value of the fusion model to obtain an optimal fusion image.

2. The near-infrared and visible light image fusion method according to claim 1, characterized in that, Solving the optimal value of the fusion model to obtain an optimal fusion image: the optimization problem of solving the optimal value of the fusion model is changed into a Lagrange extended function through a Lagrange multiplier optimization method; The Lagrangian extended function is decomposed into five sub-problems, W i , M, N, H j and X, using an alternating minimization iterative framework, specifically:

1. W sub-problem: where W i = ∇ i X - ∇ i P, M = ΔX - ΔP, H j = ∇ j X, Ai is the Lagrange multiplier multiplier of W1 in the X direction, A2 is the Lagrange multiplier multiplier of W2 in the Y direction, B is the Lagrange multiplier multiplier of M, Ci is the Lagrange multiplier multiplier of Hi in the X direction, C2 is the Lagrange multiplier multiplier of H2 in the Y direction, C3 is the Lagrange multiplier multiplier of H3 in the Z direction, ψ and η are penalty parameters, i = {1, 2}, j = {1, 2, 3}; The five sub-problems are solved iteratively, and after the (k+1)th iteration, W i , M, N, H j and X are obtained, where the iteration termination condition is specifically: where X k and X k+1 respectively represent the solution of X after the kth and (k+1)th iteration. obtaining the optimal fusion image according to the solution of X obtained after the (k+1)th iteration.

3. The near-infrared and visible light image fusion method according to any one of claims 1-2, characterized in that, the near-infrared image is specifically a near-infrared frame contained in a near-infrared video, and the visible light image is specifically a visible light frame contained in a visible light video.

4. The near-infrared and visible light image fusion method according to claim 3, characterized in that, The specific formula of the fusion model is: Where X = {X1, X2, ..., X} n }, Y = {Y1, Y2, ..., Y} n }, P = {P1, P2, ..., P} n } and N = {N1, N2, ..., N n Let} represent the estimated fusion result sequence, the input visible light frame sequence, the near-infrared frame sequence, and the estimated noise map sequence, respectively, and t={1,...,n}, where n represents the total number of input frames, X t Y t P t N t Represents the fusion result of any frame in the input video stream, the input visible light frame, the near-infrared frame, and the estimated noise map, respectively. ▽X t 、▽F t,o (X o ) and ▽P t They represent X respectively t F t,o (X o ) and P t Find the sum of the first-order gradients in the X and Y directions, i.e., ▽X t =▽1X t +▽2X t , ▽F t,o (X o )=▽1F t,o (X o )+▽2F t,o (X o ), ▽P t =▽1P t +▽2P t F t,o (X o ) indicates the establishment of X t and X o The positional correspondence between them.

5. A near-infrared and visible light image fusion system, the system comprising: a memory (10), a processor (20), and a computer program stored on the memory (10) and executable on the processor (20), characterized in that the processor (20) implements the steps of the method of any one of claims 1 to 4 when executing the computer program.

Images