Infrared light image and visible light image fusion method and system
By using preprocessing, hierarchical Bayesian model and depth expansion technology in image fusion, the problem of difficult to deal with infrared and visible images is solved, and efficient and accurate image fusion is achieved, reducing computational costs.
Patent Information
- Application Number
- CN202411749412.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-04-25
AI Technical Summary
Traditional image fusion methods are difficult to effectively deal with the heterogeneity of infrared and visible images, resulting in poor fusion results in complex scenarios. Deep learning methods are difficult to use in practical applications due to high computational costs and the need for large amounts of labeled data.
A method of fusion of infrared light images and visible light images is proposed. The forward model is formed by preprocessing images, combined with a layered Bayesian model and depth expansion technology, unstable parameters are determined, and iterated through a minimizing cost function until the parameters converge, and the final fusion image is output.
Effectively dealing with data heterogeneity in complex scenarios, reducing training time and calculation costs, improving the accuracy and interpretability of the fusion results, making the method universal in practical applications.
Smart Images
Figure CN119887536A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method and system for fusing infrared light images with visible light images. Background Art
[0002] Image fusion technology aims to fuse image data from different sensors into an image with richer information and less information loss. Traditional image fusion methods face the following major challenges: Data heterogeneity problem. There are significant differences in imaging principles and data characteristics between infrared and visible light images. Traditional methods often find it difficult to effectively handle these heterogeneities, resulting in unsatisfactory performance of fusion results in complex scenes. Limitations of deep learning methods. In recent years, deep learning methods have made some progress in the field of image fusion, but these methods usually rely on a large amount of training data and lack interpretability. The high computational cost of the training process and the need for a large amount of labeled data limit its universality in practical applications. To this end, it is necessary to propose a method and system for the fusion of infrared and visible light images, which is difficult for traditional image fusion methods to solve the data heterogeneity problem using deep learning methods. Summary of the invention
[0003] Based on this, it is necessary to propose a method and system for fusing infrared images and visible light images, as traditional image fusion methods are difficult to solve the data heterogeneity problem using deep learning methods.
[0004] The present application provides a method for fusing an infrared image and a visible light image, comprising:
[0005] Obtaining an infrared light image of a target object and a visible light image of the target object;
[0006] Pre-registering and normalizing the infrared image to obtain a pre-processed infrared image;
[0007] Pre-registering and normalizing the visible light image to obtain a preprocessed visible light image;
[0008] Forming a forward model based on the preprocessed infrared image and the preprocessed visible light image;
[0009] Using hierarchical Bayesian modeling, the unstable parameters of the forward model were determined;
[0010] Based on minimizing the cost function, unstable parameters are incorporated into the hierarchical Bayesian model;
[0011] Expand the Hierarchical Bayesian model;
[0012] After each iteration, the fused image after the iteration is obtained, and the convergence of the fused image after each iteration is judged until the unstable parameters have convergence;
[0013] Output the fused image after the last iteration.
[0014] The present application provides an infrared image and visible light image fusion system, comprising:
[0015] A server, used to execute the infrared light image and visible light image fusion method;
[0016] The memory is communicatively connected with the server.
[0017] The present application relates to a method and system for fusing infrared images and visible light images. A forward model is formed by preprocessing infrared images and preprocessing visible light images. The unstable parameters of the forward model are determined by combining the Bayesian inference algorithm and the expansion algorithm. Based on the minimization of the cost function, the unstable parameters are incorporated into the hierarchical Bayesian model. The convergence of each iterative fused image is judged until the unstable parameters converge and the fused image of the last iteration is output. In order to process the heterogeneity of data in complex scenes, the infrared image and visible light image fusion method utilizes the advantages of data-driven and model-driven fusion, improves the generalization ability of the model by introducing physical priors, and uses deep expansion technology to reduce training time and computing costs. It solves the problem that traditional image fusion methods are difficult to solve using deep learning methods due to data heterogeneity. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A flowchart of a method for fusing an infrared light image and a visible light image is provided in one embodiment of the present application.
[0019] Figure 2 A logic diagram of a method for fusing an infrared light image and a visible light image provided in one embodiment of the present application.
[0020] Figure 3 A structural diagram of a hierarchical Bayesian model of a method for fusing infrared light images and visible light images provided in one embodiment of the present application.
[0021] Figure 4 A flowchart of an expanded hierarchical Bayesian model of a method for fusing infrared light images and visible light images provided in one embodiment of the present application.
[0022] Figure 5 A structural connection diagram of an infrared light image and visible light image fusion system provided in one embodiment of the present application.
[0023] Reference numerals:
[0024] 100-server; 200-storage. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0026] The present application provides a method for fusing an infrared light image with a visible light image.
[0027] like Figure 1 As shown, in one embodiment of the present application, a method for fusing an infrared light image with a visible light image includes:
[0028] S100, obtaining an infrared light image of a target object and a visible light image of the target object.
[0029] Specifically, there are significant differences in imaging principles and data characteristics between infrared images and visible light images. In order to effectively deal with these heterogeneities, pre-registration and normalization of infrared images and visible light images can improve the accuracy of fusion results in complex scenes.
[0030] S200, pre-registering and normalizing the infrared light image to obtain a pre-processed infrared light image.
[0031] S300, pre-registering and normalizing the visible light image to obtain a pre-processed visible light image.
[0032] S400, forming a forward model based on the preprocessed infrared light image and the preprocessed visible light image.
[0033] Specifically, the forward model has high interpretability and low computational cost. The combination of deep learning methods that automatically learn complex features makes the black box characteristics of the fusion method transparent. Under the premise of maintaining interpretability, the forward model can handle complex scenes while reducing computational costs.
[0034] S500, using a hierarchical Bayesian model, determines unstable parameters of the forward model.
[0035] Specifically, the hierarchical Bayesian model can rely on the prior model, combining the Bayesian inference algorithm with the deep expansion technology, aiming to integrate the advantages of data-driven and model-driven. By introducing the prior model of physical prior, the generalization ability of the fusion method is improved, and the deep expansion technology significantly reduces the training time and computing cost.
[0036] S600, incorporates unstable parameters into the hierarchical Bayesian model based on minimizing the cost function.
[0037] S700, unfolding hierarchical Bayesian model.
[0038] S800, acquiring a fused image after each iteration, and performing convergence judgment on the fused image after each iteration until the unstable parameters have convergence.
[0039] Specifically, by judging the convergence of unstable parameters, the limitations of deep learning methods, such as high computational cost and the need for a large amount of labeled data, are solved. Convergence judgment of the fused image at each iteration reduces the amount of training data used and improves interpretability. The low computational cost and less requirement for labeled data during the training process of the fusion method make the fusion method universal in practical applications.
[0040] An unfolded network is established, wherein the unfolded network includes a cost function module and four neural network modules. The loss function of the unfolded network is iterated with the goal of minimizing the loss function. In each iteration, the cost function is minimized and the outputs of the four neural network modules are minimized.
[0041] S900, outputting the fused image after the last iteration.
[0042] This embodiment relates to a method for fusing infrared light images with visible light images. A forward model is formed by preprocessing infrared light images and preprocessing visible light images. The unstable parameters of the forward model are determined by combining the Bayesian inference algorithm and the expansion algorithm. Based on the minimization of the cost function, the unstable parameters are incorporated into the hierarchical Bayesian model. The convergence of the fused image of each iteration is judged until the unstable parameters have converged, and the fused image of the last iteration is output. In order to deal with the heterogeneity of data in complex scenes, the method for fusing infrared light images with visible light images takes advantage of the fusion of data-driven and model-driven, improves the generalization ability of the model by introducing physical priors, and uses deep expansion technology to reduce training time and computing costs. It solves the problem that traditional image fusion methods are difficult to solve using deep learning methods due to data heterogeneity.
[0043] like Figure 2 As shown, in one embodiment of the present application, S400 includes:
[0044] S411, creating a fused image original image.
[0045] S412, determining a first difference between the preprocessed infrared image and the original fused image.
[0046] S413, determining a second difference between the preprocessed visible light image and the original fused image.
[0047] S414: Generate an initial fused image based on the original fused image, the first difference portion, and the second difference portion.
[0048] Specifically, a forward model is established. Given a pair of pre-registered infrared images gU and the visible light image g V , the fused image original image f is a real but unknown image, which contains thermal information and useful textures, that is, the fused image original image f is an unknown object to be obtained. The infrared image can be represented as the fused image original image f superimposed on the observation noise and modeling uncertainty of the infrared image, and the visible light image can be represented as the fused image original image f superimposed on the observation noise and modeling uncertainty of the visible light image. The infrared image g U and the visible light image g V The expression is as follows:
[0049]
[0050] Where ξ is the observation noise and modeling uncertainty of the first difference part, ∈ is the observation noise and modeling uncertainty of the second difference part. f is the original fused image. g U is an infrared image. V It is a visible light image.
[0051] In one embodiment of the present application, S400 further includes:
[0052] S421, generating a latent variable based on the second difference part.
[0053] S422, introducing latent variables into the original fused image to generate the original fused image to be modeled.
[0054] S423, performing wavelet transform on the original fused image to be modeled to generate a fused image of the forward model.
[0055] S424, determining a modeling uncertainty using the first difference portion and the second difference portion.
[0056] Specifically, the modeling uncertainty is determined and sparse prior information is applied for latent variable estimation.
[0057] S425, generating a forward model based on the fused image, latent variables and modeling uncertainties of the forward model.
[0058] Specifically, based on wavelet transform, this embodiment proposes a hierarchical prior model based on Haar wavelet transform, in which a latent variable Z is introduced as the fusion image f. -level Haar wavelet transform coefficients, while taking into account the modeling error e. The expression of the forward model can be shown as formula 2:
[0059] f=Dz+e Formula 2
[0060] Among them, the latent variable Z represents the sparse approximation of the fused image f -level Haar wavelet transform coefficients, D represents the inverse Haar wavelet transform operator, D -1 It represents the parameters of the inverse Haar wavelet transform operator in the Haar wavelet transform. e represents the modeling uncertainty when applying sparse prior information to estimate the latent variable Z.
[0061] In one embodiment of the present application, S500 includes:
[0062] S511 , establishing Gaussian mixture distributions for the first difference portion and the second difference portion respectively, to predict characteristics of the observation noise of the first difference portion and characteristics of the observation noise of the second difference portion.
[0063] Specifically, the distribution of the features of the first difference portion has uncertainty.
[0064] The distribution of the features of the second difference portion has uncertainty.
[0065] S512, defining the moment when the predicted uncertainty converges as the moment when the image fusion is completed.
[0066] Specifically, the observation noise ∈ and ξ represent uncertainties of different distributions with different characteristics. Therefore, a Gaussian mixture distribution is used to predict these uncertainties.
[0067] S521, establish normal and inverse gamma mixed distribution for modeling uncertainty.
[0068] Specifically, after modeling the normal-inverse gamma mixed distribution of the indefinite amount e, the generated expression is shown in Formula 3:
[0069] p(e|α e ,β e )=N(e|0,v e )IG(v e |α e ,β e ), where V e =diag[v e ]
[0070] Formula 3
[0071] S522, establish normal and inverse gamma mixed distribution for latent variables.
[0072] Specifically, latent variables include principal and marginal.
[0073] The distribution of the latent variable is a mixture of normal and inverse gamma distribution.
[0074] The marginal distribution of the latent variable is the generalized Student-t distribution.
[0075] Specifically, the normal and inverse gamma mixed distribution of the observation noise of ∈ is shown in Formula 4:
[0076]
[0077] The normal and inverse gamma mixed distribution of the observation noise of ξ is shown in Formula 6:
[0078]
[0079] S523, generating a priori models based on the distribution of modeling uncertainties and the distribution of latent variables.
[0080] Specifically, the expression of the prior model is shown in Formula 6:
[0081] p(z|α z , β z )=N(z|0,V z )IG(v z |α z , β z ), where V z =diag[v z ]Formula 6
[0082] S524, define the information applied by the forward model as sparse prior information.
[0083] S525, define the model for latent variable estimation as a priori model.
[0084] S531, generating a likelihood function of the forward model based on the distribution of unstable parameters of the forward model. The unstable parameters include observation noise and modeling uncertainty.
[0085] Specifically, the likelihood function can be expressed as:
[0086]
[0087] S532, using wavelet transform to transform the likelihood function of the forward model and generate a joint posterior probability formula.
[0088] Specifically, the functional expression of the likelihood function of the wavelet transform forward model is shown in Formula 8:
[0089] p(f|z,v e )=N(f|Dz,V e )
[0090] p(f|Z,v ε )=N(f|D Z , v ε ) Formula 8
[0091] The joint posterior probability formula is shown in Formula 9:
[0092] p(f,v ∈ ,v ξ ,v e ,v z |g U ,g V )
[0093] ∝p(g U ,g V |f,v ∈ , v ξ )p(f|z,v e )p(z|v z )p(v z )p(v ∈ )p(v ξ )p(v e )
[0094] ∝p(g V |f,v ∈ )p(g U |f,v ξ )p(f|z,v e )p(z|v z )p(v z )p(v ∈ )p(v ξ )p(v e ).
[0095] Formula 9 is as follows Figure 3 As shown, in one embodiment of the present application, S600 includes:
[0096] S610: Generate a cost function of the initial fused image based on the JMAP algorithm.
[0097] Specifically, the JMAP algorithm is used to infer the fused image f, and the expression of the cost function is shown in Formula 10:
[0098] J JMAP (f,z,v ∈ , v ξ ,v e , v z )=-lnp(f,z,v ∈ , v ξ ,v e , v z |g U ,g V )
[0099] Formula 10
[0100] S620, updating each unstable parameter of the cost function to generate a minimized cost function.
[0101] Specifically, minimizing the cost function of the initial fused image includes the following parameter updating steps for each unstable parameter in turn:
[0102]
[0103]
[0104] like Figure 4 As shown, in one embodiment of the present application, S700 includes:
[0105] S710, using a learning strategy to estimate hyperparameters required to minimize the cost function.
[0106] S720, the expanded network structure of each iteration is regarded as a layer.
[0107] At S730, each layer is parameterized by quantities learned from a training database.
[0108] Specifically, the learned parameterization is task-oriented to minimize the loss function.
[0109] Specifically, we build an unfolded network. We introduce a learning strategy to estimate the hyperparameters (α ∈ , β ∈ ), (α ξ , β ξ ), (α e , β e ), (α z , β z ). The expanded network structure of each iteration is regarded as a layer, and each layer is parameterized by the quantity learned from the training database, with the task of minimizing the loss function as the guide.
[0110] The overall structure of the expanded network is Each layer L k (k∈(1,2,...K) consists of five modules, including a core JMAP module H(.) and four neural network modules NN1, NN2, NN3, NN4.
[0111] Specifically, let K>0 be the number of iterations of the JMAP algorithm, corresponding to the K layers of the neural network architecture. The expanded JMAP of iteration k∈(1,2,…,K-1) can be expressed as:
[0112]
[0113] Among them, for each L K, neural networks NN1, NN2, NN3, NN4 with f k , z k , as input and output respectively The specific structure of the neural network uses a 3x3 convolution kernel to capture local image features. Each convolution layer is followed by GroupNorm for feature normalization. Each convolution layer is followed by a ReLU activation function to ensure that the network can model nonlinear features. Each CNN network is specialized for updating a specific set of parameters.
[0114] In one embodiment of the present application, S800 includes:
[0115] S810, guided by the convergence of the predicted uncertainty, determining the pixel-level difference of the unstable parameters between the fused image and the source image.
[0116] Specifically, in the process of fusion image and source image of the pixel-level difference of the unstable parameters, the intensity loss measure is applied, and the intensity loss measure is the loss cost of minimizing the cost function.
[0117] More specifically, the source images are infrared light images and visible light images.
[0118] S820, determining whether the pixel-level difference of the unstable parameter is convergent.
[0119] S830: If the pixel-level difference of the unstable parameter does not converge, return to the unfolded hierarchical Bayesian model until the pixel-level difference of the unstable parameter converges.
[0120] S840: If the pixel-level difference of the unstable parameters is convergent, then the expansion of the hierarchical Bayesian model is terminated.
[0121] Specifically, update the network parameters. Calculate the error between the intermediate fused image obtained in step 6 and the original image through the loss function, back propagate the error, and update the neural network parameters. The loss function is defined as:
[0122] L=η1L ssim +η2L content
[0123] Among them, L ssim (I f , I s )=1-SSIM(I f ,I s ), L content =L int +η3L texture The intensity loss measures the pixel-level difference between the fused image and the source image and is defined as:
[0124]
[0125] Where H and W are the vertical and horizontal dimensions of the image, I f ,I r and I v They are fused image, infrared image and visible light image respectively. To ensure finer texture details, the texture loss is defined as:
[0126]
[0127] in, is the Sobel operator that captures fine texture details, and ||·|| represents the absolute value operation.
[0128] The present application provides a system for fusing infrared light images and visible light images.
[0129] like Figure 5 As shown, in one embodiment of the present application, a system for fusing infrared light images and visible light images includes a server 100 and a storage 200 .
[0130] The server 100 is used to execute the infrared light image and visible light image fusion method mentioned in any one of the aforementioned embodiments.
[0131] The storage 200 is communicatively connected with the server 100 .
[0132] This embodiment relates to a system for fusing infrared light images and visible light images. The server 100 forms a forward model through preprocessed infrared light images and preprocessed visible light images, and determines the unstable parameters of the forward model by combining the Bayesian inference algorithm and the expansion algorithm. Based on the minimization cost function, the unstable parameters are incorporated into the hierarchical Bayesian model, and the convergence of each iterative fused image is judged until the unstable parameters have converged, and the fused image of the last iteration is output. In order to process the heterogeneity of data in complex scenes, the infrared light image and visible light image fusion method utilizes the fusion advantages of the data and model of the memory 200, improves the generalization ability of the model by introducing physical priors, and uses deep expansion technology to reduce training time and computing costs. It solves the problem that traditional image fusion methods are difficult to solve data heterogeneity using deep learning methods.
[0133] The technical features of the above-described embodiments may be arbitrarily combined, and the execution order of the method steps is not limited. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for fusing infrared light images and visible light images, characterized in that: include: Obtaining an infrared light image of a target object and a visible light image of the target object; Pre-registering and normalizing the infrared image to obtain a pre-processed infrared image; Pre-registering and normalizing the visible light image to obtain a preprocessed visible light image; Forming a forward model based on the preprocessed infrared image and the preprocessed visible light image; Using hierarchical Bayesian modeling, the unstable parameters of the forward model were determined; Based on minimizing the cost function, unstable parameters are incorporated into the hierarchical Bayesian model; Expand the Hierarchical Bayesian model; After each iteration, the fused image after the iteration is obtained, and the convergence of the fused image after each iteration is judged until the unstable parameters have convergence; Output the fused image after the last iteration.
2. The method for fusing infrared light images and visible light images according to claim 1, characterized in that: The forming of a forward model based on the preprocessed infrared light image and the preprocessed visible light image includes: Create the original fused image; Determine a first difference between the preprocessed infrared image and the original fused image; Determine a second difference between the preprocessed visible light image and the original fused image; An initial fused image is generated based on the fused image original image, the first difference portion, and the second difference portion.
3. The method for fusing infrared light images and visible light images according to claim 2, characterized in that: The forming of the forward model based on the preprocessed infrared light image and the preprocessed visible light image also includes: generating a latent variable based on the second difference portion; Introducing latent variables into the original fused image to generate the original fused image to be modeled; Perform wavelet transform on the original fused image to be modeled to generate a fused image of the forward model; Determine a modeling uncertainty using the first difference portion and the second difference portion; A forward model is generated based on the fused image, latent variables, and modeling uncertainties of the forward model.
4. The method for fusing infrared light images and visible light images according to claim 3, characterized in that: The method of using a hierarchical Bayesian model to determine unstable parameters of a forward model includes: Establishing Gaussian mixture distributions for the first difference part and the second difference part respectively to predict the characteristics of the observation noise of the first difference part and the characteristics of the observation noise of the second difference part; The moment when the prediction uncertainty converges is defined as the moment when image fusion is completed.
5. The method for fusing infrared light images and visible light images according to claim 4, characterized in that: The method of using the hierarchical Bayesian model to determine the unstable parameters of the forward model further includes: Normal and inverse gamma mixed distributions were established for modeling uncertainty; Normal and inverse gamma mixture distributions were established for latent variables; Generate a priori models based on the distribution of modeling uncertainties and the distribution of latent variables; Define the information applied by the forward model as sparse prior information; The model for latent variable estimation is defined as the prior model.
6. The method for fusing infrared light images and visible light images according to claim 5, characterized in that: The method of using the hierarchical Bayesian model to determine the unstable parameters of the forward model further includes: Generate a likelihood function of the forward model based on the distribution of unstable parameters of the forward model; unstable parameters include observation noise and modeling uncertainty; The wavelet transform is used to perform the transformation of the likelihood function of the forward model and generate a joint posterior probability formula.
7. The method for fusing infrared light images and visible light images according to claim 6, characterized in that: The method of incorporating unstable parameters into a hierarchical Bayesian model based on minimizing the cost function includes: Based on the JMAP algorithm, the cost function of the initial fused image is generated; Each unstable parameter of the cost function is updated to generate a minimized cost function.
8. The method for fusing infrared light images and visible light images according to claim 7, characterized in that: The unfolded hierarchical Bayesian model comprises: Use a learning strategy to estimate the hyperparameters needed to minimize the cost function; The expanded network structure of each iteration is regarded as a layer; Each layer is parameterized by a quantity learned from the training database.
9. The method for fusing infrared light images and visible light images according to claim 8, characterized in that: The step of judging the convergence of each iterative fusion image until the unstable parameters have convergence includes: Guided by the convergence of the predicted uncertainty, the pixel-level differences of the unstable parameters between the fused image and the source image are determined; Determine whether the pixel-level differences of unstable parameters are convergent; If the pixel-level difference of the unstable parameter does not converge, returning to the unfolded hierarchical Bayesian model until the pixel-level difference of the unstable parameter converges; If the pixel-wise differences of the unstable parameters converge, the expansion of the hierarchical Bayesian model is terminated.
10. An infrared image and visible light image fusion system, characterized in that: include: A server, configured to execute the infrared light image and visible light image fusion method as claimed in any one of claims 1 to 9; The memory is communicatively connected with the server.