Training method of image fusion model, image fusion method and related device
By acquiring noise-free visible light and infrared images, performing layered image fusion, and training an image fusion model, the problem of low training sample quality is solved, and the performance of the image fusion model is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-05-08
AI Technical Summary
Existing deep learning algorithms suffer from low-quality training samples in the fusion of visible light and infrared images, leading to decreased model performance and difficulty in obtaining noise-free visible light and infrared images that meet the needs of the human eye.
Noise-free visible light and infrared images are acquired, and their layers are combined and fused using image layering technology to obtain a high-quality sample fused image. This sample fused image is then used to train an image fusion model.
The performance of the image fusion model has been improved, generating high-quality fused images of visible light and infrared images.
Smart Images
Figure CN121998846A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image technology, and in particular to a training method for an image fusion model, an image fusion method, and related apparatus. Background Technology
[0002] With the development of deep learning algorithms, they are finding increasingly wider applications in practice. In the field of image processing, deep learning algorithms consume more computational power but achieve better processing results compared to traditional algorithms. However, deep learning algorithms require the construction and training of training samples. The quality of these training samples directly determines the effectiveness of the deep learning algorithm.
[0003] Supervised learning requires paired training samples. Its advantage is that the trained network performs better in practical applications, but its disadvantage is the increased workload due to the need to construct paired samples. Deep learning algorithms for fusing visible light and infrared images require hardware to directly acquire noisy visible light images, noisy infrared images, and a fused image of noiseless visible light and infrared images. However, it is generally difficult to acquire a fused image of noiseless visible light and infrared images that meets human visual requirements because, in reality, the human eye can only perceive wavelengths of 390-780 nanometers, and is essentially unable to perceive the infrared band. Directly using infrared-transmitting filters to acquire visible light images containing infrared information does not meet human visual requirements because such images suffer from color casts and abnormal brightness, leading to reduced training sample quality and consequently affecting the performance of deep learning algorithms for fusing visible light and infrared images. Summary of the Invention
[0004] This invention provides a training method for an image fusion model, an image fusion method, and related apparatus. The method of this application can obtain a high-quality fused image of visible light and infrared images, thereby improving the performance of the image fusion model.
[0005] To solve the above-mentioned technical problems, the first technical solution adopted by the present invention is: to provide a training method for an image fusion model, comprising: Acquire a first visible light image and a first infrared image, wherein the first visible light image and the first infrared image are noise-free images; Based on image features, the first visible light image is layered to obtain a visible light image layer combination, and the first infrared image is layered to obtain an infrared image layer combination. The visible light image layer combination and the infrared image layer combination are fused to obtain a sample fused image; The initial image fusion model is trained using the sample fusion image to obtain the image fusion model.
[0006] To solve the above-mentioned technical problems, the second technical solution adopted by the present invention is: to provide an image fusion method, the image fusion method comprising: An image fusion model is used to process the visible light image and the infrared image to be fused, thereby obtaining a fused image; wherein the image fusion model is trained by any of the methods described above.
[0007] To solve the above-mentioned technical problems, the third technical solution adopted by the present invention is: to provide an electronic terminal, the electronic terminal including a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory, and the processor being used to execute program data to implement the steps in the training method of the image fusion model described above or the steps in the image fusion method described above.
[0008] To solve the above-mentioned technical problems, the fourth technical solution adopted by the present invention is: to provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, to implement the steps in the training method of the image fusion model or the steps in the image fusion method described above.
[0009] The beneficial effects of this invention are as follows: Unlike existing technologies, the provided image fusion model training method includes: acquiring a first visible light image and a first infrared image, wherein the first visible light image and the first infrared image are noise-free images; performing image layering on the first visible light image based on image features to obtain a visible light image layer combination, and performing image layering on the first infrared image to obtain an infrared image layer combination; fusing the visible light image layer combination and the infrared image layer combination to obtain a sample fused image; and training an initial image fusion model using the sample fused image to obtain the image fusion model. The method of this application can obtain a high-quality fused image of visible light and infrared images, and using this fused image to train the image fusion model can improve the performance of the image fusion model. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is a flowchart illustrating the first embodiment of the training method for the image fusion model provided by the present invention; Figure 2 This is the present invention. Figure 1 A schematic diagram of the structure of one embodiment of step S13; Figure 3 This is the present invention. Figure 1 A flowchart illustrating an embodiment of step S14; Figure 4 This is a schematic diagram of the structure of the first embodiment of the image fusion model provided by the present invention; Figure 5 This is a schematic diagram of the structure of the second embodiment of the image fusion model provided by the present invention; Figure 6 This is a schematic diagram of the structure of the third embodiment of the image fusion model provided by the present invention; Figure 7 This is a schematic diagram of the framework of an embodiment of the electronic terminal provided by the present invention; Figure 8 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium provided by the present invention. Detailed Implementation
[0012] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0013] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.
[0014] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "more" in this article means two or more objects.
[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0016] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.
[0017] The image fusion model training method provided in this application embodiment can be implemented by a server or terminal alone, or by a server and terminal collaboratively. In some embodiments, the terminal or server can implement the image fusion model training method provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a client that supports virtual scenes, such as a game APP; it can also be a mini-program, that is, a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.
[0018] Generally, visible light images contain a significant amount of noise and suffer from information loss. This is because of insufficient visible light illumination at night; supplementing with visible light can cause eye discomfort. Therefore, infrared illumination is needed to compensate for the loss of visible light information. Consequently, infrared images are typically fused into visible light images, requiring the training of an image fusion model that integrates both. Training this model generally employs supervised training methods, necessitating the acquisition of training samples. Directly utilizing visible light images with infrared information acquired through hardware acquisition does not meet human visual requirements, negatively impacting model training results and degrading model performance.
[0019] In view of this, this application provides a training method for an image fusion model, comprising: acquiring a first visible light image and a first infrared image, wherein the first visible light image and the first infrared image are noise-free images; performing image layering on the first visible light image based on image features to obtain a visible light image layer combination, and performing image layering on the first infrared image to obtain an infrared image layer combination; fusing the visible light image layer combination and the infrared image layer combination to obtain a sample fused image; and training an initial image fusion model using the sample fused image to obtain an image fusion model. The method of this application can obtain a high-quality fused image of the visible light image and the infrared image, and using this fused image to train the image fusion model can improve the performance of the image fusion model.
[0020] To enable those skilled in the art to better understand the technical solution of the present invention, the training method of an image fusion model provided by the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0021] Please see Figure 1 This is a flowchart illustrating the first embodiment of the training method for the image fusion model of this application, specifically including: Step S11: Acquire a first visible light image and a first infrared image, wherein the first visible light image and the first infrared image are noise-free images.
[0022] Specifically, noise-free first visible light image and first infrared image are acquired. It should be noted that the information in the first visible light image and the first infrared image is consistent. Specifically, the first visible light image and the first infrared image with consistent information can be directly acquired using visible light and infrared reflecting prisms, or the visible light image and the infrared image can be acquired separately using a visible light camera and an infrared camera, and then the information can be aligned using a registration algorithm to obtain the first visible light image and the first infrared image.
[0023] Step S12: Perform image layering on the first visible light image based on image features to obtain a visible light image layer combination, and perform image layering on the first infrared image to obtain an infrared image layer combination.
[0024] In one embodiment, the first visible light image is image-layered based on image gradient features to obtain a visible light image layer combination, and the first infrared image is image-layered to obtain an infrared image layer combination; wherein the visible light image layer combination and the infrared image layer combination respectively include at least a portion of a texture image layer, a small edge image layer, a large edge image layer, and a flat image layer.
[0025] It should be noted that flat regions generally lack gradient information, while large edges exhibit significant gradients. Small edges have relatively smaller gradients compared to large edges, and within a given range, there are no similar shapes. Textured regions have relatively small gradient information, and within a given range, similar shapes tend to repeat. Therefore, based on image gradient features, the first visible light image is layered to obtain a visible light texture image layer, a visible light small edge image layer, a visible light large edge image layer, and a visible light flat image layer; similarly, the first infrared image is layered to obtain an infrared texture image layer, an infrared small edge image layer, an infrared large edge image layer, and an infrared flat image layer.
[0026] In the above embodiments, the visible light image layer combination and the infrared image layer combination each include a texture image layer, a small edge image layer, a large edge image layer, and a flat image layer, respectively, as an example. In other embodiments, the visible light image layer combination and the infrared image layer combination may also include only a portion of the texture image layer, the small edge image layer, the large edge image layer, and the flat image layer. For example, it may include only the texture image layer and the flat image layer, or only the texture image layer, the small edge image layer, and the large edge image layer, or only the small edge image layer, the large edge image layer, and the flat image layer. The specific inclusion of these components is not limited.
[0027] Step S13: Fuse the visible light image layer combination with the infrared image layer combination to obtain the sample fused image.
[0028] Based on the above process, the first visible light image is denoted as LV, the visible light texture image layer as LV1, the visible light small edge image layer as LV2, the visible light large edge image layer as LV3, and the visible light flat image layer as LV4. The first infrared image is denoted as LN, the infrared texture image layer as LN1, the infrared small edge image layer as LN2, the infrared large edge image layer as LN3, and the infrared flat image layer as LN4.
[0029] In one embodiment, combined with Figure 2 Step S13 specifically includes: Step S131: Fuse the visible light texture image layer in the visible light image layer combination and the infrared texture image layer in the infrared image layer combination to obtain the first fused image.
[0030] There are four possible texture scenarios in the visible light texture image layer LV1 and the infrared texture image layer LN1: 1. Visible light texture image layer LV1 contains texture information, while infrared texture image layer LN1 does not; 2. Visible light texture image layer LV1 contains texture information, while infrared texture image layer LN1 contains texture information; 3. Visible light texture image layer LV1 does not contain texture information, while infrared texture image layer LN1 does not contain texture information; 4. Visible light texture image layer LV1 does not contain texture information, while infrared texture image layer LN1 contains texture information. In this application, the texture information fusion process primarily uses the texture information from the visible light texture image layer, fusing the texture information from the infrared texture image layer into the visible light texture image layer.
[0031] Understandably, if the infrared texture image layer LN1 does not contain texture information, then it is not necessary to fuse the infrared texture image layer LN1 with the visible light texture image layer LV1. Therefore, the strategy for fusing the visible light texture image layer LV1 and the infrared texture image layer LN1 in this application is as follows: 1. Fuse the visible light texture image layer LV1 containing texture information and the infrared texture image layer LN1 containing texture information; 2. Fuse the visible light texture image layer LV1 without texture information with the infrared texture image layer LN1 containing texture information.
[0032] In one embodiment, when fusing the visible light texture image layer LV1 and the infrared texture image layer LN1, it is necessary to determine the texture fusion weight. Specifically, the ratio of the infrared texture image layer LV1 to the visible light texture image layer LN1 is calculated to obtain the first fusion weight α1. The first fusion weight α1 is calculated as follows: ; in, It is an absolute value operation.
[0033] It should be noted that the visible light texture image layer LV1 and the infrared texture image layer LN1 contain texture values of several pixels, and the calculated first fusion weight α1 contains multiple elements, each of which corresponds to the fusion weight of the corresponding pixel.
[0034] Furthermore, since the dark areas of the visible light texture image layer LV1 have weak texture information, and the infrared texture image layer LN1 exhibits supplementary lighting behavior, the fusion weight can be appropriately increased for the dark areas of the visible light texture image layer LV1. Specifically, the lower the brightness of the visible light image, the larger the second fusion weight; conversely, the higher the brightness of the visible light image, the smaller the second fusion weight.
[0035] Based on this, the brightness of the first visible light image is further divided into multiple brightness intervals. For example, a mean filtering algorithm is used to filter the first visible light image to obtain its brightness. Alternatively, the brightness of the first visible light image can be calculated using a visible light brightness calculation method. The brightness of the first visible light image is further divided into 16 brightness intervals.
[0036] The visible light texture values of the visible light texture image layer and the infrared texture values of the infrared texture image layer are mapped to multiple brightness ranges. Based on the visible light and infrared texture values in each brightness range, the interval fusion weight corresponding to each brightness range is calculated. The interval fusion weights corresponding to all brightness ranges are bilinearly sampled to obtain the second fusion weight.
[0037] Specifically, the average absolute value of the visible light texture values within each brightness interval is calculated, as is the average absolute value of the infrared texture values within each brightness interval. Thus, for each brightness interval, the interval fusion weight is obtained by dividing the average absolute value of the visible light texture values by the average absolute value of the infrared texture values. Furthermore, bilinear sampling is performed on the interval fusion weights corresponding to all brightness intervals to obtain the second fusion weight, denoted as α2.
[0038] Further, the texture fusion weight is obtained based on the first fusion weight and the second fusion weight. Specifically, the first fusion weight α1 and the second fusion weight α2 are combined to obtain the texture fusion weight α. Texture fusion weight α = clip((α1×α2)(0,1)); where clip((α1×α2)(0,1)) means that the result of α1×α2 is restricted to the interval (0,1).
[0039] The visible light texture image layer and the infrared texture image layer are fused based on texture fusion weights to obtain the first fused image. The first fused image is denoted as RE1, RE1=LV1×α+LN1×(1-α).
[0040] Step S132: Fuse the visible light small edge image layer in the visible light image layer combination and the infrared small edge image layer in the infrared image layer combination to obtain the second fused image.
[0041] The most direct factor related to edge strength is brightness. Since small edges don't significantly reduce the subjective visual perception of the fused image, the focus is more on integrating image information. Therefore, the brightness of the first visible light image is used as the weight of the infrared small edge image layer, and the brightness of the first infrared image is used as the weight of the visible light small edge image layer. Based on the weights of the infrared and visible light small edge image layers, the visible light and infrared small edge image layers are fused to obtain the second fused image. Specifically, the second fused image is denoted as RE2 and represented as: ; Wherein, Nl represents the brightness of the first infrared image LN, LV2 represents the visible light small edge image layer, Vl represents the brightness of the first visible light image LV, and LN2 represents the infrared small edge image layer.
[0042] Step S133: Fuse the visible light large edge image layer in the visible light image layer combination and the infrared large edge image layer in the infrared image layer combination to obtain the third fused image.
[0043] Because large edges in infrared images can reduce the subjective visual appeal of the fused image, potentially resulting in an unnatural appearance, the fused image produced by the trained image fusion model may also appear unnatural. Therefore, it's necessary to minimize the fusion of large infrared edges. Based on this, the visible light large edge image layer and the infrared large edge image layer are fused using preset weight values to obtain a third fused image. The preset weight values are positively correlated with the proportion of the visible light large edge image layer in the third fused image. Specifically, to reduce the fusion of the infrared large edge image layer, a larger preset weight value can be set. This results in a larger proportion of the visible light large edge image layer and a smaller proportion of the infrared large edge image layer in the resulting third fused image.
[0044] Specifically, the third fused image RE3 is represented as: RE3=LV3×β+LLN3(1-β); where β represents the preset weight value, which is generally set to be close to 1, for example, 0.95.
[0045] Step S134: Fuse the visible light flat image layer in the visible light image layer combination and the infrared flat image layer in the infrared image layer combination to obtain the fourth fused image.
[0046] Typically, the fusion of multiple visible light images does not involve the fusion of flat regions. However, the fusion of visible light and infrared images does involve the fusion of flat regions. The main reason is that when visible light illumination is very weak, to the point that some areas of the captured image have brightness close to zero, even if the texture, small edges, and large edges of the infrared image are fused into the visible light image, this information will not be visible in the final fused image because the brightness is close to zero. In this case, it is necessary to fuse the brightness of the flat areas of the infrared image into the flat areas of the visible light image, primarily for brightening. However, fusing the flat layer information of the infrared image into the visible light image can easily lead to color casts in the final fused image, reducing its visual appeal to the human eye. Therefore, this application fuses the visible light flat image layer and the infrared flat image layer based on image saturation to obtain a fourth fused image.
[0047] Specifically, the visible light image is converted into an HSV image, and the flattening fusion weights are determined based on the saturation features in the HSV image.
[0048] It should be noted that HSV images decompose RGB images into three independent channels: Hue, Saturation, and Value. Hue (H) represents the type of color, such as red, green, and blue; Saturation (S) represents the purity of the color, that is, the proportion of gray in the color; and Value (V) represents the lightness or darkness of the color.
[0049] This application determines the flattening fusion weight based on the saturation features in HSV images. The flattening fusion weight is denoted as λ, and the calculation method for the flattening fusion weight λ is as follows: ; Where clip represents clipping operation, the above formula means to... Limiting to the (0.8,1) range. This indicates that for completely dark areas of visible light, the final fused image will receive up to 20% of the energy of the infrared flat image layer. It is a constant used to adjust the value of λ.
[0050] The visible light flat image layer and the infrared flat image layer are fused based on the flat fusion weight to obtain the fourth fused image. The fourth fused image is denoted as RE4, and RE4 = LV4 × λ + LN4 × (1 - λ).
[0051] Step S135: Fuse the first fused image, the second fused image, the third fused image, and the fourth fused image to obtain the sample fused image.
[0052] Let the sample fused image be denoted as RE, and RE = clip((RE1+RE2+RE3+RE4), (0,1)). This means that the values of RE1+RE2+RE3+RE4 are clipped to the interval (0,1).
[0053] Step S14: Train the initial image fusion model using the sample fusion image to obtain the image fusion model.
[0054] Combination Figure 3 Step S14 specifically includes: Step S31: Acquire a second visible light image and a second infrared image, wherein the second visible light image and the second infrared image are noisy images.
[0055] It is understandable that in practical use, the visible light image input to the image fusion model generally contains a lot of noise, while the infrared image also contains a certain amount of noise.
[0056] It should be noted that the size of the second visible light image is expressed as follows: Where 3 represents the RGB 3 channels, and the size of the second infrared image is expressed as... 1 represents one channel, and H and W represent the height and width of the image, respectively.
[0057] Step S32: Process the second visible light image and the second infrared image using the initial image fusion model to obtain the initial fused image.
[0058] Step S33: Calculate the fusion loss value using the initial fused image and the sample fused image to obtain the image fusion model.
[0059] Generally speaking, visible light images contain a lot of noise and need to be denoised. In some environments with weak infrared illumination, infrared images themselves contain a certain amount of noise and also need to be denoised. However, in some environments with sufficient infrared illumination, infrared image denoising is not necessary.
[0060] In one embodiment, the initial image fusion model includes an image fusion module. For example... Figure 4 As shown, the output of the image fusion model is a fused noise-free image. Therefore, the image fusion module in the initial image fusion model of this embodiment needs to denoise the second visible light image and the second infrared image, and also needs to fuse the denoised second visible light image and the second infrared image to obtain the initial fused image.
[0061] Specifically, a fusion loss function is established using the initial fused image and the sample fused image, and the fusion loss value is calculated. The fusion loss function is expressed as: Formula (1); Where ||1 represents the 1-norm, lossRE represents the fusion loss value, outputRE represents the initial fused image, RE represents the sample fused image, and size represents the number of pixel values.
[0062] Based on the calculated fusion loss value, the backpropagation algorithm is used to converge and adjust the parameters in the initial image fusion model until lossRE < θ1, at which point the training of the image fusion model is complete.
[0063] In one embodiment, the initial image fusion model further includes a visible light noise reduction module connected to the image fusion module, such as... Figure 5 As shown.
[0064] In this embodiment, step S32 includes: denoising the second visible light image using a visible light denoising module to obtain a denoised visible light image; and processing the denoised visible light image and the second infrared image using an image fusion module to obtain an initial fused image. Specifically, the second visible light image is denoised using a visible light denoising module, and the denoised visible light image (outputLV) and the second infrared image output by the visible light denoising module are input into the image fusion module. The image fusion module then processes the denoised visible light image (outputLV) and the second infrared image to obtain the initial fused image.
[0065] In this embodiment, step S33 includes: calculating the visible light denoising loss value using the visible light denoised image and the first visible light image, and calculating the fusion loss value using the initial fused image and the sample fused image, thereby obtaining the image fusion model.
[0066] Specifically, the calculation method for visible light noise reduction loss is expressed as follows: Formula (2); lossv represents the visible light denoising loss value, outputLV represents the visible light denoised image, and LV represents the first visible light image.
[0067] Based on the calculated visible light denoising loss value, the backpropagation algorithm is used to converge and adjust the parameters in the visible light denoising module until lossv < θ2, thus completing the training of the visible light denoising module. The parameters of the visible light denoising module are fixed, and a fusion loss function is established using the above formula (1). The fusion loss value is calculated, and based on the calculated fusion loss value, the backpropagation algorithm is used to converge and adjust the parameters in the image fusion module until lossRE < θ1, thus completing the training of the image fusion module. Then, the visible light denoising module is trained again. When lossv < θ2, the image fusion module is trained again until lossRE < θ1. After a preset number of iterations, the image fusion model is obtained.
[0068] When infrared illumination is sufficient, infrared images do not require noise reduction; therefore, this implementation only includes a visible light noise reduction module. This allows the computing power saved from the infrared noise reduction module to be allocated to the visible light noise reduction module and the fusion module, resulting in better final noise reduction performance.
[0069] In one embodiment, the initial image fusion model further includes a visible light noise reduction module and an infrared noise reduction module that connect the image fusion module, such as... Figure 6 As shown.
[0070] In this embodiment, step S32 includes: denoising the second visible light image using a visible light denoising module to obtain a visible light denoised image; processing the second infrared image using an infrared denoising module to obtain an infrared denoised image; and processing the visible light denoised image and the infrared denoised image using an image fusion module to obtain an initial fused image. Specifically, the second visible light image is denoised using a visible light denoising module, and the second infrared image is denoised using an infrared denoising module; the visible light denoised image outputLV and the infrared denoised image outputLN output by the visible light denoising module are input into the image fusion module, and the image fusion module processes the visible light denoised image outputLV and the infrared denoised image outputLN to obtain the initial fused image.
[0071] Step S33 above includes: calculating the visible light denoising loss value using the visible light denoised image and the first visible light image; calculating the infrared denoising loss value using the infrared denoised image and the first infrared image; and calculating the fusion loss value using the initial fused image and the sample fused image, thereby obtaining the image fusion model.
[0072] Specifically, the visible light denoising loss function is established using the above formula (2), the visible light denoising loss value is calculated, and the parameters in the visible light denoising module are adjusted based on the calculated visible light denoising loss value combined with the backpropagation algorithm until lossRE < θ2, thus completing the training of the visible light denoising module. After completing the training of the visible light denoising module, the parameters of the visible light denoising module are fixed.
[0073] Furthermore, the infrared noise reduction loss value is calculated as follows: Formula (3); lossn represents the infrared noise reduction loss value, outputLN represents the infrared noise-reduced image, and LN represents the first infrared image.
[0074] Based on the calculated infrared denoising loss value, the backpropagation algorithm is used to converge and adjust the parameters in the infrared denoising module until lossn < θ3, at which point the training of the infrared denoising module is complete. After the training of the infrared denoising module is complete, the parameters of the infrared denoising module are fixed.
[0075] Further utilize the above formula (1) to establish a fusion loss function, calculate the fusion loss value, and based on the calculated fusion loss value, combine the backpropagation algorithm to converge and adjust the parameters in the image fusion module until lossRE < θ1, then the training of the image fusion module is completed. Then train the visible light denoising module again, and when lossv < θ2, train the infrared denoising module again, and when lossn < θ3, train the image fusion module again until lossRE < θ1. After a preset number of iterations, the image fusion model is obtained.
[0076] It should be noted that if visible light and infrared light are directly input into the fusion module, the fusion module will inevitably also have to perform noise reduction. This makes it difficult to control the noise reduction intensity and fusion strength during training, and also increases the difficulty of the entire fusion model training convergence. In this embodiment, extracting the noise reduction function from the fusion module reduces the difficulty of model training, and the noise reduction intensity is controllable.
[0077] The training method of the image fusion model in this application can obtain a high-quality fused image of visible light and infrared images, thereby improving the performance of the image fusion model.
[0078] This application further provides an image fusion method based on the image fusion model trained as described above. Specifically, the image fusion model is used to process the visible light image and the infrared image to be fused, thereby obtaining a fused image. Specifically, the image fusion model is trained using the method described above.
[0079] In one embodiment, it can be utilized Figure 4 , Figure 5 and Figure 6 The image fusion model shown in any embodiment executes the image fusion method of this application. Specifically, it inputs the acquired visible light image and infrared image into... Figure 4 , Figure 5 and Figure 6 In any of the image fusion models shown in the embodiments, a fused image can be obtained.
[0080] Please see Figure 7 , Figure 7This is a schematic diagram of the framework of an embodiment of the electronic terminal provided by the present invention. The electronic terminal 80 includes a memory 81 and a processor 82 coupled to each other. The processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-described image fusion model training method embodiments. In a specific implementation scenario, the terminal 80 may include, but is not limited to, a microcomputer, a server, etc. In addition, the terminal 80 may also include mobile devices such as laptops and tablets, which are not limited here.
[0081] Specifically, processor 82 controls itself and memory 81 to implement the steps of any of the above-described image fusion model training method embodiments. Processor 82 can also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 82 can be implemented using integrated circuit chips.
[0082] Please see Figure 8 , Figure 8 This is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present invention. The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor. The program instructions 901 are used to implement the steps of any of the above-described image fusion model training method embodiments.
[0083] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0084] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.
[0085] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0087] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0088] The above are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A training method for an image fusion model, characterized in that, include: Acquire a first visible light image and a first infrared image, wherein the first visible light image and the first infrared image are noise-free images; Based on image features, the first visible light image is layered to obtain a visible light image layer combination, and the first infrared image is layered to obtain an infrared image layer combination. The visible light image layer combination and the infrared image layer combination are fused to obtain a sample fused image; The initial image fusion model is trained using the sample fusion image to obtain the image fusion model.
2. The method according to claim 1, characterized in that, Based on image features, the first visible light image is layered to obtain a visible light image layer combination, and the first infrared image is layered to obtain an infrared image layer combination, including: The first visible light image is layered based on image gradient features to obtain a visible light image layer combination, and the first infrared image is layered to obtain an infrared image layer combination. The visible light image layer combination and the infrared image layer combination respectively include at least some of the texture image layer, small edge image layer, large edge image layer and flat image layer.
3. The method according to claim 2, characterized in that, The step of fusing the visible light image layer combination with the infrared image layer combination to obtain a sample fused image includes: The visible light texture image layer in the visible light image layer combination and the infrared texture image layer in the infrared image layer combination are fused to obtain a first fused image; The visible light small edge image layer in the visible light image layer combination and the infrared small edge image layer in the infrared image layer combination are fused to obtain a second fused image; The visible light large edge image layer in the visible light image layer combination and the infrared large edge image layer in the infrared image layer combination are fused to obtain a third fused image; The visible light flat image layer in the visible light image layer combination and the infrared flat image layer in the infrared image layer combination are fused to obtain a fourth fused image; The first fused image, the second fused image, the third fused image, and the fourth fused image are fused to obtain the sample fused image.
4. The method according to claim 3, characterized in that, The step of fusing the visible light texture image layer in the visible light image layer combination and the infrared texture image layer in the infrared image layer combination to obtain a first fused image includes: The ratio of the infrared texture image layer to the visible light texture image layer is calculated to obtain the first fusion weight; The brightness of the first visible light image is divided into multiple brightness intervals. The visible light texture value of the visible light texture image layer and the infrared texture value of the infrared texture image layer are mapped to multiple brightness intervals. Based on the visible light texture value and infrared texture value in each brightness interval, the interval fusion weight corresponding to each brightness interval is calculated. Bilinear sampling is performed on the interval fusion weights corresponding to all brightness intervals to obtain the second fusion weight. The texture fusion weight is obtained based on the first fusion weight and the second fusion weight; The visible light texture image layer and the infrared texture image layer are fused based on the texture fusion weights to obtain the first fused image.
5. The method according to claim 3, characterized in that, The step of fusing the visible light small edge image layer in the visible light image layer combination and the infrared small edge image layer in the infrared image layer combination to obtain a second fused image includes: The brightness of the first visible light image is used as the weight of the infrared small edge image layer, and the brightness of the first infrared image is used as the weight of the visible light small edge image layer. The visible light small edge image layer and the infrared small edge image layer are fused based on the weights of the infrared small edge image layer and the visible light small edge image layer to obtain the second fused image.
6. The method according to claim 3, characterized in that, The process of fusing the visible light large-edge image layer in the visible light image layer combination and the infrared large-edge image layer in the infrared image layer combination to obtain a third fused image includes: The visible light large edge image layer and the infrared large edge image layer are fused based on a preset weight value to obtain the third fused image, wherein the preset weight value is positively correlated with the proportion of the visible light large edge image layer in the third fused image.
7. The method according to claim 3, characterized in that, The process of fusing the visible light flat image layer in the visible light image layer combination and the infrared flat image layer in the infrared image layer combination to obtain a fourth fused image includes: The first visible light image is converted into an HSV image, and the flattening fusion weights are determined based on the saturation features in the HSV image. The visible light flat image layer and the infrared flat image layer are fused based on the flat fusion weights to obtain the fourth fused image.
8. The method according to any one of claims 1 to 7, characterized in that, The initial image fusion model includes an image fusion module; The initial image fusion model is trained using the fused sample image to obtain the image fusion model, which includes: Acquire a second visible light image and a second infrared image, wherein the second visible light image and the second infrared image are noisy images; The second visible light image and the second infrared image are processed using an initial image fusion model to obtain an initial fused image; The image fusion model is obtained by calculating the fusion loss value using the initial fused image and the sample fused image.
9. The method according to claim 8, characterized in that, The initial image fusion model includes a visible light noise reduction module connected to the image fusion module; the step of processing the second visible light image and the second infrared image using the initial image fusion model to obtain the initial fused image includes: The visible light image is denoised using the visible light denoising module to obtain a denoised visible light image. The image fusion module is used to process the visible light denoised image and the second infrared image to obtain the initial fused image; The steps of calculating the fusion loss value using the initial fused image and the sample fused image to obtain the image fusion model include: The visible light denoising loss value is calculated using the visible light denoised image and the first visible light image, and the fusion loss value is calculated using the initial fused image and the sample fused image, thereby obtaining the image fusion model.
10. The method according to claim 8, characterized in that, The initial image fusion model includes a visible light noise reduction module and an infrared noise reduction module that connect the image fusion module; The steps of processing the second visible light image and the second infrared image using an initial image fusion model to obtain an initial fused image include: The visible light image is denoised using the visible light denoising module to obtain a visible light denoised image; and the infrared denoising module is used to process the second infrared image to obtain an infrared denoised image. The image fusion module is used to process the visible light denoised image and the infrared denoised image to obtain the initial fused image; The steps of calculating the fusion loss value using the initial fused image and the sample fused image to obtain the image fusion model include: The visible light denoising loss value is calculated using the visible light denoised image and the first visible light image; the infrared denoising loss value is calculated using the infrared denoised image and the first infrared image; and the fusion loss value is calculated using the initial fused image and the sample fused image, thereby obtaining the image fusion model.
11. An image fusion method, characterized in that, The image fusion method includes: An image fusion model is used to process the visible light image to be fused and the infrared image to be fused, thereby obtaining a fused image; wherein the image fusion model is trained by the method described in any one of claims 1 to 10.
12. An electronic terminal, characterized in that, The electronic terminal includes a memory and a processor coupled to each other. The processor is used to execute program instructions stored in the memory and to execute program data to implement the steps in the training method of the image fusion model as described in any one of claims 1 to 10 or the steps in the image fusion method as described in claim 11.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps in the training method of the image fusion model as described in any one of claims 1 to 10 or the steps in the image fusion method as described in claim 11.