Multi-focus image fusion method and device

Abstract

The invention provides a multi-focus image fusion method and device. The method comprises the following steps: acquiring a plurality of local focus images to be fused; performing noise prediction on the plurality of local focusing images by using a noise prediction network to obtain noise, and performing conversion processing on the noise to obtain hidden variables of the fused image; and obtaining a fused image by using the hidden variables. The device comprises an image acquisition part which is configured to acquire a plurality of local focusing images to be fused; the noise prediction part is configured to perform noise prediction on the plurality of local focusing images by using the trained noise prediction network to obtain noise obtained by the noise prediction, and perform conversion processing on the noise obtained by the noise prediction to obtain hidden variables of the fused image; and the image fusion part is configured to obtain a fused image by using the hidden variables. According to the method and the device, the principle of a diffusion model is utilized, so that rich information in multiple local focusing images can be fully utilized, and the multi-focusing image fusion effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multi-focus image fusion method and a device thereof. Background Art

[0002] In real life, when people use cameras to shoot, they hope to obtain clear images of all the objects in the same scene. However, due to the limitation of depth of field, the camera lens cannot focus on all targets at the same time, so some areas of the photos taken are clear and some areas are blurred. Multi-focus image fusion technology can fuse multiple images with different focus areas in the same scene into a fully clear image, thereby effectively solving this problem and improving the information utilization of the image. Accurately identifying and extracting the focus area in the source image is a difficult problem in the multi-focus image fusion algorithm. If the focus area is not completely extracted, artifacts and edge contours will appear in the fusion result.

[0003] Under different parameter settings, the information contained in images collected from different imaging devices or a single type of sensor is one-sided and inaccurate. These images contain different characteristic information such as color, brightness, contour, etc. The reorganization of this information can help to describe all targets in the scene more clearly and comprehensively. The goal of image fusion technology is to extract the characteristic information that people need from all source images, and then concentrate all this information on one image. In this way, people can obtain all the characteristic information they need from one picture and eliminate the interference of redundant information, which is convenient for human eye observation or subsequent image processing, such as identity authentication, target recognition, image enhancement, etc. The fused image obtained by this technology is clearer than a single source image and contains more comprehensive information. Therefore, the realization of image fusion technology is of great significance.

[0004] In the process of taking photos everyday, general imaging equipment is limited by the depth of field of the optical system, and it is difficult to clearly display all the information in the scene in the acquired image. That is to say, in this scene, if one of the targets is in focus, then the other targets are out of focus. However, a clear image of all the targets in the scene is very important for human visual perception and computer processing. Multi-focus image fusion technology can effectively solve this problem. It can fuse two or more images with different focus areas from the same scene to generate a single fully focused image, so that all targets in the same scene are clearly presented.

[0005] Multi-focus image fusion is a method of fusing two or a group of images focused on different targets into a fully focused image, which can solve this problem to a certain extent. However, different targets often have different scales and detailed textures, making them difficult to identify. Secondly, the edge blur at the junction of the focused and defocused areas will cause the fused image to lose a lot of information and cannot be fully restored, so the current fusion method still has many defects. Summary of the invention

[0006] The main technical problem solved by the present invention is how to improve the accuracy of the fused image obtained by the existing multi-focus image fusion method. The present invention provides a multi-focus image fusion method and a device thereof, which can realize fast and high-precision image fusion.

[0007] According to the first aspect, an embodiment provides a multi-focus image fusion method, including:

[0008] Acquire multiple local focused images to be fused; wherein the multiple local focused images are all obtained by shooting the same scene, and the focus areas of the multiple local focused images are different;

[0009] Using the trained noise prediction network to perform noise prediction on the multiple local focused images to obtain noise obtained by the noise prediction, and performing conversion processing on the noise obtained by the noise prediction to obtain latent variables of the fused image;

[0010] A fused image is obtained using the latent variables.

[0011] In one embodiment, the noise prediction of the multiple local focused images is performed using the trained noise prediction network to obtain the noise obtained by the noise prediction, and the noise obtained by the noise prediction is converted to obtain the latent variable of the fused image, including:

[0012] Inputting the multiple local focused images and the total number of diffusion steps T into the noise prediction network, so as to use the noise prediction network to perform T times of noise prediction and conversion processing on the multiple local focused images;

[0013] In the first noise prediction and conversion process, the multiple local focused images are input into the noise prediction network to perform noise prediction to obtain the noise predicted by the first noise prediction; the noise predicted by the first noise prediction is converted to obtain the latent variable of the fused image after the first noise prediction;

[0014] In the t-th noise prediction and conversion process, the latent variables of the multiple local focused images and the fused image after the t-1-th noise prediction are input into the noise prediction network to perform noise prediction to obtain the noise predicted by the t-th noise prediction; the noise predicted by the t-th noise prediction and the latent variables of the fused image after the t-1-th noise prediction are converted to obtain the latent variables of the fused image after the t-th noise prediction, where t=2,…,T.

[0015] In one embodiment, the converting of the noise obtained by the t-th noise prediction and the latent variables of the fused image after the t-1-th noise prediction to obtain the latent variables of the fused image after the t-th noise prediction includes:

[0016] The hidden variables of the fused image after the t-th noise prediction are converted by using the following expression:

[0017]

[0018] Among them, S1,…,S n represents the multiple local focus images, n represents the number of local focus images, ε θ (S1,…,S n , F t , t) represents the noise predicted by the t-th noise, F t represents the latent variable of the fused image after the t-1th noise prediction, α t Equal to 1-β t , β t represents a hyperparameter, 0<β t <1, F t-1 Represents the latent variable of the fused image after the t-th noise prediction.

[0019] In one embodiment, the step of obtaining a fused image by using the latent variable includes:

[0020] The latent variable of the fused image after the T-th noise prediction is taken as the final fused image.

[0021] In one embodiment, the noise prediction network includes three downsampling modules, two upsampling modules and a flattening module connected in sequence; each downsampling module includes two first convolutional layers and a downsampling layer connected in sequence; each upsampling module includes two second convolutional layers and an upsampling layer connected in sequence; the flattening module includes a third convolutional layer, and the flattening module is used to flatten the input multi-dimensional data into one-dimensional data;

[0022] The step of inputting the latent variables of the multiple local focused images and the fused image after the t-1th noise prediction into the noise prediction network to perform noise prediction and obtain the noise predicted by the tth noise prediction includes:

[0023] The latent variables of the multiple local focused images and the fused image after the t-1th noise prediction are used as the input feature map of the first downsampling module; the output feature map of the downsampling module is used as the input feature map of the first first convolutional layer, the output feature map of the first first convolutional layer is used as the input feature map of the second first convolutional layer, the output feature map of the second first convolutional layer is used as the input feature map of the downsampling layer, and the output feature map of the downsampling layer is used as the output feature map of the downsampling module; the output feature map of the previous downsampling module is used as the input feature map of the next downsampling module;

[0024] The input feature map of the upsampling module is used as the input feature map of the first second convolutional layer, the output feature map of the first second convolutional layer is used as the input feature map of the second second convolutional layer, the output feature map of the second second convolutional layer is used as the input feature map of the upsampling layer, and the output feature map of the upsampling layer is used as the output feature map of the upsampling module; the output feature map of the first upsampling module is used as the input feature map of the second upsampling module;

[0025] The third convolutional layer flattens the output feature map of the second upsampling module to obtain the noise predicted by the tth noise.

[0026] In one embodiment, the trained noise prediction network is obtained through the following training process:

[0027] A group of training data is randomly selected from the training data set, and the group of training data, the number of training noise additions N, and the total number of training diffusion steps T are input into the noise prediction network; the all-focus label image is subjected to noise addition processing using high-dimensional noise to obtain the all-focus label image after noise addition processing;

[0028] Wherein, N=T, the set of training data includes a plurality of training local focus images and fully focused label images corresponding to the plurality of training local focus images;

[0029] The noise prediction network performs noise prediction on the multiple training local focus images and the all-focus label image after the noise addition process to obtain the training noise obtained by the noise prediction, and performs conversion process on the training noise obtained by the noise prediction to obtain the training latent variable of the fused image;

[0030] The noise prediction network is trained according to a loss function to obtain the trained noise prediction network; wherein the loss function is determined by the difference between the training noise and the high-dimensional noise.

[0031] According to the second aspect, an embodiment provides a multi-focus image fusion device, comprising:

[0032] An image acquisition unit is configured to acquire a plurality of local focus images to be fused; wherein the plurality of local focus images are all obtained by photographing the same scene, and the focus areas of the plurality of local focus images are different;

[0033] A noise prediction unit is configured to use a trained noise prediction network to perform noise prediction on the multiple local focus images to obtain noise obtained by the noise prediction, and perform conversion processing on the noise obtained by the noise prediction to obtain a latent variable of the fused image;

[0034] The image fusion unit is configured to obtain a fused image using the latent variables.

[0035] In one embodiment, the noise prediction of the multiple local focused images is performed using the trained noise prediction network to obtain the noise obtained by the noise prediction, and the noise obtained by the noise prediction is converted to obtain the latent variable of the fused image, including:

[0036] Inputting the multiple local focused images and the total number of diffusion steps T into the noise prediction network, so as to use the noise prediction network to perform T times of noise prediction and conversion processing on the multiple local focused images;

[0037] In the first noise prediction and conversion process, the multiple local focused images are input into the noise prediction network to perform noise prediction to obtain the noise predicted by the first noise prediction; the noise predicted by the first noise prediction is converted to obtain the latent variable of the fused image after the first noise prediction;

[0038] In the t-th noise prediction and conversion process, the latent variables of the multiple local focused images and the fused image after the t-1-th noise prediction are input into the noise prediction network to perform noise prediction to obtain the noise predicted by the t-th noise prediction; the noise predicted by the t-th noise prediction and the latent variables of the fused image after the t-1-th noise prediction are converted to obtain the latent variables of the fused image after the t-th noise prediction, where t=2,…,T.

[0039] According to the third aspect, an embodiment provides a computer-readable storage medium, characterized in that it includes a program, and the program can be executed by a processor to implement the method as described in any embodiment of the present invention.

[0040] According to a fourth aspect, in one embodiment, a computer program product is provided, comprising a computer program and / or instructions, wherein when the computer program and / or instructions are executed by a processor, the method described in any one of the embodiments herein is implemented.

[0041] The beneficial effects of this application are:

[0042] The multi-focus image fusion method of the present application comprises: obtaining a plurality of local focus images to be fused; wherein the plurality of local focus images are all obtained by shooting the same scene, and the focus areas of the plurality of local focus images are different; using a trained noise prediction network to perform noise prediction on the plurality of local focus images to obtain the noise obtained by the noise prediction, and converting the noise obtained by the noise prediction to obtain the latent variable of the fused image; and obtaining the fused image using the latent variable; the multi-focus image fusion method of the present application utilizes the principle of the diffusion model, and can make full use of the rich information in the plurality of local focus images, and finally improve the effect (such as accuracy) of the multi-focus image fusion; compared with the traditional single focus image deblurring, the multi-focus image (such as the plurality of local focus images) has richer information, and the multi-focus image fusion needs to extract the respective clear parts from different local focus images and fuse them into one image, and similar to observing images at different scales, it can better understand the relative position and distance information between objects, and thus more accurately reflect the depth, and the diffusion model can fully retain and utilize these detail information during the reconstruction process to generate a fused high-quality image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a multi-focus image fusion method according to an embodiment;

[0044] Figure 2 A schematic diagram of a process for obtaining latent variables of a fused image according to an embodiment;

[0045] Figure 3 is a schematic diagram of multiple local focused images and a final fused image according to an embodiment;

[0046] Figure 4 A schematic diagram of a training process of a noise prediction network according to an embodiment;

[0047] Figure 5 The figure is a schematic diagram of a module of a multi-focus image fusion device according to an embodiment. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0049] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0050] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0051] The application of multi-focus image fusion technology is very extensive. In the medical field, virus cell images observed at different focal lengths can be fused together to help doctors make more reliable and accurate disease diagnosis. In the field of digital photography, images with different focus areas can be fused so that the fused image contains enhanced spatial resolution and scene information. This method is conducive to the detection of terrain and the investigation and analysis of land resources. In the military field, the rapid development of science and technology has made modern warfare evolve into a war of information technology. The image information received by different sensors is fused, which greatly improves the target detection and tracking capabilities of fighter jets. Fusion of images with different focus areas can fuse information into a fully focused image to meet the needs of intelligence collection and identity authentication, and speed up the transmission of information. In the field of machine vision and autonomous driving, multi-focus image fusion technology can provide more accurate and comprehensive visual information by fusing images at different focal points, which is helpful for tasks such as target detection, obstacle recognition and path planning.

[0052] Multi-focus image fusion is a method of fusing two or a group of images focused on different targets into a fully focused image, which can solve this problem to a certain extent. However, different targets often have different scales and detailed textures, making them difficult to identify. Secondly, the edge blur at the junction of the focused and defocused areas will cause the fused image to lose a lot of information and cannot be fully restored, so the current fusion method still has many defects. In order to eliminate the impact caused by these defects, make the image more suitable for human visual perception and computational analysis, and improve the efficiency of subsequent digital image processing, this application adopts a multi-focus image fusion method based on a diffusion model, which can achieve fast and high-precision image fusion.

[0053] The technical solution of the present application will be described in detail below in conjunction with embodiments.

[0054] Please refer to Figure 1 The present application provides a multi-focus image fusion method. The multi-focus image fusion method includes:

[0055] Step S100: Acquire multiple local focus images to be fused;

[0056] Step S200: using a noise prediction network to perform noise prediction on multiple local focus images to obtain noise, and performing conversion processing on the noise to obtain a latent variable;

[0057] Step S300: Obtain a fused image using latent variables.

[0058] Specifically, step S100: a plurality of local focus images are obtained by photographing the same scene, and the focus areas of the plurality of local focus images are different;

[0059] Specifically, step S200: using the trained noise prediction network to perform noise prediction on multiple local focus images to obtain noise obtained by the noise prediction, and performing conversion processing on the noise obtained by the noise prediction to obtain latent variables of the fused image.

[0060] In some embodiments, in step S100, those skilled in the art may obtain multiple local focus images to be fused according to actual needs.

[0061] It is understandable that the focus area of ​​an image is usually affected by the following factors: aperture size, distance from the lens, and / or focal length of the lens. Focal length refers to the distance from the optical center of the lens to the focus point of the image. The angle of view refers to the range of the picture that the lens can contain when shooting. The focus area refers to the range of the lens focus. The latent variables of the fused image represent those variables that are not directly observed in the image fusion process, and these variables play a key role in image fusion. In image fusion technology, latent variables usually refer to the information whether the corresponding pixels in the two images match. These latent variables are unobserved random variables, but they play an important role in the image fusion process. By using the ET algorithm (Expectation-Taxitization algorithm, which is a basic algorithm and the basis of many algorithms in the field of machine learning), the objective function can be maximized to find the best way to match the two images, thereby achieving image fusion. In addition, latent variables also play an important role in image generation models. In image generation models, latent variables usually refer to those variables that are not directly observed. They play the role of encoders in the model, mapping high-dimensional image data to a low-dimensional latent space. By learning the distribution of latent variables, the model can generate new image samples that have similar characteristics in the latent space but show diversity in the original image space. The use of latent variables helps to reduce the dimensionality of the data, so that a large number of observed variables can be integrated into a latent variable to represent deep concepts, making the observed data easier to understand.

[0062] Please refer to Figure 2 In some embodiments, in step S200, the noise prediction of multiple local focus images is performed using a trained noise prediction network to obtain the noise obtained by the noise prediction, and the noise obtained by the noise prediction is converted to obtain the latent variable of the fused image, including:

[0063] The multiple local focused images and the total diffusion step number T are input into the noise prediction network, so as to use the noise prediction network to predict the multiple local focused images (such as S1, ..., S n ) performs T times of noise prediction and conversion processing;

[0064] In step S210, in the first noise prediction and conversion process, the multiple local focus images (such as S1, ..., S n ) is input into the noise prediction network to perform noise prediction and obtain the noise predicted by the first noise prediction (such as ε θ (S1,…,S n , t)); the noise predicted by the first noise (such as ε θ (S1,…,S n , t)) is converted to obtain the latent variables of the fused image after the first noise prediction (such as FT );

[0065] Step S220: In the t-th noise prediction and conversion process, the multiple local focus images (such as S1, ..., S n ) and the latent variables of the fused image after the t-1th noise prediction (such as F t ) is input into the noise prediction network to perform noise prediction and obtain the noise predicted by the tth noise (such as ε θ (S1,…,S n , F t , t)); the noise predicted by the t-th noise (such as ε θ (S1,…,S n , F t , t)) and the latent variables of the fused image after the t-1th noise prediction (such as F t ) is converted to obtain the latent variables of the fused image after the t-th noise prediction (such as F t-1 ), where t=2,…,T.

[0066] In some embodiments, in step S210, a person skilled in the art determines the total number of diffusion steps T according to actual needs. The total number of diffusion steps is usually equal to the number of training noise additions when training the noise prediction network. For example, the total number of diffusion steps T can be equal to 2000. θ (S1,…,S n , t)) is converted, which can be the noise predicted by the first noise (such as ε θ (S1,…,S n , t)) is directly used as the latent variable of the fused image after the first noise prediction (such as F t ).

[0067] In some embodiments, the noise prediction network includes three down-sampling modules, two up-sampling modules and a flattening module connected in sequence; each down-sampling module includes two first convolutional layers and a down-sampling layer connected in sequence; each up-sampling module includes two second convolutional layers and an up-sampling layer connected in sequence; the flattening module includes a third convolutional layer, and the flattening module is used to flatten the input multi-dimensional data into one-dimensional data; wherein the step S220 of iteratively predicting noise: the multiple local focused images (such as S1, ..., S n ) and the latent variables of the fused image after the t-1th noise prediction (such as F t ) is input into the noise prediction network to perform the t-th noise prediction and obtain the noise predicted by the t-th noise (such as ε θ (S1,…,S n , F t , t..)), including:

[0068] Step S221: The multiple local focus images (such as S1, ..., S n ) and the latent variables of the fused image after the t-1th noise prediction (such as F t ) as the input feature map of the first downsampling module; the output feature map of the downsampling module is used as the input feature map of the first first convolutional layer, the output feature map of the first first convolutional layer is used as the input feature map of the second first convolutional layer, the output feature map of the second first convolutional layer is used as the input feature map of the downsampling layer, and the output feature map of the downsampling layer is used as the output feature map of the downsampling module; the output feature map of the previous downsampling module is used as the input feature map of the next downsampling module;

[0069] Step S222: using the input feature map of the upsampling module as the input feature map of the first second convolutional layer, using the output feature map of the first second convolutional layer as the input feature map of the second second convolutional layer, using the output feature map of the second second convolutional layer as the input feature map of the upsampling layer, and using the output feature map of the upsampling layer as the output feature map of the upsampling module; using the output feature map of the first upsampling module as the input feature map of the second upsampling module;

[0070] Step S223: The third convolutional layer flattens the output feature map of the second upsampling module to obtain the noise predicted by the tth noise (such as ε θ (S1,…,S n , F t , t)).

[0071] It should be noted that the first convolution layer can be a 3×3 convolution layer, the downsampling layer can be implemented by 2×2 maximum pooling, etc., the second convolution layer can be a 3×3 convolution layer, the upsampling layer can be implemented by 2×2 deconvolution, etc., and the third convolution layer can be a 1×1 convolution layer. The above-mentioned first convolution layer, downsampling layer, second convolution layer, upsampling layer and third convolution layer all belong to the prior art in this field, and those skilled in the art can make adjustments according to actual needs.

[0072] In some embodiments, in step S221, the scale of the input feature map of the subsequent downsampling module is half the scale of the output feature map of the previous downsampling module, and the channel dimension of the input feature map of the subsequent downsampling module is twice the channel dimension of the output feature map of the previous downsampling module.

[0073] In some embodiments, in step S222, the scale of the input feature map of the second upsampling module is twice the scale of the output feature map of the first upsampling module, and the channel dimension of the input feature map of the second upsampling module is half the channel dimension of the output feature map of the first upsampling module.

[0074] In some embodiments, the noise prediction network can adopt an improved U-Net network. U-Net adopts an encoder-decoder structure, in which the encoder part is responsible for extracting the features of the image, and the decoder part is responsible for mapping these features back to the original image size, while using skip connections. This structure not only makes U-Net very effective in extracting rich features, but also because of the use of data enhancement technology, it is particularly suitable for small sample learning. Thus, good performance can be achieved in image generation tasks. For example, the noise prediction network includes an encoder, a decoder, and a skip connection. The encoder is used to extract features from the received feature map layer by layer, the encoder includes three downsampling modules connected in sequence, and the decoder is used to restore image information from the received feature map layer by layer. The encoder includes three downsampling modules connected in sequence, the decoder includes two upsampling modules connected in sequence, the encoder and the decoder are connected through two convolutional layers, and each first convolutional layer and the second convolutional layer are connected to an activation function (such as a ReLU function) to improve the expression ability of the model. Among them, the output feature map of the second convolution layer in each downsampling module is channel-joined with the output feature map of the upsampling layer in the corresponding upsampling module after trimming (to achieve the fusion of shallow information and deep information). The output feature map of the second convolution layer in each downsampling module is transmitted to the decoder through a skip connection, and after trimming, it is channel-joined with the output feature map of the upsampling layer in the corresponding upsampling module to achieve the fusion of shallow information and deep information, providing more semantic information for the decoding process.

[0075] In some embodiments, in step S223, the output feature map of the second upsampling module can be converted into a 2D channel feature map using a third convolutional layer, and the 2D channel feature map is used as the noise predicted by the t-th noise (e.g., ε θ (S1,…,S n , F t , t..)).

[0076] Since multi-focus image fusion is a relatively deterministic process, no random noise is added during the image generation process to avoid the randomness of the fused image. Therefore, in some embodiments, in the step S230 of iteratively generating latent variables, the noise predicted by the tth noise (such as ε θ (S1,…,S n , F t , t)) and the latent variables of the fused image after the t-1th noise prediction (such as F t ) is converted to obtain the latent variables of the fused image after the t-th noise prediction (such as F t-1 ),include:

[0077] The noise predicted by the t-th noise (such as ε θ (S1,…,S n , F t , t)) and the latent variables of the fused image after the t-1th noise prediction (such as F t ) is converted to obtain the latent variables of the fused image after the t-th noise prediction (such as F t-1 ):

[0078]

[0079] Among them, S1,…,S n represents multiple local focus images, n represents the number of local focus images, ε θ (S1,…,S n , F t , t) represents the noise predicted by the tth noise, F t represents the latent variable of the fused image after the t-1th noise prediction, α t Equal to 1-β t , β t represents a hyperparameter, 0<β t <1, F t-1 Represents the latent variable of the fused image after the t-th noise prediction.

[0080] In some embodiments, the latent variable after any noise prediction before the Tth noise prediction may be used as the final fused image.

[0081] In some embodiments, in step S300, preferably, obtaining a fused image using latent variables includes:

[0082] Step S310: taking the latent variable (such as F0) of the fused image after the T-th noise prediction as the final fused image.

[0083] Please refer to Figure 3 , Figure 3This is a fusion example of the multi-focus image fusion method; wherein, Figure (a) is the left focused image (i.e., one of the multiple local focused images mentioned above), the edges of the "small clock" and the "table" are clear, and the "big clock" is blurred; Figure (b) is the right focused image (i.e., one of the multiple local focused images mentioned above), the "big clock" is very clear, while the "small clock" and the "table" outside the focused area are blurred; Figure (c) is the final fused image. In order to make the image reflect the scene information more comprehensively, clearly, and truly, the focus area in the source image is extracted by the multi-focus image fusion method, and then the new image (i.e., Figure (c)) is recombined, i.e., the final fused image. The fused image is more conducive to human eye recognition and computer analysis and processing. The fusion result contains the clear targets in the different focused images, which is a fully clear image, and the description of the scene is more accurate and reliable. It can not only improve the information utilization rate and spatial resolution of the image, improve the accuracy of target detection in a certain scene, but also reduce the amount of information transmission and reduce the requirements for storage and communication.

[0084] For some examples, please refer to Figure 4 , the trained noise prediction network is obtained through the following training process:

[0085] Step 100: randomly select a set of training data from the training data set, input the set of training data, the number of training noise additions N and the total number of training diffusion steps T into the noise prediction network; use high-dimensional noise to add noise to the fully focused label image to obtain a fully focused label image after noise addition (such as F N ); wherein N=T, the set of training data includes a plurality of training local focus images and fully focused label images corresponding to the plurality of training local focus images;

[0086] Step 200: The noise prediction network performs noise prediction on the multiple training local focus images (such as S1, ..., S n ) and the all-focus label image after the noise processing (such as F N ) performing noise prediction to obtain training noise obtained by the noise prediction, and performing conversion processing on the training noise obtained by the noise prediction to obtain training latent variables of the fused image;

[0087] Step 300: Train the noise prediction network according to the loss function to obtain a trained noise prediction network; wherein the loss function is a function of the training noise (such as ε θ (S1,…,S n , F t , t)) and high-dimensional noise (such as ε N-t )Sure.

[0088] In some embodiments, in step 100, the training data set includes multiple sets of training data. Multiple training local focus images and fully focused label images corresponding to the multiple training local focus images can be obtained from cameras with different focal lengths to form a set of training data. It can be understood that the fully focused label image is used as a reference value (ground truth) when the noise prediction network performs supervised learning.

[0089] Since the relationship between the focus area of ​​the image and the aperture, lens distance and focal length is mainly reflected in the concept of depth of field. The depth of field refers to the clear area from front to back in the photo, where the image is very clear. The depth of field is determined by three main factors: the aperture size, the distance between the lens and the subject, and the focal length of the lens. Therefore, in some embodiments, in step 100, multiple image acquisition elements (such as cameras, etc.) or the same image acquisition element can be used to shoot all targets in the same scene at different focal lengths to obtain multiple training local focus images. The above-mentioned multiple image acquisition elements may have different focal lengths and viewing angles when shooting the same scene, and different focal lengths and viewing angles can enable each image acquisition element to capture different depths of field and details of the same scene.

[0090] In some embodiments, in step 100, the aperture size of the image acquisition element may be adjusted and all objects in the same scene may be photographed to obtain a plurality of training local focus images.

[0091] In some embodiments, in step 100, a plurality of training local focus images may be obtained by adjusting the distance between the lens and the object in the image acquisition element and photographing all targets in the same scene.

[0092] In some embodiments, in step 100, the focal length, aperture size and / or distance between the lens and the subject in the image acquisition element can be adjusted to photograph all targets in the same scene to obtain multiple training local focus images.

[0093] In some embodiments, in step 100, the all-focus label image can be obtained by existing technologies such as defocus image superposition and fusion.

[0094] In some embodiments, in step 100, the all-focused label image may also be directly obtained by other existing image acquisition devices.

[0095] In some embodiments, in step 100, the number of training noise additions N and the total number of training diffusion steps T are usually equal. Those skilled in the art can determine the number of training noise additions N and the total number of training diffusion steps T according to actual needs. For example, the number of training noise additions N can be equal to 2000, and the noise variance change function in the form of cosine can be used for noise addition in the noise addition process.

[0096] It should be noted that it can be understood that in step 100, "using high-dimensional noise to perform noise processing on the all-focus label image to obtain the noise-processed all-focus label image (such as F N )” is actually equivalent to the first step of the existing diffusion model, that is, the pre-defined forward diffusion process: gradually adding high-dimensional noise to the fully focused label image until a pure noise is finally obtained. Therefore, “using high-dimensional noise to add noise to the fully focused label image to obtain the fully focused label image after adding noise (such as F N )” can directly use the existing iterative noise addition technology. For example, the all-focus label image is processed by adding noise using high-dimensional noise to obtain the all-focus label image after noise addition (such as F N ), including: using the all-focused label image as the feature map after the 0th noise addition; in the tth noise addition, adding noise to the feature map after the t-1th noise addition using the following expression: Among them, β t represents the intensity of the high-dimensional noise added during the t-th noise addition, x t-1 represents the feature map after the t-1th noise addition, x t represents the feature map after the tth noise addition, ε 1，t represents the high-dimensional noise corresponding to the t-th noise addition; where β t The expression is: Among them, β1 represents the preset intensity of the high-dimensional noise added during the first noise addition, T max represents the total number of iterative noise addition steps, Indicates the preset intensity of the high-dimensional noise added during the last noise addition; where 1≤t≤T max , T max >1, x0 represents the feature map after the 0th noise addition; As the all-focus label image after noise processing (such as F N ).

[0097] It can be understood that in step 100, “using high-dimensional noise to perform noise processing on the all-focus label image to obtain the noise-processed all-focus label image (such as F N )” That is, the real full-focus image is gradually added with Gaussian white noise to gradually obtain pure noise F n The purpose of noise processing is: because in the algorithm based on the diffusion model, Gaussian white noise is first added to form a noise map, and then reverse iteration is performed to gradually restore the required fused image.

[0098] In some embodiments, step 200: the noise prediction network performs noise prediction on the plurality of training local focus images (such as S1, ..., S n ) and the all-focus label image after the noise processing (such as FN ) to obtain the training noise obtained by the noise prediction, and convert the training noise obtained by the noise prediction to obtain the training latent variable of the fused image, including:

[0099] The multiple training local focus images and the noise-added full focus label image (such as F N ) and the total number of training diffusion steps T are input into the noise prediction network, so as to use the noise prediction network to perform the multiple training local focus images (such as S1, ..., S n ) and the all-focus label image after the noise processing (such as F N ) performs T times of noise prediction and conversion processing;

[0100] Wherein, 210: in the first noise prediction and conversion process, the multiple training local focus images and the noise-added full focus label image are input into the noise prediction network to perform noise prediction to obtain the training noise (such as ε θ (S1,…,S n , F N , t)); the training noise predicted by the first noise (such as ε θ (S1,…,S n , F N , t)) is converted to obtain the training latent variables (such as F T );

[0101] 220: In the t-th noise prediction and conversion process, the training latent variables (such as F t ) is input into the noise prediction network to perform noise prediction and obtain the training noise (such as ε θ (S1,…,S n , F t , t)); the training noise predicted by the t-th noise (such as ε θ (S1,…,S n , F t , t)) and the training latent variables of the fused image after the t-1th noise prediction are converted to obtain the training latent variables of the fused image after the tth noise prediction (such as F t-1 ), where t=2,…,T.

[0102] It should be noted that the specific process of step 220 is basically the same as the specific process of the aforementioned step S220, so step 220 will not be repeated here.

[0103] In some embodiments, the loss function is the training noise (such as ε N-t) and the training noise predicted by the Tt-th noise (such as ε θ (S1,…,S n , F t , Tt)) is determined by the difference between them (such as the difference between the two).

[0104] It should be noted that the specific process of "training the noise prediction network according to the loss function" in step 300 is the prior art in this field.

[0105] It can be seen that in some embodiments, the multi-focus image fusion method of the present application utilizes a diffusion model, fully utilizes the rich information in the image, and improves the effect of multi-focus image fusion.

[0106] It should be noted that the reason why the multi-focus image fusion method of the present application can "make full use of the rich information in the image and improve the fusion of multi-focus images" is that compared with the traditional single-focus image deblurring, the multi-focus image has richer information. Multi-focus image fusion requires extracting the clear parts of each from images of different focal lengths and merging them into one image, which is similar to observing images at different scales. It can better understand the relative position and distance information between objects, and thus more accurately reflect the depth. The diffusion model can fully retain and utilize this detail information during the reconstruction process to generate a high-quality fused image. "Making full use of the rich information in the image" corresponds to the characteristics of the diffusion model in this article that processes images at multiple levels and scales, so that it can extract detail information in the image from different resolutions and scales (that is, the feature maps output by the downsampling module and the upsampling module in the aforementioned noise prediction network). The "rich information" can specifically be edge information, corner information, color information, gradient information, etc.

[0107] It can be seen that in some embodiments, the fusion result of the multi-focus image fusion method of the present application is more globally consistent and more accurately reflects the depth and detail information of the image.

[0108] It should be noted that the reason why the multi-focus image fusion method of the present application can achieve "the fusion result is more globally consistent and more accurately reflects the depth and detail information of the image" is: compared with the traditional single-focus image deblurring, multi-focus image fusion provides image information with multiple depths of field, which can constrain the final image from multiple angles, so that the final fused image is more in line with human visual perception. And "the fusion result is more globally consistent and more accurately reflects the depth and detail information of the image" corresponds to the content of the diffusion model in this article, such as forward diffusion and backward diffusion. By learning the data distribution, the diffusion model can accurately identify the valuable clear areas in the image, retain and fuse them into the final result, while ignoring blurred or redundant information.

[0109] It can be seen that in some embodiments, the multi-focus image fusion method of the present application can be applied to various multi-focus image fusion scenarios and has strong versatility and applicability.

[0110] It should be noted that the reason why the multi-focus image fusion method of the present application "can be applied to various multi-focus image fusion scenarios and has strong versatility and applicability" is that the diffusion model has learned the distribution and characteristics of the image through a large amount of training data and has good generalization ability. This means that it can adapt to different types of images, regardless of the content, texture, brightness and other characteristics of the image, the model can play a good role in the fusion process. The diffusion model can automatically extract the clear parts of the image and fuse them without human intervention or manual adjustment. This makes the model highly applicable. In multi-focus image fusion, the image may appear in different forms of blur due to different focus settings, differences in the optical system or imaging conditions. The diffusion model can handle various types of blur, retain the clear part of each image, and perform seamless fusion.

[0111] It can be seen that in some embodiments, the multi-focus image fusion method of the present application can be widely used in photography, video shooting, virtual reality, augmented reality and other fields, providing an innovative solution for improving image quality and visual experience.

[0112] The above is some description of a multi-focus image fusion method. Please refer to Figure 5 In some embodiments of the present application, a multi-focus image fusion device is also disclosed, including:

[0113] The image acquisition unit 100 is configured to acquire a plurality of local focus images to be fused; wherein the plurality of local focus images are obtained by shooting the same scene, and the focus areas of the plurality of local focus images are different;

[0114] The noise prediction unit 200 is configured to use the trained noise prediction network to perform noise prediction on multiple local focus images to obtain noise obtained by the noise prediction, and perform conversion processing on the noise obtained by the noise prediction to obtain a latent variable of the fused image;

[0115] The image fusion unit 300 is configured to obtain a final fused image using latent variables.

[0116] In some embodiments, a noise prediction is performed on multiple local focus images using a trained noise prediction network to obtain noise obtained by the noise prediction, and the noise obtained by the noise prediction is converted to obtain a latent variable of the fused image, including:

[0117] Inputting the multiple local focused images and the total number of diffusion steps T into a noise prediction network, so as to use the noise prediction network to perform T times of noise prediction and conversion processing on the multiple local focused images;

[0118] In the first noise prediction and conversion process, the multiple local focus images (such as S1, ..., S n ) is input into the noise prediction network to perform noise prediction and obtain the noise predicted by the first noise prediction (ε θ (such as S1, ..., S n , t)); the noise predicted by the first noise (such as ε θ (S1,…,S n , t)) is converted to obtain the latent variables of the fused image after the first noise prediction (such as F T );

[0119] S220: In the t-th noise prediction and conversion process, the multiple local focus images (such as S1, ..., S n ) and the latent variables of the fused image after the t-1th noise prediction (such as F t ) is input into the noise prediction network to perform noise prediction and obtain the noise predicted by the tth noise (such as ε θ (S1,…,S n , F t , t)); the noise predicted by the t-th noise (such as ε θ (S1,…,S n , F t , t)) and the latent variables of the fused image after the t-1th noise prediction are converted to obtain the latent variables of the fused image after the tth noise prediction (such as F t-1 ), where t=2,…,T.

[0120] It should be noted that the specific processing procedures and technical effects of the image acquisition unit, the noise prediction unit and the image fusion unit are basically the same as the specific processing procedures and technical effects of step S100, step S200 and step S300 in the aforementioned multi-focus image fusion method, so they will not be repeated here.

[0121] The above is some description about a multi-focus image fusion device. In some embodiments of the present application, a computer-readable storage medium is also disclosed, including a program, which can be executed by a processor to implement a method as in any embodiment of the present invention.

[0122] In some embodiments of the present application, a computer program product is also disclosed, including a computer program and / or instructions, and when the computer program and / or instructions are executed by a processor, a method as described in any embodiment of the present invention is implemented.

[0123] This document is described with reference to various exemplary embodiments. However, those skilled in the art will recognize that changes and modifications may be made to the exemplary embodiments without departing from the scope of this document. For example, various operating steps and components for performing the operating steps may be implemented in different ways (e.g., one or more steps may be deleted, modified, or incorporated into other steps) depending on the specific application or considering any number of cost functions associated with the operation of the system.

[0124] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. In addition, as understood by those skilled in the art, the principles of this article can be reflected in a computer program product on a computer-readable storage medium, which is pre-installed with a computer-readable program code. Any tangible, non-temporary computer-readable storage medium can be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD to ROT, DVD, Blu Ray disks, etc.), flash memory and / or the like. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer or other programmable data processing device to form a machine, so that these instructions executed on a computer or other programmable data processing device can generate a device that implements a specified function. These computer program instructions can also be stored in a computer-readable memory, which can instruct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory can form a manufactured product, including an implementation device that implements a specified function. Computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device may provide steps for implementing specified functions.

[0125] Although the principles of this invention have been shown in various embodiments, many modifications of structures, arrangements, proportions, elements, materials and components particularly suitable for specific environments and operational requirements can be used without departing from the principles and scope of this disclosure. The above modifications and other changes or modifications will be included in the scope of this invention.

[0126] The foregoing specific description has been described with reference to various embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the present disclosure. Therefore, the consideration of the present disclosure will be illustrative rather than restrictive, and all these modifications will be included in its scope. Similarly, the advantages, other advantages and solutions to the problems of various embodiments have been described above. However, the benefits, advantages, solutions to the problems and any elements that can produce these, or make them more clear, should not be interpreted as critical, necessary or necessary. The term "include" and any other variants used in this article are all non-exclusive inclusions, so that the process, method, article or device including the list of elements not only includes these elements, but also includes other elements that are not explicitly listed or do not belong to the process, method, system, article or device. In addition, the term "coupled" and any other variants used in this article refer to physical connections, electrical connections, magnetic connections, optical connections, communication connections, functional connections and / or any other connections.

[0127] Those skilled in the art will appreciate that many changes may be made to the details of the above-described embodiments without departing from the basic principles of the invention. Therefore, the scope of the present invention should be determined solely by the claims.

Claims

1. A multi-focus image fusion method, characterized in that: include: Acquire multiple local focused images to be fused; wherein the multiple local focused images are all obtained by shooting the same scene, and the focus areas of the multiple local focused images are different; Using the trained noise prediction network to perform noise prediction on the multiple local focused images to obtain noise obtained by the noise prediction, and performing conversion processing on the noise obtained by the noise prediction to obtain latent variables of the fused image; A fused image is obtained using the latent variables.

2. The method according to claim 1, characterized in that The method of using the trained noise prediction network to perform noise prediction on the multiple local focused images to obtain the noise obtained by the noise prediction, and performing conversion processing on the noise obtained by the noise prediction to obtain the latent variable of the fused image includes: Inputting the multiple local focused images and the total number of diffusion steps T into the noise prediction network, so as to use the noise prediction network to perform T times of noise prediction and conversion processing on the multiple local focused images; In the first noise prediction and conversion process, the multiple local focused images are input into the noise prediction network to perform noise prediction to obtain the noise predicted by the first noise prediction; the noise predicted by the first noise prediction is converted to obtain the latent variable of the fused image after the first noise prediction; In the t-th noise prediction and conversion process, the latent variables of the multiple local focused images and the fused image after the t-1-th noise prediction are input into the noise prediction network to perform noise prediction to obtain the noise predicted by the t-th noise prediction; the noise predicted by the t-th noise prediction and the latent variables of the fused image after the t-1-th noise prediction are converted to obtain the latent variables of the fused image after the t-th noise prediction, where t=2,…,T.

3. The method according to claim 2, characterized in that The converting process of the noise predicted by the t-th noise prediction and the latent variables of the fused image after the t-1-th noise prediction to obtain the latent variables of the fused image after the t-th noise prediction includes: The hidden variables of the fused image after the t-th noise prediction are converted by using the following expression: Among them, S1,…,S n represents the multiple local focus images, n represents the number of local focus images, ε θ (S1,…,S n , F t , t) represents the noise predicted by the t-th noise, F t represents the latent variable of the fused image after the t-1th noise prediction, α t Equal to 1-β t , β t represents a hyperparameter, 0<β t <1, F t-1 Represents the latent variable of the fused image after the t-th noise prediction.

4. The method according to claim 3, characterized in that The step of obtaining a fused image by using the latent variables comprises: The latent variable of the fused image after the T-th noise prediction is taken as the final fused image.

5. The method according to claim 2, characterized in that The noise prediction network includes three down-sampling modules, two up-sampling modules and a flattening module connected in sequence; each down-sampling module includes two first convolutional layers and a down-sampling layer connected in sequence; each up-sampling module includes two second convolutional layers and an up-sampling layer connected in sequence; the flattening module includes a third convolutional layer, and the flattening module is used to flatten the input multi-dimensional data into one-dimensional data; The step of inputting the latent variables of the multiple local focused images and the fused image after the t-1th noise prediction into the noise prediction network to perform noise prediction and obtain the noise predicted by the tth noise prediction includes: The latent variables of the multiple local focused images and the fused image after the t-1th noise prediction are used as the input feature map of the first downsampling module; the output feature map of the downsampling module is used as the input feature map of the first first convolutional layer, the output feature map of the first first convolutional layer is used as the input feature map of the second first convolutional layer, the output feature map of the second first convolutional layer is used as the input feature map of the downsampling layer, and the output feature map of the downsampling layer is used as the output feature map of the downsampling module; the output feature map of the previous downsampling module is used as the input feature map of the next downsampling module; The input feature map of the upsampling module is used as the input feature map of the first second convolutional layer, the output feature map of the first second convolutional layer is used as the input feature map of the second second convolutional layer, the output feature map of the second second convolutional layer is used as the input feature map of the upsampling layer, and the output feature map of the upsampling layer is used as the output feature map of the upsampling module; the output feature map of the first upsampling module is used as the input feature map of the second upsampling module; The third convolutional layer flattens the output feature map of the second upsampling module to obtain the noise predicted by the tth noise.

6. The method according to claim 1, characterized in that The trained noise prediction network is obtained through the following training process: A group of training data is randomly selected from the training data set, and the group of training data, the number of training noise additions N, and the total number of training diffusion steps T are input into the noise prediction network; the all-focus label image is subjected to noise addition processing using high-dimensional noise to obtain the all-focus label image after noise addition processing; Wherein, N=T, the set of training data includes a plurality of training local focus images and fully focused label images corresponding to the plurality of training local focus images; The noise prediction network performs noise prediction on the multiple training local focus images and the all-focus label image after the noise addition process to obtain the training noise obtained by the noise prediction, and performs conversion process on the training noise obtained by the noise prediction to obtain the training latent variable of the fused image; The noise prediction network is trained according to a loss function to obtain the trained noise prediction network; wherein the loss function is determined by the difference between the training noise and the high-dimensional noise.

7. A multi-focus image fusion device, characterized in that: include: An image acquisition unit is configured to acquire a plurality of local focus images to be fused; wherein the plurality of local focus images are all obtained by photographing the same scene, and the focus areas of the plurality of local focus images are different; A noise prediction unit is configured to use a trained noise prediction network to perform noise prediction on the multiple local focus images to obtain noise obtained by the noise prediction, and perform conversion processing on the noise obtained by the noise prediction to obtain a latent variable of the fused image; The image fusion unit is configured to obtain a fused image using the latent variables.

8. The multi-focus image fusion device according to claim 7, characterized in that: The method of using the trained noise prediction network to perform noise prediction on the multiple local focused images to obtain the noise obtained by the noise prediction, and performing conversion processing on the noise obtained by the noise prediction to obtain the latent variable of the fused image includes: Inputting the multiple local focused images and the total number of diffusion steps T into the noise prediction network, so as to use the noise prediction network to perform T times of noise prediction and conversion processing on the multiple local focused images; In the first noise prediction and conversion process, the multiple local focused images are input into the noise prediction network to perform noise prediction to obtain the noise predicted by the first noise prediction; the noise predicted by the first noise prediction is converted to obtain the latent variable of the fused image after the first noise prediction; In the t-th noise prediction and conversion process, the latent variables of the multiple local focused images and the fused image after the t-1-th noise prediction are input into the noise prediction network to perform noise prediction to obtain the noise predicted by the t-th noise prediction; the noise predicted by the t-th noise prediction and the latent variables of the fused image after the t-1-th noise prediction are converted to obtain the latent variables of the fused image after the t-th noise prediction, where t=2,…,T.

9. A computer-readable storage medium, characterized in that: The method comprises a program which can be executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or the instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images