Multi-focus image fusion method and device, computer equipment and medium

Abstract

The invention relates to a multi-focus image fusion method and device, computer equipment and a medium. A natural image, a multi-focus image and a pseudo label image corresponding to the multi-focus image are acquired; inputting the multi-focus images into a generator, and training and fusing the multi-focus images by the generator by taking the pseudo-label image as a learning target, and then outputting a full-focus fused image; inputting the natural image and the full-focus fusion image into a discriminator to train and distinguish the difference between the full-focus fusion image and the natural image; defining a first training loss function of the discriminator, and defining a second training loss function of the generator according to the first training loss function; and iteratively training the multi-focus image fusion network model according to the first training loss function and the second training loss function until the minimum values of the first training loss function and the second training loss function are obtained, and constructing a trained multi-focus image fusion network model. The whole scheme can realize accurate multi-focus image fusion.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-focus image fusion method, device, computer equipment and medium. Background Art

[0002] Multi-focus image fusion is an important image processing technology that can combine the clear parts of images from different focal lengths into a new fully focused image. Multi-focus image fusion technology has been widely used in machine vision, digital cameras, target recognition and other fields.

[0003] Taking the application of multi-focus image fusion in nuclear power plant operation and maintenance as an example, in the operation and maintenance of modern nuclear power plants, nuclear fuel assemblies will produce strong radioactivity after combustion, which makes regular inspection and maintenance of them extremely challenging. In practical applications, in order to reduce the impact of radiation on detection equipment and ensure the safety of operators, a telephoto lens is usually used to increase the distance between the camera and the object being measured. Although this method can effectively reduce radiation exposure dose and risk, it also brings new problems, namely, the image depth of field is significantly reduced. When shooting with a telephoto lens, only a very limited area of ​​the camera image can remain clearly focused, and the rest will appear blurred to varying degrees. This limitation greatly restricts the work efficiency and information acquisition capabilities of nuclear power operation and maintenance, especially when a comprehensive assessment of the entire fuel assembly condition is required. In addition, in the complex nuclear power plant environment, the lighting conditions are poor and changeable, which further increases the difficulty of high-quality image acquisition. It is difficult for images shot with a single focus to simultaneously capture the detailed features of the fuel assembly at different depths, resulting in key defects or abnormalities that may be overlooked.

[0004] It can be seen that there is an urgent need for an accurate multi-focus image fusion solution to meet the needs of multi-focus image applications in different scenarios. Summary of the invention

[0005] Based on this, it is necessary to provide an accurate multi-focus image fusion method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a multi-focus image fusion method. The method comprises: Acquire a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image; Inputting the multi-focus image into a generator of a multi-focus image fusion network model, wherein the generator takes the pseudo-label image as a learning target, trains and fuses the multi-focus images, and then outputs a fully focused fused image; Inputting the natural image and the all-focus fusion image into the discriminator of the multi-focus image fusion network model to train the difference between the all-focus fusion image and the natural image; Defining a first training loss function of the discriminator, and defining a second training loss function of the generator according to the first training loss function; The multi-focus image fusion network model is iteratively trained according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, thereby constructing a trained multi-focus image fusion network model.

[0007] In one embodiment, obtaining a pseudo-label image corresponding to the multi-focus image includes: Segmenting the multi-focus image into preset fixed-size grids based on the spatial domain; Calculate the average gradient for each grid area and generate a sharpness mask based on the average gradient; constructing a Gaussian pyramid image of the multi-focus image, and constructing a Laplacian pyramid image of the multi-focus image based on the Gaussian pyramid image; fusing the Laplacian pyramid images layer by layer according to the clarity mask; Starting from the top layer of the Laplacian pyramid image, the Laplacian pyramid image is added to the corresponding layer image of the Gaussian pyramid image by upsampling, so as to gradually reconstruct a fused image and obtain a pseudo label image corresponding to the multi-focus image.

[0008] In one embodiment, the generator is a U-shaped network structure, and the generator includes multiple feature extraction modules, multiple feature fusion modules and multiple feature attention modules. The multiple feature extraction modules are used to extract features from feature maps of input images at different scales, and the multiple feature fusion modules are used to perform feature fusion on feature maps extracted at different scales. The multiple feature attention modules jump-connect the feature extraction modules and feature fusion modules corresponding to two scales, respectively. The feature attention modules are used to process the input feature maps, learn the importance weights in channels and spatial dimensions, and apply the learned weights to the original feature maps.

[0009] In one embodiment, the step of defining a first training loss function of the discriminator and defining a second training loss function of the generator according to the first training loss function comprises: Defining a first training loss function of the discriminator; Define edge information preservation loss function; Generating a pixel difference function of the same image between the all-focus fusion image and the natural image; The pixel difference function, the edge information preservation loss function and the first training loss function are weightedly combined to obtain a second training loss function.

[0010] In one embodiment, the first training loss function defining the discriminator includes: Acquire a constructed image having the same image size as the all-focus fused image; A binary cross entropy loss function of the constructed image and the all-focus fused image is constructed to obtain a first training loss function of the discriminator.

[0011] In one embodiment, defining the edge information preservation loss function includes: Using the Canny operator to perform image edge detection on the full-focus fusion image and the multi-focus image, respectively, to obtain a first edge information feature map corresponding to the full-focus fusion image and a second edge information feature map corresponding to the multi-focus image; An edge information preservation loss function is defined according to the first edge information feature map and the second edge information feature map.

[0012] In a second aspect, the present application also provides a multi-focus image fusion device. The device comprises: An image acquisition module, used to acquire a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image; A generator processing module, configured to input the multi-focus image into a generator of a multi-focus image fusion network model, wherein the generator takes the pseudo-label image as a learning target, trains and fuses the multi-focus images, and then outputs a fully focused fused image; A discriminator processing module, used for inputting the natural image and the all-focus fusion image into the discriminator of the multi-focus image fusion network model to train the difference between the all-focus fusion image and the natural image; A loss function construction module, used to define a first training loss function of the discriminator, and define a second training loss function of the generator according to the first training loss function; A training module is used to iteratively train the multi-focus image fusion network model according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, so as to construct a trained multi-focus image fusion network model.

[0013] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program: Acquire a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image; Inputting the multi-focus image into a generator of a multi-focus image fusion network model, wherein the generator takes the pseudo-label image as a learning target, trains and fuses the multi-focus images, and then outputs a fully focused fused image; Inputting the natural image and the all-focus fusion image into the discriminator of the multi-focus image fusion network model to train the difference between the all-focus fusion image and the natural image; Defining a first training loss function of the discriminator, and defining a second training loss function of the generator according to the first training loss function; The multi-focus image fusion network model is iteratively trained according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, thereby constructing a trained multi-focus image fusion network model.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented: Acquire a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image; Inputting the multi-focus image into a generator of a multi-focus image fusion network model, wherein the generator takes the pseudo-label image as a learning target, trains and fuses the multi-focus images, and then outputs a fully focused fused image; Inputting the natural image and the all-focus fusion image into the discriminator of the multi-focus image fusion network model to train the difference between the all-focus fusion image and the natural image; Defining a first training loss function of the discriminator, and defining a second training loss function of the generator according to the first training loss function; The multi-focus image fusion network model is iteratively trained according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, thereby constructing a trained multi-focus image fusion network model.

[0015] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented: Acquire a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image; Inputting the multi-focus image into a generator of a multi-focus image fusion network model, wherein the generator takes the pseudo-label image as a learning target, trains and fuses the multi-focus images, and then outputs a fully focused fused image; Inputting the natural image and the all-focus fusion image into the discriminator of the multi-focus image fusion network model to train the difference between the all-focus fusion image and the natural image; Defining a first training loss function of the discriminator, and defining a second training loss function of the generator according to the first training loss function; The multi-focus image fusion network model is iteratively trained according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, thereby constructing a trained multi-focus image fusion network model.

[0016] The multi-focus image fusion method, device, computer equipment, storage medium and computer program product described above obtain natural images, multi-focus images, and pseudo-label images corresponding to the multi-focus images; input the multi-focus images to the generator of the multi-focus image fusion network model, and the generator uses the pseudo-label images as learning targets to train and fuse the multi-focus images and output the fully focused fused images; input the natural images and the fully focused fused images to the discriminator of the multi-focus image fusion network model to train and distinguish the differences between the fully focused fused images and the natural images; define the first training loss function of the discriminator, and define the second training loss function of the generator according to the first training loss function; iteratively train the multi-focus image fusion network model according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, and a trained multi-focus image fusion network model is constructed. In the whole scheme, the multi-focus image fusion network model is iteratively trained based on the pseudo-label images corresponding to the multi-focus images, so as to improve the domain adaptability of the model on real-world image data, automatically learn complex feature representations, and realize accurate multi-focus image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A diagram showing an application environment of a multi-focus image fusion method in an embodiment; Figure 2 is a schematic flow chart of a multi-focus image fusion method in one embodiment; Figure 3 This is a schematic diagram of the generator network structure in the multi-focus image fusion network model; Figure 4 It is a schematic diagram of the network structure of the feature extraction module; Figure 5 Schematic diagram of the 2D residual module network structure; Figure 6 This is a schematic diagram of the network structure of the preprocessing module; Figure 7 It is a schematic diagram of the network structure of the downsampling module; Figure 8 Schematic diagram of the network structure of the feature attention module; Fig. 9 Schematic diagram of the 3D residual module network structure; Fig.10 It is a schematic diagram of the network structure of the feature fusion module; Fig.11 Schematic diagram of the network structure of the upsampling module; Fig.12 It is a schematic diagram of the network structure of the post-processing module; Fig.13 is a schematic flow chart of a multi-focus image fusion method in another embodiment; Fig.14 This is a schematic diagram of the multi-focus image fusion network model architecture; Fig.15 It is a sequence of original multi-focus images with a shallow depth of field; Fig.16 This is the effect of the fully focused image fused using the spatial domain algorithm; Fig.17 This is a full-focus image effect diagram fused using the multi-focus image fusion network model trained in this application; Fig.18 is a structural block diagram of a multi-focus image fusion device in one embodiment; Fig.19 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] The multi-focus image fusion method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 sends a multi-focus image fusion request to the server 104, and the server 104 responds to the multi-focus image fusion request to obtain a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image; the multi-focus image is input to the generator of the multi-focus image fusion network model, and the generator takes the pseudo-label image as the learning target, and outputs a fully focused fused image after training and fusion of the multi-focus image; the natural image and the fully focused fused image are input to the discriminator of the multi-focus image fusion network model to train and distinguish the difference between the fully focused fused image and the natural image; define the first training loss function of the discriminator, and define the second training loss function of the generator according to the first training loss function; iteratively train the multi-focus image fusion network model according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, and a trained multi-focus image fusion network model is constructed. Furthermore, the multi-focus image fusion request may also carry an image to be processed, and the server 104 inputs the image to be processed into the trained multi-focus image fusion network model to generate a multi-focus image fusion result, and feeds back the multi-focus image fusion result to the terminal 102. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0020] In one embodiment, Figure 2 As shown, a multi-focus image fusion method is provided, which is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included: S100: Acquire a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image.

[0021] A natural image is a high-quality, clear image that is captured by a camera at one time of a real-world scene. Here, the natural image may be a natural image dataset, which includes multiple natural images. Specifically, the high-quality, clear images in the DIV2K dataset may be used as the natural image dataset.

[0022] A multi-focus image refers to an image in which different areas of the same scene are focused. Specifically, a multi-focus image can be obtained by using a camera to shoot a group of images with different focuses. Specifically, the acquisition of multi-focus images may include the following steps: 1) Install the camera on a stable tripod and ensure that its position remains unchanged throughout the acquisition process. 2) The ideal shooting scene should contain different objects in the near, medium and distant scenes, so as to capture clear and blurred contrast at different focal lengths. 3) After the shooting scene is selected, the focus position of the camera lens is changed by adjusting the focus ring of the camera lens when the camera focal length is fixed. Start from the nearest object and gradually move the focus to the distance, and take a photo after each focus adjustment. Depending on the specific situation of the scene, focus and shoot once at a certain distance (for example, 5cm or 10cm), and take a group (for example, 6) of pictures for each shooting scene. 4) In order to ensure the consistency of all images in a group, when shooting in the same group, the camera needs to set the same exposure parameters (sensitivity, aperture, shutter speed), and the camera and objects in the shooting scene cannot move. 5) To increase the diversity and robustness of the dataset, the above steps can be repeated under different lighting conditions, such as morning, noon, and evening. In addition, the layout of objects in the scene can be changed to obtain more diverse data.

[0023] The pseudo-label image corresponding to the multi-focus image is a fully focused image made based on the multi-focus image. The use of pseudo-label technology effectively expands the scale of training data and solves the data annotation problem of model training. At the same time, in order to improve the domain adaptability of the model on real-world image data, unlike the traditional method of using "fully focused images (Ground Truth Images)" to create "multi-focus images (Source Images)" pseudo-labels, this application uses "multi-focus images (Source Images)" taken in the real world to create pseudo-labels of "fully focused images (Ground Truth Images)", which enhances the domain adaptability of the model in processing real-world images.

[0024] S200: Input the multi-focus image to the generator of the multi-focus image fusion network model, and the generator uses the pseudo-label image as the learning target, trains and fuses the multi-focus images, and outputs a fully focused fused image.

[0025] The multi-focus image fusion network model mainly includes two components: a generator and a discriminator. The multi-focus image is input to the generator of the multi-focus image fusion network model. The task of the generator is to learn how to fuse the multi-focus images into a fully focused image, and this process takes the pseudo-label image as the learning target. The multi-focus image fusion network model can be specifically a network model MultiFocusGAN.

[0026] S300: Inputting the natural image and the all-focus fusion image into the discriminator of the multi-focus image fusion network model to train the difference between the all-focus fusion image and the natural image.

[0027] After the generator outputs the fully focused fused image, it is input into the discriminator together with the natural image. The task of the discriminator is to distinguish whether the input image is a natural image (real sample) or a fully focused fused image generated by the generator (generated sample).

[0028] S400: Define a first training loss function of the discriminator, and define a second training loss function of the generator based on the first training loss function.

[0029] The discriminator receives the fully focused fused image output by the generator and natural images from the Natural Image Dataset , which is used to train the fused image output by the distinguishing generator and natural images The difference. On the basis of obtaining the first training loss function, the second training loss function of the generator is defined according to the first training loss function. Specifically, in the process of defining the second training loss function of the generator according to the training loss function of the discriminator, the loss function of this generator is associated with the loss function of the discriminator to guide the generator to improve its fusion strategy. Usually, the loss function of the generator will encourage the images it produces to be closer and closer to natural images, thereby deceiving the discriminator.

[0030] S500: Iteratively train the multi-focus image fusion network model according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, and construct a trained multi-focus image fusion network model.

[0031] The multi-focus image fusion network model is iteratively trained using the first training loss function and the second training loss function. In each iteration, the parameters of the multi-focus image fusion network model are updated according to the current loss value. The iterative process continues until a certain stop condition is reached, such as the first training loss function and the second training loss function converge to a minimum value. This indicates that the generator is able to produce fully focused fused images that are close enough to natural images, while the discriminator has difficulty distinguishing the difference between them. When the training process is completed, a trained multi-focus image fusion network model is obtained. This model can be used to fuse new multi-focus images into fully focused images.

[0032] The multi-focus image fusion method obtains a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image; the multi-focus image is input to the generator of the multi-focus image fusion network model, and the generator uses the pseudo-label image as the learning target to train and fuse the multi-focus image and output a fully focused fused image; the natural image and the fully focused fused image are input to the discriminator of the multi-focus image fusion network model to train and distinguish the difference between the fully focused fused image and the natural image; the first training loss function of the discriminator is defined, and the second training loss function of the generator is defined according to the first training loss function; the multi-focus image fusion network model is iteratively trained according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, and a trained multi-focus image fusion network model is constructed. In the whole scheme, the multi-focus image fusion network model is iteratively trained based on the pseudo-label image corresponding to the multi-focus image, so as to improve the domain adaptability of the model on real-world image data, automatically learn complex feature representations, and realize accurate multi-focus image fusion.

[0033] In one embodiment, obtaining a pseudo-label image corresponding to a multi-focus image includes: Step 1: Segment the multi-focus image into preset fixed-size grids based on the spatial domain.

[0034] First, the multifocus image is divided into a series of preset fixed-size grid areas. These grid areas can be square or rectangular, and the size is determined according to actual needs. The purpose of segmentation is to divide the image into smaller, easier-to-process parts so that the clarity of each area can be evaluated in the subsequent steps.

[0035] Step 2: Calculate the average gradient for each grid region and generate a sharpness mask based on the average gradient.

[0036] For each grid area, calculate its average gradient. Gradient is an important indicator to measure the sharpness of the image edge, so the average gradient can reflect the clarity of the area. According to the size of the average gradient, a clarity value is assigned to each grid area to generate a clarity mask. Areas with higher clarity values ​​indicate that the area is clearer.

[0037] Step 3: Construct a Gaussian pyramid image of the multi-focus image, and construct a Laplacian pyramid image of the multi-focus image based on the Gaussian pyramid image.

[0038] Gaussian pyramid is an image representation method that is constructed by smoothing and downsampling images at different scales. For multi-focus images, constructing its Gaussian pyramid image can help us analyze the clarity of the image at different scales. Laplacian pyramid is an extension of Gaussian pyramid, which is constructed by calculating the difference between adjacent layers of Gaussian pyramid. Laplacian pyramid image can highlight the detailed information of the image at different scales, which is crucial for subsequent image fusion.

[0039] Step 4: Fuse the Laplacian pyramid images layer by layer according to the clarity mask.

[0040] At each level of the Laplacian pyramid, the sharpest region is selected for fusion according to the sharpness mask.

[0041] This means that at each scale, the region with the highest clarity can be selected to construct the fused image.

[0042] Step 5: Starting from the top layer of the Laplacian pyramid image, the Laplacian pyramid image is added to the corresponding layer image of the Gaussian pyramid image by upsampling to gradually reconstruct the fused image and obtain the pseudo-label image corresponding to the multi-focus image.

[0043] Starting from the top layer of the Laplacian pyramid, the image of the current layer is enlarged to the same size as the Gaussian pyramid image of the next layer by upsampling. Then, the upsampled Laplacian pyramid image is added to the Gaussian pyramid image of the corresponding layer to reconstruct the fused image of that scale. This process is repeated until the size of the original image is reached, thus obtaining the final fused image, that is, the pseudo-label image corresponding to the multi-focus image.

[0044] In practical applications, the acquisition process of the above pseudo-label image (full-focus image) can be implemented based on the multi-focus fusion algorithm in the spatial domain. The overall process includes: 1) First, the image is divided into fixed-size grids based on the spatial domain to calculate the average gradient matrix and generate a mask; 2) Secondly, the Gaussian pyramid and Laplacian pyramid are constructed respectively, and then the Laplacian pyramid images are fused layer by layer according to the mask; 3) Finally, starting from the top layer, the fused image is gradually reconstructed by upsampling and adding the corresponding layer image of the Gaussian pyramid; 4) Each set of “multi-focus images” generates a corresponding pseudo-label image (all-focus image).

[0045] In one embodiment, the generator is a U-shaped network structure, and the generator includes multiple feature extraction modules, multiple feature fusion modules and multiple feature attention modules. The multiple feature extraction modules are used to extract features from feature maps of input images at different scales, and the multiple feature fusion modules are used to fuse features of feature maps extracted at different scales. The multiple feature attention modules jump-connect the feature extraction modules and feature fusion modules corresponding to two scales, respectively. The feature attention modules are used to process the input feature maps, learn the importance weights in channels and spatial dimensions, and apply the learned weights to the original feature maps.

[0046] The generator adopts a U-shaped structure design. Its overall structure is as follows Figure 3 As shown. The left side of the network structure mainly consists of multiple ( Figure 3 The network structure of the feature extraction module is shown in Figure 3. The three feature extraction modules are responsible for extracting features from the feature maps of the multi-focus image at different scales. Figure 4 As shown, the feature extraction module includes multiple 2D residual modules, and the 2D residual modules are as follows Figure 5 As shown in Figure 1. In the generator, the preprocessing module and downsampling module are responsible for converting the dimensions of the input image and feature map to meet the feature extraction module's requirements for the size and number of channels of the input data. At the same time, the downsampling module is also responsible for scaling the size of the image feature map so that the network model can increase the receptive field and capture a wider range of contextual information. The network structure of the preprocessing module is shown in Figure 1. Figure 6 As shown, the network structure of the downsampling module is as follows Figure 7 shown.

[0047] like Figure 3 As shown, a feature attention module is used in the middle of the U-shaped structure of the generator to realize the jump connection on the left and right sides. The feature attention module mainly processes the feature map of the input multi-focus image by learning the importance weights in the channel and spatial dimensions, and then applying these weights to the original feature map to obtain the output feature map after attention weighting. The CBAM (Convolutional Block Attention Module) attention mechanism can be used in the feature attention module. CBAM is a module that combines the channel attention mechanism and the spatial attention mechanism to enhance the feature representation ability of the convolutional neural network (CNN). In addition, the feature attention module also uses a 3D residual module to fuse the input feature map in N dimensions. The network structure of the feature attention module is shown in Figure 8 As shown. The 3D residual module network structure it contains is as follows Fig. 9 shown.

[0048] The right side of the generator network structure mainly consists of multiple ( Figure 3 The three feature fusion modules are respectively responsible for fusing the feature maps extracted at different scales of the input image. The network structure of the feature fusion module is as follows: Fig.10 The upsampling module is responsible for converting the dimension of the input feature map to meet the requirements of the network output and feature fusion modules for the size and number of channels of the input data. The specific network structure is as follows: Fig.11 The post-processing module is responsible for converting the feature map after network fusion into a normalized RGB image. Its specific network structure is as follows: Fig.12 shown.

[0049] In one embodiment, the discriminator network uses the U-Net discriminator (U-Net discriminator with spectral normalization) proposed in Real-ESRGAN. Compared with the vgg and resnet discriminators, the input resolution and output resolution of the unet discriminator are consistent, which is equivalent to the discriminator discriminating each pixel of the input image, and the discriminator is more sophisticated and powerful.

[0050] In one embodiment, if Fig.13 As shown, S400 includes: S420: Define a first training loss function of the discriminator.

[0051] like Fig.14 As shown, the generator is responsible for receiving the "multi-focus image" , and the corresponding pseudo-label image As the learning target, it is used to train fusion and output the "all-focus fusion image" The discriminator receives the fully focused fusion image output by the generator. and natural images from the Natural Image Dataset , which is used to train the fused image output by the distinguishing generator and natural images difference.

[0052] Further, defining the first training loss function of the discriminator includes: obtaining a constructed image having the same image size as the fully focused fused image; constructing a binary cross entropy loss function between the constructed image and the fully focused fused image, and obtaining the first training loss function of the discriminator.

[0053] The constructed image may specifically be an image whose image data is all 0. Construct an image with all image data being 1 , the size of the constructed image is the same as the size of the fully focused fused image. Specifically, when the input of the discriminator is the "fully focused fused image" output by the generator:

[0054] When the input of the discriminator is a high-quality "natural image":

[0055] In the formula, is the binary cross entropy loss function, is the first training loss function of the discriminator.

[0056] S440: Define an edge information preservation loss function.

[0057] Edge information preservation loss function It is used to measure the ability of the generator to retain the edge information of the fused image compared with the original multi-focus images. The addition of the image edge information preservation loss function can encourage the network model to retain as much high-frequency information as possible in all input images.

[0058] In one of the embodiments, defining the edge information preservation loss function includes: using the Canny operator to perform image edge detection on the fully focused fusion image and the multi-focus image, respectively, to obtain a first edge information feature map corresponding to the fully focused fusion image and a second edge information feature map corresponding to the multi-focus image, respectively; and defining the edge information preservation loss function according to the first edge information feature map and the second edge information feature map.

[0059] The Canny operator is used to perform edge detection on the all-focus fusion image and the multi-focus image. The Canny operator is a classic edge detection algorithm that can accurately extract edge information from the image. Through the Canny operator processing, the first edge information feature map corresponding to the all-focus fusion image can be obtained (usually a binary image or grayscale image, in which the edge part is highlighted). Similarly, the second edge information feature map corresponding to the multi-focus image can also be obtained. With the first edge information feature map and the second edge information feature map, the edge information preservation loss function can be defined. This loss function is usually used to measure the difference in edge information between the fused image and the original multi-focus image.

[0060] In practical applications, the edge information loss function The definition is as follows. Among them, It indicates that the Canny operator is used to detect the edge of the input image and output the edge information feature map of the image; N indicates the number of pixels in the image edge feature map.

[0061] .

[0062] S460: Generate a pixel difference function of the same image between the all-focus fusion image and the natural image.

[0063] Defining a function (Charbonnie loss function) is used to characterize the pixel difference between the fully focused fusion image and the natural image. Specifically, the function The definition is as follows:

[0064] In the formula, x is the image and The difference between the corresponding pixels, is a very small positive constant used to prevent the denominator from being zero.

[0065] S480: weighted combination of the pixel difference function, the edge information preservation loss function and the first training loss function to obtain a second training loss function.

[0066] Based on the function obtained before, the second training loss function of the generator is defined as follows:

[0067] in, , , is the weighting coefficient.

[0068] In general, in specific application examples, the multi-focus image fusion method of this application has the following significant effects: 1. Compared with traditional fusion algorithms based on spatial domain and transform domain, the multi-focus image fusion network model trained in this application can more accurately identify the focused area and the non-focused area in the source image, reduce misjudgment and missed judgment, and the generated fused image details are more complete and the edge transition is more natural, realizing the effective integration of global and local information, making the fused image more natural and realistic.

[0069] 2. Compared with the existing deep learning-based fusion algorithms, this application has higher domain adaptability to the input real-world images; by introducing edge information preservation loss, the fused image of the network model can retain as much high-frequency information in the input image sequence as possible; at the same time, the generative adversarial network (GAN) method is used for training, so that the output image of the model based on pseudo-label training is closer to the natural image, and the image fusion transition is more natural.

[0070] like Figure 15~Figure 17 As shown, Fig.15 It is a sequence of original multi-focus images with a shallow depth of field; Fig.16This is the full focus image effect diagram using spatial domain algorithm fusion in traditional technology. Fig.16 There is a very obvious block effect in the Fig.17 This is a full-focus image effect diagram fused using the multi-focus image fusion network model (network model MultiFocusGAN) trained in this application. The fused image is clear and natural.

[0071] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0072] Based on the same inventive concept, the embodiment of the present application also provides a multi-focus image fusion device for implementing the multi-focus image fusion method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more multi-focus image fusion device embodiments provided below can refer to the limitations of the multi-focus image fusion method above, and will not be repeated here.

[0073] In one embodiment, Fig.18 As shown, a multi-focus image fusion device is provided, comprising: An image acquisition module 100 is used to acquire a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image; A generator processing module 200 is used to input the multi-focus image into the generator of the multi-focus image fusion network model, and the generator uses the pseudo-label image as the learning target, trains and fuses the multi-focus images, and then outputs a fully focused fused image; A discriminator processing module 300 is used to input the natural image and the all-focus fusion image into the discriminator of the multi-focus image fusion network model to train the difference between the all-focus fusion image and the natural image; A loss function construction module 400, used to define a first training loss function of the discriminator, and define a second training loss function of the generator according to the first training loss function; The training module 500 is used to iteratively train the multi-focus image fusion network model according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, so as to construct a trained multi-focus image fusion network model.

[0074] In one of the embodiments, the image acquisition module 100 is further used to divide the multi-focus image into preset fixed-size grids based on the spatial domain; calculate the average gradient for each grid area, and generate a clarity mask based on the average gradient; construct a Gaussian pyramid image of the multi-focus image, and construct a Laplacian pyramid image of the multi-focus image based on the Gaussian pyramid image; fuse the Laplacian pyramid image layer by layer based on the clarity mask; starting from the top layer of the Laplacian pyramid image, add the Laplacian pyramid image and the corresponding layer image of the Gaussian pyramid image by upsampling to gradually reconstruct the fused image to obtain a pseudo-label image corresponding to the multi-focus image.

[0075] In one embodiment, the generator is a U-shaped network structure, and the generator includes multiple feature extraction modules, multiple feature fusion modules and multiple feature attention modules. The multiple feature extraction modules are used to extract features from feature maps of input images at different scales, and the multiple feature fusion modules are used to fuse features of feature maps extracted at different scales. The multiple feature attention modules jump-connect the feature extraction modules and feature fusion modules corresponding to two scales, respectively. The feature attention modules are used to process the input feature maps, learn the importance weights in channels and spatial dimensions, and apply the learned weights to the original feature maps.

[0076] In one embodiment, the loss function construction module 400 is also used to define a first training loss function of the discriminator; define an edge information preservation loss function; generate a pixel difference function of the same image between the fully focused fusion image and the natural image; and weightedly combine the pixel difference function, the edge information preservation loss function and the first training loss function to obtain a second training loss function.

[0077] In one of the embodiments, the loss function construction module 400 is also used to obtain a constructed image with the same image size as the fully focused fused image; construct a binary cross entropy loss function between the constructed image and the fully focused fused image to obtain a first training loss function of the discriminator.

[0078] In one of the embodiments, the loss function construction module 400 is also used to use the Canny operator to perform image edge detection on the fully focused fusion image and the multi-focus image, respectively, to obtain a first edge information feature map corresponding to the fully focused fusion image and a second edge information feature map corresponding to the multi-focus image, respectively; based on the first edge information feature map and the second edge information feature map, an edge information preservation loss function is defined.

[0079] Each module in the multi-focus image fusion device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0080] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.19 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store preset data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-focus image fusion method is implemented.

[0081] Those skilled in the art will understand that Fig.19 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0082] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the multi-focus image fusion method when executing the computer program.

[0083] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the multi-focus image fusion method described above is implemented.

[0084] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned multi-focus image fusion method when executed by a processor.

[0085] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0086] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-focus image fusion method, characterized in that: The method comprises: Acquire a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image; Inputting the multi-focus image into a generator of a multi-focus image fusion network model, wherein the generator takes the pseudo-label image as a learning target, trains and fuses the multi-focus images, and then outputs a fully focused fused image; Inputting the natural image and the all-focus fusion image into the discriminator of the multi-focus image fusion network model to train the difference between the all-focus fusion image and the natural image; Defining a first training loss function of the discriminator, and defining a second training loss function of the generator according to the first training loss function; The multi-focus image fusion network model is iteratively trained according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, thereby constructing a trained multi-focus image fusion network model.

2. The method according to claim 1, characterized in that Acquiring a pseudo-label image corresponding to the multi-focus image includes: Segmenting the multi-focus image into preset fixed-size grids based on the spatial domain; Calculate the average gradient for each grid area and generate a sharpness mask based on the average gradient; constructing a Gaussian pyramid image of the multi-focus image, and constructing a Laplacian pyramid image of the multi-focus image based on the Gaussian pyramid image; fusing the Laplacian pyramid images layer by layer according to the clarity mask; Starting from the top layer of the Laplacian pyramid image, the Laplacian pyramid image is added to the corresponding layer image of the Gaussian pyramid image by upsampling, so as to gradually reconstruct a fused image and obtain a pseudo label image corresponding to the multi-focus image.

3. The method according to claim 1, characterized in that The generator has a U-shaped network structure, and includes multiple feature extraction modules, multiple feature fusion modules and multiple feature attention modules. The multiple feature extraction modules are used to extract features from feature maps of input images at different scales respectively, and the multiple feature fusion modules are used to fuse features from feature maps extracted at different scales respectively. The multiple feature attention modules jump-connect the feature extraction modules and feature fusion modules corresponding to two scales respectively, and the feature attention modules are used to process the input feature maps, learn the importance weights in channels and spatial dimensions, and apply the learned weights to the original feature maps.

4. The method according to claim 1, characterized in that The step of defining a first training loss function of the discriminator and defining a second training loss function of the generator according to the first training loss function comprises: Defining a first training loss function of the discriminator; Define edge information preservation loss function; Generate a pixel difference function of the same image between the all-focus fusion image and the natural image; The pixel difference function, the edge information preservation loss function and the first training loss function are weightedly combined to obtain a second training loss function.

5. The method according to claim 4, characterized in that The first training loss function defining the discriminator includes: Acquire a constructed image having the same image size as the all-focus fused image; A binary cross entropy loss function of the constructed image and the all-focus fused image is constructed to obtain a first training loss function of the discriminator.

6. The method according to claim 4, characterized in that Defining the edge information preservation loss function includes: Using the Canny operator to perform image edge detection on the full-focus fusion image and the multi-focus image, respectively, to obtain a first edge information feature map corresponding to the full-focus fusion image and a second edge information feature map corresponding to the multi-focus image, respectively; An edge information preservation loss function is defined according to the first edge information feature map and the second edge information feature map.

7. The method according to claim 1, characterized in that The method further comprises: Multi-focus image fusion is performed based on the trained multi-focus image fusion network model.

8. A multi-focus image fusion device, characterized in that: The device comprises: An image acquisition module, used to acquire a natural image, a multi-focus image, and a pseudo-label image corresponding to the multi-focus image; A generator processing module, configured to input the multi-focus image into a generator of a multi-focus image fusion network model, wherein the generator takes the pseudo-label image as a learning target, trains and fuses the multi-focus images, and then outputs a fully focused fused image; A discriminator processing module, used for inputting the natural image and the all-focus fusion image into the discriminator of the multi-focus image fusion network model to train the difference between the all-focus fusion image and the natural image; A loss function construction module, used to define a first training loss function of the discriminator, and define a second training loss function of the generator according to the first training loss function; A training module is used to iteratively train the multi-focus image fusion network model according to the first training loss function and the second training loss function until the minimum value of the first training loss function and the second training loss function is obtained, so as to construct a trained multi-focus image fusion network model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images