A multi-focus image fusion method, device and medium applied to a microscope

Abstract

The application discloses a multi-focus image fusion method and device applied to a microscope, and a medium, and relates to the technical field of image processing, and specifically discloses the following technical scheme: a plurality of sequence images of a to-be-measured object are collected by the microscope; the sequence images are grouped; a parameter N is set according to the input image quantity of a trained deep learning network model; S multi-focus images in all sequence images are divided into a plurality of groups according to the condition that each group includes N images; each group of images including N images is input into the trained deep learning network model to perform image fusion and generate a clear image; the generated clear image is added to an original data set; images that do not participate in the last image fusion process are grouped and fused again to generate; until the number of unfused images is 1, a final clear image is obtained; the clarity map record in each fusion process is traced back according to the clarity of the final clear image; and the image number of each pixel in the final clear image is found.

Description

Technical Field

[0001] This invention relates to the field of microscope multifocus image fusion technology, and in particular to a method, device and medium for multifocus image fusion applied to microscopes. Background Technology

[0002] Knowing the microscope's focal length, the distance between the object and the microscope lens can be determined based on image sharpness. For an object with an uneven surface, images are taken at different distances to sequentially obtain sharpness at different locations on the object's surface, thus acquiring information about the surface's unevenness. Finally, these images from different distances are combined into a single sharp image; this is called multifocal image fusion. An example is shown below. Figure 2 As shown. The principle is that microscope lenses are designed with a relatively small effective depth of field. We assume that at a certain distance 'd' between the objective lens and the object, the object's surface has varying elevations. The areas of the object that meet the current focal length will be clear in the microscope image, while other parts of the object's surface that do not meet the current focal length will be blurry. Based on this principle, we can determine that the depth of the clear area of ​​the object is equal to the current focal length. By slightly moving the microscope, changing the distance between the object and the objective lens, we can obtain another clear image of the object's surface. When we obtain many images at different focal lengths, we can obtain a set of images of different clear parts of the object's surface. We then combine this set of images to create a single image where the entire object's surface is clear; this is called multi-focus fusion.

[0003] There are two technical approaches to achieving this multi-focus image fusion: the traditional image sharpness discrimination approach and the deep learning image fusion generation approach. Each approach has its drawbacks, which we will discuss in turn. The traditional image sharpness discrimination approach calculates the sharpness of a region by measuring its local gradient and sharpness, then selects the sharpest image from the set as the final fused image for that region. This method produces poor results for regions with less texture, especially when these regions are affected by dirt or particles, significantly impacting the accuracy of sharpness calculation. The deep learning image fusion approach has seen some research in recent years with the development of deep learning, but this method relies on training data, which presents challenges in practical applications due to the varying number of images. Current deep learning image fusion models take two images as input (a near-focus image and a far-focus image) and fuse them into a single sharp image, without determining whether a region is sharper in the near-focus or far-focus image. However, in practical microscope applications, more than two images are usually captured. The focus is constantly adjusted, resulting in dozens or even hundreds of images at different focal lengths. In each image, only a small portion is in focus, and the number of these images depends on the size and contours of the object. Furthermore, for deep learning models, once built, the number of input images for each computation is fixed. However, the number of images captured in a microscope is variable. Therefore, deep learning models cannot be directly applied to microscope scenarios. Summary of the Invention

[0004] Technical Objective: To address the shortcomings of existing technologies where the number of images acquired in a microscope scene is not fixed and the current deep learning model only accepts two images as input, this invention discloses a multi-focus image fusion method, device, and medium for microscopes. By grouping the images acquired by the microscope, a multi-image input image fusion model is constructed to obtain the final clear image and a pixel-level sharpness rating map.

[0005] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution.

[0006] A multi-focus image fusion method for microscopy, the method comprising: S1. Acquire several sequences of images of the object to be tested using a microscope. Each sequence of images includes several images with continuously varying distances within a depth-of-field travel range. S2. Group the sequence images. Set the parameter N according to the number of input images of the trained deep learning network model. Divide the images that are equidistant in each sequence into a group. That is, divide the S multi-focus images in all the sequence images into several groups according to the fact that each group includes N images. S3. Input each group of N images into the trained deep learning network model to perform image fusion and generate the corresponding fused image, i.e., the clear image. S4. Add the generated clear images to the original dataset and record the sharpness map corresponding to the clear images; S5. Regroup the images that did not participate in the previous image fusion process according to S2, and proceed to S6; the images that did not participate in the previous image fusion process include the clear images generated by the previously trained deep learning network model; S6: Input each group of N images into the trained deep learning network model and fuse them to generate the final image. S7: Determine if the number of unfused images is 1, meaning that all images except the last fused image have participated in the image fusion process. If yes, proceed to S8; otherwise, return to S4 and continue execution. S8: Take the clear image obtained from the last image fusion as the final clear image. Based on the clarity map of the final clear image, backtrack the clarity map records of each fusion process to find the image number of the clearest pixel in the final clear image, which is the image number in the sequence of images in S1.

[0007] Preferably, in S2, if S / N is not an integer, the last group of images is expanded so that there are N images in the last group to participate in the current group fusion process; or the last group of images is retained and does not participate in the current group fusion process, but enters the next group fusion process as images that did not participate in the previous image fusion process.

[0008] Preferably, the training process of the deep learning network model includes: S31. Construct a deep learning network model; the deep learning network model includes N interconnected feature extraction modules, feature fusion modules, sharpness recognition modules, and Decoder image generation modules; S32. Construct a sample set for the deep learning network model; the sample set includes several sequence images of different items, each sequence image includes several images with continuously varying distances within a depth-of-field range; the target ground truth clear image for training is obtained by manually removing noise points after image fusion according to current traditional image processing methods; the sharpness map required for training is obtained by calculating the similarity between the fused clear image and the original image. S33. The deep learning network model is trained using a sample set to obtain the trained deep learning network model.

[0009] Preferably, each feature extraction module is used to extract features from the input image, with an image as input and image features as output; the feature fusion module is connected to N feature extraction modules and is used to fuse and encode the input image features, with N image features as input and encoded features as output; the sharpness recognition module is connected to the feature fusion module and is used to generate a sharpness map; the Decoder image generation module is connected to the feature fusion module and is used to output a sharp image.

[0010] Preferably, the size of the sharpness map is H*W*N, where H, W, and N represent the height, width, and number of channels, respectively. The sharpness map has the same width and height as the original image. The probability distribution of each pixel on the sharpness map on N represents which image the pixel's maximum sharpness comes from. N also corresponds to N images in a set of images.

[0011] Preferably, in S33, the total loss function Loss = Loss_map * lambda + Loss_img, where lambda is the weight, Loss_map is the loss function for the sharpness map, and Loss_img is the loss function for the sharp image.

[0012] Preferably, in step S33, a sample set is used to train the deep learning network model, which includes two stages of training. The first stage is to directly train using the sample set until the total loss function Loss meets the preset threshold. In the second stage, a clear image generated by the deep learning network model trained in the first stage based on the sample set is obtained, mixed into the sample set, and then the deep learning network model is trained again until the total loss function Loss meets the preset threshold.

[0013] Preferably, the sample set contains several sequence images of different items, including acquiring a different number of sequence images of each item using a microscope, acquiring sequence images of different positions of the item based on the different undulations of different parts of the item's surface, and setting different depth-of-field travels for the microscope.

[0014] A computer device comprising a processor and a memory, the memory storing a computer program, the processor executing the computer program to implement a multi-focus image fusion method for use in a microscope as described above.

[0015] A computer storage medium storing a computer program, which, when executed on a processor, implements the above-described multi-focus image fusion method for use in a microscope.

[0016] Beneficial Effects: This invention employs a deep learning model fusion approach to improve image fusion performance and solves the problem of fusion failure in textureless regions using traditional methods. The proposed model addresses the limitation of traditional fusion models processing only two images. The proposed group fusion method allows deep learning models to be truly applied to real-world scenarios, outputting a final clear image and a pixel-level sharpness score map, indicating which input image corresponds to the clearest image in that region. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall method of an embodiment of the present invention; Figure 2 Example diagram of multi-focus image fusion; Figure 3 This is a schematic diagram of the structure of a deep learning network model according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the training process of a deep learning network model according to an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application. Example

[0019] As attached Figure 1 As shown, this embodiment of a multi-focus image fusion method applied to a microscope includes: S1. Acquire several sequences of images of the object to be tested using a microscope. Each sequence of images includes several images with continuously varying distances within a depth-of-field travel range. In this embodiment, the object to be tested is placed under a microscope. The upper and lower limits of the depth of field travel for image acquisition are adjusted according to the upper and lower limits of the object's height. The microscope is set to acquire images, resulting in a total of S multi-focus images. In order to fully extract the features of the object to be tested, different numbers of sequence images are acquired by setting different depth of field travels when acquiring multi-focus images. That is, the S multi-focus images are several sequence images, and each sequence image includes several images with continuously changing distances within a depth of field travel range. S2. Group the sequence images. Set the parameter N according to the number of input images of the trained deep learning network model. Divide the images with the same distance in each sequence image into a group. That is, divide the S multi-focus images in all sequence images into several groups according to the fact that each group includes N images. The maximum number of groups is M = ⌈S / N ⌉. In S2, S / N is not necessarily an integer, meaning there may be a case where the number of images in the last group is less than N. In this case, the last group of images can be expanded by randomly copying images from other sequences to make the last group contain N images that participate in the current group fusion process; or the last group of images can be retained and not participate in the current group fusion process, but can be used as images that did not participate in the previous image fusion process to enter the next group fusion process.

[0020] S3. Input each group of N images into the trained deep learning network model to perform image fusion and generate the corresponding fused image, i.e., the clear image. In this embodiment, each group of images, consisting of N images, is input into the trained deep learning network model. Each group of images is fused and generated in the trained deep learning network model. In each fusion generation process, the input is N images from a group of images, and the output is the fused generated image, i.e., the clear image. The training process of a deep learning network model includes: S31. Construct a deep learning network model; the deep learning network model includes N feature extraction modules, feature fusion modules, sharpness recognition modules, and Decoder image generation modules; The structure diagram of this deep learning network model is as follows: Figure 3 As shown, each feature extraction module is used to extract features from the input image. Its input is an image and its output is image features. In this embodiment, all N feature extraction modules are implemented using convolutional neural networks. The network structure can adopt the backbone structure commonly used in the current deep learning field, such as ResNet50, EfficientNetV2, etc., and there are no restrictions here. The feature fusion module is connected to N feature extraction modules and is used to fuse and encode the input image features. Its input is N image features and its output is encoded features. In this embodiment, the image features output by the N feature extraction modules are first combined together through the concat connection layer and then sent to the feature fusion module Encoder, that is, the features of all images are fused and encoded. The structure of this module can utilize a CNN-based structure or a vision transformer encoder-based structure, and no specific limitation is made in this invention. The sharpness recognition module is connected to the feature fusion module to generate a sharpness map with dimensions H*W*N, where H, W, and N represent height, width, and number of channels, respectively. The sharpness map has the same dimensions as the original image. The probability distribution of each pixel on the sharpness map across N represents which image the pixel's maximum sharpness comes from. N also corresponds to N images in a set, facilitating the tracing of the location of maximum sharpness. This embodiment uses a deconvolution layer, which is simpler in structure than a transformer decoder, reducing computational load. The image generation module of the Decoder is connected to the feature fusion module to output a clear image. In this embodiment, a deconvolution layer or vision transformer decoder structure is used. The deconvolution layer is a deconvolution layer, which is a commonly used image generation method based on CNN in previous years. This invention employs a deep learning model fusion approach to improve image fusion performance and solves the problem of fusion failure in textureless regions using traditional methods. The proposed model sets parameter N based on factors such as the microscope's depth of field and the computational power of the deep learning network model, thereby determining the number of feature extraction modules and addressing the issue of traditional fusion models processing only two images. Generally, a larger depth of field in the microscope corresponds to a larger parameter N; the specific value can be determined based on the actual situation.

[0021] S32. Construct a sample set for the deep learning network model; the sample set includes several sequence images of different items, each sequence image includes several images with continuously varying distances within a depth-of-field range; the target ground truth clear image for training is obtained by manually removing noise points after image fusion according to current traditional image processing methods; the sharpness map required for training is obtained by calculating the similarity between the fused clear image and the original image. The sample set contains several image sequences of different items. This includes acquiring a different number of image sequences for each item using a microscope, taking into account the varying elevations of different parts of the item's surface, and setting different depths of field for the microscope to acquire image sequences at different locations on the item. For different items, such as circuit boards, screws, erasers, pens, steel plates, glass plates, metal washers, wrenches, and other tools, different depths of field are set to acquire different numbers of image sequences using a microscope. Since the elevations of different parts of each item vary, this embodiment can acquire an average of 50 sets of image sequences at different locations for each item, thereby expanding the entire dataset. During training, N images are randomly selected from each set of image sequences as input images. The target ground truth image for training is a clear image, obtained by manually removing noise points after image fusion using current traditional image processing methods. The required sharpness map for training is obtained by calculating the similarity between the fused clear image and the original images. For example, if we already have the fused clear image, we convolve this clear image with the original N images at a certain pixel using a k*k size. The region with the largest similarity is the most similar, and the clearest position of that pixel can be considered to be in that image. k is selected based on the actual situation, and is generally an integer less than or equal to 5.

[0022] S33. The deep learning network model is trained using a sample set to obtain the trained deep learning network model. As attached Figure 4As shown, during training, the first stage of training is performed. This involves arbitrarily selecting N images from each image sequence as input images. The network output is then used to calculate the loss function (Loss) by subtracting the sharpness map and the sharp image from the prepared dataset. This process is similar to loss training in common deep learning image generation, and the loss function formula can be any existing formula; no restrictions are placed here. The loss function for the sharpness map is denoted as Loss_map, and the loss function for the sharp image is denoted as Loss_img. In actual training, these two Losses are added together using a single weight, i.e., the total loss function Loss = Loss_map * lambda + Loss_img, where lambda is the weight, determined empirically. Once the total loss function Loss meets a preset threshold, the first stage of training is complete, and the second stage of training begins. In the second stage of training, the deep learning network model has already developed a certain fusion capability. The clear images generated by the deep learning network model trained in the first stage will be numbered and mixed into the sample set for retraining, i.e., the second stage of training will be carried out. N images will be randomly selected from the sample set containing the clear images and fed into the deep learning network model for training until the total loss meets the preset threshold, thus completing the second stage of training.

[0023] S4. Add the generated clear images to the original dataset and record the sharpness map corresponding to the clear images; S5. Regroup the images that did not participate in the previous image fusion process according to S2 and proceed to S6. The images that did not participate in the previous image fusion process include the clear images generated by the trained deep learning network model in the previous process, such as the images numbered from S+1 in S4, and if the last group of images in the previous group was not expanded, it also includes all the images in the last group in the previous group.

[0024] S6: Input each group of N images into the trained deep learning network model and fuse them to generate the final image.

[0025] S7: Determine if the number of unfused images is 1, meaning that all images except the last fused image have participated in the image fusion process. If yes, proceed to S8; otherwise, return to S4 and continue execution. S8: Take the clear image obtained from the last image fusion as the final clear image. Based on the clarity map of the final clear image, backtrack the clarity map records of each fusion process to find the image number of the clearest pixel in the final clear image.

[0026] This invention groups images acquired by a microscope, enabling the deep learning model to be truly applied to real-world scenarios. It constructs an image fusion model with multiple image inputs, obtains the final clear image, and provides a pixel-level clarity score map. This map can retrospectively indicate which image in the initial sequence corresponds to the clearest image at each pixel in the final clear image.

[0027] The present invention also discloses a computer device comprising a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement a multi-focus image fusion method for microscopes as described above. The memory can be of various types, such as random access memory, read-only memory, flash memory, etc.

[0028] The present invention also provides a computer storage medium on which a computer program is stored. When the computer program is executed, a device running the computer program implements the aforementioned multi-focus image fusion method applied to a microscope. The computer storage medium can be a readable storage medium, a non-volatile storage medium, or a volatile storage medium. For example, the computer storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0029] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-focus image fusion method applied to microscopes, characterized in that, The methods include: S1. Acquire several sequences of images of the object to be tested using a microscope. Each sequence of images includes several images with continuously varying distances within a depth-of-field travel range. S2. Group the sequence images. Set the parameter N according to the number of input images of the trained deep learning network model. Divide the images that are equidistant in each sequence into a group. That is, divide the S multi-focus images in all the sequence images into several groups according to the fact that each group includes N images. S3. Input each group of N images into the trained deep learning network model to perform image fusion and generate the corresponding fused image, i.e., the clear image. S4. Add the generated clear images to the original dataset and record the sharpness map corresponding to the clear images; S5. Regroup the images that did not participate in the previous image fusion process according to S2, and proceed to S6; the images that did not participate in the previous image fusion process include the clear images generated by the previously trained deep learning network model; S6: Input each group of N images into the trained deep learning network model and fuse them to generate the final image. S7: Determine if the number of unfused images is 1, meaning that all images except the last fused image have participated in the image fusion process. If yes, proceed to S8; otherwise, return to S4 and continue execution. S8: Take the clear image obtained from the last image fusion as the final clear image. Based on the clarity map of the final clear image, backtrack the clarity map records of each fusion process to find the image number of the clearest pixel in the final clear image, which is the image number in the sequence of images in S1.

2. The multi-focus image fusion method for microscopes according to claim 1, characterized in that: In S2, if S / N is not an integer, the last group of images is expanded so that there are N images in the last group to participate in the current group fusion process; or the last group of images is retained and does not participate in the current group fusion process, but enters the next group fusion process as images that did not participate in the previous image fusion process.

3. The multi-focus image fusion method for microscopes according to claim 1, characterized in that: The training process of a deep learning network model includes: S31. Construct a deep learning network model; the deep learning network model includes N interconnected feature extraction modules, feature fusion modules, sharpness recognition modules, and Decoder image generation modules; S32. Construct a sample set for the deep learning network model; the sample set includes several sequence images of different items, each sequence image includes several images with continuously varying distances within a depth-of-field range; the target ground truth clear image for training is obtained by manually removing noise points after image fusion according to current traditional image processing methods; the sharpness map required for training is obtained by calculating the similarity between the fused clear image and the original image. S33. The deep learning network model is trained using a sample set to obtain the trained deep learning network model.

4. The multi-focus image fusion method for microscopes according to claim 3, characterized in that: Each feature extraction module is used to extract features from the input image. Its input is an image, and its output is the image features. feature The fusion module is connected to N feature extraction modules and is used to fuse and encode the input image features. Its input is N image features and its output is the encoded features. The sharpness recognition module is connected to the feature fusion module to generate a sharpness map, and the Decoder image generation module is connected to the feature fusion module to output a sharp image.

5. The multi-focus image fusion method for microscopes according to claim 4, characterized in that: The size of the sharpness map is H*W*N, where H, W, and N represent the height, width, and number of channels, respectively. The sharpness map has the same width and height as the original image. The probability distribution of each pixel on the sharpness map on N represents which image the pixel's maximum sharpness comes from. N also corresponds to N images in a set of images.

6. A multi-focus image fusion method for microscopes according to claim 3, characterized in that: In S33, the total loss function is Loss = Loss_map * lambda + Loss_img, where lambda is the weight, Loss_map is the loss function for the sharpness map, and Loss_img is the loss function for the sharp image.

7. The multi-focus image fusion method for microscopes according to claim 3, characterized in that: S33 uses a sample set to train the deep learning network model, which includes two stages of training. The first stage is to train directly using the sample set until the total loss function Loss meets the preset threshold. In the second stage, a clear image generated by the deep learning network model trained in the first stage based on the sample set is obtained, mixed into the sample set, and the deep learning network model is trained again until the total loss function Loss meets the preset threshold.

8. The multi-focus image fusion method for microscopes according to claim 3, characterized in that: The sample set contains several sequence images of different items, including a different number of sequence images of each item acquired through a microscope, based on the different undulations of different parts of the item's surface, and by setting different depths of field for the microscope to acquire sequence images of different positions of the item.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement a multi-focus image fusion method for a microscope according to any one of claims 1-8.

10. A computer storage medium, characterized in that, It stores a computer program that, when executed on a processor, implements a multi-focus image fusion method for a microscope according to any one of claims 1-8.

Images