A method for acquiring sharp images based on multi-focus image pixel sharpness evaluation

Abstract

This invention discloses a method for acquiring sharp images based on pixel sharpness evaluation of multi-focus images, comprising: inputting a set of multi-focus images of the object to be tested into a trained sharpness rating network model, and outputting the pixel sharpness of each image; training the sharpness rating network model using a constructed training dataset to obtain the trained sharpness rating network model, wherein several sets of multi-focus images of different objects are collected using a microscope in the training dataset, and the sharpness rating of each pixel in the image is calculated as the ground truth during training; comparing the sharpness rating of the same pixel on different images according to the pixel sharpness of each image, and determining the image number of the sharpest pixel based on the lowest sharpness rating; fusing and synthesizing a sharp image of the object to be tested according to the determined image number, and recording the depth data of each pixel.

Description

Technical Field

[0001] This invention relates to the field of microscope multifocus image fusion technology, and in particular to a method for acquiring clear images based on multifocus image pixel sharpness evaluation. Background Technology

[0002] Knowing the microscope's focal length, the distance between the object and the microscope lens can be determined based on image sharpness. For an object with an uneven surface, images are taken at different distances to sequentially obtain sharpness at different locations on the object's surface, thus acquiring information about the surface's unevenness. Finally, these images from different distances are combined into a single sharp image; this is called multifocal image fusion. An example is shown below. Figure 2 As shown. The principle is that microscope lenses are designed with a relatively small effective depth of field. We assume that at a certain distance 'd' between the objective lens and the object, the object's surface has varying elevations. The areas of the object that meet the current focal length will be clear in the microscope image, while other parts of the object's surface that do not meet the current focal length will be blurry. Based on this principle, we can determine that the depth of the clear area of ​​the object is equal to the current focal length. By slightly moving the microscope, changing the distance between the object and the objective lens, we can obtain another clear image of the object's surface. When we obtain many images at different focal lengths, we can obtain a set of images of different clear parts of the object's surface. We then combine this set of images to create a single image where the entire object's surface is clear; this is called multi-focus fusion.

[0003] There are two technical approaches to achieving this fusion of a set of multi-focus images: the traditional image sharpness assessment approach and the deep learning-based image fusion generation approach. Currently, in practical applications, the traditional image sharpness assessment approach is more commonly used, while the approach using deep learning models for direct generation is less common. Each approach has its advantages and disadvantages. This invention improves upon the traditional image sharpness assessment method.

[0004] The traditional approach to image sharpness assessment is as follows: assuming a set of S images is captured, the sharpness of each pixel in each image is calculated. Then, the sharpness of the same pixel is compared across all S images, and the image with the highest sharpness is recorded as the image with the sharpest pixel. The value from the sharpest image is then selected as the value for that pixel in the final fused image. From this process, it's clear that accurately calculating the sharpness of each pixel is crucial. Traditional methods for calculating sharpness primarily rely on high-frequency representations of surface texture information. In practical applications, when using traditional image processing methods for multi-focus image fusion, it's necessary to calculate the sharpness of each pixel in each image. However, this calculated sharpness value is inaccurate, significantly affected by background texture and dust particles. Inaccurate sharpness values ​​directly impact the subsequent fusion effect and the calculation of 3D information (surface elevation and depth) of the object. Therefore, this method results in many erroneous calculations due to inaccurate pixel sharpness calculations, especially when the object surface is smooth and the local texture is relatively simple. In addition, certain dust-like particles can also affect the calculation of sharpness in local areas. Summary of the Invention

[0005] Technical objective: To address the deficiencies in existing technologies, this invention discloses a method for acquiring clear images based on multi-focus image pixel sharpness evaluation. While acquiring a clear image of the object under test, it also records the depth data of each pixel.

[0006] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution.

[0007] A method for acquiring sharp images based on multi-focus image pixel sharpness evaluation includes the following steps:

[0008] S1. Acquire a set of multifocal images of the object under test using a microscope;

[0009] S2. Input a set of multi-focus images of the object to be tested into the trained sharpness rating network model, and output the pixel sharpness of each image. The sharpness rating network model is trained using the constructed training dataset to obtain the trained sharpness rating network model. In the training dataset, several sets of multi-focus images of different objects are collected by a microscope, and the sharpness rating of each pixel in the image is calculated as the ground truth during training. The sharpness rating of each pixel is obtained by calculating the first sharpness and the second sharpness. The first sharpness is the pixel sharpness calculated based on the actual depth obtained by the laser rangefinder, and the second sharpness is the pixel sharpness obtained by calculating the local similarity of the sharp image obtained by the existing image fusion method.

[0010] S3. Based on the pixel sharpness of each image, compare the sharpness level of the same pixel on different images, and determine the image number of the sharpest pixel based on the lowest sharpness level, that is, the image number in a set of multi-focus images of the object to be tested.

[0011] S4. Based on the determined image sequence number, fuse and synthesize a clear image of the object under test, and record the depth data of each pixel.

[0012] Preferably, the process of constructing the training dataset in S2 includes:

[0013] S21. Before acquiring images through the microscope, calibrate the reference zero point of the laser rangefinder sensor with the zero point of the microscope.

[0014] S22. Acquire several sets of multi-focus images of different objects through a microscope, and simultaneously control the laser rangefinder to move synchronously with the microscope lens, and obtain the actual depth of the object surface in each image through the laser rangefinder.

[0015] S23. For each image in each group of multifocal images, calculate the first sharpness of each pixel in all images;

[0016] S24. For each image in each group of multifocal images, calculate the second sharpness of each pixel in all images;

[0017] S25. Calculate the sharpness level of each pixel by calculating the first sharpness and the second sharpness, and calculate the sharpness level of each pixel on all images as the ground truth during training.

[0018] Preferably, the formula for calculating the first sharpness is: t1 = ΔD * α, where t1 is the first sharpness, ΔD is the difference between the actual depth and the object distance corresponding to the current microscope focal length, and α is the blur coefficient.

[0019] Preferably, the formula for calculating the second sharpness is: t2=1 / Si_s -1; where t2 is the second sharpness, Si_s is the sum of local similarities on S images of the i-th pixel, and S is the number of images in a set of multi-focus images.

[0020] Preferably, the calculation process of the local similarity of the i-th pixel in an image includes: constructing a region of size k*k centered on the pixel as a local region of the pixel in the image, and convolving the local region in the image with the local region corresponding to the i-th pixel in the clear image to obtain the local similarity, where k is an integer less than or equal to 5.

[0021] Preferably, the clear image is a clear image obtained according to an existing image fusion method, and the acquisition process includes:

[0022] For a set of multi-focus images of a certain object, existing image fusion methods are used to fuse them into a preliminary clear image. Noise is removed from the preliminary clear image, and error points are removed by using a laser rangefinder to obtain the actual depth of the object's surface in each image, resulting in a clear image.

[0023] Preferably, the formula for calculating the sharpness level of the i-th pixel is ti = t / Δt, where ti is the sharpness level of the i-th pixel, ti∈{1,2,…,T}, where 1 represents the sharpest and T represents the most blurry; the formula for calculating the pixel sharpness t is: t =λ*t1 + t2, where λ is a weighting coefficient, t1 is the first sharpness, and t2 is the second sharpness.

[0024] Preferably, the structure of the sharpness level evaluation network model includes a feature extraction module and a sharpness level output module; the feature extraction module is used to extract features from the input image and output image features; the sharpness level output module is connected to the feature extraction module and is used to classify the image features and output pixel sharpness with a size of H*W*T, where H and W are the height and width of the input image, and T is the sharpness level threshold.

[0025] Preferably, in step S3, if the same pixel has the same clarity level in two or more images, the middle image number is taken as the image number of the clearest pixel.

[0026] Beneficial effects:

[0027] (1) The present invention uses a sharpness rating network model to output the pixel sharpness of each image and constructs a training dataset for the sharpness rating network model. The first sharpness and the second sharpness calculated in the training dataset are used to obtain the sharpness rating of each pixel in the image and are used as the ground truth during training, which greatly improves the accuracy of the sharpness rating of each pixel and enhances the learning ability of the sharpness rating network model.

[0028] (2) The sharpness rating network model in this invention is a classification model, and the output is the sharpness rating. With GPU acceleration, the method of this invention improves the speed of sharpness calculation of a single image pixel.

[0029] (3) The clear image acquisition method of the present invention acquires a clear image of the object to be tested and records the depth data of each pixel. Attached Figure Description

[0030] Figure 1This is a flowchart of a method according to an embodiment of the present invention;

[0031] Figure 2 Example diagram of existing multi-focus image fusion;

[0032] Figure 3 This is a schematic diagram of the network model structure for resolution rating according to an embodiment of the present invention;

[0033] Figure 4 This is a flowchart illustrating the construction of a training dataset according to an embodiment of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application. Example

[0035] As attached Figure 1 As shown in the figure, a method for acquiring a sharp image based on multi-focus image pixel sharpness evaluation in this embodiment includes the following steps:

[0036] S1. Acquire a set of multifocal images of the object under test using a microscope;

[0037] S2. Input a set of multi-focus images of the object to be tested into the trained sharpness rating network model and output the pixel sharpness of each image.

[0038] The sharpness rating network model is trained using a constructed training dataset to obtain the trained sharpness rating network model. In the training dataset, several sets of multi-focus images of different objects are collected using a microscope, and the sharpness rating of each pixel in the image is calculated as the ground truth during training. The sharpness rating of each pixel is obtained by calculating a first sharpness and a second sharpness. The first sharpness is the pixel sharpness calculated based on the actual depth obtained by the laser rangefinder, and the second sharpness is the pixel sharpness obtained by calculating the local similarity of the sharp image obtained by the existing image fusion method.

[0039] As attached Figure 4 As shown, the process of constructing the training dataset includes:

[0040] S21. Before acquiring images through the microscope, the reference zero point of the laser rangefinder sensor is calibrated with the zero point of the microscope. This calibration process can be achieved using existing technologies and will not be elaborated here.

[0041] S22. Acquire several sets of multi-focus images of different objects through a microscope, and simultaneously control the laser rangefinder to move synchronously with the microscope lens, and obtain the actual depth of the object surface in each image through the laser rangefinder.

[0042] S23. For each image in each group of multifocal images, calculate the first sharpness of each pixel in all images;

[0043] For a specific pixel in an image, the sharpness of that pixel is calculated as the first sharpness. The formula for calculating the first sharpness is: t1 = ΔD * α, where t1 is the first sharpness, which is the pixel sharpness calculated based on the actual depth obtained from the laser rangefinder; ΔD is the difference between the actual depth and the object distance corresponding to the current microscope focal length; and α is the blur coefficient, which can be predefined and its specific value is defined according to the actual situation and is not restricted here. It should be noted that when ΔD is 0, that is, when the actual depth equals the object distance corresponding to the current microscope focal length, the image should be displayed most clearly.

[0044] S24. For each image in each group of multifocal images, calculate the second sharpness of each pixel in all images;

[0045] This invention employs a local similarity method to calculate the second sharpness of a pixel. For a specific pixel in an image, the sharpness of that pixel is calculated as the second sharpness. The formula for calculating the second sharpness is: t2 = 1 / Si_s - 1; where t2 is the second sharpness, which is the pixel sharpness obtained by calculating the local similarity of the sharp image obtained by existing image fusion methods; Si_s is the sum of the local similarities of the i-th pixel on S images; and S is the number of images in a set of multi-focus images. The calculation process for the local similarity of the i-th pixel on an image is as follows: On the image, a region of size k*k centered on the pixel is constructed as the local region of the pixel. The local region on this image is convolved with the local region corresponding to the i-th pixel on the sharp image, i.e., multiplied, summed, and normalized to obtain the local similarity; where k is a preset parameter, and the specific value is selected according to the actual situation, generally k is an integer less than or equal to 5. It should be noted that in a set of multi-focus images, when the two local regions corresponding to the i-th pixel are completely similar, that is, when the local similar region of the i-th pixel in a set of multi-focus images is completely similar to the local region in the sharp image, the local similarity of the i-th pixel in that image is 1 / S. If the two local regions corresponding to the i-th pixel in all images in a set of multi-focus images are completely similar, the calculated second sharpness t2 value is 0.

[0046] The process of obtaining a clear image based on existing image fusion methods includes:

[0047] For a set of multi-focus images of a certain object, existing image fusion methods are used to fuse them into a preliminary clear image. Noise is removed from the preliminary clear image, and error points are removed by using a laser rangefinder to obtain the actual depth of the object's surface in each image, resulting in a clear image.

[0048] Noise removal from the initially clear image can be done manually or using existing methods, which will not be elaborated here. Removing erroneous points by using a laser rangefinder to obtain the actual depth of the object surface in each image is done manually. By comparing the depth information of the initially clear image after fusion with the actual depth information of the object surface in each image obtained using the laser rangefinder, erroneous points are removed. After noise and erroneous point removal, the resulting image can be considered to represent the final clear image to some extent.

[0049] S25. Obtain the sharpness level of each pixel by calculating the first sharpness and the second sharpness, and calculate the sharpness level of each pixel on all images as the ground truth during training.

[0050] The formula for calculating the sharpness t of each pixel is: t = λ * t1 + t2, where λ is a weighting coefficient, a preset value, and its specific value is determined according to the actual situation. The formula for calculating the sharpness level of the i-th pixel is ti = t / Δt, where ti is the sharpness level of the i-th pixel, ti∈{1,2,…,T}, where 1 represents the sharpest and T represents the most blurry; Δt is the level interval, specifically the numerical difference in sharpness between two adjacent levels. It should be noted that the value calculated by t / Δt may not be an integer, in which case an additional rounding operation is performed. In this invention, rounding up or down is performed.

[0051] In fact, when t is large, ti can be taken as the most blurred level, that is, ti is taken as T. Using the above method, we can obtain a relatively accurate sharpness level of each pixel in an image as the ground truth. The training dataset constructed in this way can be used to train a sharpness level evaluation network model.

[0052] This patented model can learn information about the effects of texture and dust, which stems from our method of preparing training data. In our training data, on planes with texture and dust interference in the images, traditional methods may result in inaccurate depth judgments due to this interference. However, in our data preparation method, we can obtain the actual depth of the object surface in each image using a laser rangefinder, marking more accurate depth information. Therefore, during training, our model can learn that these texture and dust patterns are interference information, rather than true depth information.

[0053] As attached Figure 3 As shown, the sharpness rating network model is a deep learning network model. Its structure includes a feature extraction module and a sharpness rating output module. The feature extraction module extracts features from the input image and outputs image features. This module can employ common deep learning model structures, such as ResNet50 or DarkNet. The sharpness rating output module, connected to the feature extraction module, classifies the image features and outputs pixel sharpness with dimensions H*W*T, where H and W are the height and width of the input image, and T is the sharpness rating threshold. Pixel sharpness includes the sharpness rating of each pixel in the input image. The sharpness rating of the i-th pixel in the input image is defined as ti, where ti∈{1,2,…,T}, with 1 representing the sharpest and T representing the most blurry. The sharpness rating output module can adopt a transformer decoder structure, adjusting the output pixel sharpness size to H*W*T at the output layer.

[0054] During training, the sharpness rating network model is trained using the training dataset. The training process is based on existing methods and will not be described in detail here.

[0055] In this invention, the sharpness rating is defined as T levels, making the training process of the sharpness rating network model a classification process. If the sharpness rating is defined as a continuous value, then a deep learning network model needs to be trained for regression. For deep learning models, training a classification problem is easier than training a regression problem, which means that the sharpness rating network model in this invention requires less computing power.

[0056] S3. Based on the pixel sharpness of each image, compare the sharpness level of the same pixel on different images, and determine the image number of the sharpest pixel based on the lowest sharpness level, that is, the image number in a set of multi-focus images of the object to be tested.

[0057] If the same pixel has the same sharpness level in two or more images, meaning the sharpness is also the best, then the middle image number is taken as the sharpest image number for that pixel. For example, in a set of multi-focus images of the object under test, if the sharpness level is the same in the j-th and k-th images, then the sharpest image for that pixel is considered to come from the middle image of these image sequences, i.e., (j+k) / 2. If the result is not an integer, (j+k) / 2 can be rounded up or down.

[0058] S4. Based on the determined image sequence number, fuse and synthesize a clear image of the object under test, and record the depth data of each pixel.

[0059] In this invention, other image enhancement operations can be performed on the clear image of the object under test, such as tone adjustment and brightness optimization. These are also involved in traditional image fusion methods, but this invention will not describe this part in detail.

[0060] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for acquiring sharp images based on multi-focus image pixel sharpness evaluation, characterized in that, Includes the following steps: S1. Acquire a set of multifocal images of the object under test using a microscope; S2. Input a set of multi-focus images of the object to be tested into the trained sharpness rating network model and output the pixel sharpness of each image. The sharpness rating network model is trained using a constructed training dataset to obtain the trained sharpness rating network model. In the training dataset, several sets of multi-focus images of different objects are collected using a microscope, and the sharpness rating of each pixel in the image is calculated as the ground truth during training. The sharpness rating of each pixel is obtained by calculating a first sharpness and a second sharpness. The first sharpness is the pixel sharpness calculated based on the actual depth obtained by the laser rangefinder, and the second sharpness is the pixel sharpness obtained by calculating the local similarity of the sharp image obtained by the existing image fusion method. S3. Based on the pixel sharpness of each image, compare the sharpness level of the same pixel on different images, and determine the image number of the sharpest pixel based on the lowest sharpness level, that is, the image number in a set of multi-focus images of the object to be tested. S4. Based on the determined image sequence number, fuse and synthesize a clear image of the object under test, and record the depth data of each pixel.

2. The method for acquiring a sharp image based on multi-focus image pixel sharpness evaluation according to claim 1, characterized in that, The process of constructing the training dataset in S2 includes: S21. Before acquiring images through the microscope, calibrate the reference zero point of the laser rangefinder sensor with the zero point of the microscope. S22. Acquire several sets of multi-focus images of different objects through a microscope, and simultaneously control the laser rangefinder to move synchronously with the microscope lens, and obtain the actual depth of the object surface in each image through the laser rangefinder. S23. For each image in each group of multifocal images, calculate the first sharpness of each pixel in all images; S24. For each image in each group of multifocal images, calculate the second sharpness of each pixel in all images; S25. Calculate the sharpness level of each pixel by calculating the first sharpness and the second sharpness, and calculate the sharpness level of each pixel on all images as the ground truth during training.

3. The method for acquiring a sharp image based on multi-focus image pixel sharpness evaluation according to claim 2, characterized in that, The formula for calculating the first sharpness is: t1 = ΔD * α, where t1 is the first sharpness, ΔD is the difference between the actual depth and the object distance corresponding to the current microscope focal length, and α is the blur coefficient.

4. The method for acquiring a sharp image based on multi-focus image pixel sharpness evaluation according to claim 2, characterized in that, The formula for calculating the second sharpness is: t2 = 1 / Si_s - 1; where t2 is the second sharpness, Si_s is the sum of local similarities of the S images of the i-th pixel, and S is the number of images in a set of multi-focus images.

5. The method for acquiring a sharp image based on multi-focus image pixel sharpness evaluation according to claim 4, characterized in that: The process of calculating the local similarity of the i-th pixel in an image includes: constructing a k*k region centered on the pixel as the local region of the pixel in the image; convolving the local region in the image with the local region corresponding to the i-th pixel in the clear image to obtain the local similarity, where k is an integer less than or equal to 5.

6. The method for acquiring a sharp image based on multi-focus image pixel sharpness evaluation according to claim 5, characterized in that: The clear image is a clear image obtained using existing image fusion methods, and its acquisition process includes: For a set of multi-focus images of a certain object, existing image fusion methods are used to fuse them into a preliminary clear image. Noise is removed from the preliminary clear image, and error points are removed by using a laser rangefinder to obtain the actual depth of the object's surface in each image, resulting in a clear image.

7. A method for acquiring a sharp image based on multi-focus image pixel sharpness evaluation according to claim 2, characterized in that: The formula for calculating the sharpness level of the i-th pixel is ti = t / Δt, where ti is the sharpness level of the i-th pixel, ti∈{1,2,…,T}, where 1 represents the sharpest and T represents the most blurry; Δt is the level interval; the formula for calculating the pixel sharpness t is: t =λ*t1 + t2, where λ is the weighting coefficient, t1 is the first sharpness, and t2 is the second sharpness.

8. The method for acquiring a sharp image based on multi-focus image pixel sharpness evaluation according to claim 1, characterized in that: The structure of the sharpness rating network model includes a feature extraction module and a sharpness rating output module. The feature extraction module is used to extract features from the input image and output image features. The sharpness rating output module is connected to the feature extraction module and is used to classify the image features and output pixel sharpness with dimensions H*W*T, where H and W are the height and width of the input image, and T is the sharpness rating threshold.

9. The method for acquiring a sharp image based on multi-focus image pixel sharpness evaluation according to claim 1, characterized in that: In S3, if the same pixel has the same sharpness level in two or more images, the middle image number is taken as the image number of the sharpest pixel.

Images