A method and device for extending the depth of field of microscope images
By establishing the Laplace pyramid and the depth estimation pyramid, combining the natural image cutting theory to extend the depth of field of microscope images, the problem of contrast reduction caused by diffuse speckle is solved, and the effective depth of field extension effect is achieved, which is suitable for ordinary users and Reduced costs.
Patent Information
- Application Number
- CN202111232216.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-22
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2041-10-22
AI Technical Summary
The prior art has the problem of contrast reduction caused by diffuse speckle in the extension of depth of field of microscope images, and common algorithms are difficult to achieve real-time processing, which increases the cost to ordinary users.
By establishing the Laplace pyramid to obtain high-frequency information of the image, combining the natural image cutout theory for depth estimation, forming a depth estimation pyramid, fusing the relative depth map and obtaining the decision map based on the maximum value principle, guiding image sequence fusion and reducing diffuse spots Effect on the final image.
It effectively reduces the contrast effect of the diffuse speckle of the microscope image on the final large depth of field image, improves the robustness of depth of field extension, reduces the sensitivity to depth of field changes, and improves the fusion quality.
Smart Images
Figure CN113963046B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, mainly to two technical fields of image fusion and depth of field extension, and in particular to a method and device for extending the depth of field of a microscope image. Background Art
[0002] Depth of field extension technology refers to the technology of fusing images of the same observation object taken at different focusing layers into an image with a large depth of field. This technology is of great significance in the field of microscopic digital imaging.
[0003] At present, the main method for obtaining images with a large depth of field is to control the Z-axis adjustment knob of the microscope through a stepper motor, image different layers of images onto the image sensor, save them one by one after positioning, then extract the clarity area of all images, and finally fuse them to obtain an image with a large depth of field. For example, the Chinese patent with publication number CN111989608A discloses a microscope and method for performing microscopic observation of samples to present images with an extended depth of field or a three-dimensional image.
[0004] Another method is to precisely control the Z-axis focusing mechanism through a stepper motor, scan the Z-axis of an ordinary microscope with equal steps, and fuse while scanning. This method also requires the modification of a traditional microscope, which has certain significance for professional applications, but for ordinary users, it will increase costs and is of little significance. For example, the Chinese patent publication number CN112241065A discloses a microscope and a method for producing a microscopic image with an extended depth of field.
[0005] In addition, there are many difficulties in using common depth of field extension algorithms for real-time processing, such as fusion quality issues, that is, image tearing caused by local fusion. This is because the diffuse spots produced by a small depth of field lens are easily misjudged as high-frequency information and then fused into the final large depth of field image, reducing the contrast of the final result image. Summary of the invention
[0006] The object of the present invention is to provide a method and device for extending the depth of field of a microscope image, which can effectively reduce the influence of the diffuse spots of the image directly acquired by the microscope on the contrast of the final large depth of field image.
[0007] The present invention provides the following technical solutions:
[0008] A method for extending the depth of field of a microscope image, the method comprising the following steps:
[0009] (1) Establish a Laplacian pyramid for microscope images to obtain high-frequency information of the image at different scales;
[0010] (2) performing depth estimation on the topmost Gaussian blur layer of the Laplacian pyramid to obtain a depth estimation information map, wherein the residual layer of the Laplacian pyramid is used as prior information and combined with the natural image matting theory to perform depth estimation to obtain a depth estimation information map;
[0011] (3) The high-frequency information in step (1) and the depth estimation information map in step (2) are combined into a depth estimation pyramid, and the depth estimation pyramid is fused to obtain a high-resolution relative depth map;
[0012] (4) Obtaining a decision map using the large-resolution relative depth map in step (3) according to the maximum value principle;
[0013] (5) The decision graph obtained in step (4) guides the extraction of the clarity area of the microscope image, and then performs image sequence fusion to obtain the microscope image after depth of field extension.
[0014] Preferably, in the present invention, based on the two premises of allowing as many images at different focal depths as possible to participate in depth of field extension and saving computing resources as much as possible, a microscope image sequence with large focal plane changes is screened out for the depth of field extension method.
[0015] In step (1), the method for obtaining the high-frequency information is:
[0016] A single image is input into the image to build a Laplacian pyramid; the original image of the i-th layer is G i , Down() is the downsampling operation, Up() is the upsampling operation, the Laplacian pyramid i-th image L i The method to obtain is as follows:
[0017] L i =G i -Up(Down(Gi))
[0018] The obtained Laplacian pyramid is only the high-frequency component of the image. The pixel values of the residual layer are positive and negative. Regardless of the sign of the pixel value, it belongs to the high-frequency component. The absolute value of the residual layer image of the Laplacian pyramid is taken to complete the extraction of high-frequency information of the multi-scale image. The final residual layer expression of the Laplacian pyramid is:
[0019] L i =abs(G i -Up(Down(Gi))).
[0020] In step (1), the high-frequency information is used for subsequent reconstruction of a large-resolution depth estimation image.
[0021] Among them, a Laplace pyramid of the image is established, the image is downsampled to obtain a small-sized image, and then the small-sized image is upsampled, and the difference between the upsampled image and the original image is taken to obtain a Laplace difference image, which is the high-frequency component of the original image.
[0022] Preferably, image clarity includes image resolution and image sharpness. Studies have shown that the clarity of a color image mainly depends on the brightness component of the image. The details and edges of an image contain more image features and information, and are also the main areas that affect image clarity. Based on this study, the invention performs color space conversion on the input image, converts the image from RGB space to HSV space, and takes the V component in the HSV space to establish a Laplace difference pyramid.
[0023] In step (2), specifically:
[0024] The depth information of each pixel in the image is obtained by applying the Matting Laplacian matrix to the prior information. The depth information of the image is obtained by minimizing the following loss function:
[0025]
[0026] and D are the image prior information and image depth information in vector form, is a diagonal matrix. When the i pixel is at the edge of the object in the image, L is the Matting Laplacian matrix, and the element values of the matrix L are:
[0027]
[0028] δ ij is the Kronecker function, U3 is the third-order unit matrix, μ k and Σ k It is a window k The mean and covariance matrix of , ε is the regularization parameter, I i and I j is the pixel value of pixel i and j, |ω k | is windowω k size;
[0029]
[0030] Solving the above equations yields the final depth estimation information graph D.
[0031] In step (2), the depth estimation is performed on the Gaussian blur layer at the top of the Laplacian pyramid. This is because the depth estimation algorithm is computationally intensive. Depth estimation for a large-resolution image consumes a lot of memory and has a low solution efficiency. The depth estimation is performed on the small-resolution image at the top of the pyramid to obtain a depth estimation information map. The high-frequency information of different scales (the residual layer of the Laplacian pyramid) is combined to form a depth estimation pyramid. The depth estimation pyramid is fused to obtain a large-resolution relative depth map. For the theory of natural image matting: the image clarity map is used as prior information combined with the theory of natural image matting. The image foreground is the clear area, the background is the blurred area, the area with large pixel values (foreground) is closer to the lens, and the background is relatively farther away; the image clarity map is obtained based on the high-frequency area of the image as the clear area, and the residual image layer of the Laplacian pyramid contains high-frequency signals, which are regarded as prior information input to obtain the depth estimation map.
[0032] In step (4), the method for obtaining the decision map is: according to the obtained depth information image sequence, the maximum return index value is taken pixel by pixel to obtain the decision map (wherein the pixel value of the decision map is the serial number of the input image sequence):
[0033]
[0034] Wherein, n represents the number of depth information image sequence groups, i represents the i-th row in the depth information image, and j represents the j-th column in the depth information image.
[0035] In step (5), the decision graph is subjected to median filtering to remove salt and pepper noise, and the filtered decision graph is used to guide the fusion of the source image sequence. The pixel value of the fused image is the pixel value of the corresponding index image pixel. The pixel-by-pixel comparison process is not constrained by local pixels, and noise exists in the obtained decision graph. In order to obtain a better display effect, the noise needs to be removed.
[0036] The present invention also provides a device for extending the depth of field of a microscope image, the device comprising:
[0037] The high-frequency information extraction module builds a Laplacian pyramid for the microscope image, obtains the high-frequency information of the image at different scales and inputs it into the depth estimation module;
[0038] The depth estimation module performs depth estimation on the topmost Gaussian blur layer of the Laplacian pyramid to obtain a depth estimation information map; the high-frequency information in step (1) and the depth estimation information map in step (2) are combined into a depth estimation pyramid, and the depth estimation pyramid is fused to obtain a high-resolution relative depth map;
[0039] The fusion module obtains a decision map based on the maximum value principle from the large-resolution relative depth map; and uses the decision map to guide the extraction of the clarity area of the microscope image, and then performs image sequence fusion to obtain the microscope image after the depth of field extension.
[0040] Preferably, in the high-frequency information module: a single image is input into the image to establish a Laplacian pyramid; the original image at the i-th layer is G i , Down() is the downsampling operation, Up() is the upsampling operation, the Laplacian pyramid i-th image L i The method to obtain is as follows:
[0041] L i =G i -Up(Down(Gi))
[0042] The obtained Laplacian pyramid is only the high-frequency component of the image. The pixel values of the residual layer are positive and negative. Regardless of the sign of the pixel value, it belongs to the high-frequency component. The absolute value of the residual layer image of the Laplacian pyramid is taken to complete the extraction of high-frequency information of the multi-scale image. The final residual layer expression of the Laplacian pyramid is:
[0043] L i =abs(G i -Up(Down(Gi))).
[0044] Preferably, in the depth estimation module:
[0045] The method of performing depth estimation on the top Gaussian blur layer of the Laplacian pyramid and obtaining the depth estimation information map is as follows: the residual layer of the Laplacian pyramid is used as prior information combined with the natural image matting theory, specifically:
[0046] The depth information of each pixel in the image is obtained by applying the Matting Laplacian matrix to the prior information. The depth information of the image is obtained by minimizing the following loss function:
[0047]
[0048] and D are the image prior information and image depth information in vector form, is a diagonal matrix. When the i pixel is at the edge of the object in the image, L is the Matting Laplacian matrix, and the element values of the matrix L are:
[0049]
[0050] δ ij is the Kronecker function, U3 is the third-order unit matrix, μk and Σ k It is a window k The mean and covariance matrix of , ε is the regularization parameter, I i and I j is the pixel value of pixel i and j, |ω k | is windowω k size;
[0051]
[0052] Solving the above equations yields the final depth estimation information graph D.
[0053] Preferably, in the fusion module:
[0054] The method for obtaining the decision map is: according to the obtained depth information image sequence, the maximum return index value is taken pixel by pixel to obtain the decision map:
[0055]
[0056] Preferably, in the fusion module:
[0057] The decision graph is subjected to median filtering to remove salt and pepper noise, and the filtered decision graph is used to guide the fusion of the source image sequence. The pixel value of the fused image is the pixel value of the corresponding index image pixel.
[0058] Compared with the prior art, the depth of field extension method and device for microscope images provided by the present invention can effectively reduce the influence of the diffuse spots of the images directly acquired by the microscope on the contrast of the final large depth of field image, and the method and device are highly robust to depth of field changes and are not easily affected by changes in the degree or direction of the depth of field change, which may affect the final fusion quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 This is the overall flow of a one-time fusion process of the depth of field extension method in the embodiment.
[0060] Figure 2 It is an example of multi-scale high-frequency information of an image obtained during the processing in the embodiment.
[0061] Figure 3 This is an example of a small-resolution depth estimation during the processing in the embodiment.
[0062] Figure 4 This is an example of a high-resolution depth estimation during the processing in the embodiment.
[0063] Figure 5 It is an example of a decision diagram and a super depth of field image obtained during the processing in the embodiment. DETAILED DESCRIPTION
[0064] The present invention can be used as supporting software for a microscope camera. When used for depth of field extension, the microscope small depth of field image is transmitted to the input interface of the software, the object distance of the microscope stage relative to the objective lens is manually adjusted to obtain images of different focal planes, and the depth of field extension image can be generated by clicking the fusion operation.
[0065] Figure 1 It is the overall process of the depth of field extension fusion process, including:
[0066] S1. Establish a Laplacian pyramid for the microscope image in the high-frequency information module to obtain the high-frequency information of the image at different scales;
[0067] S2, in the depth estimation module, performs depth estimation on the topmost Gaussian blur layer of the Laplacian pyramid to obtain a depth estimation information map; the high-frequency information in S1 and the depth estimation information map are combined into a depth estimation pyramid, and the depth estimation pyramid is fused to obtain a high-resolution relative depth map;
[0068] S3. In the fusion module, the large-resolution relative depth map is used to obtain a decision map according to the maximum value principle; and the decision map is used to guide the extraction of the clarity area of the microscope image, and then the image sequence is fused to obtain the microscope image after the depth of field is extended.
[0069] Specifically:
[0070] In S1: Figure 2 This is a specific example of obtaining multi-scale high-frequency information of an image. The resolution of the processed image is 3072*2048. The image is converted from RGB space to HSV space, the V component image is extracted for depth estimation and the decision graph operation is obtained. The steps are as follows:
[0071] 1. Create a five-layer image pyramid for the image, with the smallest layer size being 192*148;
[0072] 2. Let the original image of the i-th layer be G i , Down() is the downsampling operation, Up() is the upsampling operation, the Laplacian pyramid i-th image L i The method to obtain is as follows:
[0073] L i =G i -Up(Down(Gi))
[0074] The obtained Laplacian pyramid is only the high-frequency component of the image. The pixel values of the residual layer are positive and negative. Regardless of the sign of the pixel value, it belongs to the high-frequency component. The absolute value of the residual layer image of the Laplacian pyramid is taken to complete the extraction of high-frequency information of the multi-scale image. The mathematical expression of the final Laplacian residual layer is:
[0075] L i =abs(Gi -Up(Down(Gi)))
[0076] 3. The pixel value of the pyramid residual layer is obtained by taking the absolute value based on the Laplacian pyramid, which contains all the high-frequency information of the current scale; the minimum size image is the depth information image obtained by applying the natural image cutout theory.
[0077] In S2: Figure 3 This is an example of depth estimation for a small-scale image in the depth estimation module. For display reasons, the obtained depth estimation image is enlarged. The specific steps are as follows:
[0078] 1. Obtain high-frequency information from the fifth-layer image of the pyramid, and then apply Matting Laplacian to the original Gaussian image of the fifth layer to obtain the depth information map;
[0079] 2. The depth information of each pixel of the image is obtained by applying Matting Laplacian to the high-frequency prior information. The depth information of the image is obtained by minimizing the following loss function:
[0080]
[0081] and D are the image prior information and image depth information in vector form, is a diagonal matrix. When the i pixel is at the edge of the object in the image, L is the Matting Laplacian matrix, and the element values of the matrix L are:
[0082]
[0083] δ ij is the Kronecker function, U3 is the third-order unit matrix, μ k and Σ k It is a window k The mean and covariance matrix of , ε is the regularization parameter, I i and I j is the pixel value of pixel i and j, |ω k | is windowω k size.
[0084]
[0085] Solve the above equation to get the final depth information map D;
[0086] Figure 4It is an example of large-scale depth estimation. The Laplacian residual layer and the depth information layer are combined into a fused pyramid. The combined fused pyramid can obtain a large-resolution relative depth map.
[0087] In S3: Figure 5 It is a decision graph obtained by taking the maximum return index value pixel by pixel based on the depth information image sequence obtained
[0088]
[0089] Wherein, n represents the number of depth information image sequence groups, i represents the i-th row in the depth information image, and j represents the j-th column in the depth information image.
[0090] The directly acquired decision graph is subjected to median filtering to remove salt and pepper noise, and the filtered decision graph is used to guide the fusion of the source image sequence. The pixel value of the fused image is the pixel value of the corresponding index image pixel point.
[0091] Completing the above steps is a complete depth of field extension process.
Claims
1. A method for extending the depth of field of a microscope image, characterized in that: The method comprises the following steps: (1) Establish a Laplacian pyramid for microscope images to obtain high-frequency information of the image at different scales; (2) performing depth estimation on the topmost Gaussian blur layer of the Laplacian pyramid to obtain a depth estimation information map, wherein the residual layer of the Laplacian pyramid is used as prior information and combined with the natural image matting theory to perform depth estimation to obtain a depth estimation information map; (3) The high-frequency information in step (1) and the depth estimation information map in step (2) are combined into a depth estimation pyramid, and the depth estimation pyramid is fused to obtain a high-resolution relative depth map; (4) Obtaining a decision map using the large-resolution relative depth map in step (3) according to the maximum value principle; (5) The decision graph obtained in step (4) guides the extraction of the clarity area of the microscope image, and then performs image sequence fusion to obtain the microscope image after depth of field extension.
2. The method for extending the depth of field of a microscope image according to claim 1, characterized in that: In step (1), the mathematical expression of the method for obtaining the high-frequency information is: A single image is input into the image to build a Laplacian pyramid; the original image of the i-th layer is G i , Down() is the downsampling operation, Up() is the upsampling operation, the Laplacian pyramid i-th image L i The method to obtain is as follows: L i =G i -Up(Down(Gi)) The obtained Laplacian pyramid is only the high-frequency component of the image. The pixel values of the residual layer are positive and negative. Regardless of whether the pixel value is positive or negative, it belongs to the high-frequency component. The absolute value of the residual layer image of the Laplacian pyramid is taken to complete the extraction of high-frequency information of the multi-scale image. The final residual layer expression of the Laplacian pyramid is: L i =abs(G i -Up(Down(Gi)))。 3. The method for extending the depth of field of a microscope image according to claim 1, characterized in that: In step (2), specifically: The depth information of each pixel in the image is obtained by applying the Matting Laplacian matrix to the prior information. The depth information of the image is obtained by minimizing the following loss function: and D are the image prior information and image depth information in vector form, is a diagonal matrix. When the i pixel is at the edge of the object in the image, L is the Matting Laplacian matrix, and the element values of the matrix L are: δ ij is the Kronecker function, U3 is the third-order unit matrix, μ k and Σ k It is a window k The mean and covariance matrix of , ε is the regularization parameter, I i and I j is the pixel value of pixel i and j, |ω k | is windowω k size; Solving the above equations yields the final depth estimation information graph D.
4. The method for extending the depth of field of a microscope image according to claim 1, characterized in that: In step (4), the method for obtaining the decision map is: according to the obtained depth information image sequence, the maximum return index value is taken pixel by pixel to obtain the decision map: Wherein, n represents the number of depth information image sequence groups, i represents the i-th row in the depth information image, and j represents the j-th column in the depth information image.
5. The method for extending the depth of field of a microscope image according to claim 1, characterized in that: In step (5), a median filter is applied to the decision graph to remove salt and pepper noise, and the filtered decision graph is used to guide the fusion of the source image sequence. The pixel value of the fused image is the pixel value of the corresponding index image pixel point.
6. A device for extending the depth of field of a microscope image, characterized in that: The device comprises: The high-frequency information extraction module is used to establish a Laplacian pyramid for the microscope image, obtain the high-frequency information of the image at different scales and input it into the depth estimation module; A depth estimation module is used to perform depth estimation on the topmost Gaussian blur layer of the Laplacian pyramid to obtain a depth estimation information map, wherein the method comprises: using the residual layer of the Laplacian pyramid as prior information and combining it with the natural image cutout theory to perform depth estimation to obtain a depth estimation information map; and forming a depth estimation pyramid with the high-frequency information and the depth estimation information map, and fusing the depth estimation pyramid to obtain a high-resolution relative depth map; The fusion module is used to obtain a decision map based on the maximum value principle for the large-resolution relative depth map; and to extract the clarity area of the microscope image guided by the decision map, and then perform image sequence fusion to obtain the microscope image after the depth of field extension.
7. The depth-of-field extension device for microscope images according to claim 6, characterized in that: In the high-frequency information module: a single image is input into the image to build a Laplacian pyramid; the original image of the i-th layer is G i , Down() is the downsampling operation, Up() is the upsampling operation, the Laplacian pyramid i-th image L i The method to obtain is as follows: L i =G i -Up(Down(Gi)) The obtained Laplacian pyramid is only the high-frequency component of the image. The pixel values of the residual layer are positive and negative. Regardless of the sign of the pixel value, it belongs to the high-frequency component. The absolute value of the residual layer image of the Laplacian pyramid is taken to complete the extraction of high-frequency information of the multi-scale image. The final residual layer expression of the Laplacian pyramid is: L i =abs(G i -Up(Down(Gi)))。 8. The depth-of-field extension device for microscope images according to claim 6, characterized in that: In the depth estimation module: The method of performing depth estimation on the top Gaussian blur layer of the Laplacian pyramid and obtaining the depth estimation information map is as follows: the residual layer of the Laplacian pyramid is used as prior information combined with the natural image matting theory, specifically: The depth information of each pixel in the image is obtained by applying the Matting Laplacian matrix to the prior information. The depth information of the image is obtained by minimizing the following loss function: and D are the image prior information and image depth information in vector form, is a diagonal matrix. When the i pixel is at the edge of the object in the image, L is the Matting Laplacian matrix, and the element values of the matrix L are: δ ij is the Kronecker function, U3 is the third-order unit matrix, μ k and Σ k It is a window k The mean and covariance matrix of , ε is the regularization parameter, I i and I j is the pixel value of pixel i and j, |ω k | is windowω k size; Solving the above equations yields the final depth estimation information graph D.
9. The depth-of-field extension device for microscope images according to claim 6, characterized in that: In the Fusion module: The method for obtaining the decision map is: according to the obtained depth information image sequence, the maximum return index value is taken pixel by pixel to obtain the decision map:
10. The device for extending the depth of field of a microscope image according to claim 6, characterized in that: In the Fusion module: The decision graph is subjected to median filtering to remove salt and pepper noise, and the filtered decision graph is used to guide the fusion of the source image sequence. The pixel value of the fused image is the pixel value of the corresponding index image pixel.