Multi-focus microscopic image fusion method, system and program based on depth map correction

Abstract

The application discloses a multi-focus microscopic image fusion method, system and program based on depth map correction. The method comprises the following steps in sequence: image sequence acquisition, texture extraction, definition evaluation, depth map generation, confidence map generation, depth correction and color image reconstruction. According to the definition curve, the depth map and the confidence map are determined, the unstable points are determined by using the confidence map, the depth data of the non-texture and weak-texture areas are corrected by HSL color space guided filtering, and thus the phenomenon that the depth map is wrong in focus plane judgment due to the non-texture and weak-texture areas of the image source is avoided.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and specifically relates to multifocal microscopy imaging technology. Background Technology

[0002] Depth of field is the depth of space within which a lens focuses, within which a sharp image of an object can be obtained. Microscope lenses have a very narrow depth of field, typically ranging from a few micrometers to several hundred micrometers. With the development of microscopic imaging technology, depth synthesis technology has emerged. Depth synthesis technology is used when the depth of field is small, and the overall image is not fully in focus, but some parts are still perfectly in focus. When the distance between the lens and the object changes, the focus also moves. By moving the lens's focus to select multiple images, then extracting and combining the sharp parts, a composite image with a clear overall depth of field can be obtained.

[0003] However, existing depth synthesis techniques still have many technical shortcomings, affecting the quality and effect of the generated images. These shortcomings mainly include: the presence of textureless or weakly textured regions in the image source, leading to incorrect focal plane determination in the depth map; and insufficiently refined depth map calculation, resulting in a "terraced effect" (the so-called "terraced effect" refers to the abrupt, step-like appearance of excessive depth in the depth map). Figure 1 As shown. Furthermore, the merged color image may also exhibit artifacts, halos, and inconsistencies in the blending process. Summary of the Invention

[0004] To address the technical problems mentioned in the background section, this invention proposes a multi-focus microscopic image fusion method, system, and program based on depth map correction.

[0005] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows:

[0006] First, this invention designs a multi-focus microscopic image fusion method, comprising the following steps:

[0007] (1) Move the lens from the lowest plane to the highest plane of the observed object in fixed steps and acquire images to obtain a color image sequence ZImages;

[0008] (2) Extract texture from each image in the color image sequence ZImages to generate a weight map corresponding to each image, and obtain the weight sequence ZImageWeights corresponding to the color image sequence ZImages;

[0009] (3) Compare the sharpness of pixels at the same position in the weight sequence ZImageWeights to obtain the sharpness curve of any image position;

[0010] (4) Perform polynomial fitting on n points near the peak of the sharpness curve at any image location, solve for the extreme points of the polynomial, and the corresponding horizontal coordinate is the floating-point value of the depth at that location, thereby obtaining the depth map ImageDepth of the observed object; accumulate all points on the sharpness curve at any image location and take the average value, then divide the maximum value of the sharpness curve by the average value to obtain the confidence level of that location, thereby obtaining the confidence map ImageConf;

[0011] (5) Correct the depth map ImageDepth using the confidence map ImageConf:

[0012] (5a) If the values ​​of all adjacent points at any image location are less than the set threshold m, then the location is determined to be an unstable point, and the row and column coordinates of the unstable point are obtained.

[0013] (5b) Each image in the color image sequence ZImages is converted from RGB color space to HSL color space. The converted images are added together and averaged to obtain three images ImageH, ImageS, and ImageL corresponding to the three components of chroma H, saturation S, and luminance L.

[0014] (5c) Obtain the chromaticity H, saturation S, and brightness L of the image ImageH, ImageS, and ImageL at the row and column coordinates of the unstable point, determine the color space region around the row and column coordinates, and search for points in the region with confidence values ​​greater than or equal to the threshold m as stable points, and use the stable points to correct the unstable points.

[0015] (6) Reconstruct the color image based on the corrected depth map.

[0016] Secondly, this invention also proposes a multi-focus microscopic image fusion system, comprising:

[0017] ① Image sequence acquisition module: used to control the lens to move from the lowest plane to the highest plane of the observed object in fixed steps and acquire images to obtain a color image sequence ZImages;

[0018] ② Texture extraction module: used to extract textures from each image in the color image sequence ZImages.

[0019] Generate a weight map for each image to obtain the color image sequence ZImages.

[0020] The corresponding weight sequence ZImageWeights;

[0021] ③ Sharpness evaluation module: used to compare the sharpness of pixels at the same position in the weight sequence ZImageWeights to obtain the sharpness curve of any image position;

[0022] ④ Depth map generation module: This module is used to perform polynomial fitting on n points near the peak of the sharpness curve at any image location, solve for the extreme points of the polynomial, and the corresponding horizontal coordinate is the floating-point value of the depth at that location, thereby obtaining the depth map ImageDepth of the observed object.

[0023] ⑤ Confidence map generation module: This module is used to sum up all points on the sharpness curve at any image location, take the average value, and then divide the maximum value of the sharpness curve by the average value to obtain the confidence level at that location, thereby obtaining the confidence map ImageConf.

[0024] ⑥ Depth Correction Module: Used to correct the depth map ImageDepth using the confidence map ImageConf;

[0025] ⑦ Image Reconstruction Module: Used to reconstruct a color image using the corrected depth map.

[0026] Furthermore, the present invention also proposes a program product for realizing multi-focus microscopic image fusion, comprising a computer program or instructions, which, when executed by a processor, implement the steps of the above method.

[0027] The beneficial effects of adopting the above technical solution are as follows:

[0028] 1. This invention determines the depth map and confidence map based on the sharpness curve, uses the confidence map to identify unstable points, and corrects the depth data of areas without texture or with weak texture by using HSL color space guided filtering, thereby avoiding the phenomenon of incorrect focal plane judgment in the depth map due to the presence of areas without texture or with weak texture in the image source.

[0029] 2. This invention uses floating-point data to represent depth data, enabling the depth map to be further subdivided from integer data, thereby solving the "terraced field phenomenon".

[0030] 3. This invention uses a floating-point depth map to guide color image fusion, which can eliminate some artifacts, halos, and uneven fusion. The weight ratio of two adjacent images can be subdivided according to "floating-point numbers," and the fusion weight of the two images can be dynamically adjusted. The fusion of color RGB values ​​is more refined, and it can achieve a fractional floating-point fusion, with performance and quality superior to the traditional "seam weight fusion." Attached Figure Description

[0031] Figure 1 This is a diagram of the "terraced field phenomenon". The red, green and blue arrows in the diagram represent the RGB components.

[0032] Figure 2 This is a simplified flowchart of the method of the present invention;

[0033] Figure 3 This is a schematic diagram of acquiring a color image sequence;

[0034] Figure 4 This is a diagram illustrating the sharpness curve;

[0035] Figure 5 This is a schematic diagram of polynomial fitting;

[0036] Figure 6 This is an example of a depth map showing unstable points;

[0037] Figure 7 yes Figure 6 3D display images;

[0038] Figure 8 This is an example image of a color image reconstruction. Detailed Implementation

[0039] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] This embodiment proposes a multi-focus microscopic image fusion method, such as... Figure 2 As shown, the method includes the following steps:

[0041] Step 1: Move the microscope along the z-axis, which in turn moves the lens from the lowest surface of the object to the highest surface, acquiring a sequence of z-axis color images (ZImages). Figure 3 As shown. In this step, the image needs to be free of jitter and the movement intervals need to be stable.

[0042] Step 2: Extract texture from each image in the color image sequence ZImages to generate a weight map corresponding to each image, thus obtaining the weight sequence ZImageWeights corresponding to the color image sequence ZImages.

[0043] In this embodiment, preferably, the texture extraction operation employs a custom filtering method. More preferably, the custom filtering can use a 7*7 Gaussian kernel minus a 3*3 Gaussian kernel, wherein the two Gaussian kernels can be generated using a standard normal distribution with mean μ = 0 and variance σ² = 5.0, and the settings of "7*7" and "3*3" can be adjusted through statistical analysis based on practical experience.

[0044] Step 3: Compare the sharpness of pixels at the same location in the weight sequence ZImageWeights to obtain the sharpness curve for any image location. For example... Figure 4As shown in the figure, this figure is the sharpness curve at a certain position in the image sequence ZImages, where the horizontal axis represents the sequence index value and the vertical axis represents the sharpness of the corresponding sequence index.

[0045] Step 4: Perform polynomial fitting on n (n is a positive integer) points near the peak of the sharpness curve at any image location, solve for the extreme points of the polynomial, and the corresponding x-coordinate is the floating-point value of the depth at that location, thus obtaining the depth map ImageDepth of the observed object.

[0046] To illustrate the above example of using floating-point depth representation, the peak index value (x-axis coordinate) of the sharpness curve is 17. However, 17 is not precise enough. Given images with indices of 14, 15, 16, 17, 18, 19, and 20, a quadratic polynomial fitting is used to solve the problem. The x-axis coordinate corresponding to the maximum value is the floating-point depth value. The final determined floating-point depth value is 17.18. (See [link to relevant documentation]). Figure 5 .

[0047] The confidence level at any location is obtained by summing the values ​​of all points on the sharpness curve at any image location, and then dividing the maximum value of the sharpness curve by the average value. This yields the confidence map ImageConf.

[0048] Step 5: Correct the depth map ImageDepth using the confidence map ImageConf:

[0049] 5a. If the values ​​of all adjacent points at any image location on the confidence graph ImageConf are less than the set threshold m, then the location is determined to be an unstable point, and the row and column coordinates of the unstable point are obtained. Figure 6 This is an example of a depth map with unstable points; the small white boxes on the map represent these unstable points. Figure 7 This is a 3D representation of the depth map.

[0050] In this embodiment, preferably, the threshold m is in the range of [1.15, 1.6], and the threshold value can also be adjusted through statistical analysis based on practical data.

[0051] 5b. Convert each image in the color image sequence ZImages from the RGB color space to the HSL color space, add the converted images together and take the average to obtain three images ImageH, ImageS, and ImageL corresponding to the three components of chroma H, saturation S, and luminance L.

[0052] 5c. Obtain the chromaticity H, saturation S, and brightness L of the image ImageH, ImageS, and ImageL at the row and column coordinates of the unstable point, determine the color space region similar around the row and column coordinates, and search for points in the region with confidence values ​​greater than or equal to the threshold m as stable points, and use the stable points to correct the unstable points.

[0053] In this embodiment, preferably, the region with similar color space is the area around the row and column coordinates of the unstable point that differs from the chromaticity H value of the unstable point by ±3. This "±3" value can also be adjusted through statistical analysis based on practical data.

[0054] In this embodiment, preferably, when searching for stable points in the region of similar color space, if the number of stable points found is less than 5, it indicates that the region of similar color space is incorrect and the corresponding unstable point cannot be corrected; if the number of stable points found is greater than or equal to 5, the average depth of the top 5 stable points with the highest confidence level is used to replace the corresponding unstable point.

[0055] In this embodiment, preferably, after step 5c, the corrected depth map can be subjected to median filtering, mean filtering, bilateral filtering, and other operations in sequence to obtain a depth map with higher accuracy.

[0056] Step 6: Reconstruct the color image based on the corrected depth map.

[0057] In this example, preferably, the operation of reconstructing the color image based on the corrected depth map is as follows: for each position on the corrected depth map, calculate the new pixel RGB value based on the depth d at that position:

[0058] The new pixel RGB value = the pixel RGB value at this position in the k1th image of the color image sequence ZImages * (depth d - depth d rounded down) + the pixel RGB value at this position in the k2th image of the color image sequence ZImages * (depth d rounded up - depth d), where k1 = depth d rounded down and k2 = depth d rounded up.

[0059] For example: If the depth of position (r, c) is 15.6, it means that this point is between the 15th and 16th images, and the pixel proportion of the 15th image is 0.6, while the pixel proportion of the 16th image is 0.4. Then:

[0060] New pixel RGB = RGB of the [r,c] pixel in frame 15 * 0.6 + RGB of the [r,c] pixel in frame 16 * 0.4.

[0061] The reconstructed color image is as follows Figure 8 As shown, it can be used with Figure 6 The comparison shows a significant improvement in image quality.

[0062] In addition to the methods described above, this embodiment also includes a multi-focus microscopic image fusion system, which includes the following modules:

[0063] ① Image sequence acquisition module: used to control the lens to move from the lowest plane to the highest plane of the observed object in fixed steps and acquire images to obtain a color image sequence ZImages.

[0064] ② Texture extraction module: used to extract textures from each image in the color image sequence ZImages.

[0065] Generate a weight map for each image to obtain the color image sequence ZImages.

[0066] The corresponding weight sequence ZImageWeights.

[0067] ③ Sharpness evaluation module: used to compare the sharpness of pixels at the same position in the weight sequence ZImageWeights to obtain the sharpness curve of any image position.

[0068] ④ Depth map generation module: This module is used to perform polynomial fitting on n points near the peak of the sharpness curve at any image location, solve for the extreme points of the polynomial, and the corresponding horizontal coordinate is the floating-point value of the depth at that location, thereby obtaining the depth map ImageDepth of the observed object.

[0069] ⑤ Confidence map generation module: This module is used to sum and average all points on the sharpness curve at any image location, then divide the maximum value of the sharpness curve by the average value to obtain the confidence level at that location, and thus obtain the confidence map ImageConf.

[0070] ⑥ Depth Correction Module: Used to correct the depth map ImageDepth using the confidence map ImageConf.

[0071] ⑦ Image Reconstruction Module: Used to reconstruct a color image using the corrected depth map.

[0072] In addition to the methods and systems described above, this embodiment also includes a program product for implementing multi-focus microscopic image fusion, comprising a computer program or instructions that, when executed by a processor, implement the steps of the methods described above.

[0073] The core technical point of this invention is to use the sharpness curve to solve the confidence level, use the confidence level to determine the unstable points of the depth map, and use HSL color space guided filtering to correct the depth data of areas without texture or with weak texture, thereby avoiding the phenomenon of incorrect focal plane judgment in the depth map due to the presence of areas without texture or with weak texture in the image source.

[0074] From a vertical perspective, the sharpness curve of each pixel is different. A normal curve is distributed in a "peak" shape, so its peak is reliable, and the depth information at this point is correct. However, exposure anomalies (overexposure, underexposure) cause partial loss of image surface information. The focus curve value of this coordinate point is too flat, and the "peak" shape is not obvious. Therefore, this point is considered unstable, incorrect, and does not express clear depth. In short, multiple consecutive images above and below this point are blurry. For all coordinate points, a confidence value (the maximum value on the sharpness curve divided by the average value) is attached to the sharpness curve to evaluate the degree of its "peak" characteristics. During correction, a threshold comparison method is used. For confidence values ​​below the threshold, some operations are performed. The core is to find coordinate points with higher confidence around this point and replace the coordinate points with lower confidence values ​​with their depth values. Because the height information of an object within a small area is unlikely to change abruptly, using the surrounding area as a substitute is feasible. By increasing the number of normal points, the resulting 3D model becomes closer to the shape of real objects. However, determining the surrounding area also requires establishing rules. Aimless searching of the surrounding area leads to excessively long search times and low algorithm efficiency. Therefore, this invention proposes using color space conversion and the characteristics of the HSL space to determine the surrounding area. Searching within this area allows for the substitution of unstable points for stable ones, thus reducing search time and improving algorithm efficiency.

[0075] The embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for fusing multifocal microscopic images, characterized in that, Includes the following steps: (1) Move the lens from the lowest plane to the highest plane of the observed object in fixed steps and acquire images to obtain a color image sequence ZImages; (2) Extract texture from each image in the color image sequence ZImages to generate a weight map corresponding to each image, and obtain the weight sequence ZImageWeights corresponding to the color image sequence ZImages; (3) Compare the sharpness of pixels at the same position in the weight sequence ZImageWeights to obtain the sharpness curve of any image position; (4) Perform polynomial fitting on n points near the peak of the sharpness curve at any image location, solve for the extreme points of the polynomial, and the corresponding horizontal coordinate is the floating-point value of the depth at that location, thereby obtaining the depth map ImageDepth of the observed object; accumulate all points on the sharpness curve at any image location and take the average value, then divide the maximum value of the sharpness curve by the average value to obtain the confidence level of that location, thereby obtaining the confidence map ImageConf; (5) Correct the depth map ImageDepth using the confidence map ImageConf: (5a) If the values ​​of all adjacent points at any image location are less than the set threshold m, then the location is determined to be an unstable point, and the row and column coordinates of the unstable point are obtained. (5b) Each image in the color image sequence ZImages is converted from RGB color space to HSL color space. The converted images are added together and averaged to obtain three images ImageH, ImageS, and ImageL corresponding to the three components of chroma H, saturation S, and luminance L. (5c) Obtain the chromaticity H, saturation S, and brightness L of the image ImageH, ImageS, and ImageL at the row and column coordinates of the unstable point, determine the color space region around the row and column coordinates, and search for points in the region with confidence values ​​greater than or equal to the threshold m as stable points, and use the stable points to correct the unstable points. (6) Reconstruct the color image based on the corrected depth map.

2. The multifocal microscopic image fusion method according to claim 1, characterized in that: In step (2), the texture extraction operation is to perform a custom filtering operation on each image in the color image sequence ZImages.

3. The multifocal microscopic image fusion method according to claim 2, characterized in that: The custom filtering operation uses a 7*7 Gaussian kernel minus a 3*3 Gaussian kernel.

4. The multifocal microscopic image fusion method according to claim 1, characterized in that: In step (5a), the threshold m ranges from [1.15, 1.6].

5. The multifocal microscopic image fusion method according to claim 1, characterized in that: In step (5c), the region with similar color space is the region around the row and column coordinates of the unstable point that differs from the chromaticity H value of the unstable point by ±3.

6. The multifocal microscopic image fusion method according to claim 1, characterized in that: In step (5c), for searching stable points in the region of similar color space, if the number of stable points found is less than 5, it means that the region of similar color space is incorrect and the corresponding unstable point cannot be corrected; if the number of stable points found is greater than or equal to 5, the average depth of the top 5 stable points with the highest confidence level is taken to replace the corresponding unstable point.

7. The multifocal microscopic image fusion method according to claim 1, characterized in that: After step (5c), the corrected depth map is subjected to median filtering, mean filtering, and bilateral filtering operations in sequence to obtain the final corrected depth map.

8. The multifocal microscopic image fusion method according to claim 1, characterized in that: In step (6), the operation of reconstructing the color image based on the corrected depth map is as follows: for each position on the corrected depth map, calculate the new pixel RGB value based on the depth d at that position: The new pixel RGB value = the pixel RGB value at this position in the k1th image of the color image sequence ZImages * (depth d - depth d rounded down) + the pixel RGB value at this position in the k2th image of the color image sequence ZImages * (depth d rounded up - depth d), where k1 = depth d rounded down and k2 = depth d rounded up.

9. A multi-focus microscopic image fusion system, characterized in that, include: Image sequence acquisition module: used to control the lens to move from the lowest plane to the highest plane of the observed object in fixed steps and acquire images to obtain a color image sequence ZImages; Texture extraction module: used to extract textures from each image in the color image sequence ZImages, generate a weight map corresponding to each image, and obtain the weight sequence ZImageWeights corresponding to the color image sequence ZImages; Sharpness evaluation module: used to compare the sharpness of pixels at the same position in the weight sequence ZImageWeights to obtain a sharpness curve for any image position; Depth map generation module: Used to perform polynomial fitting on n points near the peak of the sharpness curve at any image location, solve for the extreme points of the polynomial, and the corresponding x-coordinate is the floating-point value of the depth at that location, thereby obtaining the depth map ImageDepth of the observed object; Confidence map generation module: It is used to accumulate and average all points on the sharpness curve at any image location, and then divide the maximum value of the sharpness curve by the average value to obtain the confidence level at that location, and thus obtain the confidence map ImageConf; Depth Correction Module: This module is used to correct the depth map ImageDepth using the confidence map ImageConf. The specific processing procedure of this module is as follows: (a) If the values ​​of all adjacent points at any image location are less than the set threshold m, then the location is determined to be an unstable point, and the row and column coordinates of the unstable point are obtained. (b) Convert each image in the color image sequence ZImages from RGB color space to HSL color space, and then sum the converted images and take the average. The three images ImageH, ImageS, and ImageL corresponding to the three components of chroma H, saturation S, and luminance L are obtained; (c) Obtain the chromaticity H, saturation S, and brightness L of the image ImageH, ImageS, and ImageL at the row and column coordinates of the unstable point, determine the color space region around the row and column coordinates, and search for points in the region with confidence values ​​greater than or equal to the threshold m as stable points, and use the stable points to correct the unstable points. Image reconstruction module: Used to reconstruct a color image using the corrected depth map.

10. A program product for realizing multifocal microscopic image fusion, comprising a computer program or instructions, characterized in that: When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1-8.

Images