Multi-focus microscopic image fusion method, system and program based on depth map correction

Abstract

The invention discloses a multi-focus microscopic image fusion method, system and program based on depth map correction. The method sequentially comprises the steps of image sequence collection, texture extraction, definition evaluation, depth map generation, confidence map generation, depth correction, color image reconstruction and the like. According to the method, the depth map and the confidence map are determined according to the definition curve, the unstable points are determined by using the confidence map, and the depth data of the non-texture and weak-texture areas are corrected through the HSL color space guide filtering, so that the phenomenon of focal plane judgment error of the depth map caused by the non-texture and weak-texture areas existing in the image source is avoided.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and particularly relates to a multi-focus microscopic image technology. Background Art

[0002] Depth of field is the depth of space where the lens is focused, and within this range a clear image of the object can be obtained. The depth of field of a microscope lens is very narrow, usually between a few microns and a few hundred microns. With the development of microscopic imaging technology, depth synthesis technology came into being. The so-called depth synthesis technology is for the situation where the depth of field is small and the whole is not fully focused, but there are still some parts that can be fully focused. When the distance between the lens and the object changes, the focus also moves. By moving the focus of the lens to select multiple images, and then extracting and synthesizing the clear parts, you can get a depth of field synthesis image that is clear as a whole.

[0003] However, the existing depth synthesis technology still has many technical defects, which affect the quality and effect of the generated image. These technical defects mainly include: due to the existence of textureless or weak texture areas in the image source, the depth map produces incorrect focus judgment; the depth map calculation is not fine enough, resulting in the "terracing phenomenon" (the so-called "terracing phenomenon" refers to the excessively abrupt depth of the depth map, with obvious steps), such as Figure 1 In addition, the fused color image may have artifacts, halos, and inconsistent fusion. Summary of the invention

[0004] In order to solve the technical problems mentioned in the above background technology, the present invention proposes a multi-focus microscopic image fusion method, system and program based on depth map correction.

[0005] In order to achieve the above technical objectives, the technical solution of the present invention is:

[0006] First, the present invention designs a multi-focus microscopic image fusion method, which includes the following steps:

[0007] (1) The lens is moved from the lowest plane to the highest plane of the observed object according to a fixed step length and images are collected to obtain a color image sequence ZImages;

[0008] (2) performing texture extraction on each image in the color image sequence ZImages, generating a weight map corresponding to each image, and obtaining a weight sequence ZImageWeights corresponding to the color image sequence ZImages;

[0009] (3) Comparing the clarity of pixels at the same position in the weight sequence ZImageWeights to obtain a clarity curve at any image position;

[0010] (4) Perform polynomial fitting on n points near the peak of the clarity curve at any image position, solve the extreme point of the polynomial, and its corresponding horizontal coordinate is the floating point value of the depth of the position, thereby obtaining the depth map ImageDepth of the observed object; accumulate and average all points on the clarity curve at any image position, and then divide the maximum value of the clarity curve by the average value to obtain the confidence of the position, thereby obtaining the confidence map ImageConf;

[0011] (5) Using the confidence map ImageConf to correct the depth map ImageDepth:

[0012] (5a) If the values ​​of all adjacent points of an arbitrary image position on the confidence map ImageConf are less than the set threshold m, the position is determined to be an unstable point, and the row and column coordinates of the unstable point are obtained;

[0013] (5b) converting each image in the color image sequence ZImages from the RGB color space to the HSL color space, adding and averaging the converted images to obtain three images ImageH, ImageS, and ImageL corresponding to the three components of hue H, saturation S, and brightness L;

[0014] (5c) obtaining the chromaticity H, saturation S, and brightness L of the images ImageH, ImageS, and ImageL at the row and column coordinates of the unstable point, determining a region with a similar color space around the row and column coordinates, and searching for a point in the region whose confidence value is greater than or equal to the threshold m as a stable point, and using the stable point to correct the unstable point;

[0015] (6) Reconstruct the color image based on the corrected depth map.

[0016] Secondly, the present invention also proposes a multi-focus microscopic image fusion system, comprising:

[0017] ① Image sequence acquisition module: used to control the lens to move from the lowest plane to the highest plane of the observed object according to a fixed step length and collect images to obtain a color image sequence ZImages;

[0018] ② Texture extraction module: used to extract texture from each image in the color image sequence ZImages.

[0019] Generate a weight map corresponding to each image to obtain the color image sequence ZImages

[0020] The corresponding weight sequence ZImageWeights;

[0021] ③ Clarity evaluation module: used to compare the clarity of pixels at the same position in the weight sequence ZImageWeights to obtain a clarity curve at any image position;

[0022] ④ Depth map generation module: used to perform polynomial fitting on n points near the peak of the clarity curve at any image position, solve the extreme point of the polynomial, and its corresponding horizontal coordinate is the floating point value of the depth of the position, thereby obtaining the depth map ImageDepth of the observed object;

[0023] ⑤Confidence map generation module: It is used to accumulate and average all points on the clarity curve of any image position, and then divide the maximum value of the clarity curve by the average value to obtain the confidence of the position, and then obtain the confidence map ImageConf;

[0024] ⑥ Depth correction module: used to correct the depth map ImageDepth using the confidence map ImageConf;

[0025] ⑦Image reconstruction module: used to reconstruct a color image using the corrected depth map.

[0026] In addition, the present invention also proposes a program product for realizing multi-focus microscopic image fusion, including a computer program or instructions, and the steps of the above method are realized when the computer program or instructions are executed by a processor.

[0027] The beneficial effects brought by adopting the above technical solution are:

[0028] 1. The present invention determines a depth map and a confidence map according to a clarity curve, and uses the confidence map to determine unstable points, and corrects the depth data of texture-free and weak-texture areas through HSL color space guided filtering, thereby avoiding the phenomenon of incorrect focus judgment in the depth map due to the existence of texture-free and weak-texture areas in the image source.

[0029] 2. The present invention uses floating-point data to represent depth data, so that the depth map can be subdivided by integer data, thereby solving the "terrace phenomenon".

[0030] 3. The present invention uses a floating-point depth map to guide the fusion of color images, which can eliminate some artifacts, halos, and uneven fusion. The weight ratio of the two adjacent images can be subdivided according to the "floating point number", and the fusion weight of the two images can be dynamically adjusted. The color RGB value fusion is more refined, and thousandths floating-point fusion can be achieved. The performance and quality are better than the traditional "seam weight fusion". BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the "terracing phenomenon", in which the red, green and blue arrows represent the three components of RGB;

[0032] Figure 2 It is a simplified flow chart of the method of the present invention;

[0033] Figure 3 is a schematic diagram of obtaining a color image sequence;

[0034] Figure 4 It is a schematic diagram of the clarity curve;

[0035] Figure 5 It is a schematic diagram of polynomial fitting;

[0036] Figure 6 This is an example of a depth map with unstable points;

[0037] Figure 7 yes Figure 6 3D display diagram;

[0038] Figure 8 This is an example of a color image reconstruction. DETAILED DESCRIPTION

[0039] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] This embodiment proposes a multi-focus microscopic image fusion method, such as Figure 2 As shown, the method comprises the following steps:

[0041] Step 1: Move the microscope z-axis, which drives the lens, from the lowest surface of the object to the highest surface of the object, and obtain the z-axis color image sequence ZImages, such as Figure 3 In this step, the image needs to be free of jitter and the moving interval needs to be stable.

[0042] Step 2: Perform texture extraction on each image in the color image sequence ZImages, generate a weight map corresponding to each image, and obtain a weight sequence ZImageWeights corresponding to the color image sequence ZImages.

[0043] In this embodiment, preferably, the texture extraction operation adopts a custom filtering method. Further preferably, the custom filtering can adopt a filter kernel of 7*7 Gaussian kernel minus 3*3 Gaussian kernel, wherein the two Gaussian kernels can be generated by a standard normal distribution with a mean μ=0 and a variance σ2=5.0, and the settings of "7*7" and "3*3" can be adjusted by using statistical analysis according to practical conditions.

[0044] Step 3: Compare the clarity of pixels at the same position in the weight sequence ZImageWeights to obtain the clarity curve of any image position. Figure 4As shown in FIG. 1 , this figure is a clarity curve of a certain position in the image sequence ZImages, wherein the abscissa represents the sequence index value, and the ordinate represents the clarity of the corresponding sequence index.

[0045] Step 4: Perform polynomial fitting on n (n is a positive integer) points near the peak of the clarity curve at any image position, solve the extreme point of the polynomial, and its corresponding horizontal coordinate is the floating point value of the depth of the position, thereby obtaining the depth map ImageDepth of the observed object.

[0046] Take the above floating-point depth expression as an example. The peak index value (x-axis coordinate) of the sharpness curve is 17, but 17 is not accurate enough. The existing images with indexes of 14, 15, 16, 17, 18, 19, and 20 are solved using quadratic polynomial fitting. The x-axis coordinate corresponding to the maximum value is the floating-point value of the depth. The final depth floating-point value is determined to be 17.18. See Figure 5 .

[0047] The average value of all points on the clarity curve of any image position is accumulated, and then the maximum value of the clarity curve is divided by the average value to obtain the confidence of the position, and then the confidence map ImageConf is obtained.

[0048] Step 5: Use the confidence map ImageConf to correct the depth map ImageDepth:

[0049] 5a. If the values ​​of each adjacent point of an arbitrary image position on the confidence map ImageConf are all less than the set threshold m, then the position is determined to be an unstable point, and the row and column coordinates of the unstable point are obtained. Figure 6 This is an example of a depth map with unstable points. The small white boxes on the map are unstable points. Figure 7 is a 3D display of the depth map.

[0050] In this embodiment, preferably, the threshold m has a value range of [1.15, 1.6], and the threshold value can also be adjusted through statistical analysis according to practical data.

[0051] 5b. Convert each image in the color image sequence ZImages from the RGB color space to the HSL color space, add and average the converted images to obtain three images ImageH, ImageS, and ImageL corresponding to the three components of hue H, saturation S, and brightness L.

[0052] 5c. Obtain the chromaticity H, saturation S, and brightness L of the row and column coordinates of the images ImageH, ImageS, and ImageL at the unstable point, determine an area with a similar color space around the row and column coordinates, and search for a point in the area whose confidence value is greater than or equal to the threshold m as a stable point, and use the stable point to correct the unstable point.

[0053] In this embodiment, preferably, the region with similar color space is a region around the row and column coordinates of the unstable point whose chromaticity H value differs from the unstable point by less than ±3. The value of "±3" can also be adjusted through statistical analysis according to practical data.

[0054] In this embodiment, preferably, for searching for stable points in the area similar to the color space, if the number of stable points searched out is less than 5, it means that the determined area similar to the color space is wrong and the corresponding unstable points cannot be corrected; if the number of stable points searched out is greater than or equal to 5, the average value of the depths of the first 5 stable points with confidence levels from large to small is taken to replace the corresponding unstable points.

[0055] In this embodiment, preferably, after step 5c, the corrected depth map may be subjected to operations such as median filtering, mean filtering, and bilateral filtering in sequence, so as to obtain a depth map with higher accuracy.

[0056] Step 6: Reconstruct the color image based on the corrected depth map.

[0057] In this example, preferably, the operation of reconstructing the color image based on the modified depth map is to calculate the RGB value of a new pixel according to the depth d of each position on the modified depth map:

[0058] The RGB value of the new pixel = the RGB value of the pixel at that position in the k1th image in the color image sequence ZImages * (depth d - depth d rounded down) + the RGB value of the pixel at that position in the k2th image in the color image sequence ZImages * (depth d rounded up - depth d), where k1 = depth d rounded down, k2 = depth d rounded up.

[0059] For example: the depth of position (r,c) is 15.6, which means that this point is between the 15th and 16th images, and the pixel ratio of the 15th image is 0.6, and the pixel ratio of the 16th image is 0.4, then:

[0060] New pixel RGB = RGB of the 15th frame [r, c] pixel * 0.6 + RGB of the 16th frame [r, c] pixel * 0.4.

[0061] The reconstructed color image is Figure 8 As shown, it can be Figure 6 By comparison, we can see that the image quality has been significantly improved.

[0062] In addition to the above method, this embodiment also includes a multi-focus microscopic image fusion system, which includes the following modules:

[0063] ① Image sequence acquisition module: used to control the lens to move from the lowest plane to the highest plane of the observed object according to a fixed step size and collect images to obtain a color image sequence ZImages.

[0064] ② Texture extraction module: used to extract texture from each image in the color image sequence ZImages.

[0065] Generate a weight map corresponding to each image to obtain the color image sequence ZImages

[0066] The corresponding weight sequence ZImageWeights.

[0067] ③ Clarity evaluation module: used to compare the clarity of pixels at the same position in the weight sequence ZImageWeights to obtain a clarity curve at any image position.

[0068] ④ Depth map generation module: It is used to perform polynomial fitting on n points near the peak of the clarity curve at any image position, solve the extreme points of the polynomial, and the corresponding horizontal coordinate is the floating point value of the depth of the position, thereby obtaining the depth map ImageDepth of the observed object.

[0069] ⑤Confidence map generation module: It is used to accumulate and average all points on the clarity curve of any image position, and then divide the maximum value of the clarity curve by the average value to obtain the confidence of the position, and then obtain the confidence map ImageConf.

[0070] ⑥ Depth correction module: used to correct the depth map ImageDepth using the confidence map ImageConf.

[0071] ⑦Image reconstruction module: used to reconstruct a color image using the corrected depth map.

[0072] In addition to the above method and system, this embodiment also includes a program product for implementing multi-focus microscopic image fusion, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0073] The core technical point of the present invention is to use the clarity curve to solve the confidence, and use the confidence to determine the unstable points of the depth map, and correct the depth data of the texture-free and weak-textured areas through HSL color space guided filtering, thereby avoiding the phenomenon of incorrect focus judgment of the depth map due to the existence of texture-free and weak-textured areas in the image source.

[0074] From a vertical perspective, the clarity curve of each pixel is different. The normal curve is distributed in the shape of a "peak", so its peak is reliable, so the depth information of this point is correct. However, abnormal exposure (overexposure, underexposure) causes partial loss of image surface information. The focus curve value of this coordinate point is too flat, and the "peak" shape is not obvious. Therefore, this point is considered unstable, wrong, and does not express clear depth. In short, multiple images above and below this point are not clear. For all coordinate points, a confidence value (the maximum value on the clarity curve divided by the average value) is bound to the clarity curve to judge its "peak" characteristic degree. When correcting, the threshold comparison method is used to perform some operations on the confidence below the threshold. The core is to find the coordinate points with higher confidence around this point and use their depth values ​​to replace the coordinate points with low confidence. Because the height information of objects in a small range cannot change suddenly, it is feasible to use the surrounding range as a substitute. By increasing the number of normal points, the formed 3D model is closer to the real object shape. However, the determination of the surrounding area also requires the creation of rules to determine it. Aimlessly searching the surrounding area will result in a long search time and low algorithm efficiency. Therefore, the present invention proposes to use the conversion of the color space and the characteristics of the HSL space to determine the surrounding area, search within the area, and replace the unstable point with the stable point, which can reduce the search time and improve the algorithm efficiency.

[0075] The embodiments are only for illustrating the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A multi-focus microscopic image fusion method, characterized in that: The following steps are involved: (1) The lens is moved from the lowest plane to the highest plane of the observed object according to a fixed step length and images are collected to obtain a color image sequence ZImages; (2) performing texture extraction on each image in the color image sequence ZImages, generating a weight map corresponding to each image, and obtaining a weight sequence ZImageWeights corresponding to the color image sequence ZImages; (3) Comparing the clarity of pixels at the same position in the weight sequence ZImageWeights to obtain a clarity curve at any image position; (4) Perform polynomial fitting on n points near the peak of the clarity curve at any image position, solve the extreme point of the polynomial, and its corresponding horizontal coordinate is the floating point value of the depth of the position, thereby obtaining the depth map ImageDepth of the observed object; accumulate and average all points on the clarity curve at any image position, and then divide the maximum value of the clarity curve by the average value to obtain the confidence of the position, thereby obtaining the confidence map ImageConf; (5) Using the confidence map ImageConf to correct the depth map ImageDepth: (5a) If the values ​​of all adjacent points of an arbitrary image position on the confidence map ImageConf are less than the set threshold m, the position is determined to be an unstable point, and the row and column coordinates of the unstable point are obtained; (5b) converting each image in the color image sequence ZImages from the RGB color space to the HSL color space, adding and averaging the converted images to obtain three images ImageH, ImageS, and ImageL corresponding to the three components of hue H, saturation S, and brightness L; (5c) obtaining the chromaticity H, saturation S, and brightness L of the images ImageH, ImageS, and ImageL at the row and column coordinates of the unstable point, determining a region with a similar color space around the row and column coordinates, and searching for a point in the region whose confidence value is greater than or equal to the threshold m as a stable point, and using the stable point to correct the unstable point; (6) Reconstruct the color image based on the corrected depth map.

2. The multi-focus microscopic image fusion method according to claim 1, characterized in that: In step (2), the texture extraction operation is to perform a custom filtering operation on each image in the color image sequence ZImages.

3. The multi-focus microscopic image fusion method according to claim 2, characterized in that: The filter kernel used in the custom filtering operation is a 7*7 Gaussian kernel minus a 3*3 Gaussian kernel.

4. The multi-focus microscopic image fusion method according to claim 1, characterized in that: In step (5a), the threshold m is in the range of [1.15, 1.6].

5. The multi-focus microscopic image fusion method according to claim 1, characterized in that: In step (5c), the region close to the color space is a region around the row and column coordinates of the unstable point whose chromaticity H value differs from that of the unstable point by less than ±3.

6. The multi-focus microscopic image fusion method according to claim 1, characterized in that: In step (5c), for searching stable points in the area similar to the color space, if the number of stable points searched out is less than 5, it means that the determined area similar to the color space is wrong and the corresponding unstable points cannot be corrected; if the number of stable points searched out is greater than or equal to 5, the average value of the depths of the first 5 stable points with confidence levels from large to small is taken to replace the corresponding unstable points.

7. The multi-focus microscopic image fusion method according to claim 1, characterized in that: After step (5c), the corrected depth map is subjected to median filtering, mean filtering, and bilateral filtering operations in sequence to obtain a final corrected depth map.

8. The multi-focus microscopic image fusion method according to claim 1, characterized in that: In step (6), the operation of reconstructing the color image based on the modified depth map is to calculate the RGB value of the new pixel according to the depth d of each position on the modified depth map: The RGB value of the new pixel = the RGB value of the pixel at that position in the k1th image in the color image sequence ZImages * (depth d - depth d rounded down) + the RGB value of the pixel at that position in the k2th image in the color image sequence ZImages * (depth d rounded up - depth d), where k1 = depth d rounded down, k2 = depth d rounded up.

9. A multi-focus microscopic image fusion system, characterized in that: include: Image sequence acquisition module: used to control the lens to move from the lowest plane to the highest plane of the observed object according to a fixed step length and collect images to obtain a color image sequence ZImages; Texture extraction module: used for performing texture extraction on each image in the color image sequence ZImages, generating a weight map corresponding to each image, and obtaining a weight sequence ZImageWeights corresponding to the color image sequence ZImages; Definition evaluation module: used to compare the definition of pixels at the same position in the weight sequence ZImageWeights to obtain a definition curve at any image position; Depth map generation module: used to perform polynomial fitting on n points near the peak of the clarity curve at any image position, solve the extreme point of the polynomial, and its corresponding horizontal coordinate is the floating point value of the depth of the position, thereby obtaining the depth map ImageDepth of the observed object; Confidence map generation module: It is used to accumulate and average all points on the clarity curve of any image position, and then divide the maximum value of the clarity curve by the average value to obtain the confidence of the position, and then obtain the confidence map ImageConf; Depth correction module: used to correct the depth map ImageDepth using the confidence map ImageConf. The specific processing process of this module is as follows: (a) If the values ​​of all adjacent points of an arbitrary image position on the confidence map ImageConf are less than the set threshold m, then the position is determined to be an unstable point, and the row and column coordinates of the unstable point are obtained; (b) converting each image in the color image sequence ZImages from the RGB color space to the HSL color space, adding and averaging the converted images, Get three images ImageH, ImageS, ImageL corresponding to the three components of hue H, saturation S, and brightness L; (c) obtaining the chromaticity H, saturation S, and brightness L of the images ImageH, ImageS, and ImageL at the row and column coordinates of the unstable point, determining a region with a similar color space around the row and column coordinates, searching for a point in the region whose confidence value is greater than or equal to the threshold m as a stable point, and using the stable point to correct the unstable point; Image reconstruction module: used to reconstruct a color image using the corrected depth map.

10. A program product for implementing multi-focus microscopic image fusion, comprising a computer program or instructions, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images