Focusing morphology recovery method for converting data dimension and projection direction

Abstract

According to the focusing morphology recovery method for converting the data dimension and the projection direction, on the basis of data dimension conversion, more useful information can be obtained by changing different projection directions; firstly, a two-dimensional matrix of an image window is converted into a point cloud format of a three-dimensional space, then projection is carried out around, new different matrix information can be obtained, and a final depth map is determined by calculating the projection area and the fluctuation degree of each row in projection and carrying out weighted summation as a final focus value.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional morphology reconstruction, and in particular relates to a focusing morphology restoration method by transforming data dimension and projection direction. Background Art

[0002] Reconstructing 3D shape from 2D images is one of the basic tasks of computer vision. Shape from focus (SFF) is a passive monocular optical technique that reconstructs the 3D shape of an object from a multi-focus image sequence. These image sequences are captured by changing the focal length, image distance or object distance of the optical system. The image focus level is used as a clue for shape reconstruction. Each point in the reconstructed depth map corresponds to the focus position in the image sequence. Compared with active techniques and multi-camera systems, the main advantage of SFF is that it does not require physical interaction between the scene and the capture device, is compact in size, low in cost, and easier to implement. Only one camera and a translation stage are needed to complete the most basic operations. Compared with the shape from defocus (SFD) technique, the reconstruction result is more reliable. Therefore, SFF technology has many applications in microscopy, intelligent manufacturing, medicine and other fields.

[0003] The main process of SFF technology consists of three steps: image acquisition, focus measurement, and depth reconstruction. In the first step, by moving the object along the optical axis, a multi-focus image sequence is captured, and then the focus measurement (FM) operator is used to estimate the clarity or sharpness level of each pixel in the image sequence. The measurement result has the same size as the image sequence and is called the focus volume (FV). Then, the focus measurement value is maximized along the optical axis, and the initial depth map is obtained from the FV. In order to obtain a more accurate depth map, it is necessary to refine the FV or optimize the initial depth map. The optimization process is also called the post-processing technology of SFF. Although post-processing technology can improve the accuracy of the reconstruction results, if there are more erroneous data in the FV or depth map, there will be problems with error propagation, and the optimization effect is limited. Therefore, obtaining accurate focus values ​​is the basis of SFF morphology restoration and is the top priority. This has always been a hot topic in current research.

[0004] In SFF technology, the commonly used FM operator calculates the focus metric in the spatial domain and the transform domain. In the spatial domain, the structural edge of the image is used as the focus metric, and the edge information is obtained by calculating the first-order derivative and the second-order derivative of the image. Although the second-order derivative can obtain richer information, it is also sensitive to noise. The most famous FM operator based on statistics is the variance in the calculation window. When the region contains more textures, there is a larger variance. In the transform domain, wavelet transform and discrete cosine transform are often used to find the high-frequency components of the image, because the detail information is often contained in the high-frequency components. These operators can produce good reconstruction effects when there are rich textures on the surface of the object, but they are easily affected by noise in real scenes and have poor performance in identifying weak texture areas. In recent years, many new FM methods have been proposed. Jang et al. proposed a method using adaptive windows. First, the full-focus image calculated by the GLV method is used to determine the size of the window at each position according to the texture information of the full-focus image. This can effectively improve the running speed and make the processing results more reliable, but the reliability depends on the full-focus image. Lee et al. proposed a FM method based on spatial transformation, which transforms the image from Cartesian space to log polar coordinate space. In log polar coordinate transformation, this process is generally considered to be a form of low-pass filtering. This averaging process effectively improves the noise tolerance. When there is noise in the image, this spatial transformation can calculate a better focus value, but this spatial transformation is affected by the coordinate poles, and the focus value varies according to the position of the poles. Hae-Gon Jeon optimized the processing window shape and proposed a ring difference filter, which can obtain more accurate measurement results compared with traditional windows. Some scholars also proposed to use multiple FM operators, use genetic programming or find the maximum consensus on the calculated FV to obtain the best focus position. Using multiple FM operators will increase the processing accuracy, but it will also increase the calculation time, and the accuracy improvement is limited. Whether using adaptive windows and deformed windows, or using multiple FM operators, although the accuracy of focus measurement can be improved to a certain extent, a large amount of noise will still be generated in FV, which is reflected in the depth map as a large number of erroneous depth estimates at the edges of structures and high mutations, and slight depth fluctuations in smooth areas. In summary, due to the inherent limitations of the FM operator, it is difficult to obtain accurate focus values ​​for different objects using a single FM method.

[0005] In summary, the improvements of the FM operator are mostly in data transformation and using more methods to obtain more accurate calculation results, which will greatly increase the calculation time, and most of them use single data without expanding and enriching the data. Summary of the invention

[0006] To solve the above problems, the present invention provides a focusing morphology restoration method by transforming data dimension and projection direction, which can increase the amount of available information and obtain a better three-dimensional morphology of the object.

[0007] A method for restoring a focused shape by transforming data dimension and projection direction comprises the following steps:

[0008] Acquire an image sequence of the object to be reconstructed;

[0009] The focus value corresponding to each pixel point on each frame image in the image sequence is obtained respectively, wherein the method for obtaining the focus value corresponding to any pixel point is as follows: a pixel area with the current pixel point as the window center is extracted by using a set window size, and the pixel area is projected into a three-dimensional space to obtain three-dimensional point cloud data; the projection area and the total fluctuation degree of the broken line of the three-dimensional point cloud data on the xz projection plane and the yz projection plane are obtained, and the projection area and the total fluctuation degree of the broken line are weightedly summed, and the sum value is the focus value corresponding to the current pixel point;

[0010] The maximum value of the focus values ​​corresponding to all pixels at the same position is taken as the depth value at each position to obtain the depth map of the object to be reconstructed.

[0011] Furthermore, the pixel area is projected into the three-dimensional space to obtain the three-dimensional point cloud data q:

[0012]

[0013] Among them, N is the size of the setting window, G (1,1) ~G (N,N) is the pixel value of each pixel in the pixel area.

[0014] Furthermore, the method for obtaining the projection area of ​​the three-dimensional point cloud data on the xz projection plane is:

[0015] Project the 3D point cloud data onto the xz projection plane, and select the point cloud projection point set h{(x1,z1),(x2,z2),…,(x k ,z k )}, where k is the number of point cloud projection points that enclose the maximum convex hull;

[0016] Calculate the convex hull area A xz as follows:

[0017]

[0018] The convex hull area A xz As the projection area of ​​3D point cloud data on the xz projection plane.

[0019] Furthermore, the method for obtaining the projection area of ​​the three-dimensional point cloud data on the yz projection plane is:

[0020] Project the 3D point cloud data onto the yz projection plane, and select the point cloud projection point set h{(x1,z1),(x2,z2),…,(x k ,z k )}, where k is the number of point cloud projection points that enclose the maximum convex hull;

[0021] Calculate the convex hull area A yz as follows:

[0022]

[0023] The convex hull area A yz As the projection area of ​​3D point cloud data on the yz projection plane.

[0024] Furthermore, the method for obtaining the total fluctuation degree of the broken line of the three-dimensional point cloud data on the xz projection plane is as follows:

[0025] Treat the 3D point cloud data with the same x coordinate as one column;

[0026] Sort each column of 3D point cloud data from large to small according to the z-axis coordinate;

[0027] Connect the projection points of the three-dimensional point cloud data with the same ranking on the xz projection plane in each column of the three-dimensional point cloud data into a broken line;

[0028] Get the volatility of each broken line separately:

[0029]

[0030] Where i = 1, 2, ..., B, B is the total number of broken lines, j = 1, 2, ..., L, L is the number of points on the broken line; the total fluctuation degree of the broken line is obtained as follows:

[0031]

[0032] Among them, d xz It is the total fluctuation degree of the broken line of the 3D point cloud data on the xz projection plane.

[0033] Furthermore, the method for obtaining the total fluctuation degree of the broken line of the three-dimensional point cloud data on the yz projection plane is as follows:

[0034] Treat the 3D point cloud data with the same y coordinate as a column;

[0035] Sort each column of 3D point cloud data from large to small according to the z-axis coordinate;

[0036] Connect the projection points of the three-dimensional point cloud data with the same ranking on the yz projection plane in each column of the three-dimensional point cloud data into a broken line;

[0037] Get the volatility of each broken line separately:

[0038]

[0039] Where i = 1, 2, ..., B, B is the total number of broken lines, j = 1, 2, ..., L, L is the number of points on the broken line; the total fluctuation degree of the broken line is obtained as follows:

[0040]

[0041] Among them, d yz It is the total fluctuation degree of the broken line of the 3D point cloud data on the yz projection plane.

[0042] Furthermore, the method of performing weighted summation on the projection area and the total fluctuation degree of the broken line to obtain the focus value corresponding to the current pixel point is:

[0043] Get the focus value FM corresponding to the current pixel:

[0044] FM=λ1(A xz +A yz )+λ2(d xz +d yz )

[0045] Among them, λ1 and λ2 are set weights, A xz , A yz are the projection areas of the 3D point cloud data corresponding to the current pixel on the xz projection plane and the yz projection plane, respectively. xz d yz are the total fluctuation degree of the broken line of the three-dimensional point cloud data corresponding to the current pixel on the xz projection plane and the yz projection plane respectively;

[0046]

[0047] Among them, D(x,y) is the depth value corresponding to the current pixel.

[0048] Beneficial effects:

[0049] The present invention provides a focus morphology restoration method for transforming data dimension and projection direction. On the basis of data dimension transformation, more useful information can be obtained by changing different projection directions. First, the two-dimensional matrix of the image window is transformed into a point cloud format in three-dimensional space, and then projected to all sides to obtain new and different matrix information. The final depth map is determined by calculating the projection area and the fluctuation degree of each row in the projection, and taking the weighted sum as the final focus value. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a focusing morphology restoration method for transforming data dimension and projection direction provided by the present invention;

[0051] Figure 2 A schematic diagram of data transformation and projection in different directions provided by the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0053] The present invention proposes a method for focusing morphology restoration by transforming data dimension and projection direction. First, the dimension of the image window matrix is ​​transformed from two-dimensional space to three-dimensional point cloud. Then, projection is performed in different directions. The projection area in different projection directions and the fluctuation degree of each row of data on the projection surface are calculated respectively. The weighted sum is taken as the focus value to obtain the depth map of the object. Figure 1 As shown, the method comprises the following steps:

[0054] Acquire an image sequence of the object to be reconstructed; for example, use a monocular vision system to collect a partially focused image sequence of the object at equal intervals along the z-axis direction, and convert the image from an RGB format to a grayscale image;

[0055] The focus value corresponding to each pixel point on each frame image in the image sequence is obtained respectively, wherein the method for obtaining the focus value corresponding to any pixel point is as follows: a pixel area with the current pixel point as the window center is extracted by using a set window size, and the pixel area is projected into a three-dimensional space to obtain three-dimensional point cloud data; the projection area and the total fluctuation degree of the broken line of the three-dimensional point cloud data on the xz projection plane and the yz projection plane are obtained, and the projection area and the total fluctuation degree of the broken line are weightedly summed, and the sum value is the focus value corresponding to the current pixel point;

[0056] The maximum value of the focus values ​​corresponding to all pixels at the same position is taken as the depth value at each position to obtain the depth map of the object to be reconstructed.

[0057] That is to say, the present invention first performs data transformation on the window matrix in the grayscale image into a point cloud format, then projects it in all directions, calculates the projection area on different projection surfaces and the degree of fluctuation of each row of projection lines, and finally performs weighted summation as the focus value to ultimately determine the three-dimensional surface morphology of the object.

[0058] It should be noted that the image window matrix is ​​two-dimensional data, which only contains grayscale value information. Transforming it into three-dimensional space can contain more information, including the degree of fluctuation in space, the spatial curvature of each spatial point, and can also be projected in different directions to form new two-dimensional data. Therefore, the grayscale value G of the pixel at the (x, y) position (x,y) As the height in three-dimensional space, the position in three-dimensional space after transformation is (x, y, G (x,y) ); Specifically, the pixel area is projected into the three-dimensional space to obtain the three-dimensional point cloud data q:

[0059]

[0060] Among them, N is the size of the setting window, G (1,1) ~G (N,N) is the pixel value of each pixel in the pixel area.

[0061] For example, assuming the window size is 3×3, we get a 3×3 matrix G (x,y) , the matrix G (x,y) After deformation, the position coordinates of each position in three-dimensional space are obtained. The position in three-dimensional space after transformation is as follows Figure 2 As shown, it is represented by a blue five-pointed star.

[0062] Before transforming the data dimension, the original two-dimensional matrix can be regarded as a projection plane, which is the xy projection plane. After the data dimension is transformed, it is projected in the xz and yz directions to obtain two new data.

[0063] In the first aspect, the method for obtaining the projection area of ​​the three-dimensional point cloud data on the xz projection plane is:

[0064] Project the 3D point cloud data onto the xz projection plane, and use Graham scanning to select the point cloud projection point set h{(x1,z1),(x2,z2),…,(x k ,z k )}, where k is the number of point cloud projection points that enclose the maximum convex hull, and k is less than or equal to M, where M is the number of projection points; Figure 2 As shown, the projection points are represented by green points, and the set of projection points is represented by p{(x1,z1),(x2,z2),…,(x M ,z M )}. Figure 2 p1, p2, p3, p6, p9, p8, p7, and p4 in form the maximum convex hull.

[0065] Calculate the convex hull area A xz as follows:

[0066]

[0067] The convex hull area A xz As the projection area of ​​3D point cloud data on the xz projection plane.

[0068] Secondly, the method for obtaining the projection area of ​​the three-dimensional point cloud data on the yz projection plane is:

[0069] Project the 3D point cloud data onto the yz projection plane, and select the point cloud projection point set h{(x1,z1),(x2,z2),…,(x k ,z k )}, where k is the number of point cloud projection points that enclose the maximum convex hull;

[0070] Calculate the convex hull area A yz as follows:

[0071]

[0072] The convex hull area A yz As the projection area of ​​3D point cloud data on the yz projection plane.

[0073] That is to say, the present invention transforms the window matrix of the image into three-dimensional space and then projects it around. The window matrix of the image is used as the xy plane in the three-dimensional space, and the projection around is projection onto the xz and yz planes. Assuming that the window size is N×N, there are N×N points on the xz and yz projection planes. On the xz projection plane, each row includes N points, with a total of N rows. On the yz projection plane, each row includes N points, with a total of N rows. After projecting onto the xz and yz projection planes, the projection area is calculated. First, the convex hull point set is calculated. After finding the convex hull points, the adjacent points are connected clockwise to form a closed area, and the area of ​​the closed area on the projection surface is calculated.

[0074] Furthermore, the present invention aims to determine the fluctuation degree of each row of data on the projection surface by the total fluctuation distance method. Specifically, the method for obtaining the total fluctuation degree of the broken line of the three-dimensional point cloud data on the xz projection surface is:

[0075] Treat the 3D point cloud data with the same x coordinate as one column;

[0076] Sort each column of 3D point cloud data from large to small according to the z-axis coordinate;

[0077] The projection points of the three-dimensional point cloud data with the same ranking in each column of the three-dimensional point cloud data on the xz projection plane are connected into a broken line; the broken line is as follows Figure 2 The red broken line on the xz projection plane;

[0078] The degree of fluctuation of each broken line is obtained by the Euclidean distance between each adjacent point:

[0079]

[0080] Where i = 1, 2, ..., B, B is the total number of broken lines, j = 1, 2, ..., L, L is the number of points on the broken line; the total fluctuation degree of the broken line is obtained as follows:

[0081]

[0082] Among them, d xz It is the total fluctuation degree of the broken line of the 3D point cloud data on the xz projection plane.

[0083] The method for obtaining the total fluctuation degree of the broken line of the 3D point cloud data on the yz projection plane is:

[0084] Treat the 3D point cloud data with the same y coordinate as a column;

[0085] Sort each column of 3D point cloud data from large to small according to the z-axis coordinate;

[0086] Connect the projection points of the three-dimensional point cloud data with the same ranking on the yz projection plane in each column of the three-dimensional point cloud data into a broken line;

[0087] Get the volatility of each broken line separately:

[0088]

[0089] Where i = 1, 2, ..., B, B is the total number of broken lines, j = 1, 2, ..., L, L is the number of points on the broken line; the total fluctuation degree of the broken line is obtained as follows:

[0090]

[0091] Among them, d yz It is the total fluctuation degree of the broken line of the 3D point cloud data on the yz projection plane.

[0092] The method of weighted summing the projection area and the total fluctuation degree of the polyline to obtain the focus value corresponding to the current pixel point is:

[0093] Get the focus value FM corresponding to the current pixel:

[0094] FM=λ1(A xz +A yz )+λ2(d xz +d yz )

[0095] Among them, λ1 and λ2 are set weights, where λ1+λ2=1, usually defined as λ1=0.6, λ2=0.4, A xz , A yzare the projection areas of the 3D point cloud data corresponding to the current pixel on the xz projection plane and the yz projection plane, respectively. xz d yz are the total fluctuation degree of the broken line of the three-dimensional point cloud data corresponding to the current pixel on the xz projection plane and the yz projection plane respectively;

[0096]

[0097] Among them, D(x,y) is the depth value corresponding to the current pixel.

[0098] In summary, the present invention provides a method for restoring focus morphology using spatial transformation projection, and designs a focus evaluation function for calculating the clarity level of each pixel position in an image. The method first transforms the window matrix in the image from a two-dimensional space to a point cloud in a three-dimensional space, and then projects the transformed point cloud, so that the clarity level of the pixel in the center of the window can be calculated; then the size of the image window is determined, assuming it is M×N, and then the window matrix is ​​transformed into a point cloud format, containing a total of M×N discrete points, and then projected to the surrounding planes, and the projection area and the degree of fluctuation of each row on the projection surface are calculated, and the weighted sum is used as the calculated focus value. The method of the present invention changes the projection direction on the basis of transforming the data dimension, and while transforming the data, the amount of available information is increased.

[0099] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for restoring focused morphology by transforming data dimension and projection direction, characterized in that: The following steps are involved: Acquire an image sequence of the object to be reconstructed; The focus value corresponding to each pixel point on each frame image in the image sequence is obtained respectively, wherein the method for obtaining the focus value corresponding to any pixel point is as follows: a pixel area with the current pixel point as the window center is extracted by using a set window size, and the pixel area is projected into a three-dimensional space to obtain three-dimensional point cloud data; the projection area and the total fluctuation degree of the broken line of the three-dimensional point cloud data on the xz projection plane and the yz projection plane are obtained, and the projection area and the total fluctuation degree of the broken line are weightedly summed, and the sum value is the focus value corresponding to the current pixel point; The maximum value of the focus values ​​corresponding to all pixels at the same position is taken as the depth value at each position to obtain the depth map of the object to be reconstructed.

2. A method for restoring a focused shape by transforming data dimension and projection direction according to claim 1, characterized in that: Project the pixel area into three-dimensional space to obtain the three-dimensional point cloud data q: Among them, N is the size of the setting window, G (1,1) ~G (N,N) is the pixel value of each pixel in the pixel area.

3. A method for restoring focused morphology by transforming data dimension and projection direction as claimed in claim 1, characterized in that: The method for obtaining the projection area of ​​3D point cloud data on the xz projection plane is: Project the 3D point cloud data onto the xz projection plane, and select the point cloud projection point set h{(x1,z1),(x2,z2),…,(x k ,z k )}, where k is the number of point cloud projection points that enclose the maximum convex hull; Calculate the convex hull area A xz as follows: The convex hull area A xz As the projection area of ​​3D point cloud data on the xz projection plane.

4. A method for restoring focused morphology by transforming data dimension and projection direction according to claim 1, characterized in that: The method for obtaining the projection area of ​​3D point cloud data on the yz projection plane is: Project the 3D point cloud data onto the yz projection plane, and select the point cloud projection point set h{(x1,z1),(x2,z2),…,(x k ,z k )}, where k is the number of point cloud projection points that enclose the maximum convex hull; Calculate the convex hull area A yz as follows: The convex hull area A yz As the projection area of ​​3D point cloud data on the yz projection plane.

5. The method for restoring a focused shape by transforming data dimension and projection direction according to claim 1, characterized in that: The method for obtaining the total fluctuation degree of the broken line of the three-dimensional point cloud data on the xz projection plane is: Treat the 3D point cloud data with the same x coordinate as one column; Sort each column of 3D point cloud data from large to small according to the z-axis coordinate; Connect the projection points of the three-dimensional point cloud data with the same ranking on the xz projection plane in each column of the three-dimensional point cloud data into a broken line; Get the volatility of each broken line separately: Wherein, i=1, 2, ..., B, B is the total number of broken lines, j=1, 2, ..., L, L is the number of points on the broken line; The total fluctuation degree of the line is obtained as follows: Among them, d xz It is the total fluctuation degree of the broken line of the 3D point cloud data on the xz projection plane.

6. A method for restoring focused morphology by transforming data dimension and projection direction as claimed in claim 1, characterized in that: The method for obtaining the total fluctuation degree of the broken line of the 3D point cloud data on the yz projection plane is: Treat the 3D point cloud data with the same y coordinate as a column; Sort each column of 3D point cloud data from large to small according to the z-axis coordinate; Connect the projection points of the three-dimensional point cloud data with the same ranking on the yz projection plane in each column of the three-dimensional point cloud data into a broken line; Get the volatility of each broken line separately: Wherein, i=1, 2, ..., B, B is the total number of broken lines, j=1, 2, ..., L, L is the number of points on the broken line; The total fluctuation degree of the line is obtained as follows: Among them, dyz is the total fluctuation degree of the broken line of the three-dimensional point cloud data on the yz projection plane.

7. A method for restoring focused morphology by transforming data dimension and projection direction as claimed in claim 1, characterized in that: The method of weighted summing the projection area and the total fluctuation degree of the polyline to obtain the focus value corresponding to the current pixel point is: Get the focus value FM corresponding to the current pixel: FM=λ1(A xz +A yz )+λ2(d xz +d yz ) Among them, λ1 and λ2 are set weights, A xz , A yz are the projection areas of the 3D point cloud data corresponding to the current pixel on the xz projection plane and the yz projection plane, respectively. xz ,d yz They are the total fluctuation degree of the broken line of the three-dimensional point cloud data corresponding to the current pixel point on the xz projection plane and the yz projection plane respectively; Among them, D(x, y) is the depth value corresponding to the current pixel.

Images