A focused topography recovery method that transforms data dimensions and projection directions

Abstract

The application provides a focused shape recovery method for changing data dimension and projection direction, more useful information can be obtained by changing different projection directions on the basis of data dimension change; firstly, a two-dimensional matrix of an image window is changed into a point cloud format in a three-dimensional space, then projection is performed to the surroundings, new different matrix information can be obtained, the area of the projection and the fluctuation degree of each row in the projection are calculated, and weighted summation is taken as a final focal point value, and the final depth map is determined.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional shape reconstruction technology, and particularly relates to a focused shape recovery method that transforms data dimensions and projection direction. Background Technology

[0002] Reconstructing three-dimensional shapes from two-dimensional images is one of the fundamental tasks of computer vision. Shape from focus (SFF) is a passive, single-lens optical technique that reconstructs the three-dimensional shape of an object from a sequence of multifocal images. These image sequences are captured by varying the focal length, image distance, or object distance of the optical system. The image focus level serves as a clue for shape reconstruction, and each point in the reconstructed depth map corresponds one-to-one with the focus position in the image sequence. Compared to active techniques and multi-camera systems, SFF's main advantages are that it eliminates the need for physical interaction between the scene and the capture device, is compact, low-cost, and easier to implement, requiring only a single camera and translation stage for basic operations. Compared to shape from defocus (SFD) techniques, the reconstructed results are more reliable. Therefore, SFF technology has many applications in microscopy, smart manufacturing, and medicine.

[0003] The main workflow of SFF (Spotlight Focusing) technology consists of three steps: image acquisition, focus measurement, and depth reconstruction. In the first step, a multifocal image sequence is captured by moving the object along the optical axis. Then, the focus measurement (FM) operator is used to estimate the sharpness or clarity level of each pixel in the image sequence. The measurement result has the same size as the image sequence and is called the focus volume (FV). Next, the focus measurement value is maximized along the optical axis to obtain the initial depth map from the FV. To obtain a more accurate depth map, the FV needs to be refined or the initial depth map needs to be optimized; this optimization process is also known as SFF post-processing. Although post-processing can improve the accuracy of the reconstructed result, if there is a large amount of erroneous data in the FV or depth map, error propagation can occur, limiting the optimization effect. Therefore, obtaining accurate focus values ​​is the foundation of SFF topography restoration and is of paramount importance, which has remained a hot research topic.

[0004] In SFF (Spatial Fractional Image) techniques, commonly used FM (Factor-Minimum Focusing) operators compute focus metrics in both the spatial and transform domains. In the spatial domain, structural edges of the image are used as focus metrics, and edge information is obtained by calculating the first and second derivatives of the image. While the second derivative can capture richer information, it is also sensitive to noise. The most well-known statistical FM operator is the one that computes the variance within the window; regions containing more texture exhibit larger variances. In the transform domain, wavelet transform and discrete cosine transform are often used to find high-frequency components of the image, as detailed information is often contained within these components. These operators produce good reconstruction results when the object surface has rich texture, but they are easily affected by noise in real-world scenes and perform poorly when identifying weakly textured regions. In recent years, many new FM methods have been proposed. Jang et al. proposed an adaptive window method, first using the GLV (Gross-Lens Rendering) method to compute a full-focus image, and then determining the window size for each location based on the texture information of the full-focus image. This effectively improves the processing speed and makes the results more reliable, but the reliability depends on the full-focus image. Lee et al. proposed a spatial transformation-based FM method, transforming the image from Cartesian space to log-polar coordinate space. In log-polar coordinate transformation, this process is generally considered a form of low-pass filtering. This averaging process effectively improves noise tolerance. When noise is present in the image, this spatial transformation can calculate better focus values. However, this spatial transformation is affected by coordinate poles, and the focus value varies depending on the position of the poles. Hae-Gon Jeon optimized the processing window shape, proposing a ring difference filter that yields more accurate measurement results compared to traditional windows. Some researchers have proposed using multiple FM operators, applying genetic programming or finding the maximum consensus on the calculated FV to obtain the optimal focus position. Using multiple FM operators improves processing accuracy but also increases computation time, with limited accuracy improvement. Regardless of whether adaptive or deformable windows are used, or multiple FM operators are employed, while the accuracy of focus measurement can be improved to some extent, a significant amount of noise still arises in the FV. This manifests in the depth map as numerous erroneous depth estimates at structural edges and abrupt height changes, and slight depth fluctuations in smooth regions. In summary, due to the inherent limitations of the FM operator, it is difficult to obtain accurate focus values ​​using a single FM method for different objects.

[0005] In summary, improvements to the FM operator mainly involve data transformation and the use of more methods to obtain more accurate calculation results. This significantly increases computation time, and most of the improvements use a single data set without expanding or enriching the data. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a focused shape restoration method that transforms data dimensions and projection directions, which can increase the amount of available information and obtain a better three-dimensional shape of the object.

[0007] A method for focused topography restoration by transforming data dimensions and projection direction includes the following steps:

[0008] Obtain the image sequence of the object to be reconstructed;

[0009] The focal value of each pixel in each frame of the image sequence is obtained. The method for obtaining the focal value of any pixel is as follows: the pixel region centered on the current pixel is extracted using a set window size, and the pixel region is projected onto three-dimensional space to obtain three-dimensional point cloud data; the projection area and total fluctuation of the polyline on the xz and yz projection planes of the three-dimensional point cloud data are obtained, and the projection area and total fluctuation of the polyline are weighted and summed. The sum is the focal value of the current pixel.

[0010] The maximum value of the focal point corresponding to all pixels at the same position is used as the depth value at each position to obtain the depth map of the object to be reconstructed.

[0011] Furthermore, the pixel region is projected into three-dimensional space to obtain the three-dimensional point cloud data q, specifically:

[0012]

[0013] Where N is the size of the set window, G (1,1) ~G (N,N) This represents the pixel value of each pixel in the pixel region.

[0014] Furthermore, the method for obtaining the projected area of ​​the 3D point cloud data on the xz projection plane is as follows:

[0015] Project the 3D point cloud data onto the xz projection plane, and then select the set of point cloud projection points h{(x1,z1),(x2,z2),…,(x...z2) that form the largest convex hull on the xz projection plane. k ,z k )}, where k is the number of point cloud projection points that form the maximum convex hull;

[0016] Calculate the area A of the convex hull xz as follows:

[0017]

[0018] Let the convex hull area A xz The projected area of ​​the 3D point cloud data onto the xz projection plane.

[0019] Furthermore, the method for obtaining the projected area of ​​the 3D point cloud data on the yz projection plane is as follows:

[0020] Project the 3D point cloud data onto the yz projection plane, and select the set of point cloud projection points h{(x1,z1),(x2,z2),…,(x...z1),h{(x2,z2 ... k ,z k )}, where k is the number of point cloud projection points that form the maximum convex hull;

[0021] Calculate the area A of the convex hull yz as follows:

[0022]

[0023] Let the convex hull area A yz The projected area of ​​the 3D point cloud data onto the yz projection plane.

[0024] Furthermore, the method for obtaining the total fluctuation of the polyline in the xz projection plane of the 3D point cloud data is as follows:

[0025] Treat 3D point cloud data with the same x-coordinate as a single column;

[0026] Sort each column of 3D point cloud data from largest to smallest according to the z-axis coordinate;

[0027] Connect the projection points of the 3D point cloud data with the same ranking in each column of 3D point cloud data onto the xz projection plane with a broken line.

[0028] Obtain the fluctuation level of each line individually:

[0029]

[0030] Where i = 1, 2, ..., B, B is the total number of lines, j = 1, 2, ..., L, L is the number of points on the lines; the total fluctuation of the lines is obtained as follows:

[0031]

[0032] Where, d xz This represents the total fluctuation of the polygonal line in the xz projection plane of the 3D point cloud data.

[0033] Furthermore, the method for obtaining the total fluctuation of the polyline in the yz projection plane of the 3D point cloud data is as follows:

[0034] Treat 3D point cloud data with the same y-coordinate as a single column;

[0035] Sort each column of 3D point cloud data from largest to smallest according to the z-axis coordinate;

[0036] Connect the projection points of the 3D point cloud data with the same ranking in each column of 3D point cloud data onto the yz projection plane into a broken line.

[0037] Obtain the fluctuation level of each line individually:

[0038]

[0039] Where i = 1, 2, ..., B, B is the total number of lines, j = 1, 2, ..., L, L is the number of points on the lines; the total fluctuation of the lines is obtained as follows:

[0040]

[0041] Where, d yz This represents the total fluctuation of the polygonal line in the yz projection plane of the 3D point cloud data.

[0042] Furthermore, the method for obtaining the focal value corresponding to the current pixel by weighted summation of the projected area and the total fluctuation of the polygonal line is as follows:

[0043] Get the focus value FM corresponding to the current pixel:

[0044] FM=λ1(A xz +A yz )+λ2(d xz +d yz )

[0045] Where λ1 and λ2 are both set weights, A xz A yz d represents the projected area of ​​the 3D point cloud data corresponding to the current pixel on the xz and yz projection planes, respectively. xz d yz These represent the total fluctuation of the polyline data of the current pixel on the xz and yz projection planes, respectively.

[0046]

[0047] Where D(x,y) is the depth value corresponding to the current pixel.

[0048] Beneficial effects:

[0049] This invention provides a method for focused topography restoration by transforming data dimensions and projection directions. Based on the transformation of data dimensions, more useful information can be obtained by changing different projection directions. First, the two-dimensional matrix of the image window is transformed into a point cloud format in three-dimensional space. Then, it is projected in all directions to obtain new and different matrix information. By calculating the area of ​​the projection and the degree of fluctuation of each row in the projection, and weighted summing them as the final focus value, the final depth map is determined. Attached Figure Description

[0050] Figure 1 A flowchart of a focused topography restoration method that transforms data dimensions and projection direction provided by the present invention;

[0051] Figure 2 This is a schematic diagram of data transformation and projection in different directions provided by the present invention. Detailed Implementation

[0052] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0053] This invention proposes a focused topography restoration method that transforms data dimensions and projection directions. First, the dimensions of the image window matrix are transformed from two-dimensional space to a three-dimensional point cloud. Then, projections are performed in different directions, and the projected area and the fluctuation degree of each row of data on the projected surface are calculated in each direction. These are then weighted and summed to obtain the focus value, resulting in the object's depth map. Figure 1 As shown, the method includes the following steps:

[0054] Acquire an image sequence of the object to be reconstructed; for example, use a monocular vision system to acquire a sequence of partially focused images of the object at equal intervals along the z-axis, and convert the images from RGB format to grayscale images;

[0055] The focal value of each pixel in each frame of the image sequence is obtained. The method for obtaining the focal value of any pixel is as follows: the pixel region centered on the current pixel is extracted using a set window size, and the pixel region is projected onto three-dimensional space to obtain three-dimensional point cloud data; the projection area and total fluctuation of the polyline on the xz and yz projection planes of the three-dimensional point cloud data are obtained, and the projection area and total fluctuation of the polyline are weighted and summed. The sum is the focal value of the current pixel.

[0056] The maximum value of the focal point corresponding to all pixels at the same position is used as the depth value at each position to obtain the depth map of the object to be reconstructed.

[0057] In other words, the present invention first transforms the window matrix in the grayscale image into a point cloud format, then projects it in all directions, calculates the projection area on different projection surfaces and the fluctuation of each projection polyline, and finally performs a weighted summation as the focal value to ultimately determine the three-dimensional surface morphology of the object.

[0058] It should be noted that the image window matrix is ​​two-dimensional data, containing only grayscale value information. Transforming it into three-dimensional space allows it to contain much more information, including the degree of spatial fluctuation, the spatial curvature of each point, and the ability to project it in different directions to form new two-dimensional data. Therefore, the grayscale value G of the pixel at position (x,y) can be transformed into... (x,y) As the height in three-dimensional space, the transformed position in three-dimensional space is (x, y, G). (x,y) Specifically, the pixel region is projected onto three-dimensional space to obtain three-dimensional point cloud data q, which is as follows:

[0059]

[0060] Where N is the size of the set window, G (1,1) ~G (N,N) This represents the pixel value of each pixel in the pixel region.

[0061] For example, assuming the window size is 3×3, we get a 3×3 matrix G. (x,y) , matrix G (x,y) The transformation is performed to obtain the position coordinates of each location in three-dimensional space. The transformed position in three-dimensional space is as follows: Figure 2 As shown, it is represented by a blue pentagram.

[0062] Before the data dimension is transformed, the original two-dimensional matrix can be considered as a projection plane, namely the xy projection plane. After the data dimension is transformed, it is projected in the xz and yz directions to obtain two new types of data.

[0063] Firstly, the method for obtaining the projected area of ​​3D point cloud data on the xz projection plane is as follows:

[0064] The 3D point cloud data is projected onto the xz projection plane, and the set of projected points h{(x1,z1),(x2,z2),…,(x2,z2), which form the maximum convex hull on the xz projection plane, is selected by Graham scanning. k ,z k )}, where k is the number of point cloud projection points forming the maximum convex hull, and k is less than or equal to M, where M is the number of projection points; for example Figure 2 As shown, the projection points are represented by green dots, and the set of projection points is represented as p{(x1,z1),(x2,z2),…,(x... ... M ,z M )}. Figure 2 p1, p2, p3, p6, p9, p8, p7, and p4 form the maximum convex hull.

[0065] Calculate the area A of the convex hull xz as follows:

[0066]

[0067] Let the convex hull area A xz The projected area of ​​the 3D point cloud data onto the xz projection plane.

[0068] Secondly, the method for obtaining the projected area of ​​3D point cloud data on the yz projection plane is as follows:

[0069] Project the 3D point cloud data onto the yz projection plane, and select the set of point cloud projection points h{(x1,z1),(x2,z2),…,(x...z1),h{(x2,z2 ... k ,z k )}, where k is the number of point cloud projection points that form the maximum convex hull;

[0070] Calculate the area A of the convex hull yz as follows:

[0071]

[0072] Let the convex hull area A yz The projected area of ​​the 3D point cloud data onto the yz projection plane.

[0073] In other words, this invention transforms the window matrix of an image into three-dimensional space and projects it in all directions. The window matrix of the image serves as the xy plane in three-dimensional space, and the projection in all directions is the projection onto the xz and yz planes. Assuming the window size is N×N, there are N×N points on both the xz and yz projection planes. On the xz projection plane, each row includes N points, for a total of N rows. On the yz projection plane, each row includes N points, for a total of N rows. After projecting onto the xz and yz projection planes, the projected area is calculated. First, the set of convex hull points is calculated. After finding the convex hull points, adjacent points are connected clockwise to form a closed region. The area of ​​the closed region on the projection plane is then calculated.

[0074] Furthermore, this invention aims to determine the degree of fluctuation of each row of data on the projection plane using the total fluctuation distance method. Specifically, the method for obtaining the total fluctuation degree of the polyline of the 3D point cloud data on the xz projection plane is as follows:

[0075] Treat 3D point cloud data with the same x-coordinate as a single column;

[0076] Sort each column of 3D point cloud data from largest to smallest according to the z-axis coordinate;

[0077] Connect the projection points of the 3D point cloud data with the same ranking in each column onto the xz projection plane with a polyline; the polyline is as follows: Figure 2 The red broken line on the xz projection plane;

[0078] The degree of fluctuation of each broken line is obtained by measuring the Euclidean distance between each adjacent point:

[0079]

[0080] Where i = 1, 2, ..., B, B is the total number of lines, j = 1, 2, ..., L, L is the number of points on the lines; the total fluctuation of the lines is obtained as follows:

[0081]

[0082] Where, d xz This represents the total fluctuation of the polygonal line in the xz projection plane of the 3D point cloud data.

[0083] The method for obtaining the total fluctuation of the polyline on the yz projection plane of 3D point cloud data is as follows:

[0084] Treat 3D point cloud data with the same y-coordinate as a single column;

[0085] Sort each column of 3D point cloud data from largest to smallest according to the z-axis coordinate;

[0086] Connect the projection points of the 3D point cloud data with the same ranking in each column of 3D point cloud data onto the yz projection plane into a broken line.

[0087] Obtain the fluctuation level of each line individually:

[0088]

[0089] Where i = 1, 2, ..., B, B is the total number of lines, j = 1, 2, ..., L, L is the number of points on the lines; the total fluctuation of the lines is obtained as follows:

[0090]

[0091] Where, d yz This represents the total fluctuation of the polygonal line in the yz projection plane of the 3D point cloud data.

[0092] The method for obtaining the focal value of the current pixel by weighted summation of the projected area and the total fluctuation of the polygonal line is as follows:

[0093] Get the focus value FM corresponding to the current pixel:

[0094] FM=λ1(A xz +A yz )+λ2(d xz +d yz )

[0095] Where λ1 and λ2 are set weights, and λ1 + λ2 = 1, usually defined as λ1 = 0.6 and λ2 = 0.4, A xz A yzd represents the projected area of ​​the 3D point cloud data corresponding to the current pixel on the xz and yz projection planes, respectively. xz d yz These represent the total fluctuation of the polyline data of the current pixel on the xz and yz projection planes, respectively.

[0096]

[0097] Where D(x,y) is the depth value corresponding to the current pixel.

[0098] In summary, this invention provides a focused topography restoration method using spatial transformation projection. It designs a focused evaluation function to calculate the sharpness level of each pixel position in an image. This method first transforms the window matrix in the image from two-dimensional space to a point cloud in three-dimensional space, and then projects the transformed point cloud to calculate the sharpness level of the central pixel of the window. Next, the size of the image window is determined, assuming it to be M×N, and the window matrix is ​​transformed into a point cloud format containing M×N discrete points. Then, it is projected onto the surrounding planes, and the area of ​​the projection and the degree of fluctuation of each row on the projection surface are calculated, and a weighted sum is used as the calculated focus value. This invention changes the projection direction while transforming the data dimension, increasing the amount of usable information while transforming the data.

[0099] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A focused profile recovery method that transforms data dimensions and projection directions, characterized by, The method comprises the following steps: obtaining an image sequence of an object to be reconstructed; obtaining a focal point value corresponding to each pixel point in each frame image in the image sequence respectively, wherein the focal point value corresponding to any pixel point is obtained by: extracting a pixel region with the current pixel point as the center of the window by using a set window size, projecting the pixel region to a three-dimensional space to obtain three-dimensional point cloud data, obtaining a projection area and a total fluctuation degree of a broken line of the three-dimensional point cloud data on an xz projection plane and a yz projection plane, and performing weighted summation on the projection area and the total fluctuation degree of the broken line, wherein a sum value obtained is the focal point value corresponding to the current pixel point; taking a maximum value of the focal point values corresponding to all pixel points at the same position as a depth value at each position to obtain a depth map of the object to be reconstructed; Projecting the pixel region into a three-dimensional space to obtain three-dimensional point cloud data Specifically: N is the size of the set window, is the pixel value of each pixel point in the pixel region; the method for obtaining the total fluctuation degree of the broken line of the three-dimensional point cloud data on the xz projection plane comprises: regarding three-dimensional point cloud data with the same x coordinate as a column; sorting each column of three-dimensional point cloud data according to the z-axis coordinate from large to small; connecting the projection points of the three-dimensional point cloud data with the same ranking in each column of three-dimensional point cloud data on the xz projection plane into a broken line; obtaining the fluctuation degree of each broken line respectively; wherein, , is the total number of broken lines, , is the number of points on the broken line; the total fluctuation degree of the broken line is obtained as follows: wherein, is the total fluctuation degree of the polyline of the three-dimensional point cloud data on the xz projection plane; the method for obtaining the total fluctuation degree of the broken line of the three-dimensional point cloud data on the yz projection plane comprises: regarding three-dimensional point cloud data with the same y coordinate as a column; sorting each column of three-dimensional point cloud data according to the z-axis coordinate from large to small; connecting the projection points of the three-dimensional point cloud data with the same ranking in each column of three-dimensional point cloud data on the yz projection plane into a broken line; obtaining the fluctuation degree of each broken line respectively; wherein, , is the total number of broken lines, , is the number of points on the broken line; the total fluctuation degree of the broken line is obtained as follows: wherein, is the total fluctuation degree of the polyline of the three-dimensional point cloud data on the yz projection plane.

2. The focused profile recovery method of transforming data dimensionality and projection direction as claimed in claim 1, wherein, the method for obtaining the projection area of the three-dimensional point cloud data on the xz projection plane comprises: Project the 3D point cloud data onto the xz projection plane, and then select the set of point cloud projection points that form the largest convex hull on the xz projection plane. h {( x 1, z 1),( x 2, z 2),…,( x k , z k )},in, k The number of point cloud projection points that form the largest convex hull; Computing a convex hull area As follows: The convex hull area is the projected area of the three-dimensional point cloud data on the xz projection plane.

3. The method of claim 1, wherein the method further comprises: the method for obtaining the projection area of the three-dimensional point cloud data on the yz projection plane comprises: Project the 3D point cloud data onto the yz projection plane, and then select the set of point cloud projection points that form the largest convex hull on the yz projection plane. h {( y 1, z 1),( y 2, z 2),…,( y k , z k )},in, k The number of point cloud projection points that form the largest convex hull; Computing a convex hull area As follows: The convex hull area is the projected area of the three-dimensional point cloud data on the yz projection plane. The method for transforming data dimension and projecting direction according to claim 1, wherein the method for obtaining the focal point value corresponding to the current pixel point by weighting and summing the projection area and the total fluctuation degree of the broken line is: obtaining a focus value corresponding to the current pixel point : wherein, , are set weights, , are projection areas of the three-dimensional point cloud data corresponding to the current pixel point on the xz projection plane and the yz projection plane, respectively, , are total fluctuation degrees of the broken line of the three-dimensional point cloud data corresponding to the current pixel point on the xz projection plane and the yz projection plane, respectively. wherein, is the depth value corresponding to the current pixel point.

Images