Three-dimensional imaging method based on metasurface holographic projection

Abstract

The application discloses a kind of three-dimensional imaging methods based on metasurface holographic projection, belong to micro-nano optics, optical detection and machine vision application technical field.The three-dimensional imaging system based on metasurface holographic projection is established by the present application, since the patterned light beam of holographic projection includes the characteristic pattern of super surface coding design and the laser speckle introduced by laser light source coherence, and the high randomness of speckle makes the details of characteristic pattern more abundant, combined with the characteristics of metasurface holographic reconstruction pattern and triangulation technology, according to the spatial shift and deformation of characteristic pattern, the three-dimensional information of object is recovered, the dense three-dimensional reconstruction point cloud is obtained, the active three-dimensional imaging insensitive to the texture information of measured object is realized, with the advantages of high three-dimensional reconstruction precision;Based on local subarea nonlinear optimization and global continuity constraint, the algorithm calculation amount is small, and can be parallel processing, not only can obtain dense three-dimensional data, but also be suitable for mobile device and other light weight computing platform.

Description

TECHNICAL FIELD

[0001] The present application relates to a three-dimensional imaging method based on metasurface holographic projection, belonging to the technical field of micro-nano optics, optical detection and machine vision application. BACKGROUND

[0002] Three-dimensional information as a key means to describe the real physical world is the focus of today's information acquisition technology, and is widely used in optical detection, autonomous driving, human-computer interaction and machine vision fields. Among them, three-dimensional information acquisition technology is divided into active imaging technology and passive imaging technology, passive imaging technology cannot process weak texture target objects, and is easily affected by the environment, resulting in limited precision, so active imaging technology is more and more widely used in complex real scenes. With the wide application prospect of active imaging technology, in recent years, new hardware structures combined with metasurfaces and other micro-nano devices have gradually shown the advantages of miniaturization and high integration, which provides a very clear development direction for the hardware upgrade of three-dimensional imaging technology.

[0003] The metasurface is a kind of planar optical element with subwavelength size, which has flexible light field control ability. The active point cloud projection of the metasurface is realized by laser light field modulation and propagation of the metasurface, and the essence is holographic reconstruction based on diffraction theory to realize projection. However, there are still many problems to be solved in the combination of metasurface-based point cloud projection and three-dimensional imaging technology. The overall parameter design and precision analysis of the three-dimensional imaging system of the metasurface point cloud projector is the primary work to solve the engineering design of the three-dimensional imaging system, which is used for customized design and standardization of application-oriented objects; in addition, the physical characteristics of the projected patterned light beam and the mathematical algorithm design of three-dimensional reconstruction, the development of a generalizable computing framework based on diffraction optics theory will help the large-scale application of this technology; at the same time, the system parameter storage design of light weight computing platform can be used to solve the practical problems of engineering hardware and realize the upgrade of biometric technology on mobile devices and other platforms. SUMMARY

[0004] The main purpose of the present application is to provide a kind of three-dimensional imaging method based on metasurface holographic projection, since the patterned light beam of holographic projection contains the feature pattern of metasurface encoding design and the laser speckle introduced by laser source coherence, and the high randomness of speckle makes the details of feature pattern more abundant, combined with the characteristics of metasurface holographic reconstruction pattern and triangulation technology, according to the spatial shift and deformation of feature pattern, the three-dimensional information of object is recovered, the dense three-dimensional reconstruction point cloud is obtained, and the texture information of the object to be measured is insensitive to active three-dimensional imaging.The present application has the following advantages: (1) based on the geometric model of triangulation and the accuracy of three-dimensional reconstruction algorithm, the mathematical relationship between the parameters and performance of effective three-dimensional imaging system is established, and the general system configuration technology route is provided; (2) make full use of the spatial characteristics and speckle features of patterned light beam, realize high-precision sub-pixel registration accuracy, and have the advantage of high three-dimensional reconstruction accuracy; (3) based on local sub-area nonlinear optimization and global continuity constraint, the algorithm has small amount of calculation and can be used in parallel, and can be used as a general algorithm across platforms.

[0005] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0006] The three-dimensional imaging method based on metasurface holographic projection disclosed by the present application comprises the following steps:

[0007] Step one: establish a three-dimensional imaging system based on metasurface holographic projection, the three-dimensional imaging system comprises a metasurface and an acquisition module. The metasurface is used for holographic projection to generate an encoded feature patterned light beam. The acquisition module comprises an acquisition lens and a camera, and is used for acquiring the feature pattern reflected by the object.

[0008] The feature patterned light beam generated by the metasurface holographic projection has global uniqueness, and the local feature pattern has very low correlation with the patterns in other areas, which is used to determine the spatial position of the pattern.

[0009] The relationship between the field of view FOV (field of view) of the metasurface holographic projection and the distance L is FOV=L·NA, wherein FOV is the side length of the projection field of view, L is the distance between the projection plane and the metasurface, and NA is the numerical aperture of the metasurface hologram.

[0010] The distance between the center of the acquisition lens and the center of the metasurface in the acquisition module composed of the acquisition lens and the camera is B=L / tanα, wherein α is the included angle between the optical axis of the acquisition lens and the line connecting the center of the acquisition lens and the center of the metasurface.

[0011] The depth resolution Δz of the three-dimensional imaging module satisfies the formula: Δz=B·f / Δp, wherein f is the focal length of the optical lens in the acquisition module, and Δp is the smallest distinguishable pixel size, so B and f are appropriately selected according to the required accuracy of the three-dimensional imaging system.

[0012] Step two: Because the patterned light beam of holographic projection contains the feature pattern of super surface coding design and the laser speckle introduced by the coherence of laser light source, and the high randomness of speckle makes the details of the feature pattern more rich, combined with the characteristics of super surface holographic reconstruction pattern and triangulation technology, the depth information of the object is determined based on the position offset and deformation of the original projection pattern and the image plane after reflection of the object, the spatial position is determined based on the global uniqueness of the local pattern itself, the dense three-dimensional reconstruction point cloud is obtained, and the active three-dimensional imaging insensitive to texture information of the measured object is realized.

[0013] According to the feature pattern design, a feature descriptor composed of vectors is designed, and the corresponding positions of the feature points are determined by the minimum cosine distance; on this basis, based on the local spatial continuity, an optimization target (as shown in formula 1) and a nonlinear optimization algorithm are designed to obtain the spatial position corresponding relationship of the sub-pixel points on the image plane, and more dense three-dimensional point cloud information is obtained. Wherein, f(x,y) and g(x,y) of formula (1) correspond to the shooting image and the reference image respectively, u * (x,y) is the optimal deformation field to be solved, the optimal deformation field needs to satisfy that the reference image is transformed by the sub-region deformation field u i (x,y) in each sub-region Ω i (x,y) and the reference image are the smallest, and satisfy the continuous constraint Fc in the whole space.

[0014]

[0015]

[0016] As preferred, in the lightweight three-dimensional reconstruction algorithm, the original projection pattern is obtained by prior shooting of the reference plane and stored, and only two pictures of storage amount can realize the three-dimensional reconstruction algorithm, the data storage and calculation amount are small, and the lightweight requirement is met.

[0017] Advantages:

[0018] 1. The three-dimensional imaging method based on super surface holographic projection disclosed in the application establishes a three-dimensional imaging system based on super surface holographic projection, on the basis of holographic projection characteristics and triangulation principle analysis, a clear and flexible active three-dimensional imaging system configuration design method is established, including the relationship between the field of view, resolution and system configuration parameters of imaging, and the generalization of the three-dimensional imaging method based on super surface holographic projection is realized.

[0019] 2. The three-dimensional imaging method based on metasurface holographic projection disclosed in the present application uses the patterned local uniqueness feature of holographic projection to design a reconstruction algorithm based on local space continuity and speckle feature matching, which is universal for imaging systems based on metasurface holographic projection and can be used as a basic algorithm process for cross-platform applications.

[0020] 3. The three-dimensional imaging method based on metasurface holographic projection disclosed in the present application only needs to use two pictures, and the data storage amount is small. The algorithm is based on an optimization algorithm for each sub-region and can be processed in parallel, which is a lightweight three-dimensional reconstruction algorithm. Not only can it obtain dense three-dimensional data, but also is suitable for mobile devices and other lightweight computing platforms, and has a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a schematic diagram of the working principle of the three-dimensional imaging method based on metasurface holographic projection provided in the present example;

[0022] Figure 2 is a schematic diagram of the reference image and the deformed image in the three-dimensional imaging method provided in the present example;

[0023] Figure 3 is a photograph of a metasurface holographic projection provided in the present example;

[0024] Figure 4 is a Hamming distance calculation result diagram of the patterned light beam of the metasurface holographic projection provided in the present example;

[0025] Figure 5 is a flowchart of the three-dimensional reconstruction algorithm based on metasurface holographic projection provided in the present example;

[0026] Figure 6 is a feature descriptor schematic diagram in the three-dimensional reconstruction algorithm provided in the present example;

[0027] Figure 7 is a coarse matching schematic diagram based on the feature descriptor in the three-dimensional reconstruction algorithm provided in the present example;

[0028] Figure 8 is a three-dimensional reconstruction point cloud diagram of a step-shaped object based on metasurface holographic projection provided in the present example. DETAILED DESCRIPTION

[0029] In order to more clearly illustrate the purpose, technical scheme and advantages of the present application, the inventive method is further described in detail below in combination with the drawings and examples.

[0030] As Figure 1As shown, the three-dimensional imaging method based on metasurface holographic projection disclosed in the embodiment first includes a projection device composed of a metasurface and an acquisition module composed of an acquisition lens and a camera. The projection device projects a structured light beam with a known code, and after being reflected by the object to be measured, a deformed image is acquired by the acquisition module. According to the principle of triangulation, the image deformation information is caused by the three-dimensional topography of the object to be measured. As shown in the figure, Figure 1 As shown, the same pixel of the acquired image corresponds to points A, B, C, and D at different depths in the real physical space, which correspond to different projection pattern points A', B', C', and D' respectively. By acquiring the image reflected by the standard plane on the reference surface as the reference image, and by algorithm design to find the corresponding points of the reference image and the deformed image, the depth information is obtained in combination with the system calibration data. According to the principle of triangulation, the depth z of the object to be measured relative to the reference plane is B·f / Δx, where Δx is the relative displacement of the measured point in the image, that is, Figure 1 The relative displacement of B', C', and D' to A' in the reference plane image and the deformed image of the projection pattern after being modulated by the three-dimensional object is shown in the figure. Figure 2 As shown in the figure, the deformed object is a cylindrical object, and the deformed pattern changes obviously with the depth of the object, showing the movement of the same projection point in the deformed image relative to the reference image.

[0031] The overall parameter design of the three-dimensional imaging system based on metasurface holographic projection is essentially based on the theory of diffraction optics and the principle of triangulation. The subwavelength size advantage of the metasurface enables it to fill the entire space when the holographic image is reproduced, but limited by the limited field of view of the acquisition module and the quality of the projection pattern, the numerical aperture for realizing high signal-to-noise ratio reproduction image is selected, and in this example, the numerical aperture is selected as 0.6. According to the best working distance of the system, the installation position and installation angle of the acquisition module need to satisfy the relationship: B=L / tanα, so that the center of the field of view of the acquisition module and the projection device coincide, so as to fully utilize the projection field of view for three-dimensional reconstruction. According to the principle of triangulation, the depth resolution Δz satisfies the formula: Δz=B·f / Δp, which is proportional to the installation position of the acquisition module and the focal length of the acquisition lens, so the system parameters need to be designed comprehensively according to the working distance and depth accuracy required by the system.

[0032] Figure 3 For the deformed image actually shot by the three-dimensional imaging system based on metasurface holographic projection, it can be seen that the pattern is deformed after being diffusely reflected by a convex object and is acquired. The spot in the image is a holographic reconstruction image designed by the metasurface code, which has global uniqueness. To verify its global uniqueness, we use Hamming distance as the experimental verification. The Hamming distance is a difference representation method of two local windows, and its calculation formula is:

[0033]

[0034]

[0035] where H is the Hamming distance between two sub-windows centered at (i1, j1) and (i2, j2) with the size of n x n. By traversing the Hamming distance of all sub-windows, the histogram distribution of Hamming distance is obtained as shown in FIG. 2. In this example, we choose the size of the window as 4 x 4, so the maximum Hamming distance is 16. As can be seen from the figure, the Hamming distance of 0 does not exist, indicating that there are no two identical sub-windows, and the proportion of Hamming distance < 4 is less than 5%, indicating that the coded pattern has good recognition characteristics, ensuring the accuracy of the subsequent algorithm reconstruction. Figure 4

[0036] In addition to the global uniqueness of the local window, the coherence of the laser light source introduces a large number of speckles into the super-holographic projection pattern, providing more detailed information for the projection pattern as shown in FIG. 3. Because the speckle of the projection pattern has strong randomness, it is impossible to characterize each speckle by an accurate mathematical description. In view of the characteristics of the projection pattern and the speckle, we provide a reconstruction algorithm based on feature description and local spatial continuity, in which the feature descriptor is designed according to the global unique feature of each pattern feature point, and the feature descriptor is used for coarse matching, and then the local spatial deformation function is constructed by assuming local spatial continuity, to obtain sub-pixel level matching. The algorithm flowchart of the algorithm is shown in FIG. 4, and the specific steps are as follows: Figure 3 Figure 5

[0037] 1) According to the characteristics of the coded pattern, a feature descriptor is constructed, which is composed of the intensity information and information gradient of its neighborhood as shown in FIG. 5. Each bright spot is a feature point of the projection pattern, and its nearest neighbor is found according to the eight equal angle of the circumference, and the distance P n P(n = 1, 2, …, 8) constitutes a vector as the feature descriptor V P of the bright spot, and then the feature descriptor is used to transform the deformed image and the reference image into the feature space respectively; Figure 6

[0038] V P = [P1P; P2P; … P8P]

[0039] 2) The cosine distance is used to find the corresponding feature descriptor of each feature descriptor in the deformed image in the reference image. This process needs to use path constraints to eliminate as many false matching points as possible. The path constraint is that when selecting the feature descriptor to be calculated each time, the descriptor with the highest information density in the neighborhood of the calculated descriptor is selected for calculation until all the descriptors are matched, and the coarse matching based on the feature description is completed. The initial effect of the coarse matching is shown in FIG. 6.​​​​Figure 7 as shown;

[0040] 3) Coarse matching obtains the corresponding information of each bright spot center of gravity, and the deformation parameters of the neighborhood around each bright spot center of gravity are calculated to delete points with large deformation parameter differences, which are mostly false matching points. At the same time, the neighborhood window is gradually contracted to find the best neighborhood range that satisfies local spatial continuity. The deformation parameter refers to the key parameter in the shape function, which is used to represent the spatial coordinate changes of the deformed image and the reference image. In this example, a second-order shape function is used:

[0041]

[0042]

[0043] where (x d ,y d ) and (x,y) are the spatial coordinates of the corresponding points in the deformed image and the reference image, respectively. Δx = x - x0, Δy = y - y0. (x0, y0) is the center coordinate of the local region window. The deformation parameter p is:

[0044] p = (u0, u x ,u y ,u xx ,u xy ,u yy ,v0, v x ,v y ,v xx ,v xy ,v yy ) T

[0045] 4) Within the determined best local region range, the initial value of the deformation parameter p is estimated using the least squares method using the corresponding coordinates of all feature points in the region, and then an optimization function is established with the deformation parameter p as the variable. The nonlinear iterative method is used to determine the deformation parameter p of each local region, thereby completing the accurate sub-pixel matching.

[0046] Based on the above pattern accurate sub-pixel matching, combined with the relationship between depth z and relative displacement Δx, the corresponding depth value of each spatial position is calculated. Figure 8 is a three-dimensional reconstruction point cloud diagram of a stepped object realized by this example. The specific parameters of the example realized are as follows: working distance 300 mm, installation angle of the acquisition module 30°, and height difference of the stepped object 2 mm.

[0047] In summary, the three-dimensional imaging method based on the metasurface holographic projection disclosed in the embodiment provides a complete technical route of the three-dimensional imaging method based on the metasurface holographic projection, solves the blank problems of the technology in the actual development and application, and opens the application market of the technology in the fields of automatic driving, human-computer interaction and machine vision, and promotes the upgrading of the three-dimensional detection technology industry.

[0048] The above specific description further details the purposes, technical solutions and beneficial effects of the application. It should be understood that the above description is only a specific embodiment of the application and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A method for three-dimensional imaging based on metasurface holographic projection, characterized in that: Comprising the following steps, Step one: Establishing a three-dimensional imaging system based on metasurface holographic projection, which includes a metasurface and an acquisition module; the metasurface is used for holographic projection to generate an encoded characteristic patterning light beam; the acquisition module includes an acquisition lens and a camera, which are used to acquire the characteristic pattern reflected by the object; The characteristic patterning light beam generated by the metasurface holographic projection has global uniqueness, and the local characteristic pattern has very low correlation with other areas of the pattern, which is used to determine the spatial position of the pattern; The relationship between the field of view FOV and the distance L of the metasurface holographic projection is: FOV=L·NA, where FOV is the side length of the projection field of view, L is the distance between the projection plane and the metasurface, and NA is the numerical aperture of the metasurface holographic; The acquisition lens and the camera constitute an acquisition module, and the distance between the lens center and the metasurface center is: B=L / tanα, where α is the included angle between the optical axis of the acquisition lens and the line connecting the lens center and the metasurface center; The depth resolution Δz of the three-dimensional imaging module satisfies the formula: Δz=B·f / Δp, where f is the focal length of the optical lens in the acquisition module, and Δp is the smallest distinguishable pixel size, so B and f are appropriately selected according to the required accuracy of the three-dimensional imaging system; Step two: Since the patterning light beam of holographic projection contains the characteristic pattern designed by the metasurface coding and the laser speckle introduced by the coherence of the laser light source, and the high randomness of the speckle makes the details of the characteristic pattern more rich, combined with the characteristics of the metasurface holographic reconstruction pattern and the triangulation technology, the depth information of the object is determined based on the original projection pattern and the position offset and deformation on the image plane obtained after the object is reflected, the spatial position of the local pattern is determined based on its global uniqueness, and a dense three-dimensional reconstruction point cloud is obtained, realizing active three-dimensional imaging which is not sensitive to the texture information of the measured object; According to the characteristic pattern design feature descriptor composed of vectors, and through the minimum cosine distance to determine the corresponding position of the feature points; on this basis, based on the local space continuity, design optimization target and nonlinear optimization algorithm, obtain the spatial position correspondence of sub-pixel points on the image plane, and obtain more dense three-dimensional point cloud information; wherein f(x, y) and g(x, y) of formula (1) correspond to the reference image and the shooting image respectively, u * (x,y) is the optimal deformation field to be solved, the optimal deformation field needs to satisfy that the reference image is transformed after the sub-region deformation field u i (x,y) in each sub-region Ω i The difference with the reference image is the smallest, and it satisfies the continuity constraint Fc in the whole space.

2. The metasurface holography-based three-dimensional imaging method of claim 1, wherein: In the lightweight three-dimensional reconstruction algorithm, the original projection pattern is obtained by pre-shooting the reference plane and stored, and only two pictures of storage capacity can realize the three-dimensional reconstruction algorithm, meeting the requirement of lightweight.

Images