Metasurface-regulated dual-mode structured light projection and three-dimensional reconstruction ranging integrated system and method

Abstract

The invention relates to the technical field of three-dimensional reconstruction and ranging, in particular to a metasurface-regulated dual-mode structured light projection and three-dimensional reconstruction ranging integrated system and method, and the system comprises a laser, a beam expander, a metasurface optical element, a binocular camera module, an IMU module, and a computing device. The metasurface generates grazing emergent light and normal emergent light at the same time by regulating and controlling the phase of incident light waves, the structured light projection range is expanded, and key optical characteristics are provided. The binocular camera captures light spot information and calculates depth data through parallax, and vision-inertia fusion is realized in combination with the IMU module; a multi-modal point cloud enhancement algorithm optimizes the three-dimensional reconstruction quality in a complex environment. The system can break through the limitation of the measurement range of a traditional structured light system, achieves the integration of real-time three-dimensional reconstruction and forward distance measurement, improves the robustness and adaptability of the system in a complex environment, achieves the miniaturization and high integration of hardware, and is suitable for the dynamic measurement task of a mobile platform.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction and ranging, and particularly relates to a dual-mode structured light projection and three-dimensional reconstruction ranging integrated system and method regulated by a metasurface. Background Technique

[0002] Due to its high authenticity, three-dimensional scene reconstruction technology has become an important cornerstone in many application fields such as human-computer interaction, virtual reality, and automotive manufacturing. Within the current technical scope, three-dimensional scene reconstruction mainly relies on three major methods: lidar reconstruction, visual reconstruction, and structured light reconstruction. The lidar reconstruction method uses a radar system to scan the surrounding environment and calculates three-dimensional data such as the distance and angle of the target by measuring the time interval (TOF) of the laser pulse from emission to return. However, this method is limited by the rotational scanning mechanism and the requirement for dense pulse emission, resulting in a complex and large system structure, making it difficult for the system to achieve breakthroughs in miniaturization and precision, and facing challenges in the development of compact integrated devices.

[0003] The visual reconstruction method captures multi-view images of the target scene through a camera, extracts depth values using the geometric relationship between the image and the real space, and obtains the three-dimensional information of the entire scene through feature matching and point cloud stitching. The image captured by the camera in a single shot is already a two-dimensional picture. Compared with the one-dimensional pulse of the lidar, the visual reconstruction is easier to obtain a larger field of view (FOV). The larger the field of view, the more three-dimensional information of the target can be included, improving the information capacity of the reconstruction result. However, the camera is sensitive to light, and too strong or too weak light will cause the system to introduce large calculation errors or even be unable to obtain depth information. In addition, for scenes lacking visual features, this pure visual scheme based on image texture information is also difficult to obtain a reliable reconstruction effect.

[0004] In recent years, thanks to the development of diffractive optics and on-chip image processing technologies, important breakthroughs have been made in three-dimensional reconstruction technology based on structured light. This technology effectively reduces the interference of ambient light by integrating dot matrix structured light into visual reconstruction and actively assigns texture features to the target. It can perform single-shot surface modeling without mechanical scanning, making up for the deficiencies of the radar scheme and the pure visual scheme, and greatly improving the efficiency and accuracy of three-dimensional reconstruction. However, traditional optical elements regulate the phase principle of light waves based on Snell's law, resulting in a large device thickness and difficulty in miniaturization. Moreover, under the current technical level, the coverage range of the dot matrix structured light projected by it is still limited (the field of view angle is generally only about 65°), making it difficult to meet the measurement requirements of a wider space and integration for three-dimensional reconstruction systems.

[0005] In addition, traditional diffractive optical elements (DOEs) have limited beam propagation and projection capabilities in large-scale scenarios, resulting in spot distortion and affecting measurement accuracy. Traditional structured light systems rely on the precise combination of multiple components, increasing the complexity and volume of the system and reducing the reliability and portability of the system. In reflective or complex environments, the traditional method has poor robustness, and holes or errors may occur in the point cloud data, affecting the quality and accuracy of 3D reconstruction.

[0006] To this end, the present invention proposes an adaptive structured light system based on metasurface technology, combined with a multimodal point cloud enhanced reconstruction algorithm, which can achieve real-time integration of 3D reconstruction and forward ranging. Through a single metasurface device, light rays with grazing emergence and normal emergence are generated simultaneously, realizing a larger range of structured light projection and breaking through the beam propagation limitation of the traditional system. Through the fine regulation of the optical wave phase by the metasurface, the present invention can achieve a deflection angle close to 90°, effectively improving the adaptability and system performance of the structured light. At the same time, through the multimodal point cloud enhanced reconstruction algorithm, this system can effectively fill the point cloud missing due to optical distortion or environmental factors, improving the robustness and adaptability of the system in complex environments. The present invention integrates a metasurface, a binocular camera and an IMU module to achieve the dual functions of 3D reconstruction and forward ranging, and has high integration, low power consumption and high real-time performance, and is suitable for dynamic measurement tasks on mobile platforms. Summary of the Invention

[0007] Aiming at the deficiencies of existing 3D reconstruction technologies in terms of measurement range, system complexity and environmental adaptability, the present invention proposes a dual-mode structured light projection and 3D reconstruction ranging integrated system and method based on metasurface regulation. The system realizes high-precision, real-time and miniaturized 3D reconstruction and forward ranging functions by introducing a geometric phase metasurface as the core optical element, combining a binocular vision and inertial measurement unit (IMU) fusion algorithm and a multimodal point cloud enhanced reconstruction algorithm.

[0008] The present invention provides a dual-mode structured light projection and 3D reconstruction ranging integrated system and method based on metasurface regulation, and its overall solution includes the following components:

[0009] The system consists of a laser, a beam expander, a metasurface optical element, a binocular camera module, an IMU module, and a computing device. Among them, the laser is used to emit incident light waves, and the beam expander expands the laser beam into a uniform planar light wave to cover the entire effective area of the metasurface. Further, the metasurface optical element realizes the phase double-angle regulation of the incident light wave through the precise arrangement of sub-wavelength scale nano-units, and simultaneously generates grazing outgoing light rays and normal outgoing light rays. The deflection angle of the grazing outgoing light ray is close to 90°, which can significantly expand the projection range of the structured light; the normal outgoing light ray forms a central bright spot on the surface of the target object, providing key optical features for the target.

[0010] Further, the binocular camera module consists of two OV9282 sensors and 150° wide-angle lenses, and is used to capture the spot information of the structured light generated by the metasurface irradiating on the surface of the target object, and obtain depth data through parallax calculation. In particular, the binocular camera module works in cooperation with the IMU module, and the IMU module uses the BMI270 model to collect the attitude information of the system. Through the visual-inertial fusion method, the calibration between the camera and the IMU is completed using the Kalibr tool to improve the robustness and accuracy in the dynamic environment. Among them, the semi-global matching (SGM) algorithm is applied to optimize the matching accuracy of binocular vision to ensure that the system realizes real-time three-dimensional reconstruction and forward ranging at a speed of 30 frames per second.

[0011] In particular, the present invention proposes a multi-modal point cloud enhanced reconstruction algorithm for solving the problem of point cloud missing caused by surface reflectivity change or structured light distortion in complex environments. The algorithm divides the point cloud along the axis and gradually fills the missing area through the minimum distance and direction selection method, thereby optimizing the three-dimensional reconstruction quality. Specifically, S1: According to the initial point cloud data, determine the spatial distribution characteristics of the point cloud missing area; S2: Divide the point cloud along the axis and extract local point cloud segments; S3: Select the optimal filling path through the direction selection method; S4: Gradually fill the missing area to finally generate a complete three-dimensional model.

[0012] Further, the integrated design of the present invention realizes the dual functions of three-dimensional reconstruction and forward ranging by integrating the metasurface, binocular camera, IMU module, and computing device. The system streamlines the hardware configuration, reduces the volume and cost, improves the reliability of the system, and is suitable for dynamic measurement tasks on mobile platforms. Among them, the design of the metasurface contains 1075×1075 nano-units, which has high transmission efficiency and can generate matching structured light patterns in circular and rectangular cross-section tunnel environments to meet the three-dimensional reconstruction requirements in different scenarios.

[0013] The technical effects of the present invention are as follows:

[0014] Through the metasurface light wave phase control technology, the system can achieve grazing emission and normal emission in the same device at the same time, greatly improving the adaptability and system performance of structured light. The grazing emission light can cover a wider range of scenes, while the normal emission light beam provides key optical features for the target, meeting the three-dimensional reconstruction and ranging requirements in different scenes. Furthermore, through the binocular vision and IMU fusion algorithm, the system achieves real-time three-dimensional reconstruction and forward ranging at a speed of 30 frames per second, meeting the measurement requirements in dynamic environments. In particular, the multimodal point cloud enhanced reconstruction algorithm effectively copes with the missing point cloud caused by optical distortion or environmental factors, and improves the robustness and adaptability of the system in complex environments. In addition, through the integrated design, the system achieves miniaturization and high integration of hardware, reduces volume and cost, improves system reliability, and is suitable for compact devices and mobile platforms.

[0015] Furthermore, the key technical points of the present invention include the following:

[0016] First, the metasurface light wave phase control technology. The technology uses the metasurface to finely control the phase of the light wave, breaking through the limitation of the light deflection angle, and simultaneously generating grazing outgoing and normal outgoing light to meet the three-dimensional reconstruction and ranging needs in different scenarios. Second, the multimodal point cloud enhanced reconstruction algorithm. The algorithm divides the point cloud axially and gradually fills in the missing areas by minimizing the distance and direction selection method, thereby optimizing the three-dimensional reconstruction quality. Third, integrated design. The design integrates metasurfaces, binocular cameras, IMU modules and computing devices to achieve the dual functions of three-dimensional reconstruction and forward ranging. It has high integration, low power consumption and high real-time performance, and is suitable for dynamic measurement tasks on mobile platforms.

[0017] In summary, the present invention solves the shortcomings of existing 3D reconstruction technology in terms of measurement range, system complexity and environmental adaptability through innovative technical means such as metasurface control, binocular vision and IMU fusion, and multimodal point cloud enhancement algorithm, and realizes a high-precision, real-time, miniaturized 3D reconstruction and forward ranging integrated system, which has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is the transmission efficiency curve of the metasurface of the present invention;

[0019] Figure 2 It is a schematic diagram of the binocular vision principle of the present invention;

[0020] Figure 3 It is a schematic diagram of the system module of the present invention;

[0021] Figure 4 It is a schematic diagram of the system working process of the present invention;

[0022] Figure 5Schematic diagram of the multi-modal point cloud enhancement reconstruction algorithm of the present invention;

[0023] The reference numerals are as follows:

[0024] P, projection point; X L , image point captured by the left camera; X R , image point captured by the right camera; f, camera focal length; B, baseline distance between the two cameras; z, distance from the projection point to the camera plane. Specific embodiments

[0025] The present invention provides a dual-mode structured light projection and three-dimensional reconstruction ranging integrated system and method based on metasurface regulation, which realizes high-precision, real-time and miniaturized three-dimensional reconstruction and forward ranging functions by combining geometric phase metasurfaces, binocular vision technology, inertial measurement units and multi-modal point cloud enhancement algorithms. The following will be combined with the attached Figure 1 to the attached Figure 5 to describe the specific embodiments of the present invention in detail.

[0026] The overall architecture of this system is as shown in the attached Figure 3 and mainly consists of a laser, a beam expander, a metasurface optical element, a binocular camera module, an IMU module and a computing device. The laser selects a laser source with a wavelength of 632 nm. After the emitted light wave is expanded by the beam expander, the laser energy can cover the entire area of the metasurface throughout the movement of the platform. The metasurface optical element is the core component of this system, and its design adopts a sub-wavelength scale nano-brick unit arrangement scheme. The size and direction of each nano-brick unit are precisely calculated, and the simultaneous generation of grazing outgoing light and normal outgoing light is achieved by modulating the phase double angle of the incident light wave. The attached Figure 1 shows the transmission efficiency curve of the metasurface, and it can be seen that this design has high transmission efficiency, especially reaching the peak transmittance at a wavelength of 632 nm, meeting the actual application requirements.

[0027] The design of the metasurface contains 1075×1075 nano-units, and these units are fabricated by electron beam direct writing technology. The attached Figure 1 shows the transmission efficiency curve of the metasurface, verifying that it can simultaneously generate grazing outgoing light with a deflection angle close to 90° and normal outgoing light in the vertical direction. The grazing outgoing light is used to expand the projection range of the structured light, while the normal outgoing light forms a central bright spot on the surface of the target object, providing key optical features for the target. This design of dual-mode structured light significantly improves the adaptability of the system, enabling it to flexibly switch working modes in different scenarios. For example, in a tunnel environment, the grazing outgoing light can cover the inner wall of a tunnel with a circular or rectangular cross-section, while the normal outgoing light provides clear optical features for the target in the center of the tunnel.

[0028] The binocular camera module consists of two OV9282 sensors and a 150° wide-angle lens. Its function is to capture the spot information of the structured light generated by the metasurface irradiating the surface of the target object and obtain depth data through parallax calculation. Figure 2 The working principle of the binocular camera is shown, where the image points captured by the left camera are marked with X L , the image point captured by the right camera is marked as X R , the baseline distance between the two cameras is B, and the focal length of the camera is f. According to the basic principle of binocular vision, the distance z from the projection point P to the camera plane can be calculated by the formula Calculated. In order to improve the matching accuracy, this system uses the semi-global matching (SGM) algorithm to optimize the binocular vision matching process. The SGM algorithm significantly reduces the probability of mismatching by constructing a cost function and performing path accumulation optimization, thereby improving the accuracy of the depth data.

[0029] The IMU module uses the BMI270 model to collect the system's attitude information. The IMU module works in conjunction with the binocular camera to improve robustness and accuracy in dynamic environments through the visual-inertial fusion method. Figure 3 The system module prompt diagram, the calibration process between the camera and IMU modules are shown, and the calibration is completed using the Kalibr tool to ensure the accuracy of data fusion. The visual-inertial fusion method combines the high-frequency attitude update characteristics of the IMU and the high-precision depth information of the binocular vision, enabling the system to maintain stable performance in fast-moving or vibrating environments. For example, on a drone platform, the IMU module monitors flight attitude changes in real time and transmits the data to the computing device, which adjusts the external parameter matrix of the binocular camera based on the attitude information to correct the parallax error caused by motion.

[0030] For the specific workflow of this system, please refer to Figure 4 , including: the laser is incident on the designed metasurface, which will generate normal and grazing structured light; the light spot is captured by a binocular camera, and the computing device calculates the three-dimensional spatial information of the irradiation position to obtain a three-dimensional point cloud model; the three-dimensional point cloud of the entire space is spliced ​​according to the IMU and adjacent frame information; based on the obtained three-dimensional point cloud, the point cloud model is further optimized using the point cloud enhancement algorithm we proposed, and finally three-dimensional reconstruction is performed.

[0031] The present invention also proposes a multimodal point cloud enhancement reconstruction algorithm to solve the problem of missing point clouds caused by surface reflectivity changes or structured light distortion in complex environments. The algorithm first determines the spatial distribution characteristics of the missing point cloud area based on the initial point cloud data, and then divides the point cloud along the axial direction to extract local point cloud fragments. The optimal filling path is selected through the direction selection method, and the missing area is gradually filled to finally generate a complete three-dimensional model. Figure 5As shown in the figure, the specific implementation steps are as follows: S1 calculates the spatial coordinates and normal vector of each point according to the initial point cloud data, and identifies the boundary points of the missing area; S2 divides the point cloud into multiple layers along the axial direction, and extracts coplanar point cloud fragments to ensure the integrity of local details; S3 performs direction selection analysis on each layer of point cloud, and selects the direction with the smallest normal vector change as the filling path; S4 gradually generates new points along the selected path and adds them to the point cloud until the missing area is completely filled.

[0032] The hardware design of this system focuses on miniaturization and high integration, and all modules are optimized to reduce volume and cost. For example, the size of the metasurface optical element is only 0.03 centimeters (cm), which is significantly smaller than the size of traditional optical elements.

[0033] In actual application scenarios, this system can be widely used in human-computer interaction, virtual reality, automobile manufacturing and other fields. For example, in the field of autonomous driving, this system can obtain three-dimensional information of the vehicle's surrounding environment in real time and perform forward ranging, providing key data for path planning and obstacle detection. In tunnel inspection tasks, this system uses grazing light to cover the inner wall of the tunnel and detects the central target through normal light to generate a complete three-dimensional model of the tunnel. In drone mapping tasks, this system combines visual-inertial fusion methods to overcome the errors caused by rapid movement and generate high-precision three-dimensional terrain models.

[0034] In summary, the present invention solves the shortcomings of existing 3D reconstruction technology in terms of measurement range, system complexity and environmental adaptability through innovative technical means such as metasurface control, binocular vision and IMU fusion, and multimodal point cloud enhancement algorithm. The system has the characteristics of high precision, real-time performance and miniaturization, and can meet the needs of 3D reconstruction and ranging in different scenarios.

[0035] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of the present invention is defined by the attached claims and their equivalents.

[0036] As used in the specification and claims, certain terms are used to refer to specific components. Those skilled in the art should understand that hardware manufacturers may use different terms to refer to the same component. The specification and claims do not use the difference in names as a way to distinguish components, but rather use the difference in the functions of components as the criterion for distinction. As used throughout the specification and claims, the term "comprising" is an open-ended term and should be interpreted as "comprising but not limited to". "Substantially" means within an acceptable error range. Those skilled in the art can solve technical problems within a certain error range and basically achieve the technical effects.

[0037] It should be noted that the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a commodity or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such commodity or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the commodity or system including the element.

[0038] The above description shows and describes several preferred embodiments of the present invention. However, as previously mentioned, it should be understood that the present invention is not limited to the forms disclosed herein, should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be changed within the scope of the inventive concept described herein through the above teachings or the technology or knowledge in the relevant field. And any changes and variations made by those skilled in the art without departing from the spirit and scope of the present invention shall fall within the protection scope of the appended claims of the present invention.

Claims

1. A dual-mode structured light projection and three-dimensional reconstruction ranging integrated system based on metasurface control, characterized in that It includes a laser, a beam expander, a metasurface optical element, a binocular camera module, an IMU module and a computing device, wherein the laser is used to emit an incident light wave, the beam expander expands the laser beam to cover all areas of the metasurface optical element, the metasurface optical element achieves double phase angle control of the incident light wave and generates grazing outgoing light and normal outgoing light through the precise arrangement of sub-wavelength scale nano-units, the binocular camera module is used to capture the spot information of the structured light generated by the metasurface irradiated on the surface of the target object and obtain depth data through parallax calculation, the IMU module is used to collect the posture information of the system and work with the binocular camera module, and the computing device is used to process data and realize three-dimensional reconstruction and forward ranging functions.

2. The system according to claim 1, characterized in that The metasurface optical element contains 1075×1075 nanounits, each of which is prepared by electron beam direct writing technology, and the size of the metasurface optical element is only 0.03 centimeters (cm).

3. The system according to claim 2, characterized in that The nano-units of the metasurface optical element are designed to achieve peak transmittance at a wavelength of 632 nanometers (nm) and simultaneously generate grazing light with a deflection angle close to 90 degrees and normal light in a perpendicular direction.

4. The system according to claim 1, characterized in that The binocular camera module consists of two OV9282 sensors and a 150-degree wide-angle lens. The binocular camera module optimizes the matching accuracy of binocular vision through a semi-global matching algorithm to ensure that the system can achieve real-time 3D reconstruction and forward ranging at a speed of 30 frames per second.

5. The system according to claim 4, characterized in that The binocular camera module and IMU module are calibrated using the Kalibr tool. The IMU module uses the BMI270 model and uses the visual-inertial fusion method to improve robustness and accuracy in dynamic environments.

6. The system of claim 1, wherein The computing device runs a multimodal point cloud enhanced reconstruction algorithm, which divides the point cloud axially and gradually fills in the missing areas to optimize the three-dimensional reconstruction quality by minimizing the distance and direction selection method.

7. The system according to claim 6, characterized in that The multimodal point cloud enhanced reconstruction algorithm includes the following steps: The spatial distribution characteristics of the missing area of ​​the point cloud are determined according to the initial point cloud data, the point cloud is segmented along the axial direction to extract local point cloud fragments, and the optimal filling path is selected through the direction selection method to gradually fill the missing area until a complete three-dimensional model is generated.

Images