Single-frame real-time three-dimensional imaging method based on deep learning

Abstract

The invention discloses a single-frame real-time three-dimensional imaging method based on deep learning. The method comprises the following steps: constructing a lightweight phase recovery network model based on a physical model; building a multi-vision structure light stripe projection system; and constructing a training data set, completing training of the lightweight phase recovery network model, and deploying the lightweight phase recovery network model by using TensorRT. Three wrapped phase images are obtained in real time through parallel reasoning of three monochromatic view angles; unwrapping the wrapped phase into an absolute phase by using a multi-view geometric constraint and three-dimensional phase matching algorithm; and generating a three-dimensional point cloud with color texture information through phase-height mapping and a color image. According to the method, the deep learning technology and a traditional fringe projection physical model are combined, rapid phase acquisition can be completed only through a single frame of image, motion interference is overcome, and online real-time high-precision three-dimensional imaging of a complex dynamic scene is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional imaging, and specifically relates to a single-frame real-time three-dimensional imaging method based on deep learning. Background Art

[0002] Human beings live in a three-dimensional space. Eyes are windows to perceive the world, but they can only make qualitative judgments and cannot perform quantitative analysis. Therefore, three-dimensional imaging and sensing systems have emerged to help people better understand objects and conduct quantitative analysis by reconstructing the three-dimensional shape of objects. Structured Light Profilometry (SLP) has become an important optical three-dimensional imaging and measurement technology. It stores the three-dimensional shape information of the object to be measured in the phase of the stripe pattern and recovers the three-dimensional information by decoding the phase. It has the advantages of non-contact, easy integration, and high precision. It is widely used in industrial inspection, aerospace, biomedicine and other fields.

[0003] With the development of optoelectronic technology, the research on 3D shape measurement has gradually expanded from static and slowly changing scenes to more flexible scenes such as complex and high-speed motion, which has put forward higher requirements for structured light projection 3D imaging technology. How to quickly, accurately and unambiguously obtain 3D shape information in complex and high-speed motion scenes has become an urgent problem to be solved. One of the main limiting factors of SLP measurement speed is the number of patterns required for each 3D reconstruction, and it is necessary to reduce coding redundancy to increase speed.

[0004] In recent years, the rapid development of artificial intelligence and deep learning technology has opened a new door for optical imaging technology. Different from traditional physical model-driven imaging methods, deep learning is a powerful data-driven machine learning technology that achieves multi-level functional learning through deep neural networks and has achieved remarkable success in data-rich applications. With its excellent nonlinear fitting ability, deep learning can solve many complex problems that are difficult to deal with in traditional optical imaging. With the advancement of artificial intelligence theory and technology, the improvement of computing hardware level, and the development of optical information processing technology, deep learning has gradually penetrated into the field of optical 3D measurement and has shown great application potential. Specifically in the field of structured light 3D imaging, deep learning uses single-frame fringe patterns for phase analysis, fringe denoising, phase unwrapping and 3D reconstruction, showing the potential of single-frame measurement. However, so far, most of these existing single-frame deep learning methods are offline or have only roughly demonstrated preliminary feasibility. None of them have shown the ability to achieve real-time, high-precision and robust imaging of samples with complex and reflective surfaces or dynamic scenes, nor have they shown comprehensive practical application potential in optical metrology fields such as industrial defect detection and human-computer interaction. Summary of the invention

[0005] The present invention aims to provide a single-frame real-time three-dimensional imaging method based on deep learning.

[0006] The technical solution to achieve the purpose of the present invention is: A single-frame real-time three-dimensional imaging method based on deep learning, comprising:

[0007] Step 1, constructing a lightweight phase recovery network model based on a physical model;

[0008] Step 2: Build a multi-vision structured light fringe projection system and complete the calibration of the multi-vision structured light fringe projection system;

[0009] Step 3, construct a training data set to complete the training of the lightweight phase recovery network model;

[0010] Step 4: Use TensorRT to deploy the trained lightweight phase recovery network model to complete the engineering migration of the deep learning network;

[0011] Step 5, using a multi-vision structured light fringe projection system to collect grating fringe images from multiple perspectives, and using the deployed lightweight phase recovery network model to predict phase information;

[0012] Step 6, through multi-view geometric constraints and stereo phase matching algorithm, the wrapped phases predicted by different views at the same time are unfolded into absolute phases;

[0013] Step 7, obtain three-dimensional data according to the system calibration parameters, perform data rendering in combination with the RGB information obtained from the color image, obtain three-dimensional data of specific color texture, and realize real-time three-dimensional display.

[0014] Preferably, the lightweight phase recovery network model based on the physical model adopts an encoder-decoder structure, wherein the encoder part includes two convolution blocks, 6 STDC modules and a convolution layer connected in sequence, and the original input image is subjected to two convolution blocks and downsampled by 2 times respectively to extract shallow appearance feature information; then, the 6 STDC modules perform fast downsampling to obtain feature information of different scales; finally, the convolution layer outputs the feature map predicted by the encoder;

[0015] The decoder part uses the attention refinement module to refine the outputs of the last two STDC modules in the encoder, fuses the refined results with the output features of the encoder through the feature fusion module, and upsamples the fused results to obtain the output.

[0016] Preferably, each convolution block includes a convolution layer, a normalization function and a linear rectification unit, wherein the convolution kernel size of the block is 3 and the displacement is 2; each STDC module contains 1 1×1 convolution block and 3 3×3 convolution blocks in sequence; the feature map obtained after each convolution operation is connected using a skip connection structure as the output of the STDC module.

[0017] Preferably, the constructed multi-vision structured light stripe projection system includes a main camera 1, an auxiliary camera 2, an auxiliary camera 4, a color camera 3 and a DLP projector. The main camera 1 maintains a set distance from the projector, and the auxiliary camera 2 is located between the main camera 1 and the projector and close to the projector; the auxiliary camera 4 and the main camera 1 are placed symmetrically relative to the projector, and the color camera 3 and the auxiliary camera 2 are placed symmetrically on both sides of the projector.

[0018] Preferably, the specific method for constructing the training data set required for the phase recovery network using the multi-vision structured light fringe projection system is:

[0019] generating a series of sinusoidal fringe patterns for projection, and projecting the sinusoidal fringe patterns onto a target object through a digital projector;

[0020] The main camera 1, the auxiliary camera 2 and the auxiliary camera 4 are used to collect the fringe image of the target surface after modulation and deformation;

[0021] Use the modulation function to define a binary mask, perform shadow noise detection and filtering on the original stripe image, and remove background noise from the stripe image;

[0022] The wrapped phase is extracted from the fringe image after background noise removal by the least square method;

[0023] The stripe image after background removal is used as the network input, and the numerator and denominator of the wrapped phase inverse tangent function are the network output.

[0024] Preferably, the specific method of predicting phase information using the deployed lightweight phase recovery network model is:

[0025]

[0026] In the formula, represents the predicted wrapping phase corresponding to the image captured by the i-th monochrome camera, , The numerator and denominator of the predicted wrapped phase inverse tangent function corresponding to the image captured by the i-th monochrome camera are shown.

[0027] Preferably, the specific method of unfolding the wrapped phase predicted by different perspectives at the same time into the absolute phase through multi-perspective geometric constraints and stereo phase matching algorithm is:

[0028] For each point in the image captured by the main camera 1, The possible number of fringe levels k, i.e. the absolute phase of possibilities, corresponding to 3D candidate points;

[0029] Establishing Absolute Phase With projector horizontal coordinate The linear mapping relationship is:

[0030]

[0031] in, is the lateral resolution of the projector, represents the total number of fringe cycles;

[0032] According to the preset depth measurement range and linear mapping relationship of the multi-vision structured light fringe projection system, the three-dimensional candidate points outside the depth measurement range are excluded, and the number of three-dimensional candidate points is calculated by Reduced to indivual;

[0033] Will The three-dimensional candidate points are projected onto auxiliary camera 2 to obtain the corresponding two-dimensional candidate points, and the corresponding wrapping phase of each two-dimensional candidate point is calculated. , and perform similarity checks on these package phases, is the phase difference, and the number of two-dimensional candidate points and three-dimensional candidate points is changed from Down to indivual;

[0034] Will The three-dimensional candidate points are projected onto the auxiliary camera 4 using epipolar mapping to obtain Two-dimensional candidate points, Two-dimensional candidate points are used for phase similarity test. For phase difference, the number of 3D candidate points is further reduced to ;

[0035] According to the phase difference obtained from the two phase similarity tests, the sum of the weighted phase differences is calculated:

[0036]

[0037] in is the sum of the phase difference weights, , The phase difference corresponding to the two phase similarity tests , The weight factor of The smallest candidate point is the correct corresponding point, and the unique fringe order is determined. The wrapped phase predicted by deep learning is phase-unwrapped to obtain the absolute phase:

[0038]

[0039] Among them, The absolute phase obtained from the perspective of main camera 1, is the wrapped phase corresponding to the image captured by main camera 1, For the stripe level.

[0040] Preferably, the specific method of phase similarity test is:

[0041] Calculate the midpoint of main camera 1 The wrapping phase With the auxiliary camera 2 Candidate points The wrapping phase The phase difference between

[0042]

[0043] Determine whether the phase difference satisfies , exclude the two-dimensional candidate points that do not satisfy the inequality,

[0044] in Indicates the midpoint of main camera 1 The wrapping phase With the auxiliary camera 2 Candidate points The wrapping phase The phase difference between them.

[0045] Preferably, the three-dimensional points under the perspective of the main camera 1 are mapped to the auxiliary camera 4 by using epipolar mapping to obtain the whole pixel matching points of the main camera 1 in the auxiliary camera 4, and the refined sub-pixel matching points are found by interpolation operation;

[0046] The three-dimensional reconstruction of the object under test in the viewing angle of the main camera 1 is restored according to the system calibration parameters to obtain three-dimensional data;

[0047] The corresponding points of the main camera 1 on the color camera 3 are solved according to the obtained three-dimensional data, and the (R, G, B) color texture information of the corresponding points is obtained to complete the true color display of the three-dimensional data.

[0048] Preferably, the three-dimensional point cloud of the object under test under the viewing angle of the main camera 1 is restored according to the system calibration parameters. , the specific formula is:

[0049]

[0050]

[0051] in, is the calibration coefficient matrix calculated in step 2;

[0052] The corresponding points of the main camera 1 on the color camera 3 are solved according to the obtained three-dimensional data, and the (R, G, B) color texture information of the corresponding points is obtained to complete the true color display of the three-dimensional data.

[0053] Compared with the prior art, the present invention has the following significant advantages: (1) A comprehensive framework for deep learning structured light 3D imaging (DL-SL3D) is designed to overcome the limitations of traditional structured light 3D imaging, which requires multiple shots and consumes a lot of computing resources, and verifies the feasibility of achieving real-time, artifact-free 3D reconstruction from a single structured light projection, making it more suitable for real-time application scenarios. (2) A lightweight phase recovery network (LPR-Net) based on a physical model is proposed, which has low computational complexity and robustness to high surface reflection characteristics, thereby achieving efficient phase retrieval of a single frame. (3) A detailed deep learning model deployment scheme and system implementation are provided, which can instantly display reconstructed high-density 3D point cloud data. Compared with the existing structured light imaging system, this system achieves real-time, high-precision 3D imaging through a single shot, and is completely unaffected by motion.

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the process of the comprehensive framework of single-frame real-time three-dimensional imaging method based on deep learning (DL-SL3D).

[0056] Figure 2 This is the network structure diagram of LPR-Net.

[0057] Figure 3 Flowchart for deep learning model deployment and inference as well as 3D reconstruction.

[0058] Figure 4 This is the dynamic ceramic ball accuracy test result.

[0059] Figure 5 This is the real-time imaging result in a complex dynamic scene.

[0060] Figure 6 Real-time 3D imaging under deep learning system. DETAILED DESCRIPTION

[0061] A single-frame real-time 3D imaging method based on deep learning is designed with a comprehensive framework, such as Figure 1As shown in the figure, an integrated solution of deep learning technology and structured light projection profilometry is provided, which can achieve real-time, high-precision three-dimensional imaging of complex dynamic scenes through a structured light coded stripe pattern. The method includes the following seven steps:

[0062] Step 1: Build a lightweight phase recovery network (LPR-Net) based on the physical model, such as Figure 2 shown.

[0063] The physical model of the traditional phase shift method is integrated into the network framework, and the network input is set as a high-frequency grating fringe image, and the network output is the numerator (a sine mapping) and denominator (a cosine mapping) of the phase inverse tangent function. By comparing and performing inverse tangent operations on each pair of predicted images, the final predicted phase result can be obtained, and the influence of noise items such as the surface reflectivity of the object can be completely offset, effectively improving the prediction accuracy.

[0064] The proposed LPR-Net is an efficient lightweight backbone network. Figure 2 As shown in the figure. The network adopts a typical encoder-decoder structure, introduces a fast downsampling feature map module and a multi-scale feature information fusion module, effectively reduces the computational complexity, improves the model prediction speed, and provides a scalable receptive field and multi-scale spatial information to ensure prediction accuracy. In the encoder, multiple convolution blocks and stacked short-term dense connection modules (STDC modules) are used. First, the original input image is downsampled by 2 times through two convolution blocks (ConvX1, ConvX2) to extract shallow appearance feature information; then, 6 STDC modules perform fast downsampling to obtain feature information of different scales (STDC3.1, STDC4.1, and STDC5.1 modules perform 2 times spatial resolution downsampling respectively, and STDC3.0, STDC4.0, and STDC5.0 modules keep the spatial resolution unchanged); finally, a 1×1 convolution layer with a stride of 1 outputs the feature map of the prediction encoder.

[0065] In the decoder, a U-shaped structure is used to upsample the feature maps output by the encoder. First, the output feature maps of the STDC4.0 and STDC5.0 modules in the encoder are refined using the attention refinement module. Then, the refined results are fused with the output features of the encoder using the feature fusion layer to achieve the fusion of low-level detail features and high-level semantic features, so that the fused features contain more detail information and more semantic information. Then, the fused results are upsampled. The network uses transposed convolution to complete fast upsampling, which can improve the model inference speed. In addition, in order to avoid the checkerboard effect that may appear in the image generated after transposed convolution, a normal convolution layer is added after each transposed convolution operation to alleviate such problems.

[0066] In a further embodiment, each convolution block is composed of a convolution layer plus a batch normalization function and a linear rectifier unit, wherein the convolution kernel size of the block is 3 and the displacement is 2; each STDC module contains 1 1×1 convolution block and 3 3×3 convolution blocks in sequence; in order to enrich the feature information, the STDC module uses a skip connection structure to connect the feature map obtained after each convolution operation as the output of the STDC module. Among them, the original image range that each unit in the first convolution layer can see is 1×1, the convolution kernel size of the second convolution layer is 3×3, and the original image range that each unit can see is 3×3. Each unit of the third convolution layer is composed of the second convolution layer with a range of 3×3, so tracing back to the original input feature map of the STDC module, it can see the original image range of 5×5, and so on. Therefore, the STDC module retains the multi-scale receptive field and multi-scale information of the original input image. In addition, it replaces the parallel use of larger-sized 5×5 and 7×7 filters in a geometric series manner, thereby greatly reducing the computational complexity.

[0067] Step 2, build a multi-vision structured light stripe projection system, reasonably arrange the spatial pose and baseline distance between multiple cameras and projectors, and complete the system calibration. For the hardware configuration of the multi-view structured light (stripe projection) system, it includes multiple cameras (three dedicated high-speed monochrome CMOS cameras and one color CMOS camera) and 1 DLP projector. The pose and baseline distance between them are carefully arranged to facilitate the subsequent high-precision phase reconstruction from the obtained image. Before starting any data collection or processing procedures, the system must be calibrated first. Calibration is essentially a simple mapping process from image coordinates to world coordinates to facilitate subsequent three-dimensional reconstruction. However, the use of deep learning will introduce more uncertainty, making the calibration process more complicated, and may not necessarily improve the calibration accuracy. Therefore, this method uses traditional non-deep learning methods to obtain the intrinsic and extrinsic parameters of the system, including each camera and projector.

[0068] The specific process is as follows: the projector projects a stripe pattern and generates a trigger signal, which is transmitted to each camera through a signal line. After receiving the trigger signal, each camera synchronously collects images, and the collected images are transmitted to the computer for processing, and finally three-dimensional data is obtained. In a multi-view system, the spatial position between each view affects its spatial geometric constraint relationship: the closer the view distance, the shorter the depth projection line segment between adjacent views; the farther the view distance, the more accurate the three-dimensional reconstruction. Based on this principle, the position of the camera and the projector in the system is optimized. The three dedicated high-speed monochrome CMOS cameras are the main camera 1, the auxiliary camera 2, the auxiliary camera 4, and the color camera 3: the main camera 1 maintains a set distance from the projector, and the auxiliary camera 2 is located between the main camera 1 and the projector and close to the projector; the auxiliary camera 4 and the main camera 1 are placed symmetrically relative to the projector, and the color camera 3 and the auxiliary camera 2 are placed symmetrically on both sides of the projector. Main camera 1 is the main camera. After completing the spatial position optimization, the plane-based calibration method is used to complete the system calibration, including camera calibration, projector calibration, lens distortion correction and system calibration (camera-camera joint calibration and camera-projector joint calibration). The entire system is calibrated to a unified world coordinate system to obtain the calibration parameters of the four cameras and the projector in the same coordinate system. Subsequently, these parameters are mapped into different lookup table forms to simplify the complex mathematical calculations of the data.

[0069] Step 3: Use the multi-vision structured light fringe projection system to build a high-quality training data set required for the phase recovery network and complete the LPR-Net network model training. First, a series of sinusoidal fringe patterns are generated by the computer for projection and projected onto the target object through a digital projector. The projector projects N phase-shifted patterns per cycle, and the camera captures these images to calculate the labels for network training.

[0070] The three dedicated high-speed monochrome CMOS cameras and one color CMOS camera of the multi-vision structured light fringe projection system are used to capture the modulated and deformed fringe images passing through the surface of the object at multiple angles:

[0071]

[0072] in It is the first captured images, , represents the image pixel coordinates, is the average intensity related to the pattern brightness and background illumination, is the intensity modulation related to pattern contrast and surface reflectivity, is the wrapping phase ( ).

[0073] First, use the modulation function Define a binary mask image to detect and filter the shadow noise of the original stripe image and remove the background noise in the stripe image:

[0074]

[0075]

[0076] represents the mask image. Pixels larger than a certain threshold are used as valid pixels to determine the effective phase information, and finally the 3D surface data is reconstructed only through these valid pixels. It is set in the range of [0,20] and can be automatically adjusted according to the changes and needs of the actual measurement scene.

[0077] Next, wrap the phase It can be extracted from the stripe image after removing the background noise by the least squares method:

[0078]

[0079] In order to obtain high-precision phase distribution, N=12 is generally selected.

[0080] The dataset is collected by the built multi-vision structured light stripe projection system, including: the input of network training is the deformed stripe image captured by the first stripe sequence projection pattern in each projection cycle The output of the training is the numerator of the wrapped phase inverse tangent function and the denominator :

[0081]

[0082] The final dataset contains about 1,000 image pairs, which are randomly divided into training and validation datasets at a ratio of 8:1. Each dataset contains the true value data of the numerator and denominator of the single-frequency fringe image and phase information. It should be noted that the data in the training set, validation set, and test set are not reused to ensure the generalization of the network. The loss function of the network is set to mean square error (MSE), the optimizer is Adam, and the training cycle of the network is set to 500 rounds to complete the LPR-Net network model training.

[0083] Step 4: Use TensorRT to deploy the model and complete the engineering migration of the deep learning network model. The deep learning-based reconstruction algorithms currently proposed all complete predictions through training models under offline data. In order to use deep learning models to achieve real-time prediction of dynamic data, the trained models need to be imported into the system engineering.

[0084] First, the training model needs to be converted into a format that converts the deep learning model trained with Python into a model file that runs in the main programming code C++ environment of the system engineering. Specifically, the model is trained using Pytorch and converted into the ONNX model format, and then the model file is parsed using TensorRT to obtain the engine. After the inference engine is deserialized, it can be used for inference operations on single-frame stripe images. Among them, TensorRT is a C++ deep learning inference library provided by NVIDIA that can run on the GPU hardware platform. It can achieve high throughput, low latency, and low device memory usage. It uses the GPU side to complete the parallel accelerated inference of the model, effectively improving the system's inference computing capabilities for single-frame stripe images. In order to successfully load the pre-trained model of the deep learning network on the system, it is also necessary to pre-configure the attribute table under the system engineering. The specific process is as follows.

[0085] Due to the migration of network data and the acceleration of data calculation, the model import work and data processing code work are distributed on three platforms, namely C++, Python, CUDA and TensorRT. The code work of the Python platform is mainly for network training and network format conversion, network structure and network weight file preservation. The Python platform has a large number of library files available and excellent data set processing functions, so Python is used as a deep learning model training platform. C++ is the main working code and system operation code platform. Its main task is to call the camera to collect single-frame stripe images in real time and collect the collected The image format is converted into a four-dimensional array data type that can be input into the GPU memory and used for deep learning. The network model and weight file are loaded using TensorRT, and the network trained on the Python platform is migrated to the platform, thereby realizing the system's single-frame streak image calculation package phase function. Among them, TensorRT, as a C++ inference framework launched by NVIDIA, parses the trained model into an engine, which can improve the speed of deep learning inference on the NVIDIA GPU.

[0086] Step 5: Use the multi-vision structured light fringe projection system to collect grating fringe images from multiple perspectives, load the trained LPR-Net network model, and quickly predict the phase information. Three monochrome cameras synchronously collect three images in a single exposure. , load the trained network model, and use the network model to predict the phase information of three perspectives :

[0087]

[0088] in denotes the predicted parcel phase, with the superscript Used to distinguish between the three monochrome cameras.

[0089] Step 6: Through multi-view geometric constraints and stereo phase matching algorithm, the wrapped phases calculated from different views at the same time are unfolded into absolute phases.

[0090] For each point in the image captured by the main camera 1, there is The possible number of fringe levels k, i.e. the absolute phase of possibility, corresponding to Each 3D candidate point.

[0091] In absolute phase With projector horizontal coordinate Establish an accurate linear mapping relationship between them:

[0092]

[0093] in, is the lateral resolution of the projector, Indicates the total number of fringe cycles.

[0094] Preset depth measurement range based on multi-vision structured light fringe projection system ( ) and linear mapping relationship, the 3D candidate points outside the depth measurement range are excluded, so the number of 3D candidate points is calculated by Reduced to Secondly, The three-dimensional candidate points are projected onto auxiliary camera 2 to obtain the corresponding two-dimensional candidate points, and the corresponding wrapping phase of each two-dimensional candidate point is calculated. , and perform phase similarity tests on these wrapped phases.

[0095]

[0096]

[0097] in Indicates the midpoint of main camera 1 The wrapping phase With the auxiliary camera 2 Candidate points The wrapping phase The two-dimensional candidate points that do not satisfy the above inequality will be excluded ( ). Therefore, the number of two-dimensional candidate points and three-dimensional candidate points is changed from Down to In order to more accurately determine the final candidate points, another auxiliary camera 4 is introduced. The three-dimensional candidate points are projected onto the auxiliary camera 4 using epipolar mapping to obtain Two-dimensional candidate points, similarly, for this Two-dimensional candidate points are tested for phase similarity ( ), the number of 3D candidate points can be further reduced to For any point in the main camera 1, its corresponding is not necessarily 1, that is, the stripe level cannot be uniquely determined. To determine a unique candidate point among candidate points, this system makes full use of the phase difference obtained by two phase similarity tests and adopts the weighted phase difference sum algorithm:

[0098]

[0099] in is the sum of the phase difference weights, , The phase difference corresponding to the two phase similarity tests , The higher the projection point accuracy, the higher the reliability of its phase difference, and the larger the weight factor is assigned. The smaller candidate point is the correct corresponding point, and the unique fringe order is determined. The wrapped phase predicted by deep learning is phase-unwrapped using the following formula to obtain the absolute phase.

[0100]

[0101] Step 7, obtain three-dimensional data according to the system calibration parameters, perform data rendering in combination with the RGB information of the color image collected by the color camera 3, obtain three-dimensional data of specific color texture, and realize real-time three-dimensional display on the GUI.

[0102] After obtaining the absolute phase, the three-dimensional points in the perspective of the main camera 1 (reconstructed between the projector) are mapped to the auxiliary camera 4 using epipolar mapping to obtain the whole pixel matching points of the main camera 1 in the auxiliary camera 4, and the refined sub-pixel matching points are found through interpolation. Finally, the three-dimensional reconstruction of the object under the perspective of the main camera 1 is restored according to the system calibration parameters.

[0103]

[0104]

[0105] in, is the calibration coefficient matrix pre-calculated by the lookup table in step 4.

[0106] Furthermore, the corresponding points of the main camera 1 on the color camera 3 are solved according to the obtained three-dimensional data, and the (R, G, B) color texture information of the corresponding points is obtained to complete the true color display of the three-dimensional data.

[0107] The overall implementation of the system is written based on CUDA and accelerated by GPU. Compared with traditional CPU core computing, the computing speed is increased by dozens of times. The efficient computing power of GPU significantly improves the real-time performance of the 3D reconstruction process, allowing the network to obtain the 3D reconstruction results of each frame more quickly, thereby meeting the high-performance requirements of dynamic scenes. This acceleration optimization is particularly critical in complex scenes, effectively ensuring the real-time performance and accuracy of the system.

[0108] like Figure 3 As shown, in order to achieve low latency and high throughput performance of deep learning model reasoning on GPU, the present invention uses the TensorRT reasoning engine library developed by NVIDIA to optimize and accelerate the pre-trained deep learning model. (1) Input image format conversion. Before using TensorRT for reasoning on the GPU, the data needs to undergo a certain format conversion to convert the input two-dimensional image into a four-dimensional tensor. (2) TensorRT engine model loading. Convert the training model developed using deep learning frameworks such as PyTorch to an open neural network exchange format to ensure compatibility with TensorRT, and then use TensorRT's C++ high-level application programming interface (API) to parse the ONNX model to generate an engine. (3) Model reasoning. Perform forward reasoning through deserialization and execution engine to output three sets of phase-related items . (4) Dimension transformation of prediction data. (5) Wrapping phase calculation based on deep learning. Wrapping phase is calculated by the inverse tangent function formula (6)Calculate the absolute phase (7) Reconstructing the three-dimensional point cloud. Obtain the three-dimensional shape data of the object to be measured by phase-height mapping.

[0109] The present invention aims to introduce artificial intelligence technology into structured light 3D imaging, build a real-time 3D imaging system based on deep learning, and achieve high-precision and fast 3D shape measurement for complex and moving objects. Compared with traditional multi-frame 3D reconstruction, the present invention compresses 3D reconstruction to the physical limit in the time dimension, and completes the 3D reconstruction of complex objects with only one structured light image. By constructing a multi-vision structured light stripe projection system that integrates a deep learning model and combining it with computer parallel processing technology, real-time high-precision 3D imaging based on a single-frame stripe image is achieved.

[0110] Example:

[0111] In order to verify the effectiveness of the patented method, the inventors have built a real-time multi-view structured light system based on deep learning, which is suitable for three-dimensional imaging of complex dynamic scenes and static scenes. Figure 6 As shown in the figure, it consists of a DLP projector (DLP Lightcraft 4500Pro), four industrial cameras (3 Basler ace acA640-750μm, 1 Baslerace acA640-750μm), a personal computer (PC) and a graphics processing unit (NVIDIA GeForce RTX4090GPU). The DLP projector is configured in Pattern Sequence mode and can project a resolution of 912 1140, projecting 8-bit patterns at 120 fps. Basler cameras with CMOS sensors at 640 751 fps is available at 480 image resolution, and a 12mm Computar lens is used. In this hardware system, the computer is a self-assembled desktop computer equipped with an Intel Core i7-11700K CPU (3.6GHz processor base frequency, 32GB RAM) and an NVIDIAGeForce RTX4090 GPU (24GB memory). The projector is connected to each camera via a signal line and connected via a trigger line so that each camera can simultaneously acquire the deformed stripe image modulated by the surface of the object. In addition, in the system, by porting orders of magnitude of calculations to a GPU that supports the Unified Compute Device Architecture (CUDA), the system is able to have real-time computing capabilities.

[0112] Firstly, a pair of standard ceramic balls swinging up and down are used to perform 3D reconstruction and frame-by-frame dynamic accuracy analysis to verify the 3D dynamic accuracy of the proposed method. Figure 4 (a) is calibrated by a three-dimensional coordinate measuring machine. According to the metrology report, the radii of the ceramic balls are RA = 25.3967mm and RB = 25.3989mm, the center distance between the two balls is D0 = 100.1166mm, and the uncertainty is 2.0μm. Figure 4 (b) is the image captured by the main camera 1 of the system at a certain moment. Figure 4(c) is the 3D reconstruction result of two standard measuring balls at this moment and its corresponding error distribution. The measurement accuracy is distinguished by calculating the difference between the predicted 3D point cloud data and the true value obtained by fitting the standard sphere to the point cloud. The reconstructed sphere radii are 25.9057 mm (sphere A) and 25.5587 mm (sphere B), respectively, and the corresponding root mean square error (RMSE) is 69.6 μm (sphere A) and 61.2 μm (sphere B). The measured center-to-center distance of the sphere is 100.081 mm, and the absolute deviation is 35.5 μm. During the free movement of the ceramic ball, the average dynamic measurement error of the radius of sphere B is 63.1 μm, while the measurement accuracy is still better than 80 μm. Therefore, it is proved that the method proposed in the present invention can achieve a 3D reconstruction result with an accuracy better than 80 μm in complex dynamic scenes.

[0113] Next, two sets of real-time dynamic three-dimensional imaging experiments were carried out using the system constructed by the present invention. Figure 5 , Figure 6 ).

[0114] Figure 5 (a) The measurement results of the single-frame real-time 3D imaging method based on deep learning proposed in this invention for a complex scene consisting of a stationary ceramic ball, a hand, and a moving tablet (without motion artifact errors). Figure 5 (a1), Figure 5 (a2) are Figure 5 (a) Magnified display of the corresponding area. Figure 5 (b) is the result measured by the traditional three-step phase shift method in the same scene (with obvious motion artifact errors). Figure 5 (b1) and (b2) are Figure 5 (b) Magnified display of the corresponding area. Figure 5 (a3), Figure 5 (b3) are Figure 5 (a2), Figure 5 (b2) Depth cross-section curve corresponding to the dashed line.

[0115] Figure 6 (a) shows the real-time structured light 3D imaging system based on deep learning, the Windows-based graphical user interface, and the real-time 3D imaging results with pseudo color. This method is compared with the traditional method (three-step phase shift method) in similar dynamic scenes. Figure 6 (b), Figure 6 (d) is the result of the traditional method, Figure 6 (c), Figure 6 (e) is the overall 3D imaging result of our method at two moments. The comparison results show that both can image static parts (such as Figure 6 (b), Figure 6(c) Static plaster David) for robust 3D imaging. However, for dynamic parts (such as Figure 6 (b)- Figure 6 (e) Rapid hand movements, and Figure 6 (d), Figure 6 (e) Moving plaster David), the traditional method has obvious motion ripples, while this method is based on the advantage of single-frame reconstruction, is not affected by the phase shift error caused by motion, and can still robustly restore the 3D shape. In addition, by using an auxiliary color camera to capture the texture information in the scene, high-precision real-time 3D imaging with texture can also be achieved. Figure 6 (f) shows the clear 3D imaging results of hair, eyes, and hands with texture information in fast-moving scenes, and can also overcome motion artifacts and retrieve 3D point clouds with high accuracy and fidelity in dynamic and complex scenes.

Claims

1. A single-frame real-time three-dimensional imaging method based on deep learning, characterized in that: include: Step 1, constructing a lightweight phase recovery network model based on a physical model; Step 2: Build a multi-vision structured light fringe projection system and complete the calibration of the multi-vision structured light fringe projection system; Step 3, construct a training data set to complete the training of the lightweight phase recovery network model; Step 4: Use TensorRT to deploy the trained lightweight phase recovery network model to complete the engineering migration of the deep learning network; Step 5, using a multi-vision structured light fringe projection system to collect grating fringe images from multiple perspectives, and using the deployed lightweight phase recovery network model to predict phase information; Step 6, through multi-view geometric constraints and stereo phase matching algorithm, the wrapped phases predicted by different views at the same time are unfolded into absolute phases; Step 7, obtain three-dimensional data according to the system calibration parameters, perform data rendering in combination with the RGB information obtained from the color image, obtain three-dimensional data of specific color texture, and realize real-time three-dimensional display.

2. The single-frame real-time three-dimensional imaging method based on deep learning according to claim 1, characterized in that: The lightweight phase recovery network model based on the physical model adopts an encoder-decoder structure. The encoder part includes two convolution blocks, 6 STDC modules and a convolution layer connected in sequence. The original input image passes through two convolution blocks and is downsampled by 2 times respectively to extract shallow appearance feature information. Then, 6 STDC modules perform fast downsampling to obtain feature information of different scales. Finally, the convolutional layer outputs the feature map predicted by the encoder. The decoder part uses the attention refinement module to refine the outputs of the last two STDC modules in the encoder, fuses the refined results with the output features of the encoder through the feature fusion module, and upsamples the fused results to obtain the output.

3. The single-frame real-time three-dimensional imaging method based on deep learning according to claim 2, characterized in that: Each convolution block consists of a convolution layer, a normalization function, and a linear rectifier unit, where the convolution kernel size of the block is 3 and the displacement is 2; each STDC module contains 1 1×1 convolution block and 3 3×3 convolution blocks in sequence; the feature map obtained after each convolution operation is connected using a skip connection structure as the output of the STDC module.

4. The single-frame real-time three-dimensional imaging method based on deep learning according to claim 1, characterized in that: The constructed multi-vision structured light stripe projection system includes a main camera 1, an auxiliary camera 2, an auxiliary camera 4, a color camera 3 and a DLP projector. The main camera 1 maintains a set distance from the projector, and the auxiliary camera 2 is located between the main camera 1 and the projector and close to the projector; the auxiliary camera 4 and the main camera 1 are placed symmetrically relative to the projector, and the color camera 3 and the auxiliary camera 2 are placed symmetrically on both sides of the projector.

5. The single-frame real-time three-dimensional imaging method based on deep learning according to claim 4, characterized in that: The specific method of using the multi-vision structured light fringe projection system to construct the training data set required for the phase recovery network is: generating a series of sinusoidal fringe patterns for projection, and projecting the sinusoidal fringe patterns onto a target object through a digital projector; The main camera 1, the auxiliary camera 2 and the auxiliary camera 4 are used to collect the fringe image of the target surface after modulation and deformation; Use the modulation function to define a binary mask, perform shadow noise detection and filtering on the original stripe image, and remove background noise from the stripe image; The wrapped phase is extracted from the fringe image after background noise removal by the least square method; The stripe image after background removal is used as the network input, and the numerator and denominator of the wrapped phase inverse tangent function are the network output.

6. The single-frame real-time three-dimensional imaging method based on deep learning according to claim 1, characterized in that: The specific method of predicting phase information using the deployed lightweight phase recovery network model is as follows: ， In the formula, represents the predicted wrapping phase corresponding to the image captured by the i-th monochrome camera, , They respectively represent the numerator and denominator of the predicted wrapped phase inverse tangent function corresponding to the image captured by the i-th monochrome camera.

7. The single-frame real-time three-dimensional imaging method based on deep learning according to claim 4, characterized in that: The specific method of unfolding the wrapped phase predicted by different perspectives at the same time into the absolute phase through multi-view geometric constraints and stereo phase matching algorithm is as follows: For each point in the image captured by the main camera 1, The possible number of fringe levels k, i.e. the absolute phase of possibilities, corresponding to 3D candidate points; Establishing Absolute Phase With projector horizontal coordinate The linear mapping relationship is: ， in, is the lateral resolution of the projector, represents the total number of fringe cycles; According to the preset depth measurement range and linear mapping relationship of the multi-vision structured light fringe projection system, the three-dimensional candidate points outside the depth measurement range are excluded, and the number of three-dimensional candidate points is calculated by Reduced to indivual; Will The three-dimensional candidate points are projected onto auxiliary camera 2 to obtain the corresponding two-dimensional candidate points, and the corresponding wrapping phase of each two-dimensional candidate point is calculated. , and perform similarity checks on these package phases, is the phase difference, and the number of two-dimensional candidate points and three-dimensional candidate points is changed from Down to indivual; Will The three-dimensional candidate points are projected onto the auxiliary camera 4 using epipolar mapping to obtain Two-dimensional candidate points, Two-dimensional candidate points are used for phase similarity test. For phase difference, the number of 3D candidate points is further reduced to ; According to the phase difference obtained from the two phase similarity tests, the sum of the weighted phase differences is calculated: ， in is the sum of the phase difference weights, , The phase difference corresponding to the two phase similarity tests , The weight factor of The smallest candidate point is the correct corresponding point, and the unique fringe order is determined. The wrapped phase predicted by deep learning is phase-unwrapped to obtain the absolute phase: ， Among them, The absolute phase obtained from the perspective of main camera 1, is the wrapped phase corresponding to the image captured by main camera 1, For the stripe level.

8. The single-frame real-time three-dimensional imaging method based on deep learning according to claim 7, characterized in that: The specific method of phase similarity test is: Calculate the midpoint of main camera 1 The wrapping phase With the auxiliary camera 2 Candidate points The wrapping phase The phase difference between ， Determine whether the phase difference satisfies , exclude the two-dimensional candidate points that do not satisfy the inequality, in Indicates the midpoint of main camera 1 The wrapping phase With the auxiliary camera 2 Candidate points The wrapping phase The phase difference between them.

9. The single-frame real-time three-dimensional imaging method based on deep learning according to claim 1, characterized in that: Using epipolar mapping, the 3D points in the perspective of the main camera 1 are mapped to the auxiliary camera 4 to obtain the whole pixel matching points of the main camera 1 in the auxiliary camera 4, and the refined sub-pixel matching points are found through interpolation operation; The three-dimensional reconstruction of the object under test in the viewing angle of the main camera 1 is restored according to the system calibration parameters to obtain three-dimensional data; The corresponding points of the main camera 1 on the color camera 3 are solved according to the obtained three-dimensional data, and the (R, G, B) color texture information of the corresponding points is obtained to complete the true color display of the three-dimensional data.

10. The single-frame real-time three-dimensional imaging method based on deep learning according to claim 9, characterized in that: Restore the 3D point cloud of the object under the view of main camera 1 according to the system calibration parameters , the specific formula is: ， ， in, is the calibration coefficient matrix calculated in step 2; The corresponding points of the main camera 1 on the color camera 3 are solved according to the obtained three-dimensional data, and the (R, G, B) color texture information of the corresponding points is obtained to complete the true color display of the three-dimensional data.

Images