Double / multi-projection synchronous structured light three-dimensional reconstruction method and system based on deep learning

Abstract

The invention belongs to the technical field of three-dimensional reconstruction, and particularly relates to a double / multi-projection synchronous structured light three-dimensional reconstruction method and system based on deep learning. The method comprises the following steps: firstly, acquiring an aliasing image, wherein the aliasing image is an image obtained by shooting the surface of an object after double / multiple projectors project an actual structured light image with required frequency to the surface of the object at the same time; then inputting the aliasing image into a recovery model to recover the aliasing image to obtain each predicted structured light image; the recovery model is obtained by training a deep learning model by using an image data set, and the image data set is composed of aliasing images and corresponding actual structured light images; and finally, reconstructing the three-dimensional point cloud of the object by using all the predicted structured light images. According to the method, the effect of aliasing stripe recovery can be achieved by using the deep learning model, a complex image processing and de-aliasing algorithm is not needed, only time synchronization is required for hardware such as a projector and a camera, and other special requirements do not exist.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, specifically relating to a deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method and system. Background Technology

[0002] In recent years, structured light-based 3D reconstruction technology has been widely applied and has made groundbreaking progress. However, existing scenarios mostly rely on a single camera and a single projector, resulting in incomplete information acquisition and low efficiency.

[0003] To address the issues of low measurement speed, limited measurement range, and blind spots in single-camera, single-projector 3D measurement structures, some researchers have proposed single-camera, dual-projector structures to increase the measurement range. For example, Zheng Shiwei's 2018 master's thesis, "Research on High Dynamic Range Imaging Technology and Dual-Projection Measurement Method Based on Digital Micromirrors," proposes a dual-projection measurement system to solve the problems of blind spots and measurement accuracy in traditional measurements. This system is an asynchronous structured light system where the projector and camera operate independently, without strict time synchronization. The projector continuously projects structured light images, and the camera independently captures the deformation of these images on the object's surface. During this process, it is desirable for the two projectors to project in separate regions and at different times to avoid aliasing of striped images. Once aliasing occurs, it results in full aliasing. In this case, it is necessary to separate (or recover) the structured light patterns from different time points in the mixed images. This typically requires complex image processing and dealiasing algorithms or places high demands on the hardware of the projector and camera.

[0004] For example, traditional methods for dealiasing grating fringe separation include the six-image separation method, the dual-projection superimposed grating separation method based on a color camera, and the derivative-based separation method. The six-image separation method obtains the principal phase information of the object under test when projected individually by the left and right projectors by designing the projection order of the gratings. This method has complex separation steps and strict requirements on the projection order of the projectors. The dual-projection superimposed grating separation method based on a color camera requires the projection fringes of the left projector to be red and the projection fringes of the right projector to be blue. After forming a color superimposed grating on the surface of the object under test, it is acquired using a color camera. Then, based on the color characteristics, the obtained color superimposed grating is directly separated, and the principal phases of the left and right projectors are calculated separately for the 3D reconstruction of the object under test. This separation method mainly utilizes the three-channel principle of the color camera to separate the superimposed gratings, but this method has high requirements for camera configuration and the sinusoidal nature of the camera's three channels. The grating separation method based on differentiation requires obtaining effective phase information from the superimposed gratings. Then, the phase-height conversion model can be used to complete the three-dimensional measurement. This separation method has strict requirements on the placement of the projector and the projection direction of the grating. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning-based method and system for 3D reconstruction using dual / multi-projection synchronous structured light, in order to solve the technical problem of requiring complex image processing and dealiasing algorithms or high requirements on hardware such as projectors and cameras when aliasing occurs in asynchronous structured light systems.

[0006] To address the aforementioned technical problems, this invention provides a deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method, comprising the following steps: 1) acquiring aliased images, wherein the aliased images are images obtained by simultaneously projecting actual structured light images of the required frequency onto the object surface using dual / multi-projectors; 2) inputting the aliased images into a recovery model to recover the aliased images, thereby obtaining each predicted structured light image; wherein the recovery model is obtained by training a deep learning model using an image dataset, and the image dataset consists of aliased images and corresponding actual structured light images; 3) reconstructing the 3D point cloud of the object using the predicted structured light images obtained in step 2).

[0007] Furthermore, the deep learning model is an improved conditional diffusion model, which is obtained by improving the traditional conditional diffusion model. The improvement of the traditional conditional diffusion model includes using an improved U-Net to predict noise. The improved U-Net is obtained by improving the traditional U-Net. The improvement of the traditional U-Net includes connecting the deepest layer of the encoder to the shallowest layer of the decoder through a residual module and a self-attention module connected in sequence.

[0008] Furthermore, the deep learning model is a traditional conditional diffusion model, in which traditional U-Net is used to predict noise.

[0009] Furthermore, the training effect of a deep learning model can be measured in any of the following ways:

[0010] The root mean square error between the predicted structured light image output by the deep learning model and the actual projected structured light image is calculated to evaluate the degree of difference between the output predicted structured light image and the actual projected structured light image, and the training effect of the deep learning model is judged based on the degree of difference.

[0011] The Frechet Inception distance between the predicted structured light image output by the deep learning model and the actual projected structured light image is calculated to evaluate the quality of the output image of the deep learning model, and the training effect of the deep learning model is judged based on the quality evaluation result.

[0012] Furthermore, the learning rate changes during the training of a deep learning model. Specifically, all iteration stages are divided into segments, with the learning rate of the later segment being smaller than that of the earlier segment.

[0013] Furthermore, the aliased image input to the restoration model is a preprocessed aliased image, and the preprocessing methods include denoising and filtering.

[0014] Furthermore, the projected structured light image is a stripe pattern.

[0015] Furthermore, the process of reconstructing the 3D point cloud of an object using each predicted structured light image includes: calculating the wrapping phase and unwrapping based on each predicted structured light image, and reconstructing the 3D point cloud of the object in combination with system calibration data.

[0016] To address the aforementioned technical problems, this invention provides a deep learning-based dual / multi-projection synchronous structured light 3D reconstruction system, comprising a computing device, at least two cameras, and at least two projectors. The computing device includes a processor, which executes a computer program to implement the following steps: 1) acquiring aliased images captured by the cameras, wherein the aliased images are images obtained by simultaneously projecting actual structured light images of the required frequency onto the object surface by all projectors; 2) inputting the aliased images into a reconstruction model to reconstruct the aliased images, obtaining each predicted structured light image; wherein the reconstruction model is obtained by training a deep learning model using an image dataset, and the image dataset consists of aliased images and corresponding actual structured light images; 3) reconstructing the 3D point cloud of the object using the predicted structured light images obtained in step 2).

[0017] Furthermore, the deep learning model is an improved conditional diffusion model, which is obtained by improving the traditional conditional diffusion model. The improvement of the traditional conditional diffusion model includes using an improved U-Net to predict noise. The improved U-Net is obtained by improving the traditional U-Net. The improvement of the traditional U-Net includes connecting the deepest layer of the encoder to the shallowest layer of the decoder through a residual module and a self-attention module connected in sequence.

[0018] This invention is a pioneering invention with the following advantages: It innovatively proposes a synchronous structured light 3D reconstruction system, which includes at least two cameras and at least two projectors. Unlike the asynchronous structured light 3D reconstruction systems in the prior art, in the synchronous system, all projectors simultaneously project structured light images onto the object surface, ensuring omnidirectional and blind-spot-free projection and minimizing blind spots. While aliasing inevitably occurs in the images captured by the cameras, this is a localized aliasing phenomenon. Since this aliasing is caused by simultaneous projection from all projectors, complex algorithms are not needed to handle different time points; the focus is on how to recover the structured light images at the same time point. Therefore, a simple deep learning model can solve this problem. Accordingly, the measurement method formed by the present invention is to acquire aliased images and input them into the recovery model obtained by training a deep learning model to obtain each predicted structured light image. Using each predicted structured light image, a simple and accurate 3D point cloud reconstruction of an object can be achieved without the need for complex image processing and dealiasing algorithms. It only requires the projector, camera and other hardware to work synchronously, without any other special requirements, thereby solving the problems existing in the prior art. Attached Figure Description

[0019] Figure 1 This is a three-dimensional measurement optical path diagram of the dual-projector single-camera system in the system embodiment of the present invention;

[0020] Figure 2 This is a flowchart of the method in a system embodiment of the present invention;

[0021] Figure 3 This is a diagram of the deep learning network structure used in the system embodiments of the present invention;

[0022] Figure 4 This is a basic processing flowchart of the conditional diffusion model used in the system embodiments of the present invention;

[0023] Figure 5 This is a schematic diagram of the forward diffusion process of the conditional diffusion model used in the system embodiments of the present invention;

[0024] Figure 6 This is a schematic diagram of the reverse diffusion process of the conditional diffusion model used in the system embodiments of the present invention;

[0025] Figure 7 This is the network diagram of the improved U-Net in the conditional diffusion model used in the system embodiments of the present invention;

[0026] Figure 8 This is a schematic diagram of the aliasing effect in an embodiment of the present invention. Detailed Implementation

[0027] The main concept of this invention lies in employing a synchronous structured light 3D reconstruction method, unlike the asynchronous structured light 3D reconstruction methods in the prior art. In this system, all projectors simultaneously project structured light images onto the object surface. Therefore, to address aliasing, a trained deep learning model can be used to recover the aliased images to obtain the predicted structured light images. Then, using the predicted structured light images of the required frequency, a simple and accurate 3D point cloud reconstruction of the object can be achieved, without the need for complex image processing and dealiasing algorithms, and with lower requirements for projectors, cameras, and other hardware. To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings.

[0028] Example of a deep learning-based dual / multi-projection synchronous structured light 3D reconstruction system:

[0029] The following describes the method of the present invention using a system with dual projection devices as an example. A projection device includes a projector and a camera, with different cameras located in different positions. This method can also be applied to systems with multiple projection devices, and the principle is the same. It should be noted that projectors and cameras are generally paired, i.e., one projector paired with one camera (i.e., one projection device). If the number of cameras is small, it is impossible to capture the information of the object under test from multiple perspectives. If the number of projectors is small, the stripe information cannot cover the object as completely as possible. Once the stripe information does not cover the object surface, the three-dimensional information of the object under test will be missing. Furthermore, the entire system operates synchronously. This synchronization requires two points: first, all projectors need to simultaneously project structured light images onto the object surface. To achieve this synchronization, synchronous triggering software is needed to achieve simultaneous projection; second, after the projector projects, the camera paired with it needs to immediately take a picture. This is achieved through a synchronous trigger line to synchronize the camera and projector in a projection device (this synchronization will actually have a slight time difference, which can be ignored). Of course, there is a situation where, under certain special operating conditions, not all cameras in the system are working to obtain aliased images. Only a small number of cameras can work, but the projectors that are working must be synchronized (i.e., projecting simultaneously), and the working cameras and their corresponding projectors must be synchronized.

[0030] The structure of the entire system is as follows Figure 1As shown, the system includes two projectors, projector 1 and projector 2, and two cameras, camera 1 and camera 2. Furthermore, the system also includes a computing device connected to the cameras to acquire aliased images captured by the cameras. This computing device includes a processor that executes a computer program to implement the steps of the deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method of the present invention. The process of the deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method implemented by this system is as follows: Figure 2 As shown, the details are as follows:

[0031] Step 1: Design the structured light images to be projected by the two projectors based on the phase-shifting method.

[0032] Called as Figure 1 In the dual-projection system shown, projector 1 is the left projector, and projector 2 is the right projector. Furthermore, in this embodiment, the structured light images projected by both projectors are sinusoidal fringe patterns, as shown below. Figure 1 As shown in A and B, a four-step phase-shifting method is used. The specific pattern of the structured light image can be adjusted according to the actual situation.

[0033] The design of the left projector includes: first, generating the phase-shifted fringe pattern to be projected by the left projector, letting the four phase-shifted fringe patterns to be projected by the left projector be M1, M2, M3, and M4, which are sinusoidal fringes with a phase difference of π / 2. Then, the left projector projects in the sequence M1, M2, M3, M4, M1, M2, M3, M4. The design of the right projector includes: first, generating the phase-shifted fringe pattern to be projected by the right projector, letting the four phase-shifted fringe patterns to be projected by the right projector be N1, N2, N3, and N4, which are sinusoidal fringes with a phase difference of π / 2. Then, the right projector projects in the sequence N1, N2, N3, N4, N1, N2, N3, N4. Among them, M1 and N1 are two fringe patterns with equal phase shift, M2 and N2 are two fringe patterns with equal phase shift, M3 and N3 are two fringe patterns with equal phase shift, and M4 and N4 are two fringe patterns with equal phase shift.

[0034] Step two: Write the designed stripe pattern into the left and right projectors.

[0035] Step 3: The left and right projectors simultaneously project onto the surface of the object being measured, creating an overlap. At this point, the camera captures the aliased image formed by the overlapping projection areas (referred to as an aliased stripe pattern in this embodiment). The aliased stripe patterns are as follows: Figure 1 As shown in C and D. Furthermore, by using steps one through three, a large image dataset can be obtained, including a significant number of original projected structured light images and corresponding aliased images. This image dataset is then divided into training and testing sets according to a certain ratio. Specifically:

[0036] Mixed stripe pattern L i (That is, an aliased image) consists of two projected stripes M i and N i (That is, the actual structured light image projected) is aliased together, and its aliasing intensity value depends on M. i and N i Projection intensity values ​​(i = 1, 2, 3, 4). After projection, the camera is triggered to collect the aliased stripe pattern to obtain the aliased stripe patterns L1, L2, L3, and L4, i.e.: L1 = k*M1 + (1-k)N1, L2 = k*M2 + (1-k)N2, L3 = k*M3 + (1-k)N3, L4 = k*M4 + (1-k)N4, where k is used to control the degree of aliasing. Taking two projectors projecting simultaneously as an example, k = 0.5.

[0037] Furthermore, during this process, each image in the constructed image dataset can be denoised and filtered to reduce noise interference and improve the model's prediction accuracy.

[0038] Step four: Build a deep learning model, and use the training set and test set built in step three to train and test the deep learning model to obtain a recovery model with recovery function.

[0039] The root cause of aliasing is that when two projectors simultaneously project two structured light images of striped patterns, the striped patterns will produce image intensity aliasing. The intensity of the aliased image depends on the intensity values ​​of all projected images within the shooting time. Therefore, the intensity recovery of the aliased image is equivalent to the blind source separation problem. Considering these factors, this embodiment uses a deep learning model.

[0040] The diffusion model is a likelihood-based model with a relatively stable training process and learnable forward and reverse processes, exhibiting good scalability. Therefore, this embodiment selects the diffusion model to handle the aliasing problem caused by dual projection. Specifically, a traditional conditional diffusion model can be selected, and the traditional U-Net predicts noise in the conditional diffusion model.

[0041] Furthermore, since the experiment focused more on the network model's recovery of local aliasing fringes under specified conditions, a better approach was to select a conditional diffusion model. This model is as follows: Figure 3 As shown, this is a method that can be used in datasets. A conditional diffusion model R is used for supervised learning to realize the input aliased image L. i Output two restored predicted structured light images M i 'and N i ',Right now:

[0042] M i ',N i '=R(Li ;θ) (1)

[0043] Where θ represents the parameters of the trained conditional diffusion model. For example... Figure 4 As shown, the model mainly consists of two stages: the forward diffusion stage and the backward diffusion stage (also known as the reconstruction stage). Forward diffusion stage: Real images are input into the network, and noise is continuously added to the input real images using a diffusion method. Through multiple iterations, a noise map conforming to a normal distribution is obtained. That is, Gaussian noise is continuously added to the sample data in the forward diffusion stage until the data is completely destroyed. Backward diffusion stage: Real images are sampled from the real dataset. By randomly determining the number of samples, the corresponding noise map is obtained using a diffusion method. That is, the backward diffusion stage randomly samples noise data from a Gaussian distribution and reconstructs the original sample through continuous iteration. This process requires the use of the U-Net network, which can effectively remove noise. When a noisy map is input to the network, the network can obtain the predicted fake image. This will be described in detail below.

[0044] The forward diffusion process can be represented as a series of conditional probability distributions that gradually transition from the initial data distribution to the noise distribution over time step t. The conditional probability distribution at each time step t depends only on the state of the previous time step t-1, which conforms to the definition of a Markov chain. Therefore, it can be represented as a Markov chain as follows:

[0045]

[0046] In the formula, t represents the time step. β is used to determine the noise intensity introduced at each diffusion step. t and σ t Let q(x) represent the diffusion coefficient and the standard deviation of the noise at each step, respectively. t |x t-1 ) represents forward diffusion, N represents the normal distribution, I represents the identity matrix, and x t This represents the diffusion state at time t. For conditional diffusion models, such as... Figure 5 As shown, the forward diffusion stage is the training process. After inputting the selected initial data x0 and the corresponding conditions y, a time step t is selected first, and noise z is randomly sampled from the Gaussian distribution; then the diffusion state x at time t is calculated according to formula (2). t This process helps the model simulate the noise distribution that changes over time step t, thereby learning richer data features.

[0047] The back-diffusion stage, also known as the generation stage, requires the network to accurately estimate the noise at each time step t in order to reconstruct the original image. Noise is gradually removed using a network, ultimately generating samples that match the distribution of the training data. To achieve this, the noisy image x at time step t and the current time step are used.t The conditional signal y is input into the improved U-Net network, and the noise estimate is obtained through the prediction of the improved U-Net. Then, the data distribution x of the previous time step by step is backtracked using formula (3). t-1 Through continuous iteration, a high-quality reconstruction result x0 approaching the original image is obtained. This process is demonstrated in... Figure 6 In this context, the backward diffusion process can be described using a conditional probability formula within a Bayesian inference framework:

[0048]

[0049] In the formula, p(x) t |x t-1 ) indicates reverse diffusion, α t =1-β t ,β t This indicates the noise intensity added at each diffusion step. R θ This represents a deep learning network.

[0050] This study employs an improved U-Net structure to predict noise, thereby achieving the desired effect of image restoration and aliasing reduction. The traditional U-Net structure is characterized by a symmetrical encoder and decoder. During the encoder's downsampling process, the spatial dimension of the input image is progressively reduced, while the depth of the feature map gradually increases. This process aims to capture multi-level feature information in the image. Correspondingly, during the decoder's upsampling stage, the spatial dimension of the image is progressively enlarged, while the depth of the feature map gradually decreases. This process is considered a gradual reconstruction and restoration of the original image information because some detailed information is lost during downsampling. Furthermore, this improved U-Net structure... Figure 7 As shown, the specific improvement is to embed residual modules and self-attention modules into the U-Net network structure to improve the training efficiency and feature extraction capability of the model. The specific embedding position is that the deepest layer of the encoder is connected to the shallowest layer of the decoder through the sequentially connected residual modules and self-attention modules.

[0051] During the training of this model, error analysis can be performed from two aspects: First, using the loss function as an indicator to evaluate the system's performance, the mean squared error between the model's output predicted structured light image and the projected structured light image can be calculated. This determines the degree of difference between the current predicted structured light image and the actual projected structured light image, and the magnitude of the difference determines whether the model needs further training and optimization. Second, the Frechet Inception distance score can be calculated to determine the distance between the model's output predicted structured light image and the actual projected structured light image, thus measuring the quality of the model's output image and evaluating the model's training effect. Both of these error analysis methods can be used to evaluate the model's training effect, i.e., the model's accuracy.

[0052] Furthermore, during model training, an automatic learning rate is used for hyperparameter tuning. This involves dynamically changing the learning rate at different training stages to help the model reach its optimal state more quickly. Specifically, the learning rate is adjusted by dividing each iteration stage into segments, with the learning rate of each subsequent segment being smaller than that of the previous one. For example, the learning rate is set to 0.9 for iterations 1-200, 0.09 for iterations 201-400, and 0.009 for iterations 401-600, and so on.

[0053] Step 5: Each projector projects the actual structured light image of the required frequency onto the object surface, and two cameras respectively capture two aliased images of the object surface; the obtained aliased images are preprocessed, specifically denoising and filtering to reduce noise interference; one preprocessed aliased image is input into the restoration model, and two predicted structured light images are output to restore the intensity information of the aliased stripes, aiming to obtain a structured light image consistent with the original projected stripes. The same operation is performed on the other preprocessed aliased image.

[0054] Step 6: Calculate the wrapping phase and unwrap the image based on the recovery fringe pattern (i.e., the predicted structured light image) output by the recovery model.

[0055] Among them, the wrapped phase is the phase value containing the three-dimensional information of the measured object obtained by using phase shift profilometry, and the unwrapping is to obtain the absolute phase that is continuous and unique throughout the field, and the phase is unfolded using the multi-frequency heterodyne principle.

[0056] Step 7: Reconstruct the 3D point cloud of the object under test based on the system calibration data.

[0057] The calibration data of the camera and projector are obtained using Zhang Zhengyou's calibration algorithm, and then combined with the absolute phase to obtain the three-dimensional information of the object. Steps six and seven are existing technologies and will not be described in detail in this embodiment.

[0058] The following experiments demonstrate the effectiveness of the method of this invention. A sinusoidal fringe with a simulated pixel size of 512*512 is used. By changing the initial phase of two fringe images, different fringe projections are simulated. Finally, by changing the blending region of the two fringe images, the aliased fringe effect image captured by the camera is obtained. The dataset can be expanded by changing the fringe period, initial phase, and degree of aliasing. The recovery effect of the aliased fringe image using the method of this invention is as follows: Figure 8 As shown, C and D are the aliased images captured by the left and right structured light systems, respectively; C1 and D1 are the results of the network's reconstruction of the aliased stripes, and C2 and D2 are the original images. It can be seen that the fringe pattern reconstructed by the network is very similar to the original fringe pattern, and the reconstruction effect is superior.

[0059] Example of a deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method:

[0060] The deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method implemented in this embodiment is based on the deep learning-based dual / multi-projection synchronous structured light 3D reconstruction system described in the "Deep Learning-based Dual / Multi-Projection Synchronous Structured Light 3D Reconstruction System Embodiment". The principle and implementation process of this method have been described in detail in the "Deep Learning-based Dual / Multi-Projection Synchronous Structured Light 3D Reconstruction System Embodiment", and will not be repeated in this embodiment.

[0061] In summary, this invention differs from the asynchronous structured light 3D reconstruction systems in existing technologies. It innovatively proposes synchronous structured light 3D reconstruction, where all projectors simultaneously project structured light images onto the object's surface. Furthermore, the camera must be synchronized with its corresponding projector, resulting in denser subsequent reconstruction data, particularly effective for uneven surfaces and reducing blind spots. Moreover, the aliased image of the object's surface captured by the camera in this synchronous method is locally aliased, unlike the fully aliased images captured in asynchronous methods. Therefore, complex algorithms are not needed to handle different time points; only the problem of restoring the structured light image at the same time point needs to be addressed. This can be achieved using a deep learning model, restoring image intensity. This method also has no special requirements for projectors, cameras, or other hardware, making aliasing handling more convenient and cost-effective. It allows for more comprehensive reconstruction of the object's surface contours, enabling faster and wider-ranging 3D measurements.

[0062] Specific implementation methods have been given above, but the present invention is not limited to the described implementation methods. The basic idea of ​​the present invention lies in the above basic scheme. For those skilled in the art, designing various modified models, formulas, and parameters based on the teachings of the present invention does not require creative effort. Changes, modifications, substitutions, and variations made to the implementation methods without departing from the principles and spirit of the present invention still fall within the protection scope of the present invention.

Claims

1. A deep learning-based method for dual / multi-projection synchronous structured light 3D reconstruction, characterized in that, Includes the following steps: 1) Acquire aliased images, which are images obtained by simultaneously projecting actual structured light images of the required frequency onto the object surface using two / multiple projectors; 2) Input the aliased image into the restoration model to restore the aliased image and obtain each predicted structured light image; wherein, the restoration model is obtained by training a deep learning model using an image dataset, and the image dataset consists of aliased images and corresponding actual structured light images; 3) Reconstruct the three-dimensional point cloud of the object using the predicted structured light image obtained in step 2).

2. The deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method according to claim 1, characterized in that, The deep learning model is an improved conditional diffusion model, which is obtained by improving the traditional conditional diffusion model. The improvement of the traditional conditional diffusion model includes using an improved U-Net to predict noise. The improved U-Net is obtained by improving the traditional U-Net. The improvement of the traditional U-Net includes connecting the deepest layer of the encoder to the shallowest layer of the decoder through a residual module and a self-attention module connected in sequence.

3. The deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method according to claim 1, characterized in that, The deep learning model is a traditional conditional diffusion model, which uses traditional U-Net to predict noise.

4. The deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method according to claim 1, characterized in that, The training effect of a deep learning model can be measured in any of the following ways: The root mean square error between the predicted structured light image output by the deep learning model and the actual projected structured light image is calculated to evaluate the degree of difference between the output predicted structured light image and the actual projected structured light image, and the training effect of the deep learning model is judged based on the degree of difference. The Frechet Inception distance between the predicted structured light image output by the deep learning model and the actual projected structured light image is calculated to evaluate the quality of the output image of the deep learning model, and the training effect of the deep learning model is judged based on the quality evaluation result.

5. The deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method according to claim 1, characterized in that, The learning rate changes during the training of a deep learning model. Specifically, it changes by dividing all iteration stages into segments, with the learning rate of the later segment being smaller than that of the earlier segment.

6. The deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method according to claim 1, characterized in that, The aliased image input to the restoration model is a preprocessed aliased image, and the preprocessing methods include denoising and filtering.

7. The deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method according to any one of claims 1 to 6, characterized in that, The projected structured light image is a stripe pattern.

8. The deep learning-based dual / multi-projection synchronous structured light 3D reconstruction method according to any one of claims 1 to 6, characterized in that, The process of reconstructing the 3D point cloud of an object using each predicted structured light image includes: calculating the wrapping phase and unwrapping based on each predicted structured light image, and reconstructing the 3D point cloud of the object in combination with system calibration data.

9. A deep learning-based dual / multi-projection synchronous structured light 3D reconstruction system, comprising a computing device, at least two cameras, and at least two projectors, wherein the computing device includes a processor, characterized in that, The processor executes a computer program to perform the following steps: 1) Acquire the aliased image captured by the camera, and the aliased image is the image obtained by photographing the surface of the object after all projectors simultaneously project the actual structured light image of the required frequency onto the surface of the object. 2) Input the aliased image into the restoration model to restore the aliased image and obtain each predicted structured light image; wherein, the restoration model is obtained by training a deep learning model using an image dataset, and the image dataset consists of aliased images and corresponding actual structured light images; 3) Reconstruct the three-dimensional point cloud of the object using the predicted structured light image obtained in step 2).

10. The deep learning-based dual / multi-projection synchronous structured light 3D reconstruction system according to claim 9, characterized in that, The deep learning model is an improved conditional diffusion model, which is obtained by improving the traditional conditional diffusion model. The improvement of the traditional conditional diffusion model includes using an improved U-Net to predict noise. The improved U-Net is obtained by improving the traditional U-Net. The improvement of the traditional U-Net includes connecting the deepest layer of the encoder to the shallowest layer of the decoder through a residual module and a self-attention module connected in sequence.

Images