A speckle structured light three-dimensional reconstruction method, electronic equipment and storage medium

Abstract

The application discloses a speckle structured light three-dimensional reconstruction method, an electronic device and a storage medium, and the method comprises the following steps: 1, shooting a scene image and a reference pattern; 2, constructing a teacher network composed of a binocular depth estimation network; 3, constructing a student network composed of a brightness reconstruction branch and a depth reconstruction branch; 4, constructing a training loss function; 5, a self-distillation training process. The application can accurately reconstruct scene depth and brightness information from an image shot by a speckle structured light, thereby realizing high-quality and high-fidelity three-dimensional scene reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computational imaging technology, and in particular to a three-dimensional reconstruction method, electronic device, and storage medium based on speckle structured light. Background Technology

[0002] Three-dimensional information plays a vital role in many applications, such as autonomous driving, robotics, and virtual / augmented reality. A typical structured light system consists of a projector and a camera. Due to its advantages such as high spatial resolution, high measurement accuracy, and simple configuration, structured light systems have been widely deployed on various devices.

[0003] In speckle structured light 3D reconstruction, recovering reliable depth information is a current research hotspot. However, images captured by speckle structured light often contain both depth and brightness information of the scene. Current traditional methods focus on matching the scene image with a reference pattern, thus only recovering depth information. Furthermore, traditional algorithms struggle to reconstruct high-quality depth information.

[0004] Deep learning-based reconstruction algorithms suffer from a lack of analysis of the data characteristics acquired by speckle structured light systems, and require additional scene brightness information as supervision information during training. As a result, they cannot surpass some traditional reconstruction algorithms in terms of depth information reconstruction performance and generalization ability, and they cannot recover brightness information.

[0005] Therefore, existing solutions cannot effectively achieve high-quality scene depth reconstruction while simultaneously completing brightness reconstruction. Summary of the Invention

[0006] The present invention addresses the shortcomings of the prior art by proposing a three-dimensional reconstruction method, electronic device, and storage medium using speckle structured light. The aim is to accurately reconstruct scene depth and brightness information from an image captured by speckle structured light, thereby achieving high-quality, high-fidelity three-dimensional scene reconstruction.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] The three-dimensional reconstruction method of speckle structured light of the present invention is characterized by its application in a structured light system consisting of a projector and a camera, and is carried out according to the following steps:

[0009] Step 1: Project the design pattern onto the scene using the projector, and capture the projected scene image I with the camera; project the design pattern onto a reference plane at a known distance using the projector, and capture the reference pattern P with the camera.

[0010] Step 2: Construct a teacher network consisting of a binocular depth estimation network to recover the depth of the scene image I and obtain the teacher disparity map D. t ;

[0011] Step 3: Construct a student network consisting of a brightness reconstruction branch and a depth reconstruction branch to recover the depth and brightness of the scene image I, and obtain the final depth image D. s ;

[0012] Step 4: Constructing the training loss function;

[0013] Step 4.1: Construction of the teacher network loss function;

[0014] Step 4.1.1: Construct photometric loss using equation (1)

[0015]

[0016] In equation (1), N is the total number of pixels in scene image I, p represents any pixel in scene image I, C represents the Census transform operation of the image block centered at pixel p, and I LCN (p) represents the normalized scene image I. LCN The pixel value of pixel p, P LCN (p) is the normalized reference pattern P. LCN The pixel value of the middle pixel p;

[0017] Step 4.1.2: Construct the edge smoothing loss using equation (2).

[0018]

[0019] In equation (2), and These are the teacher parallax images D. t The gradient along the x and y axes of the image coordinate system; D t It is a teacher parallax diagram;

[0020] Step 4.1.3: Construct the teacher network loss function using equation (3).

[0021]

[0022] In equation (3), λ1 is a hyperparameter;

[0023] Step 4.2: Construction of the student network loss function;

[0024] Step 4.2.1: Constructing the illumination consistency loss function;

[0025] Step a: Construct photometric loss using equation (4)

[0026]

[0027] In equation (1), P C (p) is the deformed pattern P C The pixel value of the middle pixel p;

[0028] Step b: Construct the edge smoothing loss using equation (5)

[0029]

[0030] In equation (2), and These are student parallax images D. s The gradient on the x-axis and y-axis of the image coordinate system;

[0031] Step c: Construct the teacher network loss function using equation (5)

[0032]

[0033] In equation (6), λ2 is a hyperparameter;

[0034] Step 4.2.2 Construction of the luminance loss function:

[0035] Based on the scene's requirements for brightness recovery results, if brightness image I pred If the peak signal-to-noise ratio (PSNR) is less than the threshold ε, then proceed to step A; if the brightness image I... pred If the peak signal-to-noise ratio (PSNR) is greater than the threshold ε, then proceed to step B.

[0036] Step A: Construction of the unsupervised brightness loss function;

[0037] Step A.1: Based on the teacher's parallax diagram D t By mapping the reference pattern P using equation (7), a pseudo-deformation pattern is obtained.

[0038]

[0039] In equation (4), Warp() is the mapping operator;

[0040] Step A.2: Construct the pseudo-luminosity map I of the luminosity reconstruction branch using equation (8). pse :

[0041]

[0042] In equation (5), β is a coefficient that measures the degree of linear combination;

[0043] Step A.3: Construct an unsupervised luminance loss function L using equation (9). I :

[0044]

[0045] In equation (9), λ3 is a hyperparameter, DI() represents the dilation operator; S is the SSIM operator that measures the structural similarity between two images; I pred (p) represents the predicted brightness image I. pred The pixel value of the middle pixel p; I pse (p) represents pseudo-luminance map I pse The pixel value of the middle pixel p;

[0046] Step B: Construct a supervised luminance loss function L using equation (10). I :

[0047]

[0048] In equation (10), I gt (p) represents the true brightness diagram I gt The pixel value of the middle pixel p;

[0049] Step 4.2.3: Construct the distillation loss function L using equation (11). S :

[0050]

[0051] In equation (11), λ4 and λ5 are two hyperparameters that measure the corresponding proportions;

[0052] Step 4.2.4: Construct the student network loss function L using equation (12). total :

[0053]

[0054] In equation (9), α1 and α2 are two hyperparameters that measure the corresponding proportions;

[0055] Step 5: Self-distillation training process;

[0056] Step 5.1, Training for Teacher Networks:

[0057] To minimize the teacher network loss function To achieve this, the teacher network is trained separately using the Adam optimization algorithm, and its network parameters are updated until the illumination consistency loss L is reached. DThe process continues until convergence, thus obtaining the trained teacher network model.

[0058] Step 5.2, Training for student networks:

[0059] To minimize the student network loss function L total With the objective set and the weights of the trained teacher network model fixed, the student network is trained using the Adam optimization algorithm, and its network parameters are updated until the total loss function L is reached. total The process continues until convergence, resulting in a trained student network. This trained student network is then used to reconstruct the depth and brightness of the scene image.

[0060] The three-dimensional reconstruction method of speckle structured light described in this invention is also characterized in that step 2 includes:

[0061] Step 2.1: First, use the Local Contrast Normalization (LCN) operator to perform local contrast normalization on the scene image I and the reference pattern P respectively, to obtain the normalized scene image I. LCN and the normalized reference pattern P LCN ;

[0062] Step 2.2: Convert the normalized scene image I LCN and the normalized reference pattern P LCN The data is fed into the teacher network for processing, and the teacher disparity map D is calculated. t .

[0063] Step 3 includes:

[0064] Step 3.1: Construct a brightness reconstruction branch; the brightness reconstruction branch includes: a feature extractor, M1 convolutional residual blocks, and an optimization module; wherein, the feature extractor consists of N1 convolutional layers, N2 ReLU layers, and N3 normalization layers, each convolutional residual block consists of N4 convolutional layers, N5 ReLU layers, N6 normalization layers, and a residual connection, and the optimization module consists of N7 convolutional layers;

[0065] The scene image I is input into the brightness reconstruction branch. After being processed by the feature extractor, the encoded feature map is calculated. Then, the encoded feature map is input into M1 convolutional residual blocks for processing and calculation to recover a rough deformed pattern.

[0066] The rough deformation pattern is input into the optimization module for refinement to eliminate brightness effects, and the final deformation pattern P is calculated and generated. C ;

[0067] The rough deformed pattern and the scene image I are residually concatenated to calculate a rough brightness image, which is then input into the optimization module for brightness reconstruction, thereby obtaining the predicted brightness image I. pred ;

[0068] Step 3.2: Construct a depth reconstruction branch consisting of a depth estimation network, wherein the depth estimation network is the same as the binocular depth estimation network of the teacher network;

[0069] The reference pattern P and the predicted deformed pattern P C The data is fed into the depth reconstruction branch for processing, resulting in the final depth image D. s .

[0070] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing any of the three-dimensional reconstruction methods, and the processor is configured to execute the program stored in the memory.

[0071] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs any of the steps of the three-dimensional reconstruction method.

[0072] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0073] 1. This invention proposes for the first time a multi-task deep learning network, consisting of a teacher network and a student network, for 3D reconstruction of speckle structured light images captured in a single shot. This network can accurately recover the depth image of the scene and simultaneously recover the brightness image, thereby overcoming the problems of poor accuracy and inability to recover brightness information in current algorithms.

[0074] 2. This invention designs a brightness reconstruction branch, including a feature extractor, a residual convolutional block, and an optimization module. The brightness reconstruction branch can resolve the deformation pattern and brightness image. The deformation pattern can be further used for depth estimation, improving the accuracy of the depth image. Recovering the brightness image through the brightness reconstruction branch solves the problem that current methods cannot recover the brightness image.

[0075] 3. This invention designs a self-distillation strategy. First, the teacher network is pre-trained to obtain the teacher disparity map, which generates a supervision signal for the student network. Then, the student network is trained together, and finally, the student network predicts high-precision student disparity and brightness images. The self-distillation strategy can realize unsupervised training of multi-task networks and better reconstruct depth images.

[0076] 4. This invention can be applied to a variety of different devices and has good effectiveness and generalization on different real data. Attached Figure Description

[0077] Figure 1 This is a schematic diagram of a speckle structured light three-dimensional reconstruction method according to the present invention;

[0078] Figure 1a This is a diagram of the brightness reconstruction branch network framework of the present invention;

[0079] Figure 2 This is a schematic diagram of the speckle structured light imaging system of the present invention;

[0080] Figure 3 This is a visualization of the depth reconstruction results on the synthetic data of this invention;

[0081] Figure 4 This is a visualization of the depth reconstruction results of the present invention in a real imaging system;

[0082] Figure 5 This is a visualization of the brightness reconstruction results on the synthesized data of this invention;

[0083] Figure 6 This is a visualization of the brightness reconstruction results of the present invention in a real imaging system. Detailed Implementation

[0084] In this embodiment, as Figure 1 As shown, a three-dimensional reconstruction method for speckle structured light is presented, and the following steps are followed:

[0085] Step 1: Project the design pattern onto the scene using a projector, and capture the projected scene image I with a camera; Project the design pattern onto a reference plane at a known distance using a projector, and capture the reference pattern P with a camera.

[0086] The monocular structured light system of the present invention, as follows Figure 2 As shown, assuming a virtual image plane displays a reference pattern, this system calculates a disparity map between the captured image and the reference pattern. The depth map is then recovered from the disparity map using triangulation.

[0087] Step 2: Construct a teacher network consisting of a binocular depth estimation network to recover the depth of scene image I;

[0088] This paper models speckle structured light depth estimation as a stereo correspondence search problem and selects PSMNet as the stereo depth estimation network to reconstruct the depth. PSMNet is a supervised learning method for stereo matching. This algorithm utilizes the photometric consistency loss function to achieve unsupervised network training.

[0089] Step 2.1: First, use the Local Contrast Normalization (LCN) operator to perform local contrast normalization on both the scene image I and the reference pattern P to obtain the normalized scene image I. LCN and the normalized reference pattern P LCN ;

[0090] Step 2.2: Convert the normalized scene image I LCN and the normalized reference pattern P LCN The data is fed into the teacher network, first passing through a weight-sharing feature extractor to extract feature maps. Then, a 4D cost volume is constructed by moving along the disparity range and summing the feature maps. The cost volume is then aggregated by several 3D convolutional layers. Finally, the disparity map is calculated using the differential softargmax operator, resulting in the teacher disparity map D. t ;

[0091] Step 3: Construct a student network consisting of a brightness reconstruction branch and a depth reconstruction branch to recover the depth and brightness of scene image I;

[0092] Step 3.1, as follows Figure 2 As shown, a brightness reconstruction branch is constructed; the brightness reconstruction branch includes: a feature extractor, a convolutional residual block, and an optimization module, such as... Figure 1a As shown; the feature extractor consists of N1 = 3 convolutional layers, N2 = 1 ReLU layer and N3 = 1 normalization layer, the convolutional residual block consists of N4 = 1 convolutional layer, N5 = 1 ReLU layer, N6 = 1 normalization layer and a residual connection, and the optimization module consists of N7 = 3 convolutional layers;

[0093] The scene image I is input into the brightness reconstruction branch. After being processed by the feature extractor, the encoded feature map is calculated. Then, the encoded feature map is input into M1 = 2 convolutional residual blocks for processing and calculation to recover a rough deformed pattern.

[0094] The rough deformation pattern is input into the optimization module for refinement to eliminate brightness effects, and the final deformation pattern P is calculated and generated. C ;

[0095] The rough deformed pattern and scene image I are residually joined to calculate a rough brightness image, which is then input into the optimization module for brightness reconstruction, thereby obtaining the predicted brightness image I. pred ;

[0096] Step 3.2: Construct a depth reconstruction branch consisting of a depth estimation network, and the depth estimation network is the same as the binocular depth estimation network of the teacher network;

[0097] The reference pattern P and the predicted deformed pattern P CThe data is fed into the depth reconstruction branch for processing, resulting in the final depth image D. s ;

[0098] Step 4: Constructing the training loss function;

[0099] Step 4.1: Construction of the teacher network loss function;

[0100] Step 4.1.1: Construct photometric loss using equation (1) :

[0101]

[0102] In equation (1), N is the total number of pixels in scene image I, p represents any pixel in scene image I, C represents the Census transform operation of the image block centered at pixel p, and I LCN (p) represents the normalized scene image I. LCN The pixel value of pixel p, P LCN (p) is the normalized reference pattern P. LCN The pixel value of the middle pixel p;

[0103] Step 4.1.2: Construct the edge smoothing loss using equation (2). :

[0104]

[0105] In equation (2), and These are the teacher parallax images D. t The gradient on the x-axis and y-axis of the image coordinate system;

[0106] Step 4.1.3: Construct the teacher network loss function using equation (3). :

[0107]

[0108] In equation (3), λ1 is a hyperparameter;

[0109] Step 4.2: Construction of the student network loss function;

[0110] Step 4.2.1: Constructing the illumination consistency loss function;

[0111] Step a: Construct photometric loss using equation (4)

[0112]

[0113] In equation (1), P C (p) is the deformed pattern PC The pixel value of the middle pixel p;

[0114] Step b: Construct the edge smoothing loss using equation (5)

[0115]

[0116] In equation (2), and These are student parallax images D. s The gradient on the x-axis and y-axis of the image coordinate system;

[0117] Step c: Construct the teacher network loss function using equation (5)

[0118]

[0119] In equation (6), λ2 is a hyperparameter;

[0120] Step 4.2.2 Construction of the luminance loss function:

[0121] Based on the scene's requirements for brightness recovery results, if brightness image I pred If the peak signal-to-noise ratio (PSNR) is less than the threshold ε, then proceed to step A; if the brightness image I... pred If the peak signal-to-noise ratio (PSNR) is greater than the threshold ε, then proceed to step B.

[0122] Step A: Construction of the unsupervised brightness loss function;

[0123] Step A.1: Based on the teacher's parallax diagram D t By mapping the reference pattern P using equation (7), a pseudo-deformation pattern is obtained.

[0124]

[0125] In equation (4), Warp() is the mapping operator;

[0126] Step A.2: Construct the pseudo-luminosity map I of the luminosity reconstruction branch using equation (8). pse :

[0127]

[0128] In equation (5), β is a coefficient that measures the degree of linear combination;

[0129] Step A.3: Construct an unsupervised luminance loss function L using equation (9). I :

[0130]

[0131] In equation (9), λ3 is a hyperparameter, DI() represents the dilation operator; S is the SSIM operator that measures the structural similarity between two images; I pred (p) represents the predicted brightness image I. pred The pixel value of the middle pixel p; I pse (p) represents pseudo-luminance map I pse The pixel value of the middle pixel p;

[0132] B. Construct a supervised luminance loss function L using equation (10). I :

[0133]

[0134] In equation (10), I gt (p) represents the true brightness diagram I gt The pixel value of the middle pixel p;

[0135] Step 4.2.3: Construct the distillation loss function L using equation (11). S :

[0136]

[0137] In equation (11), λ4 and λ5 are two hyperparameters that measure the corresponding proportions;

[0138] Step 4.2.4: Construct the student network loss function L using equation (12). total :

[0139]

[0140] In equation (9), α1 and α2 are two hyperparameters that measure the corresponding proportions;

[0141] In this embodiment, α1, α2, and β are 0.5. λ1, λ2, λ3, λ4, and λ5 are 0.2, 0.2, 0.15, 0.15, and 0.2, respectively.

[0142] Step 5: Self-distillation training process;

[0143] This method introduces a self-distillation strategy to address the challenge of unsupervised learning (the lack of real values ​​as guidance) by generating pseudo-real disparity maps. The self-distillation training strategy is as follows: Figure 1 As shown, the entire network extracts the teacher network as knowledge to generate supervision signals for the student network.

[0144] Step 5.1, Training for teachers on the network;

[0145] The teacher network takes a reference pattern and a captured image as input to minimize the teacher network loss function. To achieve this, the teacher network is trained separately using the Adam optimization algorithm, and its network parameters are updated until the illumination consistency loss L is reached. D The process continues until convergence, thus obtaining the trained teacher network model.

[0146] Step 5.2, Training students on the network;

[0147] The teacher network takes the morphing pattern and the captured image as input to minimize the student network loss function L. total To achieve this, the teacher network is loaded with the teacher network model trained in step 5.1, and the teacher network weights are fixed. The student network is then trained using the Adam optimization algorithm, and its network parameters are updated until the total loss function L is reached. total The process continues until convergence, resulting in a trained student network. This trained student network is then used to reconstruct the depth and brightness of the scene image.

[0148] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor in executing the above-described three-dimensional reconstruction method, and the processor is configured to execute the program stored in the memory.

[0149] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above-described three-dimensional reconstruction method.

[0150] The test results of the present invention are further described in conjunction with the following charts;

[0151] To demonstrate the effectiveness of the above-mentioned scheme of this invention, the depth and brightness characteristics of the network reconstruction were tested on synthetic data. The comparison methods were divided into three categories: traditional methods (block matching BM and semi-global matching SGM), machine learning-based methods (HyperDepth), and deep learning-based methods (CTD). HyperDepth is a fully supervised method that transforms the corresponding problem into a classification and regression task. CTD was originally designed for multi-view depth reconstruction with edge detection, but it can also be used for single-view scenes. Furthermore, this algorithm also compared the depth branch of CTD without edge detection, named CTD-NE. For the method proposed in this example, three versions are provided: Our d Our us and Our ws Our d This represents a teacher-trained deep network, trained in an unsupervised manner. us and Our wsThe difference lies in Our us The brightness branch in the model is trained in an unsupervised manner, while Our... ws The brightness branch is trained in a weakly supervised manner.

[0152] 1. Deep Reconstruction Comparison Test

[0153] The deep reconstruction results are shown in Tables 1, 2, and 3. As can be seen from the tables, this method achieves the best results compared to the comparison methods on different simulation datasets, and also achieves the best results on real datasets.

[0154] Table 1 shows the depth reconstruction results on simulation dataset 1.

[0155]

[0156] Table 2 shows the depth reconstruction results on simulation dataset 2.

[0157]

[0158]

[0159] Table 3 shows the deep reconstruction results on real dataset 1.

[0160]

[0161] Visualization results of different comparison algorithms on simulated and real datasets, as follows: Figure 3 and Figure 4 As shown, SGM has low accuracy but recovers the overall structure well. HyperDepth performs poorly in regions of parallax discontinuity. Comparing the results of CTD-NE with those of CTD, it can be seen that the problem of edge hypertrophy is effectively alleviated by introducing an edge detector. However, the edge detector in CTD offsets the advantage of the depth model in textured regions. It can be observed that the model in this example achieves the best performance, especially in regions with complex textures and contour edges.

[0162] 2. Brightness Reconstruction Comparison Test

[0163] This algorithm is the first to reconstruct a luminance image of a face from a captured image. To demonstrate its superiority, this example compares it with the Frequency Domain Filtering (FDF) method, which originates from Fourier Transform profilometry. Specifically, the captured image is first transformed to the frequency domain using a Fourier Transform. Then, the values ​​of the maximum 1% frequencies are retained as the deformed pattern frequencies. Following this, an inverse Fourier Transform is performed to obtain the deformed pattern, which is then used to generate the luminance image. Table 4 lists the quantitative results of the different methods. For FDF, the PSNR and SSIM values ​​are low due to image frequency aliasing and filtering frequency leakage. In contrast, this algorithm achieves better results.

[0164] Table 4 shows the brightness reconstruction results on both simulation and real data.

[0165]

[0166] exist Figure 5 This example demonstrates the luminance image recovery results of FDF and our algorithm on a simulated dataset. ws The best performance is achieved with FDF. FDF performs well in simple scenes; however, FDF's image frequency aliasing leads to noisy and indistinct images in complex scenes. us The brightness image is superior to the brightness image of the FDF. Although Our us The brightness of the image is greater than Our ws The brightness image has more noise, but Our us It still produces better brightness images in some background areas, such as Figure 5 The last column. Brightness images recovered from the real scene, such as... Figure 6 As shown. FDF and Our us This produces a coarse brightness image. This is because frequency filtering methods are difficult to use in real-world, complex scenes. us The performance was not satisfactory, possibly due to the small size of the real-world dataset for fine-tuning. However, when the brightness recovery branch became weakly supervised, the recovery of the brightness image was encouraging, with PSNR and SSIM metrics of 26.10 dB and 0.87, respectively. Overall, both training methods proposed in this algorithm can recover deformed patterns well and are more favorable for depth estimation.

Claims

1. A three-dimensional reconstruction method using speckle structured light, characterized in that, It is applied to a structured light system consisting of a projector and a camera, and is carried out according to the following steps: Step 1: Project the design pattern onto the scene using the projector, and capture the projected scene image I with the camera; project the design pattern onto a reference plane at a known distance using the projector, and capture the reference pattern P with the camera. Step 2: Construct a teacher network consisting of a binocular depth estimation network to recover the depth of the scene image I and obtain the teacher disparity map D. t ; Step 3: Construct a student network consisting of a brightness reconstruction branch and a depth reconstruction branch to recover the depth and brightness of the scene image I, and obtain the final depth image D. s ; Step 4: Constructing the training loss function; Step 4.1: Construction of the teacher network loss function; Step 4.1.1: Construct photometric loss using equation (1) In equation (1), N is the total number of pixels in scene image I, p represents any pixel in scene image I, C represents the Census transform operation of the image block centered at pixel p, and I LCN (p) represents the normalized scene image I. LCN The pixel value of pixel p, P LCN (p) is the normalized reference pattern P. LCN The pixel value of the middle pixel p; Step 4.1.2: Construct the edge smoothing loss using equation (2). In equation (2), and These are the teacher parallax images D. t The gradient along the x and y axes of the image coordinate system; D t It is a teacher parallax diagram; Step 4.1.3: Construct the teacher network loss function using equation (3). In equation (3), λ1 is a hyperparameter; Step 4.2: Construction of the student network loss function; Step 4.2.1: Constructing the illumination consistency loss function; Step a: Construct photometric loss using equation (4) In equation (1), P C (p) is the deformed pattern P C The pixel value of the middle pixel p; Step b: Construct the edge smoothing loss using equation (5) In equation (2), and These are student parallax images D. s The gradient on the x-axis and y-axis of the image coordinate system; Step c: Construct the teacher network loss function using equation (5) In equation (6), λ2 is a hyperparameter; Step 4.2.2 Construction of the luminance loss function: Based on the scene's requirements for brightness recovery results, if brightness image I pred If the peak signal-to-noise ratio (PSNR) is less than the threshold ε, then proceed to step A; if the brightness image I... pred If the peak signal-to-noise ratio (PSNR) is greater than the threshold ε, then proceed to step B. Step A: Construction of the unsupervised brightness loss function; Step A.1: Based on the teacher's parallax diagram D t By mapping the reference pattern P using equation (7), a pseudo-deformation pattern is obtained. In equation (4), Warp() is the mapping operator; Step A.2: Construct the pseudo-luminosity map I of the luminosity reconstruction branch using equation (8). pse : In equation (5), β is a coefficient that measures the degree of linear combination; Step A.3: Construct an unsupervised luminance loss function L using equation (9). I : In equation (9), λ3 is a hyperparameter, DI() represents the dilation operator; S is the SSIM operator that measures the structural similarity between two images; I pred (p) represents the predicted brightness image I. pred The pixel value of the middle pixel p; I pse (p) represents pseudo-luminance map I pse The pixel value of the middle pixel p; Step B: Construct a supervised luminance loss function L using equation (10). I : In equation (10), I gt (p) represents the true brightness diagram I gt The pixel value of the middle pixel p; Step 4.2.3: Construct the distillation loss function L using equation (11). S : In equation (11), λ4 and λ5 are two hyperparameters that measure the corresponding proportions; Step 4.2.4: Construct the student network loss function L using equation (12). total : In equation (9), α1 and α2 are two hyperparameters that measure the corresponding proportions; Step 5: Self-distillation training process; Step 5.1, Training for Teacher Networks: To minimize the teacher network loss function To achieve this, the teacher network is trained separately using the Adam optimization algorithm, and its network parameters are updated until the illumination consistency loss L is reached. D The process continues until convergence, thus obtaining the trained teacher network model. Step 5.2, Training for student networks: To minimize the student network loss function L total With the objective set and the weights of the trained teacher network model fixed, the student network is trained using the Adam optimization algorithm, and its network parameters are updated until the total loss function L is reached. total The process continues until convergence, resulting in a trained student network. This trained student network is then used to reconstruct the depth and brightness of the scene image.

2. The three-dimensional reconstruction method of speckle structured light according to claim 1, characterized in that, Step 2 includes: Step 2.1: First, use the Local Contrast Normalization (LCN) operator to perform local contrast normalization on the scene image I and the reference pattern P respectively, to obtain the normalized scene image I. LCN and the normalized reference pattern P LCN ; Step 2.2: Convert the normalized scene image I LCN and the normalized reference pattern P LCN The data is fed into the teacher network for processing, and the teacher disparity map D is calculated. t .

3. The three-dimensional reconstruction method of speckle structured light according to claim 1, characterized in that, Step 3 includes: Step 3.1: Construct a brightness reconstruction branch; the brightness reconstruction branch includes: a feature extractor, M1 convolutional residual blocks, and an optimization module; wherein, the feature extractor consists of N1 convolutional layers, N2 ReLU layers, and N3 normalization layers, each convolutional residual block consists of N4 convolutional layers, N5 ReLU layers, N6 normalization layers, and a residual connection, and the optimization module consists of N7 convolutional layers; The scene image I is input into the brightness reconstruction branch. After being processed by the feature extractor, the encoded feature map is calculated. Then, the encoded feature map is input into M1 convolutional residual blocks for processing and calculation to recover a rough deformed pattern. The rough deformation pattern is input into the optimization module for refinement to eliminate brightness effects, and the final deformation pattern P is calculated and generated. C ; The rough deformed pattern and the scene image I are residually concatenated to calculate a rough brightness image, which is then input into the optimization module for brightness reconstruction, thereby obtaining the predicted brightness image I. pred ; Step 3.2: Construct a depth reconstruction branch consisting of a depth estimation network, wherein the depth estimation network is the same as the binocular depth estimation network of the teacher network; The reference pattern P and the predicted deformed pattern P C The data is fed into the depth reconstruction branch for processing, resulting in the final depth image D. s .

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing any of the three-dimensional reconstruction methods of claims 1-3, and the processor is configured to execute the programs stored in the memory.

5. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by a processor, performs the steps of any of the three-dimensional reconstruction methods described in claims 1-3.

Images