Three-dimensional reconstruction method, device, apparatus and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and provides a three-dimensional reconstruction method, device, equipment and storage medium. The method comprises the following steps: constructing a high-fidelity structured light simulation environment based on a rendering engine to automatically generate a simulation data set with accurate levels and depth annotations; constructing a level prediction model, training the level prediction model through a joint loss function composed of a level classification loss and a depth reconstruction loss based on a structured light physical principle, and obtaining a trained level prediction model; acquiring a wrapped phase map collected by a real structured light system; inputting the wrapped phase map into the trained level prediction model to obtain a stripe level prediction result; and performing three-dimensional reconstruction based on the stripe level prediction result and the wrapped phase map. The technical scheme of the application effectively improves the stripe level determination accuracy and generalization performance of the model in a real scene through high-fidelity simulation and global context modeling.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a three-dimensional reconstruction method, apparatus, device, and storage medium. Background Technology

[0002] Structured light 3D measurement technology, with its advantages of high precision, high speed, and non-contact operation, is widely used in industrial inspection, robot guidance, and 3D modeling. Its basic principle involves projecting an coded structured light pattern onto the object being measured, and a camera capturing images of deformed fringes modulated by the object's surface topography. The 3D shape of the object is then reconstructed by calculating the phase information of the deformed fringes. Phase unwrapping is a crucial step, aiming to determine the absolute order of each pixel's fringes from the wrapped phase image, thereby obtaining continuous absolute phase.

[0003] Traditional phase unwrapping algorithms, such as temporal phase unwrapping and multi-frequency heterodyne methods, typically require projecting multiple sets of fringe patterns at different frequencies and determining the absolute phase step-by-step through constraints between frequencies. While these methods are mature, they are prone to misjudging order when dealing with complex surfaces, discontinuous objects, or the presence of noise interference, leading to phase jumps and errors in 3D reconstruction. Furthermore, deep learning-based methods have been introduced in recent years to address the order determination problem in complex scenes. These methods usually rely on large amounts of real-world data with precise order annotations to train the model. However, obtaining large-scale, diverse, and pixel-level precisely annotated real-world data is extremely costly and almost impossible in certain extreme or dangerous scenarios, severely limiting the performance ceiling and practical application scope of data-driven methods.

[0004] To overcome the bottleneck of real-world data annotation, the Sim2Real paradigm was proposed, which utilizes simulated data to train models and aims to generalize them to real-world scenes. Some existing Sim2Real solutions attempt to generate simulated data through simple graphics rendering. However, these solutions suffer from the following drawbacks: 1) There are significant domain differences between the simulated environment and the real physical world. For example, lighting models, material reflection properties, and sensor noise are not adequately modeled, leading to inconsistencies between the distribution of simulated and real data, and poor model generalization ability; 2) Existing deep learning-based order determination models are mostly convolutional neural networks with local receptive fields, making it difficult to effectively model the inherent periodic structural information spanning the entire field of view in sinusoidal fringe images. When dealing with broken, occluded, or low-texture areas, they are prone to local order errors due to a lack of global context constraints. These errors propagate and accumulate during the reconstruction process, ultimately affecting the overall accuracy and robustness of 3D reconstruction. Summary of the Invention

[0005] Based on this, it is necessary to address the shortcomings of the existing technologies, such as the large difference between the simulation data and the real scene, and the lack of modeling ability for the global periodicity of stripe images, which leads to insufficient accuracy in order discrimination and generalization ability. Therefore, a three-dimensional reconstruction method, device, equipment and storage medium are proposed.

[0006] Firstly, a three-dimensional reconstruction method is provided, the method comprising: A structured light simulation environment is built based on the rendering engine to simulate the optical projection and imaging process of the structured light system. The material, pose and occlusion relationship of the object are parameterized and randomized to automatically generate a simulation dataset including level annotation and depth annotation. A hierarchical prediction model is constructed to capture the global periodic dependencies of an image. Using the simulation dataset, the hierarchical prediction model is trained by a joint loss function consisting of hierarchical classification loss and depth reconstruction loss based on the physical principle of structured light, and the trained hierarchical prediction model is obtained. To obtain a wrapping phase map acquired by a real structured light system, the following steps are taken: controlling a projector to project at least three different frequencies of sinusoidal fringe patterns onto the object under test, and simultaneously acquiring corresponding deformed fringe images by a camera; performing phase shift calculation on the deformed fringe images of each frequency group to obtain multiple wrapping phase maps of different frequencies. The wrapped phase map is input into the trained level prediction model to obtain the stripe level prediction result; Based on the stripe order prediction results and the package phase map, three-dimensional reconstruction is performed.

[0007] Secondly, a three-dimensional reconstruction apparatus is provided, the apparatus comprising: The generation module is used to build a structured light simulation environment based on the rendering engine, simulate the optical projection and imaging process of the structured light system, and perform parameterized randomization on the object material, pose and occlusion relationship to automatically generate a simulation dataset including level annotation and depth annotation. The training module is used to construct a hierarchical prediction model, which is used to capture the global periodic dependencies of an image. Using the simulation dataset, the hierarchical prediction model is trained by a joint loss function consisting of hierarchical classification loss and depth reconstruction loss based on the physical principle of structured light, so as to obtain the trained hierarchical prediction model. The acquisition module is used to acquire the wrapping phase map collected by the real structured light system, including: controlling the projector to project at least three different frequency sinusoidal fringe patterns onto the object under test, and the camera synchronously acquiring the corresponding deformed fringe images, performing phase shift calculation on the deformed fringe images of each frequency to obtain multiple wrapping phase maps of different frequencies. The prediction module is used to input the package phase map into the trained level prediction model to obtain the stripe level prediction result; The reconstruction module is used to perform three-dimensional reconstruction based on the stripe level prediction results and the package phase map.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described target stripe level determination method.

[0009] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described target stripe level determination method.

[0010] As can be seen from the technical solutions provided in this application above: On the one hand, by employing a structured light simulation environment based on a rendering engine and parameterizing and randomizing the material, pose, and occlusion relationships of objects, the optical projection and imaging processes of a real structured light system, as well as various complex object surface properties and scene layouts, can be simulated highly realistically in a virtual environment. The data distribution of the automatically generated simulation dataset is closer to the complexity of the real physical world, thereby significantly reducing the domain difference between simulation data and real data. This allows the deep learning network model trained using this dataset to learn data features and patterns that are closer to real scenes, thus greatly improving the model's generalization performance and practicality when facing unseen real objects and scenes, overcoming the dependence on expensive and difficult-to-obtain real labeled data. On the other hand, the hierarchical prediction model is used to capture the global periodic dependencies of the image. It is no longer limited to the visual features of local areas and can model long-range dependencies of features along the row and column directions of the image, thereby understanding the periodic pattern and continuous change of sinusoidal stripes in the entire image. This utilization of global contextual information enables the order prediction model to infer based on a wider range of image structural information when determining the fringe order. In particular, it can effectively suppress local misjudgments and maintain the overall consistency of order changes at fringe breaks, object edges, occluded areas, or noise interference, thereby significantly improving the accuracy of order determination results and robustness to complex situations. Thirdly, the technical solution of this application organically combines physical simulation-driven data generation, global context-aware order prediction model training, and preprocessing and inference processes for real data into a coherent method. After the model is trained on physically realistic simulation data, it can directly process the wrapping phase map acquired by the real structured light system and output accurate fringe order prediction results. Combining the order prediction results with the original wrapping phase map, subsequent 3D reconstruction can be completed stably and reliably. The entire process does not require tedious annotation or additional model fine-tuning for real data, realizing efficient and high-precision transfer from the simulation environment to real application scenarios, and providing a powerful solution for automated 3D measurement. In summary, the technical solution of this application effectively improves the accuracy and generalization performance of the model in determining stripe order in real-world scenarios through high-fidelity simulation and global context modeling. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1Here is a flowchart of a three-dimensional reconstruction method in one embodiment; Figure 2 This is a structural block diagram of a three-dimensional reconstruction device in one embodiment; Figure 3 This is a structural block diagram of a computer device in one embodiment; Figure 4 This is a schematic diagram of the deep learning network inference and deep reconstruction process in one embodiment. Figure 5 This is a comparison diagram of the effects of traditional algorithms and the three-dimensional reconstruction of embodiments of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Traditional phase unwrapping algorithms, such as time-phase unwrapping and multi-frequency heterodyne methods, typically require projecting multiple sets of fringe patterns at different frequencies and determining the absolute phase step-by-step through constraints between frequencies. While these methods are mature, they are prone to misjudging order when dealing with complex surfaces, discontinuous objects, or noise interference, leading to phase jumps and 3D reconstruction errors. Furthermore, deep learning-based methods have been introduced in recent years to address order determination in complex scenes. These methods usually rely on large amounts of real-world data with precise order annotations to train the model. However, obtaining large-scale, diverse, and pixel-level precisely annotated real-world data is extremely costly and almost impossible in certain extreme or dangerous scenarios, severely limiting the performance ceiling and practical application scope of data-driven methods. To overcome the bottleneck of real-world data annotation, the Sim2Real paradigm has been proposed, which uses simulated data to train a model and expects it to generalize to real-world scenes. Some existing Sim2Real schemes attempt to generate simulated data through simple graphics rendering. However, these solutions have the following drawbacks: 1) There are serious domain differences between the simulation environment and the real physical world. For example, the lighting model, material reflection properties, and sensor noise are not fully modeled, resulting in inconsistent simulation data distribution with real data distribution and poor model generalization ability; 2) Existing deep learning-based order determination models are mostly convolutional neural networks with local receptive fields, which are difficult to effectively model the inherent periodic structural information in sinusoidal stripe images that spans the entire field of view. When dealing with broken, occluded, or low-texture areas, they are prone to local order errors due to the lack of global context constraints. These errors will propagate and accumulate during the reconstruction process, ultimately affecting the overall accuracy and robustness of 3D reconstruction.

[0014] To address the aforementioned problems in the existing technology, this application proposes a three-dimensional reconstruction method, the main process of which is as follows: Figure 1 As shown, it mainly includes steps S1 to S5, which are detailed below: Step S1: Construct a structured light simulation environment based on the rendering engine to simulate the optical projection and imaging process of the structured light system, and perform parameterized randomization on the object material, pose and occlusion relationship to automatically generate a simulation dataset including level annotation and depth annotation.

[0015] In existing technologies, a common practice for obtaining simulation data to train deep learning models is to use simplified graphics rendering, ignoring the real physical processes of optical imaging, or using a simple diffuse reflection model for object materials. For example, schemes that use simplified graphics rendering (i.e., non-physical rendering) to generate simulation data produce data that differs greatly from the real world in terms of physical properties such as lighting, material reflection, and optical distortion. This leads to a severe performance degradation of models trained on such data when encountering real data. Even with data augmentation methods such as generative adversarial networks, their shortcomings remain significant. The generated data may be visually realistic, but it lacks pixel-level accurate physical ground truth (e.g., precise depth, order), and the generation process is uncontrollable, making it difficult to guarantee consistency with the physical principles of measurement.

[0016] To address the significant domain discrepancy (Sim2Real domain offset) between simulation and real data at its source, this application abandons simple graphics rendering and instead employs a rendering engine-based approach to construct the simulation environment. Specifically, a structured light simulation environment is built using the rendering engine to simulate the optical projection and imaging process of a structured light system. Object materials, poses, and occlusion relationships are parameterized and randomized to automatically generate a simulation dataset including hierarchy and depth annotations. This approach not only achieves automated and low-cost generation of large-scale, precisely annotated datasets but also covers complex and rare industrial scenarios such as high reflectivity and multi-object occlusion through parameter randomization, laying a data foundation for training high-performance models. Furthermore, by importing real hardware parameters and using physically based rendering, the simulation data achieves a high degree of consistency with the real system in optical characteristics, significantly reducing the distributional differences between simulation and real data. This provides the possibility for cross-domain generalization of subsequent models. The simulation environment directly calculates error-free absolute phase, depth, and other ground truth values ​​without any manual intervention, avoiding errors and subjectivity introduced by manual annotation.

[0017] As one embodiment of this application, a structured light simulation environment is constructed based on a rendering engine to simulate the optical projection and imaging process of a structured light system. The object material, pose, and occlusion relationship are parameterized and randomized to automatically generate a simulation dataset including level annotation and depth annotation. This can be achieved through the following steps S11 to S14: Step S11: Import the intrinsic and extrinsic parameters and distortion coefficients of the camera and projector in the real structured light system into the physical rendering engine to accurately reproduce the optical projection and imaging process.

[0018] Specifically, step S11 can be: creating two camera objects in Blender, representing a real measuring camera and a projector respectively; precisely assigning the parameters obtained by calibrating the real hardware to these two objects; for the camera, setting its intrinsic parameter matrix (which includes focal length fx, fy and principal point coordinates cx, cy), image sensor size, resolution, and lens distortion coefficients (e.g., radial distortion k1, k2, k3 and tangential distortion p1, p2); for the projector, similarly setting its intrinsic parameters and treating it as a reverse camera used to project images onto the scene rather than acquire them. Through this step, the virtual optical system in the simulation environment is completely consistent with the real system in terms of geometric imaging model, ensuring that the simulated fringe image is comparable to the real image in terms of perspective and distortion characteristics.

[0019] Step S12: In the rendering engine, define at least one optical property parameter among diffuse reflection coefficient, specular reflection coefficient, metallicity, and roughness for the simulated object.

[0020] The reflective properties of an object's surface are one of the key factors causing differences in the image domain. In physically based rendering, physically based rendering (PBR) material models are typically used to describe object surfaces. In this application, to simulate various materials commonly found in industry, corresponding PBR material parameters need to be configured for each simulated object. For example, for highly reflective metallic materials, a high metallicity value (close to 1.0) and a low roughness value (close to 0.0) are set to simulate strong specular reflection and weak diffuse reflection; for transparent or translucent plastic materials, a non-zero refractive index parameter is set, along with appropriate transmission parameters, to simulate the refraction and internal scattering effects of light. For ordinary plastics, rubber, ceramics, etc., their realistic appearance is approximated by adjusting diffuse color, specular intensity, and specular roughness. By parameterizing these material properties, the simulation data can cover a wide range of materials, from diffuse to specular metals, and from opaque to transparent / translucent, greatly enhancing the diversity and realism of the data.

[0021] Step S13: Based on the defined optical property parameters, randomly set the size of the object, the three-dimensional spatial pose, and the degree of mutual occlusion between multiple objects in the structured light simulation environment.

[0022] To simulate the complex situation of disordered stacking and mutual occlusion of workpieces in an industrial setting, and to avoid overfitting the model to specific poses, step S13 randomizes the scene layout. Specifically, based on defined optical attribute parameters, the size of objects, their three-dimensional spatial pose, and the degree of mutual occlusion between multiple objects are randomly set in the structured light simulation environment through steps S131 to S133, as detailed below: Step S131: Load multiple 3D object models into the simulation scene.

[0023] These models can be derived from common computer-aided design (CAD) model libraries, covering various industrial parts such as nuts, bolts, gears, and connectors.

[0024] Step S132: Randomly assign a translation and rotation transformation in three-dimensional space to each three-dimensional object model to determine its pose.

[0025] That is, a position coordinate (x, y, z) and a set of Euler angles (rx, ry, rz) are randomly generated for each object, and then applied to the object.

[0026] Step S133: Based on the pose assigned to each object, calculate the bounding box overlap region between the 3D object models to quantify and record the degree of occlusion.

[0027] After each randomly generated pose, the program calculates the pairwise intersection volume of the bounding boxes along the 3D axes of all objects. By setting a threshold, severely occluded scenes can be filtered out, or the degree of occlusion can be recorded as metadata for subsequent analysis of the model's performance in occluded regions. This parameterized randomization strategy can automatically generate massive amounts of diverse training samples, ensuring that the model can learn the deformation patterns of stripes under various poses and occlusion relationships, thereby improving its robustness.

[0028] Step S14: Project a multi-step phase-shift sinusoidal fringe pattern onto the scene containing the simulated object, acquire the modulated deformed fringe image, and automatically calculate and generate the corresponding wrapping phase map, integer fringe order map, depth map, and effective region mask based on the absolute phase and depth information generated by the structured light simulation environment to form a simulation dataset.

[0029] First, the program controls a virtual projector to sequentially project a set (usually 3 or 4 steps) of sinusoidal fringe patterns with fixed phase differences (e.g., 2π / 3 or π / 2) onto the scene. For each projected pattern, the virtual camera captures a corresponding image of deformed fringe patterns modulated by the object's surface shape. After acquiring all phase-shifted images, a wrap-around phase map can be calculated from this set of images using a standard multi-step phase-shifting algorithm. The key advantage is that the simulation environment can directly access the absolutely correct 3D spatial coordinates (i.e., depth values) corresponding to each pixel, as well as the original phase values ​​originating from the projector. These error-free depth and original phase values, directly provided by the simulation system, constitute the ground truth. Using these ground truths, an unambiguous, globally continuous absolute phase map can be calculated in reverse. Taking the absolute phase map modulo 2π yields the simulated wrap-around phase map; dividing the absolute phase map by 2π and rounding down yields a completely correct integer fringe order map. Simultaneously, the depth map can be directly obtained from the rendering engine's depth channel. The effective region mask is generated by determining whether a pixel is on the object's surface (not the background) and whether stripes have been successfully projected onto it. The entire process is fully automated and error-free, perfectly solving the pain point of high costs associated with real-world data annotation.

[0030] In the technical solutions exemplified by steps S11 to S14 above, the consistency between the simulation environment and the optical geometric characteristics of the real system is ensured by importing real hardware parameters. The diversity and complexity of the dataset are guaranteed by parameterizing and randomizing the material and pose, thus achieving fully automated and zero-error generation of data from simulation to annotation.

[0031] Furthermore, after step S14, the simulation dataset constructed in step S14 can be divided to obtain a training subset for model training and a validation subset for verifying the model's generalization ability on real data. After generating tens of thousands of samples, the dataset needs to be reasonably divided. Typically, most samples (e.g., 80%) are randomly assigned to the training set for model parameter learning. The remaining samples (e.g., 20%) serve as the validation set, but their function differs from that of a traditional validation set. The validation set in this application is mainly used to periodically evaluate the model's performance on unseen simulation scenarios during training to monitor whether the training process is overfitting. To objectively evaluate the model's transfer from simulation to reality (Sim2Real) effect, after model training is completed, final validation can be performed on its performance on a small amount of real-world data (referred to as the "real validation set"). This perfectly aligns with the core advantage of the technical solution mentioned in this application, which eliminates the need for real-world level labeling. Evaluation of the real-world validation set only requires registration and error comparison between the model-predicted levels and the reference 3D point cloud reconstructed from the wrapped phase map, and a reference point cloud obtained using a high-precision 3D scanner or similar equipment. Pixel-level stripe level labels for these real-world images are completely unnecessary. In this way, the effectiveness of Sim2Real transfer can be objectively measured.

[0032] Step S2: Construct a hierarchical prediction model. The hierarchical prediction model is used to capture the global periodic dependencies of the image. Using a simulation dataset, the hierarchical prediction model is trained by a joint loss function consisting of hierarchical classification loss and depth reconstruction loss based on the physical principle of structured light, and the trained hierarchical prediction model is obtained.

[0033] In the stripe order determination task, the wrapped phase map exhibits a significant periodic stripe structure. While traditional convolutional neural networks excel at extracting local features, their limited receptive field makes it difficult to model the long-range dependencies of stripes spanning multiple periods across the entire field of view. This leads to localized order misclassifications in areas such as stripe breaks, occlusion boundaries, or low-texture regions, and these errors lack global consistency constraints. To address this issue, this application introduces a sequence modeling module on top of a classic encoder-decoder (e.g., U-Net) skeleton, enabling the network to possess global context awareness. Furthermore, to combat the domain shift of Sim2Real and allow the model to learn fundamental physical laws beyond simulated textures, this application proposes a unique joint physical supervision mechanism. Specifically, this application employs the following technical solution: constructing an order prediction model to capture the global periodic dependencies of the image; using a simulated dataset, training the order prediction model using a joint loss function consisting of order classification loss and depth reconstruction loss based on structured light physics principles, resulting in a trained order prediction model. The aforementioned technical solution enables the order prediction model to capture long-range contextual information along the row and column directions of the stripes, thereby achieving an understanding of the overall periodic pattern of the stripes. This allows the model to refer to the global stripe pattern when determining local orders, effectively reducing isolated misjudgments caused by local noise, occlusion, or reflection, and improving the overall consistency of the order map. At the same time, the joint loss function based on depth reconstruction associates discrete order prediction with the ultimate physical goal of continuous depth reconstruction. This supervision forces the features learned by the model to not only correctly classify orders but also satisfy the basic geometric constraints of structured light measurement. As a result, the model learns more essential, domain-independent physical laws rather than the apparent features in the simulation data, significantly enhancing the robustness and generalization ability of the model when transferring from the simulation domain to the real domain.

[0034] As an embodiment of this application, step S2 can be implemented by the following steps S21 to S24, as detailed below: Step S21: Construct a hierarchical prediction model with an encoder-decoder structure, wherein the encoder contains multiple downsampled convolutional blocks to extract multi-scale features of the image.

[0035] The encoder in the encoder-decoder architecture is responsible for extracting hierarchical features from the input wrapped phase map, specifically designed as four cascaded convolutional blocks. Each convolutional block contains, in sequence: a 3x3 convolutional layer (for feature transformation), a batch normalization layer (for accelerating training and improving stability), a ReLU activation function (for introducing non-linearity), and a 2x2 average pooling layer (for spatial downsampling and expanding the receptive field). The number of output channels of the four convolutional blocks can be set to 16, 32, 64, and 128 respectively, thereby continuously increasing the abstract expressive power of its feature dimensions while reducing the spatial resolution of the feature map, capturing multi-scale information from edges and textures to higher-level semantics.

[0036] Step S22: On the end feature map of the encoder, perform sequence modeling operations along the row and column directions of the image respectively to capture the global periodic context information of the stripes and output a fused feature map that incorporates the global periodic context information.

[0037] In this embodiment, although the feature map output by the encoder contains high-level semantics, its spatial structure is still aggregated through local convolution operations. To explicitly model the periodic continuity of the stripes along the row and column directions, this application introduces sequence modeling on the feature map. Specifically, on the end feature map of the encoder, performing sequence modeling operations along the image row and column directions can be achieved through the following steps S221 to S223: Step S221: Unfold the encoder end feature map into feature sequences along the horizontal and vertical directions respectively.

[0038] Specifically, for a feature map of size [H, W, C], it can be reshaped into H sequences of length W (row direction) and W sequences of length H (column direction), with the feature dimension of each sequence point being C.

[0039] Step S222: Use a bidirectional long short-term memory network to model the feature sequences in the horizontal and vertical directions respectively, so as to fuse global context information.

[0040] A bidirectional long short-term memory (Bi-LSTM) network is configured for each direction. Taking the row direction as an example, the Bi-LSTM processes each row sequence in both front-to-back and back-to-front directions, so that the features at each position can aggregate information from all other positions in that row. The same processing is performed for the column direction.

[0041] Step S223: Recombine the feature sequences processed in both directions into a two-dimensional feature map.

[0042] The row and column sequences output by the Bi-LSTM are restored to the shape [H, W, C], respectively. Then, the two feature maps are concatenated or added along the channel dimension to obtain a fused feature map that incorporates global contextual information from the rows and columns. This fused feature map is then fed into the decoder for upsampling and detail restoration.

[0043] In another embodiment of this application, to further improve the prediction accuracy of the model in complex scenes, especially in highly reflective edge regions, the above-mentioned sequence modeling operation is performed not only on the end feature map of the encoder, but also on multiple levels of the encoder. That is, another implementation of step S22 in the above embodiment can be: performing sequence modeling operations along the image row and column directions at multiple levels of the encoder to capture the global periodic context information of the stripes at each level, and fusing the global context information obtained at each level with the local features of the corresponding level to obtain multiple multi-scale feature maps that have fused global context information. Specifically, at each stage of the encoder's downsampling, multiple intermediate level feature maps are selected, and the sequence modeling operations described in steps S221 to S223 are performed on these feature maps along the image row and column directions. In this way, shallow feature maps can also obtain global periodic information, thereby better recovering the local stripe structure damaged by high reflectivity, occlusion, and other factors during subsequent decoding, significantly improving the model's level determination accuracy in these difficult regions. The feature maps obtained at each level, which incorporate global context information, will be fused with the features of the corresponding level in the decoder through skip connections and other methods to form multi-scale global-local features, which will be used for the final level prediction.

[0044] Step S23: During model training, based on the fused feature map, calculate the stripe level classification cross-entropy loss for each pixel as the level classification loss.

[0045] The decoder's final output is a feature map of dimensions [H, W, K], where K is the maximum possible number of stripe levels (i.e., the number of classes). This is transformed into a probability distribution for each pixel across K levels using the Softmax function. Level classification loss is then applied. The standard cross-entropy loss function is used for calculation, but only within the effective region mask generated in step S14 above. The formula is as follows: Where i represents the number of pixels traversed, and N is the total number of valid pixels. It is the mask value (1 for valid, 0 for background). It is the probability that the model predicts pixel i to be of order k. The loss is the true level label provided by the simulation data. This loss directly supervises the model to make the correct level classification.

[0046] Step S24: During model training, based on the fused feature map, the stripe level distribution predicted by the model is converted into a continuous soft level through differentiable operations. The predicted depth map is reconstructed by combining the wrapped phase map, and the mean square error between the predicted depth map and the true depth map is calculated as the depth reconstruction loss.

[0047] Since level classification is discrete while depth reconstruction is continuous, the discrete level predictions can be "softened" to establish a connection. The above-described scheme for converting the model-predicted stripe level distribution into continuous soft levels through differentiable operations is as described in steps S241 to S242: Step S241: Apply the Gumbel-Softmax function to the classification logits output by the model to obtain a smooth, differentiable order probability distribution.

[0048] Gumbel-Softmax is a differentiable approximation of Softmax. During training, it adds Gumbel noise and uses a smoothed version of argmax, allowing gradient backpropagation for the "sampling" operation from the classification distribution. The temperature coefficient τ is a key hyperparameter that controls the smoothness of the output distribution: a larger τ value results in a smoother distribution (approaching a uniform distribution); a smaller τ value results in a sharper distribution (approaching a one-hot distribution). Initially, a large τ (e.g., 1.0) is typically set to allow gradients to flow freely between classes. As training progresses, an annealing strategy can be used to gradually decrease τ (e.g., decaying to 0.1 according to a cosine law), causing the distribution to gradually approximate the discrete decisions, thereby stabilizing training and improving final performance.

[0049] Step S242: Perform a weighted summation of the order probability distribution and the integer order value sequence to calculate the continuous soft order values.

[0050] Specifically, the probability vectors of the K categories are multiplied by an integer sequence [0, 1, 2, …, K-1] to obtain a continuous soft-level subvalue. Then, based on the basic formula for structured light measurement... Using the input wrap phase and soft level The soft absolute phase is calculated. Finally, the soft absolute phase is converted into a predicted depth map using a pre-calibrated lookup table describing the phase-depth mapping. Deep reconstruction loss Calculate the predicted depth map Compared with simulated real depth map Mean square error within the effective region: Where i represents all pixels in the image being traversed, and N is the total number of valid pixels. It is the effective area mask value generated in step S14 (effective pixels are 1, background or invalid area is 0). and and represent the predicted depth value and the true depth value at pixel i, respectively. This loss function constrains the model from the physical essence of 3D measurement, requiring that its predicted levels must be able to reconstruct the correct 3D shape, thereby greatly enhancing the generalization and physical rationality of the model's learning.

[0051] Please see Figure 4 This diagram illustrates the deep learning network inference and deep reconstruction process. The entire process achieves a leap from 2D striped images to 3D geometric point clouds. Its core innovation lies in using Gumbel-Softmax to solve the differentiability problem of discrete order determination, and combining it with a lookup table to accelerate efficient physical reconstruction. The network receives a four-channel tensor as input, including three normalized wrapper phase maps (p11, p12, and p13) and a binarized order mask used to mask invalid regions. Before entering the backbone network, these four independent data streams are merged into a high-dimensional composite tensor through tensor concatenation operations in the channel dimension, thereby achieving early fusion of multimodal information. The concatenated tensor is then fed into the core order prediction model. This model typically employs a fully convolutional architecture, responsible for automatically extracting multi-scale features from the input data, ranging from low-level texture to high-level semantics, and outputting a prediction score of dimension (B, N, H, W), where B is the batch size, N is the number of order categories, and H and W are the image height and width, respectively. To address the non-differentiability problem caused by the discreteness of phase orders, Figure 4 A dual-branch output structure was designed: Branch 1 (i.e., Gumbel-Softmax sampling): This branch utilizes the F.gumbel_softmax operation, a reparameterization technique for discrete variables. It allows gradient backpropagation through the sampling process, outputting an approximately one-hot soft code, which is then decoded into continuous order predictions via the (one_hot * index).sum operation.

[0052] Branch 2 (i.e., cross-entropy classification): This branch is directly connected to the cross-entropy loss function as an auxiliary supervision signal, driving the network to learn a more accurate class probability distribution.

[0053] Regardless of whether the order prediction results are obtained through Gumbel-Softmax sampling or cross-entropy classification, they all enter the physical post-processing stage. First, the absolute phase is calculated using the system's calibration parameters. Then, a fast lookup operation (reconstructPointCloudViaLUT) is performed using a pre-built depth lookup table (LookupTableTorch::readTable) to directly map the absolute phase to physical depth values, generating preliminary depth reconstruction results. To improve the robustness of the reconstruction, the network uses the calculated order to generate an order mask. This mask is multiplied element-wise with the depth mask generated based on the depth (mask = order mask * depth mask), thereby accurately removing background noise and invalid measurement areas, ensuring the reliability of subsequent calculations.

[0054] To achieve simultaneous optimization of level determination and depth reconstruction, Figure 4 A weighted joint loss function is proposed to calculate the joint loss, for example, Among them, the loss function calculated from the mean squared error is... Used to constrain the numerical deviation between the predicted and actual levels, ensuring the accuracy of periodic unwrapping. This is used to measure the geometric error between the reconstructed depth and the true depth map. By setting a small weight coefficient (i.e., 0.01), the dominance of hierarchical classification is ensured, while the depth information provides physical regularization constraints, significantly improving the model's generalization ability in complex scenes.

[0055] It should be noted that in the above embodiments, the level classification loss and the depth reconstruction loss together constitute the joint loss function, which is used to optimize the deep learning network model through backpropagation. The total loss function is the weighted sum of the two: ,in, It is a weighting coefficient that balances the two losses, and is usually set to a small value (e.g., 0.01) to ensure the dominance of the hierarchical classification task while introducing physical constraints for deep reconstruction.

[0056] It should also be noted that in this embodiment, the above description uses a bidirectional LSTM as an example, but those skilled in the art will understand that other sequence models with long-range dependency modeling capabilities, such as GRU or Transformer, can also be applied here.

[0057] The technical solutions exemplified in steps S21 to S24 above clearly define the fusion feature map as the basis for calculating the two losses, ensuring that the features learned by the model serve both classification and reconstruction tasks. Steps S23 and S24 specify the calculation methods for the two losses, especially linking discrete-level prediction with continuous-depth reconstruction during training. This design forces the model to learn from the physical essence of structured light measurement rather than merely fitting the data appearance, thereby greatly enhancing the model's generalization ability from the simulation domain to the real domain and its prediction robustness in complex scenarios.

[0058] In the above embodiments, training the level prediction model may include the iterative optimization process exemplified by steps S25 to S27: Step S25: In each training iteration, associate the previous prediction output of the hierarchical prediction model with the input features to generate residual information.

[0059] Drawing inspiration from optimization algorithms, a lightweight update module can be designed. This module is conditioned on the model prediction results of the current iteration (e.g., the order probability map) and the original input features.

[0060] Step S26: Utilize the residual information to adjust the parameters of the network computation model through a lightweight update.

[0061] The update network (which may consist of several convolutional layers) calculates a set of update amounts Δ based on the residuals between the current prediction and the ideal state, which are used to fine-tune certain features or parameters within the main network.

[0062] Step S27: Iteratively update the model's internal state or intermediate prediction results based on the adjustment amount, and output the final training loss after multiple iterations to optimize the model weights.

[0063] The updated values ​​are applied to the main network to obtain the updated intermediate state, and then forward propagation is performed again to calculate the new predictions and loss. This process is repeated several times (e.g., 3-5 times) in the forward propagation of a single training sample, and finally the loss from the last round is used for backpropagation. This mechanism enables the model to learn to perform multi-step inference and self-correction internally.

[0064] For the complex deep learning model that includes a sequence modeling module and a joint loss, in order to configure an optimization strategy to ensure stable and efficient training convergence and achieve the best performance, in the above embodiment, the level prediction model is trained using a simulation dataset through a joint loss function consisting of level classification loss and depth reconstruction loss based on the physical principle of structured light. The trained level prediction model can be obtained through the following steps S28 and S29: Step S28: Perform model parameter updates using the AdamW (Adaptive Moment Estimation with Weight Decay) optimizer.

[0065] The AdamW optimizer was used for model training. It decouples weight decay from Adam, which can more effectively prevent overfitting and help the model learn more generalized features.

[0066] Step S29: Configure a cosine annealing learning rate scheduler for the AdamW optimizer to decay the learning rate from its initial value to its minimum value according to a cosine law during training.

[0067] The initial learning rate can be set to The minimum value is set as The cosine annealing strategy keeps the learning rate relatively high in the early stages of training so that it decreases rapidly, and then decreases it slowly in the later stages of training so that parameters can be finely tuned, which helps the model converge to a better performance point.

[0068] The training schemes in steps S28 to S29 employ the AdamW optimizer and weight decay to help prevent overfitting and improve the model's generalization ability. Combined with cosine annealing learning rate scheduling, the learning rate can decrease rapidly in the early stages of training and be finely adjusted in the later stages, making the model more likely to converge to a better local optimum. This carefully designed training strategy ensures that the innovative model architecture potential in this application is fully explored, thereby obtaining a final model with better performance and greater stability.

[0069] Step S3: Obtain the package phase map acquired by the real structured light system.

[0070] The model trained in the simulation environment eventually needs to be deployed to a real fringe projection 3D measurement system. The purpose of step S3 is to prepare real data that meets the input requirements of the model. Traditional deep learning-based order determination methods usually only input a single-frequency wrap-around phase map, which may result in more phase ambiguity areas under high-frequency fringes or complex shapes.

[0071] As an embodiment of this application, obtaining the package phase map acquired by a real structured light system can be achieved through the following steps S31 to S33.

[0072] Step S31: Control the projector to project at least three different frequencies of sinusoidal stripe patterns onto the object being measured, and have the camera simultaneously acquire the corresponding deformed stripe images.

[0073] Specifically, the projector is controlled to project at a frequency of [frequency value missing]. , , (For example, < < The camera and projector synchronize to capture images of all the deformed fringes, which consist of three sets of sinusoidal fringes, each containing multiple phase-shifted patterns (e.g., 3-step or 4-step).

[0074] Step S32: Perform phase shift calculation on the deformed fringe images of each frequency group to obtain multiple wrap-around phase images of different frequencies.

[0075] For each set of frequency phase-shifted images, the same phase-shifting algorithm (such as the three-step phase-shifting method) used when generating the simulation data was employed to calculate the wrap-around phase map. , and .

[0076] Step S33: Use multiple wrap-around phase maps of different frequencies as multi-channel inputs to the order prediction model.

[0077] The calculated wrapper phase maps of different frequencies are stitched together along the channel dimension to form a multi-channel input tensor, for example, with an input shape of [H, W, 3]. This multi-frequency input provides the model with phase information from coarse to fine. Low-frequency phase maps have fewer orders and lower ambiguity, which helps to determine the approximate range; high-frequency phase maps are rich in detail, which helps to pinpoint the exact location. The model can learn to fuse this information to make more robust order determinations.

[0078] After obtaining the wrapped phase map, preprocessing can be performed to align the distribution of the real data with the distribution of the simulation data seen during model training, thereby reducing the impact of domain differences. Preprocessing typically includes normalization operations, such as linearly scaling the numerical range of the wrapped phase map to the interval [-π, π] or [0, 1], which is consistent with the processing method used when generating the simulation data.

[0079] Step S4: Input the wrapped phase map into the trained level prediction model to obtain the stripe level prediction results.

[0080] Step S4 is the core of model inference, which can be implemented through steps S41 to S43, as detailed below: Step S41: Normalize the wrapping phase map so that its numerical distribution is consistent with the distribution of the simulation data used during model training.

[0081] Step S41 is a continuation and confirmation of the preprocessing, ensuring that the input values ​​are within the statistical range expected by the model.

[0082] Step S42: Input the normalized package phase map into the trained level prediction model and perform forward propagation operation.

[0083] The optimal model weights saved in step S2 are loaded, and the prepared input data is fed into the network. The network undergoes a series of forward computations, including encoding, global context fusion, and decoding.

[0084] Step S43: Perform the argmax operation on the channel dimension of the classification logits output by the model to obtain the integer stripe level at each pixel position.

[0085] The network's final output is a tensor of shape [H, W, K], representing the unnormalized score (logits) of each pixel belonging to K levels. By taking the index corresponding to the maximum value in the channel dimension (i.e., the K dimension) (the argmax operation), the model prediction can be transformed into a final integer stripe level map for each pixel. This is the most basic reasoning process.

[0086] To further improve the global consistency of prediction results during the inference stage, especially when dealing with challenging real-world scenarios, a preferred embodiment of this application introduces a process for iteratively optimizing the stripe-level prediction results, as described in steps S44 to S46.

[0087] Step S44: Perform iterative initialization by generating low-resolution initial predictions and geometric results.

[0088] Specifically, the argmax operation is performed on the 1 / 4 resolution level probability map output by the level prediction model to obtain the initial fringe level. At the same time, the wrapped phase map and the effective region mask are downsampled synchronously. The initial fringe level and the downsampled wrapped phase map are combined to perform phase unwrapping. Then, the initial disparity map is obtained by converting through calibration parameters and lookup table, thus completing the iterative initialization.

[0089] Step S45: Starting with iterative initialization, perform multiple rounds of iterative updates.

[0090] The aforementioned iterative update (e.g., 3 rounds) can specifically involve constructing disparity distribution features based on the current wrapping phase, current fringe level, and camera parameters, extracting spatial geometric constraint features through a geometric encoding module, and then feeding the geometric constraint features, the current disparity map, and intermediate network features into the update module to learn the level probability update. ,according to Update the order probability, upsample the updated low-resolution order features and fuse them with the backbone features to output the full-resolution prediction result for this round. Use the new stripe order to re-unwrap the phase and reconstruct the disparity map as the input for the next iteration.

[0091] Step S46: Repeat the iteration for a preset number of times. After the iteration is completed, upsample the initial low-resolution level features to the original image size and output the final stripe level prediction result. Will As a new The update module is then input again to calculate the new update field, and this process is repeated iteratively a preset number of times (e.g., 5-10 times). After the final iteration, argmax is applied to the final feature map to obtain the optimized level prediction result. This process simulates the iterative refinement in optimization algorithms, which can gradually correct local inconsistencies, making the prediction result more consistent with the periodicity constraint of the stripes globally, thereby obtaining a more accurate level map.

[0092] The iterative optimization process for the stripe level prediction results exemplified in steps S44 to S46 above mimics the process of gradually approaching the optimal solution in optimization algorithms by constructing a level update field and performing multiple rounds of iterative correction. This mechanism allows the model to self-correct potentially contradictory local regions during the inference phase using intermediate features and current predictions, thereby outputting a stripe level map with higher global consistency and fewer errors. This is particularly beneficial for handling difficulties such as occluded boundaries and noisy regions, significantly improving the final accuracy of 3D reconstruction. Step S5: Perform three-dimensional reconstruction based on the stripe order prediction results and the wrapped phase map.

[0093] Once a reliable fringe order is obtained, the three-dimensional shape of the object can be recovered according to the standard physical process of structured light three-dimensional measurement. Specifically, based on the fringe order prediction results and the enveloping phase map, three-dimensional reconstruction can be achieved through the following steps S51 to S53: Step S51: Combine the fringe level prediction results with the wrapping phase map pixel by pixel to calculate the absolute phase map.

[0094] According to the formula For each valid pixel, the wrapping phase value obtained in step S3 is... In addition to its corresponding prediction level This is multiplied by 2π, thus obtaining a globally continuous and unambiguous absolute phase value. This forms an absolute phase diagram.

[0095] Step S52: Based on the pre-calibrated system parameters, generate an initial depth map using the mapping relationship from absolute phase Φ to depth Z established during the calibration phase.

[0096] In the embodiments of this application, absolute phase The mapping to depth Z can be achieved using a high-resolution lookup table to support fast indexing, or it can be accurately described using polynomial function fitting. In practical applications, each phase value in the absolute phase map calculated in step S51 is used... By substituting the mapping relationship (by looking up a table or calculating a polynomial), the corresponding depth value Z can be obtained, thereby generating a two-dimensional initial depth map.

[0097] In another embodiment of this application, a parallax-based 3D reconstruction method can be employed to optimize parallel computing efficiency. In this method, the parallax plane originates from epipolar correction. Epipolar correction is the process of rotating and aligning the imaging / projection planes of the camera and projector. Traditional epipolar correction methods directly apply to the 2D image, obtaining the corrected coordinates (x', y') by querying the pixel coordinates (x, y) before correction. However, the projector itself does not generate 2D images and cannot be directly applied to this method. Therefore, this application utilizes a pre-calibrated lookup table relationship between phase and camera coordinates to directly obtain the projector's coordinates after epipolar correction, thereby avoiding the 2D image correction step for the projector in traditional methods. This makes the parallax calculation process more convenient and efficient, significantly improving the overall computational speed of 3D reconstruction.

[0098] Step S53: Filter the 3D point cloud obtained from the initial depth map transformation to remove isolated noise points.

[0099] Using the camera's intrinsic parameter matrix, each pixel (u, v, Z) in the depth map can be back-projected into 3D space to obtain a set of 3D point clouds. Due to measurement noise, order misjudgment, or surface specular highlights, there may be outlier noise points in the point cloud.

[0100] To verify the technical effect of this application, the following comparative experiment was conducted, and the experimental results are as follows: Figure 5 As shown. Figure 5 The comparison includes three sets of images: The first row shows a depth map comparison. The traditional algorithm's depth map on the left has blurry details and significant noise, while the depth map provided in this embodiment has clear details and good noise suppression. The blue areas in the right-hand comparison image represent regions with small depth errors. The second row shows a scene annotation comparison. The traditional algorithm's annotation boundaries on the left are blurry and the categories are confused, while the annotation boundaries provided in this embodiment are clear and the categories are accurate. The right-hand comparison image is highlighted with a blue background. The third row shows a point cloud comparison. The traditional algorithm's point cloud on the left is sparse and structurally distorted, while the point cloud provided in this embodiment is dense and structurally complete. The red areas on the right represent regions with high point cloud overlap and improved accuracy. This comparison visually demonstrates the significant advantages of the proposed method in complex real-world scenes from three dimensions: depth estimation, scene understanding, and point cloud reconstruction.

[0101] To obtain clean and usable point cloud data, post-processing filtering is required. An effective method is radius filtering, which is implemented through the following steps S531 and S532: Step S531: For each point in the 3D point cloud, count the number of neighboring points within a sphere of a specified radius around it.

[0102] For each point in the point cloud P Define a search radius r (For example, 2 mm), calculate the area falling within... P For the center of the ball, r The number of other points inside a sphere with radius . N .

[0103] Step S532: If the number of neighboring points of a certain point is lower than a preset threshold, then the point is determined to be a noise point and removed.

[0104] Set a threshold T (For example, 5 points). If points P Number of neighborhood points N < T Then the point is considered P It is isolated and likely noise; remove it from the point cloud. Keep all neighboring points with a count greater than or equal to [a certain value]. T The points are used to form the final 3D point cloud after denoising. This filtering method can effectively remove floating isolated points while preserving the main structure of the object's surface.

[0105] like Figure 5 As shown, this is a comparison chart (3 rows and 3 columns) of the deep reconstruction results based on deep learning proposed in the embodiments of this application, using the traditional algorithm. Row 1 is the depth map. The intersection of row 1 and column 1 indicates that the depth map of the "traditional algorithm" has local blurring / noise. The intersection of row 1 and column 2 indicates that the depth map of the "deep learning" has clearer details. The "comparison" at the intersection of row 1 and column 3 highlights the difference area with a blue background. Row 2 represents the depth error. The intersection of row 2 and column 1 indicates that the error of the "traditional algorithm" is concentrated at the edge (red / yellow heatmap). The intersection of row 2 and column 2 indicates that the error distribution of the "deep learning" is more uniform (green / yellow heatmap). The intersection of row 2 and column 3 indicates the quantization error range in the "comparison". Row 3 represents the point cloud. The intersection of row 3 and column 1 indicates that the point cloud of the "traditional algorithm" (blue) has missing / distorted parts. The intersection of row 3 and column 2 indicates that the point cloud of the "deep learning" (red) has higher integrity. The "comparison" at the intersection of row 3 and column 3 shows the red / white overlapping area (the overlapping part is high-precision reconstruction). The "Load Traditional Data", "Start Inference" and "View Comparison" buttons at the bottom support interactive verification, intuitively demonstrating the reconstruction advantages of this invention in complex scenarios.

[0106] Figure 1The example method can be applied to industrial automation scenarios; therefore, after step S5, the following steps S6 and S7 can also be included as examples of technical solutions: Step S6: Register the reconstructed 3D point cloud with the preset 3D model of the workpiece, and calculate the spatial pose of the workpiece.

[0107] In industrial automation applications, the ultimate goal of 3D reconstruction is to guide robot operations. This requires identifying and locating the target workpiece from the point cloud. A precise 3D CAD model of the workpiece is pre-stored in the system. Using a point cloud registration algorithm, such as the Iterative Closest Point (ICP) algorithm or its variants, the denoised point cloud obtained from real-time reconstruction is aligned with the workpiece's CAD model. By minimizing the distance error between the two point clouds, the rotation matrix R and translation vector T required to transform the model to the point cloud's location are calculated. This (R, T) represents the workpiece's spatial pose (including 3D position and orientation) in the robot's base coordinate system.

[0108] Step S7: Send the spatial pose to the robot control system to guide the robot to perform grasping or assembly operations. Finally, the calculated workpiece pose (R, T) is sent to the robot controller in real time via a communication interface (e.g., TCP / IP, EtherCAT, etc.). Based on this pose information, the robot controller plans the robot's motion trajectory and controls the end effector (e.g., suction cup, gripper) to move to the specified position and posture, completing the precise gripping, placement, or assembly operation of the workpiece, thereby realizing a fully automated production process.

[0109] From the above appendix Figure 1As demonstrated by the example of 3D reconstruction methods, by employing a structured light simulation environment built on a rendering engine and parameterizing and randomizing object materials, poses, and occlusion relationships, it is possible to realistically simulate the optical projection and imaging processes of a real structured light system, as well as various complex object surface properties and scene layouts, within a virtual environment. The automatically generated simulation dataset's data distribution more closely resembles the complexity of the real physical world, significantly reducing the domain difference between simulation and real data. This allows deep learning network models trained on this dataset to learn data features and patterns closer to real scenes, thereby greatly improving the model's generalization performance and practicality when facing unseen real objects and scenes, overcoming the dependence on expensive and difficult-to-obtain real labeled data. On the other hand, the hierarchical prediction model is used to capture the global periodic dependencies of an image. It is no longer limited to the visual features of local regions but can model long-range dependencies of features along the row and column directions of the image, thus understanding the periodic pattern and continuous variation of sinusoidal stripes throughout the image. This comprehensive approach... The utilization of local context information enables the fringe level prediction model to infer based on a wider range of image structure information when determining the fringe level. In particular, at fringe breaks, object edges, occluded areas, or noise interference, it can effectively suppress local misjudgments and maintain the overall consistency of level changes, thereby significantly improving the accuracy of level determination results and robustness to complex situations. Thirdly, the technical solution of this application organically combines physical simulation-driven data generation, global context-aware fringe level prediction model training, and preprocessing and inference processes for real data into a coherent method. After the model is trained on physically realistic simulation data, it can directly process the wrapping phase map acquired by the real structured light system and output accurate fringe level prediction results. Combining the level prediction results with the original wrapping phase map, subsequent 3D reconstruction can be completed stably and reliably. The entire process does not require tedious annotation or additional model fine-tuning for real data, realizing efficient and high-precision transfer from the simulation environment to real application scenarios, and providing a powerful solution for automated 3D measurement. In summary, the technical solution of this application effectively improves the accuracy and generalization performance of the model in determining stripe order in real-world scenarios through high-fidelity simulation and global context modeling.

[0110] Please see Figure 2 As shown, in one embodiment, a three-dimensional reconstruction device is provided, which may include a generation module 201, a training module 202, an acquisition module 203, a prediction module 204, and a reconstruction module 205, as detailed below: The generation module 201 is used to build a structured light simulation environment based on the rendering engine, simulate the optical projection and imaging process of the structured light system, and perform parameterized randomization on the object material, pose and occlusion relationship to automatically generate a simulation dataset including level annotation and depth annotation. Training module 202 is used to construct a hierarchical prediction model. The hierarchical prediction model is used to capture the global periodic dependencies of the image. Using a simulation dataset, the hierarchical prediction model is trained by a joint loss function consisting of hierarchical classification loss and depth reconstruction loss based on the physical principle of structured light, and the trained hierarchical prediction model is obtained. The acquisition module 203 is used to acquire the package phase map collected by the real structured light system; Prediction module 204 is used to input the package phase map into the trained level prediction model to obtain the stripe level prediction result; The reconstruction module 205 is used to perform three-dimensional reconstruction based on the stripe order prediction results and the wrapping phase map.

[0111] From the above appendix Figure 2As seen in the example of the 3D reconstruction device, on the one hand, by employing a technique that uses a rendering engine to construct a structured light simulation environment and parametrically randomizes the material, pose, and occlusion relationships of objects, it is possible to simulate the optical projection and imaging processes of a real structured light system, as well as various complex object surface properties and scene layouts, in a highly realistic virtual environment. The data distribution of the automatically generated simulation dataset is closer to the complexity of the real physical world, thus significantly reducing the domain difference between simulation data and real data. This allows the deep learning network model trained using this dataset to learn data features and patterns that are closer to real scenes, thereby greatly improving the model's generalization performance and practicality when facing unseen real objects and scenes, overcoming the dependence on expensive and difficult-to-obtain real labeled data. On the other hand, the hierarchical prediction model is used to capture the global periodic dependencies of the image. It is no longer limited to the visual features of local areas and can model long-range dependencies of features along the row and column directions of the image, thereby understanding the periodic pattern and continuous variation of sinusoidal stripes throughout the image. The utilization of global contextual information enables the order prediction model to infer based on a wider range of image structural information when determining the fringe order. In particular, at fringe breaks, object edges, occluded areas, or noise interference, it can effectively suppress local misjudgments and maintain the overall consistency of order changes, thereby significantly improving the accuracy of order determination results and robustness to complex situations. Thirdly, the technical solution of this application organically combines physical simulation-driven data generation, global context-aware order prediction model training, and preprocessing and inference processes for real data into a coherent method. After the model is trained on physically realistic simulation data, it can directly process the wrapping phase map acquired by the real structured light system and output accurate fringe order prediction results. Combining the order prediction results with the original wrapping phase map, subsequent 3D reconstruction can be completed stably and reliably. The entire process does not require tedious annotation or additional model fine-tuning for real data, realizing efficient and high-precision transfer from the simulation environment to real application scenarios, and providing a powerful solution for automated 3D measurement. In summary, the technical solution of this application effectively improves the accuracy and generalization performance of the model in determining stripe order in real-world scenarios through high-fidelity simulation and global context modeling.

[0112] In one embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 3As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a target stripe level determination method.

[0113] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the following steps: A structured light simulation environment is built based on the rendering engine to simulate the optical projection and imaging process of the structured light system. The material, pose and occlusion relationship of the object are parameterized and randomized to automatically generate a simulation dataset including level annotation and depth annotation. A hierarchical prediction model is constructed to capture the global periodic dependencies of an image. Using a simulation dataset, the hierarchical prediction model is trained by a joint loss function consisting of hierarchical classification loss and depth reconstruction loss based on the physical principle of structured light, and the trained hierarchical prediction model is obtained. Obtain the wrapping phase map acquired by a real structured light system; input the wrapping phase map into the trained order prediction model to obtain the stripe order prediction result; Three-dimensional reconstruction is performed based on the stripe order prediction results and the package phase map.

[0114] The aforementioned computer program effectively improves the accuracy and generalization performance of the model in determining stripe order in real-world scenarios through high-fidelity simulation and global context modeling.

[0115] In one embodiment, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, performs the following steps: A structured light simulation environment is built based on the rendering engine to simulate the optical projection and imaging process of the structured light system. The material, pose and occlusion relationship of the object are parameterized and randomized to automatically generate a simulation dataset including level annotation and depth annotation. A hierarchical prediction model is constructed to capture the global periodic dependencies of an image. Using a simulation dataset, the hierarchical prediction model is trained by a joint loss function consisting of hierarchical classification loss and depth reconstruction loss based on the physical principle of structured light, and the trained hierarchical prediction model is obtained. Obtain the wrapping phase map acquired by a real structured light system; input the wrapping phase map into the trained order prediction model to obtain the stripe order prediction result; Three-dimensional reconstruction is performed based on the stripe order prediction results and the package phase map.

[0116] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0119] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: A structured light simulation environment is built based on a rendering engine to simulate the optical projection and imaging process of a structured light system. The object's material, pose, and occlusion relationships are parameterized and randomized to automatically generate a simulation dataset including level and depth annotations. This process includes importing the intrinsic and extrinsic parameters of the camera and projector, as well as distortion coefficients from a real structured light system, into the rendering engine to accurately reproduce the optical projection and imaging process. In the rendering engine, at least one optical property parameter among diffuse reflection coefficient, specular reflection coefficient, metallicity, and roughness is defined for the simulated object. Based on the defined optical property parameters, the size, three-dimensional spatial pose, and mutual occlusion degree between multiple objects are randomly set in the structured light simulation environment. A multi-step phase-shift sinusoidal fringe pattern is projected onto the scene containing the simulated object, and the modulated deformed fringe image is acquired. Based on the absolute phase and depth information generated by the structured light simulation environment, the corresponding wrapping phase map, integer fringe order map, depth map, and effective region mask are automatically calculated and generated to constitute the simulation dataset. A hierarchical prediction model is constructed to capture the global periodic dependencies of an image. Using the simulation dataset, the hierarchical prediction model is trained using a joint loss function consisting of hierarchical classification loss and depth reconstruction loss based on structured light physics principles. The trained hierarchical prediction model is obtained by: constructing an encoder-decoder structure for the hierarchical prediction model, where the encoder contains multiple downsampling convolutional blocks to extract multi-scale features of the image; and performing sequence modeling operations along the row and column directions of the image on the end feature map of the encoder to capture the global periodic context information of the stripes, and outputting an output that incorporates the global periodicity. The model training process involves using a fused feature map of contextual information. During model training, based on the fused feature map, the stripe level classification cross-entropy loss for each pixel is calculated as the level classification loss. During model training, based on the fused feature map, the stripe level distribution predicted by the model is converted into a continuous soft level through differentiable operations. The predicted depth map is reconstructed using the wrapped phase map, and the mean squared error between the predicted depth map and the actual depth map is calculated as the depth reconstruction loss. The level classification loss and the depth reconstruction loss together constitute the joint loss function, used for backpropagation optimization of the level prediction model. The conversion of the stripe level distribution predicted by the model into a continuous soft level through differentiable operations includes: applying the Gumbel-Softmax function to the model output classification logits to obtain a smooth, differentiable level probability distribution; and weighted summing of the level probability distribution with an integer level value sequence to calculate continuous soft level values. To obtain a wrapping phase map acquired by a real structured light system, the following steps are taken: controlling a projector to project at least three different frequencies of sinusoidal fringe patterns onto the object under test, and simultaneously acquiring corresponding deformed fringe images by a camera; performing phase shift calculation on the deformed fringe images of each frequency group to obtain multiple wrapping phase maps of different frequencies. The wrapped phase map is input into the trained level prediction model to obtain the stripe level prediction result; Based on the stripe order prediction results and the package phase map, three-dimensional reconstruction is performed.

2. The three-dimensional reconstruction method according to claim 1, characterized in that, The step of inputting the wrapped phase map into the trained level prediction model to obtain the stripe level prediction result includes: The package phase map is normalized to make its numerical distribution consistent with the distribution of the simulation data used during model training. The normalized package phase map is input into the trained level prediction model, and forward propagation is performed. The classification logits output by the model are subjected to an argmax operation in the channel dimension to obtain the integer stripe levels at each pixel position.

3. The three-dimensional reconstruction method according to claim 1, characterized in that, The step of training the hierarchical prediction model using the simulation dataset through a joint loss function consisting of hierarchical classification loss and depth reconstruction loss based on the physical principles of structured light, to obtain the trained hierarchical prediction model, includes: Model parameter updates are performed using the AdamW optimizer with weight decay. Configure a cosine annealing learning rate scheduler for the AdamW optimizer, which will decay the learning rate from its initial value to its minimum value according to a cosine law during training.

4. The three-dimensional reconstruction method according to claim 2, characterized in that, The step of inputting the wrapped phase map into the trained level prediction model to obtain the stripe level prediction result also includes the following process of iteratively optimizing the stripe level prediction result: Iterative initialization is performed by generating low-resolution initial predictions and geometric results; Starting with iterative initialization, multiple rounds of iterative updates are performed in a loop; Repeat the iteration a preset number of times. After the iteration is completed, upsample the initial low-resolution level features to the original image size and output the final stripe level prediction result.

5. The three-dimensional reconstruction method according to claim 1, characterized in that, The step of performing sequence modeling operations along the row and column directions of the image on the end feature map of the encoder to capture the global periodic context information of the stripes and outputting a fused feature map that incorporates the global periodic context information includes: performing sequence modeling operations along the row and column directions of the image on multiple levels of the encoder to capture the global periodic context information of the stripes at each level, and fusing the global context information obtained at each level with the local features of the corresponding level to obtain multiple multi-scale feature maps that incorporate the global context information.

6. A three-dimensional reconstruction apparatus, the apparatus comprising: The generation module is used to construct a structured light simulation environment based on the rendering engine, simulate the optical projection and imaging process of a structured light system, and parametrically randomize the object's material, pose, and occlusion relationships to automatically generate a simulation dataset including level and depth annotations. This includes importing the intrinsic and extrinsic parameters and distortion coefficients of the camera and projector in a real structured light system into the rendering engine to accurately reproduce the optical projection and imaging process. The imaging process involves defining at least one optical property parameter among diffuse reflection coefficient, specular reflection coefficient, metallicity, and roughness for the simulated object in the rendering engine. Based on the defined optical property parameters, the object's size, three-dimensional spatial pose, and mutual occlusion degree between multiple objects are randomly set in the structured light simulation environment. A multi-step phase-shift sinusoidal fringe pattern is projected onto the scene containing the simulated object, and the modulated deformed fringe image is acquired. Based on the absolute phase and depth information generated by the structured light simulation environment, the corresponding wrapping phase map, integer fringe order map, depth map, and effective region mask are automatically calculated and generated to constitute the simulation dataset. A training module is used to construct a hierarchical prediction model, which captures the global periodic dependencies of an image. Using the simulation dataset, the hierarchical prediction model is trained using a joint loss function consisting of hierarchical classification loss and depth reconstruction loss based on structured light physics principles. The trained hierarchical prediction model is then obtained. This training includes: constructing an encoder-decoder structure for the hierarchical prediction model, where the encoder contains multiple downsampled convolutional blocks to extract multi-scale features of the image; and performing sequence modeling operations along the row and column directions of the image on the end feature map of the encoder to capture the global periodic context information of the stripes, and outputting a fused version of the global periodic context information. A fusion feature map of local periodic context information is generated. During model training, based on the fusion feature map, the stripe level classification cross-entropy loss for each pixel is calculated as the level classification loss. During model training, based on the fusion feature map, the stripe level distribution predicted by the model is converted into a continuous soft level through differentiable operations. The predicted depth map is reconstructed by combining the wrapped phase map, and the mean square error between the predicted depth map and the true depth map is calculated as the depth reconstruction loss. The level classification loss and the depth reconstruction loss together constitute the joint loss function, which is used to optimize the level prediction model through backpropagation. The step of converting the stripe level distribution predicted by the model into a continuous soft level through differentiable operations includes: applying the Gumbel-Softmax function to the classification logits output by the model to obtain a smooth, differentiable level probability distribution; and weighting the level probability distribution with the integer level value sequence to calculate the continuous soft level value. The acquisition module is used to acquire the wrapping phase map collected by the real structured light system, including: controlling the projector to project at least three different frequency sinusoidal fringe patterns onto the object under test, and the camera synchronously acquiring the corresponding deformed fringe images, performing phase shift calculation on the deformed fringe images of each frequency to obtain multiple wrapping phase maps of different frequencies. The prediction module is used to input the package phase map into the trained level prediction model to obtain the stripe level prediction result; The reconstruction module is used to perform three-dimensional reconstruction based on the stripe level prediction results and the package phase map.

7. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the three-dimensional reconstruction method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional reconstruction method as described in any one of claims 1 to 5.

Images