A binocular speckle scene reconstruction method based on Gaussian splash

Abstract

The application discloses a binocular speckle scene reconstruction method based on Gaussian splatting, and relates to the technical fields of computer vision and three-dimensional reconstruction, which comprises the following steps: collecting binocular speckle structured light data and generating initial point clouds; converting the obtained initial point clouds into three-dimensional Gaussian sets, initializing corresponding three-dimensional Gaussian primitives and encoding; rendering speckle images under each view angle by using a differentiable Gaussian splatting method, comparing the speckle images with real speckle images, constructing a joint optimization target to optimize the Gaussian primitives, and supplementing the three-dimensional structure of the binocular matching failure area; establishing a gray appearance parameter for each Gaussian primitive, optimizing the gray appearance parameter by using multi-view gray images, and obtaining the completed three-dimensional scene reconstruction and rendering result. By using the above method, the problems of point cloud holes, local structure loss and insufficient rendering expression quality of the existing binocular speckle structured light three-dimensional scene reconstruction method under a complex actual environment light scene are solved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, and in particular to a binocular speckle scene reconstruction method based on Gaussian splashing. Background Technology

[0002] In several important fields such as robotics, surveying, 3D scanning, and 3D printing, binocular speckle scene reconstruction methods have been widely used commercially due to their advantages such as active texture projection, non-contact measurement, and high reconstruction accuracy. This method typically enhances the surface texture of the target through speckle projection and recovers the 3D point cloud using the parallax relationship between the left and right cameras.

[0003] However, existing binocular speckle structured light methods are highly dependent on the shared viewing area of ​​the left and right cameras and the quality of the speckle image. When the target surface is disturbed by external ambient light, the speckle contrast decreases. Furthermore, when the surface is obstructed, reflected, or in areas with weak texture, binocular matching is prone to failure, leading to holes, breaks, and missing local structures in the point cloud. Traditional point cloud completion methods often rely on interpolation, filtering, surface fitting, or mesh repair, making it difficult to fully utilize the contour, grayscale, and appearance information in multi-view images, easily resulting in inaccurate boundaries and loss of detail.

[0004] In recent years, differentiable 3D representation methods such as 3D Gaussian splashing have shown great potential in 3D reconstruction and new perspective rendering by utilizing multi-view image supervision to optimize scene representation. However, existing methods typically rely on relatively complete initial point clouds and stable image appearance information. When the initial point cloud contains holes caused by binocular matching failures, the Gaussian model tends to inherit the geometric defects of the original point cloud, making it difficult to effectively fill in the missing areas. Furthermore, in binocular speckle structured light scenes, the speckle image mainly provides geometric constraints, while the multi-view grayscale image mainly provides appearance information. Since their imaging conditions differ, directly mixing and optimizing them can easily lead to interference between geometric reconstruction and appearance rendering. Therefore, how to combine binocular speckle initial point clouds, multi-view speckle images, and multi-view grayscale images to achieve point cloud completion, geometric accuracy preservation, and high-quality grayscale rendering is a problem urgently needing to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a binocular speckle scene reconstruction method based on Gaussian splashing, so as to solve the problems of point cloud holes, local structure loss and insufficient rendering quality that existing binocular speckle structured light 3D scene reconstruction methods are prone to in complex real-world ambient light scenes.

[0006] To achieve the above objectives, on the one hand, the present invention provides a binocular speckle scene reconstruction method based on Gaussian splashing, comprising the following steps: S1. Acquire binocular speckle structured light data and generate an initial point cloud; S2. Convert the obtained initial point cloud into a 3D Gaussian set, initialize the corresponding 3D Gaussian primitives, and encode the Gaussian primitives. S3. Differentiable Gaussian splashing method is used to render speckle images from various perspectives and compare them with real speckle images. A joint optimization objective is constructed to optimize Gaussian elements and supplement the three-dimensional structure of the binocular matching failure area. S4. Establish grayscale appearance parameters for each Gaussian element that are independent of the speckle appearance and shared across multiple viewpoints. Optimize the grayscale appearance parameters using multi-view grayscale images to obtain the completed 3D scene reconstruction and rendering results.

[0007] Preferably, S1 specifically includes: S11. Calibrate the left and right cameras and projection units in the data acquisition system to obtain the camera intrinsic parameters, extrinsic parameters, distortion parameters and binocular baseline relationship, and acquire speckle images and grayscale images of the target scene and their poses from multiple perspectives. S12. Based on the pre-calibrated binocular intrinsic parameters, extrinsic parameters and distortion parameters, perform epipolar correction and binocular matching on the left and right speckle images to obtain local depth maps or point clouds under each viewpoint. S13. By using multi-view pose estimation and point cloud registration methods, local point clouds from different viewpoints are fused into an initial 3D point cloud.

[0008] Preferably, initializing the corresponding three-dimensional Gaussian primitives specifically includes: For the first point cloud The 3D point, the th The first perspective Each Gaussian element is represented as: ; in, Indicates the center position of the Gaussian unit. Indicates the scale parameter. These three parameters—rotation parameters, rotation parameters, and rotation parameters—together describe the spatial geometric distribution of Gaussian elements. Indicates opacity, used to control the contribution weight of Gaussian elements to the rendering result. Indicates the first The first Gaussian Yuan in the The speckle color parameters from different perspectives.

[0009] Preferably, the Gaussian meta-coding employs a multi-viewpoint shared geometric parameter encoding method, while the speckle color parameters are encoded independently for each viewpoint. , , and Shared across different viewpoints, with each viewpoint having its corresponding speckle appearance parameters set and optimized separately. .

[0010] Preferably, S3 specifically includes: S31. Differentiable Gaussian splashing is used to render speckle images from various viewpoints, and the results are compared with the corresponding real speckle images. A joint optimization objective is obtained by integrating speckle image reconstruction, depth distribution, surface normal, multi-view local correlation, and Gaussian position stability. ; in, This represents the loss in speckle image reconstruction. This represents the depth distortion regularization term. This represents the loss of surface normal consistency. This represents the loss of local correlation consistency. This indicates that the Gaussian reference position is maintained at a loss. The weights represent the speckle image reconstruction loss. This represents the weight of the depth distortion regularization term. This represents the surface normal consistency loss weight. This represents the weight of the local correlation consistency loss. This indicates that the Gaussian reference position retains the loss weights; The position holding loss is expressed as: ; ; in, This indicates the initial measurement position of the static Gaussian. This represents the position during dynamic Gaussian generation. Indicates the number of Gaussian elements. Indicates candidate Gaussian, This represents the static Gaussian set, which is the Gaussian set obtained by directly initializing the initial point cloud. This represents the dynamic Gaussian element set, which is the set of Gaussian elements newly added during the optimization process through Gaussian compaction. This indicates the reference position of the i-th Gaussian element; S32. Define the Gaussian generated from the initial point cloud as a static Gaussian and the Gaussian added through densification during the optimization process as a dynamic Gaussian. For a static Gaussian, use its initial measurement position as the reference position, and for a dynamic Gaussian, use its position when it is generated as the reference position. S33. For regions in the real speckle image that have valid observations but are not covered by the initial Gaussian, select candidate Gaussians to be projected onto these regions. The decision to perform densification is based on the speckle image reconstruction error, Gaussian position gradient, and multi-view visibility information. S34. After the densification condition is satisfied, a new dynamic Gaussian is generated by cloning or splitting, and its position, scale, rotation, opacity and speckle appearance parameters are updated under the joint optimization objective constraint, thereby supplementing the three-dimensional structure of the binocular matching failure area.

[0011] The preferred densification conditions can be summarized as follows: ; in, Indicates the first Pixels from a single perspective Does it belong to a region with effective speckle observations but not covered by the initial Gaussian? This indicates the speckle reconstruction error; Indicates candidate Gaussian Magnitude of position gradient; Indicates the effective number of views of the Gaussian; This represents the threshold for speckle reconstruction error; Indicates the Gaussian position gradient threshold; This represents the threshold for the number of times the image is visible.

[0012] Preferably, S4 specifically includes: S41. Fix the center position, scale, rotation, and opacity of the Gaussian elements, and establish grayscale appearance parameters for each Gaussian element that are independent of the speckle appearance and shared across multiple viewpoints. The grayscale rendering result from each viewpoint is represented as follows: ; in, This represents the rendering weights calculated with fixed geometric parameters. Indicates the first The grayscale appearance parameters shared by each Gaussian element across all viewpoints. Indicates the number of Gaussian elements; S42. Optimize grayscale appearance parameters using the multi-view grayscale images acquired in step S11. The optimization process is as follows: ; in, A set of grayscale appearance parameters shared from multiple perspectives. For the first A true grayscale image from a single perspective. For grayscale image reconstruction loss, For the number of viewpoints, To fix the geometric parameters, Indicates the first The grayscale image is obtained by rendering from a single viewpoint.

[0013] Preferably, the center position, scale, rotation, and opacity of the Gaussian elements are not updated during the optimization of grayscale appearance parameters.

[0014] Therefore, the binocular speckle scene reconstruction method based on Gaussian splashing using the above-described structure of the present invention has the following beneficial effects: (1) Improve point cloud integrity: This invention utilizes multi-view speckle image supervision for three-dimensional Gaussian optimization and generates new Gaussian through Gaussian densification, which can fill the point cloud holes caused by binocular speckle matching failure, occlusion or ambient light interference.

[0015] (2) Maintaining the geometric accuracy of the main region: The present invention uses the Gaussian corresponding to the initial point cloud as the static Gaussian and applies geometric anchoring constraints to keep it near the initial measurement position during the optimization process, thereby avoiding significant geometric drift in the main region.

[0016] (3) Improve the quality of grayscale appearance rendering: After completing the geometric completion under the supervision of speckle image, the present invention further fixes the Gaussian geometric parameters and optimizes the grayscale appearance parameters separately using multi-view grayscale images, thereby decoupling speckle geometric reconstruction and grayscale appearance rendering, and obtaining more stable and clear grayscale rendering results.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the workflow of a binocular speckle scene reconstruction method based on Gaussian splashing according to the present invention. Figure 2 This is a schematic diagram of the structure of a binocular speckle scene reconstruction system based on Gaussian splashing according to the present invention; Figure 3 This is a comparison of input and output point clouds in one embodiment of the present invention, where (a) and (c) are point clouds with holes obtained by traditional binocular speckle matching, and (b) and (d) are complete point clouds optimized by Gaussian splashing. Figure 4 This is a rendering quality change curve during the grayscale appearance optimization stage in one embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] like Figure 1As shown, this invention provides a binocular speckle scene reconstruction method based on Gaussian splashing, comprising: S1. Acquire binocular speckle structured light data and generate an initial point cloud. Specifically, this includes: S11. Calibrate the left and right cameras and projection units in the data acquisition module to obtain the camera intrinsic parameters, extrinsic parameters, distortion parameters and binocular baseline relationship, and acquire speckle images and grayscale images of the target scene and their poses from multiple perspectives.

[0021] S12. Based on the pre-calibrated binocular intrinsic parameters, extrinsic parameters, and distortion parameters, epipolar correction and binocular matching are performed on the left and right speckle images to obtain local depth maps or point clouds under each viewpoint.

[0022] S13. By using multi-view pose estimation and point cloud registration methods, local point clouds from different viewpoints are fused into an initial 3D point cloud.

[0023] S2. Convert the obtained initial point cloud into a 3D Gaussian set, initialize the corresponding 3D Gaussian primitives, and encode the Gaussian primitives. Specifically, this includes: The obtained initial point cloud is converted into a 3D Gaussian set, and the first Gaussian point in the initial point cloud is... Each 3D point initializes a corresponding 3D Gaussian element, and the 3D point initializes a corresponding 3D Gaussian element. The first perspective Each Gaussian element is represented as: ; in, Indicates the center position of the Gaussian unit. Indicates the scale parameter. These three parameters—rotation parameters, rotation parameters, and rotation parameters—together describe the spatial geometric distribution of Gaussian elements. Indicates opacity, used to control the contribution weight of Gaussian elements to the rendering result. Indicates the first The first Gaussian Yuan in the The speckle color parameters from different perspectives.

[0024] Gaussian meta-coding employs a multi-viewpoint sharing of geometric parameters and a viewpoint-specific independent encoding method for speckle color parameters. , , and Shared across different viewpoints, with each viewpoint having its corresponding speckle appearance parameters set and optimized separately. Therefore, different viewpoints jointly constrain the same 3D structure, while the inconsistent speckle brightness and texture variations between viewpoints are expressed by independent speckle color parameters, thereby reducing the interference of inconsistent speckle appearance on geometric optimization.

[0025] This invention proposes an encoding method for binocular speckle in Gaussian splashing. Gaussian geometric parameters are shared across multiple views to represent a unified 3D spatial structure; speckle appearance parameters are independently encoded to describe speckle responses under different views, thereby reducing the impact of inconsistent speckle textures across multiple views on geometric optimization. This mechanism is fundamental to enabling binocular speckle images to participate in differentiable optimization within the Gaussian splashing framework. Compared to traditional binocular speckle reconstruction methods, this invention does not rely solely on matching left and right camera images to generate point clouds. Instead, when holes exist in the initial point cloud, it further utilizes multi-view speckle images to supervise 3D Gaussian optimization, filling in areas where binocular matching fails.

[0026] S3. A differentiable Gaussian splashing method is used to render speckle images from various viewpoints, and these images are compared with real speckle images. A joint optimization objective is constructed to optimize the Gaussian primitives, supplementing the 3D structure of regions where binocular matching fails. Specifically, this includes: S31. Differentiable Gaussian splashing is used to render speckle images from various viewpoints, and the results are compared with the corresponding real speckle images. A joint optimization objective is obtained by integrating speckle image reconstruction, depth distribution, surface normal, multi-view local correlation, and Gaussian position stability. ; in, This represents the loss in speckle image reconstruction. This represents the depth distortion regularization term. This represents the loss of surface normal consistency. This represents the loss of local correlation consistency. This indicates that the Gaussian reference position is maintained at a loss. The weights represent the speckle image reconstruction loss. This represents the weight of the depth distortion regularization term. This represents the surface normal consistency loss weight. This represents the weight of the local correlation consistency loss. This indicates that the Gaussian reference position maintains the loss weight.

[0027] The speckle image reconstruction loss is expressed as: ; In the formula, V represents the number of viewpoints. For the first Rendering speckle images from multiple perspectives For true speckle images, This is the balance coefficient between the pixel reconstruction term and the structural similarity term.

[0028] The depth distortion regularization term is expressed as follows: ; In the formula, For the first Gauss in the th... Pixels from various perspectives Rendering weights; Indicates the first The rendering weights corresponding to each Gaussian; Indicates the first The depth of a Gaussian under the first perspective; Indicates the first Gauss in the th... Depth from a single perspective; Indicates pixel position; Indicates the first Effective pixel area from each viewpoint; This indicates the number of pixels in the valid pixel area.

[0029] The surface normal consistency loss is expressed as: ; In the formula, Indicates the first Pixels from a single perspective The surface normal obtained by Gaussian rendering; Indicates the first Pixels from a single perspective The surface normal is calculated from the rendering depth.

[0030] The local correlation consistency loss is expressed as: ; In the formula, Indicates reference perspective Pixels A local speckle image patch centered on the image; Indicates adjacent viewpoints The corresponding local speckle image patch; Indicates the reprojection of the reference viewpoint pixel to the first... The corresponding position after each perspective; Indicates reference perspective and adjacent perspectives The perspective that constitutes the composition; This represents a set of perspectives involved in locally related computations. Indicates perspective versus quantity; Indicates reference perspective and adjacent perspectives The effective corresponding pixel areas between them; Indicates the number of valid corresponding pixels; This indicates normalized cross-correlation.

[0031] The position holding loss is expressed as: ; ; in, This indicates the initial measurement position of the static Gaussian. This represents the position during dynamic Gaussian generation. Indicates the number of Gaussian elements. Indicates candidate Gaussian, This represents the static Gaussian set, which is the set of Gaussian elements obtained by directly initializing the initial point cloud; This represents a dynamic Gaussian set, which is the set of Gaussian elements newly added during the optimization process through Gaussian compaction. Indicates the first The reference position of a Gaussian element. For a static Gaussian, the reference position is its initial measurement position. For dynamic Gaussians, the reference position is the position at which they were generated. This constraint serves both to maintain the geometric accuracy of the original measurement region and to suppress excessive drift of the newly added dynamic Gaussian during the optimization process.

[0032] S32. Define the Gaussian generated from the initial point cloud as a static Gaussian, and define the Gaussian added during the optimization process through densification as a dynamic Gaussian. For a static Gaussian, use its initial measurement position as the reference position, and for a dynamic Gaussian, use its position when it is generated as the reference position.

[0033] S33. For regions in the real speckle image that have valid observations but are not covered by the initial Gaussian, select candidate Gaussians to be projected onto these regions. The decision to perform densification is based on the speckle image reconstruction error, Gaussian position gradient, and multi-view visibility information. The densification conditions are summarized as follows: ; in, Indicates the first Pixels from a single perspective Does it belong to a region with effective speckle observations but not covered by the initial Gaussian? This indicates the speckle reconstruction error; Indicates candidate Gaussian Large positional gradient; Indicates the effective number of views of the Gaussian; This represents the speckle reconstruction error threshold, used to determine whether the speckle rendering error in the current region is large enough. Only when the error exceeds this threshold will densification be considered. This represents the Gaussian position gradient threshold, used to determine whether the position gradient of a candidate Gaussian is large enough, reflecting whether the current geometry still needs further adjustment or refinement; This represents the effective visibility threshold, used to ensure that candidate Gaussians have effective observations in a sufficient number of viewpoints, avoiding erroneous densification triggered by noise from a single viewpoint.

[0034] S34. After the densification condition is satisfied, a new dynamic Gaussian is generated by cloning or splitting, and its position, scale, rotation, opacity and speckle appearance parameters are updated under the joint optimization objective constraint, thereby supplementing the three-dimensional structure of the binocular matching failure area.

[0035] This invention employs a static-dynamic Gaussian co-optimization method. The Gaussian corresponding to the initial point cloud is used as the static Gaussian, and geometric anchoring constraints are applied to maintain the original measurement accuracy of the main region. The Gaussian added during the optimization process is used as the dynamic Gaussian, optimized under multi-view speckle image supervision to fill in missing areas of the initial point cloud. Compared to ordinary 3D Gaussian splashing methods, this invention uses binocular speckle structured light point cloud as the initial geometric prior and maintains the geometric accuracy of the original measurement region through static Gaussian anchoring constraints, thus avoiding significant drift of the main region during the optimization process.

[0036] like Figure 3 As shown in (a) and (c), the point cloud reconstructed by binocular speckle under ambient light interference has large holes. As shown in (b) and (d) in the figure, introducing Gaussian splashing into structured light binocular speckle can generate new Gaussian sputtering in the binocular speckle matching failure area, thereby improving the integrity of the point cloud.

[0037] As shown in Table 1, within the main area covered by both the input and output point clouds, the average error of the optimized point cloud decreased from 0.9343 mm to 0.9199 mm, the standard deviation decreased from 0.8591 mm to 0.8523 mm, and the RMSE decreased from 1.2693 mm to 1.2541 mm. This indicates that the proposed method, while performing Gaussian enrichment and point cloud completion, does not compromise the geometric accuracy of the original binocular speckle reconstruction region and can improve the geometric consistency of the main area to some extent. Furthermore, the overall average error of the output point cloud, including the main area and the hole completion area, is 0.9541 mm, which is close to the 0.9343 mm of the input point cloud. This suggests that although the newly added completed point cloud introduces errors in more difficult areas, the overall geometric accuracy remains within a stable range.

[0038] Table 1 Error Statistics

[0039] Wherein, input point cloud: represents the initial input point cloud generated by binocular speckle structured light; main region input point cloud: represents the main region jointly covered by the input point cloud and the optimized point cloud; hole area output point cloud: represents the hole area point cloud obtained by this method after the initial point cloud was missing; overall output point cloud: represents the entire output point cloud after Gaussian optimization and completion. The error unit is mm.

[0040] S4. Establish grayscale appearance parameters for each Gaussian primitive that are independent of the speckle appearance but shared across multiple viewpoints. Optimize these grayscale appearance parameters using multi-view grayscale images to obtain the completed 3D scene reconstruction and rendering results. Specifically, this includes: S41. After completing Gaussian compaction and obtaining a stable geometry, fix the center position, scale, rotation, and opacity of the Gaussian elements. Establish grayscale appearance parameters for each Gaussian element that are independent of the speckle appearance and shared across multiple viewpoints. The grayscale rendering result from each viewpoint is represented as follows: ; in, This represents the rendering weights calculated with fixed geometric parameters. Indicates the first The grayscale appearance parameters shared by each Gaussian element across all viewpoints. Indicates the number of Gaussian elements; S42. Optimize grayscale appearance parameters using the multi-view grayscale images acquired in step S11. The optimization process is as follows: ; in, A set of grayscale appearance parameters shared from multiple perspectives. For the first A true grayscale image from a single perspective. For grayscale image reconstruction loss, For the number of viewpoints, To fix the geometric parameters, Indicates the first The grayscale image is rendered from multiple perspectives. In this stage, only the shared grayscale appearance parameters are updated, and the center position, scale, rotation, and opacity of the Gaussian elements are no longer updated. This enables directional supervision of geometric completion by speckle observation and independent optimization of appearance expression by grayscale observation, and avoids the grayscale appearance loss from inversely altering the obtained geometric structure.

[0041] This invention proposes a grayscale appearance optimization method under fixed geometric conditions: after speckle geometric completion, the geometric parameters of Gaussian primitives are fixed, and grayscale color parameters are optimized only using multi-view grayscale images. This avoids interference from speckle texture residues on the rendering results, achieving phased decoupling between speckle geometric reconstruction and grayscale appearance rendering, improving the consistency and clarity of the final grayscale rendering result. Compared with traditional point cloud completion methods such as point cloud interpolation, surface fitting, or mesh repair, this invention does not simply fill points based on geometric rules, but uses differentiable rendering errors to guide Gaussian compaction using multi-view speckle image information, making the completion result subject to both 3D geometric priors and 2D image observation constraints. Compared with traditional texture mapping or simple grayscale rendering methods, this invention first uses speckle images to complete geometric completion, then fixes the geometric parameters and uses multi-view grayscale images to optimize appearance parameters, achieving decoupling between geometric reconstruction and appearance rendering. Figure 4 As shown, with the iteration of grayscale appearance optimization, the PSNR of the rendered image continues to improve and eventually reaches about 32dB, indicating that the present invention can obtain a relatively stable, clear and grayscale consistent 3D scene rendering result.

[0042] like Figure 2 As shown, the present invention also provides a binocular speckle scene reconstruction system based on Gaussian splashing, comprising: The data acquisition module is used to acquire speckle images, grayscale images, and poses of the target scene from multiple perspectives.

[0043] The initial point cloud generation module is used to perform epipolar correction and binocular matching on the left and right speckle images based on the pre-calibrated binocular intrinsic parameters, extrinsic parameters and distortion parameters to obtain local depth maps or point clouds under each view. Then, through multi-view pose estimation and point cloud registration methods, the local point clouds under different viewpoints are fused into an initial 3D point cloud.

[0044] The Gaussian initialization module is used to convert the initial point cloud into a 3D Gaussian set, initializing a corresponding 3D Gaussian primitive for each 3D point in the initial point cloud.

[0045] The Gaussian coding module is used to encode the three-dimensional Gaussian primitives in the three-dimensional Gaussian set using a coding method that shares geometric parameters across multiple views and allows speckle color parameters to be encoded independently for each view.

[0046] The speckle optimization module is used to render speckle images of the 3D Gaussian set using the differentiable Gaussian splashing method at various viewpoints and compare them with the real speckle image to construct a joint optimization objective for optimization. During the optimization process, the Gaussian corresponding to the initial point cloud is defined as static Gaussian and geometric anchoring constraints are applied. The Gaussian added during the optimization process through densification is defined as dynamic Gaussian. Based on the speckle image reconstruction error, Gaussian position gradient, and multi-view visibility information, it is determined whether to perform densification to supplement the 3D structure of the area where the binocular matching failed.

[0047] The grayscale appearance optimization module is used to establish grayscale appearance parameters for each Gaussian element that are independent of the speckle appearance and shared across multiple viewpoints. It then uses multi-view grayscale images to optimize the grayscale appearance parameters, thereby obtaining the 3D scene reconstruction and rendering results.

[0048] This invention uses the initial point cloud obtained by binocular speckle structured light as a geometric prior through the above reconstruction framework, introduces a three-dimensional Gaussian splash representation, completes geometric optimization and point cloud completion through multi-view speckle image supervision, and optimizes appearance rendering by multi-view grayscale images after geometric stabilization, thus realizing a complete reconstruction process from "speckle geometric completion" to "grayscale appearance expression".

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A binocular speckle scene reconstruction method based on Gaussian splashing, characterized in that, Includes the following steps: S1. Acquire binocular speckle structured light data and generate an initial point cloud; S2. Convert the obtained initial point cloud into a 3D Gaussian set, initialize the corresponding 3D Gaussian primitives, and encode the Gaussian primitives. S3. Differentiable Gaussian splashing method is used to render speckle images from various perspectives and compare them with real speckle images. A joint optimization objective is constructed to optimize Gaussian elements and supplement the three-dimensional structure of the binocular matching failure area. S4. Establish grayscale appearance parameters for each Gaussian element that are independent of the speckle appearance and shared across multiple viewpoints. Optimize the grayscale appearance parameters using multi-view grayscale images to obtain the completed 3D scene reconstruction and rendering results.

2. The binocular speckle scene reconstruction method based on Gaussian splashing according to claim 1, characterized in that, S1 specifically includes: S11. Calibrate the left and right cameras and projection unit in the data acquisition module to obtain the camera intrinsic parameters, extrinsic parameters, distortion parameters and binocular baseline relationship, and acquire speckle images and grayscale images of the target scene and their poses from multiple perspectives. S12. Based on the pre-calibrated binocular intrinsic parameters, extrinsic parameters and distortion parameters, perform epipolar correction and binocular matching on the left and right speckle images to obtain local depth maps or point clouds under each viewpoint. S13. By using multi-view pose estimation and point cloud registration methods, local point clouds from different viewpoints are fused into an initial 3D point cloud.

3. The binocular speckle scene reconstruction method based on Gaussian splashing according to claim 1, characterized in that, Initializing the corresponding 3D Gaussian primitives specifically includes: For the first point cloud The 3D point, the th The first perspective Each Gaussian element is represented as: ； in, Indicates the center position of the Gaussian unit. Indicates the scale parameter. These three parameters—rotation parameters, rotation parameters, and rotation parameters—together describe the spatial geometric distribution of Gaussian elements. Indicates opacity, used to control the contribution weight of Gaussian elements to the rendering result. Indicates the first The first Gaussian Yuan in the The speckle color parameters from different perspectives.

4. The binocular speckle scene reconstruction method based on Gaussian splashing according to claim 3, characterized in that, Gaussian meta-coding employs a multi-viewpoint sharing of geometric parameters and a viewpoint-specific independent encoding method for speckle color parameters. , , and Shared across different viewpoints, with each viewpoint having its corresponding speckle appearance parameters set and optimized separately. .

5. The binocular speckle scene reconstruction method based on Gaussian splashing according to claim 1, characterized in that, S3 specifically includes: S31. Differentiable Gaussian splashing is used to render speckle images from various viewpoints, and the results are compared with the corresponding real speckle images. A joint optimization objective is obtained by integrating speckle image reconstruction, depth distribution, surface normal, multi-view local correlation, and Gaussian position stability. ； in, This represents the loss in speckle image reconstruction. This represents the depth distortion regularization term. This represents the loss of surface normal consistency. This represents the loss of local correlation consistency. This indicates that the Gaussian reference position is maintained at a loss. The weights represent the speckle image reconstruction loss. This represents the weight of the depth distortion regularization term. This represents the surface normal consistency loss weight. This represents the weight of the local correlation consistency loss. This indicates that the Gaussian reference position retains the loss weights; The position holding loss is expressed as: ； ； in, Indicates the initial measurement position of the static Gaussian; This represents the position during dynamic Gaussian generation; Indicates the number of Gaussian elements; Indicates a candidate Gaussian; Represents a static Gaussian set. Represents a dynamic Gaussian set. Indicates the first The reference position of each Gaussian element; S32. Define the Gaussian generated from the initial point cloud as a static Gaussian and the Gaussian added through densification during the optimization process as a dynamic Gaussian. For a static Gaussian, use its initial measurement position as the reference position, and for a dynamic Gaussian, use its position when it is generated as the reference position. S33. For regions in the real speckle image that have valid observations but are not covered by the initial Gaussian, select candidate Gaussians to be projected onto these regions. The decision to perform densification is based on the speckle image reconstruction error, Gaussian position gradient, and multi-view visibility information. S34. After the densification condition is satisfied, a new dynamic Gaussian is generated by cloning or splitting, and its position, scale, rotation, opacity and speckle appearance parameters are updated under the joint optimization objective constraint, thereby supplementing the three-dimensional structure of the binocular matching failure area.

6. The binocular speckle scene reconstruction method based on Gaussian splashing according to claim 5, characterized in that: The conditions for densification can be summarized as follows: ； in, Indicates the first Pixels from a single perspective Does it belong to a region with valid speckle observations but not covered by the initial Gaussian? This indicates the speckle reconstruction error. Indicates candidate Gaussian Magnitude of position gradient This represents the effective number of times Gaussian is visible. This represents the threshold for speckle reconstruction error; Indicates the Gaussian position gradient threshold; This represents the threshold for the number of times the image is visible.

7. The binocular speckle scene reconstruction method based on Gaussian splashing according to claim 2, characterized in that, S4 specifically includes: S41. Fix the center position, scale, rotation, and opacity of the Gaussian elements, and establish grayscale appearance parameters for each Gaussian element that are independent of the speckle appearance and shared across multiple viewpoints. The grayscale rendering result from each viewpoint is represented as follows: ； in, This represents the rendering weights calculated with fixed geometric parameters. Indicates the first The grayscale appearance parameters shared by each Gaussian element across all viewpoints. Indicates the number of Gaussian elements; S42. Optimize grayscale appearance parameters using the multi-view grayscale images acquired in step S11. The optimization process is as follows: ； in, A set of grayscale appearance parameters shared from multiple perspectives. For the first A true grayscale image from a single perspective. For grayscale image reconstruction loss, For the number of viewpoints, To fix the geometric parameters, Indicates the first The grayscale image is obtained by rendering from a single viewpoint.

8. The binocular speckle scene reconstruction method based on Gaussian splashing according to claim 7, characterized in that: During the optimization of grayscale appearance parameters, the center position, scale, rotation, and opacity of the Gaussian elements are not updated.

Images