Three-dimensional model surface reconstruction method, storage medium, electronic equipment and program product

Abstract

The invention provides a three-dimensional model surface reconstruction method, a storage medium, electronic equipment and a program product, and relates to the technical field of computer vision and three-dimensional reconstruction. The method comprises the steps that a 3D Gaussian splash model is used for simulating a plurality of collection view angles in a target scene, a target image pair under each collection view angle is generated, the target image pair at least comprises a first view and a second view, and the first view and the second view meet epipolar constraints; determining a disparity map of the target image pair under each collection view angle by using a stereo matching model; determining a depth map of the target image pair under each acquisition view angle based on the disparity map of the target image pair under each acquisition view angle; and fusing the depth maps of the target image pairs under the plurality of collection view angles to obtain a three-dimensional surface model. The geometric accuracy of the reconstructed surface is improved, the contradiction between the efficiency and the quality is solved, and the dependence on the accuracy of the collection view angle is reduced.

Description

Technical Field

[0001] This application relates to the fields of computer vision and 3D reconstruction technology, specifically to a 3D model surface reconstruction method, storage medium, electronic device, and program product. Background Technology

[0002] 3D surface reconstruction technology mainly revolves around multi-view visual computing, aiming to efficiently and accurately recover the 3D geometric information of objects or scenes from image data.

[0003] However, current solutions generally face the common challenge of balancing reconstruction quality, efficiency, and robustness. For example, they are prone to geometric noise and artifacts, have high computational costs, fail to meet the needs of rapid processing in practical applications, and lack adaptability to real-world scenarios. Summary of the Invention

[0004] In view of this, embodiments of this application provide a three-dimensional model surface reconstruction method, a storage medium, an electronic device, and a program product.

[0005] In a first aspect, one embodiment of this application provides a method for reconstructing a three-dimensional model surface. The method includes: simulating multiple acquisition perspectives in a target scene using a 3D Gaussian splash model, generating a target image pair under each acquisition perspective, wherein the target image pair includes at least a first view and a second view, and the first view and the second view satisfy epipolar constraints; determining a disparity map of the target image pair under each acquisition perspective using a stereo matching model; determining a depth map of the target image pair under each acquisition perspective based on the disparity map of the target image pair under each acquisition perspective; and fusing the depth maps of the target image pairs under multiple acquisition perspectives to obtain a three-dimensional surface model.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, a 3D Gaussian splash model is used to simulate multiple acquisition perspectives in the target scene to generate target image pairs under each acquisition perspective, including: determining the view pose corresponding to the first view under each acquisition perspective; determining the baseline distance based on the scene scale corresponding to the multiple acquisition perspectives; determining the view pose of the second view based on the view pose corresponding to the first view and the baseline distance; and using the 3D Gaussian splash model, rendering and generating the first view based on the view pose of the first view, and rendering and generating the second view based on the view pose of the second view.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the baseline distance is determined based on the scene scale corresponding to multiple acquisition perspectives, including: determining the point cloud data corresponding to the target scene; determining the scene scale of the target scene based on the point cloud data corresponding to the target scene; and multiplying the scene scale of the target scene by the target scaling factor to obtain the baseline distance.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, before simulating multiple acquisition perspectives in the target scene using a 3D Gaussian splash model and generating target image pairs under each acquisition perspective, the method further includes: acquiring a multi-view image sequence of the target scene and estimating the view pose corresponding to each frame image in the multi-view image sequence; and determining multiple acquisition perspectives from the view pose corresponding to each frame image in the multi-view image sequence.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the depth map of the target image pair under each acquisition viewpoint is determined based on the disparity map of the target image pair under each acquisition viewpoint, including: converting the disparity map of each target image pair into an initial depth map based on the baseline distance and imaging system parameters of each target image pair; and performing reliability screening on the initial depth map of each target image pair to obtain the depth map of each target image pair.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, reliability screening is performed on the initial depth map of each target image pair to obtain the depth map of each target image pair, including: determining the occlusion region of the target image pair based on the disparity map of each target image pair; determining the effective range of the depth values ​​corresponding to the initial depth map of each target image pair based on the baseline distance of each target image pair; and filtering the initial depth map of each target image pair based on the occlusion region of the target image pair and the effective range of the depth values ​​corresponding to the initial depth map of each target image pair to obtain the depth map of each target image pair.

[0011] In conjunction with the first aspect, in certain implementations of the first aspect, the depth maps of target image pairs from multiple acquisition perspectives are fused to obtain a three-dimensional surface model, including: dividing the target scene into multiple voxels; for each acquisition perspective, determining the truncated symbolic distance field of that acquisition perspective; integrating the truncated symbolic distance fields of multiple acquisition perspectives into multiple voxels of the target scene, and updating the fused geometric value and cumulative confidence of each voxel, wherein the fused geometric value represents the scalar value of the surface spatial relationship between the voxel and the target scene, and the cumulative confidence represents the sum of the confidence of the contribution of all target image pairs from all acquisition perspectives to the fused geometric value of the voxel; and extracting isosurfaces based on the fused geometric value and cumulative confidence of each voxel to obtain a three-dimensional surface model.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, isosurfaces are extracted based on the fused geometric values ​​and cumulative confidence of each voxel to obtain a three-dimensional surface model, including: filtering multiple voxels of the target scene based on cumulative confidence to obtain effective voxels with cumulative confidence higher than the target threshold; traversing each effective voxel and extracting an initial triangular mesh based on the fused distance value of each effective voxel; and post-processing the initial triangular mesh to generate a three-dimensional surface model.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the depth maps of target image pairs from multiple acquisition perspectives are fused to obtain a three-dimensional surface model, including: obtaining a mask of the target object in a first view from the target acquisition perspective, where the target acquisition perspective is one of multiple acquisition perspectives; based on the depth map from the target acquisition perspective, propagating the mask of the target object to other acquisition perspectives from the multiple acquisition perspectives to obtain a mask of the target object from each acquisition perspective; and fusing the depth maps within the mask of the target object from each acquisition perspective to obtain a three-dimensional surface model of the target object.

[0014] Secondly, one embodiment of this application provides a three-dimensional model surface reconstruction device, the device comprising: a generation module, configured to simulate multiple acquisition perspectives in a target scene using a 3D Gaussian splash model, and generate a target image pair under each acquisition perspective, the target image pair including at least a first view and a second view, the first view and the second view satisfying epipolar constraints; a first determination module, configured to determine the disparity map of the target image pair under each acquisition perspective using a stereo matching model; a second determination module, configured to determine the depth map of the target image pair under each acquisition perspective based on the disparity map of the target image pair under each acquisition perspective; and a fusion module, configured to fuse the depth maps of the target image pairs under multiple acquisition perspectives to obtain a three-dimensional surface model.

[0015] Thirdly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the three-dimensional model surface reconstruction method described in the first aspect.

[0016] Fourthly, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to perform the three-dimensional model surface reconstruction method described in the first aspect.

[0017] Fifthly, one embodiment of this application provides a computer program product including instructions that, when executed on an electronic device, cause the electronic device to implement the three-dimensional model surface reconstruction method described in the first aspect.

[0018] In this application, by generating target image pairs that are strictly aligned with the acquisition viewpoint and utilizing the geometric prior of a pre-trained stereo matching model, the noise problem inherent in directly extracting the surface based on Gaussian positions is effectively avoided, thereby generating accurate depth information and significantly improving the geometric accuracy of the reconstructed surface. Secondly, based on the efficient rendering of 3DGS, the scheme in this application only requires adding lightweight operations such as target image pair generation and depth calculation, which can improve the processing speed by more than an order of magnitude while ensuring reconstruction quality, successfully resolving the contradiction between efficiency and quality. Finally, by introducing epipolar constraints in stereo matching, the dependence on the accuracy of the input acquisition viewpoint is reduced, enhancing the processing capability for data acquired by ordinary camera devices such as mobile phones, and improving the robustness of the method in this application in practical applications. Attached Figure Description

[0019] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 The diagram shown is a flowchart of a three-dimensional model surface reconstruction method provided in an embodiment of this application.

[0021] Figure 2 The diagram shown is a complete reconstruction flowchart from a two-dimensional image sequence to a three-dimensional surface model provided in an embodiment of this application.

[0022] Figure 3 The diagram shown is a structural schematic of a three-dimensional model surface reconstruction device provided in an embodiment of this application.

[0023] Figure 4 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] With the rapid development of technologies such as digital twins and virtual reality, image-based 3D model surface reconstruction methods have received widespread attention. Currently, the mainstream technical approaches mainly revolve around three categories: rendering techniques based on explicit representations, reconstruction methods based on neural implicit surfaces, and traditional multi-view geometric reconstruction methods.

[0026] Specifically, explicit representation-based rendering techniques optimize the appearance and geometric properties of numerous spatial elements in a scene, achieving photorealistic visual fidelity while maintaining high rendering speed. However, these methods have significant shortcomings in surface reconstruction quality, and the extracted surfaces need improvement in physical consistency and smoothness. Consequently, they cannot form continuous and consistent geometric surfaces, often resulting in noticeable noise and fluctuations in the directly extracted mesh models.

[0027] Reconstruction methods based on neural implicit surfaces learn the geometric representation of a scene through deep neural networks, enabling the reconstruction of detailed and high-quality surface models. These methods offer excellent reconstruction accuracy, but require hours or even days of training time. The enormous computational cost limits their value in real-time applications, while efficient methods cannot guarantee geometric accuracy.

[0028] Traditional multi-view geometric reconstruction methods rely on pixel-level matching between images to recover the 3D structure through triangulation. These methods have extremely stringent requirements for the quality of the input data, especially the accuracy of camera pose estimation, which directly affects the reconstruction results. Therefore, in practical applications, even minute pose errors can lead to severe noise and voids on the reconstructed surface.

[0029] Based on the current state of 3D reconstruction technology and its shortcomings such as geometric inconsistency, efficiency-quality contradiction, and insufficient robustness, the surface reconstruction scheme provided in this application demonstrates significant application value in multiple fields that require high-quality 3D surface models by combining efficient rendering with geometric priors.

[0030] For example, in the field of cultural relic digitization, traditional reconstruction methods, due to their stringent requirements for pose accuracy and poor reconstruction results in areas with weak texture, struggle to meet the demands for high-precision, non-contact digitization of cultural relics. This application generates high-quality virtual views using a 3D Gaussian splash model and combines it with the geometric priors of a stereo matching model. This effectively overcomes challenges such as complex patterns and fragile structures on the surface of cultural relics. Only ordinary equipment is needed to acquire images, enabling complete reconstruction of geometric details and high-fidelity preservation of texture features, thus supporting subsequent virtual display, restoration research, and long-term conservation.

[0031] Furthermore, in the field of industrial inspection, current solutions suffer from insufficient reconstruction quality on reflective metallic surfaces and areas with weak textures, and struggle to balance inspection efficiency and accuracy requirements. The surface reconstruction scheme provided in this application significantly improves the robustness of reconstruction on complex industrial material surfaces by generating stereo image pairs that conform to epipolar constraints and introducing strong geometric constraints from a stereo matching model. In addition, this method can accurately reconstruct the three-dimensional contours of industrial parts, enabling automated detection of key parameters such as dimensional tolerances, assembly gaps, and surface defects. It effectively solves the problems of poor adaptability and unstable accuracy of traditional methods in industrial environments, providing reliable quality control assurance for intelligent manufacturing.

[0032] In addition to the above, the technical solutions of this application are also applicable to application scenarios that have high requirements for the surface quality and generation efficiency of 3D models, such as real-scene 3D modeling, digital twins, and augmented reality, which will not be elaborated here.

[0033] Below, in conjunction with Figure 1 and Figure 2 This paper elaborates on the implementation of the three-dimensional model surface reconstruction method of this application.

[0034] Specifically, Figure 1 The diagram shown is a flowchart illustrating a three-dimensional model surface reconstruction method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps.

[0035] Step S110: Use a 3D Gaussian splash model to simulate multiple acquisition perspectives in the target scene and generate target image pairs for each acquisition perspective.

[0036] 3D Gaussian Splatting (3DGS) is a neural rendering model that optimizes the parameters (including position, covariance, opacity, and spherical harmonics) of multiple Gaussian spatial elements in a scene. This model can generate novel views with photorealistic quality.

[0037] The acquisition viewpoint refers to the discrete, calibrated position and orientation of each observation point during data acquisition of the target scene in the 3D reconstruction process. Specifically, each acquisition viewpoint is defined by a unique view pose, which is usually represented as a matrix containing a rotation matrix R and a translation vector T. It clarifies the relative spatial relationship between the imaging system coordinate system and the world coordinate system under that acquisition viewpoint.

[0038] A target image pair refers to a set of stereo images, including a first view and a second view, generated for each acquisition viewpoint. The first and second views satisfy epipolar constraints. Specifically, epipolar constraints restrict the optical center of the imaging system of the second view to a positional relationship with the optical center of the imaging system of the first view, allowing only horizontal translation. This ensures that the image matching problem between the two views is constrained to the same scan line, thereby guaranteeing the correctness and efficiency of the subsequent stereo matching process. For example, the second view is generated by introducing a horizontal displacement (i.e., baseline distance) along the X-axis of the imaging system based on the same pose as the first view.

[0039] Optionally, in some embodiments, multiple acquisition viewpoints are input into a 3D Gaussian splash model, driving it to render the virtual observation image corresponding to each viewpoint. Then, for each acquisition viewpoint, the 3D Gaussian splash model, while maintaining the original viewpoint rendering to obtain a first view (e.g., the left view), calculates the acquisition viewpoint of the second viewpoint by applying a preset translation along the X-axis of the imaging system coordinate system, and renders the corresponding second view (e.g., the right view). Through this pose construction method, it is ensured that the generated first and second views naturally satisfy epipolar constraints, that is, the corresponding pixels between the two views are located on the same image row, thereby providing ideal input data with accurate geometric relationships and high radiometric consistency for subsequent stereo matching algorithms.

[0040] Step S120: Using a stereo matching model, determine the disparity map of the target image pair under each acquisition viewpoint.

[0041] A stereo matching model refers to a deep neural network pre-trained on a large amount of real or synthetic stereo data. This network can establish pixel-level correspondences between a first view and a second view, and has the ability to infer scene geometric information from target image pairs. For example, in this embodiment, the stereo matching model includes, but is not limited to, architectures such as Deep Local and Non-local Rectification Network (DLNR), Recurrent All-Pairs Field Transforms for Stereo (RAFT-Stereo), or Iterative Geometry Encoding Volume (IGEV).

[0042] The disparity map of a target image pair is a two-dimensional data field that uses the first view of the target image pair as a reference coordinate system and quantitatively represents the horizontal coordinate offset of the corresponding matching point of each pixel in the second view along the epipolar direction.

[0043] Optionally, in some embodiments, the generated target image pairs from each acquisition viewpoint are input as input data into a pre-trained stereo matching model. This stereo matching model extracts multi-level visual features from the target image pairs through its internal multi-level feature extraction module, then calculates the matching cost between features of the target image pairs in the feature space, optimizes the cost volume using two-dimensional or three-dimensional convolution to enhance the discriminative power of the matching cost, and finally calculates the optimal disparity value for each pixel based on the optimized cost volume, thereby generating a disparity map with the same resolution as the input target image pairs.

[0044] Step S130: Based on the disparity map of the target image pair under each acquisition viewpoint, determine the depth map of the target image pair under each acquisition viewpoint.

[0045] Optionally, the disparity map for each acquisition viewpoint is converted into a depth map using a geometric transformation relationship. Specifically, based on the baseline distance B corresponding to each target image pair and the imaging system parameters (e.g., if the imaging system is a camera, the imaging system parameters include the camera focal length f and the principal point coordinates), a mathematical relationship between disparity d and depth Z is established. Furthermore, for each valid pixel location in the disparity map, based on its disparity value... The corresponding depth value can be directly calculated using the above mathematical relationships. This generates a depth map with the same resolution as the disparity map.

[0046] Here, the baseline distance B is the translation amount used when generating the second view in step S110, and the imaging system parameters are derived from the intrinsic parameter matrix obtained through imaging system calibration during the target scene acquisition stage. This conversion process realizes the transformation from a two-dimensional disparity representation based on pixel displacement to three-dimensional geometric information based on physical distance, providing geometric data that conforms to the real-world scale for subsequent multi-view depth fusion.

[0047] Step S140: The depth maps of target image pairs from multiple acquisition perspectives are fused to obtain a three-dimensional surface model.

[0048] Depth map fusion refers to the process of integrating depth observations from different acquisition perspectives, which may contain noise and viewpoint limitations, into a consistent three-dimensional geometric representation using a specific algorithm. This process aims to eliminate blind spots and noise in depth maps acquired from a single perspective, generating a complete and accurate three-dimensional surface model.

[0049] Correspondingly, a three-dimensional surface model refers to the final output, which is a digital three-dimensional model represented in the form of a triangular mesh that can completely describe the surface geometry of the target scene.

[0050] Optionally, in some embodiments, the depth map from each acquisition viewpoint is first back-projected to generate a 3D point cloud, and the 3D point clouds from all acquisition views are unified to the same coordinate system based on a point cloud registration algorithm (such as the iterative nearest point algorithm). Subsequently, statistical filtering is performed on the registered dense point cloud to remove noise, and a closed triangular mesh model is generated based on the denoised point cloud.

[0051] In this embodiment, by generating target image pairs that are strictly aligned with the acquisition viewpoint and utilizing the geometric prior of a pre-trained stereo matching model, the noise problem inherent in directly extracting the surface based on Gaussian positions is effectively avoided, thereby generating accurate depth information and significantly improving the geometric accuracy of the reconstructed surface. Secondly, based on the efficient rendering of 3DGS, the scheme in this embodiment only requires adding lightweight operations such as target image pair generation and depth calculation, which can improve the processing speed by more than an order of magnitude while ensuring reconstruction quality, successfully resolving the contradiction between efficiency and quality. Finally, by introducing epipolar constraints in stereo matching, the dependence on the accuracy of the input acquisition viewpoint is reduced, enhancing the processing capability for data acquired by ordinary camera devices such as mobile phones, and improving the robustness of the method in this embodiment in practical applications.

[0052] To ensure that the generated target images possess optimal stereoscopic observation conditions and improve the accuracy of subsequent depth estimation, this application provides an optimized stereoscopic view generation method. Specifically, the view pose corresponding to the first view under each acquisition perspective is determined; the baseline distance is determined based on the scene scale corresponding to multiple acquisition perspectives; the view pose of the second view is determined based on the view pose corresponding to the first view and the baseline distance; using a 3D Gaussian splash model, the first view is rendered and generated based on the view pose of the first view, and the second view is rendered and generated based on the view pose of the second view.

[0053] The view pose corresponding to the first view refers to the spatial position and orientation parameters of the imaging system (such as the camera) at each acquisition viewpoint, which are estimated in advance by the Structure from Motion (SFM) algorithm. These parameters are represented in the form of rotation matrix and translation vector.

[0054] Furthermore, the technique of determining the view pose of the second view based on the view pose corresponding to the first view and the baseline distance is grounded in the geometric constraints of a binocular stereo vision system. Specifically, the view pose of the first view defines the rotation matrix R and translation vector T of the first imaging system (e.g., the left camera) in the world coordinate system. To obtain the pose of the second view (e.g., the right view), the same rotation matrix R is maintained to ensure the optical axes of the two imaging systems are parallel. Only the translation vector T is increased along the positive X-axis of the first imaging system coordinate system by the offset of the baseline distance B, i.e., the new translation vector... .

[0055] By maintaining the same rotation parameters, the imaging planes of the two imaging systems are made completely parallel, satisfying the basic conditions of epipolar constraint. Secondly, by precisely controlling the horizontal translation, the line connecting the optical centers of the two imaging systems is made parallel to the imaging plane, ensuring that corresponding pixels in the generated target image pair lie on the same scan line. Finally, the pose determination method of the second view ensures that the generated virtual image pair (i.e., the target image pair) has geometric characteristics completely consistent with the real stereo imaging system, eliminating the need for subsequent epipolar correction processing and guaranteeing the accuracy and efficiency of the subsequent stereo matching process.

[0056] The following section uses the 3D reconstruction of an indoor piece of furniture as an example to illustrate the specific process of generating a target image.

[0057] For example, firstly, the target sofa is photographed from multiple angles using a camera, obtaining an image sequence containing 65 acquisition viewpoints. This sequence is processed using the SFM algorithm to obtain the view pose corresponding to each acquisition viewpoint, while simultaneously reconstructing a scene point cloud containing 12,000 3D points. Next, based on the spatial distribution of the scene point cloud, the average distance from each point in the point cloud to the center of the point cloud is calculated to obtain the scene radius R (e.g., 2.5 meters) as the scene scale parameter, and the baseline distance B is determined based on the scene scale parameter. Finally, for each acquisition viewpoint, the view pose estimated by SFM at that viewpoint is used as the view pose corresponding to the first view. Keeping the rotation parameters of this pose unchanged, an offset along the camera's X-axis is added to the translation parameters to obtain the view pose of the second view. The two view poses are then input into a trained 3D Gaussian splash model to render and generate the first and second views, respectively.

[0058] In this embodiment, the second view pose is generated by maintaining the first view pose unchanged and applying a baseline distance, ensuring that the generated target image pair strictly satisfies the epipolar constraint, thus establishing a correct geometric basis for subsequent stereo matching. Furthermore, the baseline distance is adaptively determined based on the scene scale. When the scene scale is large, the baseline distance calculated proportionally increases accordingly, ensuring sufficient stereo parallax range when observing large-scale scenes. Conversely, when the scene scale is small, the correspondingly reduced baseline distance avoids matching difficulties caused by excessive parallax, while effectively controlling the occlusion area range between different acquisition viewpoints. Finally, the first and second views are generated by rendering using a 3D Gaussian splash model under the specified view pose, ensuring that the generated target image pair maintains high consistency in lighting, color, and texture representation, and that the generated target image pair strictly satisfies multi-view geometric constraints, especially the epipolar constraint relationship.

[0059] To optimize stereo matching and ensure the effectiveness of depth estimation, this application further provides an adaptive baseline distance determination method. Specifically, the method involves determining the point cloud data corresponding to the target scene; determining the scene scale of the target scene based on the point cloud data; and multiplying the scene scale of the target scene by a target scaling factor to obtain the baseline distance.

[0060] The point cloud data corresponding to the target scene refers to a sparse set of three-dimensional points that characterize the spatial structure of the target scene.

[0061] Scene scale refers to the spatial extent of a target scene based on the spatial distribution statistical characteristics of a 3D point cloud. It is determined by calculating one or a combination of the diagonal length of the point cloud bounding box, the distance from the center of the point cloud to the farthest point, or the eigenvalues ​​of the point cloud covariance matrix, as well as the scene size (such as the scene radius).

[0062] The target scaling factor is an optimized parameter determined in advance through experiments. For example, its value ranges from 0.03 to 0.12, with a preferred value of 0.07. This factor reflects the optimal scaling relationship between the baseline distance and the scene scale.

[0063] For example, 50 multi-view images of a target part are acquired using a camera mounted at the end of a robotic arm. These images are then processed using a structure-based motion reconstruction (SPC) method to obtain a sparse point cloud containing 8000 3D points. Based on the root mean square distance from each point to the center of the point cloud, the scene scale parameter R = 0.5 meters is obtained. This scale parameter is then compared with a preset target scale factor. Multiplying by 0.07 yields a baseline distance B of 0.035 meters. This baseline distance configuration ensures sufficient parallax measurement accuracy during subsequent stereo matching while effectively controlling occlusion areas between viewpoints, providing optimized stereo observation conditions for the 3D reconstruction of the part's surface.

[0064] In this embodiment, the scene scale parameters are calculated based on 3D point cloud computing, ensuring an optimal match between the baseline configuration and the scene space size. This guarantees sufficient parallax range for effective depth measurement while avoiding occlusion issues between viewpoints caused by excessively long baselines. Furthermore, by multiplying the scene scale by an optimized and verified target scaling factor, automatic optimization calculation of the baseline distance is achieved, effectively improving the configuration efficiency and reliability of stereoscopic observation.

[0065] To ensure the accuracy and completeness of multiple acquisition perspectives, this embodiment of the application also requires preprocessing of the multi-view image sequence of the target scene. Specifically, before simulating multiple acquisition perspectives in the target scene using a 3D Gaussian splash model and generating target image pairs under each acquisition perspective, the method further includes: acquiring the multi-view image sequence of the target scene and estimating the view pose corresponding to each frame image in the multi-view image sequence; and determining multiple acquisition perspectives from the view poses corresponding to each frame image in the multi-view image sequence.

[0066] A multi-view image sequence refers to a series of two-dimensional images captured from different spatial locations and angles of the same target scene, which together cover the complete visual information of the scene. In this embodiment, multi-view images can be captured by consumer-grade shooting devices such as digital cameras or mobile phones surrounding the target scene. It is understood that a multi-view image sequence can also be an image sequence from other sources, as long as it can provide scene information from multiple perspectives.

[0067] Optionally, in some embodiments, multiple acquisition viewpoints are determined from the view poses corresponding to each frame of the multi-view image sequence, including: The view poses of images in a multi-view image sequence that meet the target conditions are determined as multiple acquisition viewpoints, wherein the target conditions include at least one of the following: The reprojection error of the view pose of each frame is lower than the target error threshold (e.g., 0.5 pixels). The view pose of an image that can cover all key areas of the target scene; The view pose of the image is uniformly distributed in the spherical coordinate system, and the view pose between two adjacent frames is within the target angle range (e.g., 15° to 30°).

[0068] Taking the digitization of cultural relics as an example, 120 multi-view images of a bronze ding were first taken by holding a mobile phone around it. After obtaining the view poses of all images through motion reconstruction, unreliable view poses with reprojection errors greater than 0.5 pixels were removed. Based on the visibility analysis of feature points on the surface of the bronze ding, it was ensured that key areas such as the ding body decorations, ding ears, and ding feet were covered by at least 3 acquisition viewpoints. Forty spatially evenly distributed viewpoints were selected from all view poses as the final acquisition viewpoints. It was verified that the average angle between the selected acquisition viewpoints was 22.5°, which met the spatial distribution requirements.

[0069] In this embodiment, firstly, by automatically estimating the view pose corresponding to each frame of the image, a reliable imaging system parameter basis is provided for the subsequent generation of virtual target image pairs, ensuring the geometric accuracy of the entire reconstruction process. Secondly, multiple acquisition viewpoints are determined from the view poses corresponding to each frame of the multi-view image sequence, ensuring that the selected viewpoints have optimal spatial distribution and quality reliability, providing complete scene coverage for subsequent 3D reconstruction. Finally, this preprocessing mechanism effectively reduces the dependence of the method in this application on professional acquisition equipment, enabling it to adapt to image data acquired by ordinary acquisition equipment, thus improving the applicability and robustness of the method in practical applications.

[0070] To ensure the accuracy and reliability of depth information, this application embodiment, based on the obtained disparity map, achieves high-precision conversion and data optimization from disparity to depth through the following method. Specifically, based on the baseline distance and imaging system parameters of each target image pair, the disparity map of each target image pair is converted into an initial depth map; the initial depth map of each target image pair is then subjected to reliability screening to obtain the depth map of each target image pair.

[0071] For example, the imaging system is a binocular camera.

[0072] Optionally, in some embodiments, based on the geometric principles of stereo vision, the disparity map of each target image pair is converted into an initial depth map using the baseline distance of each target image pair and imaging system parameters. For example, the formula is: Furthermore, reliability screening refers to the process of assessing the reliability of depth values ​​in the initial depth map and filtering the data by establishing geometric constraints.

[0073] In this embodiment, the disparity map is converted into a depth map based on the baseline distance and imaging system parameters, ensuring the physical accuracy and metric consistency of the depth value calculation. Secondly, a reliability screening mechanism is introduced, effectively improving the quality and credibility of the depth data. Finally, through a combination of geometric transformation and data optimization, reliable input data is provided for subsequent multi-view depth fusion, thereby ultimately ensuring the reconstruction accuracy of the 3D surface model.

[0074] To improve the quality of the depth map and suppress noise interference, this application embodiment provides the following reliability screening mechanism based on the obtained initial depth map. Specifically, based on the disparity map of each target image pair, the occlusion region of the target image pair is determined; based on the baseline distance of each target image pair, the effective range of the depth values ​​corresponding to the initial depth map of each target image pair is determined; based on the occlusion region of the target image pair and the effective range of the depth values ​​corresponding to the initial depth map of each target image pair, the initial depth map of each target image pair is filtered to obtain the depth map of each target image pair.

[0075] Occlusion regions are identified by comparing the corresponding values ​​of the disparity maps of the first and second views after transformation. Regions with differences exceeding a target threshold are considered unreliable areas due to viewing angle differences. For example, using the disparity map (d1) of the first view, the disparity map (d2) of the second view is distorted to the perspective of the left view, resulting in the transformed and aligned disparity map warp(d1). Theoretically, in non-occluded regions, the value of a pixel in d1 should be very close to the value at the corresponding position in warp(d1). Assuming an occlusion threshold... ,if If the pixel's parallax is inconsistent in the two views, it is considered an occluded area. If the disparity of the pixel is consistent in both views, the pixel is considered to be located in the visible area, and its depth information is reliable. For example, [0.5, 2] pixels, preferably 1 pixel.

[0076] The effective range of depth values ​​defines a reasonable physical interval for depth values, excluding depth anomalies caused by parallax measurement errors. For example, the effective depth range is limited to... .

[0077] Optionally, in this embodiment, an occlusion mask is generated based on a consistency comparison of the disparity maps of the first and second views. Then, the upper and lower bounds of the depth values ​​are calculated based on the baseline distance to determine the effective depth range. Finally, the occlusion mask and the effective depth range are combined to perform joint filtering on the initial depth map, removing depth values ​​that do not meet the double verification conditions, and outputting an optimized depth map.

[0078] For example, when generating target image pairs, the baseline distance =0.08 meters. Using the previous example, the minimum depth value can be obtained. =4 × 0.08 = 0.32 meters, maximum depth value The effective range for depth values ​​is [0.32, 1.6] meters. During the reliability screening step, regions in the initial depth map with depth values ​​below 0.32 meters or above 1.6 meters will be marked as invalid and filtered out, ensuring that subsequent fusion processes use only reliable depth data that conforms to physical constraints.

[0079] In this embodiment, occlusion region detection based on disparity maps effectively identifies and eliminates unreliable depth estimation regions caused by viewing angle differences, thus solving the inherent occlusion problem in stereo matching. The effective depth range is determined by baseline distance, eliminating abnormal depth values ​​beyond the reasonable measurement range. By jointly applying these two filtering mechanisms—occlusion region detection and effective depth range limitation—a synergistically optimized data filtering effect is achieved, significantly improving the data quality and reliability of the depth map and providing more accurate and consistent geometric information input for subsequent 3D reconstruction processes.

[0080] To achieve accurate reconstruction from multi-view depth information to a unified surface model, this application provides a voxel fusion-based 3D reconstruction method. Specifically, the target scene is divided into multiple voxels; for each acquisition viewpoint, the truncated symbolic distance field of that viewpoint is determined; the truncated symbolic distance fields of multiple acquisition views are integrated into multiple voxels of the target scene, and the fused geometric value and cumulative confidence of each voxel are updated; based on the fused geometric value and cumulative confidence of each voxel, isosurfaces are extracted to obtain a 3D surface model.

[0081] A voxel is a cubic unit formed by uniformly dividing the three-dimensional space of a target scene into a regular grid. For example, the size of the voxel is determined according to the scale of the target scene. For example, the voxel is 1 / 1500 to 1 / 500 of the diameter of the target scene, preferably 1 / 1000.

[0082] The truncated sign distance field represents the geometric information of the scene surface under each acquisition view. Each voxel stores the signed distance to the nearest surface under that acquisition view. Positive values ​​indicate that the voxel is outside the surface, and negative values ​​indicate that it is inside the surface. The truncation mechanism limits the distance to an effective range near the surface.

[0083] The fused geometric value is a scalar value obtained by integrating truncated symbolic distance fields from multiple acquisition perspectives using a weighted average algorithm. This value characterizes the spatial positional relationship between the voxel and the real target scene surface.

[0084] The cumulative confidence score represents the sum of the confidence levels of the contribution of the target images from all acquisition perspectives to the fused geometry of the voxel. For example, this confidence score is obtained by summing the reliability weights of the depth maps from each acquisition perspective, reflecting the reliability of the fused geometry. It is understood that high-confidence regions correspond to reliable surface regions consistently observed from multiple perspectives.

[0085] An isosurface is a continuous surface formed by all points with the same specific value in a three-dimensional scalar field.

[0086] Optionally, in some embodiments, a three-dimensional voxel grid is first established based on the spatial extent of the target scene, with each voxel used to store the fused geometric information and confidence data. Next, for each acquisition viewpoint, the signed distance from each voxel to the observation surface is calculated based on the depth map and camera pose parameters at that viewpoint. For example, a positive distance value indicates that the voxel is located outside the surface, and a negative value indicates that it is located inside the surface. Then, a truncation mechanism is used to limit the distance values ​​to an effective range near the surface; for example, the truncation distance is set to 2 to 6 times the voxel size, preferably 3 times. Next, in the multi-view data fusion stage, the truncated signed distance fields from each acquisition viewpoint are integrated into a unified voxel grid, and the fused geometric value of each voxel is updated using a weighted average algorithm, while simultaneously updating the cumulative confidence.

[0087] Taking the digitization of cultural relics as an example, when reconstructing a bronze ding (a type of ancient Chinese cooking vessel), a grid space containing 200×200×150 voxels was first established. For each acquisition viewpoint, the distance from each voxel to the surface of the ding was calculated based on the depth information of that viewpoint, and unreliable data far from the surface was eliminated through truncation. Subsequently, the observation data from 40 different viewpoints were merged into a unified grid. The high-confidence areas were concentrated on the surface of the ding body observed by multiple viewpoints, while the low-confidence areas were located on the inside of the ding's ears, which were severely obscured.

[0088] In this embodiment, by dividing the target scene into a voxel mesh and integrating truncated symbolic distance fields from multiple perspectives, the discrete depth observation data is transformed into a continuous spatial representation, effectively solving the consistency problem of multi-view data. Through a collaborative update mechanism that integrates geometric values ​​and cumulative confidence, noise and outliers in single-view observations are effectively suppressed while preserving reliable geometric information. Finally, the confidence-weighted isosurface extraction method can automatically distinguish between high-confidence surface regions and low-confidence regions, significantly improving the geometric accuracy and completeness of the final 3D surface model.

[0089] To improve the geometric quality of the 3D surface model and ensure the integrity of the output mesh, this application optimizes the isosurface extraction process. Specifically, multiple voxels of the target scene are filtered based on cumulative confidence to obtain effective voxels with a cumulative confidence higher than the target threshold; each effective voxel is traversed, and an initial triangular mesh is extracted based on the fusion distance value of each effective voxel; the initial triangular mesh is post-processed to generate a 3D surface model.

[0090] Effective voxels refer to voxel units whose cumulative confidence value is higher than a preset target threshold. These voxels represent scene areas that are observed by multiple acquisition perspectives and whose depth estimates are consistent.

[0091] The initial triangular mesh is a preliminary surface model generated by traversing all valid voxels and detecting the zero isosurfaces of their fusion distance values.

[0092] Optionally, in some embodiments, a target threshold for the cumulative confidence score is set according to application requirements, and the voxel mesh is filtered based on this threshold, retaining valid voxels with confidence scores higher than the threshold. Then, the Marching Cubes algorithm is used to traverse these valid voxels, generating a continuous triangular mesh surface based on the spatial distribution of their fusion distance values. Finally, the extracted initial triangular mesh undergoes post-processing operations including smoothing, hole filling, and mesh simplification to eliminate the step-like artifacts caused by voxel discretization and fill in small missing areas, thereby generating a complete and smooth 3D surface model.

[0093] For example, taking the 3D reconstruction of a bronze artifact as an example, after completing the multi-view depth map fusion, a scene mesh containing 150×120×100 voxels is obtained. Each voxel stores the fused geometry value and cumulative confidence score. A target threshold of 0.7 is set, and voxels with a cumulative confidence score higher than 0.7 are marked as valid voxels, totaling 85,000, while voxels below this threshold (mainly distributed in severely occluded areas and scene edges) are excluded. Subsequently, the Marching Cubes algorithm is used to traverse these valid voxels. For each valid voxel, the sign change of its fused geometry value with that of its neighboring voxels is detected, and triangular patches are generated at the boundaries of the sign change. For example, in the belly region of the bronze ding (a type of ancient Chinese cooking vessel), the fused geometry values ​​of multiple valid voxels transition from positive values ​​(outside the surface) to negative values ​​(inside the surface), and the algorithm accurately generates triangular patches representing the curved surface of the ding's belly at these locations. Finally, the extracted initial triangular mesh is post-processed: for example, the Laplacian smoothing algorithm is first applied to eliminate the step-like artifacts caused by voxel discretization; then the small holes inside the ding ears are filled; finally, the mesh is appropriately simplified by the edge folding algorithm, and the number of mesh faces is optimized from the initial 1.2 million faces to 800,000 faces while maintaining the geometric features, and finally a bronze ding 3D model with complete details and a smooth surface is generated.

[0094] In this embodiment, a voxel filtering mechanism based on cumulative confidence effectively distinguishes between high-quality data and low-reliability regions with insufficient observations, significantly reducing the impact of noise and outliers on the reconstruction results. By extracting isosurfaces only within the effective voxel range, computational efficiency is greatly improved while ensuring reconstruction accuracy. Finally, combined with mesh post-processing, geometric defects caused by voxel discretization are effectively eliminated. The generated triangular mesh surface maintains geometric accuracy while possessing good visual integrity and smoothness, ultimately achieving efficient reconstruction of a high-quality 3D surface model.

[0095] To achieve accurate reconstruction of specific target objects in a scene, this application provides an optional embodiment that introduces object segmentation and mask propagation mechanisms during the reconstruction process to achieve surface reconstruction at the target object level. The specific implementation is as follows: In a first view from the target acquisition perspective, a mask of the target object is obtained. The target acquisition perspective is one of multiple acquisition perspectives. Based on the depth map from the target acquisition perspective, the mask of the target object is propagated to other acquisition perspectives among the multiple acquisition perspectives, obtaining the mask of the target object in each acquisition perspective. The depth maps within the mask of the target object in each acquisition perspective are fused to obtain a three-dimensional surface model of the target object.

[0096] The target object mask refers to the binary image obtained on the first view of the target acquisition perspective through an interactive segmentation model, where the target object region is marked as the valid region and the background region is marked as the invalid region.

[0097] Optionally, in some embodiments, firstly, an initial mask of the target object is obtained through interactive segmentation at a selected target acquisition viewpoint. Then, based on the depth map and camera pose of this acquisition viewpoint, the initial mask is back-projected into 3D space to form an initial surface hypothesis. This surface hypothesis is then transferred to the 2D image space of other acquisition viewpoints via orthographic projection. After morphological optimization, masks of the target object corresponding to each viewpoint are obtained. Finally, only the depth values ​​falling within the effective area of ​​the mask are retained, excluding interference from the background area.

[0098] For example, in an industrial parts quality inspection scenario, it is necessary to perform 3D reconstruction of bolts of a specific model on an assembly line. First, a clear acquisition viewpoint is selected as the target acquisition viewpoint. On the first view of this viewpoint, the bolt area is selected using an interactive segmentation tool to obtain an initial mask. Based on the depth map of this viewpoint, the corresponding 3D point cloud of the mask is projected onto adjacent viewpoints. Specifically, assuming a 30° angle between the adjacent viewpoint and the target viewpoint, the projection area of ​​the initial mask onto the image plane of the adjacent viewpoint is calculated through coordinate system transformation. Then, morphological dilation is used to compensate for the projection error, resulting in the bolt mask for that viewpoint. This process is iterated until bolt masks are obtained for all 40 acquisition viewpoints. Finally, only the depth values ​​within the mask areas of each viewpoint are fused. The reconstructed 3D bolt model is completely separated from the background, with a clear and complete thread structure, and can be directly used for subsequent dimensional tolerance inspection.

[0099] In this embodiment, an automatic identification and segmentation of target objects under multiple viewpoints is achieved through a mask propagation mechanism based on depth information, effectively avoiding the problem of manual annotation for each viewpoint required in traditional methods. Secondly, by fusing depth information only within the mask area, the interference of background noise on the reconstruction process is significantly reduced, improving the accuracy and completeness of target object reconstruction. Finally, the scheme in this embodiment achieves accurate separation and reconstruction of specific target objects in complex scenes, resulting in a reconstructed 3D surface model with clear semantic boundaries and a complete geometric structure.

[0100] To enable those skilled in the art to more clearly understand how the technical solutions protected by this application are specifically implemented, the implementation of this application will be described in detail below with reference to two embodiments.

[0101] Figure 2 The diagram shown is a complete reconstruction flowchart from a two-dimensional image sequence to a three-dimensional surface model provided in an embodiment of this application. Below, in conjunction with... Figure 2 First, a complete method embodiment will be used to demonstrate how to specifically implement the various technical steps defined in the above solution.

[0102] Example 1: First, input a video or image sequence of the target scene, process it using the SFM algorithm, and output the view pose corresponding to each frame and the sparse point cloud of the target scene. Then, train a 3DGS model based on this data. Next, for each acquisition viewpoint determined by SFM (i.e., each known view pose), perform the following operations: This pose is used as the view pose of the first view, and the baseline distance is adaptively determined according to the scene scale. The view pose of the second view is calculated through translation. The trained 3DGS model is used to render the images of these two poses respectively, generating target image pairs that satisfy epipolar constraints.

[0103] The target image is input into a pre-trained stereo matching model (such as DLNR, RAFT-Stereo, etc.) to obtain a disparity map. Subsequently, the disparity map is converted into a depth map based on the baseline distance, and after reliability screening, unreliable data is filtered out to obtain a high-quality depth map.

[0104] Reliable depth maps obtained from all acquisition perspectives are fused using a truncated signed distance function algorithm to update the fused geometric values ​​and cumulative confidence scores of each voxel. Finally, isosurfaces are extracted based on these data (e.g., using the Marching Cubes algorithm) to generate an initial triangular mesh. After post-processing such as denoising and smoothing, the final smooth 3D surface model is output.

[0105] The above embodiment one has fully described the method flow of this application. In order to transform the technical solutions in the method embodiment into an implementable system architecture, this application further provides embodiment two. This embodiment concretizes each step in the method flow into specific functional modules, forming a complete system implementation scheme.

[0106] Specifically, this embodiment provides a modular 3D surface reconstruction system, comprising six core components: a rendering module, a baseline determination module, a binocular depth estimation module, a reliability discrimination module, a voxel fusion module, and a mesh extraction module. Specifically, the rendering module generates target image pairs conforming to epipolar constraints based on a trained 3D Gaussian splash model; the baseline determination module adaptively calculates the optimal baseline distance based on the scene point cloud; the binocular depth estimation module extracts disparity and depth information from the target image pairs; the reliability discrimination module filters reliable depth data through occlusion detection and depth range verification; the voxel fusion module integrates multi-view depth information using a weighted truncated signed distance function algorithm; and the mesh extraction module finally generates an optimized 3D surface model based on confidence filtering and isosurface extraction. These modules work collaboratively to form a complete processing pipeline from data input to model output.

[0107] The above text combined Figure 1 and Figure 2 The present application describes in detail the embodiments of the three-dimensional model surface reconstruction method. The following is a combination of... Figure 3 This application provides a detailed description of embodiments of the three-dimensional model surface reconstruction apparatus. It should be understood that the descriptions of the three-dimensional model surface reconstruction method embodiments correspond to the descriptions of the three-dimensional model surface reconstruction apparatus embodiments; therefore, any parts not described in detail can be found in the preceding method embodiments.

[0108] Figure 3 The diagram shown is a structural schematic of a three-dimensional model surface reconstruction device provided in an embodiment of this application. Figure 3 As shown, the three-dimensional model surface reconstruction device 30 provided in this application embodiment includes: The generation module 310 is used to simulate multiple acquisition perspectives in the target scene using a 3D Gaussian splash model, and generate a target image pair under each acquisition perspective. The target image pair includes at least a first view and a second view, and the first view and the second view satisfy the epipolar constraint. The first determining module 320 is used to determine the disparity map of the target image pair under each acquisition view using a stereo matching model; The second determining module 330 is used to determine the depth map of the target image pair under each acquisition view based on the disparity map of the target image pair under each acquisition view. The fusion module 340 is used to fuse the depth maps of target image pairs from multiple acquisition perspectives to obtain a three-dimensional surface model.

[0109] In one embodiment of this application, the generation module 310 is further configured to: determine the view pose corresponding to the first view under each acquisition viewpoint; determine the baseline distance based on the scene scale corresponding to multiple acquisition viewpoints; determine the view pose of the second view based on the view pose corresponding to the first view and the baseline distance; and generate the first view by rendering based on the view pose of the first view using a 3D Gaussian splash model, and generate the second view by rendering based on the view pose of the second view.

[0110] In one embodiment of this application, the generation module 310 is further configured to: determine the point cloud data corresponding to the target scene; determine the scene scale of the target scene based on the point cloud data corresponding to the target scene; and multiply the scene scale of the target scene by the target scaling factor to obtain the baseline distance.

[0111] In one embodiment of this application, the generation module 310 is further configured to acquire a multi-view image sequence of the target scene and estimate the view pose corresponding to each frame of the multi-view image sequence; and determine multiple acquisition perspectives from the view poses corresponding to each frame of the multi-view image sequence.

[0112] In one embodiment of this application, the second determining module 330 is further configured to convert the disparity map of each target image pair into an initial depth map based on the baseline distance and imaging system parameters of each target image pair; and to perform reliability screening on the initial depth map of each target image pair to obtain the depth map of each target image pair.

[0113] In one embodiment of this application, the second determining module 330 is further configured to: determine the occlusion region of the target image pair based on the disparity map of each target image pair; determine the effective range of the depth values ​​corresponding to the initial depth map of each target image pair based on the baseline distance of each target image pair; and filter the initial depth map of each target image pair based on the occlusion region of the target image pair and the effective range of the depth values ​​corresponding to the initial depth map of each target image pair to obtain the depth map of each target image pair.

[0114] In one embodiment of this application, the fusion module 340 is further configured to: divide the target scene into multiple voxels; for each acquisition viewpoint, determine the truncated symbolic distance field of that acquisition viewpoint; integrate the truncated symbolic distance fields of multiple acquisition viewpoints into multiple voxels of the target scene, and update the fused geometric value and cumulative confidence of each voxel, wherein the fused geometric value represents the scalar value of the surface spatial relationship between the voxel and the target scene, and the cumulative confidence represents the sum of the confidence of the contribution of the target image of all acquisition viewpoints to the fused geometric value of the voxel; and extract isosurfaces based on the fused geometric value and cumulative confidence of each voxel to obtain a three-dimensional surface model.

[0115] In one embodiment of this application, the fusion module 340 is further configured to: filter multiple voxels of the target scene based on the cumulative confidence score to obtain effective voxels with a cumulative confidence score higher than the target threshold; traverse each effective voxel and extract an initial triangular mesh based on the fusion distance value of each effective voxel; and perform post-processing on the initial triangular mesh to generate a three-dimensional surface model.

[0116] In one embodiment of this application, the fusion module 340 is further configured to: obtain a mask of a target object in a first view under a target acquisition view, wherein the target acquisition view is one of multiple acquisition views; propagate the mask of the target object to other acquisition views among the multiple acquisition views based on the depth map under the target acquisition view, thereby obtaining a mask of the target object under each acquisition view; and fuse the depth maps within the mask of the target object under each acquisition view to obtain a three-dimensional surface model of the target object.

[0117] Below, for reference Figure 4 This describes an electronic device according to embodiments of the present application. Figure 4 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0118] like Figure 4 As shown, the electronic device 40 includes one or more processors 401 and memory 402.

[0119] The processor 401 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 40 to perform desired functions.

[0120] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may execute the program instructions to implement the three-dimensional model surface reconstruction methods of the various embodiments of this application described above, and / or other desired functions.

[0121] In one example, the electronic device 40 may also include an input device 403 and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0122] The input device 403 may include, for example, a keyboard, a mouse, etc.

[0123] The output device 404 can output various information to the outside. The output device 404 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0124] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device 40 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 40 may include any other suitable components depending on the specific application.

[0125] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the three-dimensional model surface reconstruction methods according to various embodiments of this application described above.

[0126] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0127] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the three-dimensional model surface reconstruction methods according to various embodiments of this application described above.

[0128] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0129] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0130] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0131] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0132] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0133] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method of surface reconstruction of a three-dimensional model, characterized by, The method comprises: simulating multiple collection perspectives in a target scene by using a 3D Gaussian splash model, generating a target image pair under each of the collection perspectives, the target image pair comprising at least a first view and a second view, the first view and the second view satisfying an epipolar constraint; determining a disparity map of the target image pair under each of the collection perspectives by using a stereo matching model; determining a depth map of the target image pair under each of the collection perspectives based on the disparity map of the target image pair under each of the collection perspectives; fusing the depth maps of the target image pairs under the multiple collection perspectives to obtain a three-dimensional surface model.

2. The three-dimensional model surface reconstruction method according to claim 1, characterized by, The simulating multiple collection perspectives in a target scene by using a 3D Gaussian splash model comprises: determining a view pose corresponding to the first view under each of the collection perspectives; determining a baseline distance based on scene scales corresponding to the multiple collection perspectives; determining a view pose of the second view based on the view pose corresponding to the first view and the baseline distance; generating the first view based on the view pose corresponding to the first view and generating the second view based on the view pose of the second view by using the 3D Gaussian splash model.

3. The three-dimensional model surface reconstruction method according to claim 2, wherein, The determining a baseline distance based on scene scales corresponding to the multiple collection perspectives comprises: determining point cloud data corresponding to the target scene; determining scene scales of the target scene based on the point cloud data corresponding to the target scene; multiplying the scene scales of the target scene by a target scale factor to obtain the baseline distance.

4. The method of claim 1, wherein Before the simulating multiple collection perspectives in a target scene by using a 3D Gaussian splash model, the method further comprises: obtaining a multi-view image sequence of the target scene and estimating a view pose corresponding to each frame of image in the multi-view image sequence; determining the multiple collection perspectives from the view poses corresponding to each frame of image in the multi-view image sequence.

5. The method of claim 1, wherein The determining a depth map of the target image pair under each of the collection perspectives based on the disparity map of the target image pair under each of the collection perspectives comprises: converting the disparity map of each of the target image pairs into an initial depth map based on a baseline distance and imaging system parameters of each of the target image pairs; performing reliability screening on the initial depth map of each of the target image pairs to obtain a depth map of each of the target image pairs.

6. The three-dimensional model surface reconstruction method according to claim 5, wherein, The performing reliability screening on the initial depth map of each of the target image pairs to obtain a depth map of each of the target image pairs comprises: determining an occlusion region of the target image pair based on the disparity map of each of the target image pairs; determining an effective range of depth values corresponding to the initial depth map of each of the target image pairs based on the baseline distance of each of the target image pairs; filtering the initial depth map of each of the target image pairs based on the occlusion region of the target image pair and the effective range of depth values corresponding to the initial depth map of each of the target image pairs to obtain a depth map of each of the target image pairs.

7. The method of claim 1, wherein The depth maps of the target image pairs under the plurality of collection perspectives are fused to obtain a three-dimensional surface model, including: The target scene is divided into a plurality of voxels; For each collection perspective, a truncated signed distance field of the collection perspective is determined; The truncated signed distance fields of the plurality of collection perspectives are integrated into the plurality of voxels of the target scene, and a fused geometry value and a cumulative confidence of each voxel are updated, wherein the fused geometry value represents a scalar value of a surface spatial relationship of the voxel with the target scene, and the cumulative confidence represents an accumulation of a fused geometry value contribution credibility of the target image pairs of all collection perspectives to the voxel; Based on the fused geometry value and the cumulative confidence of each voxel, an isosurface is extracted to obtain the three-dimensional surface model.

8. The three-dimensional model surface reconstruction method according to claim 7, wherein, The three-dimensional surface model is obtained based on the fused geometry value and the cumulative confidence of each voxel, including: Based on the cumulative confidence, the plurality of voxels of the target scene are filtered to obtain effective voxels with a cumulative confidence higher than a target threshold; Each of the effective voxels is traversed, and an initial triangular mesh is extracted based on the fused distance value of each effective voxel; The initial triangular mesh is post-processed to generate the three-dimensional surface model.

9. The method of claim 1 to 8, wherein, The depth maps of the target image pairs under the plurality of collection perspectives are fused to obtain a three-dimensional surface model, including: In a first view under a target collection perspective, a mask of a target object is obtained, the target collection perspective being one of the plurality of collection perspectives; Based on the depth map under the target collection perspective, the mask of the target object is propagated to other collection perspectives in the plurality of collection perspectives to obtain a mask of the target object under each collection perspective; The depth map in the mask of the target object under each collection perspective is fused to obtain a three-dimensional surface model of the target object.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is used to execute the three-dimensional model surface reconstruction method in any one of claims 1 to 9.

11. An electronic device, comprising: Including: a processor; a memory for storing instructions executable by the processor; the processor is configured to execute the three-dimensional model surface reconstruction method in any one of claims 1 to 9.

12. A computer program product, characterised in that, The computer program product includes instructions that, when executed on an electronic device, cause the electronic device to implement the three-dimensional model surface reconstruction method in any one of claims 1 to 9.

Images