A method, apparatus, equipment and medium for scene surface reconstruction

Abstract

This application provides a method, apparatus, device, and medium for scene surface reconstruction, relating to the field of computer graphics technology. First, an initialized 3D Gaussian model is introduced, and combined with a multi-view image sequence of the target scene, the attribute and position information of the 2D Gaussian model corresponding to the 3D anchor point Gaussian is determined. Second, using the attribute and position information, a rendered image is generated using the Gaussian splashing method. Subsequently, based on the rendered image and images with the same viewpoint as the rendered image in the multi-view image sequence, the initialized 3D Gaussian model is updated by minimizing the rendering loss function. Finally, depth information is determined based on the updated 3D Gaussian model and the multi-view image sequence, and based on the depth information and the rendered image, a 3D reconstruction algorithm is used to construct the surface model of the target scene. This method can more accurately reflect the structure and details of the target scene, avoid the occurrence of floating objects, and significantly improve the quality and accuracy of scene reconstruction.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, and in particular to a method, apparatus, device and medium for scene surface reconstruction. Background Technology

[0002] With the rapid development of Virtual Reality (VR) and Augmented Reality (AR) technologies, 3D Gaussian Splatting (3DGS) technology has emerged. 3DGS technology can explicitly represent 3D scenes using a 3D Gaussian sphere and synthesize new views of the 3D scene from any perspective through Gaussian splashing, resulting in high-quality visual effects and faster rendering speeds.

[0003] However, in practical applications, if the real image contains weakly textured areas with very smooth or almost constant pixel value changes, or poorly lit areas, the reconstructed scene may not be accurately aligned with the real objects. This can cause the 3D Gaussian sphere to appear to float in the air, forming incorrect "floating objects," resulting in a lower quality reconstructed scene. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method, apparatus, device, and medium for scene surface reconstruction, which can more accurately reflect the structure and details of the target scene, avoid the occurrence of floating objects, and significantly improve the quality and accuracy of scene reconstruction.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] In a first aspect, this application discloses a method for reconstructing a scene surface, the method comprising:

[0007] Given an initialized 3D Gaussian model obtained from a multi-view image sequence of the target scene, the attribute information and position information of the 2D Gaussian corresponding to the 3D anchor point Gaussian in the initialized 3D Gaussian model are determined.

[0008] Based on the attribute and position information of the two-dimensional Gaussian, a rendered image is generated using the Gaussian splashing method, wherein the rendered image includes color information, semantic information, depth information, and normal information.

[0009] Based on the rendered image and the images in the multi-view image sequence with the same viewpoint as the rendered image, the initialized 3D Gaussian model is updated by minimizing the rendering loss function to obtain the updated 3D Gaussian model. The rendering loss function includes a color loss function, a semantic loss function, a depth loss function, and a normal loss function.

[0010] Based on the updated 3D Gaussian model and the multi-view image sequence, the depth information of the target scene is determined;

[0011] Based on the depth information and the rendered image, a surface model of the target scene is constructed using a 3D reconstruction algorithm.

[0012] Optionally, determining the attribute information and position information of the two-dimensional Gaussian corresponding to the three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model includes:

[0013] The projection radius of each 3D anchor point Gaussian in the initialized 3D Gaussian model is determined by projecting each 3D anchor point Gaussian onto the image plane.

[0014] Determine the attribute information and position information of the two-dimensional Gaussian corresponding to the three-dimensional anchor point Gaussian with a projection radius greater than 0.

[0015] Optionally, determining the depth information of the target scene based on the updated 3D Gaussian model and the multi-view image sequence includes:

[0016] Determine the average value of the alpha mixture weights of the two-dimensional Gaussian corresponding to each three-dimensional anchor point Gaussian in the updated three-dimensional Gaussian model;

[0017] By deleting the three-dimensional anchor point Gaussians corresponding to the two-dimensional Gaussians whose average value is less than the first preset threshold, a three-dimensional Gaussian model after secondary update is obtained.

[0018] The depth information of the target scene is determined based on the second-updated 3D Gaussian model and the multi-view image sequence.

[0019] Optionally, determining the depth information of the target scene based on the updated 3D Gaussian model and the multi-view image sequence includes:

[0020] Generate an opacity map corresponding to the rendered image, wherein the opacity map represents the opacity of each pixel in the rendered image;

[0021] For a pixel whose opacity is less than a second preset threshold, determine the target pixel that is closest to the pixel and whose opacity is greater than the second preset threshold;

[0022] By back-projecting the target pixel into the updated 3D Gaussian model, a 3D Gaussian model with three updates is obtained.

[0023] The depth information of the target scene is determined based on the three-dimensional Gaussian model updated three times and the multi-view image sequence.

[0024] Optionally, the attribute information of the two-dimensional Gaussian includes color attribute information, geometric attribute information, semantic attribute information, and opacity attribute information; the attribute information of the two-dimensional Gaussian is determined as follows:

[0025] Determine the color feature vector and semantic feature vector of the target scene;

[0026] The color feature vector is input into the color fully connected network and the opacity fully connected network of the initialized 3D Gaussian model, respectively, to obtain the color attribute information and opacity attribute information.

[0027] The semantic feature vectors are input into the geometric fully connected network and the semantic fully connected network of the initialized 3D Gaussian model, respectively, to obtain the geometric attribute information and the opacity attribute information.

[0028] Optionally, the position information of the two-dimensional Gaussian is determined as follows:

[0029] Based on the multi-view image sequence of the target scene, a sparse point cloud of the target scene is generated;

[0030] Based on the coordinates of the sparse point cloud and the preset distance between Gaussian anchor points, determine the anchor point position of the three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model.

[0031] The position information of the two-dimensional Gaussian is determined based on the anchor point position and the position offset.

[0032] Optionally, the formulas for the depth loss function and the normal loss function are respectively shown below:

[0033]

[0034] Among them, L d-smooth For depth loss function, For a depth image, w h Let d(x) be the cosine similarity of the semantic features of adjacent pixels in the vertical direction. i,j Let be the pixel value at position (i,j) in the depth image, and w be the pixel value at position (i,j). w L is the cosine similarity of the semantic features of horizontally adjacent pixels. n-smooth The normal loss function is... For the normal graph, n(x) i,j) represents the pixel value at position (i,j) in the normal image.

[0035] Secondly, this application discloses a scene surface reconstruction device, the device comprising: a first determining module, an image generation module, a model updating module, a second determining module, and a model building module;

[0036] The first determining module is used to determine the attribute information and position information of the two-dimensional Gaussian corresponding to the three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model when an initialized three-dimensional Gaussian model is obtained based on the multi-view image sequence of the target scene.

[0037] The image generation module is used to generate a rendered image using the Gaussian splashing method based on the attribute information and position information of the two-dimensional Gaussian. The rendered image includes color information, semantic information, depth information, and normal information.

[0038] The model update module is used to update the initialized 3D Gaussian model by minimizing the rendering loss function based on the rendered image and images in the multi-view image sequence that have the same viewpoint as the rendered image, thereby obtaining the updated 3D Gaussian model. The rendering loss function includes a color loss function, a semantic loss function, a depth loss function, and a normal loss function.

[0039] The second determining module is used to determine the depth information of the target scene based on the updated three-dimensional Gaussian model and the multi-view image sequence;

[0040] The model building module is used to construct a surface model of the target scene based on the depth information and the rendered image using a 3D reconstruction algorithm.

[0041] Optionally, the first determining module is specifically used to: determine the projection radius of each three-dimensional anchor point Gaussian by projecting each three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model onto the image plane;

[0042] Determine the attribute information and position information of the two-dimensional Gaussian corresponding to the three-dimensional anchor point Gaussian with a projection radius greater than 0.

[0043] Optionally, the second determining module specifically includes: a first determining submodule, a second determining submodule, and a third determining submodule;

[0044] The first determining submodule is used to determine the average value of the alpha mixture weights of the two-dimensional Gaussian corresponding to each three-dimensional anchor point Gaussian in the updated three-dimensional Gaussian model;

[0045] The second determining submodule is used to obtain a three-dimensional Gaussian model after a second update by deleting the three-dimensional anchor point Gaussians corresponding to the two-dimensional Gaussians whose average value is less than the first preset threshold.

[0046] The third determining submodule is used to determine the depth information of the target scene based on the second-updated 3D Gaussian model and the multi-view image sequence.

[0047] Optionally, the second determining module specifically includes: a fourth determining submodule, a fifth determining submodule, a sixth determining submodule, and a seventh determining submodule;

[0048] The fourth determining submodule is used to generate an opacity map corresponding to the rendered image, wherein the opacity map represents the opacity of each pixel in the rendered image;

[0049] The fifth determination submodule is used to determine, for pixels whose opacity is less than the second preset threshold, the target pixel that is closest to the pixel and whose opacity is greater than the second preset threshold;

[0050] The sixth determining submodule is used to obtain a three-dimensional Gaussian model after three updates by back-projecting the target pixel into the updated three-dimensional Gaussian model.

[0051] The seventh determination submodule is used to determine the depth information of the target scene based on the three-dimensional Gaussian model after the three updates and the multi-view image sequence.

[0052] Optionally, the attribute information of the two-dimensional Gaussian includes color attribute information, geometric attribute information, semantic attribute information, and opacity attribute information; the specific unit for determining the attribute information of the two-dimensional Gaussian is as follows:

[0053] The first determining unit is used to determine the color feature vector and semantic feature vector of the target scene;

[0054] The second determining unit is used to input the color feature vector into the color fully connected network and the opacity fully connected network of the initialized three-dimensional Gaussian model respectively, and obtain the color attribute information and opacity attribute information respectively;

[0055] The third determining unit is used to input the semantic feature vector into the geometric fully connected network and the semantic fully connected network of the initialized three-dimensional Gaussian model, respectively, to obtain the geometric attribute information and the opacity attribute information.

[0056] Optionally, the unit for determining the position information of the two-dimensional Gaussian is as follows:

[0057] The fourth determining unit is used to generate a sparse point cloud of the target scene based on a multi-view image sequence of the target scene;

[0058] The fifth determining unit is used to determine the anchor point position of the three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model based on the coordinates of the sparse point cloud and the preset distance between Gaussian anchor points.

[0059] The sixth determining unit is used to determine the position information of the two-dimensional Gaussian based on the anchor point position and the position offset.

[0060] Optionally, the formulas for the depth loss function and the normal loss function are respectively shown below:

[0061]

[0062] Among them, L d-smooth For depth loss function, For a depth image, w h Let d(x) be the cosine similarity of the semantic features of adjacent pixels in the vertical direction. i,j Let be the pixel value at position (i,j) in the depth image, and w be the pixel value at position (i,j). w L is the cosine similarity of the semantic features of horizontally adjacent pixels. n-smooth The normal loss function is... For the normal graph, n(x) i,j ) represents the pixel value at position (i,j) in the normal image.

[0063] Thirdly, this application discloses a scene surface reconstruction device, the device comprising: a memory and a processor;

[0064] The memory is used to store programs;

[0065] The processor is configured to execute the program to implement the various steps of the scene surface reconstruction method as described in the first aspect.

[0066] Fourthly, this application discloses a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the various steps of the scene surface reconstruction method as described in the first aspect.

[0067] Compared with the prior art, this application has the following advantages:

[0068] This application provides a method, apparatus, device, and medium for scene surface reconstruction. The method first introduces an initialized 3D Gaussian model and, combined with a multi-view image sequence of the target scene, determines the attribute and position information of a 2D Gaussian model corresponding to the 3D anchor point Gaussian. Second, using the attribute and position information, a rendered image is generated through a Gaussian splashing method. Subsequently, based on the rendered image and images from the multi-view image sequence with the same viewpoint as the rendered image, the initialized 3D Gaussian model is updated by minimizing the rendering loss function. Finally, depth information is determined based on the updated 3D Gaussian model and the multi-view image sequence. Based on the depth information and the rendered image, a 3D reconstruction algorithm is used to construct the surface model of the target scene. This method can more accurately reflect the structure and details of the target scene, avoids the occurrence of floating objects, and significantly improves the quality and accuracy of scene reconstruction. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 A flowchart illustrating a scene surface reconstruction method provided in this application embodiment;

[0071] Figure 2 A flowchart of another scene surface reconstruction method provided in the embodiments of this application;

[0072] Figure 3 This is a schematic diagram of a scene surface reconstruction device provided in an embodiment of this application;

[0073] Figure 4 This is a schematic diagram of a computer-readable medium provided in an embodiment of this application. Detailed Implementation

[0074] First, let me explain the technical terms used in this application:

[0075] 3D Gaussian Splatting (3DGS) is an advanced rendering technique used in computer vision and graphics, particularly suitable for reconstructing high-quality 3D scenes from a set of sparse 2D images. 3DGS utilizes a Gaussian distribution to represent local features in the scene and renders them by "splattering" these Gaussian functions onto the image plane, thus generating realistic views. In 3DGS, each 3D Gaussian sphere represents a local feature in the scene, such as color or shape. It is defined by multiple parameters, including position, size (standard deviation), and orientation. These Gaussian spheres are used to approximate the surfaces of objects in the scene.

[0076] As described earlier, in practical applications, if the real image contains weakly textured areas with very smooth or almost constant pixel value changes, or poorly lit areas, the reconstructed scene may not be accurately aligned with the real objects. This can cause the 3D Gaussian sphere to appear to float in the air, forming incorrect "floating objects," resulting in a lower quality reconstructed scene.

[0077] The inventors, through research, proposed a method, apparatus, device, and medium for scene surface reconstruction. This method first introduces an initialized 3D Gaussian model and, combined with a multi-view image sequence of the target scene, determines the attribute and position information of the 2D Gaussian model corresponding to the 3D anchor point Gaussian. Second, using the attribute and position information, a rendered image is generated through a Gaussian splashing method. Subsequently, based on the rendered image and images from the multi-view image sequence with the same viewpoint as the rendered image, the initialized 3D Gaussian model is updated by minimizing the rendering loss function. Finally, depth information is determined based on the updated 3D Gaussian model and the multi-view image sequence. Based on the depth information and the rendered image, a 3D reconstruction algorithm is used to construct the surface model of the target scene, which more accurately reflects the structure and details of the target scene, avoids the occurrence of floating objects, and significantly improves the quality and accuracy of scene reconstruction. Furthermore, based on the updated 3D Gaussian model, this method further determines the depth information of the target scene through various strategies (such as average filtering using alpha mixed weights and utilizing opacity maps), which helps to eliminate inaccurate depth information and retain more reliable depth data, again significantly improving the quality and accuracy of scene reconstruction.

[0078] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0079] See Figure 1 This figure is a flowchart of a scene surface reconstruction method provided in an embodiment of this application. The method includes:

[0080] S101: Given an initialized 3D Gaussian model obtained from a multi-view image sequence of the target scene, determine the attribute and position information of the 2D Gaussian corresponding to the 3D anchor point Gaussian in the initialized 3D Gaussian model.

[0081] Based on a multi-view image sequence of the target scene (the scene requiring surface reconstruction), an initialized 3D Gaussian model is generated using the Structure From Motion (SFM) method. This initialized 3D Gaussian model includes several 3D anchor-point Gaussians. A 3D anchor-point Gaussian refers to a Gaussian distribution centered on a set of anchor points in 3D space. Each anchor point represents a point in 3D space, and the Gaussian distribution describes the uncertainty of that point in space.

[0082] Subsequently, for each 3D anchor point Gaussian in the initialized 3D Gaussian model, its projection onto the 2D image is found using camera parameters and the 3D-to-2D projection relationship; this projection is the 2D Gaussian. Next, the attribute and position information of the 2D Gaussian is determined. The attribute information of the 2D Gaussian includes color attributes (describing the color information of the 2D Gaussian), geometric attributes (describing the shape and size of the 2D Gaussian), semantic attributes (describing the semantic information of the 2D Gaussian, such as object category, material, etc.), and opacity attributes (describing the transparency of the 2D Gaussian).

[0083] S102: Based on the attribute and position information of the two-dimensional Gaussian, a rendering image is generated using the Gaussian splashing method. The rendering image includes color information, semantic information, depth information, and normal information.

[0084] The rendered image generated using the Gaussian splashing method includes color information (the color value of each pixel), semantic information (the semantic label of the object category for each pixel), depth information (the depth value of each pixel), and normal information (the surface normal direction for each pixel).

[0085] As can be understood, S102 uses the example of first generating the color channel, semantic label channel, depth channel, and normal channel separately, and then summing the contributions of all two-dimensional Gaussian channels (i.e., color information in the color channel, semantic information in the semantic label channel, depth information in the depth channel, and normal information in the normal channel) to generate a final rendered image. In practical applications, it is also possible not to sum the contributions of all two-dimensional Gaussian channels separately, but to process each channel separately and generate a separate rendered image. semantic feature image Depth image and normal image (i.e., four images). This application does not limit this.

[0086] S103: Based on the rendered image and the images with the same viewpoint as the rendered image in the multi-view image sequence, update the initialized 3D Gaussian model by minimizing the rendering loss function to obtain the updated 3D Gaussian model. The rendering loss function includes the color loss function, semantic loss function, depth loss function and normal loss function.

[0087] In some specific implementations, the color loss function can be represented by the following formula (1):

[0088]

[0089] in, Let λ be the color loss function, and λ be the weight. The loss measures the absolute difference between the pixel values ​​of the rendered image and scene images with the same viewpoint as the rendered image in the multi-view image sequence, while The loss factor takes into account the structural similarity between the rendered image and scene images in the multi-view image sequence that share the same viewpoint as the rendered image.

[0090] Understandably, λ is used to adjust the proportion of L1 loss and D-SSIM loss in the total color loss, and is usually set to 0.7 (meaning that L1 loss accounts for a large proportion of the total color loss, but D-SSIM loss is also taken into account to balance the brightness / color difference and structural similarity of the image). This application does not limit the specific value of λ.

[0091] In some specific implementations, the semantic loss function can be represented by the following formula (2):

[0092]

[0093] in, For semantic loss function, For semantic information, v is the true category of the pixel, and i is all categories of the pixel. Formula (2) is the cross-entropy loss function.

[0094] In some specific implementations, the depth loss function and the normal loss function can be represented by the following formulas (3) and (4), respectively:

[0095]

[0096] Among them, L d-smooth For depth loss function, For depth information, wh The cosine similarity of the semantic features of vertically adjacent pixels is calculated using the following formula (5): d(x i,j Let be the pixel value at position (i,j) in the depth image, and w be the pixel value at position (i,j). w The cosine similarity of the semantic features of horizontally adjacent pixels is calculated using the following formula (6), where L n-smooth The normal loss function is... For normal information, n(x) i,j ) represents the pixel value at position (i,j) in the normal image.

[0097]

[0098] Among them, S i,j Let (i,j) be the semantic feature of pixel (i,j).

[0099] After obtaining the color loss, semantic loss, depth loss, and normal loss values ​​using the formulas above, these values ​​can be minimized using gradient descent or other optimization algorithms to update the initialized 3D Gaussian model, resulting in an updated 3D Gaussian model. This updated 3D Gaussian model can more accurately represent the 3D structure, color information, semantic information, depth information, and normal information of the target scene.

[0100] Understandably, by minimizing the depth loss and normal loss values, it is possible to improve rendering quality by semantically constraining the Gaussians of the same object to have similar depths and smooth normals.

[0101] S104: Determine the depth information of the target scene based on the updated 3D Gaussian model and the multi-view image sequence.

[0102] Depth information reveals the distances of points in the target scene relative to the observer (or camera). The updated 3D Gaussian model provides 3D structural information about the target scene, while the multi-view image sequence provides observation data from different perspectives. Combining the two allows for a more accurate estimation of depth information.

[0103] S105: Construct a surface model of the target scene using a 3D reconstruction algorithm based on depth information and rendered images.

[0104] The surface model of a target scene is typically a three-dimensional mesh (such as a triangular mesh, quadrilateral mesh, etc.) that approximates the surface of objects in the target scene. Specifically, the coordinates of each pixel in the depth information and their positions in the rendered image are calculated. These points are then connected to form a triangular or quadrilateral mesh to approximate the surface of the object, thereby constructing the surface model of the target scene.

[0105] Understandably, because rendered images include semantic information, this information enables the surface model of the target scene to understand and distinguish different objects and regions. This helps to more accurately identify and manipulate objects in the scene in subsequent applications, thereby enhancing the overall understanding of the scene. Furthermore, during 3D reconstruction, ambiguity or errors may arise due to factors such as occlusion and noise. Semantic information can help to more accurately judge and handle these situations, thereby improving the geometric accuracy and detail of the model.

[0106] In summary, this application discloses a scene surface reconstruction method. First, an initialized 3D Gaussian model is introduced, and combined with a multi-view image sequence of the target scene, the attribute and position information of the 2D Gaussian model corresponding to the 3D anchor point Gaussian model is determined. Second, using the attribute and position information, a rendered image is generated through a Gaussian splashing method. Subsequently, based on the rendered image and images with the same viewpoint as the rendered image in the multi-view image sequence, the initialized 3D Gaussian model is updated by minimizing the rendering loss function. Finally, depth information is determined based on the updated 3D Gaussian model and the multi-view image sequence, and based on the depth information and the rendered image, a 3D reconstruction algorithm is used to construct the surface model of the target scene. This method can more accurately reflect the structure and details of the target scene, avoid the occurrence of floating objects, and significantly improve the quality and accuracy of scene reconstruction.

[0107] See Figure 2 This figure is a flowchart of another scene surface reconstruction method provided in an embodiment of this application. The method includes:

[0108] S201: Acquire a multi-view image sequence of the target scene.

[0109] In one specific implementation, electronic devices such as mobile phones and cameras can be used to capture a series of scene images of the target scene, obtaining a multi-view image sequence I = {I...} k |k=1,2,…N}. It should be noted that the above scene images can be images obtained by a series of electronic devices in different positions simultaneously capturing the target scene, or images obtained by a single electronic device capturing the target scene while moving. This application does not limit the specific capturing method.

[0110] In another specific implementation, electronic devices such as mobile phones and cameras can be used to capture scene videos of the target scene, and by performing frame extraction processing on the scene videos, a multi-view image sequence I = {I...} can be obtained. k |k=1,2,…N}. For example, after acquiring scene video of the target scene, OpenCV software can be used to perform frame extraction processing on the scene video. For instance, images with a 5-frame interval can be selected as keyframe images, and the keyframe image sequence can be used as a multi-view image sequence. It should be noted that images with intervals of 3 frames, 10 frames, or other frame numbers can also be selected as keyframe images; this application does not limit the specific number of frames.

[0111] It should also be noted that all images in the multi-view image sequence are images of the same target scene from different angles. Furthermore, all the above images can be RGB images or images in other formats; this application does not limit the specific image format.

[0112] S202: Preprocess the multi-view image sequence to obtain camera parameters, sparse point cloud of the target scene, and semantic segmentation map of each scene image.

[0113] Camera parameters can be categorized into intrinsic camera parameters, extrinsic camera parameters, near-plane parameters, and far-plane parameters. Intrinsic camera parameters represent the camera's fixed parameters, typically a 3×3 matrix, including focal length and principal point. Extrinsic camera parameters represent the camera's rotation and position relative to the world coordinate system, typically a 4×4 matrix, including rotation and translation vectors. Near-plane and far-plane parameters define the depth range of the target scene, helping to determine which parts should be included during scene surface reconstruction. Sparse point clouds represent key feature points in the target scene and their 3D coordinates. In one specific implementation, COLMAP software can be used to perform sparse reconstruction on multi-view image sequences to obtain camera parameters and sparse point clouds.

[0114] Semantic segmentation maps classify each scene image at the pixel level, obtaining information about which pixel belongs to which object category (e.g., walls, furniture). In one specific implementation, the Grounded SAM model can be used to perform semantic segmentation on each scene image in a multi-view image sequence, obtaining a semantic segmentation map for each scene image.

[0115] S203: Generate an initial 3D Gaussian model based on the sparse point cloud of the target scene.

[0116] The initialized 3D Gaussian model includes several 3D anchor point Gaussians. Each 3D anchor point Gaussian includes the following information: anchor point position P. anchor(representing the position of the Gaussian sphere in three-dimensional space), color feature vector f c (Description of color information of the Gaussian sphere), semantic feature vector f s (Semantic information describing the object category to which the Gaussian sphere belongs), and the learnable position offset p offset (Each position offset is a three-dimensional vector, and the position offsets allow the position of the Gaussian sphere to be adjusted during training to better fit the scene.)

[0117] The initialized 3D Gaussian model also includes four fully connected networks (Multilayer Perceptron, MLP), including a color fully connected network MLP. color Geometric Fully Connected Network MLP geometry Semantic Fully Connected Network (MLP) semantic and opaque fully connected network MLP opacity .

[0118] S204: Filter the 3D anchor point Gaussians in the initialized 3D Gaussian model to obtain the filtered 3D anchor point Gaussians.

[0119] For each Gaussian 3D anchor point in the initial scene model of the target scene, calculate its radius radii projected onto the image plane. The formula for calculating radii is shown in formula (7):

[0120] radii=3*max(δ1,δ2)(7)

[0121] Where radii is the radius of the Gaussian projection of the three-dimensional anchor point onto the image plane, and δ1 and δ2 are the eigenvalues ​​of the covariance matrix Σ′ of the two-dimensional Gaussian projection onto the image plane. Specifically, the formula for the covariance matrix Σ′ can be shown in formula (8) below:

[0122] Σ′=JWRXX T R T W T J T (8)

[0123] Where Σ′ is the covariance matrix, J is the Jacobian matrix, W is the transformation matrix from world coordinates to image plane coordinates, R is the identity matrix (obtained from the quaternion r), and X is a 3×3 matrix representing the diagonal matrix diag(D anchor D anchor D anchor Specifically, J and W are calculated using camera parameters (including camera intrinsic and camera extrinsic parameters).

[0124] If radii ≤ 0, then the Gaussian sphere at the 3D anchor point will not participate in the subsequent rendering process. This significantly improves the algorithm's efficiency because unnecessary calculations on invalid Gaussian spheres are eliminated.

[0125] S205: Based on the fully connected network, determine the attribute information and position information of the two-dimensional Gaussian corresponding to the filtered three-dimensional anchor point Gaussian. The attribute information of the two-dimensional Gaussian includes color attribute information, geometric attribute information, semantic attribute information, and opacity attribute information.

[0126] In some specific implementations, the position information μ of the two-dimensional Gaussian is determined as follows:

[0127] First, based on the preset distance D between Gaussian anchor points anchor (e.g., 0.5 unit length), initialize anchor point position P anchor The specific formula is shown in formula (9) below:

[0128]

[0129] Among them, P anchor For anchor point positions, the Unique function indicates the deletion of duplicate anchor point positions P. anchor , P represents rounding down. colmap Let P be the coordinates of the sparse point cloud. From formula (9), we know that the anchor point position P... anchor D is the distance between Gaussian anchor points. anchor The integer multiple of. Therefore, the preset Gaussian anchor point distance D... anchor and COLMAP sparse point cloud coordinates P colmap It can calculate and remove duplicates to obtain unique and uniformly distributed anchor point positions P. anchor .

[0130] Subsequently, based on the aforementioned anchor point position P anchor and learnable position offset p offset The position information μ of the two-dimensional Gaussian can be obtained, and the specific formula is shown in formula (10) below:

[0131] μ = p offset +P anchor (10)

[0132] In some specific implementations, the method for determining the attribute information of a two-dimensional Gaussian is as follows:

[0133] First, the color feature vector f c Input color fully connected network MLP color In the process, a three-dimensional vector c (representing RGB color) is output, and the specific formula is shown in formula (11) below:

[0134] c = MLP color (f c (11)

[0135] Secondly, the semantic feature vector f s Input geometry fully connected network MLP geometry In the process, a four-dimensional vector r (representing the quaternion of rotation) and a two-dimensional vector s (representing the two scale factors s1 and s2 of the elliptical patch) are output, as shown in the following formula (12):

[0136] r,s=MLP geometry (f s (12)

[0137] Then, the color feature vector f c Input Opacity Fully Connected Network MLP opacity In the output, a scalar o (representing opacity) is given, as shown in formula (13) below:

[0138] o=MLP opacity (f c (13)

[0139] Finally, the semantic feature vector f s Input Semantic Fully Connected Network (MLP) semantic In the process, a 64-dimensional multidimensional vector S (representing semantic features) is output, and the specific formula is shown in formula (14) below:

[0140] S = MLP semantic (f s (14)

[0141] Therefore, color attributes, geometric attributes, semantic attributes, and opacity attributes (i.e., two-dimensional Gaussian attribute information) can be generated through different MLP networks.

[0142] S206: Based on the positional and attribute information of the two-dimensional Gaussian, generate a rendered image, a semantic feature image, a depth image, and a normal image using the Gaussian splashing algorithm.

[0143] In some specific implementations, the generated rendering image semantic feature image Depth image and normal image The formulas can be shown as follows: Formula (15) - Formula (18)

[0144]

[0145] Among them, c iLet α be the color of the i-th two-dimensional Gaussian color. i It is the opacity factor, T i S represents the cumulative transmittance. i Let d be the semantic feature of the i-th two-dimensional Gaussian. i Let n be the depth of the intersection point of the emitted ray from the pixel and the two-dimensional Gaussian ray. i Let be the normal of a two-dimensional Gaussian vector. Where the opacity factor α... i =o i G i (u), u=[u,v] T Let x be the intersection point of the ray projected from pixel x on the corresponding two-dimensional Gaussian plane. It is a Gaussian distribution function.

[0146] S207: Based on the rendered image and scene images with the same viewpoint as the rendered image in the multi-view image sequence, determine the color loss value by calculating the L1 loss function and the D-SSIM structural dissimilarity loss function.

[0147] It is understandable that steps S207-S210 are similar to step S103, and will not be described again here.

[0148] S208: Based on the semantic feature image and the semantic segmentation map of scene images with the same viewpoint as the rendered image in the multi-view image sequence, determine the semantic loss value by calculating the pixel-wise cross-entropy loss function.

[0149] S209: Based on the depth image and normal image, determine the depth loss value and normal loss value by calculating the depth loss function and normal loss function respectively.

[0150] It is understandable that the depth smoothing loss value and the normal smoothing loss value make regions with similar semantic features have similar values ​​in the rendered depth image and normal image, respectively.

[0151] S210: Update the initialized 3D Gaussian model by minimizing the color loss, semantic loss, depth loss and normal loss, and obtain the updated 3D Gaussian model.

[0152] S211: Determine the depth information of the target scene based on the updated 3D Gaussian model and the multi-view image sequence.

[0153] During training, it is necessary to control the density of 3D anchor points:

[0154] In some specific implementations, firstly, the average value of the alpha mixture weights of the two-dimensional Gaussians corresponding to each three-dimensional anchor Gaussian in the updated three-dimensional Gaussian model is determined. Specifically, the alpha mixture weights of the two-dimensional Gaussians corresponding to the three-dimensional anchor Gaussians can be represented by the following formula (19):

[0155]

[0156] Among them, w g For alpha mixed weights, N k α represents the number of images that can be observed by this Gaussian. k It is the opacity factor, T k This represents the cumulative transmittance.

[0157] Subsequently, by deleting the three-dimensional anchor point Gaussians corresponding to the two-dimensional Gaussians whose average value is less than a first preset threshold, a second-updated three-dimensional Gaussian model is obtained. Less than the first preset threshold τ d The deletion of a 3D anchor point means that the anchor point likely represents a floating object or noise in the air, and therefore will be deleted.

[0158] Finally, based on the updated 3D Gaussian model and the multi-view image sequence, the depth information of the target scene is determined. This depth information can be a depth image. This application does not limit this.

[0159] In some other specific implementations, in order to maintain the density and accuracy of the target scene, firstly, an opacity map corresponding to the rendered image is generated. In this context, the opacity map represents the opacity of each pixel in the rendered image. Since infinity is not an objective feature in the target scene, the opacity map should not contain values ​​close to 0. Therefore, for pixels with opacity less than a second preset threshold (i.e., areas through which light passes have almost no Gaussian coverage), the target pixel closest to the pixel and with an opacity greater than the second preset threshold is identified. Subsequently, the target pixel is back-projected into the updated 3D Gaussian model to obtain a 3D Gaussian model that has been updated three times. Finally, based on the 3D Gaussian model updated three times and the multi-view image sequence, the depth information of the target scene is determined. This depth information can be a depth image. This application does not limit this aspect.

[0160] It should be noted that for the two methods of controlling the density of the three-dimensional anchor points mentioned above, both methods can be implemented, or one method can be selected for implementation. This application does not limit this.

[0161] S212: Construct a surface model of the target scene using a 3D reconstruction algorithm based on depth information and rendered images.

[0162] Specifically, the coordinates of each pixel in the depth information, along with their positions in the rendered image, can be calculated. Then, using Delaunay triangulation, Poisson surface reconstruction, or Marching Cubes algorithms, these points are connected to form a triangular or quadrilateral mesh to approximate the surface of the object, thereby constructing a surface model of the target scene.

[0163] It should be noted that, in order to further improve the accuracy of the surface model of the target scene, the faces in the triangular or quadrilateral mesh can be filtered, for example, the top 500 clusters with the largest number of faces can be selected to eliminate inaccurate surfaces and floating objects in the air.

[0164] Understandably, similar to the first embodiment, since the rendered image includes semantic information, this information enables the surface model of the target scene to understand and distinguish different objects and regions. This helps to more accurately identify and manipulate objects in the scene in subsequent applications, thereby enhancing the overall understanding of the scene. Furthermore, during 3D reconstruction, ambiguity or errors may arise due to factors such as occlusion and noise. Semantic information can help to more accurately judge and handle these situations, thereby improving the geometric accuracy and detail representation of the model.

[0165] In summary, this application discloses a scene surface reconstruction method. First, an initialized 3D Gaussian model is introduced, and combined with a multi-view image sequence of the target scene, the attribute and position information of the 2D Gaussian model corresponding to the 3D anchor point Gaussian is determined. Second, using the attribute and position information, a rendered image is generated through a Gaussian splashing method. Subsequently, based on the rendered image and images with the same viewpoint as the rendered image in the multi-view image sequence, the initialized 3D Gaussian model is updated by minimizing the rendering loss function. Finally, depth information is determined based on the updated 3D Gaussian model and the multi-view image sequence, and a 3D reconstruction algorithm is used to construct the surface model of the target scene based on the depth information and the rendered image. This method can more accurately reflect the structure and details of the target scene, avoid the occurrence of floating objects, and significantly improve the quality and accuracy of scene reconstruction. Furthermore, based on the updated 3D Gaussian model, this scene surface reconstruction method further determines the depth information of the target scene through various strategies (such as average filtering of alpha mixed weights and utilization of opacity maps), which helps to eliminate inaccurate depth information and retain more reliable depth data, thus significantly improving the quality and accuracy of scene reconstruction.

[0166] See Figure 3The figure shows a scene surface reconstruction device provided in an embodiment of this application. The scene surface reconstruction device 300 includes: a first determining module 301, an image generation module 302, a model updating module 303, a second determining module 304, and a model building module 305.

[0167] The first determining module 301 is used to determine the attribute information and position information of the two-dimensional Gaussian corresponding to the three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model when an initialized three-dimensional Gaussian model is obtained based on the multi-view image sequence of the target scene.

[0168] The image generation module 302 is used to generate a rendered image using the Gaussian splashing method based on the attribute information and position information of the two-dimensional Gaussian. The rendered image includes color information, semantic information, depth information and normal information.

[0169] The model update module 303 is used to update the initialized 3D Gaussian model by minimizing the rendering loss function based on the rendered image and images with the same viewpoint as the rendered image in the multi-view image sequence. The rendering loss function includes color loss function, semantic loss function, depth loss function and normal loss function.

[0170] The second determining module 304 is used to determine the depth information of the target scene based on the updated three-dimensional Gaussian model and the multi-view image sequence.

[0171] The model building module 305 is used to construct a surface model of the target scene based on depth information and rendered images using a 3D reconstruction algorithm.

[0172] In some specific implementations, the first determining module 301 is specifically used to: determine the projection radius of each three-dimensional anchor point Gaussian by projecting each three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model onto the image plane; and determine the attribute information and position information of the two-dimensional Gaussian corresponding to the three-dimensional anchor point Gaussian with a projection radius greater than 0.

[0173] In some specific implementations, the second determining module 304 specifically includes: a first determining submodule, a second determining submodule, and a third determining submodule;

[0174] The first determining submodule is used to determine the average value of the alpha mixture weights of the two-dimensional Gaussian corresponding to each three-dimensional anchor point Gaussian in the updated three-dimensional Gaussian model;

[0175] The second determining submodule is used to obtain a three-dimensional Gaussian model after a second update by deleting the three-dimensional anchor point Gaussians corresponding to the two-dimensional Gaussians whose average value is less than the first preset threshold.

[0176] The third determination submodule is used to determine the depth information of the target scene based on the second-updated 3D Gaussian model and the multi-view image sequence.

[0177] In some specific implementations, the second determining module 304 specifically includes: a fourth determining submodule, a fifth determining submodule, a sixth determining submodule, and a seventh determining submodule;

[0178] The fourth determination submodule is used to generate an opacity map corresponding to the rendered image, wherein the opacity map represents the opacity of each pixel in the rendered image;

[0179] The fifth determination submodule is used to determine the target pixel that is closest to the pixel and has an opacity greater than the second preset threshold for pixels with opacity less than the second preset threshold.

[0180] The sixth determination submodule is used to obtain the three-dimensional Gaussian model after three updates by back-projecting the target pixels into the updated three-dimensional Gaussian model.

[0181] The seventh determination submodule is used to determine the depth information of the target scene based on the three-dimensional Gaussian model updated three times and the multi-view image sequence.

[0182] In some specific implementations, the attribute information of a two-dimensional Gaussian includes color attribute information, geometric attribute information, semantic attribute information, and opacity attribute information; the specific unit for determining the attribute information of a two-dimensional Gaussian is as follows:

[0183] The first determining unit is used to determine the color feature vector and semantic feature vector of the target scene;

[0184] The second determining unit is used to input the color feature vector into the color fully connected network and the opacity fully connected network of the initialized three-dimensional Gaussian model respectively, and obtain color attribute information and opacity attribute information respectively.

[0185] The third determining unit is used to input the semantic feature vector into the geometric fully connected network and the semantic fully connected network of the initialized 3D Gaussian model, respectively, to obtain geometric attribute information and opacity attribute information.

[0186] In some specific implementations, the unit for determining the position information of a two-dimensional Gaussian is as follows:

[0187] The fourth determining unit is used to generate a sparse point cloud of the target scene based on the multi-view image sequence of the target scene;

[0188] The fifth determining unit is used to determine the anchor point position of the three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model based on the coordinates of the sparse point cloud and the preset distance between Gaussian anchor points.

[0189] The sixth determining unit is used to determine the position information of the two-dimensional Gaussian based on the anchor point position and the position offset.

[0190] In some specific implementations, the formulas for the depth loss function and the normal loss function are shown in formulas (20) and (21) respectively:

[0191]

[0192]

[0193] Among them, L d-smooth For depth loss function, For a depth image, w h Let d(x) be the cosine similarity of the semantic features of adjacent pixels in the vertical direction. i,j Let be the pixel value at position (i,j) in the depth image, and w be the pixel value at position (i,j). w L is the cosine similarity of the semantic features of horizontally adjacent pixels. n-smooth The normal loss function is... For the normal graph, n(x) i,j ) represents the pixel value at position (i,j) in the normal image.

[0194] In summary, this application discloses a scene surface reconstruction device. First, an initialized 3D Gaussian model is introduced, and combined with a multi-view image sequence of the target scene, the attribute and position information of the 2D Gaussian model corresponding to the 3D anchor point Gaussian is determined. Second, using the attribute and position information, a rendered image is generated through a Gaussian splashing device. Subsequently, based on the rendered image and images with the same viewpoint as the rendered image in the multi-view image sequence, the initialized 3D Gaussian model is updated by minimizing the rendering loss function. Finally, depth information is determined based on the updated 3D Gaussian model and the multi-view image sequence, and based on the depth information and the rendered image, a 3D reconstruction algorithm is used to construct the surface model of the target scene. This more accurately reflects the structure and details of the target scene, avoids the occurrence of floating objects, and significantly improves the quality and accuracy of scene reconstruction. Furthermore, based on the updated 3D Gaussian model, this scene surface reconstruction device further determines the depth information of the target scene through various strategies (such as average filtering of alpha mixed weights and utilization of opacity maps), which helps to eliminate inaccurate depth information and retain more reliable depth data, thus significantly improving the quality and accuracy of scene reconstruction.

[0195] This application also provides a corresponding scene surface reconstruction device and a computer-readable medium for implementing the scene surface reconstruction method provided in this application.

[0196] The scene surface reconstruction device includes a memory and a processor. The memory is used to store instructions or code, and the processor is used to execute the instructions or code so that the device performs a scene surface reconstruction method according to any embodiment of this application.

[0197] See Figure 4 This figure is a schematic diagram of a computer-readable medium provided in an embodiment of this application. The computer-readable medium 400 stores a computer program 411, which, when executed by a processor, implements the above-described... Figure 1 The steps of the scene surface reconstruction method.

[0198] It should be noted that, in the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0199] It should be noted that the machine-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0200] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0201] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0202] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0203] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for reconstructing a scene surface, characterized in that, The method includes: Given an initialized 3D Gaussian model obtained from a multi-view image sequence of the target scene, the attribute information and position information of the 2D Gaussian corresponding to the 3D anchor Gaussian in the initialized 3D Gaussian model are determined. The attribute information of the 2D Gaussian includes color attribute information, geometric attribute information, semantic attribute information, and opacity attribute information. The determination of the 2D Gaussian attribute information is as follows: The color feature vector and semantic feature vector of the target scene are determined; the color feature vector is input into the color fully connected network and the opacity fully connected network of the initialized 3D Gaussian model, respectively, to obtain the color attribute information and opacity attribute information; the semantic feature vector is input into the geometric fully connected network and the semantic fully connected network of the initialized 3D Gaussian model, respectively, to obtain the geometric attribute information and semantic attribute information. Based on the attribute and position information of the two-dimensional Gaussian, a rendered image is generated using the Gaussian splashing method, wherein the rendered image includes color information, semantic information, depth information, and normal information. Based on the rendered image and the images in the multi-view image sequence with the same viewpoint as the rendered image, the initialized 3D Gaussian model is updated by minimizing the rendering loss function to obtain the updated 3D Gaussian model. The rendering loss function includes a color loss function, a semantic loss function, a depth loss function, and a normal loss function. Based on the updated 3D Gaussian model and the multi-view image sequence, the depth information of the target scene is determined; Based on the depth information of the target scene and the rendered image, a surface model of the target scene is constructed using a 3D reconstruction algorithm.

2. The method according to claim 1, characterized in that, The determination of the attribute information and position information of the two-dimensional Gaussian corresponding to the three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model includes: The projection radius of each 3D anchor point Gaussian in the initialized 3D Gaussian model is determined by projecting each 3D anchor point Gaussian onto the image plane. Determine the attribute information and position information of the two-dimensional Gaussian corresponding to the three-dimensional anchor point Gaussian with a projection radius greater than 0.

3. The method according to claim 1, characterized in that, Determining the depth information of the target scene based on the updated 3D Gaussian model and the multi-view image sequence includes: Determine the average value of the alpha mixture weights of the two-dimensional Gaussian corresponding to each three-dimensional anchor point Gaussian in the updated three-dimensional Gaussian model; By deleting the three-dimensional anchor point Gaussians corresponding to the two-dimensional Gaussians whose average value is less than the first preset threshold, a three-dimensional Gaussian model after secondary update is obtained. The depth information of the target scene is determined based on the second-updated 3D Gaussian model and the multi-view image sequence.

4. The method according to claim 1, characterized in that, Determining the depth information of the target scene based on the updated 3D Gaussian model and the multi-view image sequence includes: Generate an opacity map corresponding to the rendered image, wherein the opacity map represents the opacity of each pixel in the rendered image; For a pixel whose opacity is less than a second preset threshold, determine the target pixel that is closest to the pixel and whose opacity is greater than the second preset threshold; By back-projecting the target pixel into the updated 3D Gaussian model, a 3D Gaussian model with three updates is obtained. The depth information of the target scene is determined based on the three-dimensional Gaussian model updated three times and the multi-view image sequence.

5. The method according to claim 1, characterized in that, The method for determining the position information of the two-dimensional Gaussian is as follows: Based on the multi-view image sequence of the target scene, a sparse point cloud of the target scene is generated; Based on the coordinates of the sparse point cloud and the preset distance between Gaussian anchor points, determine the anchor point position of the three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model. The position information of the two-dimensional Gaussian is determined based on the anchor point position and the position offset.

6. The method according to any one of claims 1-5, characterized in that, The formulas for the depth loss function and the normal loss function are shown below: ； ； in, For depth loss function, For depth information, Cosine similarity of semantic features between vertically adjacent pixels. Let be the pixel value at position (i, j) in the depth image. Cosine similarity of semantic features between horizontally adjacent pixels. The normal loss function is... For normal image, Let be the pixel value at position (i, j) in the normal image.

7. A scene surface reconstruction device, characterized in that, The device includes: a first determining module, an image generation module, a model updating module, a second determining module, and a model building module; The first determining module is used to determine the attribute information and position information of the two-dimensional Gaussian corresponding to the three-dimensional anchor point Gaussian in the initialized three-dimensional Gaussian model, given an initialized three-dimensional Gaussian model obtained from a multi-view image sequence of the target scene. The attribute information of the two-dimensional Gaussian includes color attribute information, geometric attribute information, semantic attribute information, and opacity attribute information. The unit for determining the attribute information of the two-dimensional Gaussian is specifically as follows: a first determining unit is used to determine the color feature vector and semantic feature vector of the target scene; a second determining unit is used to input the color feature vector into the color fully connected network and the opacity fully connected network of the initialized three-dimensional Gaussian model, respectively, to obtain the color attribute information and opacity attribute information; a third determining unit is used to input the semantic feature vector into the geometric fully connected network and the semantic fully connected network of the initialized three-dimensional Gaussian model, respectively, to obtain the geometric attribute information and semantic attribute information. The image generation module is used to generate a rendered image using the Gaussian splashing method based on the attribute information and position information of the two-dimensional Gaussian. The rendered image includes color information, semantic information, depth information, and normal information. The model update module is used to update the initialized 3D Gaussian model by minimizing the rendering loss function based on the rendered image and images in the multi-view image sequence that have the same viewpoint as the rendered image, thereby obtaining the updated 3D Gaussian model. The rendering loss function includes a color loss function, a semantic loss function, a depth loss function, and a normal loss function. The second determining module is used to determine the depth information of the target scene based on the updated three-dimensional Gaussian model and the multi-view image sequence; The model building module is used to construct a surface model of the target scene based on the depth information of the target scene and the rendered image using a 3D reconstruction algorithm.

8. A scene surface reconstruction device, characterized in that, The device includes: a memory and a processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the scene surface reconstruction method as described in any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements each step of the scene surface reconstruction method as described in any one of claims 1 to 6.

Images