Structural perception three-dimensional scene reconstruction method and device

Abstract

The invention discloses a three-dimensional scene reconstruction method and device based on structure perception. The method comprises the following steps: constructing a sparse point cloud generation model according to a diffusion model based on a Unet architecture; inputting the information into a sparse point cloud generation model, and generating camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding view angle; constructing a rough three-dimensional structure grid according to the sparse point cloud data; carrying out random sampling on the rough three-dimensional structure grid, and carrying out Gaussian field initialization on the Gaussian primitives according to a random sampling result to obtain an initialized Gaussian field; the loss between the current image frame and the RGB color image is calculated, parameters of Gaussian primitives of all the RGB image frames in the information are iteratively optimized in sequence according to the loss, all the optimized Gaussian primitives are combined, and a reconstructed three-dimensional scene is obtained. According to the invention, the spatial consistency and accuracy of reconstruction results are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional scene reconstruction, and in particular to a structure-aware three-dimensional scene reconstruction method and device. Background Art

[0002] In recent years, 3D scene reconstruction technology has made rapid progress in the fields of computer vision and graphics, and has shown wide application potential in the fields of virtual reality, augmented reality, film and television production, game development, architectural design, etc. In order to achieve more realistic 3D scene rendering effects, it is necessary to reconstruct a 3D scene model with high-precision geometric structure. However, scene reconstruction does face some significant challenges in terms of accurate geometric structure. The shortcomings of existing 3D scene reconstruction methods are as follows: 1) Scene reconstruction method based on neural radiation field: When dealing with large-scale and complex scenes, the geometric details of sparse texture areas are limited, and the reconstruction ability of the decoder may be limited for irregular geometric shapes, making it difficult for the model to accurately capture the geometric relationship of these areas; 2) Scene reconstruction method based on Gaussian: In some areas with high precision requirements, it is unable to capture enough geometric details, and the details processed in the initial stage are relatively few, resulting in some subtle geometric and texture details not being accurately captured. 3) GeoGaussian method: Its initialization relies on the smoothness assumption of the local point cloud area, which may lead to the loss of local geometric details when dealing with irregular surfaces.

[0003] Therefore, a structure-aware three-dimensional scene reconstruction method and device are developed to solve the above problems. Summary of the invention

[0004] The present invention proposes a structure-aware three-dimensional scene reconstruction method and device to solve the existing problem of difficulty in accurately capturing geometric details.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] The present invention provides a structure-aware three-dimensional scene reconstruction method, comprising:

[0007] Acquiring information, the information comprising a multi-view RGB image frame sequence of a scene;

[0008] Construct a sparse point cloud generation model based on the diffusion model based on the Unet architecture;

[0009] Inputting the information into the sparse point cloud generation model to generate camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding viewing angle;

[0010] constructing a rough three-dimensional structure grid according to the sparse point cloud data;

[0011] Randomly sampling the rough three-dimensional structure grid, and performing Gaussian field initialization on the Gaussian basis element according to the random sampling result to obtain an initialized Gaussian field;

[0012] A frame of RGB image frame is randomly selected from the information as the current image frame, the initialized Gaussian field is projected onto the imaging plane of the camera according to the current image frame and the camera internal and external parameters, an RGB color image rendered by the camera external parameters of the current image frame is generated, the loss between the current image frame and the RGB color image is calculated, the parameters of the Gaussian basis units of all RGB image frames in the information are iteratively optimized according to the loss, all the optimized Gaussian basis units are combined to obtain a reconstructed three-dimensional scene.

[0013] Specifically, the information is input into the sparse point cloud generation model to generate camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding viewing angle, including:

[0014] Through the geometric constraints between the RGB image frames of the scene's multiple perspectives, the camera's internal and external parameters of each perspective are restored based on the SIFT (Scale Invariant Feature Transform) feature point matching algorithm and the ICP (Iterative Closest Point) pose optimization algorithm.

[0015] Extracting multi-scale features of each of the RGB image frames based on a UNet encoder to obtain multi-view features;

[0016] The multi-view features are mapped to a three-dimensional space based on a diffusion model and a UNet decoder to generate the sparse point cloud data.

[0017] Specifically, constructing a rough three-dimensional structure grid according to the sparse point cloud data includes:

[0018] Performing spatial analysis on the sparse point cloud data, extracting the geometric position relationship and spatial distribution characteristics of each point in the sparse point cloud data, and obtaining the overall framework of the scene;

[0019] A triangulation algorithm or a surface fitting algorithm is used to transform the overall framework of the scene into a grid structure consisting of vertices, edges and faces, thereby constructing a rough three-dimensional structure grid.

[0020] Specifically, the rough three-dimensional structure grid is randomly sampled, and the Gaussian field is initialized for the Gaussian basis element according to the random sampling result to obtain the initialized Gaussian field, including:

[0021] Reconstructing the three-dimensional surface of the entire object in the scene according to the coarse three-dimensional structure grid;

[0022] Randomly sample 70% of the triangular faces of all face elements in the three-dimensional surface of the entire object in the scene, and randomly initialize weights of the three vertices on each triangular face;

[0023] Using the initialized weights, the vertices are interpolated to obtain the coordinates of the sampling points within the triangle surface;

[0024] Interpolate the normal of the vertex using the initialized weight to obtain the normal vector of the sampling point;

[0025] Randomly shifting the coordinates of the sampling points along the direction of the normal vector to obtain the center position of the Gaussian primitive;

[0026] The distance between the center position of the Gaussian primitive and the nearest Gaussian primitive is used as a scaling value, and the rotation value, opacity and spherical harmonic function of the Gaussian primitive are randomly initialized according to the scaling value to obtain an initialized Gaussian field.

[0027] Specifically, reconstructing the three-dimensional surface of the entire object in the scene according to the coarse three-dimensional structure grid includes:

[0028] segmenting the coarse three-dimensional grid into a voxel grid of fixed resolution;

[0029] Processing each voxel cube in the voxel grid one by one, marking each vertex of each voxel cube as internal or external according to the comparison result of the attribute values ​​of the eight vertices of each voxel cube with the object isosurface, forming an 8-bit binary number, wherein the binary number is used to look up a triangulation table, wherein the triangulation table predefines the number of triangles corresponding to each number and the connection mode of their vertices;

[0030] The intersection positions of the entire object surface and the edges of the voxel cube are calculated using a linear interpolation method. These intersection points will be used as the vertices of the triangles. This process is repeated to process the cubes in all voxel grids one by one to generate local triangles.

[0031] All the local triangles are stitched together to form a complete mesh model, generating a three-dimensional surface of the entire object in the scene.

[0032] Specifically, the rough three-dimensional structure grid is randomly sampled, and the Gaussian field is initialized on the Gaussian basis element according to the random sampling result to obtain the initialized Gaussian basis element, including: ,

[0033] ,

[0034] ,

[0035] ,

[0036] u, v, 1-uv are the weights of the three vertices p1, p2, and p3 on each sampled triangle face, respectively, which are randomly initialized. x is the sampling point, n p is the normal vector, d is the random offset distance, is the center position of the Gaussian primitive, and the distance to the nearest Gaussian primitive is used as the scaling value s, where The center position of the nearest Gaussian primitive, the initial rotation value q of each Gaussian primitive is set to the quaternion (1,0,0,0), and the spherical harmonic function h is initialized to (N,3,(d+1) 2 )-dimensional tensor, where N is the number of Gaussian basis elements, the order of spherical harmonics is d=3, the value of the tensor at (N,3,0) is 0.5, and the rest are 0, and the opacity α of each Gaussian basis element is initialized to 0.1.

[0037] Specifically, projecting the initialized Gaussian field onto an imaging plane of a camera according to the current image frame and the camera extrinsic parameters to generate an RGB color image rendered by the camera extrinsic parameters of the current image frame includes:

[0038] Calculate the center position of the Gaussian primitive of the current image frame according to the weights of the three vertices on the sampling triangle and the offset distance of the random offset;

[0039] The position, scale and rotation of the Gaussian basis element of the current scene are obtained by calculating the rotation and area scaling changes of the triangular face corresponding to the mesh model of the current image frame and the mesh model of the initialization scene, and adding them to the properties of the Gaussian basis element under the geometric form of the current scene;

[0040] The color image, the depth map and the normal map of the current image frame are rendered according to the position, scaling and rotation of the Gaussian primitives and the opacity of the current image frame.

[0041] Specifically, the Gaussian primitive of the current scene can be expressed as:

[0042] ,

[0043] in is the final position of the Gaussian primitive, s is the final scale of the Gaussian primitive, and q is the final rotation of the Gaussian primitive.

[0044] Specifically, calculating the loss between the current image frame and the RGB color image, and iteratively optimizing the parameters of the Gaussian primitives of all RGB image frames in the information according to the loss, including:

[0045] Perform L1 loss and mean square error loss on the color image of the current image frame and the RGB image rendered under the camera external parameters of the current frame , Perceptual Similarity Loss , Gaussian scaling loss and normal consistent loss L n ,in:

[0046] The L1 loss is:

[0047] ,

[0048] Where H, W, and C are the height, width, and number of channels of the image respectively. is the pixel value of the rendered image, The pixel value of the picture taken by the camera;

[0049] for:

[0050] ,

[0051] is the pixel value of the rendered image, The pixel value of the picture taken by the camera;

[0052] for:

[0053] ,

[0054] in represents the feature map of the i-th layer of image I in the VGG deep network, N is the number of feature layers for calculating the loss, To render the image, Take pictures for the camera;

[0055] for:

[0056] ,

[0057] in is the maximum value of the Gaussian basis element scaling, is the minimum value of the Gaussian basis element scaling, To specify the maximum scaling value, The maximum value of the ratio of the maximum scaling value to the minimum scaling value is specified;

[0058] for:

[0059] ,

[0060] Where N is the normal map derived from the rendered depth map, is the normal of the Gaussian primitive inside the camera frustum, is the weight of the i-th Gaussian basis.

[0061] The present invention also provides a structure-aware three-dimensional scene reconstruction device, comprising:

[0062] An acquisition module, the acquisition module is used to acquire information, the information including a multi-view RGB image frame sequence of a scene;

[0063] A first building module, wherein the first building module is used to build a sparse point cloud generation model according to a diffusion model based on a Unet architecture;

[0064] A generation module, the generation module is used to input the information into the sparse point cloud generation model to generate camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding viewing angle;

[0065] A second construction module, the second construction module is used to construct a rough three-dimensional structure grid according to the sparse point cloud data;

[0066] An initialization module, the initialization module is used to randomly sample the rough three-dimensional structure grid, and initialize the Gaussian field of the Gaussian basis element according to the random sampling result to obtain an initialized Gaussian field;

[0067] An optimization module is used to randomly select a frame of RGB image frames from the information as the current image frame, project the initialized Gaussian field to the imaging plane of the camera according to the current image frame and the camera internal and external parameters, generate an RGB color image rendered by the camera external parameters of the current image frame, calculate the loss between the current image frame and the RGB color image, iteratively optimize the parameters of the Gaussian primitives of all RGB image frames in the information according to the loss, combine all the optimized Gaussian primitives, and obtain a reconstructed three-dimensional scene.

[0068] The beneficial effects of the present invention are:

[0069] The present invention proposes a structure-aware three-dimensional scene reconstruction method and device, which solves the problem of being unable to accurately capture subtle geometric and texture details. It significantly improves the geometric accuracy and global consistency of sparse point clouds through a diffusion model, and improves the spatial consistency of reconstruction results by constructing a rough three-dimensional structure grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a flow chart of a structure-aware three-dimensional scene reconstruction method for the present application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0072] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0074] In the description of the present invention, it should be understood that the terms "upper", "lower", "inside", "outside", "left", "right", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use, or are directions or positional relationships commonly understood by those skilled in the art. These directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0075] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and should not be understood as indicating or implying relative importance.

[0076] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0077] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0078] like Figure 1 As shown, a structure-aware 3D scene reconstruction method includes:

[0079] S1: Acquire information, the information including a multi-view RGB image frame sequence of a scene;

[0080] S2: Construct a sparse point cloud generation model based on the diffusion model based on the Unet architecture;

[0081] S3: inputting the information into the sparse point cloud generation model to generate camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding viewing angle;

[0082] S4: constructing a rough three-dimensional structure grid according to the sparse point cloud data;

[0083] S5: randomly sampling the rough three-dimensional structure grid, and performing Gaussian field initialization on the Gaussian basis element according to the random sampling result to obtain an initialized Gaussian field;

[0084] S6: randomly selecting a RGB image frame from the information as the current image frame, projecting the initialized Gaussian field onto the imaging plane of the camera according to the current image frame and the camera internal and external parameters, generating an RGB color image rendered by the camera external parameters of the current image frame, calculating the loss between the current image frame and the RGB color image, iteratively optimizing the parameters of the Gaussian primitives of all RGB image frames in the information according to the loss, combining all the optimized Gaussian primitives, and obtaining a reconstructed three-dimensional scene.

[0085] Through the above-mentioned reconstructed three-dimensional scene, the RGB image frame of the new perspective can be rendered. For example, under the provided new perspective, the optimized Gaussian primitive center is projected to the camera imaging plane through the known camera intrinsic parameters and extrinsic parameters, and the two-dimensional position coordinates of each Gaussian primitive in the pixel space are calculated. The shape characteristics of the Gaussian primitive are determined by its final scaling parameters and rotation parameters. In the new perspective rendering process, each Gaussian primitive is represented as a two-dimensional Gaussian distribution on the imaging plane, and the shape of the distribution is jointly determined by the scaling, rotation and camera perspective relationship of the three-dimensional Gaussian primitive. The preliminary image under the new perspective is generated by weighted accumulation of the Gaussian primitive influence area of ​​each pixel, where the weight is related to the opacity, distribution characteristics and depth information of the Gaussian primitive, and the calculation formula is as follows:

[0086] ,

[0087] is the weight of the i-th Gaussian basis element; is the opacity, which is related to the depth of the primitive. The farther away the Gaussian kernel is, the higher its transparency is usually, and the smaller its contribution to reconstruction is. The degree of influence of the Gaussian kernel on the final reconstructed model is is the mean of the pixel depth covered by the Gaussian kernel; μ is the mean of the Gaussian kernel distribution; σ is the standard deviation of the Gaussian distribution, which controls the width of the Gaussian kernel;

[0088] Spherical harmonics (SH) are introduced to model the color information of Gaussian primitives. The color distribution and illumination interaction of Gaussian primitives are effectively represented by low-order polynomials, and the final color value of each pixel is generated through continuous mixing of color and opacity.

[0089] ,

[0090] is the color distribution based on i on the sphere; L, The order of spherical harmonics, are the spherical harmonic coefficients, which represent the distribution of colors at different orders and azimuths; is the spherical harmonic basis function, which represents the directional change of color.

[0091] The final rendered color is calculated using the following formula:

[0092] ,

[0093] is the opacity of the ith Gaussian basis element, C i is the color of the i-th Gaussian basis element, The final rendered color of the j-th point in the scene.

[0094] In some embodiments, the information is input into the sparse point cloud generation model to generate camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding viewing angle, including:

[0095] Through the geometric constraints between the RGB image frames of the scene's multiple perspectives, the camera's internal and external parameters of each perspective are restored based on the SIFT (Scale Invariant Feature Transform) feature point matching algorithm and the ICP (Iterative Closest Point) pose optimization algorithm.

[0096] Extracting multi-scale features of each of the RGB image frames based on a UNet encoder to obtain multi-view features;

[0097] The multi-view features are mapped to a three-dimensional space based on a diffusion model and a UNet decoder to generate the sparse point cloud data.

[0098] In some embodiments, constructing a coarse three-dimensional structure grid according to the sparse point cloud data includes:

[0099] Performing spatial analysis on the sparse point cloud data, extracting the geometric position relationship and spatial distribution characteristics of each point in the sparse point cloud data, and obtaining the overall framework of the scene;

[0100] A triangulation algorithm or a surface fitting algorithm is used to transform the overall framework of the scene into a grid structure consisting of vertices, edges and faces, thereby constructing a rough three-dimensional structure grid.

[0101] In some embodiments, the rough three-dimensional structure grid is randomly sampled, and Gaussian field initialization is performed on the Gaussian basis element according to the random sampling result to obtain the initialized Gaussian basis element, including:

[0102] Reconstructing the three-dimensional surface of the entire object in the scene according to the coarse three-dimensional structure grid;

[0103] Randomly sample 70% of the triangular faces of all face elements in the three-dimensional surface of the entire object in the scene, and randomly initialize weights of the three vertices on each triangular face;

[0104] Using the initialized weights, the vertices are interpolated to obtain the coordinates of the sampling points within the triangle surface;

[0105] Interpolate the normal of the vertex using the initialized weight to obtain the normal vector of the sampling point;

[0106] Randomly shifting the coordinates of the sampling points along the direction of the normal vector to obtain the center position of the Gaussian primitive;

[0107] The distance between the center position of the Gaussian primitive and the nearest Gaussian primitive is used as a scaling value, and the rotation value, opacity and spherical harmonic function of the Gaussian primitive are randomly initialized according to the scaling value to obtain an initialized Gaussian field.

[0108] In some embodiments, reconstructing the three-dimensional surface of the entire object in the scene according to the coarse three-dimensional structure grid includes:

[0109] segmenting the coarse three-dimensional grid into a voxel grid of fixed resolution;

[0110] Processing each voxel cube in the voxel grid one by one, marking each vertex of each voxel cube as internal or external according to the comparison result of the attribute values ​​of the eight vertices of each voxel cube with the object isosurface, forming an 8-bit binary number, wherein the binary number is used to look up a triangulation table, wherein the triangulation table predefines the number of triangles corresponding to each number and the connection mode of their vertices;

[0111] The intersection positions of the entire object surface and the edges of the voxel cube are calculated using a linear interpolation method. These intersection points will be used as the vertices of the triangles. This process is repeated to process the cubes in all voxel grids one by one to generate local triangles.

[0112] All the local triangles are stitched together to form a complete mesh model, generating a three-dimensional surface of the entire object in the scene.

[0113] In some embodiments, the rough three-dimensional structure grid is randomly sampled, and Gaussian field initialization is performed on the Gaussian basis element according to the random sampling result to obtain the initialized Gaussian basis element, including:

[0114] ,

[0115] ,

[0116] ,

[0117] ,

[0118] u, v, 1-uv are the weights of the three vertices p1, p2, and p3 on each sampled triangle face, respectively, which are randomly initialized. x is the sampling point, n p is the normal vector, d is the random offset distance, is the center position of the Gaussian primitive, and the distance to the nearest Gaussian primitive is used as the scaling value s, where The center position of the nearest Gaussian primitive, the initial rotation value q of each Gaussian primitive is set to the quaternion (1,0,0,0), and the spherical harmonic function h is initialized to (N,3,(d+1) 2 )-dimensional tensor, where N is the number of Gaussian basis elements, the order of spherical harmonics is d=3, the value of the tensor at (N,3,0) is 0.5, and the rest are 0, and the opacity α of each Gaussian basis element is initialized to 0.1.

[0119] In some embodiments, projecting the initialized Gaussian field onto an imaging plane of a camera according to the current image frame and the camera extrinsic parameters to generate an RGB color image rendered by the camera extrinsic parameters of the current image frame includes:

[0120] Calculate the center position of the Gaussian primitive of the current image frame according to the weights of the three vertices on the sampling triangle and the offset distance of the random offset;

[0121] The position, scale and rotation of the Gaussian basis element of the current scene are obtained by calculating the rotation and area scaling changes of the triangular face corresponding to the mesh model of the current image frame and the mesh model of the initialization scene, and adding them to the properties of the Gaussian basis element under the geometric form of the current scene;

[0122] The color image, the depth map and the normal map of the current image frame are rendered according to the position, scaling and rotation of the Gaussian primitives and the opacity of the current image frame.

[0123] In some embodiments, the Gaussian primitive representation of the current scene is:

[0124] ,

[0125] in is the position of the Gaussian basis element, s is the scale of the Gaussian basis element, and q is the rotation of the Gaussian basis element.

[0126] In some embodiments, calculating the loss between the current image frame and the RGB color image, and iteratively optimizing the parameters of the Gaussian primitives of all RGB image frames in the information according to the loss, comprises:

[0127] Perform L1 loss and mean square error loss on the color image of the current image frame and the RGB image rendered under the camera external parameters of the current frame , Perceptual Similarity Loss , Gaussian scaling loss and normal consistent loss L n ,in:

[0128] The L1 loss is:

[0129] ,

[0130] Where H, W, and C are the height, width, and number of channels of the image respectively. is the pixel value of the rendered image, The pixel value of the picture taken by the camera;

[0131] for:

[0132] ,

[0133] is the pixel value of the rendered image, The pixel value of the picture taken by the camera;

[0134] for:

[0135] ,

[0136] in represents the feature map of the i-th layer of image I in the VGG deep network, N is the number of feature layers for calculating the loss, To render the image, Take pictures for the camera;

[0137] for:

[0138] ,

[0139] in is the maximum value of the Gaussian basis element scaling, is the minimum value of the Gaussian basis element scaling, To specify the maximum scaling value, The maximum value of the ratio of the maximum scaling value to the minimum scaling value is specified;

[0140] L n for:

[0141] ,

[0142] Where N is the normal map derived from the rendered depth map, is the normal of the Gaussian primitive inside the camera frustum, is the weight of the i-th Gaussian basis element. This loss enables Gaussian to achieve a smoother surface with consistent geometry. The parameter weights u, v, offset distance d, scaling parameter s, rotation parameter q, spherical harmonic function h and opacity α are continuously optimized through the above loss. After the training, the final scene model can be obtained.

[0143] The total loss of training is:

[0144] + ,

[0145] Among them, the scene rendered image and the RGB image of the i-th frame are subjected to L1 loss. is the mean square error loss, is the perceptual similarity loss, is the Gaussian scaling loss, L n is the normal consistency loss, and λ1, λ2, λ3, λ4, and λ5 are all constants.

[0146] According to the RGB image frame of the information and the corresponding camera internal and external parameters and the initialization parameters of the Gaussian primitives, the random initialization weights of the sampling points and the random offset distances of the sampling points along the normal vector direction, a training strategy for adding random perturbation of the data includes:

[0147] An RGB image frame and the corresponding camera intrinsic and extrinsic parameters are randomly selected from the information.

[0148] According to the Gaussian field and the camera internal and external parameters, a rendering image of the current frame scene is obtained.

[0149] The parameters of the Gaussian field model are optimized by minimizing the RGB pixel loss between the current frame and the human body rendering image of the current frame.

[0150] In some examples, looping the above steps 30,000 times can output a scene model of good quality.

[0151] The embodiment of the present invention further provides a structure-aware three-dimensional scene reconstruction device, comprising:

[0152] An acquisition module, the acquisition module is used to acquire information, the information including a multi-view RGB image frame sequence of a scene;

[0153] A first building module, wherein the first building module is used to build a sparse point cloud generation model according to a diffusion model based on a Unet architecture;

[0154] A generation module, the generation module is used to input the information into the sparse point cloud generation model to generate camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding viewing angle;

[0155] A second construction module, the second construction module is used to construct a rough three-dimensional structure grid according to the sparse point cloud data;

[0156] An initialization module, wherein the initialization module is used to randomly sample the rough three-dimensional structure grid, and perform Gaussian field initialization on the Gaussian basis element according to the random sampling result to obtain the initialized Gaussian basis element;

[0157] An optimization module is provided, wherein the optimization module is used to randomly select a frame of RGB image frames from the information as a current image frame, project the initialized Gaussian primitives of the current RBG image frame to an imaging plane of a camera according to the current image frame and internal and external parameters of the camera, generate a color image, a depth map and a normal map of the current image frame, take the color image, the depth map and the normal map of the RGB image frame of the first frame scene in the information as a reference, calculate the loss between the color image, the depth map and the normal map of the current image frame and the color image, the depth map and the normal map of the RGB image frame of the first frame scene, iteratively optimize the parameters of the initialized Gaussian primitives of all RGB image frames in the information according to the loss, and obtain a reconstructed three-dimensional scene.

[0158] The advantages of the present invention compared to the prior art are:

[0159] 1. Introduce diffusion model and UNet architecture for sparse point cloud generation.

[0160] The present invention achieves high-precision reconstruction from multi-view images to three-dimensional sparse point clouds by combining the generation characteristics of the diffusion model's progressive denoising with the multi-scale feature extraction capability of the UNet network. The diffusion model gradually restores the scene structure during the noise optimization process, significantly improving the geometric accuracy and global consistency of the sparse point cloud.

[0161] 2. Gaussian primitive initialization guided by coarse grid model.

[0162] The present invention constructs a rough three-dimensional structure grid, performs random sampling on the triangular facets of the grid, and uses the vertex normal vectors of the facets for interpolation and offset to initialize the center, scaling and rotation parameters of the Gaussian primitives. This method ensures the rationality of the initial position of the Gaussian primitives and improves the spatial consistency of the reconstruction results.

[0163] 3. Gaussian primitive optimization under multiple loss constraints.

[0164] The present invention introduces multiple loss functions of color map, depth map and normal map, and combines normal consistency loss with scaling constraint loss to iteratively optimize the position, scaling, rotation and opacity of Gaussian primitives. Through the joint constraints of these geometric and color losses, the surface smoothness and geometric accuracy of Gaussian field reconstruction are ensured.

[0165] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A structure-aware three-dimensional scene reconstruction method, characterized in that: include: Acquiring information, the information comprising a multi-view RGB image frame sequence of a scene; Construct a sparse point cloud generation model based on the diffusion model based on the Unet architecture; Inputting the information into the sparse point cloud generation model to generate camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding viewing angle; constructing a rough three-dimensional structure grid according to the sparse point cloud data; Randomly sampling the rough three-dimensional structure grid, and performing Gaussian field initialization on the Gaussian basis element according to the random sampling result to obtain an initialized Gaussian field; A frame of RGB image frame is randomly selected from the information as the current image frame, the initialized Gaussian field is projected onto the imaging plane of the camera according to the current image frame and the camera internal and external parameters, an RGB color image rendered by the camera external parameters of the current image frame is generated, the loss between the current image frame and the RGB color image is calculated, the parameters of the Gaussian basis units of all RGB image frames in the information are iteratively optimized according to the loss, all the optimized Gaussian basis units are combined to obtain a reconstructed three-dimensional scene.

2. The structure-aware 3D scene reconstruction method according to claim 1, characterized in that: The information is input into the sparse point cloud generation model to generate camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding viewing angle, including: Through the geometric constraint relationship between the RGB image frames of the scene's multiple perspectives, based on the feature point matching algorithm and the pose optimization algorithm, the camera's internal and external parameters of each perspective are restored; Extracting multi-scale features of each of the RGB image frames based on a UNet encoder to obtain multi-view features; The multi-view features are mapped to a three-dimensional space based on a diffusion model and a UNet decoder to generate the sparse point cloud data.

3. The structure-aware 3D scene reconstruction method according to claim 1, characterized in that: Constructing a rough three-dimensional structure grid according to the sparse point cloud data, including: Performing spatial analysis on the sparse point cloud data, extracting the geometric position relationship and spatial distribution characteristics of each point in the sparse point cloud data, and obtaining the overall framework of the scene; A triangulation algorithm or a surface fitting algorithm is used to transform the overall framework of the scene into a grid structure consisting of vertices, edges and faces, thereby constructing a rough three-dimensional structure grid.

4. The structure-aware 3D scene reconstruction method according to claim 1, characterized in that: The rough three-dimensional structure grid is randomly sampled, and Gaussian field initialization is performed on Gaussian basis elements according to the random sampling result to obtain an initialized Gaussian field, including: Reconstructing the three-dimensional surface of the entire object in the scene according to the rough three-dimensional structure grid; Randomly sample 70% of the triangular faces of all face elements in the three-dimensional surface of the entire object in the scene, and randomly initialize weights of the three vertices on each triangular face; Using the initialized weights, the vertices are interpolated to obtain the coordinates of the sampling points within the triangle surface; Interpolate the normal of the vertex using the initialized weight to obtain the normal vector of the sampling point; Randomly shifting the coordinates of the sampling points along the direction of the normal vector to obtain the center position of the Gaussian primitive; The distance between the center position of the Gaussian primitive and the nearest Gaussian primitive is used as a scaling value, and the rotation value, opacity and spherical harmonic function of the Gaussian primitive are randomly initialized according to the scaling value to obtain an initialized Gaussian field.

5. The structure-aware 3D scene reconstruction method according to claim 4, characterized in that: Reconstructing the three-dimensional surface of the entire object in the scene according to the rough three-dimensional structure grid, comprising: segmenting the coarse three-dimensional grid into a voxel grid of fixed resolution; Processing each voxel cube in the voxel grid one by one, marking each vertex of each voxel cube as internal or external according to the comparison result of the attribute values ​​of the eight vertices of each voxel cube with the object isosurface, forming an 8-bit binary number, wherein the binary number is used to look up a triangulation table, wherein the triangulation table predefines the number of triangles corresponding to each number and the connection mode of their vertices; The intersection positions of the entire object surface and the edges of the voxel cube are calculated using a linear interpolation method. These intersection points will be used as the vertices of the triangles. This process is repeated to process the cubes in all voxel grids one by one to generate local triangles. All the local triangles are stitched together to form a complete mesh model, generating a three-dimensional surface of the entire object in the scene.

6. The structure-aware 3D scene reconstruction method according to claim 5, characterized in that: The rough three-dimensional structure grid is randomly sampled, and Gaussian field initialization is performed on the Gaussian basis element according to the random sampling result to obtain the initialized Gaussian basis element, including: ， ， ， ， u, v, 1-uv are the weights of the three vertices p1, p2, and p3 on each sampled triangle face, respectively, which are randomly initialized. x is the sampling point, n p is the normal vector, d is the random offset distance, is the center position of the Gaussian primitive, and the distance to the nearest Gaussian primitive is used as the scaling value s, where The center position of the nearest Gaussian primitive, the initial rotation value q of each Gaussian primitive is set to the quaternion (1,0,0,0), and the spherical harmonic function h is initialized to (N,3,(d+1) 2 )-dimensional tensor, where N is the number of Gaussian basis elements, the order of spherical harmonics is d=3, the value of the tensor at (N,3,0) is 0.5, and the rest are 0, and the opacity α of each Gaussian basis element is initialized to 0.

1.

7. The structure-aware 3D scene reconstruction method according to claim 6, characterized in that: The method projects the initialized Gaussian field onto an imaging plane of a camera according to the current image frame and the camera extrinsic parameters to generate an RGB color image rendered by the camera extrinsic parameters of the current image frame, including: Calculate the center position of the Gaussian primitive of the current image frame according to the weights of the three vertices on the sampling triangle and the offset distance of the random offset; The position, scaling and rotation of the Gaussian basis element of the current scene are obtained by calculating the rotation and scaling changes of the triangular face corresponding to the mesh model of the current image frame and the mesh model of the initialization scene, and adding them to the properties of the Gaussian basis element under the geometric form of the current scene; The color image, the depth map and the normal map of the current image frame are rendered according to the position, scaling and rotation of the Gaussian primitives and the opacity of the current image frame.

8. The structure-aware 3D scene reconstruction method according to claim 7, characterized in that: The Gaussian primitive representation of the current scene is: ， ， ， in is the position of the Gaussian element, is the scaling of the Gaussian basis, is the rotation of the Gaussian basis element, Δs and Δq respectively represent the changes in the rotation and area scaling of the triangle surface corresponding to the mesh model of the current scene and the mesh model of the initialization scene.

9. The structure-aware 3D scene reconstruction method according to claim 8, characterized in that: Calculating the loss between the current image frame and the RGB color image, and iteratively optimizing the parameters of the Gaussian primitives of all RGB image frames in the information according to the loss, including: Perform L1 loss and mean square error loss on the color image of the current image frame and the RGB image rendered under the camera external parameters of the current frame , Perceptual Similarity Loss , Gaussian scaling loss and normal consistent loss L n ,in: The L1 loss is: ， Where H, W, and C are the height, width, and number of channels of the image respectively. is the pixel value of the rendered image, The pixel value of the picture taken by the camera; for: ， is the pixel value of the rendered image, The pixel value of the picture taken by the camera; for: ， in represents the feature map of the i-th layer of image I in the VGG deep network, N is the number of feature layers for calculating the loss, To render the image, Take pictures for the camera; for: ， in is the maximum value of the Gaussian basis element scaling, is the minimum value of the Gaussian basis element scaling, To specify the maximum scaling value, The maximum value of the ratio of the maximum scaling value to the minimum scaling value is specified; L n for: ， Where N is the normal map derived from the depth map, is the normal of the Gaussian primitive inside the camera frustum, is the weight of the i-th Gaussian basis.

10. A structure-aware three-dimensional scene reconstruction device, characterized in that: include: An acquisition module, the acquisition module is used to acquire information, the information including a multi-view RGB image frame sequence of a scene; A first building module, wherein the first building module is used to build a sparse point cloud generation model according to a diffusion model based on a Unet architecture; A generation module, the generation module is used to input the information into the sparse point cloud generation model to generate camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding viewing angle; A second construction module, the second construction module is used to construct a rough three-dimensional structure grid according to the sparse point cloud data; An initialization module, wherein the initialization module is used to randomly sample the rough three-dimensional structure grid, and initialize the Gaussian field of the Gaussian basis element according to the random sampling result to obtain an initialized Gaussian field; An optimization module is used to randomly select a frame of RGB image frames from the information as the current image frame, project the initialized Gaussian field to the imaging plane of the camera according to the current image frame and the camera internal and external parameters, generate an RGB color image rendered by the camera external parameters of the current image frame, calculate the loss between the current image frame and the RGB color image, iteratively optimize the parameters of the Gaussian primitives of all RGB image frames in the information according to the loss, combine all the optimized Gaussian primitives, and obtain a reconstructed three-dimensional scene.

Images