Multi-dimensional feature and three-plane representation enhanced three-dimensional grid reconstruction method and device

Abstract

The invention discloses a multi-dimensional feature and three-plane representation enhanced three-dimensional grid reconstruction method and device, and belongs to the technical field of three-dimensional reconstruction, and the method comprises the steps: obtaining three-dimensional priori knowledge: after a multi-view image sequence is generated based on a single image, extracting the multi-dimensional features of the image sequence through a multi-dimensional feature extraction and fusion module, and obtaining the three-dimensional priori knowledge; performing feature fusion on the multi-dimensional features and the initial three-plane features by using a high-resolution three-plane representation construction module containing a double-flow Transform, a convolutional layer and a pixel rearrangement operation so as to organize the initial three-plane features into high-resolution three-plane representation; a three-dimensional grid construction module is used for generating a three-dimensional grid based on the three-plane representation, a color decoder is used for decoding and generating color textures of space points based on the three-plane representation, and the three-dimensional grid and the color textures are combined to obtain a high-precision three-dimensional grid with textures. The model effectively reduces the fuzzy and artifact phenomena of the grid surface and improves the accuracy of the geometric structure. The method can be widely applied to modeling processes of game and movie and television industries.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional mesh reconstruction, and in particular relates to a three-dimensional mesh reconstruction method and device with enhanced multi-dimensional features and three-plane representation. Background Art

[0002] In the field of computer graphics and computer vision, 3D reconstruction is one of the fundamental problems, which aims to build a corresponding 3D model from input data such as images or text. The key technologies for 3D object modeling are mainly divided into traditional methods and deep learning-based methods. Traditional methods include using stereo vision, structured light and other technologies to reconstruct 3D meshes from multiple images, or manually created by professionals using tools such as Maya. However, traditional methods have defects such as limited reconstruction accuracy, high learning cost, and low production efficiency. Although physical scanning technology can obtain 3D data of real objects with high precision through professional equipment, its equipment is expensive and has poor portability, which limits its widespread application.

[0003] In recent years, the rapid development of deep learning has promoted 3D reconstruction into a new paradigm, and a series of excellent 3D generation models have emerged, which can generate matching 3D models based on multiple modal conditions. Images and text are the two main driving conditions. Compared with text, images can provide richer and more accurate 3D model information and provide more detailed guidance for the generation of 3D models. At the same time, single-image-driven 3D mesh generation technology is more in line with current industry needs. For example, in application scenarios such as game modeling, modelers need to construct 3D meshes based on original paintings. Single-image-driven 3D mesh generation technology has achieved good results. For example, the LRM (Large Reconstruction Model) model uses the DINO model (a deep learning model for self-supervised visual learning) to extract image features through the input of a single image, and combines the Transformer-based three-plane representation construction network to construct a three-plane representation from image features. Finally, the Marching Cubes algorithm (MC, isosurface extraction algorithm) is used to construct a 3D mesh from the three-plane representation. This method provides a network design paradigm for single-image driven 3D mesh construction solutions. Based on this work, InstantMesh (a tool for quickly generating 3D meshes based on a single image) introduces a multi-view image diffusion model to obtain more 3D priors, and uses FlexiCubes (an open source project for high-quality isosurface representation of gradient-optimized meshes) to obtain and display 3D meshes from three-plane representations. During training, supervision is performed on the 3D mesh to complete 3D mesh reconstruction with higher accuracy.

[0004] However, most of the work in this field usually considers the global information of the driving image and ignores the local information, which makes it impossible to reconstruct sufficient three-dimensional prior knowledge; in addition, since the three-plane representation construction network is usually based on the Transformer architecture, the computational storage complexity and the resolution of the three-plane representation are in a quadratic relationship, and a trade-off needs to be made between the efficiency of the network architecture and the expressiveness of the three-plane representation. These two main reasons lead to the insufficient accuracy of the generated mesh geometry and the presence of blur or artifacts on the surface, that is, the three-dimensional mesh is not accurate enough. Summary of the invention

[0005] In view of the problems existing in the prior art field, the purpose of the present invention is to provide a three-dimensional mesh reconstruction method and device with enhanced multi-dimensional features and three-plane representation, which specifically includes: designing a multi-dimensional feature extraction and fusion module to obtain information of multiple dimensions from a single image, which contains spatial information, global information and local information. At the same time, a high-resolution three-plane representation construction module is also designed. The calculation and storage complexity of this module is linearly related to the resolution of the three-plane representation, and it can efficiently construct a three-plane representation with strong expression ability. By integrating the above two parts of technology, the present invention can generate high-precision three-dimensional meshes based on a single image, effectively reduce the blur and artifacts on the mesh surface, and improve the accuracy of its geometric structure. The generated high-precision three-dimensional mesh can be widely used in the modeling process of the game and film and television industries, simplifying the traditional modeling steps, and improving work efficiency, thereby helping the industry to achieve cost reduction and efficiency improvement.

[0006] To achieve the above-mentioned object of the invention, an embodiment provides a three-dimensional mesh reconstruction method with enhanced multi-dimensional features and three-plane representation, comprising the following steps: Acquisition of three-dimensional prior knowledge: After generating a multi-view image sequence based on a single image, a multi-dimensional feature extraction and fusion module is used to extract multi-dimensional features of the image sequence, specifically including: allowing the modified DINO model to accept camera parameter input, and considering spatial information when extracting image features in combination with the image sequence, to obtain image global features containing global information and spatial information, and also encoding based on the image sequence to obtain image local features containing local information, and splicing the image global features and image local features to obtain multi-dimensional features as prior knowledge; Three-plane representation construction: Use a high-resolution three-plane representation construction module consisting of a two-stream Transformer, a convolutional layer, and a pixel rearrangement operation to fuse the multi-dimensional features and the initial three-plane features to organize the initial three-plane features into a high-resolution three-plane representation; Texture and mesh generation: Use the 3D mesh construction module to generate a 3D mesh based on the three-plane representation. Use the color decoder to generate the color texture of the spatial points based on the three-plane representation decoding. Combine the 3D mesh and color texture to get a textured 3D mesh.

[0007] Preferably, generating a multi-view image sequence based on a single image includes: A pre-trained multi-view image diffusion model is used to generate a multi-view image sequence based on a single image, where the multi-view image diffusion model includes a Zero123++ model.

[0008] Preferably, the modified DINO model accepts camera parameter input, and simultaneously considers spatial information when extracting image features in combination with the image sequence, so as to obtain image global features containing global information and spatial information, including: A camera condition injection module is added to the DINO model to obtain a modified DINO model. The camera condition injection module obtains a scaling vector and an offset vector based on camera parameters. The scaling vector and the offset vector are used to scale and offset the features extracted based on the image sequence in the DINO model, thereby injecting spatial information into it, so that the obtained global features of the image contain spatial information and global information.

[0009] Preferably, encoding is performed based on the image sequence to obtain local features of the image containing local information, and the global features of the image and the local features of the image are spliced ​​and fused to obtain multi-dimensional features as prior knowledge, including: A lightweight convolutional network including convolutional layers is used to encode the image sequence. Specifically, the convolutional layers are used to perform sliding encoding on different image blocks in the image sequence to obtain local image features containing local information. The local image features are concatenated with the global image features and then mapped into multi-dimensional features as prior knowledge through layer normalization and a fully connected layer.

[0010] Preferably, a high-resolution three-plane representation building module including a two-stream Transformer, a convolutional layer, and a pixel rearrangement operation is used to perform feature fusion on the multi-dimensional features and the initial three-plane features to organize the initial three-plane features into a high-resolution three-plane representation, including: The attention mechanism in the two-stream Transformer adopts a fast attention mechanism. The two-stream Transformer takes the input initial three-plane features as one stream, and the input multi-dimensional features and randomly initialized features as another stream, and divides the two-stream features into blocks through the feature fusion basic block of the fast attention mechanism to achieve feature fusion. The obtained fused features are further feature encoded by the convolutional layer, and then the resolution is adjusted by the pixel rearrangement operation to obtain a high-resolution three-plane representation, which contains the geometric structure and texture detail information of the three-dimensional object.

[0011] Preferably, generating a three-dimensional mesh based on a three-plane representation using a three-dimensional mesh construction module comprises: A three-dimensional mesh is generated based on a three-plane representation using a three-dimensional mesh construction module including a signed distance field (SDF) decoder, a weight decoder, an offset decoder, and a FlexiCubes model. Specifically, a set of spatial points is initialized, and the three-plane representation features of the spatial points are queried from the three-plane representation. The SDF decoder, the weight decoder, and the offset decoder are respectively decoded based on the three-plane representation of the spatial points to obtain SDF values, weight values, and offset values, which constitute FlexiCubes parameters. The FlexiCubes model constructs a three-dimensional mesh based on the FlexiCubes parameters.

[0012] Preferably, the multi-dimensional feature extraction and fusion module, the high-resolution three-plane representation construction module, the three-dimensional grid construction module, and the color decoder are trained in two stages before being applied. The training process does not require the use of a multi-view image diffusion model. The training process includes: The first stage of training is as follows: the multi-dimensional feature extraction and fusion module accepts the input of multi-view images and extracts the corresponding multi-dimensional features from them. The high-resolution three-plane representation construction module accepts the multi-dimensional features to construct a three-plane representation, initializes the spatial point set, and queries the three-plane representation features of the spatial points from the three-plane representation; the neural radiation field model including the density decoder and the color decoder is used to decode the three-plane representation features of the spatial points to obtain the density value and the color value, and the volume rendering is performed based on the density value and the color value and the camera parameters to obtain the first rendered color image, the first rendered depth image, and the first rendered mask image corresponding to the camera perspective, and the mean square error loss and the perceptual loss of the first rendered color image and the mean square error loss of the first rendered mask image are constructed to form the first stage training loss and conduct training; The second stage of training is as follows: the multi-dimensional feature extraction and fusion module accepts the input of multi-view images and extracts the corresponding multi-dimensional features from them. The high-resolution three-plane representation construction module accepts the multi-dimensional features to construct a three-plane representation, initializes the spatial point set, and queries the three-plane representation features of the spatial points from the three-plane representation; the SDF decoder, weight decoder, and offset decoder are used to decode the features of the spatial point set to obtain the SDF value, weight value, and offset value, which constitute the FlexiCubes parameters. The FlexiCubes model constructs a three-dimensional grid based on the FlexiCubes parameters; the Nvdiffrast algorithm is used to render based on the three-dimensional grid, color texture, and camera parameters to obtain the second rendered color image, the second rendered depth image, the second rendered mask image, and the rendered normal image, and the mean square error loss and perceptual loss of the second rendered color image, the mean square error loss of the second rendered mask image, the L1 loss of the second rendered depth image, the Cosine loss of the rendered normal image, and the regularization loss of the FlexiCubes parameters are constructed to constitute the second stage training loss and train.

[0013] Preferably, the first stage training loss It is expressed as: ; in, Represents a sequence of rendered color images, with superscript represents an image sequence consisting of M images, express The corresponding color image true value sequence, represents the loss weight corresponding to the mask image, represents the sequence of mask images for rendering, express The corresponding mask image true value sequence, represents the weight parameter corresponding to the perceptual loss, It means using VGG network to extract deep features of rendered color image and true value of color image, and use them to calculate perceptual loss. represents mean square error; The loss in the second training phase It is expressed as: ; in, represents the rendered second depth image sequence, express The corresponding second depth image true value sequence, represents a sequence of rendered normal images, express The corresponding normal image true value sequence, symbol represents element-wise multiplication, represents the L1 loss of the second rendered depth image, represents the weight parameter corresponding to the L1 loss, represents the cosine loss of the rendered normal image, Represents the weight parameter corresponding to the cosine loss, represents the regularization loss of FlexiCubes parameters, The weight parameter representing the regularization loss; When calculating the first training loss, The value is the first color image sequence , is the first mask image sequence , when calculating the second training loss, The value is the second color image sequence , is the second mask image sequence .

[0014] To achieve the above-mentioned purpose of the invention, an embodiment also provides a three-dimensional mesh reconstruction device with enhanced multi-dimensional features and three-plane representations, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned three-dimensional mesh reconstruction method with enhanced multi-dimensional features and three-plane representations.

[0015] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-mentioned three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane representation.

[0016] Compared with the prior art, the present invention has the following beneficial effects: Combining the image sequence generation part with the multi-dimensional feature extraction and fusion module, the present invention can obtain more comprehensive multi-dimensional features as prior knowledge through a single image input, covering the spatial information, global information and local information of three-dimensional objects from multiple perspectives, providing sufficient and rich conditions for subsequent three-dimensional mesh reconstruction. Secondly, the high-resolution three-plane representation construction module designed based on dual-stream Transformer, convolutional layer and pixel rearrangement operation not only greatly improves the resolution of the three-plane representation, but also ensures that the calculation and storage complexity only increases linearly. Through this optimized design, the present invention significantly improves the expression ability of the three-plane representation while ensuring the efficiency of the network model, so that the subsequent three-dimensional mesh construction module can construct a high-precision three-dimensional mesh from the three-plane representation. The improved three-plane representation resolution effectively enhances the accuracy of the generated three-dimensional mesh, reduces blurring and artifacts, and improves the accuracy of the geometric structure, further improving the performance of the three-dimensional mesh in geometric structure and texture details. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a general process framework diagram of a three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane characterization provided in an embodiment; Figure 2 is a structural diagram of a multidimensional feature extraction and fusion module provided in an embodiment; Figure 3 is a structural diagram of a camera condition injection module provided in an embodiment; Figure 4is a structural diagram of a high-resolution three-plane representation building block provided by an embodiment; Figure 5 It is a process framework diagram of the first stage training of the network model provided in the embodiment; Figure 6 It is a flow chart of the second phase training of the network model provided in the embodiment; Figure 7 It is an example diagram of three-dimensional mesh reconstruction provided by the embodiment. DETAILED DESCRIPTION

[0019] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0020] The inventive concept of the present invention is that the existing single-image driven three-dimensional mesh generation technology has the problem of low precision in the generated results, that is, the geometric structure accuracy of the mesh is insufficient and its surface is prone to blurring and artifacts. This is mainly because the prior art fails to fully extract the multi-dimensional information in the image and the three-plane representation used is insufficient in expression ability, resulting in poor mesh accuracy. To solve this problem, the present invention provides a three-dimensional mesh reconstruction solution enhanced by multi-dimensional features and three-plane representation. The solution is improved in two aspects: feature extraction and three-plane representation. On the one hand, three-dimensional prior knowledge is fully acquired through multi-dimensional feature extraction and fusion modules. On the other hand, high-resolution three-plane representation is generated through a high-resolution three-plane representation construction module while being efficient. This significantly improves the accuracy of the generated three-dimensional mesh, ensuring that the mesh is more delicate and realistic in terms of geometric structure and texture details. Finally, the present invention can better meet the needs of individual creators and the requirements of industry application scenarios, and provide a high-precision, high-quality three-dimensional mesh reconstruction solution.

[0021] like Figure 1 As shown, a 3D mesh reconstruction method enhanced by multi-dimensional features and three-plane representation provided in the embodiment can obtain sufficient 3D prior knowledge and construct a high-resolution three-plane representation under the input of a single image, and finally generate a high-precision 3D mesh, which specifically includes the following steps: Step 1, acquisition of three-dimensional prior knowledge: After generating a multi-view image sequence based on a single image, a multi-dimensional feature extraction and fusion module is used to extract the multi-dimensional features of the image sequence.

[0022] In the embodiment, a pre-trained multi-view image diffusion model is used based on a single input image. Generate a multi-view image sequence, where the multi-view image diffusion model can be a Zero123++ model. Specifically, a single image Input into the Zero123++ model to generate image sequences of 6 perspectives , image sequence It is used to provide richer three-dimensional prior knowledge. Input into the multi-dimensional feature extraction and fusion module to extract features and obtain multi-dimensional features Among them, a single image Before input, it is also preprocessed, including background removal, cropping and scaling, to make it meet the input format of the multi-view image diffusion model.

[0023] In the embodiment, the following process is performed in the multi-dimensional feature extraction and fusion module: For a single image , where H is the image height, W is the image width, and C is the number of image channels, usually 3. Figure 2 As shown, the image First, after the convolution layer processing, it is divided into non-overlapping image blocks for encoding to obtain the initial image embedding vector , is the total number of blocks in a single image, P is the side length of each block, Represents the feature dimension after encoding each block; then the camera parameters Mapped to camera embedding vector through fully connected layers , to provide spatial information of three-dimensional objects, where the camera parameters Including camera internal and external parameters, the specific settings are: For external parameters, the azimuth angles are set to {30, 90, 150, 210, 270, 330} degrees, the pitch angles are set to {20, -10, 20, -10, 20, -10} degrees, and the distance from the camera to the origin is set to 4. It should be emphasized that the object in the input image is at the origin of the world coordinate system by default and is scaled to The internal reference field of view is set to 30 degrees, the focal length is about 1.866, and the principal point (image center) is in the normalized coordinate location.

[0024] In order to achieve effective fusion of spatial information and image features, a camera condition injection module is added to the DINO model to obtain a modified DINO model, wherein the DINO model can adopt the DINOv2 model, and the structure of the camera condition injection module is as follows: Figure 3 As shown, the camera is embedded into the vector through two fully connected layers Mapped to twice the image encoding dimension, and split into scaling vectors and offset vectors for channel-level scaling and translation operations on image features, thereby obtaining image global features containing spatial information and global information. The modified DINO model not only pays attention to the semantic level information of the image (i.e., global information), but also outputs the global features of the image that contain spatial information and global information.

[0025] However, for the 3D reconstruction task, the local information of the image is also crucial. Existing methods often ignore the representation of local information, resulting in the lack of geometric structure and texture details in the generated 3D mesh. Therefore, in order to capture the local information in the image, a lightweight convolutional network with convolutional layers is designed to encode the image sequence, ensuring that the setting of the convolutional layer is consistent with the convolutional layer in the DINO model, so as to achieve alignment with the global features of the image in the channel dimension, and adopt a Patchify operation similar to ViT to extract local information of different regions by sliding the convolution kernel on the image block to obtain the local features of the image. Finally, the local features of the image and the global features of the image are spliced ​​in the channel dimension, and after layer normalization and full connection layer, the final multi-dimensional features are obtained. As prior knowledge.

[0026] The obtained multi-dimensional features The spatial information, global image information and local image information are integrated to significantly improve the accuracy and detail expression of 3D reconstruction. The present invention injects camera parameters into the DINO model to obtain spatial information, and extracts local information of the image by combining lightweight convolutional layers and Patchify operations to generate feature vectors containing multi-dimensional information, which provides a solid foundation for the subsequent three-plane representation construction and high-precision 3D mesh (3D mesh) reconstruction.

[0027] Step 2, three-plane representation construction: Use a high-resolution three-plane representation construction module consisting of a two-stream Transformer, a convolutional layer, and a pixel reordering operation to fuse the multi-dimensional features and the initial three-plane features to organize the initial three-plane features into a high-resolution three-plane representation.

[0028] In getting the feature vector After that, it is necessary to use the high-resolution three-plane representation construction module to construct the three-plane representation. The constructed three-plane representation is used as the intermediate representation of three-dimensional generation. The shape of the three-plane representation is set to , where 3 represents three mutually perpendicular planes, Represents the height and width of each plane, that is, the resolution, and also represents the number of features stored in each plane. The three-plane representation is composed of the XY plane, XZ plane and YZ plane of the Cartesian coordinate system, and stores various information of the spatial point.

[0029] In order to obtain the three-plane representation features of the spatial point, it is necessary to query in the three-plane representation. First, the coordinates of the spatial point are scaled to within the range of the cube; project the scaled spatial point coordinates onto the three planes represented by the three planes to obtain the two-dimensional coordinates on different planes; since the plane contains The feature vectors cannot cover every coordinate point on the plane, so it is necessary to use bilinear interpolation to obtain the corresponding feature vectors; finally, the feature vectors obtained from each plane are spliced ​​to obtain the feature vectors of the three-plane representation of the spatial point. It can be seen that the resolution of the three-plane representation determines the number of features on each plane. If the resolution is too low, it will affect the expressiveness of the spatial point features. Existing methods usually use a Transformer-based network to construct a three-plane representation, in which the three-plane representation is used as the query matrix (Query), and the computational storage complexity of the self-attention operation is in a square relationship with the sequence length, that is, the resolution of the three-plane representation ( ) will lead to a quadratic increase in the computational storage complexity of the Transformer structure, which will seriously affect the training and reasoning efficiency of the model. Therefore, the three-plane representation resolution of existing methods is usually low. For example, the three-plane representation resolution of the InstantMesh and LDM models is , however, this also makes the results generated by these models less accurate.

[0030] To this end, the structure of the high-resolution three-plane representation building block constructed in the embodiment of the present invention is as follows: Figure 4 As shown in the figure, it includes a two-stream Transformer, a convolutional layer, and a pixel rearrangement operation. The two-stream Transformer splits the self-attention mechanism and the cross-attention mechanism in the traditional Transformer structure into two independent streams, and significantly reduces the computational and storage complexity by adjusting the execution order of the attention mechanism, so that when the resolution of the three-plane representation is improved, the complexity only increases linearly, thereby supporting higher-resolution three-plane representations. Specifically, the attention mechanism in the two-stream Transformer adopts a fast attention mechanism, including a fast attention mechanism 2 (i.e., Flash Attention2), i.e. Figure 4The two-stream Transformer uses the input initial three-plane features as one stream, and the input multi-dimensional features and randomly initialized features as another stream. The two-stream features are processed in blocks by using the feature fusion basic block of the fast attention mechanism to achieve feature fusion. The obtained fused features are further encoded by the convolution layer, and then the resolution is adjusted by the pixel rearrangement operation to obtain a high-resolution three-plane representation, which effectively improves the resolution and detail expression ability of the three-plane representation. The three-plane representation contains all information related to the three-dimensional object, including spatial information, geometric structure, and texture detail information. Specifically: The input includes multi-dimensional features , Randomly initialize features and the initial three-plane features The initial three-plane feature shape is , , which is the number of features in the three-plane representation, is the dimension of the feature, which is set to 1024 in the model. and After layer normalization and full connection layer, concatenation is performed on the sequence dimension to obtain the conditional feature vector , the shape is , the conditional eigenvector For initial three-plane features The spatial information, geometric structure and texture information of the three-dimensional grid are injected into the

[0031] The two-stream Transformer contains four feature fusion basic blocks, each of which consists of two fusion blocks and three calculation blocks. The attention operation in each block is implemented based on the fast attention mechanism. Through block processing, the efficient fusion of three-plane features and image features is gradually completed, thereby significantly reducing the overall computational complexity. The first fusion block contains a fast cross attention operation using the fast attention mechanism. As the query matrix (Query) of the attention operation, As the key-value matrix (Key, Value), the intermediate features are fast self-attention and fast cross-attention operations are performed on the intermediate features in the calculation block. In the fast cross-attention operation of the calculation block, the intermediate features are used as the query matrix (Query), and the multi-dimensional features It will be input into the calculation block as the key-value matrix (Key, Value) of the fast cross attention operation using the fast attention mechanism, and then in the second fusion block, The query matrix for fast cross attention is taken as the key matrix, and the feature vector from the computation block is taken as the key matrix. Through the above attention operation organization, the attention operation of the traditional Transformer is placed in two different streams, which can achieve feature fusion while maintaining the efficiency of the structure. In the above calculation process, the computational complexity of the first fusion block is , the storage complexity is , where d is the feature dimension; the computational complexity of the computational block is , the storage complexity is ; The computational complexity of the second fusion block is , the storage complexity is . It can be seen that when the resolution of the three-plane representation is improved (i.e. Compared with the existing technology, the overall computational and storage complexity is significantly reduced.

[0032] The resolution of the initial three-plane features is set to After passing through the two-stream Transformer, the spatial, geometric structure and texture information of the three-dimensional object are obtained, and a three-plane representation is obtained. After the convolution layer, the resolution is maintained unchanged for further feature encoding, and the shape of the three-plane representation is obtained as There are 4 convolution layers in total, and the convolution kernel size of each layer is , the stride is 1, the padding is 1, the input dimension and output dimension of the first three layers are 1024, the input dimension of the last layer is 1024, and the output dimension is 1280. Finally, the three-plane representation is further improved by pixel rearrangement operation T The resolution is . The refinement of feature expression is significantly enhanced, ensuring that the generated three-dimensional mesh has higher precision, reducing blur and artifacts, and improving the accuracy of the geometric structure. The two-stream Transformer structure effectively reduces the computational and storage complexity, so that while the resolution of the three-plane representation is improved, the time and space complexity only increases linearly. Combined with the convolutional layer and pixel rearrangement operation, the refinement of feature expression and the accuracy of model generation are further improved.

[0033] Step 3, texture and mesh generation: Use the 3D mesh construction module to generate a 3D mesh based on the three-plane representation, use the color decoder to decode and generate the color texture of the spatial point based on the three-plane representation, and combine the 3D mesh and the color texture to obtain a textured 3D mesh.

[0034] After obtaining the three-plane representation After that, it is necessary to generate a three-dimensional mesh. Specifically, a three-dimensional mesh construction module including an SDF decoder, a weight decoder, an offset decoder, and a FlexiCubes model is used to generate a three-dimensional mesh based on a three-plane representation. Specifically, the spatial point set is initialized, the spatial points are represented by coordinates in a Cartesian coordinate system, and the features of the spatial points are queried from the three-plane representation. The SDF decoder, the weight decoder, and the offset decoder are used to decode the three-plane representation of the spatial points to obtain the signed distance field (i.e., the SDF value). , weight value , and offset values , which constitutes the FlexiCubes parameters, where Used to determine the relationship between vertices in internal and external space. Used to control the interpolation and quadrilateral splitting method in the DualMarching Cubes algorithm (an improved isosurface extraction algorithm), offset It is used to deform the vertices of the reference mesh, input the three FlexiCubes parameters into the FlexiCubes model to complete the mesh construction, and obtain the constructed three-dimensional mesh, which is stored in OBJ format or GLB format.

[0035] In the embodiment, the designs of the three decoders, namely, the SDF decoder, the weight decoder, and the offset decoder, uniformly adopt four fully connected layers and combine with the ReLU activation function, wherein the dimension of each hidden layer is 64, and the final output channels are 1 ( ),twenty one( ) and 3 ( ). In the core process of the FlexiCubes model, the three-dimensional space is first discretized into a regular voxel grid according to the preset resolution and scaling factor, thereby obtaining the vertices and cube units of the reference grid. On this basis, the offset value are added to each reference vertex one by one to form the deformed mesh vertex coordinates; at the same time, the weight value obtained by analysis , the interpolation position and mesh topology (especially the splitting strategy of quadrilaterals) can be controlled in the subsequent extraction process, thus giving the entire mesh construction process enough differentiable freedom. The sign information (positive and negative values) is used to accurately identify surface voxels and surface mesh edges. At each mesh edge that intersects the surface, the SDF zero crossing point and weight value are used to identify the surface voxels and surface mesh edges. Linear interpolation or least squares calculation is performed to solve the dual vertex, and the triangles or quadrilaterals of the mesh surface are spliced ​​based on them to obtain a three-dimensional mesh.

[0036] In the embodiment, it is also necessary to obtain the texture information (i.e., color texture) of the mesh surface. For the three-plane representation features corresponding to the spatial points, a color decoder (composed of 4 fully connected layers) is used to decode the three-plane representation to obtain the color texture corresponding to the spatial points. Finally, the trimesh library is used to combine the three-dimensional mesh and color texture to output a three-dimensional mesh with texture.

[0037] In the above-mentioned 3D mesh reconstruction method enhanced by multi-dimensional features and three-plane representation, the multi-dimensional feature extraction and fusion module, high-resolution three-plane representation construction module, 3D mesh construction module, and color decoder are trained in two stages before being applied to optimize model parameters. Among them, both the first stage training and the second stage training require three parts, namely, 3D prior knowledge acquisition, three-plane representation construction, and differentiable rendering. Among them, the acquisition of 3D prior knowledge is different from reasoning. At this time, there is no need to introduce a pre-trained multi-view image diffusion model, and the input is a multi-view image; the three-plane representation construction is consistent with the reasoning; the last part of the reasoning is the generation of texture and mesh. However, since the training is supervised on the image data, the differentiable rendering part is introduced to obtain the rendered image. The difference between the first stage training and the second stage training lies in the differentiable rendering technology. The first stage uses volume rendering technology, and the second stage uses grid-based differentiable rendering (Nvdiffrast).

[0038] The goal of the first phase of training is to enable the network to capture the rough 3D grid information under the image drive, so that the network model can initially build a mapping relationship from the image to the three-plane representation. In order to conduct more stable training, the explicit 3D grid is not built first, and the NeRF model is introduced to replace the FlexiCubes part. The specific process framework of the first phase of training is as follows: Figure 5 As shown, the multi-view image sequence After multi-dimensional feature extraction and fusion module, multi-dimensional features are obtained , combined with the initial three-plane features Perform three-plane representation construction to obtain a high-resolution three-plane representation The density decoder and color decoder in the Neural Radiance Field model (NeRF model) are used to represent the Decode the spatial point features into density values ​​and color values, and render the camera parameters through volume rendering based on the density values, color values ​​and camera parameters. The rendered image of the specified perspective includes the first rendered color image, the first rendered depth image, and the first rendered mask image. The mean square error loss (MSE loss) and perceptual loss of the first rendered color image and the mean square error loss of the first rendered mask image are constructed to form the first stage training loss and are trained. The MSE loss is used to provide supervision of geometric structure and texture details, and the perceptual loss is used to provide supervision at the semantic level. Specific first training stage loss It is expressed as: ; in, Represents a sequence of rendered color images, with superscript represents an image sequence consisting of M images, express The corresponding color image true value sequence, represents the loss weight corresponding to the mask image, represents the sequence of mask images for rendering, express The corresponding mask image true value sequence, represents the weight parameter corresponding to the perceptual loss, It means using VGG network to extract deep features of rendered color image and true value of color image, and use them to calculate perceptual loss. Represents the mean square error.

[0039] The purpose of the second stage of training is to enable the network to capture more detailed 3D mesh information under the input of multi-view images and map it into a more informative three-dimensional plane representation, thereby improving the final generation effect. At this time, FlexiCubes is used instead of NeRF to construct the 3D mesh (the 3D mesh generation process is the same as the inference stage). The process framework of the second stage of training is as follows: Figure 6 As shown, the multi-view image sequence After multi-dimensional feature extraction and fusion module, multi-dimensional features are obtained After that, combined with the initial three-plane features Perform three-plane representation construction to obtain a high-resolution three-plane representation Initialize the spatial point set, query the features of the spatial point set from the three-plane representation, and decode the spatial point features into SDF values ​​using the SDF decoder, weight decoder, and offset decoder respectively. , weight value and offset value , which constitutes FlexiCubes parameters. Combining these FlexiCubes parameters, the FlexiCubes model can construct a 3D mesh (without texture). Using a color decoder to decode the three-plane representation of the spatial points on the mesh can obtain the color texture. The Nvdiffrast algorithm can render the camera parameters based on the 3D mesh, color texture and camera parameters. The rendered image of the specified perspective includes a second rendered color image, a second rendered depth image, a second rendered mask image, and a rendered normal image. The mean square error loss and perceptual loss of the second rendered color image and the mean square error loss of the second rendered mask image, that is, the loss of the first training stage, are constructed. In addition, the L1 loss of the second rendered depth image, the Cosine loss of the rendered normal image, and the regularization loss of the FlexiCubes parameters are constructed to constitute the second stage training loss and are trained. The regularization loss makes the generated mesh shape more reasonable.

[0040] Second stage training loss It is expressed as: ; in, represents the rendered second depth image sequence, express The corresponding second depth image true value sequence, represents a sequence of rendered normal images, express The corresponding normal image true value sequence, symbol represents element-wise multiplication, represents the L1 loss of the second rendered depth image, represents the weight parameter corresponding to the L1 loss, Represents the Cosine loss of rendering normal images, Represents the weight parameter corresponding to the Cosine loss, represents the regularization loss of FlexiCubes parameters, The weight parameter representing the regularization loss; When calculating the first training loss, The value is the first color image sequence , is the first mask image sequence , when calculating the second training loss, The value is the second color image sequence , is the second mask image sequence .

[0041] After the above two stages of training, Figure 1The process framework shown inputs a single image and outputs a high-precision 3D mesh. Figure 7 The final generated result is displayed, and it can be seen that the generated result has no blur or artifacts and the geometric structure is accurate.

[0042] Based on the same inventive concept, an embodiment further provides a three-dimensional mesh reconstruction device with multi-dimensional features and three-plane representation enhancement, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the three-dimensional mesh reconstruction method with multi-dimensional features and three-plane representation enhancement is implemented, specifically including the following steps: Step 1, 3D prior knowledge acquisition: after generating a multi-view image sequence based on a single image, a multi-dimensional feature extraction and fusion module is used to extract the multi-dimensional features of the image sequence; Step 2, three-plane representation construction: Use a high-resolution three-plane representation construction module including a two-stream Transformer, a convolutional layer, and a pixel rearrangement operation to fuse the multi-dimensional features and the initial three-plane features to organize the initial three-plane features into a high-resolution three-plane representation; Step 3, texture and mesh generation: Use the 3D mesh construction module to generate a 3D mesh based on the three-plane representation, use the color decoder to decode and generate the color texture of the spatial point based on the three-plane representation, and combine the 3D mesh and the color texture to obtain a textured 3D mesh.

[0043] The three-dimensional mesh reconstruction device enhanced by multi-dimensional features and three-plane characterization provided in the embodiment belongs to a hardware device. At the hardware level, in addition to the processor and memory, it also includes hardware required for other services such as internal bus, network interface, memory, etc. The memory is a non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane characterization of steps 1 to 3 above. Of course, in addition to software implementation methods, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0044] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned three-dimensional mesh reconstruction method with enhanced multi-dimensional features and three-plane representation is implemented, specifically including the following steps: Step 1, 3D prior knowledge acquisition: after generating a multi-view image sequence based on a single image, a multi-dimensional feature extraction and fusion module is used to extract the multi-dimensional features of the image sequence; Step 2, three-plane representation construction: Use a high-resolution three-plane representation construction module including a two-stream Transformer, a convolutional layer, and a pixel rearrangement operation to fuse the multi-dimensional features and the initial three-plane features to organize the initial three-plane features into a high-resolution three-plane representation; Step 3, texture and mesh generation: Use the 3D mesh construction module to generate a 3D mesh based on the three-plane representation, use the color decoder to decode and generate the color texture of the spatial point based on the three-plane representation, and combine the 3D mesh and the color texture to obtain a textured 3D mesh.

[0045] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0046] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane representation, characterized in that: The following steps are involved: Acquisition of three-dimensional prior knowledge: After generating a multi-view image sequence based on a single image, a multi-dimensional feature extraction and fusion module is used to extract multi-dimensional features of the image sequence, specifically including: allowing the modified DINO model to accept camera parameter input, and considering spatial information when extracting image features in combination with the image sequence, to obtain image global features containing global information and spatial information, and also encoding based on the image sequence to obtain image local features containing local information, and splicing the image global features and image local features to obtain multi-dimensional features as prior knowledge; Three-plane representation construction: Use a high-resolution three-plane representation construction module consisting of a two-stream Transformer, a convolutional layer, and a pixel rearrangement operation to fuse the multi-dimensional features and the initial three-plane features to organize the initial three-plane features into a high-resolution three-plane representation; Texture and mesh generation: Use the 3D mesh construction module to generate a 3D mesh based on the three-plane representation. Use the color decoder to generate the color texture of the spatial points based on the three-plane representation decoding. Combine the 3D mesh and color texture to get a textured 3D mesh.

2. The three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane representation according to claim 1 is characterized in that: Generate multi-view image sequences based on a single image, including: A pre-trained multi-view image diffusion model is used to generate a multi-view image sequence based on a single image, where the multi-view image diffusion model includes a Zero123++ model.

3. The three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane representation according to claim 1 is characterized in that: The modified DINO model accepts camera parameter input and considers spatial information when extracting image features in combination with the image sequence, thereby obtaining image global features containing global information and spatial information, including: A camera condition injection module is added to the DINO model to obtain a modified DINO model. The camera condition injection module obtains a scaling vector and an offset vector based on camera parameters. The scaling vector and the offset vector are used to scale and offset the features extracted based on the image sequence in the DINO model, thereby injecting spatial information into it, so that the obtained global features of the image contain spatial information and global information.

4. The three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane representation according to claim 1, characterized in that: Based on the image sequence, the local features of the image containing local information are obtained by encoding, and the global features of the image and the local features of the image are spliced ​​to obtain multi-dimensional features as prior knowledge, including: A lightweight convolutional network including convolutional layers is used to encode the image sequence. Specifically, the convolutional layers are used to perform sliding encoding on different image blocks in the image sequence to obtain local image features containing local information. The local image features are concatenated with the global image features and then mapped into multi-dimensional features as prior knowledge through layer normalization and a fully connected layer.

5. The three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane representation according to claim 1, characterized in that: The high-resolution three-plane representation building module including a two-stream Transformer, a convolutional layer, and a pixel rearrangement operation is used to fuse the multi-dimensional features and the initial three-plane features to organize the initial three-plane features into a high-resolution three-plane representation, including: The attention mechanism in the two-stream Transformer adopts a fast attention mechanism. The two-stream Transformer takes the input initial three-plane features as one stream, and the input multi-dimensional features and randomly initialized features as another stream, and realizes feature fusion by dividing the two-stream features into blocks through the feature fusion basic block of the fast attention mechanism. The obtained fused features are further feature encoded by the convolutional layer, and then the resolution is adjusted by the pixel rearrangement operation to obtain a high-resolution three-plane representation, which contains the geometric structure and texture detail information of the three-dimensional object.

6. The three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane representation according to claim 1, characterized in that: Generate a 3D mesh based on a tri-planar representation using the 3D Mesh Building Blocks, including: A three-dimensional grid is generated based on a three-plane representation using a three-dimensional grid construction module including a signed distance field decoder, a weight decoder, an offset decoder, and a FlexiCubes model, specifically: a set of spatial points is initialized, and the three-plane representation features of the spatial points are queried from the three-plane representation, and the signed distance field value, the weight value, and the offset value are obtained by decoding the three-plane representation features of the spatial points using a signed distance field decoder, a weight decoder, and an offset decoder, respectively, to form FlexiCubes parameters, and the FlexiCubes model constructs a three-dimensional grid based on the FlexiCubes parameters.

7. The three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane representation according to claim 6, characterized in that: The multi-dimensional feature extraction and fusion module, the high-resolution three-plane representation construction module, the three-dimensional mesh construction module, and the color decoder are trained in two stages before being applied, including: The first stage of training is as follows: the multi-dimensional feature extraction and fusion module accepts the input of multi-view images and extracts the corresponding multi-dimensional features from them. The high-resolution three-plane representation construction module accepts the multi-dimensional features to construct a three-plane representation, initializes the spatial point set, and queries the three-plane representation features of the spatial points from the three-plane representation; the neural radiation field model including the density decoder and the color decoder is used to decode the three-plane representation features of the spatial points to obtain the density value and the color value, and the volume rendering is performed based on the density value and the color value and the camera parameters to obtain the first rendered color image, the first rendered depth image, and the first rendered mask image corresponding to the camera perspective, and the mean square error loss and the perceptual loss of the first rendered color image and the mean square error loss of the first rendered mask image are constructed to form the first stage training loss and conduct training; The second stage of training is as follows: the multi-dimensional feature extraction and fusion module accepts the input of multi-view images and extracts the corresponding multi-dimensional features from them. The high-resolution three-plane representation construction module accepts the multi-dimensional features to construct a three-plane representation, initializes the spatial point set, and queries the three-plane representation features of the spatial points from the three-plane representation. The three-plane representation features of the spatial points are decoded using the signed distance field decoder, the weight decoder, and the offset decoder to obtain the SDF value, weight value, and offset value. These values ​​constitute the FlexiCubes parameters. The FlexiCubes model constructs a three-dimensional grid based on the FlexiCubes parameters; the Nvdiffrast algorithm is used to render based on the three-dimensional grid, color texture, and camera parameters to obtain the second rendered color image, the second rendered depth image, the second rendered mask image, and the rendered normal image. The mean square error loss and perceptual loss of the second rendered color image, the mean square error loss of the second rendered mask image, the L1 loss of the second rendered depth image, the cosine loss of the rendered normal image, and the regularization loss of the FlexiCubes parameters are constructed to constitute the second stage training loss and training is performed.

8. The three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane representation according to claim 7, characterized in that: First stage training loss It is expressed as: ; in, Represents a sequence of rendered color images, with superscript represents an image sequence consisting of M images, express The corresponding color image true value sequence, represents the loss weight corresponding to the mask image, represents the sequence of mask images for rendering, express The corresponding mask image true value sequence, represents the weight parameter corresponding to the perceptual loss, It means using VGG network to extract deep features of rendered color image and true value of color image, and use them to calculate perceptual loss. represents mean square error; The loss in the second training phase It is expressed as: ; in, represents the rendered second depth image sequence, express The corresponding second depth image true value sequence, represents a sequence of rendered normal images, express The corresponding normal image true value sequence, symbol represents element-wise multiplication, represents the L1 loss of the second rendered depth image, represents the weight parameter corresponding to the L1 loss, represents the cosine loss of the rendered normal image, Represents the weight parameter corresponding to the cosine loss, represents the regularization loss of FlexiCubes parameters, The weight parameter representing the regularization loss; When calculating the first training loss, The value is the first color image sequence , is the first mask image sequence , when calculating the second training loss, The value is the second color image sequence , is the second mask image sequence .

9. A three-dimensional mesh reconstruction device enhanced by multi-dimensional features and three-plane representation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the three-dimensional mesh reconstruction method with enhanced multi-dimensional features and three-plane representations as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the three-dimensional mesh reconstruction method enhanced by multi-dimensional features and three-plane representations as described in any one of claims 1 to 8 is implemented.

Images