Three-dimensional reconstruction method and device, electronic equipment, storage medium and program product

Abstract

The embodiments of the present disclosure disclose a three-dimensional reconstruction method and device, electronic equipment, storage medium and program product. A low-resolution 3D Gaussian model (sparse point cloud data) is used to reconstruct a fine (or high-resolution) 3D Gaussian model (dense point cloud data; the point cloud is more and the details are rich), the dependence on 2D image pseudo-labels or intermediate rendering steps is avoided, 3D information is directly used for enhancement, and multi-view consistency is ensured. In addition, the intermediate step (such as rendering an image sequence, applying a 2D super-resolution model, and reconstructing a 3D model) in the traditional “3D→2D→3D” process is skipped, the point cloud model (low-resolution 3D Gaussian model) is directly used to process 3D data, the computing overhead is reduced, and end-to-end processing avoids multiple conversions, so the processing efficiency is high. A high-resolution ground truth image is used as supervision to train a Gaussian splash super-resolution model, so that the model optimization direction is more clear and accurate, the trained Gaussian splash super-resolution model has higher stability and higher prediction efficiency.

Description

Technical Field

[0001] This disclosure relates to three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In recent years, 3D Gaussian Splatting (3DGS) has been widely adopted in novel perspective synthesis tasks due to its efficient rendering capabilities and excellent representational properties. However, as its performance limitations with low-resolution inputs have become increasingly apparent, researchers have proposed various 3D super-resolution methods to enhance its detail representation. Some methods employ 2D (Two-Dimensional) image prior-guided optimization, generating high-resolution pseudo-labels from low-resolution images, which are then used for direct optimization in 3DGS. For example, SRGS (Super-Resolution 3D Gaussian Splatting) utilizes a SISR (Single Image Super-Resolution) model to synthesize pseudo-labels and introduces sub-pixel consistency constraints. SuperGS further incorporates variational residual features and uncertainty modeling to enhance detail and robustness. These methods follow a typical "2D→3D" process, applying image spatial supervision to Gaussian point clouds. However, the reference images generated by the 2D AI model to guide the optimization of the 3D model contain contradictory geometric or texture information from different perspectives (i.e., the pseudo-labels may be inconsistent across multiple views), which makes the 3D model confused and ultimately affects the reconstruction quality. Summary of the Invention

[0003] To address the technical problems in related technologies, embodiments of this disclosure provide a three-dimensional reconstruction method, apparatus, electronic device, storage medium, and program product.

[0004] According to a first aspect of the present disclosure, a three-dimensional reconstruction method is provided, the method comprising:

[0005] Obtain a model training sample set, which includes several data triples. Each data triple is a first three-dimensional Gaussian model of any first target object, a corresponding set of high-resolution images, and its camera extrinsic parameters. The first three-dimensional Gaussian model is sparse point cloud data. The set of high-resolution images includes two-dimensional high-resolution images of the first target object taken by two or more camera extrinsic parameters.

[0006] The first three-dimensional Gaussian model of any first target object is input into the first pre-trained model, so as to obtain the second three-dimensional Gaussian model by means of the coordinate parameters and attribute features of the point cloud based on the first three-dimensional Gaussian model. The second three-dimensional Gaussian model is dense point cloud data.

[0007] Image rendering is performed based on the coordinate parameters and attribute features of the point cloud of the second three-dimensional Gaussian model to obtain a rendered composite image corresponding to any of the camera extrinsic parameters of the first target object;

[0008] Calculate the loss function value between the two-dimensional high-definition image corresponding to the camera extrinsic parameters and the rendered composite image, and adjust the network parameters of the first pre-trained model based on the loss function value through backpropagation to obtain the second pre-trained model, which is a Gaussian splash super-resolution model.

[0009] In response to the received third 3D Gaussian model of the second target object, a fourth 3D Gaussian model is predicted and reconstructed using the Gaussian splash super-resolution model. The fourth 3D Gaussian model is the target 3D model of the second target object. The third 3D Gaussian model is sparse point cloud data, and the fourth 3D Gaussian model is dense point cloud data.

[0010] As an optional embodiment, the first three-dimensional Gaussian model of any first target object is input into a first pre-trained model to obtain a second three-dimensional Gaussian model based on the coordinate parameters and attribute features of the point cloud of the first three-dimensional Gaussian model. The second three-dimensional Gaussian model is dense point cloud data, including:

[0011] For any first target object, a first three-dimensional Gaussian model is used to preprocess all point cloud data in the first three-dimensional Gaussian model to normalize all point cloud data to a preset coordinate range, thus obtaining a fifth three-dimensional Gaussian model.

[0012] The first pre-trained model is used to perform feature enhancement processing on the point cloud data in the fifth three-dimensional Gaussian model based on the attribute features of the point cloud, so as to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model.

[0013] Based on the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model, the point cloud data of the fifth three-dimensional Gaussian model is processed by doubling the number of point clouds through the first pre-trained model to obtain the second three-dimensional Gaussian model, which is dense point cloud data.

[0014] As an optional embodiment, the first three-dimensional Gaussian model for any first target object involves preprocessing all point cloud data in the first three-dimensional Gaussian model to normalize all point cloud data to a preset coordinate range, resulting in a fifth three-dimensional Gaussian model, including:

[0015] Based on the coordinate parameters of all point cloud data in the first 3D Gaussian model, determine the maximum and minimum values ​​of the coordinates of all point cloud data.

[0016] Based on the maximum and minimum values ​​of the coordinates of all point cloud data, the position coordinates of each point cloud in the first three-dimensional Gaussian model are linearly transformed to scale the coordinates of all point cloud data to a preset coordinate range, thereby obtaining the fifth three-dimensional Gaussian model.

[0017] As an optional embodiment, the step of performing feature enhancement processing on the point cloud data in the fifth 3D Gaussian model using the first pre-trained model with attribute features based on the point cloud to obtain the enhanced feature vector of each point cloud in the fifth 3D Gaussian model includes:

[0018] Based on the attribute features of point clouds, all point cloud data in the fifth three-dimensional Gaussian model are sorted using the space filling curve of the first pre-trained model to obtain sorted point cloud sequence data.

[0019] The sorted point cloud sequence data is divided into data blocks according to a preset block length to obtain a preset number of point cloud data blocks;

[0020] For each point cloud data block, the coordinate position of each point cloud in the point cloud data block is enhanced by encoding to obtain a position encoding vector of each point cloud that has the same feature dimension as the point cloud.

[0021] Based on the location encoding vector of each point cloud, a corresponding query vector, key vector, and value vector are generated for all point clouds of each point cloud data block.

[0022] For any point cloud in each point cloud data block, the relevance of the point cloud to other point clouds is determined based on the query vector of the point cloud and the key vectors of other point clouds in the same point cloud data block, thus obtaining the attention weight.

[0023] Using the attention weight as the weight coefficient, the value vectors of all point clouds in the point cloud data block are weighted and summed to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model.

[0024] As an optional embodiment, the enhanced feature vector of each point cloud in the fifth 3D Gaussian model is used to double the number of point clouds in the point cloud data of the fifth 3D Gaussian model through the first pre-trained model to obtain the second 3D Gaussian model. The second 3D Gaussian model is dense point cloud data, including:

[0025] Based on the enhanced feature vector of each point cloud, the target feature vector of a preset length for each point cloud is predicted by the first pre-trained model.

[0026] The first pre-trained model is used to split the target feature vector of each point cloud into a preset length to obtain the first feature vector and the second feature vector of each point cloud.

[0027] For each point cloud, the corresponding augmented point cloud is determined based on the first and second feature vectors of the point cloud to obtain the second three-dimensional Gaussian model. The second three-dimensional Gaussian model is dense point cloud data, including all point clouds of the fifth three-dimensional Gaussian model and the augmented point cloud corresponding to each point cloud.

[0028] As an optional embodiment, the step of rendering an image based on the coordinate parameters and attribute features of the point cloud of the second three-dimensional Gaussian model to obtain a rendered composite image corresponding to any of the camera extrinsic parameters of the first target object includes:

[0029] Based on any of the camera extrinsic parameters of the first target object and the attribute features of each point cloud in the second three-dimensional Gaussian model, any point cloud in the second three-dimensional Gaussian model is projected onto a preset two-dimensional plane to obtain a two-dimensional Gaussian distribution of each point cloud.

[0030] Based on the coordinate parameters of each point cloud in the second three-dimensional Gaussian model, the two-dimensional Gaussian distribution of each point cloud is sorted by depth according to a preset sorting rule;

[0031] Based on the depth sorting result, the pixel value of each point cloud in the preset two-dimensional plane is determined according to the two-dimensional Gaussian distribution of each point cloud, and the rendered composite image corresponding to any of the camera extrinsic parameters of the first target object is obtained.

[0032] As an optional embodiment, the step of calculating the loss function value between the two-dimensional high-resolution image corresponding to the camera extrinsic parameters and the rendered composite image, and adjusting the network parameters of the first pre-trained model based on the loss function value through backpropagation to obtain a second pre-trained model, wherein the second pre-trained model is a Gaussian splash super-resolution model, includes:

[0033] Calculate the pixel loss, structural loss, and perceptual loss between the two-dimensional high-definition image corresponding to the camera extrinsic parameters and the rendered composite image, respectively. The pixel loss is the pixel distance between the two-dimensional high-definition image and the rendered composite image, the structural loss is the structural similarity loss between the two-dimensional high-definition image and the rendered composite image, and the perceptual loss is the visual perceptual similarity loss between the two-dimensional high-definition image and the rendered composite image.

[0034] The comprehensive loss is calculated based on the pixel loss, structural loss, and perceptual loss. The comprehensive loss is the value of the loss function. The network parameters of the first pre-trained model are adjusted by backpropagation based on the loss function value to obtain the second pre-trained model, which is a Gaussian splash super-resolution model.

[0035] According to a second aspect of the present disclosure, a three-dimensional reconstruction apparatus is provided, the apparatus comprising:

[0036] The sample set acquisition module is used to acquire the model training sample set, which includes several data triples. The data triples are a first three-dimensional Gaussian model of any first target object, a corresponding high-definition image set and its camera extrinsic parameters. The first three-dimensional Gaussian model is sparse point cloud data. The high-definition image set includes two-dimensional high-definition images of the first target object taken by two or more camera extrinsic parameters.

[0037] The first model training module is used to input the first three-dimensional Gaussian model of any first target object into the first pre-trained model, so as to obtain a second three-dimensional Gaussian model through the coordinate parameters and attribute features of the point cloud based on the first three-dimensional Gaussian model by the first pre-trained model. The second three-dimensional Gaussian model is dense point cloud data.

[0038] The image rendering module is used to render images based on the coordinate parameters and attribute features of the point cloud of the second three-dimensional Gaussian model, and to obtain a rendered composite image corresponding to any of the camera extrinsic parameters of the first target object.

[0039] The second model training module is used to calculate the loss function value between the two-dimensional high-definition image corresponding to the camera extrinsic parameters and the rendered synthetic image, and to adjust the network parameters of the first pre-trained model based on the loss function value through backpropagation to obtain the second pre-trained model. The second pre-trained model is a Gaussian splash super-resolution model.

[0040] The 3D reconstruction module is used to respond to the received third 3D Gaussian model of the second target object and predict and reconstruct a fourth 3D Gaussian model through the Gaussian splash super-resolution model. The fourth 3D Gaussian model is the target 3D model of the second target object. The third 3D Gaussian model is sparse point cloud data, and the fourth 3D Gaussian model is dense point cloud data.

[0041] As an optional embodiment, the first model training module includes:

[0042] The data preprocessing unit is used to perform data preprocessing on all point cloud data in the first three-dimensional Gaussian model for any first target object, so as to normalize all point cloud data to a preset coordinate range and obtain a fifth three-dimensional Gaussian model.

[0043] The feature enhancement unit is used to perform feature enhancement processing on the point cloud data in the fifth three-dimensional Gaussian model using the first pre-trained model with attribute features based on the point cloud, so as to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model.

[0044] The point cloud doubling unit is used to double the number of points in the point cloud data of the fifth three-dimensional Gaussian model based on the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model, and to obtain the second three-dimensional Gaussian model, which is dense point cloud data.

[0045] As an optional embodiment, the data preprocessing unit is further configured to:

[0046] Based on the coordinate parameters of all point cloud data in the first 3D Gaussian model, determine the maximum and minimum values ​​of the coordinates of all point cloud data.

[0047] Based on the maximum and minimum values ​​of the coordinates of all point cloud data, the position coordinates of each point cloud in the first three-dimensional Gaussian model are linearly transformed to scale the coordinates of all point cloud data to a preset coordinate range, thereby obtaining the fifth three-dimensional Gaussian model.

[0048] As an optional embodiment, the feature enhancement unit is further configured to:

[0049] Based on the attribute features of point clouds, all point cloud data in the fifth three-dimensional Gaussian model are sorted using the space filling curve of the first pre-trained model to obtain sorted point cloud sequence data.

[0050] The sorted point cloud sequence data is divided into data blocks according to a preset block length to obtain a preset number of point cloud data blocks;

[0051] For each point cloud data block, the coordinate position of each point cloud in the point cloud data block is enhanced by encoding to obtain a position encoding vector of each point cloud that has the same feature dimension as the point cloud.

[0052] Based on the location encoding vector of each point cloud, a corresponding query vector, key vector, and value vector are generated for all point clouds of each point cloud data block.

[0053] For any point cloud in each point cloud data block, the relevance of the point cloud to other point clouds is determined based on the query vector of the point cloud and the key vectors of other point clouds in the same point cloud data block, thus obtaining the attention weight.

[0054] Using the attention weight as the weight coefficient, the value vectors of all point clouds in the point cloud data block are weighted and summed to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model.

[0055] As an optional embodiment, the point cloud doubling unit is further used for:

[0056] Based on the enhanced feature vector of each point cloud, the target feature vector of a preset length for each point cloud is predicted by the first pre-trained model.

[0057] The first pre-trained model is used to split the target feature vector of each point cloud into a preset length to obtain the first feature vector and the second feature vector of each point cloud.

[0058] For each point cloud, the corresponding augmented point cloud is determined based on the first and second feature vectors of the point cloud to obtain the second three-dimensional Gaussian model. The second three-dimensional Gaussian model is dense point cloud data, including all point clouds of the fifth three-dimensional Gaussian model and the augmented point cloud corresponding to each point cloud.

[0059] As an optional embodiment, the image rendering module includes:

[0060] A two-dimensional projection unit is used to project any point cloud in the second three-dimensional Gaussian model onto a preset two-dimensional plane based on any of the camera extrinsic parameters of the first target object and the attribute features of each point cloud in the second three-dimensional Gaussian model, so as to obtain the two-dimensional Gaussian distribution of each point cloud.

[0061] The depth sorting unit is used to perform depth sorting of the two-dimensional Gaussian distribution of each point cloud according to a preset sorting rule based on the coordinate parameters of each point cloud in the second three-dimensional Gaussian model.

[0062] The pixel determination unit is used to determine the pixel value of each point cloud in the preset two-dimensional plane based on the two-dimensional Gaussian distribution of each point cloud according to the depth sorting result, so as to obtain the rendered composite image corresponding to any of the camera extrinsic parameters of the first target object.

[0063] As an optional embodiment, the second model training module includes:

[0064] The first loss calculation unit is used to calculate the pixel loss, structural loss and perceptual loss between the two-dimensional high-definition image corresponding to the camera extrinsic parameters and the rendered composite image, respectively. The pixel loss is the pixel distance between the two-dimensional high-definition image and the rendered composite image, the structural loss is the structural similarity loss between the two-dimensional high-definition image and the rendered composite image, and the perceptual loss is the visual perceptual similarity loss between the two-dimensional high-definition image and the rendered composite image.

[0065] The second loss calculation unit is used to calculate and determine the comprehensive loss based on the pixel loss, structural loss and perceptual loss, wherein the comprehensive loss is the loss function value, and to adjust the network parameters of the first pre-trained model through backpropagation based on the loss function value to obtain the second pre-trained model, wherein the second pre-trained model is a Gaussian splash super-resolution model.

[0066] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0067] Memory, used to store computer program products;

[0068] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in the first aspect above.

[0069] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the method described in the first aspect.

[0070] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method described in the first aspect above.

[0071] The technical solution provided in this disclosure achieves image-supervised super-resolution 3D reconstruction of a Gaussian splash super-resolution model. Specifically, it reconstructs a detailed (or high-resolution) 3D Gaussian model (dense point cloud data; numerous and detailed points) from a low-resolution 3D Gaussian model (sparse point cloud data). This avoids reliance on 2D image pseudo-labels or intermediate rendering steps, directly utilizing 3D information for enhancement, ensuring multi-view...Figure 1 Furthermore, by skipping intermediate steps in the traditional "3D→2D→3D" process (such as rendering image sequences, applying 2D super-resolution models, and reconstructing 3D models), the point cloud model (low-resolution 3D Gaussian model) is used directly to process 3D data, reducing computational overhead. End-to-end processing avoids multiple conversions, resulting in high processing efficiency. Additionally, the Gaussian splash super-resolution model is trained based on a high-resolution image set (high-resolution ground truth images). Since the details contained therein are 100% self-consistent, the optimization direction of the model is clearer and more accurate. The supervision signals from all perspectives point to the same real 3D geometry, thus avoiding contradictions. This results in higher stability and higher prediction efficiency for the Gaussian splash super-resolution model.

[0072] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0073] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0074] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0075] Figure 1 This is one of the schematic flowcharts of a three-dimensional reconstruction method according to an embodiment of the method disclosed herein.

[0076] Figure 2 This is a second schematic flowchart of a three-dimensional reconstruction method according to an embodiment of the method disclosed herein.

[0077] Figure 3 This is a schematic structural block diagram of a three-dimensional reconstruction apparatus according to one embodiment of the present disclosure.

[0078] Figure 4 This is a schematic structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0079] The performance bottleneck of 3D Gaussian Splatting (3DGS) with low-resolution input can be addressed by several approaches. One approach involves using 2D image priors to guide optimization, generating high-resolution pseudo-labels from low-resolution images, and then directly optimizing the 3DGS. Another approach starts with an existing 3DGS model, renders it as an image sequence, enhances it using an image or video super-resolution model, and then performs 3D reconstruction, forming a closed loop of "3D→2D→3D". For example, SuperGaussian uses a video super-resolution model to process multi-view sequences rendered from low-resolution 3D (or sparse 3D) representations to improve overall detail and consistency. Sequence Matters designs a pseudo-video sequence sorting strategy to improve the adaptability of the VSR (Video Super-Resolution) model while avoiding fine-tuning it. However, while these methods enhance spatial continuity, they rely on intermediate image representations and reconstruction processes, resulting in a long optimization path. In summary, some progress has been made in the methods for enhancing the details of 3DGS assets, but the following drawbacks exist: (1) Methods based on 2D image super-resolution models or 2D video super-resolution models suffer from spatial inconsistencies in the added details across multiple views; (2) Methods based on 2D image super-resolution models or 2D video super-resolution models are not fast.

[0080] To address the technical problems or shortcomings in related technologies, this disclosure provides a three-dimensional reconstruction method, apparatus, electronic device, storage medium, and program product. The technical solutions provided by the embodiments of this disclosure are described in detail below with reference to the accompanying drawings, so that those skilled in the art can clearly and accurately understand the technical solutions of the embodiments of this disclosure.

[0081] Figure 1 This is one of the schematic flowcharts illustrating a three-dimensional reconstruction method according to an embodiment of the method disclosed herein. Figure 1 As shown, a three-dimensional reconstruction method according to an embodiment of this disclosure may include the following steps:

[0082] Step 101: Obtain the model training sample set, wherein the model training sample set includes several data triples, and the data triples are the first three-dimensional Gaussian model of any first target object, the corresponding high-definition image set and its camera extrinsic parameters.

[0083] The first 3D Gaussian model consists of sparse point cloud data, and the high-resolution image set includes two-dimensional high-resolution images of the first target object taken by two or more camera extrinsic parameters. These two-dimensional high-resolution images are high-resolution ground truth images. The first target object is the main object or person contained in the images in the training dataset.

[0084] It should be noted that the first 3D Gaussian model is sparse point cloud data. It can be a small number of point clouds of any target object (the first target object; such as a statue, vehicle, etc.) obtained by devices such as LiDAR or RGB-D cameras. It can also be a few photos of the first target object, or samples of blurry low-resolution photos. The low-quality 3DGS asset obtained by running the 3DGS reconstruction algorithm on these low-quality data (low-resolution photos) is the first 3D Gaussian model.

[0085] A high-resolution image set can be a series of high-resolution ground truth images taken from different camera perspectives (camera extrinsic parameters or poses) around any first target object. For example, using a high-resolution camera, a series of high-resolution ground truth images can be taken from multiple different angles (V1, V2, V3...) around a statue, while accurately recording the camera's position and angle (i.e., camera pose or camera extrinsic parameters) at each shot.

[0086] In this way, the first 3D Gaussian model, the corresponding set of high-resolution images, and their camera extrinsic parameters are combined into triples, and a large number of triple data of the first target object constitute the training sample set. The construction of this training sample set cleverly transforms the quality evaluation of the 3D model into a visual fidelity problem in the 2D image space, providing a reliable and direct signal source for supervised learning, and fundamentally avoiding optimization conflicts caused by inconsistent pseudo-labels in related technologies.

[0087] Step 102: Input the first three-dimensional Gaussian model of any first target object into the first pre-trained model, so as to obtain the second three-dimensional Gaussian model by using the coordinate parameters and attribute features of the point cloud based on the first three-dimensional Gaussian model through the first pre-trained model. The second three-dimensional Gaussian model is dense point cloud data.

[0088] In this embodiment, the first pre-trained model can be a neural network architecture based on powerful pre-trained models such as Point TransformerV3 (PTv3). This allows for a deep understanding of the local and global contextual information of the point cloud, and through its unique feature vector segmentation and point doubling mechanism, it achieves an efficient and controllable upsampling process from sparse to dense points. This not only significantly increases the model's geometric representation capability, providing the necessary physical space for detail enhancement, but also ensures the semantic coherence between the newly added points and the original points.

[0089] As one possible implementation, step 102 can be implemented as follows: For any first target object, perform data preprocessing on all point cloud data in the first three-dimensional Gaussian model to normalize all point cloud data to a preset coordinate range, thereby obtaining a fifth three-dimensional Gaussian model; perform feature enhancement processing on the point cloud data in the fifth three-dimensional Gaussian model based on the attribute features of the point cloud using the first pre-trained model, thereby obtaining an enhanced feature vector for each point cloud in the fifth three-dimensional Gaussian model; based on the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model, double the number of point clouds in the point cloud data of the fifth three-dimensional Gaussian model using the first pre-trained model, thereby obtaining a second three-dimensional Gaussian model, wherein the second three-dimensional Gaussian model is dense point cloud data.

[0090] As an optional embodiment, for any first target object, the first three-dimensional Gaussian model, all point cloud data in the first three-dimensional Gaussian model are preprocessed to normalize all point cloud data to a preset coordinate range to obtain a fifth three-dimensional Gaussian model. Specifically, based on the coordinate parameters of all point cloud data in the first three-dimensional Gaussian model, the maximum and minimum values ​​of the coordinates of all point cloud data are determined, and then the position coordinates of each point cloud in the first three-dimensional Gaussian model are linearly transformed according to the maximum and minimum values ​​of the coordinates of all point cloud data to scale the coordinates of all point cloud data to a preset coordinate range to obtain a fifth three-dimensional Gaussian model. For example, a bounding box that can contain all point cloud can be determined according to the maximum and minimum values ​​of the coordinates of all point cloud data. Then, the center point (i.e., the center coordinates of the bounding box) and the dimensions (length, width, height) of the bounding box in each direction can be further determined according to the coordinate parameters of the point cloud, and then linear transformation is performed. For example, the linear transformation can be performed on the position coordinates of each point in the point cloud as follows: (1) Translation: Subtract the center point coordinates from each point. That is, the point cloud of the entire first three-dimensional Gaussian model is "moved" to the vicinity of the coordinate origin (0, 0, 0); (2) Scaling: the coordinates of each point after translation are divided by a scale factor (which can be a fixed value), and the point cloud of the entire first three-dimensional Gaussian model is "scaled down" to a standard size (for example, it can fit into a cube with a side length of 2 and a center at the origin). In this way, the data can be linearly scaled to a fixed range, usually [0, 1] or [-1, 1], while preserving the distribution shape of the original point cloud data.

[0091] As another embodiment, normalization can also be based on the mean and standard deviation. This involves calculating the mean and standard deviation of the coordinates of all point clouds, then subtracting the mean from the coordinates of each point cloud, and finally dividing by the standard deviation (or scaling to a fixed interval). This normalizes all point cloud data to a specific range. For example, given the original point cloud coordinate range: x∈[-10,10], y∈[0,20], z∈[5,15], calculate the mean: μ_x=0, μ_y=10, μ_z=10; calculate the standard deviation: σ_x=5, σ_y=5, σ_z=3, and then use the normalization formula: x norm =(x-μ) x ) / σ x Similarly, by processing the y and z dimensions, all point cloud data are ultimately normalized to a mean of 0 and a standard deviation of 1. This embodiment can better handle features at different scales and is insensitive to outliers.

[0092] In this embodiment of the disclosure, as one implementation, feature enhancement processing is performed on the point cloud data in the fifth three-dimensional Gaussian model based on the attribute features of the point cloud using the first pre-trained model to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model. This can be achieved as follows: based on the attribute features of the point cloud, all point cloud data in the fifth three-dimensional Gaussian model are sorted using the space-filling curve of the first pre-trained model to obtain sorted point cloud sequence data; the sorted point cloud sequence data is then divided into data blocks according to a preset block length to obtain a preset number of point cloud data blocks; for each point cloud data block, each point cloud in the point cloud data block is processed... The coordinate positions are enhanced and encoded to obtain a position encoding vector for each point cloud with the same feature dimension as the point cloud. Based on the position encoding vector of each point cloud, a corresponding query vector, key vector, and value vector are generated for all point clouds in each point cloud data block. For any point cloud in each point cloud data block, the relevance of the point cloud to other point clouds is determined according to the query vector of the point cloud and the key vectors of other point clouds in the point cloud data block, and an attention weight is obtained. Using the attention weight as the weight coefficient, the value vectors of all point clouds in the point cloud data block are weighted and summed to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model.

[0093] Specifically, the feature vector (with added position encoding vector) of each point cloud within the point cloud data block can be subjected to three different linear transformations (multiplied by three different weight matrices), generating three new vectors for each point: a query vector representing "what information the point wants to find"; a key vector representing "what information the point can provide"; and a value vector representing "the actual information content" of the point. Since the input feature vector incorporates the position encoding vector, the generated query vector, key vector, and value vector also contain precise geometric information of the point cloud. Furthermore, for each point cloud within the point cloud data block, its query vector (e.g., Q) is multiplied by the key vectors (e.g., K) of all point clouds within the block to obtain the correlation or matching degree between that point and every other point in the block. The resulting integral score can then be scaled (to prevent gradient vanishing) and normalized using a Softmax function, ultimately yielding a set of attention weights (e.g., an array of scores between 0 and 1). This set of attention weights defines the characteristic information that the current point cloud acquires from other point clouds in its respective point cloud database. For example, a point cloud on a "table corner" will have a high attention weight with another point cloud on a "table corner" (due to geometric symmetry and feature similarity), while its attention weight with a point cloud on a distant "lampshade" will be very low. Thus, the positional encoding vector ensures that spatially adjacent point clouds receive higher attention weights. Furthermore, the attention weights obtained in the previous step are used as coefficients to perform a weighted summation of the value vectors (e.g., V) of all point clouds within the point cloud data block, ultimately generating an enhanced feature vector for that point cloud. This enhanced feature vector is no longer isolated feature information of that point cloud, but rather a weighted fusion of its own feature information with the feature information of all point clouds within the point cloud data block. Its weights are determined by the feature similarity and spatial proximity between point clouds.

[0094] As another implementation, the first pre-trained model performs feature enhancement processing on the point cloud data in the fifth 3D Gaussian model based on the attribute features of the point cloud to obtain the enhanced feature vector of each point cloud in the fifth 3D Gaussian model. Alternatively, for each point cloud in the point cloud data of the fifth 3D Gaussian model, the first pre-trained model traverses the target neighbor points of each point cloud and obtains the attribute features of all target neighbor points. The attribute features of all target neighbor points are then added to the feature vector of the point cloud to obtain the enhanced feature vector of each point cloud. Here, the target neighbor points are the neighbor points within a preset search range of any point cloud.

[0095] Furthermore, based on the enhanced feature vector of each point cloud, the target feature vector of a preset length for each point cloud is predicted by the first pre-trained model. Then, the target feature vector of the preset length for each point cloud is split by the first pre-trained model to obtain the first feature vector and the second feature vector of each point cloud. For each point cloud, the corresponding augmented point cloud is determined according to the first feature vector and the second feature vector of the point cloud to obtain the second three-dimensional Gaussian model. The second three-dimensional Gaussian model is dense point cloud data, including all point clouds of the fifth three-dimensional Gaussian model and the augmented point cloud corresponding to each point cloud.

[0096] For example, the output vector of each point cloud feature can be expanded into two parts (twice the number of channels) during the output stage of the first pre-trained model to achieve a double representation of the same point cloud, and then the number of points can be decoded twice. Specifically, assuming the dimension of the enhanced feature vector for each point cloud is C, in the final output layer of the first pre-trained model, the first pre-trained model predicts a feature vector (target feature vector) of dimension 2C for each point cloud. For example, assuming the original feature dimension C = 256, the predicted feature vector is a super feature vector of length 512. Then, channel splitting is performed, and the target feature vector of dimension 2C is subjected to tensor partitioning. For example, the first half (the first C channels) is the final, enhanced attribute features of the original input point cloud (corresponding to the enhanced feature vector), and the second half (the last C channels) is the attribute features of a newly added point cloud. This results in two independent feature vectors, each with a dimension of C. Then, feature decoding and point cloud materialization are performed. Specifically, the two independent feature vectors, each with a dimension of C, are input into a decoder network (usually several fully connected layers) to make them based on, for example, appearance attributes (color, opacity), geometry (rotation, 3D scale). For each feature vector, the system outputs a set of specific parameters (including but not limited to color (RGB), opacity (Alpha), rotation (represented by quaternions or three rotation angles), and scale (scaling factors in the X, Y, and Z directions)) and coordinate parameters (3D coordinates). Then, it determines the new point cloud position (i.e., the position of the augmented point cloud). Specifically, for the original point, its position remains unchanged, preserving its original coordinate parameters. The position of the augmented point cloud can be achieved in two optional ways: Method A (direct copy): directly setting the position of the augmented point cloud equal to the position of the original point. Thus, the two point clouds overlap, and in subsequent optimization, their shapes and colors will merge, producing a richer visual effect. Method B (predicted offset): decoding a new 3D displacement vector (Δx, Δy, Δz) from the feature vector of the augmented point cloud (or an additional branch). That is, the position of the augmented point cloud = the position of the original point cloud + the displacement vector. This method can immediately generate geometric details. In this way, through the above process, each point cloud is transformed into two points, thus precisely doubling the number of point clouds from N to 2N.

[0097] The above embodiments not only enrich the feature vectors (feature information) of each point cloud, giving it a richer visual effect, but also increase the number of point clouds, transforming it from sparse point cloud data to dense point cloud data, thus more clearly expressing the information of the target object. This provides a more stable and effective data foundation for subsequent analysis and processing.

[0098] Step 103: Based on the coordinate parameters and attribute features of the point cloud of the second three-dimensional Gaussian model, perform image rendering to obtain the rendered composite image corresponding to any camera extrinsic parameter of the first target object.

[0099] As one embodiment, step 103 can be implemented as follows: based on the extrinsic parameters of any camera of the first target object and the attribute features of each point cloud in the second three-dimensional Gaussian model, project any point cloud in the second three-dimensional Gaussian model onto a preset two-dimensional plane to obtain the two-dimensional Gaussian distribution of each point cloud; sort the two-dimensional Gaussian distribution of each point cloud according to a preset sorting rule based on the coordinate parameters of each point cloud in the second three-dimensional Gaussian model; and determine the pixel value of each point cloud in the preset two-dimensional plane based on the two-dimensional Gaussian distribution of each point cloud according to the depth sorting result to obtain the rendered composite image corresponding to any camera extrinsic parameters of the first target object.

[0100] In this process, any camera extrinsic parameter can be selected from the camera extrinsic parameters in the model training sample set, i.e., one camera extrinsic parameter is randomly chosen as the camera pose for rendering. Given the camera pose (camera extrinsic parameter), the center position (x, y, z) of each point cloud is projected onto the 2D image plane (i.e., the preset 2D plane), resulting in a 2D coordinate (u, v). Based on the point cloud's attribute features (rotation and scaling attributes, etc.) and the camera pose (camera extrinsic parameter), the elliptical shape of the point cloud projected onto the 2D image plane is calculated, i.e., the 2D Gaussian distribution of the point cloud. At the same time, the color and opacity of the point cloud are also assigned to a 2D Gaussian distribution. For each pixel on the 2D image plane, all 2D Gaussian distributions that contribute to the color of this pixel are identified (e.g., those point clouds that are covered by light spots). Then, based on the original 3D depth (z value) of these point clouds, all point clouds are sorted from far to near. According to this sorting, the color and opacity of each 2D Gaussian distribution are sequentially mixed onto the pixel to obtain the rendered composite image.

[0101] Step 104: Calculate the loss function value between the two-dimensional high-definition image and the rendered composite image corresponding to the camera extrinsic parameters. Based on the loss function value, adjust the network parameters of the first pre-trained model through backpropagation to obtain the second pre-trained model, which is a Gaussian splash super-resolution model.

[0102] The quality of the rendered synthetic image reflects the quality of the second 3D Gaussian model output by the first pre-trained model. To evaluate its quality and further optimize the first pre-trained model, this disclosure provides a 2D high-resolution image with the same camera extrinsic parameters (camera viewpoint or camera pose) as a supervisory signal.

[0103] As one embodiment, the pixel loss, structural loss, and perceptual loss between the two-dimensional high-definition image and the rendered composite image corresponding to the camera's extrinsic parameters can be calculated separately. The pixel loss is the pixel distance between the two-dimensional high-definition image and the rendered composite image, the structural loss is the structural similarity loss between the two-dimensional high-definition image and the rendered composite image, and the perceptual loss is the visual perceptual similarity loss between the two-dimensional high-definition image and the rendered composite image. Then, the comprehensive loss is calculated and determined based on the pixel loss, structural loss, and perceptual loss. The comprehensive loss is the loss function value. The network parameters of the first pre-trained model are adjusted through backpropagation based on the loss function value to obtain the second pre-trained model. The second pre-trained model is a Gaussian splash super-resolution model.

[0104] For example, the loss function value is used to compare the difference between the rendered synthetic image and the real high-resolution image (two-dimensional high-definition image), guiding the optimization of the first pre-trained model. The loss function value can be expressed as: L total =L1+L SSIM +L LPIPS Among them, L1 (pixel-level L1 loss): calculates the absolute error of each pixel between the rendered composite image and the 2D high-resolution image to ensure color accuracy; L SSIM (Structural similarity loss): Measures the consistency of local structure in an image (such as edges and texture); L LPIPS (Perceptual Loss): Features are extracted using a pre-trained first pre-trained model, and high-level semantic differences are compared.

[0105] Step 105: In response to the received third 3D Gaussian model of the second target object, a fourth 3D Gaussian model is obtained by prediction and reconstruction using the Gaussian splash super-resolution model. The fourth 3D Gaussian model is the target 3D model of the second target object, the third 3D Gaussian model is sparse point cloud data, and the fourth 3D Gaussian model is dense point cloud data.

[0106] The process of processing the third three-dimensional Gaussian model of the second target object can be found in the model training process of steps 101 to 104 above, and will not be repeated here for the sake of brevity.

[0107] To enable those skilled in the art to more clearly understand the technical solutions of the embodiments of this disclosure, the following will further illustrate... Figure 2 Briefly describe the implementation process again. For example... Figure 2As shown, the coordinate parameters and attribute features of the sparse point cloud of any 3D Gaussian model are used as input, such as the 3D Gaussian model of the second target object. A Gaussian splash super-resolution model (including a Gaussian encoder and a Gaussian decoder; trained and optimized from a first pre-trained model; and using PTv3 as the model basis) is used to process the augmented point cloud (position coordinates A, B) and the enhanced point cloud feature vectors (features A, B) to form the dense point cloud data of the 3D Gaussian model. A rendered composite image is obtained based on the dense point cloud data of the 3D Gaussian model. The loss function value is calculated by comparing the rendered composite image with a corresponding 2D high-resolution image of the camera extrinsic parameters, and this loss function value is used to further guide the optimization of the Gaussian splash super-resolution model.

[0108] It should be noted that the first pre-trained model and the second pre-trained model in this embodiment are the same neural network architecture, namely a neural network architecture based on powerful pre-trained models such as Point TransformerV3 (PTv3). The terms "first pre-trained model" and "second pre-trained model" are used only to highlight the training and tuning process of this neural network architecture, enabling the network to generate high-resolution Gaussian models based on low-resolution Gaussian models. The models that are finally trained and tuned are collectively referred to as Gaussian splash super-resolution models.

[0109] In summary, the technical solution provided by this disclosure achieves image-supervised super-resolution 3D reconstruction of a Gaussian splash super-resolution model. Specifically, it reconstructs a detailed (or high-resolution) 3D Gaussian model (dense point cloud data; numerous and detailed points) based on a low-resolution 3D Gaussian model (sparse point cloud data). This avoids reliance on 2D image pseudo-labels or intermediate rendering steps, directly utilizing 3D information for enhancement, and ensuring multi-view... Figure 1 Furthermore, by skipping intermediate steps in the traditional "3D→2D→3D" process (such as rendering image sequences, applying 2D super-resolution models, and reconstructing 3D models), the point cloud model (low-resolution 3D Gaussian model) is used directly to process 3D data, reducing computational overhead. End-to-end processing avoids multiple conversions, resulting in high processing efficiency. Additionally, the Gaussian splash super-resolution model is trained based on a high-resolution image set (high-resolution ground truth images). Since the details contained therein are 100% self-consistent, the optimization direction of the model is clearer and more accurate. The supervision signals from all perspectives point to the same real 3D geometry, thus avoiding contradictions. This results in higher stability and higher prediction efficiency for the Gaussian splash super-resolution model.

[0110] Correspondingly, this disclosure also provides a three-dimensional reconstruction apparatus. Figure 3 This is a schematic structural block diagram of a three-dimensional reconstruction apparatus according to one embodiment of the present disclosure. Figure 3As shown, a three-dimensional reconstruction apparatus according to an embodiment of this disclosure may include:

[0111] The sample set acquisition module 301 is used to acquire the model training sample set, which includes several data triples. The data triples are a first three-dimensional Gaussian model of any first target object, a corresponding high-definition image set and its camera extrinsic parameters. The first three-dimensional Gaussian model is sparse point cloud data. The high-definition image set includes two-dimensional high-definition images of the first target object taken by two or more camera extrinsic parameters.

[0112] The first model training module 302 is used to input the first three-dimensional Gaussian model of any first target object into the first pre-trained model, so as to obtain a second three-dimensional Gaussian model through the coordinate parameters and attribute features of the point cloud based on the first three-dimensional Gaussian model by the first pre-trained model. The second three-dimensional Gaussian model is dense point cloud data.

[0113] The image rendering module 303 is used to perform image rendering based on the coordinate parameters and attribute features of the point cloud of the second three-dimensional Gaussian model, so as to obtain a rendered composite image corresponding to any of the camera extrinsic parameters of the first target object.

[0114] The second model training module 304 is used to calculate the loss function value between the two-dimensional high-definition image corresponding to the camera extrinsic parameters and the rendered synthetic image, and to adjust the network parameters of the first pre-trained model based on the loss function value through backpropagation to obtain the second pre-trained model, wherein the second pre-trained model is a Gaussian splash super-resolution model.

[0115] The 3D reconstruction module 305 is used to respond to the received third 3D Gaussian model of the second target object and predict and reconstruct a fourth 3D Gaussian model through the Gaussian splash super-resolution model. The fourth 3D Gaussian model is the target 3D model of the second target object. The third 3D Gaussian model is sparse point cloud data, and the fourth 3D Gaussian model is dense point cloud data.

[0116] As an optional embodiment, the first model training module includes:

[0117] The data preprocessing unit is used to perform data preprocessing on all point cloud data in the first three-dimensional Gaussian model for any first target object, so as to normalize all point cloud data to a preset coordinate range and obtain a fifth three-dimensional Gaussian model.

[0118] The feature enhancement unit is used to perform feature enhancement processing on the point cloud data in the fifth three-dimensional Gaussian model using the first pre-trained model with attribute features based on the point cloud, so as to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model.

[0119] The point cloud doubling unit is used to double the number of points in the point cloud data of the fifth three-dimensional Gaussian model based on the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model, and to obtain the second three-dimensional Gaussian model, which is dense point cloud data.

[0120] As an optional embodiment, the data preprocessing unit is further configured to:

[0121] Based on the coordinate parameters of all point cloud data in the first 3D Gaussian model, determine the maximum and minimum values ​​of the coordinates of all point cloud data.

[0122] Based on the maximum and minimum values ​​of the coordinates of all point cloud data, the position coordinates of each point cloud in the first three-dimensional Gaussian model are linearly transformed to scale the coordinates of all point cloud data to a preset coordinate range, thereby obtaining the fifth three-dimensional Gaussian model.

[0123] As an optional embodiment, the feature enhancement unit is further configured to:

[0124] Based on the attribute features of point clouds, all point cloud data in the fifth three-dimensional Gaussian model are sorted using the space filling curve of the first pre-trained model to obtain sorted point cloud sequence data.

[0125] The sorted point cloud sequence data is divided into data blocks according to a preset block length to obtain a preset number of point cloud data blocks;

[0126] For each point cloud data block, the coordinate position of each point cloud in the point cloud data block is enhanced by encoding to obtain a position encoding vector of each point cloud that has the same feature dimension as the point cloud.

[0127] Based on the location encoding vector of each point cloud, a corresponding query vector, key vector, and value vector are generated for all point clouds of each point cloud data block.

[0128] For any point cloud in each point cloud data block, the relevance of the point cloud to other point clouds is determined based on the query vector of the point cloud and the key vectors of other point clouds in the same point cloud data block, thus obtaining the attention weight.

[0129] Using the attention weight as the weight coefficient, the value vectors of all point clouds in the point cloud data block are weighted and summed to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model.

[0130] As an optional embodiment, the point cloud doubling unit is further used for:

[0131] Based on the enhanced feature vector of each point cloud, the target feature vector of a preset length for each point cloud is predicted by the first pre-trained model.

[0132] The first pre-trained model is used to split the target feature vector of each point cloud into a preset length to obtain the first feature vector and the second feature vector of each point cloud.

[0133] For each point cloud, the corresponding augmented point cloud is determined based on the first and second feature vectors of the point cloud to obtain the second three-dimensional Gaussian model. The second three-dimensional Gaussian model is dense point cloud data, including all point clouds of the fifth three-dimensional Gaussian model and the augmented point cloud corresponding to each point cloud.

[0134] As an optional embodiment, the image rendering module includes:

[0135] A two-dimensional projection unit is used to project any point cloud in the second three-dimensional Gaussian model onto a preset two-dimensional plane based on any of the camera extrinsic parameters of the first target object and the attribute features of each point cloud in the second three-dimensional Gaussian model, so as to obtain the two-dimensional Gaussian distribution of each point cloud.

[0136] The depth sorting unit is used to perform depth sorting of the two-dimensional Gaussian distribution of each point cloud according to a preset sorting rule based on the coordinate parameters of each point cloud in the second three-dimensional Gaussian model.

[0137] The pixel determination unit is used to determine the pixel value of each point cloud in the preset two-dimensional plane based on the two-dimensional Gaussian distribution of each point cloud according to the depth sorting result, so as to obtain the rendered composite image corresponding to any of the camera extrinsic parameters of the first target object.

[0138] As an optional embodiment, the second model training module includes:

[0139] The first loss calculation unit is used to calculate the pixel loss, structural loss and perceptual loss between the two-dimensional high-definition image corresponding to the camera extrinsic parameters and the rendered composite image, respectively. The pixel loss is the pixel distance between the two-dimensional high-definition image and the rendered composite image, the structural loss is the structural similarity loss between the two-dimensional high-definition image and the rendered composite image, and the perceptual loss is the visual perceptual similarity loss between the two-dimensional high-definition image and the rendered composite image.

[0140] The second loss calculation unit is used to calculate and determine the comprehensive loss based on the pixel loss, structural loss and perceptual loss, wherein the comprehensive loss is the loss function value, and to adjust the network parameters of the first pre-trained model through backpropagation based on the loss function value to obtain the second pre-trained model, wherein the second pre-trained model is a Gaussian splash super-resolution model.

[0141] In summary, the technical solution provided by this disclosure achieves image-supervised super-resolution 3D reconstruction of a Gaussian splash super-resolution model. Specifically, it reconstructs a detailed (or high-resolution) 3D Gaussian model (dense point cloud data; numerous and detailed points) based on a low-resolution 3D Gaussian model (sparse point cloud data). This avoids reliance on 2D image pseudo-labels or intermediate rendering steps, directly utilizing 3D information for enhancement, and ensuring multi-view... Figure 1 Furthermore, by skipping intermediate steps in the traditional "3D→2D→3D" process (such as rendering image sequences, applying 2D super-resolution models, and reconstructing 3D models), the point cloud model (low-resolution 3D Gaussian model) is used directly to process 3D data, reducing computational overhead. End-to-end processing avoids multiple conversions, resulting in high processing efficiency. Additionally, the Gaussian splash super-resolution model is trained based on a high-resolution image set (high-resolution ground truth images). Since the details contained therein are 100% self-consistent, the optimization direction of the model is clearer and more accurate. The supervision signals from all perspectives point to the same real 3D geometry, thus avoiding contradictions. This results in higher stability and higher prediction efficiency for the Gaussian splash super-resolution model.

[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this disclosure. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0143] Below, for reference Figure 4 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0144] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0145] like Figure 4As shown, the electronic device includes one or more processors and memory.

[0146] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0147] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the three-dimensional reconstruction methods of the various embodiments of this disclosure described above and / or other desired functions.

[0148] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0149] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0150] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0151] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0152] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the three-dimensional reconstruction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0153] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0154] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the three-dimensional reconstruction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0155] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0156] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0157] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0158] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0159] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0160] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0161] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0162] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: Obtain a model training sample set, which includes several data triples. Each data triple is a first three-dimensional Gaussian model of any first target object, a corresponding set of high-resolution images, and its camera extrinsic parameters. The first three-dimensional Gaussian model is sparse point cloud data. The set of high-resolution images includes two-dimensional high-resolution images of the first target object taken by two or more camera extrinsic parameters. The first three-dimensional Gaussian model of any first target object is input into the first pre-trained model, so as to obtain the second three-dimensional Gaussian model by means of the coordinate parameters and attribute features of the point cloud based on the first three-dimensional Gaussian model. The second three-dimensional Gaussian model is dense point cloud data. Image rendering is performed based on the coordinate parameters and attribute features of the point cloud of the second three-dimensional Gaussian model to obtain a rendered composite image corresponding to any of the camera extrinsic parameters of the first target object; Calculate the loss function value between the two-dimensional high-definition image corresponding to the camera extrinsic parameters and the rendered composite image, and adjust the network parameters of the first pre-trained model based on the loss function value through backpropagation to obtain the second pre-trained model, which is a Gaussian splash super-resolution model. In response to the received third three-dimensional Gaussian model of the second target object, a fourth three-dimensional Gaussian model is predicted and reconstructed through the Gaussian splash super-resolution model. The fourth three-dimensional Gaussian model is the target three-dimensional model of the second target object. The third three-dimensional Gaussian model is sparse point cloud data, and the fourth three-dimensional Gaussian model is dense point cloud data. The first three-dimensional Gaussian model of any first target object is input into a first pre-trained model to obtain a second three-dimensional Gaussian model based on the coordinate parameters and attribute features of the point cloud of the first three-dimensional Gaussian model. The second three-dimensional Gaussian model is dense point cloud data, including: For any first target object, a first three-dimensional Gaussian model is used to preprocess all point cloud data in the first three-dimensional Gaussian model to normalize all point cloud data to a preset coordinate range, thus obtaining a fifth three-dimensional Gaussian model. The first pre-trained model is used to perform feature enhancement processing on the point cloud data in the fifth three-dimensional Gaussian model based on the attribute features of the point cloud, so as to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model. Based on the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model, the point cloud data of the fifth three-dimensional Gaussian model is processed by doubling the number of point clouds through the first pre-trained model to obtain the second three-dimensional Gaussian model, which is dense point cloud data; The step of performing feature enhancement processing on the point cloud data in the fifth 3D Gaussian model using the first pre-trained model based on the attribute features of the point cloud to obtain the enhanced feature vector of each point cloud in the fifth 3D Gaussian model includes: Based on the attribute features of point clouds, all point cloud data in the fifth three-dimensional Gaussian model are sorted using the space filling curve of the first pre-trained model to obtain sorted point cloud sequence data. The sorted point cloud sequence data is divided into data blocks according to a preset block length to obtain a preset number of point cloud data blocks; For each point cloud data block, the coordinate position of each point cloud in the point cloud data block is enhanced by encoding to obtain a position encoding vector of each point cloud that has the same feature dimension as the point cloud. Based on the location encoding vector of each point cloud, a corresponding query vector, key vector, and value vector are generated for all point clouds of each point cloud data block. For any point cloud in each point cloud data block, the relevance of the point cloud to other point clouds is determined based on the query vector of the point cloud and the key vectors of other point clouds in the same point cloud data block, thus obtaining the attention weight. Using the attention weight as the weight coefficient, the value vectors of all point clouds in the point cloud data block are weighted and summed to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model.

2. The method according to claim 1, characterized in that, The first three-dimensional Gaussian model for any first target object involves preprocessing all point cloud data within the first three-dimensional Gaussian model to normalize all point cloud data to a preset coordinate range, resulting in a fifth three-dimensional Gaussian model, including: Based on the coordinate parameters of all point cloud data in the first 3D Gaussian model, determine the maximum and minimum values ​​of the coordinates of all point cloud data. Based on the maximum and minimum values ​​of the coordinates of all point cloud data, the position coordinates of each point cloud in the first three-dimensional Gaussian model are linearly transformed to scale the coordinates of all point cloud data to a preset coordinate range, thereby obtaining the fifth three-dimensional Gaussian model.

3. The method according to claim 1, characterized in that, The enhanced feature vectors of each point cloud in the fifth 3D Gaussian model are used to double the number of point clouds in the point cloud data of the fifth 3D Gaussian model through the first pre-trained model to obtain the second 3D Gaussian model. The second 3D Gaussian model is dense point cloud data, including: Based on the enhanced feature vector of each point cloud, the target feature vector of a preset length for each point cloud is predicted by the first pre-trained model. The first pre-trained model is used to split the target feature vector of each point cloud into a preset length to obtain the first feature vector and the second feature vector of each point cloud. For each point cloud, the corresponding augmented point cloud is determined based on the first and second feature vectors of the point cloud to obtain the second three-dimensional Gaussian model. The second three-dimensional Gaussian model is dense point cloud data, including all point clouds of the fifth three-dimensional Gaussian model and the augmented point cloud corresponding to each point cloud.

4. The method according to claim 1, characterized in that, The image rendering based on the coordinate parameters and attribute features of the point cloud of the second three-dimensional Gaussian model to obtain a rendered composite image corresponding to any of the camera extrinsic parameters of the first target object includes: Based on any of the camera extrinsic parameters of the first target object and the attribute features of each point cloud in the second three-dimensional Gaussian model, any point cloud in the second three-dimensional Gaussian model is projected onto a preset two-dimensional plane to obtain a two-dimensional Gaussian distribution of each point cloud. Based on the coordinate parameters of each point cloud in the second three-dimensional Gaussian model, the two-dimensional Gaussian distribution of each point cloud is sorted by depth according to a preset sorting rule; Based on the depth sorting result, the pixel value of each point cloud in the preset two-dimensional plane is determined according to the two-dimensional Gaussian distribution of each point cloud, and the rendered composite image corresponding to any of the camera extrinsic parameters of the first target object is obtained.

5. The method according to claim 1, characterized in that, The calculation of the loss function value between the 2D high-resolution image corresponding to the camera's extrinsic parameters and the rendered composite image, and the adjustment of the network parameters of the first pre-trained model based on the loss function value through backpropagation to obtain a second pre-trained model, wherein the second pre-trained model is a Gaussian splash super-resolution model, including: Calculate the pixel loss, structural loss, and perceptual loss between the two-dimensional high-definition image corresponding to the camera extrinsic parameters and the rendered composite image, respectively. The pixel loss is the pixel distance between the two-dimensional high-definition image and the rendered composite image, the structural loss is the structural similarity loss between the two-dimensional high-definition image and the rendered composite image, and the perceptual loss is the visual perceptual similarity loss between the two-dimensional high-definition image and the rendered composite image. The comprehensive loss is calculated based on the pixel loss, structural loss, and perceptual loss. The comprehensive loss is the value of the loss function. The network parameters of the first pre-trained model are adjusted by backpropagation based on the loss function value to obtain the second pre-trained model, which is a Gaussian splash super-resolution model.

6. A three-dimensional reconstruction device, characterized in that, The device includes: The sample set acquisition module is used to acquire the model training sample set, which includes several data triples. The data triples are a first three-dimensional Gaussian model of any first target object, a corresponding high-definition image set and its camera extrinsic parameters. The first three-dimensional Gaussian model is sparse point cloud data. The high-definition image set includes two-dimensional high-definition images of the first target object taken by two or more camera extrinsic parameters. The first model training module is used to input the first three-dimensional Gaussian model of any first target object into the first pre-trained model, so as to obtain a second three-dimensional Gaussian model through the coordinate parameters and attribute features of the point cloud based on the first three-dimensional Gaussian model by the first pre-trained model. The second three-dimensional Gaussian model is dense point cloud data. The image rendering module is used to render images based on the coordinate parameters and attribute features of the point cloud of the second three-dimensional Gaussian model, and to obtain a rendered composite image corresponding to any of the camera extrinsic parameters of the first target object. The second model training module is used to calculate the loss function value between the two-dimensional high-definition image corresponding to the camera extrinsic parameters and the rendered synthetic image, and to adjust the network parameters of the first pre-trained model based on the loss function value through backpropagation to obtain the second pre-trained model. The second pre-trained model is a Gaussian splash super-resolution model. The three-dimensional reconstruction module is used to respond to the received third three-dimensional Gaussian model of the second target object, and predict and reconstruct a fourth three-dimensional Gaussian model through the Gaussian splash super-resolution model. The fourth three-dimensional Gaussian model is the target three-dimensional model of the second target object. The third three-dimensional Gaussian model is sparse point cloud data, and the fourth three-dimensional Gaussian model is dense point cloud data. The first model training module includes: The data preprocessing unit is used to perform data preprocessing on all point cloud data in the first three-dimensional Gaussian model for any first target object, so as to normalize all point cloud data to a preset coordinate range and obtain a fifth three-dimensional Gaussian model. The feature enhancement unit is used to perform feature enhancement processing on the point cloud data in the fifth three-dimensional Gaussian model using the first pre-trained model with attribute features based on the point cloud, so as to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model. The point cloud doubling unit is used to double the point cloud number of the point cloud data of the fifth three-dimensional Gaussian model based on the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model, and to obtain the second three-dimensional Gaussian model, which is dense point cloud data. The feature enhancement unit is further used for: Based on the attribute features of point clouds, all point cloud data in the fifth three-dimensional Gaussian model are sorted using the space filling curve of the first pre-trained model to obtain sorted point cloud sequence data. The sorted point cloud sequence data is divided into data blocks according to a preset block length to obtain a preset number of point cloud data blocks; For each point cloud data block, the coordinate position of each point cloud in the point cloud data block is enhanced by encoding to obtain a position encoding vector of each point cloud that has the same feature dimension as the point cloud. Based on the location encoding vector of each point cloud, a corresponding query vector, key vector, and value vector are generated for all points in each point cloud data block. For any point cloud in each point cloud data block, the relevance of the point cloud to other point clouds is determined based on the query vector of the point cloud and the key vectors of other point clouds in the same point cloud data block, thus obtaining the attention weight. Using the attention weight as the weight coefficient, the value vectors of all point clouds in the point cloud data block are weighted and summed to obtain the enhanced feature vector of each point cloud in the fifth three-dimensional Gaussian model.

7. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein, when the computer program product is executed, it implements the method described in any one of claims 1-5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-5.

9. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-5.

Images