Method and apparatus for three-dimensional reconstruction of a scene, electronic device, and storage medium

Abstract

The present application provides a method and apparatus for three-dimensional (3D) reconstruction of a scene, an electronic device, and a storage medium. The method includes: obtaining a two-dimensional (2D) image and 3D point cloud data of a scene to be reconstructed, where the 2D image is a foreground image including a specified object; identifying an object in the 2D image, obtaining 2D data of the object, and performing 3D reconstruction based on the 2D data to obtain first feature data; performing point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data; performing 3D reconstruction based on the object 3D point cloud data to obtain second feature data, and performing 3D reconstruction based on the background 3D point cloud data to obtain third feature data; fusing the first feature data and the second feature data to obtain fusion feature data; and rendering the third feature data and the fusion feature data into a virtual space based on a position relationship between the object and a background to realize 3D reconstruction of the scene to be reconstructed. The method can improve the accuracy of 3D reconstruction of the scene.

Description

METHOD AND APPARATUS FOR THREE-DIMENSIONAL RECONSTRUCTION OF A SCENE, ELECTRONIC DEVICE, AND STORAGE MEDIUM

[0001] Embodiments of the present disclosure relate to the field of computer vision, and in particular to a method and apparatus for three-dimensional (3D) reconstruction of a scene, an electronic device, and a storage medium.

[0002] 3D reconstruction refers to establishing a mathematical model of a 3D object that can be represented and processed by a computer. 3D reconstruction serves as the foundation for processing, operating, and analyzing properties of the 3D object in a computer environment, which is a key technology for establishing virtual reality (VR, virtual reality) that represents the objective world in a computer. There are two main methods to create a 3D model in the computer. One method involves that professional operators use modeling software to generate specific models through human-computer interaction, while the other method utilizes two-dimensional (2D) images or 3D information of objective entities, along with relevant algorithms, for reconstruction.

[0003] Currently, 3D reconstruction techniques can only provide a visual experience. Single-image 3D reconstruction, although cost-effective, poses challenges in achieving precise modeling of scenes or objects and cannot be trained using neural radiance fields. Point cloud 3D reconstruction does not differentiate between scenes and objects, which impacts the overall accuracy of the objects. Therefore, the accuracy of current 3D reconstruction solutions is relatively low.

[0004] In view of the above, the present application provides a method and apparatus for 3D reconstruction of a scene, an electronic device, and a storage medium, which can improve the accuracy of 3D reconstruction of the scene.

[0005] To solve the above technical problem, the technical solution of the present application is realized as follows.

[0006] In one embodiment, a method for 3D reconstruction of a scene is provided, the method including:

[0007] acquiring a 2D image and 3D point cloud data of a scene to be reconstructed, where the 2D image is a foreground image of the scene to be reconstructed, and the foreground image is an image including a specified object;

[0008] identifying an object in the 2D image, obtaining 2D data of the object, and performing 3D reconstruction based on the 2D data to obtain first feature data;

[0009] performing point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data; performing 3D reconstruction based on the object 3D point cloud data to obtain second feature data, and performing 3D reconstruction based on the background 3D point cloud data to obtain third feature data;

[0010] fusing the first feature data and the second feature data to acquire fusion feature data; and

[0011] rendering the third feature data and the fusion feature data into a virtual space based on a position relationship between the object and a background to realize 3D reconstruction of the scene to be reconstructed.

[0012] The method further includes:

[0013] storing the fusion feature data and the third feature data in a database; and

[0014] acquiring, when a change in attributes of the object in the scene is detected, changed attributes of the object and updating the changed attributes into the database.

[0015] The performing 3D reconstruction based on the background 3D point cloud data to obtain third feature data includes:

[0016] performing, in response to determining that the background is a static background, 3D reconstruction based on the background 3D point cloud data using 3D Gaussian splatting (3D-GS) to obtain the third feature data; and

[0017] performing, in response to determining that the background is a dynamic background, 3D reconstruction based on the background 3D point cloud data using a four-dimensional Gaussian splatting (4D-GS) to obtain the third feature data.

[0018] The identifying the object in the 2D image and acquiring 2D data of the object includes:

[0019] denoising redundant information of the 2D image using a non-local means (NLM) algorithm; and

[0020] applying a 2D3DMatch-Net on denoised data to obtain a corresponding relationship between an image and a point cloud; and segmenting the object using a segmentation model to obtain the 2D data of the object.

[0021] The performing point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data includes:

[0022] down-sampling the 3D point cloud data using a voxel grid filter;

[0023] removing outliers of down-sampled data using a statistical outlier removal filter;

[0024] fusing point clouds under different conditions using normal distribution transform; and

[0025] segmenting fused point clouds into fine semantic segmentation data and background segmentation data through a segmentation network, where the fine semantic segmentation data is the object 3D point cloud data, and the background segmentation data is the background 3D point cloud data.

[0026] The performing 3D reconstruction based on the 2D data to obtain first feature data includes:

[0027] obtaining the first feature data corresponding to the 2D data through a single-image 3D reconstruction model, where the single-image 3D reconstruction model is supervised by reference view reconstruction and guided by 2D and 3D joint priors.

[0028] The performing 3D reconstruction based on the object 3D point cloud data to obtain second feature data includes:

[0029] determining each voxel and internal point clouds thereof based on the object 3D point cloud data;

[0030] aggregating all the internal point clouds into a feature vector through an encoder network;

[0031] acquiring four branches corresponding to the feature vector using a decoder network, including: a structure prediction branch, a normal prediction branch, a kernel prediction branch, and a mask prediction branch, where the structure prediction branch outputs 3D features to determine an output hierarchy; the normal prediction branch outputs 3D normal for a subsequent linear system; the kernel prediction branch outputs feature fields according to kernel fields; the mask prediction branch outputs a 16-dimensional feature for determining a relationship between a query position and an actual surface; and

[0032] acquiring features of each point from the hierarchy and the feature fields, and taking a set of features of all points as the second feature data.

[0033] The fusing the first feature data and the second feature data to obtain fusion feature data includes:

[0034] fusing the first feature data and the second feature data using an attention-based network to obtain the fusion feature data.

[0035] In another embodiment, an apparatus for 3D reconstruction of a scene is provided, the apparatus including:

[0036] a data collection unit, configured to acquire a two-dimensional (2D) image and 3D point cloud data of a scene to be reconstructed, where the 2D image is a foreground image of the scene to be reconstructed, and the foreground image is an image including a specified object; identify an object in the 2D image, and obtain 2D data of the object; and perform point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data;

[0037] a data processing unit, configured to perform 3D reconstruction based on the 2D data to obtain first feature data; perform 3D reconstruction based on the object 3D point cloud data to obtain second feature data; perform 3D reconstruction based on the background 3D point cloud data to obtain third feature data; and fuse the first feature data and the second feature data to acquire fusion feature data; and

[0038] a data rendering unit, configured to render the third feature data and the fusion feature data into a virtual space based on a position relationship between the object and a background to realize 3D reconstruction of the scene to be reconstructed.

[0039] In another embodiment, an electronic device is provided that includes a memory, a processor, and computer programs stored on the memory and executable on the processor that, when executed, implement the method for 3D reconstruction of a scene.

[0040] In another embodiment, a computer-readable storage medium is provided that stores thereon computer programs which, when executed by a processor, implement the method for 3D reconstruction of a scene.

[0041] It can be seen from the above technical solution that in the above embodiment, data of the same scene to be reconstructed is divided into a 2D image and 3D point cloud data; the 2D image is a foreground image including a specified object in the scene to be reconstructed, and an object in the 2D image is identified and subjected to 3D reconstruction to obtain first feature data; the 3D point cloud data is segmented into object 3D point cloud data and background 3D point cloud data and then subjected to 3D reconstruction to obtain second feature data and third feature data, respectively; and then the first feature data and the second feature data related to the object are fused to acquire fusion feature data; finally the third feature data and the fusion feature data are rendered into a virtual space based on a position relationship between the object and a background to realize 3D reconstruction of the scene to be reconstructed. Data of two different dimensions is used in this method to perform 3D reconstruction on foreground objects, which is separated from the 3D reconstruction of the background. Then, 3D reconstruction of the scene is realized by associating objects and background through the position relationship, thereby improving the accuracy of 3D reconstruction of the scene.

[0042] To explain the technical solutions in the embodiments of the present application more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments. The drawings in the description below are only some embodiments of the present application; the ordinarily skilled in the art may acquire other drawings according to these drawings without involving any inventive effort.

[0043] Fig. 1 is a flowchart for 3D reconstruction of a scene in an embodiment of the present disclosure;

[0044] Fig. 2 is a flowchart for obtaining 2D data of an object in an embodiment of the present disclosure;

[0045] Fig. 3 is a structural diagram for obtaining 2D data in an embodiment of the present disclosure;

[0046] Fig. 4 is a flowchart for performing point cloud segmentation on 3D point cloud data to acquire object 3D point cloud data and background 3D point cloud data in an embodiment of the present disclosure;

[0047] Fig. 5 is a structural diagram for performing point cloud segmentation on 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data in an embodiment of the present disclosure;

[0048] Fig. 6 is a structural diagram for training a single-image 3D reconstruction model in an embodiment of the present disclosure;

[0049] Fig. 7 is a flowchart for performing 3D reconstruction based on object 3D point cloud data to obtain second feature data in an embodiment of the present disclosure;

[0050] Fig. 8 is a structural diagram for performing 3D reconstruction based on object 3D point cloud data to acquire second feature data in an embodiment of the present disclosure;

[0051] Fig. 9 is a structural diagram of feature fusion in an embodiment of the present disclosure;

[0052] Fig. 10 is a structural diagram of an apparatus for 3D reconstruction of a scene in an embodiment of the present disclosure; and

[0053] Fig. 11 is a physical structure diagram of an electronic device provided by an embodiment of the present disclosure.

[0054] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings of the embodiments of the present application. The embodiments described are only part of but not all embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by the ordinarily skilled in the art without involving any inventive effort fall within the scope of protection of the present application.

[0055] The terms "first", "second", "third", "fourth", and the like in the specification and claims of the present invention and the above drawings, if present, are used for distinguishing between similar objects and not necessarily for describing a sequential or chronological order of targets. The data used in this way may be interchanged in appropriate cases so that the embodiments of the present invention described herein, for example, may be implemented in order other than those illustrated or described here. Furthermore, the terms "include" and "have" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device that includes a list of steps or units is not necessarily limited to those steps or units expressly listed. Still, it may include other steps or units not expressly listed or inherent to such process, method, product, or device.

[0056] The technical solutions of the present invention will be described in detail concerning specific embodiments. The following specific embodiments may be combined, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0057] 3D reconstruction refers to establishing a mathematical model of a 3D object that can be represented and processed by a computer. 3D reconstruction serves as the foundation for processing, operating, and analyzing properties of the 3D object in a computer environment, which is a key technology for establishing VR that represents the objective world in a computer. There are two main methods to create a 3D model in the computer. One method involves professional operators using modeling software to generate specific models through human-computer interaction, while the other method utilizes two-dimensional (2D) images or 3D information of objective entities, along with relevant algorithms, for reconstruction.

[0058] Currently, 3D reconstruction techniques can only provide a visual experience. Single-image 3D reconstruction, although cost-effective, poses challenges in achieving precise modeling of scenes or objects and is incompatible with training using neural radiance fields. Point cloud 3D reconstruction does not differentiate between scenes and objects, which impacts the overall accuracy of the objects. Therefore, the accuracy of current 3D reconstruction solutions is relatively low.

[0059] Based on the above technical problem, a method for 3D reconstruction of a scene is proposed in an embodiment of the present disclosure. Data of the same scene to be reconstructed is divided into a 2D image and 3D point cloud data; the 2D image is a foreground image including a specified object in the scene to be reconstructed, and an object in the 2D image is identified and subjected to 3D reconstruction to obtain first feature data; the 3D point cloud data is segmented into object 3D point cloud data and background 3D point cloud data and then subjected to 3D reconstruction to obtain second feature data and third feature data, respectively; and the first feature data and the second feature data related to the object are fused to obtain fusion feature data; finally the third feature data and the fusion feature data are rendered into a virtual space based on a position relationship between the object and background to realize 3D reconstruction of the scene to be reconstructed. Data of two different dimensions is used in this method to perform 3D reconstruction on foreground objects, which is separated from the 3D reconstruction of the background. Then, 3D reconstruction of the scene is realized by associating objects and background through the position relationship, thereby improving the accuracy of 3D reconstruction of the scene.

[0060] The scene in an embodiment of the present application includes a foreground and a background; the foreground refers to a pre-specified object, for example, the pre-specified object for a certain scene is a chair and a table, then the table and the chair are the foreground, and the contents of pictures other than the specified object serve as the background. In an embodiment of the present application, a foreground (an object) is distinguished from a background based on a pre-specified object and 3D reconstruction is performed using different methods for the foreground and the background.

[0061] In addition, an embodiment of the present application is not limited to 3D reconstruction of a scene with an object, and if there is no object, only 3D reconstruction of the background in the embodiment of the present application is performed; if there is no background, it is only sufficient to perform 3D reconstruction of an object, namely, a foreground, in the embodiment of the present application, that is, the embodiment of the present application is compatible with 3D reconstruction of a scene in different situations.

[0062] A process of 3D reconstruction for a scene in an embodiment of the present application will be described in detail concerning the drawings.

[0063] Referring to Fig. 1, Fig. 1 is a flowchart for 3D reconstruction of a scene in an embodiment of the present application. The specific steps are as follows:

[0064] Step 101: Obtain a 2D image and 3D point cloud data of a scene to be reconstructed, wherein the 2D image is a foreground image of the scene to be reconstructed, and the foreground image is an image including a specified object.

[0065] In the embodiment of the present application, an object is pre-specified, and when performing the 3D reconstruction of the scene, the specified object is taken as a foreground to divide the scene into a foreground and a background. The 2D image and the 3D point cloud are obtained once by the device, for example, a panoramic image of the scene to be reconstructed is scanned by a camera, a 2D image and 3D point cloud data of the scene to be reconstructed may be obtained; the 2D image is one or more images including a specified object among a plurality of images corresponding to the 3D point cloud data. It is usually sufficient to select one image for one object, select one image for a plurality of objects if all of them are in one image, or select a plurality of images if not, that is, select one image for each object, or select only one image if a plurality of objects are in one image.

[0066] Step 102: Identify an object in the 2D image, obtain 2D data of the object, and perform 3D reconstruction based on the 2D data to obtain first feature data.

[0067] In this step, object identification processing is performed based on the 2D image to obtain 2D data of the object; if there is a plurality of objects in the whole scene, each object needs to be labeled to distinguish.

[0068] Step 103: Perform point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data; perform 3D reconstruction based on the object 3D point cloud data to obtain second feature data, and perform 3D reconstruction based on the background 3D point cloud data to obtain third feature data.

[0069] In this step, the foreground (object) and background in the 3D point cloud data are separated, and then subjected to 3D reconstruction to obtain corresponding feature data; if there is a plurality of objects, each object needs to be labeled to distinguish.

[0070] In the specific implementation, steps 102 and 103 may be performed sequentially or simultaneously without any prior order, and this is not limited to the specific implementation of the present application.

[0071] Step 104: Fuse the first feature data and the second feature data to obtain fusion feature data.

[0072] In this step, a multi-modal fusion of object features is adopted. Multi-modal refers to various forms of combination of two or more modalities. In this step, two modalities are used to realize fusion. Different modalities have different performances and can obtain complementary features, to achieve a better reconstruction effect.

[0073] Step 105: Render the third feature data and the fusion feature data into a virtual space based on a position relationship between the object and a background to realize 3D reconstruction of the scene to be reconstructed.

[0074] In the step, the feature data (fusion feature data) related to the foreground (object) and the feature data related to the background is rendered together based on the position relationship to realize the 3D reconstruction of the scene to be reconstructed.

[0075] In the embodiment of the present application, data of the same scene to be reconstructed is divided into a 2D image and 3D point cloud data; the 2D image is a foreground image including a specified object in the scene to be reconstructed, and an object in the 2D image is identified and subjected to 3D reconstruction to obtain first feature data; the 3D point cloud data is segmented into object 3D point cloud data and background 3D point cloud data and then subjected to3D reconstruction to obtain second feature data and third feature data, respectively; and the first feature data and the second feature data related to the object are fused to obtain fusion feature data; finally the third feature data and the fusion feature data are rendered into a virtual space based on a position relationship between the object and a background to realize 3D reconstruction of the scene to be reconstructed. Data of two different dimensions is used in this method to perform 3D reconstruction on foreground objects, which is separated from the 3D reconstruction of the background. Then, 3D reconstruction of the scene is realized by associating objects and background through the position relationship, thereby improving the accuracy of 3D reconstruction of the scene.

[0076] In one example,

[0077] the method for 3D reconstruction of a scene further includes:

[0078] storing the fusion feature data and the third feature data in a database; and

[0079] obtaining, when a change in attributes of the object in the scene is detected, the changed attributes of the object, and updating the changed attributes of the object into the database.

[0080] The relevant scene is usually detected by a detecting device, and whether the scene changes and what changes are determined by comparing the detecting contents of pre-detecting and post-detecting.

[0081] In this way, for a scene in which the attributes of an object change, only the attribute parameters of the object are updated and re-rendered, and the entire model does not need to be reconstructed; the attribute change of the object includes position, color, status, and the like.

[0082] If the scene is an exhibition hall, the exhibits therein are the objects specified in the embodiment of the present application, and it is assumed that a certain exhibit is placed upside down at the time of initial 3D reconstruction, and then after discovering this error, the staff correct it; at this time, the detecting device will discover that the attribute of the object changes, and then obtain the changed attribute and update the changed attribute into the database, therefore, the re-rendering of the scene is triggered without performing 3D reconstruction again, which greatly reduces the time for 3D reconstruction and improves the efficiency.

[0083] In one example,

[0084] a specific process of performing 3D reconstruction based on the background 3D point cloud data to obtain third feature data in step 103 is as follows:

[0085] It is necessary to firstly determine whether the background is a static background or a dynamic background.

[0086] Firstly, in response to determining that the background is a static background, 3D reconstruction is performed based on the background 3D point cloud data using 3D-GS to obtain the third feature data.

[0087] 3D-GS is the smallest element of a scene representation in a 3D Gaussian system, whose attributes include center position μ, opacity α, covariance Σ, and color c.

[0088] The process of performing 3D reconstruction using 3D-GS is as follows:

[0089] Firstly, the 3D Gaussian of all point clouds is obtained.

[0090] Second, these 3D Gaussians are projected onto a pixel-based image plane, a process known as "splatting". To reduce cost, 3D-GS shifts accuracy from pixel-level to patch-level detail and divides the image into a plurality of "tiles".

[0091] Then, the Gaussians are sorted and the value of each pixel is calculated.

[0092] Finally, rendering is performed according to the sorted Gaussians.

[0093] Secondly, in response to determining that the background is a dynamic background, 3D reconstruction is performed based on the background 3D point cloud data using a 4D-GS to obtain the third feature data.

[0094] The 4D-GS converts the original 3D GaussiansGinto another set of 3D GaussiansG' based on a given timestampt. Specifically, it uses an efficient space-time structure encoder and a Gaussian deformation decoder to predict the deformation of each 3D Gaussian and finally concatenates the transformed GaussianG' according to the time stamp to obtain a continuous 3D dynamic rendering scene.

[0095] Both 3D-GS and 4D-GS support real-time rendering.

[0096] A specific process of identifying an object in the 2D image and obtaining 2D data of the object in step 102 is illustrated in Fig. 2; Fig. 2 is a flowchart for obtaining 2D data of an object in an embodiment of the present application. The specific steps are as follows:

[0097] Step 201: Redundant information of the 2D image is denoised using an NLM algorithm.

[0098] Step 202: A 2D3DMatch-Net is applied on the denoised data to obtain a corresponding relationship between an image and a point cloud.

[0099] The step is a specific implementation of image registration.

[0100] Step 203: The object is segmented using a segmentation model to obtain the 2D data of the object.

[0101] In this step, object recognition is achieved using a segmentation model (such as dense prediction transformer (DPT)) to segment objects, and in order to distinguish and match objects, each object may also be labeled; in the specific implementation, each object may be labeled using YOLOv9.

[0102] Referring to Fig. 3, Fig. 3 is a structural diagram for obtaining 2D data in an embodiment of the present application. In Fig. 3, the specified object is a chair; an image including the object is firstly obtained from a scene, then denoising processing is performed on the object image, then image registration is performed to establish a relationship between the image and a point cloud, and finally, a target object (a chair) is determined through target recognition, and 2D data of the target object is obtained, and the object is labeled.

[0103] A specific process of performing point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data in step 103 is illustrated in Fig. 4; Fig. 4 is a flowchart for performing point cloud segmentation on 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data in an embodiment of the present application. The specific steps are as follows:

[0104] Step 401: Down-sample the 3D point cloud data using a voxel grid filter.

[0105] The step is implemented to compress large-capacity samples without destroying the sample geometry and to leave a representative sample therein.

[0106] Step 402: Remove outliers of the down-sampled data using a statistical outlier removal filter.

[0107] The step is implemented to avoid complex local point cloud feature estimation or failures of point cloud registration.

[0108] Step 403: Fuse point clouds under different conditions using normal distribution transform.

[0109] The point cloud registration step implements in this step. The different conditions here involve device, time perspective, coordinates, and the like.

[0110] Step 404: Segment the fused point clouds into fine semantic segmentation data and background segmentation data through a segmentation network, wherein the fine semantic segmentation data is the object 3D point cloud data, and the background segmentation data is the background 3D point cloud data.

[0111] The segmentation network herein may use LidarMultiNet but is not limited thereto.

[0112] Referring to Fig. 5, Fig. 5 is a structural diagram for performing point cloud segmentation on 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data in an embodiment of the present application. In Fig. 5, the 3D point cloud data is filtered through a statistical outlier removal filter, and then point clouds under different conditions are fused through point cloud registration; the fused point clouds are segmented into fine semantic segmentation data and background segmentation data by the segmentation network.

[0113] A specific process of performing 3D reconstruction based on the 2D data to obtain first feature data in step 102 is as follows:

[0114] the first feature data corresponding to the 2D data is obtained through a single-image 3D reconstruction model, wherein the single-image 3D reconstruction model is supervised by reference view reconstruction and guided by 2D and 3D joint priors.

[0115] Referring to Fig. 6, Fig. 6 is a structural diagram for training a single-image 3D reconstruction model in an embodiment of the present application. In Fig. 6, at the 3D reconstruction stage of single-image, the model is supervised by the reference view reconstruction and guided by 2D and 3D joint priors. A new view of noise rendering is generated for both 2D and 3D priors, while separately adding text prompts and camera poses. The training of the single-image 3D reconstruction model includes the training of a coarse-stage 3D model and the training of a fine-stage 3D reconstruction model.

[0116] In the coarse stage, the model will focus on optimizing the underlying geometric structure of an image, and a low-resolution semi-finished 3D model is generated using a neural radiation field; specifically, a new visual angle and a source visual angle are obtained through the neural radiation field, then noise rendering is performed for the new visual angle, and then text prompts and camera poses are separately added at the same time to acquire 2D priors and 3D priors for determining a 2D loss function and a 3D loss function; a first model training loss function is determined in the coarse stage according to the 2D loss function and the 3D loss function together with a reconstruction loss function determined by a reconstruction visual angle of a source visual angle to train the model in the coarse stage. In the fine stage, the model will focus on fine texture rendering and use deep marching tetrahedra (DMTet) to generate a high-resolution and high-precision 3D model. The specific implementation of the fine stage is as follows: A new visual angle and a source visual angle are obtained through a DMTet network, then noise rendering is performed for the new visual angle, and then text prompts and camera poses are separately added at the same time to acquire 2D priors and 3D priors for determining a 2D loss function and a 3D loss function; a second model training loss function is determined in the fine stage according to the 2D loss function and the 3D loss function together with a reconstruction loss function determined by a reconstruction visual angle of a source visual angle to train the model in the fine stage.

[0117] After the training of the coarse stage model and the fine stage model, a single-image 3D reconstruction model is completed, that is, 2D image data may be input to the single-image 3D reconstruction model to obtain first feature data.

[0118] The single-image 3D reconstruction model may be generated using Magic123 but is not limited thereto. 3D features for post-multimodal fusion may be extracted by the single-image 3D reconstruction model without a complete rendering process.

[0119] A specific process of performing 3D reconstruction based on object 3D point cloud data to obtain second feature data in step 103 is illustrated in Fig. 7; Fig. 7 is a flowchart for performing 3D reconstruction based on object 3D point cloud data to obtain second feature data in an embodiment of the present application. The specific steps are as follows:

[0120] Step 701: Determine each voxel and internal point clouds thereof based on the object 3D point cloud data.

[0121] Step 702: Aggregate all the internal point clouds into a feature vector through an encoder network.

[0122] Step 703: Obtain four branches corresponding to the feature vector using a decoder network, namely, a structure prediction branch, a normal prediction branch, a kernel prediction branch, and a mask prediction branch.

[0123] The structure prediction branch can output 3D features to determine output hierarchy; the normal prediction branch can output 3D normal for a subsequent linear system; the kernel prediction branch can output feature fields according to kernel fields; the mask prediction branch can output a 16-dimensional feature for determining a relationship between a query position and an actual surface.

[0124] Step 704: Obtain features of each point from the hierarchy and the feature fields, and take a set of features of all points as the second feature data.

[0125] Step 705: Segment the fused point cloud into fine semantic segmentation data and background segmentation data through a segmentation network; wherein the fine semantic segmentation data is object 3D point cloud data, and the background segmentation data is background 3D point cloud data.

[0126] Referring to Fig. 8, Fig. 8 is a structural diagram for performing 3D reconstruction based on object 3D point cloud data to obtain second feature data in an embodiment of the present application. In Fig. 8, voxels and internal point clouds thereof are determined for an object 3D point cloud data; all the internal point clouds are then aggregated into feature vectors using a PointNet encoder; four branches are output using a sparse U-Net decoder, including structure prediction, normal prediction, kernel prediction, and mask prediction. The structure prediction branch outputs 3D features to determine and the output hierarchy; the normal prediction branch outputs 3D normal for a subsequent linear system; the kernel prediction branch outputs feature fields according to kernel fields; the mask prediction branch outputs a 16-dimensional feature for determining a relationship between a query position and an actual surface. 3D features of each point are acquired from the predicted hierarchy and the feature fields, and the 3D features of all the points constitute second feature data, for subsequent multi-modal analysis.

[0127] The object point cloud reconstruction method herein may be implemented using neural kernel surface reconstruction (NKSR) but is not limited thereto. Through the method, the 3D features of multi-modal fusion may be extracted without a complete surface reconstruction process.

[0128] A specific flow of fusing the first feature data and the second feature data to acquire fusion feature data in step 104 includes:

[0129] fusing the first feature data and the second feature data using an attention-based network to obtain the fusion feature data.

[0130] Multi-modal refers to various forms of combination of two or more modalities. Different modes have different performances, so we use late fusion based on feature level, in order to obtain complementary features to achieve better results.

[0131] In the previous steps, the first 3D feature (first feature data) is obtained from a single image containing 2D information and a second 3D feature (second feature data) is obtained from a point cloud containing 3D shapes and structures.

[0132] Feature fusion is achieved through an attentional network, and reference is made specifically to Fig. 9; Fig. 9 is a structural diagram of feature fusion in an embodiment of the present application. Firstly, all the features will be processed by different long short-term memory (LSTM) (first LSTM and second LSTM) to extract time-dependent latent features; the latent features will then be passed to the self-attention layer to achieve parallel training while having global information; after that, the latent features will be connected through the connection layer and transferred to the next linear layer; finally, Softmax is used as the activation function and the last layer of the network.

[0133] After the model training is complete, the features of the connection layer will be used for rendering.

[0134] The method for 3D reconstruction of a scene in the embodiment of the present application may be applied to a metaverse, and has the following advantages compared with the related art:

[0135] Firstly, the object and the scene can be processed separately by this method, and ensure the high-precision reconstruction of the both can be ensured, thereby avoiding the impact of the whole scene on the object.

[0136] Secondly, the reconstruction of static scenes and dynamic scenes can be supported by this method.

[0137] Thirdly, fully automatic or semi-automatic modeling can be achieved by simply collecting all the data at once and the data can be updated through multi-device collaboration without the need for 3D reconstruction again by this method.

[0138] Fourthly, different interaction requirements between users and object scenes can be met, and it is easy for users to manage and experience through this method.

[0139] The units of the above embodiments may be integrated as a whole or deployed separately, which may be combined into a single unit or further split into a plurality of sub-units.

[0140] The method for 3D reconstruction of a scene in the embodiment of the present application may be applied to a smart home scene, a VR game entertainment scene, a VR tourism scene, an automated factory management scene, and the like.

[0141] The 3D reconstruction application for smart homes is as follows:

[0142] When users are outdoors or have limited mobility, they can enter the metaverse through VR and interact with objects in the real world to control them. For example, when pressing a switch in the metaverse, the users can directly turn off the lights in the home without moving. In addition, they can control air conditioners, rice cookers, and other Internet of Things (IoT) devices.

[0143] When users want to adjust the layout of their home, they can open the model directly on the tablet, interact with the furniture (drag and drop, zoom, change colors, and the like), and preview the final effect. In addition, by cooperating with the furniture supplier, the users can also preview the decorative effect of the new furniture desired to be purchased.

[0144] The 3D reconstruction application for VR game entertainment is as follows:

[0145] Escape games and live role-playing games / murder mystery games require a large amount of space while the number of people in the same space is also limited. Through the method for 3D reconstruction of a scene of the present application, a provider can make richer scenes and scripts with less time and cost, and reconstruct a real place and relevant props in a virtual space, and a user can invite friends to play together in a metaverse.

[0146] The 3D reconstruction application for VR tourism is as follows:

[0147] Augmented reality (AR) tourism and 360° panoramic VR have been widely used, but the need for interactive tourist attractions to be experienced by users has not been realized. With the method for 3D reconstruction of a scene provided in the present application, a provider does not need to use complex modeling software and merely needs to "move" places and devices into VR through 3D reconstruction, so that a user can enjoy beauty scenes or play related entertainment devices at any time and anywhere.

[0148] The 3D reconstruction application for automated factory management is as follows:

[0149] VR technology is gradually being applied to traditional factory management. Through the above method, users can easily reconstruct their factories, and achieve remote factory monitoring and automatic factory management through smartphones. The users can supervise the factory at any time through the materialization of information to determine whether the operation status of different workshops, production lines, machines, goods, and the like is normal, and can directly interact with the faulty machine.

[0150] High-precision maps may be constructed by 3D reconstruction and applied to the field of automatic driving.

[0151] The method may further be used to provide a user with an efficient home storage solution by adding a recommendation system based on historical position information and query / usage frequency of parts of the items.

[0152] The method may further be used to construct diverse metaverse communities by adding large language models (LLMs).

[0153] The solution provided by the embodiments of the present application does not need to use an additional chip to track the position and status of each object for real-time updating, only several laser radar devices are required to perform multi-device cooperative updating; all the data only need to be collected once, which greatly saves hardware device and time costs.

[0154] The object and scene are processed separately, and different loss functions and algorithms are adopted for different tasks, which solves the problem that the reconstruction effect of different entities can not be balanced by the previous single algorithm, and avoids the impact of real-time dynamic scene modeling on a single object.

[0155] The solution also enables people with disabilities and living difficulties to experience different scenes of the real world in the metaverse and interact freely with objects using only intelligent devices.

[0156] All of the above alternative technical solutions may be combined arbitrarily to form alternative embodiments of the present disclosure, and will not be described in detail herein.

[0157] Based on the same inventive concept, an apparatus for 3D reconstruction of a scene is further provided in the embodiments of the present application. Referring to Fig. 10, Fig. 10 is a structural diagram of apparatus for 3D reconstruction of a scene in an embodiment of the present application. The apparatus includes:

[0158] a data collection unit 1001, configured to obtain a two-dimensional (2D) image and 3D point cloud data of a scene to be reconstructed, where the 2D image is a foreground image of the scene to be reconstructed, and the foreground image is an image including a specified object; identify an object in the 2D image, and obtain 2D data of the object; and perform point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data;

[0159] a data processing unit 1002, configured to perform 3D reconstruction based on the 2D data to obtain first feature data; perform 3D reconstruction based on the object 3D point cloud data to obtain second feature data; perform 3D reconstruction based on the background 3D point cloud data to obtain third feature data; and fuse the first feature data and the second feature data to obtain fusion feature data; and

[0160] a data rendering unit 1003, configured to render the third feature data and the fusion feature data into a virtual space based on a position relationship between the object and a background to realize 3D reconstruction of the scene to be reconstructed.

[0161] In another embodiment, the apparatus includes:

[0162] a storage unit 1004, configured to store the fusion feature data and the third feature data in a database; and

[0163] a detecting unit 1005, configured to obtain, when a change in attributes of the object in the scene is detected, changed attributes of the object and updating the changed attributes into the database.

[0164] In another embodiment, the apparatus includes:

[0165] a data processing unit 1002, configured to, when specifically performing 3D reconstruction based on the background 3D point cloud data to obtain third feature data, perform, in response to determining that the background is a static background, 3D reconstruction based on the background 3D point cloud data using3D-GS to obtain the third feature data, and perform, in response to determining that the background is a dynamic background, 3D reconstruction based on the background 3D point cloud data using a 4D-GS to obtain the third feature data.

[0166] In another embodiment, the apparatus includes:

[0167] a data processing unit 1002, configured to, when specifically identifying the object in the 2D image and acquiring 2D data of the object, denoise redundant information of the 2D image using an NLM algorithm, apply a 2D3DMatch-Net on denoised data to obtain a corresponding relationship between an image and a point cloud, and segment the object using a segmentation model to obtain the 2D data of the object.

[0168] In another embodiment, the apparatus includes:

[0169] a data processing unit 1002, configured to, when specifically performing point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data, convert the 3D point cloud data into data in a PCD format, down-sample the format-converted data using a voxel grid filter, remove outliers of down-sampled data using a statistical outlier removal filter, fuse point clouds under different conditions using normal distribution transform, segment fused point clouds into fine semantic segmentation data and background segmentation data through a segmentation network, where the fine semantic segmentation data is the object 3D point cloud data, and the background segmentation data is the background 3D point cloud data.

[0170] In another embodiment, the apparatus includes:

[0171] a data processing unit 1002, configured to, when specifically performing 3D reconstruction based on the 2D data to obtain first feature data, acquire the first feature data corresponding to the 2D data through a single-image 3D reconstruction model, where the single-image 3D reconstruction model is supervised by reference view reconstruction and guided by 2D and 3D joint priors.

[0172] In another embodiment, the apparatus includes:

[0173] a data processing unit 1002, configured to, when specifically performing 3D reconstruction based on the object 3D point cloud data to acquire second feature data, determine each voxel and internal point clouds thereof based on the object 3D point cloud data; aggregate all the internal point clouds into a feature vector through an encoder network; obtain four branches corresponding to the feature vector using a decoder network, including: a structure prediction branch, a normal prediction branch, a kernel prediction branch, and a mask prediction branch, where the structure prediction branch outputs 3D features to determine an output hierarchy; the normal prediction branch outputs 3D normal for a subsequent linear system; the kernel prediction branch outputs feature fields according to kernel fields; the mask prediction branch outputs a 16-dimensional feature for determining a relationship between a query position and an actual surface; and acquire features of each point from the hierarchy and the feature fields, and take a set of features of all points as the second feature data.

[0174] In another embodiment, the apparatus includes:

[0175] a data processing unit 1002, configured to, when specifically fusing the first feature data and the second feature data to acquire fusion feature data, fuse the first feature data and the second feature data using an attention-based network to obtain the fusion feature data.

[0176] In another embodiment, there is further provided an electronic device including a memory, a processor, and computer programs stored on the memory and executable on the processor; the processor, when executing the computer programs, executes the method for 3D reconstruction of a scene.

[0177] In another embodiment, there is further provided a computer-readable storage medium storing thereon computer instructions that, when executed by a processor, implement a method for 3D reconstruction of a scene.

[0178] Fig. 11 is a physical structure diagram of an electronic device provided by an embodiment of the present invention. As shown in Fig. 11, the electronic device may include a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140, where the processor 1110, the communication interface 1120, and the memory 1130 complete communication with each other via the communication bus 1140. The processor 1110 may call logic instructions in memory 1130 to perform the following method:

[0179] obtaining a 2D image and 3D point cloud data of a scene to be reconstructed, where the 2D image is a foreground image of the scene to be reconstructed, and the foreground image is an image including a specified object;

[0180] identifying an object in the 2D image, obtaining 2D data of the object, and performing 3D reconstruction based on the 2D data to obtain first feature data;

[0181] performing point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data; performing 3D reconstruction based on the object 3D point cloud data to obtain second feature data, and performing 3D reconstruction based on the background 3D point cloud data to obtain third feature data;

[0182] fusing the first feature data and the second feature data to obtain fusion feature data; and

[0183] rendering the third feature data and the fusion feature data into a virtual space based on a position relationship between the object and a background to realize 3D reconstruction of the scene to be reconstructed.

[0184] Moreover, the logic instructions in the above memory 1130, if implemented in the form of a software function unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, in part contributing to the prior art, or in part thereof, may be embodied in the form of a software product stored in a storage medium including instructions for causing a computer device (which may be a personal computer, a server, or a network device, and the like) to execute all or part of the steps of the method of various embodiments of the present invention. The foregoing storage medium may include various media that can store the program code, such as U-disk, removable hard disk, read-only memory (ROM), random-access memory (RAM), and magnetic or optical disks.

[0185] The apparatus embodiments described above are merely schematic, where the elements illustrated as separate elements may or may not be physically separated, and the elements shown as units may or may not be physical elements, that is, may be located in one place, or may also be distributed over a plurality of network units. Some or all of the modules may be selected to achieve the objectives of the embodiments according to actual needs. The ordinarily skilled in the art would have been able to understand and implement the same without involving any inventive effort.

[0186] From the above description of the embodiments, it will be clear to the skilled in the art that the implementations may be achieved via software plus a necessary general hardware platform, but of course also via hardware. Based on such an understanding, the above technical solution, in essence or part contributing to the prior art, may be embodied in the form of a software product; the computer software product may be stored in a computer-readable storage medium, such as a ROM / RAM, and a magnetic or optical disk, including several instructions to cause a computer device, which may be a personal computer, a server, or a network device, to execute the methods of the various embodiments or portions of the embodiments.

[0187] The flowcharts and block diagrams in the drawings of the present application illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments disclosed in the present application. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or portion of code, which includes one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the various drawings. For example, two connectively represented blocks may be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functionality involved. It will also be noted that each block in the block diagrams or flowcharts, and combinations of blocks in the block diagrams or flowcharts, may be implemented by special hardware-based systems which perform the specified functions or operations, or by combinations of special hardware and computer instructions.

[0188] It will be appreciated by the skilled in the art that various combinations of features recited in the various embodiments and / or claims disclosed in the present application may be made even if such combinations are not expressly recited in the present application. Specifically, various combinations of features recited in the various embodiments and / or claims of the present application may be made without departing from the spirit and teachings of the present application, and all such combinations fall within the scope disclosed by the present application.

[0189] While the principles and implementations of the present invention have been described herein in connection with specific embodiments, illustration of the foregoing embodiments is intended to aid in the understanding of the methods and principles of the present invention, and is not intended to limit the present application. For the skilled in the art, the implementations and application scope may be changed according to the idea, spirit, and principle of the present invention, and any modification, equivalent replacement, and improvement made by the skilled shall be included in the scope of protection of the present application.

Claims

1.A method for three-dimensional (3D) reconstruction of a scene, comprising:obtaining a two-dimensional (2D) image and 3D point cloud data of a scene to be reconstructed, wherein the 2D image is a foreground image of the scene to be reconstructed, and the foreground image is an image comprising a specified object;identifying an object in the 2D image, obtaining 2D data of the object, and performing 3D reconstruction based on the 2D data to obtain first feature data;performing point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data;performing 3D reconstruction based on the object 3D point cloud data to obtain second feature data, and performing 3D reconstruction based on the background 3D point cloud data to obtain third feature data;fusing the first feature data and the second feature data to obtain fusion feature data; andrendering the third feature data and the fusion feature data into a virtual space based on a position relationship between the object and a background to realize 3D reconstruction of the scene to be reconstructed.2.The method according to claim 1, further comprising:storing the fusion feature data and the third feature data in a database; andobtaining, when a change in attributes of the object in the scene is detected, changed attributes of the object and updating the changed attributes into the database.3.The method according to claim 1, wherein the performing 3D reconstruction based on the background 3D point cloud data to obtain third feature data comprises:performing, in response to determining that the background is a static background, 3D reconstruction based on the background 3D point cloud data using 3D Gaussian splatting (3D-GS) to obtain the third feature data; andperforming, in response to determining that the background is a dynamic background, 3D reconstruction based on the background 3D point cloud data using a four-dimensional Gaussian splatting (4D-GS) to obtain the third feature data.4.The method according to claim 1, wherein the identifying the object in the 2D image and acquiring 2D data of the object comprises:denoising redundant information of the 2D image using a non-local means (NLM) algorithm; andapplying a 2D3DMatch-Net on denoised data to obtain a corresponding relationship between an image and a point cloud; andsegmenting the object using a segmentation model to obtain the 2D data of the object.5.The method according to claim 1, wherein the performing point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data comprises:down-sampling the 3D point cloud data using a voxel grid filter;removing outliers of down-sampled data using a statistical outlier removal filter;fusing point clouds under different conditions using normal distribution transform; andsegmenting fused point clouds into fine semantic segmentation data and background segmentation data through a segmentation network, wherein the fine semantic segmentation data is the object 3D point cloud data, and the background segmentation data is the background 3D point cloud data.6.The method according to claim 1, wherein the performing 3D reconstruction based on the 2D data to obtain first feature data comprises:obtaining the first feature data corresponding to the 2D data through a single-image 3D reconstruction model, wherein the single-image 3D reconstruction model is supervised by reference view reconstruction and guided by 2D and 3D joint priors.7.The method according to claim 1, wherein the performing 3D reconstruction based on the object 3D point cloud data to obtain second feature data comprises:determining each voxel and internal point clouds thereof based on the object 3D point cloud data;aggregating all the internal point clouds into a feature vector through an encoder network;obtaining four branches corresponding to the feature vector using a decoder network, including: a structure prediction branch, a normal prediction branch, a kernel prediction branch, and a mask prediction branch, wherein the structure prediction branch outputs 3D features to determine an output hierarchy; the normal prediction branch outputs 3D normal for a subsequent linear system; the kernel prediction branch outputs feature fields according to kernel fields; the mask prediction branch outputs a 16-dimensional feature for determining a relationship between a query position and an actual surface; andobtaining features of each point from the hierarchy and the feature fields, and taking a set of features of all points as the second feature data.8.The method according to claim 1, wherein the fusing the first feature data and the second feature data to obtain fusion feature data comprises:fusing the first feature data and the second feature data using an attention-based network to obtain the fusion feature data.9.An apparatus for three-dimensional (3D) reconstruction of a scene, comprising:a data collection unit, configured to acquire a two-dimensional (2D) image and 3D point cloud data of a scene to be reconstructed, wherein the 2D image is a foreground image of the scene to be reconstructed, and the foreground image is an image comprising a specified object; identify an object in the 2D image, and obtain 2D data of the object; and perform point cloud segmentation on the 3D point cloud data to obtain object 3D point cloud data and background 3D point cloud data;a data processing unit, configured to perform 3D reconstruction based on the 2D data to obtain first feature data; perform 3D reconstruction based on the object 3D point cloud data to obtain second feature data; perform 3D reconstruction based on the background 3D point cloud data to obtain third feature data; and fuse the first feature data and the second feature data to acquire fusion feature data; anda data rendering unit, configured to render the third feature data and the fusion feature data into a virtual space based on a position relationship between the object and a background to realize 3D reconstruction of the scene to be reconstructed.10.The apparatus according to claim 9, further comprising:a storage unit configured to store the fusion feature data and the third feature data in a database; anda detecting unit configured to obtain, when a change in attributes of the object in the scene is detected, changed attributes of the object and updating the changed attributes into the database.11.The apparatus according to claim 9, wherein the data processing unit is configured to:perform 3D reconstruction based on the background 3D point cloud data to obtain third feature data comprises,perform, in response to determining that the background is a static background, 3D reconstruction based on the background 3D point cloud data using 3D Gaussian splatting (3D-GS) to obtain the third feature data, andperform, in response to determining that the background is a dynamic background, 3D reconstruction based on the background 3D point cloud data using a four-dimensional Gaussian splatting (4D-GS) to obtain the third feature data.12.The apparatus according to claim 9, wherein the data processing unit is configured to:denoise redundant information of the 2D image using a non-local means (NLM) algorithm,apply a 2D3DMatch-Net on denoised data to obtain a corresponding relationship between an image and a point cloud, andsegment the object using a segmentation model to obtain the 2D data of the object.13.The apparatus according to claim 9, wherein the data processing unit is configured to:down-sample the 3D point cloud data using a voxel grid filter,remove outliers of down-sampled data using a statistical outlier removal filter,fuse point clouds under different conditions using normal distribution transform, andsegment fused point clouds into fine semantic segmentation data and background segmentation data through a segmentation network, wherein the fine semantic segmentation data is the object 3D point cloud data, and the background segmentation data is the background 3D point cloud data.14.The apparatus according to claim 9, wherein the data processing unit is configured to:obtain the first feature data corresponding to the 2D data through a single-image 3D reconstruction model, wherein the single-image 3D reconstruction model is supervised by reference view reconstruction and guided by 2D and 3D joint priors.15.A computer-readable storage medium storing thereon computer programs, wherein the programs, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images