Scene construction method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a scene construction method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of three-dimensional scene reconstruction. The method comprises the following steps: acquiring a true color image and a depth image of the true color image; generating a symbol distance field and a plane parameter according to the true color image and the depth image; and generating a plane field representing a scene according to the symbol distance field and the plane parameters. The dense plane field representing the scene can be generated in the three-dimensional space in a global consistent mode, and the completeness and the accuracy of reconstructing the three-dimensional scene are improved.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application No. 63 / 578,678, filed on August 25, 2023, and application name “PLANEPRIMITIVE FIELD FOR SCENE REPRESENTATION AND INTERACTION IN XR ENVIRONMENT,” which application will be incorporated by reference into this application. Technical Field

[0003] The present application relates to the technical field of three-dimensional scene reconstruction, and more specifically, to a scene construction method, device, electronic device and computer-readable storage medium. Background Art

[0004] Planes (primitives) are key elements in structured environments, such as indoor rooms and urban buildings. Capturing these planes allows for 3D scene reconstruction, which has a significant impact on applications such as virtual reality and augmented reality. However, current 3D scene reconstruction has limitations, for example, only isolated 2D plane segments per view are produced, or the reconstructed scene is incomplete. Summary of the invention

[0005] The embodiments of the present application propose a scene construction method, device, electronic device and computer-readable storage medium to improve the above-mentioned problems.

[0006] In a first aspect, an embodiment of the present application provides a scene construction method, the method comprising: acquiring a true color image and a depth image of the true color image; generating a signed distance field and plane parameters based on the true color image and the depth image; and generating a plane field representing the scene based on the signed distance field and the plane parameters.

[0007] In a second aspect, an embodiment of the present application provides a scene construction device, which includes: an image acquisition module, used to acquire a true color image and a depth image of the true color image; a data generation module, used to generate a signed distance field and plane parameters based on the true color image and the depth image; a scene construction module, used to generate a plane field representing the scene based on the signed distance field and the plane parameters.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein an application is stored in the memory, and the application is configured to cause the processor to execute the method of the present application when called by the processor.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having program code stored thereon, wherein the program code is configured to cause the processor to execute the method of the present application when called by the processor.

[0010] The scene construction method provided in the embodiment of the present application generates a signed distance field and plane parameters based on a true color image and a depth image, and generates a plane field representing the scene according to the signed distance field and the plane parameters, so that a dense plane field representing the scene can be generated in a globally consistent manner in three-dimensional space, and a complete three-dimensional scene can be reconstructed, thereby improving the integrity and accuracy of the reconstructed three-dimensional scene, and facilitating a series of virtual reality interactions such as virtual replacement and games. In addition, the method of the present application can be executed online, which is conducive to matching with the virtual reality experience, so that the user can gradually restore the scene plane while enjoying the virtual reality experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments and drawings obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] Figure 1 A framework diagram of a planar neural radiation field (PlanarNeRF) provided in an embodiment of the present application is shown;

[0013] Figure 2 A schematic diagram of a process flow of a scene construction method provided by an embodiment of the present application is shown;

[0014] Figure 3 A schematic diagram of a global memory bank provided by an exemplary embodiment of the present application is shown;

[0015] Figure 4 A schematic diagram showing a planar field construction process of a three-dimensional scene provided by an exemplary embodiment of the present application is shown;

[0016] Figure 5 A schematic diagram showing a planar field construction process of a three-dimensional scene provided by another exemplary embodiment of the present application is shown;

[0017] Figure 6 A schematic diagram of a process flow of a scene construction method provided by another embodiment of the present application is shown;

[0018] Figure 7 A schematic diagram showing a flow chart of a scene construction method provided by another embodiment of the present application is shown;

[0019] Figure 8 A schematic diagram of the structure of a scene construction device provided in an embodiment of the present application is shown;

[0020] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0022] As mentioned above, current plane detection methods have limitations, for example, only producing isolated 2D plane segments per view, or incomplete reconstructed scenes due to the lack of spatial consistency in correspondence across views. The related technologies of plane detection methods mainly include the following four:

[0023] (1) Single-view plane detection. Single-view plane detection focuses on directly segmenting a plane from a single 2D image. For example, algorithms such as PlaneNet, PlanerRecover, PlaneRCNN, PlaneAE, and PlaneTR. The plane instances detected by single-view plane detection lack consistency between different frames.

[0024] (2) Multi-view plane detection. For example, algorithms such as SparsePlanes, PlaneFormers, NOPE-SAC, PlaneMVS, and PlanarRecon. Multi-view plane detection requires 3D ground truth as a prerequisite and tends to oversimplify the results.

[0025] (3) Dense 3D fitting. For example, RANSAC, GoCoPP and other algorithms. Although dense 3D fitting methods can work effectively without annotations, their computational requirements are large. In addition, some dense 3D fitting methods have poor generalization ability.

[0026] (4) Neural scene reconstruction. The NeRF algorithm introduces an innovative solution for 3D environment representation, and many studies have demonstrated the excellent performance of the NeRF algorithm in scene reconstruction. For example, the NiceSLAM algorithm constructs a series of learnable grid architectures as hierarchical feature encoders and performs pose optimization and dense mapping. The Nicer-SLAM algorithm improves the NeRF algorithm, reduces the need for depth images, and obtains similar reconstruction results. The CoSLAM algorithm uses hash maps instead of grids as feature containers, and introduces coordinate and parameter encoding to speed up convergence and query.

[0027] Based on this, this application proposes Planar Neural Radiance Fields (PlanarNeRF), an online 3D plane detection framework based on Neural Radiance Fields (NeRF), to overcome the limitations of current plane detection methods. Specifically, PlanarNeRF extends the neural field representation to regress planes (primitives) with appearance and geometry, thereby obtaining more complete and accurate plane detection and 3D reconstruction results.

[0028] Here we first introduce the current NeRF. NeRF (Neural Radiance Fields) conceptualizes the scene as a continuous function, usually represented by a multi-layer perceptron (MLP). The function is defined as Map the 3D point x and 2D viewing direction v to the corresponding RGB color c and volume density σ. It is worth noting that the color and density prediction share the same network. For a ray R(t) = o + tv with origin o, the rendered color C(R) is obtained by integrating the points along the ray (the ray emitted by the camera) through volume rendering:

[0029]

[0030] in, is the proximal boundary t n Cumulative transmittance from t to t, t f is the distal boundary. To improve the learning performance of high-frequency details, NeRF encodes the position of 3D coordinates as input. The NeRF model is then trained to minimize the difference between the observed color values ​​and the predicted color values.

[0031] The latest NeRF enhances rendering and reconstruction by modifying the original framework. The latest NeRF modifications include: using signed distance fields (SDF) for prediction, using separate neural networks for enhanced input RGB and geometry (ie depth), recalculating weights in SDF-based rendering equations, and using hashing and one-blob encoding for position data. In addition, deep rendering is used to improve geometric learning of scenes. In the PlanarNeRF of this application, the above latest modifications are incorporated to obtain an updated and optimized color rendering equation:

[0032]

[0033] Where M is the number of sampling points along the ray; ω i is the sampling weight calculated based on SDF: Among them, si is the predicted SDF value along the ray; tr is the predefined SDF cutoff threshold; σ(·) is the sigmoid function. Similar to equation (2), the depth rendering equation is:

[0034]

[0035] Among them, dp i is the depth of the sample point along the ray.

[0036] See also Figure 1 , Figure 1 FIG. 2 shows a framework diagram of a planar neural radiation field (PlanarNeRF) provided in an embodiment of the present application. Figure 1 As shown, the latest NeRF includes hash coding module, one-blob coding, SDF prediction branch (including SDFMLP and weight calculation module), color rendering branch (including color MLP and color rendering equation (2)), depth rendering branch (depth rendering equation (3)). This application adds a plane rendering branch (including plane MLP and plane rendering equation) to NeRF to obtain a planar neural radiation field (PlanarNeRF). Figure 1 As shown, the inputs of the plane MLP and the color MLP are the same, including the one-blob encoded 3D coordinates and the SDF feature vector output by the SDF prediction branch.

[0037] In some embodiments, the input of PlanarNeRF may include a true color image and a depth image, and the output may include an estimated camera pose, color information, SDF, and plane parameters. It is understood that NeRF can use the positioning module on the device (such as a synchronous positioning and mapping module) to estimate the camera pose, so the input of PlanarNeRF may not include the camera pose, and the estimated camera pose can be further optimized.

[0038] In some other embodiments, the input of PlanarNeRF may include camera pose, true color image and depth image, and the output may include color information, SDF and plane parameters. Directly taking the actual accurate camera pose as input can improve the accuracy of PlanarNeRF without optimizing the camera pose.

[0039] It should be noted that the PlanarNeRF of the present application allows dual operation modes, one of which is a supervised mode that utilizes sparse two-dimensional plane annotations (i.e., PlanarNeRF-S mode), and the other is a self-supervised mode that directly extracts planes from depth images (i.e., PlanarNeRF-SS mode). In each mode, PlanarNeRF can effectively learn from sparse training signals, significantly improving training efficiency. In PlanarNeRF-SS, the present application designs a lightweight plane fitting module to estimate plane parameters from a highly sparse set of sampling points in each iteration. Then, a global memory library is maintained to ensure consistent tracking of planes from different views, and labels (including plane parameters and indices) are generated for sparse points to facilitate the generation of plane labels. The multi-view inherent in NeRF Figure 1 The consistency and smoothness are helpful for the propagation of sparse labels.

[0040] PlanarNeRF uses Co-SLAM as its backbone and is able to simultaneously predict pose, geometry, and plane instance segmentation. PlanarNeRF relies on samples in image space to train the MLP. This application adopts a hierarchical training scheme that combines local optimization and global optimization. Local optimization uses samples from each local frame, and global optimization uses the accumulation of historical samples. In PlanarNeRF, lightweight plane fitting, global memory library updates, and plane rendering training only occur in global optimization.

[0041] PlanarNeRF has three MLPs, namely, SDF MLP, color MLP and plane MLP, each of which is a 2-layer shallow MLP with 32 hidden units. During training, we use a learning rate of 1e-3 (i.e. 0.001) to optimize the pose vector and a learning rate of 1e-2 (i.e. 0.01) to optimize all MLPs. The cutoff distance tr in the sampling weights is set to 10 cm.

[0042] Next, the scene construction method of the present application will be introduced. The scene construction method of the present application can be applied to a scene construction device or an electronic device, and the scene construction device can be integrated in an electronic device. Electronic devices may include but are not limited to smart phones, tablet computers, laptops, etc. Electronic devices can communicate with other electronic devices or servers via wired or wireless networks. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, referred to as CDN), and big data and artificial intelligence platforms.

[0043] See also Figure 2 , Figure 2 The schematic diagram of the flow of the scene construction method provided by an embodiment of the present application is shown. The scene construction method may include steps S110 to S130.

[0044] Step S110: Acquire a true color image and a depth image of the true color image.

[0045] The electronic device is integrated with a camera, and the electronic device can obtain true color images (i.e., RGB images) and corresponding depth images in three-dimensional space through the camera. For example, assuming that the electronic device is a virtual reality acquisition device, the user wearing the virtual reality acquisition device can obtain true color images and corresponding depth images under different views in three-dimensional space by adjusting the user's posture (i.e., adjusting the camera posture). Among them, the camera can be a monocular camera.

[0046] In some embodiments, the electronic device can collect true color images and depth images under different views, so as to obtain true color images, depth images and camera poses under different views. In some embodiments, since NeRF can use the positioning module on the electronic device to estimate the camera pose, only the true color image and the depth image can be used as the input of PlanarNeRF. In other embodiments, in order to reduce the optimization of the camera pose, the camera pose, true color image and depth image can be used as the input of PlanarNeRF.

[0047] It should be noted that the method of the present application constructs the scene in an online learning manner. That is, each time a frame of true color image and its depth image are obtained, steps S110 to S130 are repeated once, so that local planes under different views are continuously integrated while the user is experiencing virtual reality, and the three-dimensional scene plane is gradually reconstructed until a complete three-dimensional scene plane field is constructed. It can be understood that, in general, a frame of true color image and depth image can reflect the local plane field of the three-dimensional scene but cannot reflect the complete plane field. In order to subsequently reconstruct the complete three-dimensional scene plane, the number of true color images and depth images obtained under different views should not be too small. The more true color images and depth images obtained under different views, the more complete and accurate the reconstructed three-dimensional scene is, but on the contrary, the longer the scene reconstruction time is.

[0048] Step S120: Generate a signed distance field and plane parameters according to the true color image and the depth image.

[0049] The signed distance field (SDF) samples the distance to the surface of an object in space, using negative values ​​to represent the inside of the object, positive values ​​to represent the outside of the object, and 0 to represent the surface of the object, thereby distinguishing different areas of the object.

[0050] The plane parameters may include the plane parameters of all planes captured by the image, and different plane parameters are used to represent different planes, thereby being used to distinguish and segment different planes. The plane parameters may include parameters of the plane equation, such as the normal vector of the plane, the components of the normal vector of the plane on the X, Y, and Z axes, and the distance between the plane and the origin (i.e., the camera position).

[0051] The electronic device may generate a signed distance field according to the true color image and the depth image; and generate a plane parameter based on the signed distance field.

[0052] In some embodiments, Figure 1 As shown, the electronic device can perform hash coding on the true color image and the depth image; based on the hash coding result, generate a signed distance field and a feature vector of the signed distance field through SDF MLP.

[0053] In some embodiments, Figure 1 As shown, the electronic device can perform one-blob encoding on the true color image and the depth image; and generate plane parameters through plane MLP based on the one-blob encoding result and the feature vector of the signed distance field.

[0054] The plane parameters can be extracted from the depth image through the lightweight plane fitting module in the plane MLP. The plane fitting module can be defined as the following mapping function for constructing the input point to the plane:

[0055]

[0056] Among them, p represents the plane parameter, p i represents the plane parameters of the i-th plane, Forming the plane equation in Represents the components of the plane normal vector of the i-th plane on the X, Y, and Z axes, d i represents the distance between the i-th plane and the ray origin (i.e., the camera position); N is the total number of planes estimated this time; PO represents the input point set of the depth image (points in the signed distance field), PO i represents the set of points belonging to the i-th plane, in n j is the number of points in the set, represents the i-th point in the plane point set; Θ is a set of hyperparameters, Where: m min refers to the minimum number of points that belong to a plane instance; ∈ refers to the absolute maximum tolerance Euclidean distance between a point and the plane; ∈cluster Refers to controlling the connectivity of the points covered by an estimated plane. Larger values ​​usually lead to under-segmentation, while smaller values ​​usually lead to over-segmentation. n Refers to the threshold of normal deviation between a point's normal and the estimated plane normal; is the probability of missing the maximum plane. To obtain satisfactory sparse point plane fitting performance, the hyperparameter Θ must be carefully tuned. As an example, m min =1,∈=0.02,∈ cluster =1,τ n =0.7,

[0057]

[0058] In some embodiments, at each iteration (i.e., each time the plane parameters corresponding to a frame of image are generated), the plane parameters can be incrementally integrated into a global memory library, which is used to store the plane parameters of each plane in the scene. This application adopts a cyclic training iteration method to learn the plane online (based on), and the plane estimates in different iterations using formula (4) are independent of each other. With the continuous influx of new data, there will be a lack of consistency between the planes estimated across frames. Based on this, please refer to Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a global memory library provided by an exemplary embodiment of the present application. The present application proposes a global memory library to maintain plane parameters between different frames, such as Figure 3 As shown in , the global memory is updated with each batch of new estimated planes. The key part of maintaining the global memory is the similarity measure between two planes. According to equation (4), we can get the plane parameters (also called plane vector) of each plane instance.

[0059] In some embodiments, the plane parameters (e.g., p new ) and the plane parameters in the global memory (such as p bank ) (e.g., dist(p new ,p bank )); If the distance is not less than the distance threshold (e.g. τ dist ), the plane parameters extracted by the plane fitting module are stored in the global memory library and the corresponding index is generated to obtain the plane label, thereby integrating the local plane parameters into the globally consistent global memory library. The index can be a label, which can be Given (see equation (5) below). It can be understood that if the PlanarNeRF-S mode is used, it can be assumed that is known. For example, the distance threshold may be 0.1.

[0060] In some embodiments, according to the plane parameters (eg Get a first plane normal vector (e.g., n1) and a first distance (e.g., d1), wherein the first plane normal vector n1 can be based on the components of the plane normal vector of the plane p1 on the X, Y, and Z axes Obtain; According to the plane parameters in the global memory library (such as Get a second plane normal vector (e.g., n2) and a second distance (e.g., d1), wherein the second plane normal vector n2 can be based on the components of the plane normal vector of plane p2 on the X, Y, and Z axes Obtain; The distance between the plane parameters extracted by the plane fitting module and the plane parameters in the global memory library (for example, dist'(p1, p2)) can be determined according to the following formula, based on the first plane normal vector, the first distance, the second plane normal vector and the second distance:

[0061]

[0062] Where p1 = [n1, d1] T ,p2=[n2,d2] T All offset values ​​must be non-negative, since the plane parameter vector and its negative describe the same plane in space, regardless of the normal direction.

[0063] In some embodiments, considering that formula (5) works well as a similarity measure, it is too sensitive to estimation noise and directly comparing two plane parameters lacks robustness to noise. To solve this problem, a simple and robust method can be used to calculate the similarity measure. It can be observed from formula (4) that the plane fitting result also tells the identity of each point on the plane, which means that a plane has two representations, namely, the plane parameters (p i ) or a point belonging to a plane instance The new similarity measure is based on the distance from a point to a plane. Assuming that p1 represents a plane and PO2 represents another plane, we can obtain the points (e.g., ); as shown in the following formula (6), the plane parameters (such as ) and the point Determine the distance between the plane parameters extracted by the plane fitting module and the plane parameters in the global memory:

[0064]

[0065] Where p1 = [n1, d1] T ,p2=[n2,d2]T All offset values ​​must be non-negative, since the plane parameter vector and its negative describe the same plane in space, regardless of the normal direction.

[0066] In some embodiments, in order to further improve the robustness of the global memory library, if a plane with highly similar plane parameters to those extracted by the plane fitting module is found in the memory library, that is, within the distance dist(p new ,p bank ) is less than the distance threshold, as shown in the following formula, the exponential moving average (EMA) is used to update the plane parameters in the global memory according to the plane parameters extracted by the plane fitting module:

[0067] p bank =ψp new +(1-ψ)p bank (7)

[0068] Where ψ is the EMA coefficient. It should be noted that before using EMA to update, the offset value must satisfy the constraints in equation (5).

[0069] Step S130: Generate a plane field representing a scene according to the signed distance field and the plane parameters.

[0070] The signed distance field can be converted into a three-dimensional mesh; based on the three-dimensional mesh and the plane parameters, a plane field representing the scene is generated. The three-dimensional mesh includes the position information of a point in the three-dimensional space. Based on the position information of the point and the plane parameters, it can be determined to which plane the point belongs, thereby realizing the reconstruction of the three-dimensional scene. Specifically, a plane primitive can be generated based on the three-dimensional mesh and the plane parameters; based on the plane primitive, a plane field representing the scene is generated. The present application uses an iterative method to gradually construct a plane field of a three-dimensional scene.

[0071] For example, for the first frame of true color image and depth image, the plane parameters corresponding to the first frame of image can be generated, the plane parameters corresponding to the first frame of image can be stored in the global memory library (initial value is 0), and the plane primitives corresponding to the first frame of image are generated, and based on the plane primitives corresponding to the first frame of image, a part of the plane of the scene is constructed; for the second frame of true color image and depth image, the plane parameters corresponding to the second frame of image can be generated, and the distances between the plane parameters corresponding to the second frame of image (i.e., new plane parameters) and the plane parameters in the global memory library can be calculated one by one, and the plane parameters in the global memory library can be updated according to the plane parameters of the plane with a closer distance, and the new plane parameters in the plane parameters of the plane with a farther distance are added to the global memory library, and the plane primitives corresponding to the second frame of image are generated based on the plane parameters corresponding to the second frame of image, and the plane primitives corresponding to the first frame of image and the second frame of image are fused based on the plane parameters in the global memory library, and so on, until a complete plane field of the three-dimensional scene is constructed. Through the global memory library, the spatial consistency between different frames can be maintained, so as to generate an accurate and complete plane field of the three-dimensional scene.

[0072] As an example, see Figure 4 , Figure 4 A schematic diagram of a planar field construction process of a three-dimensional scene provided by an exemplary embodiment of the present application is shown. The upper row on the right side of PlanarNeRF shows the planar field reconstruction progress of the three-dimensional scene 1, and the lower row shows 2D segmented images at different time steps. Figure 4 It shows the construction of a plane field of a three-dimensional scene based on four frames of true color images and their depth images collected in real time from different viewing angles.

[0073] As another example, see Figure 5 , Figure 5 A schematic diagram of a planar field construction process of a three-dimensional scene provided by another exemplary embodiment of the present application is shown. The upper row on the right side of PlanarNeRF shows the planar field reconstruction progress of the three-dimensional scene 2, and the lower row shows 2D segmented images at different time steps. Figure 5 It shows the construction of a plane field of a three-dimensional scene based on four frames of true color images and their depth images collected in real time from different viewing angles.

[0074] It is understood that, generally, in actual interactive scenarios (such as virtual reality interactive scenarios or augmented reality interactive scenarios), PlanarNeRF can support the output of a planar field of a three-dimensional scene. In particular, if there is a need to view the effect of a 2D scene (such as 2D printing), PlanarNeRF can also support converting a 3D scene into a 2D image.

[0075] The scene construction method provided in the embodiment of the present application can generate a signed distance field and plane parameters based on a true color image and a depth image, and generate a plane field representing the scene according to the signed distance field and the plane parameters, so that a dense plane field representing the scene can be generated in a globally consistent manner in a three-dimensional space, and a complete three-dimensional scene can be reconstructed, thereby improving the integrity and accuracy of the reconstructed three-dimensional scene, and facilitating a series of virtual reality interactions such as virtual replacement and games. In addition, the method of the present application can be executed online, which is conducive to matching with the virtual reality experience, so that the user can gradually restore the scene plane while enjoying the virtual reality experience.

[0076] See also Figure 6 , Figure 6 A schematic flow chart of a scene construction method provided by another embodiment of the present application is shown. The scene construction method may include steps S210 to S240.

[0077] Step S210: Acquire a true color image and a depth image of the true color image.

[0078] Step S220: Generate a signed distance field and plane parameters according to the true color image and the depth image.

[0079] Step S230: Generate a plane field representing a scene according to the signed distance field and the plane parameters.

[0080] Step S240: Optimizing the signed distance field according to the plane field.

[0081] In some embodiments, based on the signed distance field, a sampling weight (ie, the above ω i ); based on the sampling weight and the plane parameters, a plane rendering equation as shown in formula (8) is used to generate a rendering plane; based on the rendering plane and the plane true value, the signed distance field is optimized. Specifically, the loss between the rendering plane (i.e., the plane parameters of the rendering plane) and the plane true value (the plane parameters in the global memory library) can be calculated, and the loss can be calculated according to Figure 1 The gradient is back-propagated in the direction shown, thereby optimizing the signed distance field.

[0082] Based on the sampling weight and the plane parameters, the following plane rendering equation is used to generate a rendering plane:

[0083]

[0084] Where P(R) represents the plane parameters of the sampling points along the ray (i.e., the ray emitted by the camera with the camera position as the origin), M represents the number of sampling points, and ω irepresents the weight of the output of the signed distance field prediction branch of the neural radiation field, R(t) represents the ray emitted by the camera, v represents the viewing direction of the camera, and p i A plane classification probability vector representing a sampling point, wherein the plane classification probability vector represents the probability that the sampling point belongs to a plane in the global memory library.

[0085] In some embodiments, before calculating the loss between the plane parameters (predicted values) output by PlanarNeRF and the true values ​​(i.e., the plane parameters in the global memory library), a correspondence between the predicted values ​​and the true values ​​can be established based on a bipartite graph matching method. The plane segmentation loss based on bipartite graph matching is as follows:

[0086]

[0087] Where Q is the number of pixels; C is the number of classes; is the cth element in the true value y, where m(·) is the matching function and the assignment cost can be given by the intersection-over-union ratio between the predicted value and the true value of each plane. is the predicted probability vector The cth element in . The use of bipartite graph matching stems from the inherent difference in index values ​​between plane segmentation predictions and true values. It is only necessary to match the segmented regions and distinguish different planes.

[0088] In the embodiment of the present application, PlanarNeRF adopts a unique plane segmentation method, that is, a fixed matching technique similar to semantic segmentation is used to calculate the plane segmentation loss, so that spatially consistent 3D planes can be learned. It is necessary for the rendered 2D plane segmentation to be consistent between different frames. In order to maintain this consistency, it is necessary to ensure that the predicted value strictly matches the true value during the loss calculation.

[0089] In some embodiments, according to the depth image and the sampling weight, a rendering equation shown in formula (3) is used to generate a depth rendering image; based on the depth rendering image and the depth image, the signed distance field is optimized. Specifically, the loss between the depth rendering image and the depth image can be calculated, and the loss can be calculated according to Figure 1 The gradient is back-propagated in the direction shown, thereby optimizing the signed distance field.

[0090] In some embodiments, color information is generated based on the depth image and the true color image, such as Figure 1As shown, the one-blob encoding result and the feature vector of SDF can be input into the color MLP to obtain the color information corresponding to the true color image; according to the color information and the sampling weight, the rendering equation shown in formula (2) is used to generate the true color rendering image; based on the true color rendering image and the true color image, the signed distance field is optimized. Specifically, the loss between the depth rendering image and the depth image can be calculated, and the loss can be calculated according to Figure 1 The gradient is back-propagated in the direction shown, thereby optimizing the signed distance field.

[0091] The scene construction method provided in the embodiment of the present application can generate a signed distance field and plane parameters based on a true color image and a depth image, and generate a plane field representing the scene according to the signed distance field and the plane parameters, so that a dense plane field representing the scene can be generated in a globally consistent manner in three-dimensional space, and a complete three-dimensional scene can be reconstructed, thereby improving the integrity and accuracy of the reconstructed three-dimensional scene, and facilitating a series of virtual reality interactions such as virtual replacement and games. In addition, the method of the present application can be executed online, which is conducive to matching with the virtual reality experience, so that the user can gradually restore the scene plane while enjoying the virtual reality experience. In addition, optimizing the signed distance field using the plane field can realize the joint optimization of the entire scene construction process in a self-supervised mode, and improve the smoothness of the signed distance field.

[0092] See also Figure 7 , Figure 7 A schematic flow chart of a scene construction method provided by another embodiment of the present application is shown. The scene construction method may include steps S310 to S340.

[0093] Step S310: Acquire a true color image and a depth image of the true color image.

[0094] Step S320: Generate estimated camera pose, signed distance field and plane parameters according to the true color image and the depth image.

[0095] The electronic device is integrated with a positioning module (eg, a synchronous positioning and mapping module), and the true color image and the depth image can be input into the positioning module to obtain the camera pose estimated by the positioning module.

[0096] Step S330: Generate a plane field representing a scene according to the signed distance field and the plane parameters.

[0097] Step S340: Optimizing the signed distance field and the camera pose according to the plane field.

[0098] In some embodiments, based on the signed distance field, a sampling weight (ie, the above ω i); based on the sampling weights and the plane parameters, the plane rendering equation shown in formula (8) is used to generate a rendering plane; based on the rendering plane and the plane truth value, the signed distance field is optimized. Specifically, the loss between the rendering plane (i.e., the plane parameters of the rendering plane) and the plane truth value (the plane parameters in the global memory library) can be calculated, and the gradient of the loss can be back-propagated to the PlanarNeRF corresponding module and the positioning module, so as to achieve the optimization of the signed distance field and the camera pose.

[0099] In some embodiments, according to the depth image and the sampling weight, a rendering equation shown in formula (3) is used to generate a depth rendering image; based on the depth rendering image and the depth image, the signed distance field is optimized. Specifically, the loss between the depth rendering image and the depth image can be calculated, and the loss can be back-propagated to the PlanarNeRF corresponding module and the positioning module through gradient, so as to achieve the optimization of the signed distance field and the camera pose.

[0100] In some embodiments, color information is generated based on the depth image and the true color image, such as Figure 1 As shown, the one-blob encoding result and the feature vector of SDF can be input into the color MLP to obtain the color information corresponding to the true color image; according to the color information and the sampling weight, the rendering equation shown in formula (2) is used to generate the true color rendering image; based on the true color rendering image and the true color image, the signed distance field is optimized. Specifically, the loss between the depth rendering image and the depth image can be calculated, and the gradient of the loss can be back-propagated to the PlanarNeRF corresponding module and the positioning module, so as to achieve the optimization of the signed distance field and the camera pose.

[0101] The scene construction method provided in the embodiment of the present application can generate a signed distance field and plane parameters based on a true color image and a depth image, and generate a plane field representing the scene according to the signed distance field and the plane parameters, so that a dense plane field representing the scene can be generated in a globally consistent manner in three-dimensional space, and a complete three-dimensional scene can be reconstructed, thereby improving the integrity and accuracy of the reconstructed three-dimensional scene, and facilitating a series of virtual reality interactions such as virtual replacement and games. In addition, the method of the present application can be executed online, which is conducive to matching with the virtual reality experience, so that the user can gradually restore the scene plane while enjoying the virtual reality experience. In addition, the use of the plane field to optimize the signed distance field and the camera pose can realize the joint optimization of the entire scene construction process in a self-supervised mode, and improve the smoothness of the signed distance field and the camera pose.

[0102] In some embodiments, in addition to the self-supervised mode described in steps S210 to S240 and steps S310 to S340, the scene construction method can also support a supervised mode based on plane annotations, including: after each time the plane neural radiation field obtains the plane parameters, the plane segmentation loss is calculated using manually annotated plane labels and plane parameters, thereby performing supervised learning on the plane neural radiation field.

[0103] See also Figure 8 , Figure 8 The schematic diagram of the structure of the scene construction device provided by the embodiment of the present application is shown. The scene construction device 100 includes: an image acquisition module 110, a data generation module 120 and a scene construction module 130. Among them: the image acquisition module 110 is used to acquire a true color image and a depth image of the true color image. The data generation module 120 is used to generate a signed distance field and plane parameters according to the true color image and the depth image. The scene construction module 130 is used to generate a plane field representing the scene according to the signed distance field and the plane parameters.

[0104] In some embodiments, the scene construction module 130 is further configured to convert the signed distance field into a three-dimensional grid; and generate a plane field representing the scene according to the three-dimensional grid and the plane parameters.

[0105] In some embodiments, the scene construction module 130 is further used to generate a plane primitive according to the three-dimensional grid and the plane parameters; and generate a plane field representing the scene based on the plane primitive.

[0106] In some embodiments, the data generation module 120 is further configured to generate a signed distance field according to the true color image and the depth image; and generate plane parameters based on the signed distance field.

[0107] In some embodiments, the data generation module 120 is further used to merge the plane parameters into a global memory library, where the global memory library is used to store the plane parameters of each plane in the scene.

[0108] In some embodiments, the data generation module 120 is further configured to perform hash coding on the true color image and the depth image; and generate a signed distance field and a feature vector of the signed distance field based on the hash coding result.

[0109] In some embodiments, the data generation module 120 is further configured to perform one-blob encoding on the true color image and the depth image; and generate plane parameters based on the one-blob encoding result and the feature vector of the signed distance field.

[0110] In some embodiments, the scene construction device 100 may further include an optimization module, and the optimization module is used to optimize the signed distance field according to the plane field.

[0111] In some embodiments, the optimization module is further configured to generate sampling weights based on the signed distance field; generate a rendering plane based on the sampling weights and the plane parameters; and optimize the signed distance field based on the rendering plane and the plane true value.

[0112] In some embodiments, the optimization module is further configured to generate a depth rendering image according to the depth image and the sampling weights; and optimize the signed distance field based on the depth rendering image and the depth image.

[0113] In some embodiments, the optimization module is further used to generate color information according to the depth image and the true color image; generate a true color rendering image according to the color information and the sampling weights; and optimize the signed distance field based on the true color rendering image and the true color image.

[0114] In some embodiments, the scene construction device 100 may further include a pose estimation module, which is used to estimate the camera pose based on the depth image and the true color image; the optimization module is also used to optimize the camera pose based on the plane field.

[0115] Those skilled in the art can clearly understand that the above device provided in the embodiment of the present application can implement the method provided in the embodiment of the present application. The specific working process of the above-described device and module can refer to the corresponding process of the method in the embodiment of the present application, which will not be repeated here.

[0116] In the embodiments provided in the present application, the coupling, direct coupling or communication connection between the modules shown or discussed may be indirect coupling or communication coupling through some interfaces, devices or modules, and may be electrical, mechanical or other forms, and the embodiments of the present application do not impose specific limitations on this.

[0117] In addition, each functional module in the embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0118] See also Fig. 9 , Fig. 9 The electronic device 200 may include a memory 210 and a processor 220, wherein the memory 210 stores an application program, and the application program is configured to cause the processor to execute the method of the present application when called by the processor.

[0119] The processor 220 may include one or more processing cores. The processor 220 uses various interfaces and lines to connect various parts of the entire electronic device 200, and is used to run or execute instructions, programs, code sets or instruction sets stored in the memory 210, and call to run or execute data stored in the memory 210, perform various functions of the electronic device 200 and process data.

[0120] The processor 220 can be implemented in at least one of the following hardware forms: digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 220 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 220, but may be implemented separately through a communication chip.

[0121] The memory 210 may include a random access memory (RAM) or a read-only memory (ROM). The memory 210 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 210 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may store data created by the electronic device 200 during use, etc.

[0122] An embodiment of the present application also provides a computer-readable storage medium, on which a program code is stored. The program code is configured to execute the method provided in the embodiment of the present application when called by a processor.

[0123] The computer-readable storage medium may be an electronic memory such as a flash memory, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a hard disk or a ROM.

[0124] In some embodiments, the computer readable storage medium includes a non-volatile computer readable medium (Non-Transitory Computer-Readable Storage Medium, referred to as Non-TCRSM). The computer readable storage medium has a storage space for the program code that executes any method step in the above method. These program codes can be read from or written into one or more computer program products. The program code can be compressed in an appropriate form.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A scene construction method, characterized in that: include: Acquire a true color image and a depth image of the true color image; Generate a signed distance field and plane parameters according to the true color image and the depth image; A plane field representing a scene is generated according to the signed distance field and the plane parameters.

2. The method according to claim 1, characterized in that The step of generating a plane field representing a scene according to the signed distance field and the plane parameters comprises: Converting the signed distance field into a three-dimensional grid; A plane field representing a scene is generated according to the three-dimensional grid and the plane parameters.

3. The method according to claim 2, characterized in that The step of generating a plane field representing a scene according to the three-dimensional grid and the plane parameters comprises: Generate a plane primitive according to the three-dimensional grid and the plane parameters; Based on the plane primitives, a plane field representing a scene is generated.

4. The method according to claim 1, characterized in that: The generating a signed distance field and a plane parameter according to the true color image and the depth image comprises: generating a signed distance field according to the true color image and the depth image; Plane parameters are generated based on the signed distance field.

5. The method according to claim 4, characterized in that After generating the plane parameters based on the signed distance field, the method further includes: The plane parameters are merged into a global memory library, where the global memory library is used to store the plane parameters of each plane in the scene.

6. The method according to claim 4, characterized in that The step of generating a signed distance field according to the true color image and the depth image comprises: Performing hash encoding on the true color image and the depth image; Based on the hash coding result, a signed distance field and a feature vector of the signed distance field are generated.

7. The method according to claim 4, characterized in that The generating a plane parameter based on the signed distance field comprises: One-blob encoding the true color image and the depth image; Generate plane parameters based on the one-blob encoding result and the feature vector of the signed distance field.

8. The method according to claim 1, characterized in that The method further comprises: The signed distance field is optimized according to the planar field.

9. The method according to claim 8, characterized in that The optimizing the signed distance field according to the plane field comprises: Based on the signed distance field, generating sampling weights; Generate a rendering plane based on the sampling weight and the plane parameters; A signed distance field is optimized based on the rendering plane and the plane ground truth.

10. The method according to claim 8, characterized in that The method further comprises: Generate a depth rendering image according to the depth image and the sampling weights; A signed distance field is optimized based on the depth rendered image and the depth image.

11. The method according to claim 8, characterized in that The method further comprises: Generate color information according to the depth image and the true color image; Generate a true color rendering image according to the color information and the sampling weight; A signed distance field is optimized based on the true color rendered image and the true color image.

12. The method according to claim 1, characterized in that The method further comprises: Estimate a camera pose based on the depth image and the true color image; The camera pose is optimized according to the planar field.

13. A scene construction device, characterized in that: include: An image acquisition module, used to acquire a true color image and a depth image of the true color image; A data generation module, used for generating a signed distance field and plane parameters according to the true color image and the depth image; The scene construction module is used to generate a plane field representing the scene according to the signed distance field and the plane parameters.

14. An electronic device, characterized in that: include: A memory and a processor, wherein an application is stored in the memory, and the application is configured to cause the processor to execute the method according to any one of claims 1 to 12 when called by the processor.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes are configured to cause the processor to execute the method according to any one of claims 1 to 12 when called by the processor.

Images