Metacosmic dense-level instance modeling method based on neural implicit representation

Abstract

The invention belongs to the field of computer vision and robots, and discloses a meta-universe dense-level instance modeling method based on neural implicit representation. Obtaining a spatial point coordinate by using the instance image, the depth image and the camera internal reference, generating an object surrounding frame, and converting the spatial point to an object coordinate system; a coordinate and plane joint coding method is adopted to carry out feature coding on the spatial points, multi-level linear interpolation is carried out to obtain object feature vectors and scene feature vectors, feature fusion is carried out, and the feature vectors are decoded to reconstruct an original image; and generating a scene Mesh and an instance scene Mesh through an SDF network by using an error optimization network of output information and a true value and a camera pose estimation result. According to the invention, tracking and map reconstruction are realized at the same time, and the local scene reconstruction precision and the camera tracking precision are improved. And an unknown area in the scene is effectively complemented based on coordinate and plane joint coding. Finally, accurate tracking precision, a high-fidelity reconstruction result and a complete semantic instance graph can be ensured.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and robotics, and in particular to a metaverse dense-level instance modeling method based on neural implicit representation. Background Art

[0002] Three-dimensional modeling is the cornerstone of building the metaverse. In the space of the metaverse, the process of digitizing physical objects is the process of their three-dimensional reconstruction in the virtual space. Combined with digital twins, physical objects in the real world can be mapped in the virtual space to achieve a 1:1 three-dimensional simulation model.

[0003] Accurate scene recognition helps to achieve good 3D reconstruction in the space of the metaverse and better complete the 3D mapping of the real scene. The intervention of instance attributes improves the mapping ability of objects in the 3D scene and realizes a 1:1 3D simulation model. The so-called scene refers to the data of a certain time and place recorded by the sensor in the real world. It contains a combination of various objects. Scene recognition generally revolves around "where is this", and analyzes and judges the current scene by detecting and analyzing the targets in the scene, or performing stable feature extraction. Traditional visual SLAM algorithms can estimate the camera pose with high accuracy and robustness in various scenarios, but cannot predict unobserved viewpoints, and the storage cost of the reconstruction results is high. Neural Radiance Fields provides a novel solution to achieve continuous and dense representation of scenes by using limited memory. "Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM (CVPR), 13293-13302, 2023" uses multi-resolution hash grids for scene representation, which can achieve high convergence speed and the ability to represent high-frequency local features. It also combines one-blob encoding to encourage surface coherence and completion of unobserved areas. This joint parametric coordinate encoding achieves real-time and robust performance by introducing the advantages of two worlds: fast convergence and surface hole filling. However, due to the characteristics of the hash network, it cannot perform an effective completion, nor can it reconstruct the details in the scene in a refined manner. "vMAP: Vectorised Object Mapping for Neural Field SLAM (CVPR), 952-961, 2023" implements object-level SLAM. By representing each object with a small MLP, efficient and dense object modeling can be achieved without 3D priors. When an RGB-D camera browses a scene without prior information, the algorithm detects object instances in real time and dynamically adds them to its map, but the pose is provided by the true value and the reconstruction of occluded areas is poor. "Neural Implicit Dense Semantic SLAM (arXiv)" enables online semantic segmentation. 3D vision-based tracking and loop closure are combined with neural field-based mapping. This method can provide robust tracking, mapping, and semantic labeling by using multiple local mapping networks to expand the pipeline to large scenes. However, the frequency encoding method is used, the code runs slowly, and occupies a large amount of video memory. Summary of the invention

[0004] Aiming at the shortcomings of large data storage and inability to complete unseen areas in previous scene reconstruction, the present invention proposes a dense-level instance modeling method for the metaverse based on neural implicit representation. The method uses coordinate and plane joint encoding to represent scene information. The scene point cloud is obtained by depth and camera intrinsic parameters, and the scene point cloud is clustered by introducing instance information to obtain the object bounding box. The object point cloud is converted to the object coordinate system through the bounding box. The scene feature vector and the object feature vector are obtained by inputting the scene point cloud and the object point cloud into the feature encoding network respectively. In this way, the reconstruction effect of the local scene can be improved while ensuring the scene reconstruction effect. At the same time, in the design of the encoding network, the algorithm uses coordinate and plane joint encoding to represent the scene, and performs multi-level linear interpolation to extract feature vectors to save scene features. At the same time, in order to achieve better reconstruction effect of unseen areas, the algorithm combines One-blob encoding into the encoding network. The scene representation network is optimized by the loss of input image and prediction result. The method can generate high-fidelity scene reconstruction and accurate camera pose estimation, and realize high-fidelity scene reconstruction mapping in the metaverse to serve application scenarios such as navigation and positioning.

[0005] The technical solution of the present invention is as follows: a method for modeling a dense instance of a metaverse based on neural implicit representation, the specific steps are as follows:

[0006] Step 1: Obtain the object point cloud containing instance attributes and the background scene point cloud x through the camera intrinsic parameters, instance image and depth image w ; Use PCA principal component analysis to generate an object bounding box for the object point cloud, and get the object point cloud based on the object bounding box Coordinates in the object coordinate system

[0007] Step 2: The background scene point cloud and the object point cloud are jointly input into three encoding networks based on feature fusion to obtain feature vectors, and the feature vectors are decoded by decoding networks respectively; the decoding networks are SDF network, Color network and Instance decoding network, and a global loss function is designed to optimize the encoding network and decoding network based on feature fusion;

[0008] Step 3: Set the image keyframes through the common view relationship of the scene, use the network prediction results in step 2 and the true value error to predict the current camera pose, and use the BA algorithm to optimize the global pose;

[0009] Step 4: Reconstruct the scene Mesh and instance Mesh through the optimized feature fusion-based encoding network and decoding network.

[0010] Furthermore, the specific process of step 1 is as follows:

[0011] The scene point cloud is obtained based on the camera intrinsic parameters and depth image, and then the instance attributes are assigned to the scene point cloud through the instance image to obtain the object point cloud and the background scene point cloud containing the object ID information; for each type of object point cloud in the space, the PCA principal component analysis method is used to obtain the three main directions of the object point cloud and obtain the center of mass Calculate the covariance and obtain the covariance matrix Find the eigenvalue ε of the covariance matrix n and eigenvectors, the eigenvectors are used as the main directions; according to the calculated main directions and centroids The object point cloud is transformed into the object coordinate system, and the main direction coincides with the direction of the object coordinate system, and the object bounding box of the point cloud transformed to the origin of the object coordinate system is established; by giving the object point cloud Set the main direction and object bounding box, and transform the object point cloud Convert to the object coordinate system;

[0012]

[0013] Furthermore, the specific process of step 2 is as follows:

[0014] In each frame of the image, the same number of background pixels and object pixels are randomly selected through the object mask, and all rays x(t k )=o+t k d, select N sampling rays, and select K points on each sampling ray as sampling points, for example, K = 2048. Predict the color by integrating the predicted value along the sampling ray depth and examples The weight of each sampling point on the sampling ray is parameterized as:

[0015]

[0016] Among them, tr is the cutoff distance, which is set to 0.05cm; s k is the predicted SDF;

[0017] The depth value of each sampling ray is {t1…t K}, the predicted color is {c1…c K}, the prediction instance is {i1…i K}, predict the color, depth and instance of each pixel as:

[0018]

[0019] Furthermore, the encoding network based on feature fusion is specifically:

[0020] First, the 3D space is decomposed into three planes (plane XY, plane XZ, and plane YZ) using orthographic projection, and then the Mipmap (M XY , M XZ and M YZ ) to represent each plane to simulate the pre-filtered feature space; for each Mipmap, the base level Is a feature map of shape H×W×C, where H, W, C are the height, width, and number of channels respectively; basic level For random initialization, other levels From the previous layer The output is scaled down by a factor of 2 along the height and width; this pre-filtering strategy maintains consistency between mipmap levels, which is key to making the reconstructed object consistent at different distances. To query the feature vector f(x) corresponding to a sample point S(x,r), first orthogonally project S on three planes to obtain three disks D = {D XY ,D XZ ,D YZ}; For each disk, query the feature vector from the corresponding Mipmap;

[0021] Based on the properties of orthogonal projection, disk D XY has the same radius r as the sampling sphere, D XY center The two-dimensional coordinates are the partial coordinates (x, y) of the sampling sphere center x(x, y, z); for Mipmap M XY Disc D XY Query level l and assign it to:

[0022]

[0023] Among them, r is Mipmap The radius of the base feature element, B max and B min are the maximum and minimum angles of the axis-aligned bounding box of the 3D space of interest, respectively; after obtaining the query coordinates (x, y, 1), trilinear interpolation is used to obtain the coordinates from Mipmap M XY Get the eigenvector f XY ; Repeat the operation to obtain the eigenvectors f of the other two projection planes XZ and f YZ ; The result f obtained by encoding the spatial coordinates through Oneblod encoding x , the eigenvector is:

[0024] f=f x +(f XY +f XZ +f YZ)(5)

[0025] The final scene expression feature obtained by fusing the scene feature vector and the object feature vector is: Where n represents the instance ID of the object.

[0026] Furthermore, the SDF network outputs the predicted SDF value and feature vector h:

[0027] D S (f sdf (x))→(sdf,h) (6)

[0028] Among them, f sdf (x) is the feature vector input from the encoding network to the SDF network based on feature fusion, D S It is a SDF network based on two-layer MLP;

[0029] The decoded outputs of the Color network and Instance decoding network are:

[0030] D c (f c (x),h)→c D i (f i (x),h)→i (7)

[0031] Among them, f c (x) is the feature vector input to the Color network by the encoding network based on feature fusion, D c is a Color network based on two layers of MLP; f i (x) is the feature vector input from the encoding network to the instance decoding network based on feature fusion; D i It is an Instance decoding network based on two-layer MLP;

[0032] By minimizing the mean square error between the true value and the predicted value and updating the camera pose and image rendering through gradient backpropagation;

[0033] For a valid real depth sampling point r, its real color value is c r , the true depth value is t r ; The rendering loss function for color and depth is defined as follows:

[0034]

[0035] Since instance prediction is a classification problem, the one-hot encoding network H is used to process the decoding results to obtain the real instance results. The rendering loss function of instance rendering is calculated as follows:

[0036]

[0037] To improve the accuracy of scene geometry reconstruction, the truncated signed distance error is used for the sampling depth near the surface, and the truncation region is where |t r - t p | < tr; where t p is the depth ground truth of the distance from the camera center to the object surface, and the distance between the sample point and the ground truth is used as the SDF ground truth for supervision;

[0038]

[0039] where, s p is the sdf value of the sampling point decoded from the SDF feature. For the region where far from the surface with |t r - t p | > tr, the free space loss is used to force the SDF prediction to truncate the distance tr:

[0040]

[0041] Therefore, the global loss function is defined as:

[0042] L = λ fs L fs + λ sdf L sdf + λ t L t + λ rgb L rgb + λ i L i (12)

[0043] where {λ fs , λ sdf , λ t , λ rgb , λ i} are weight factors. It should be noted that the loss calculation requires the light to have the actual depth.

[0044] Furthermore, the specific process of the third step is as follows:

[0045] The estimation of the camera pose is a process of minimizing a loss function optimized by a global Adam optimizer. The current estimation of the camera pose is represented by the transformation matrix from the camera coordinate system to the world coordinate system ; the prediction of the camera pose at time t is initialized by the constant velocity model hypothesis, and a relative pose deviation is calculated from the poses of the previous two frames. Therefore, the camera pose at time t is initialized as:

[0046]

[0047] A total of K rays are randomly sampled from the global keyframe list to optimize the encoding network, decoding network and camera pose. First, the encoding network and decoding network of the current frame are optimized to obtain the rendering effect that meets the requirements of the current frame. Then, the camera parameters {x t} based on the accumulated gradient to update the camera pose T t ; By selecting K pixels from M frames, including the current frame, the first two key frames, and M-3 frames randomly selected from the key frame list, the scene features, object features and parameters of the three decoding networks of M frames are jointly optimized by minimizing the global loss function.

[0048] Furthermore, the specific process of step 4 is as follows:

[0049] By using the scale information of the scene to perform high-resolution space segmentation, the spatial points are input into the optimized encoding network, and the features are decoded through the SDF network, Color network and Instance network to obtain the SDF value, color value and instance value of each scene spatial point, and the scene Mesh and instance Mesh are generated according to the predicted values.

[0050] Beneficial effects of the present invention: In the selection of tracking sample points, the selection is made by using bounding box guidance. Compared with depth-guided sampling, it can track the target better. In addition, key frames are selected for pose optimization. Each key frame is represented by storing only a subset of pixels (about 5%). This enables more frequent insertion of new key frames and maintains a larger key frame database. For joint optimization, a total of K rays are randomly sampled from the global key frame list to optimize the scene representation and camera pose. The joint optimization is performed in an alternating manner. Specifically, k rays are first optimized. m The scene representation of steps θ is used, and the camera parameters {x t} is used to update the camera pose. Since only 6 parameters are used for each camera pose, this method can improve the robustness of camera pose optimization, while the additional computational cost of gradient accumulation is negligible. Selection of global keyframe strategy. By selecting K pixels from M frames, including the current frame, the first two keyframes, and M-3 frames randomly selected from the keyframe list. We jointly optimize the scene features, target features, and MLP decoder of M frames by minimizing the overall loss function. The algorithm does not require a phase optimization strategy and can optimize the entire scene at the same time. On this basis, a method based on coordinate and plane joint encoding is used to fill in the unobserved viewpoints of the scene, which improves the completeness of the scene and achieves high-fidelity reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1This is a schematic diagram of the simulated change of the coordinate transformation of the sampling point object of the present invention;

[0052] Figure 2 This is a schematic diagram of the coordinate and plane joint coding network structure of the present invention;

[0053] Figure 3 Comparison of experimental results of different feature vectors input into the network of the present invention; (a) is the object feature vector; (b) is the scene feature vector; (c) is the feature fusion vector.

[0054] Figure 4 The performance comparison of plane coding and coordinate and plane joint coding using the present invention; (a) is plane coding, (b) is coordinate and plane joint coding.

[0055] Figure 5 Schematic diagram of the algorithm execution flow of the present invention. DETAILED DESCRIPTION

[0056] The following is a specific embodiment, and the specific steps are as follows:

[0057] Step 1: Test the algorithm on Room0 in the Replica dataset. First, read the color, depth, and instance information of the dataset. Use the camera intrinsics and depth image to obtain the spatial coordinate information of the sampling points of the current frame scene, and confirm the instance attributes of each sampling point through the instance image. For each type of object in space, use the PCA principal component analysis method to obtain the three main directions of each object point cloud and obtain the center of mass. Calculate the covariance and obtain the covariance matrix Find the eigenvalue ε of the covariance matrix n and the characteristic vector, the eigenvector is the principal direction. Figure 1 As shown in the figure, by transforming the input point cloud to the origin, and the main direction is back to the coordinate system direction, the bounding box of the point cloud transformed to the origin is established. And by setting the main direction and bounding box for the input point cloud, the inverse transformation of the input point cloud to the object point cloud is realized, and the point cloud is transformed to the object coordinate system.

[0058]

[0059] Step 2: Transform the spatial point x w and object coordinates They are input into the feature encoding network respectively, and the instance is realized by integrating the predicted values ​​along the sampling ray depth and color In each frame of the image, an equal number of background and object pixels are randomly selected using the object mask and all rays x(t k )=o+tk Select 2048 sampling points on d, and parameterize the weight of each point on the ray as:

[0060]

[0061] Where tr is the cutoff distance. k is our predicted SDF. The depth value of each sampled ray is {t1…t K}, the predicted color is {c1…c K}, the prediction instance is {i1…i K} and renders each pixel’s color c, depth t, and instance i as predicted respectively

[0062]

[0063] For the acquisition of feature vectors, the algorithm adopts a method based on coordinate and plane joint encoding, such as Figure 2 As shown, first, the 3D space is decomposed into three planes (plane XY, plane XZ and plane YZ) using orthogonal projection, and then Mipmap (M XY , M XZ and M YZ ) to represent each plane to simulate the pre-filtered feature space. For each Mipmap, the base level It is a feature map of shape H×W×C, where H, W, C are the height, width, and number of channels, respectively. are randomly initialized and can be trained during reconstruction. Other levels From the previous layer derive by downscaling by a factor of 2 along both height and width. This pre-filtering strategy maintains consistency between mipmap levels, which is key to making the reconstructed objects consistent at different distances. To query the feature vector f corresponding to the sphere S(x,r), we first orthogonally project S on three planes to obtain three disks D = {D XY ,D XZ ,D YZ}. For each disk, we query the feature vector from the corresponding mipmap. XY For example, we start from MipmapM XY Query its feature f XY Based on the properties of orthogonal projection, disk D XY has the same radius r as the sampling sphere, D XY center The two-dimensional coordinates of are the partial coordinates (x, y) of the sampling sphere center x(x, y, z). XY Disc DXY Query level l, we assign it to:

[0064]

[0065] Among them, r is Mipmap The radius of the base feature element, B max and B min are the maximum and minimum angles of the axially aligned bounding box (AABB) of the 3D space of interest, respectively. After obtaining the query coordinates (x, y, 1), trilinear interpolation is performed from the Mipmap M XY Get the eigenvector f XY Repeat the operation to obtain the eigenvectors f of the other two projection planes. XZ and f YZ The result obtained by encoding the spatial coordinates through Oneblod encoding is f x , so the eigenvector can be obtained as:

[0066] f=f x +(f XY +f XZ +f YZ ) (18)

[0067] By fusing scene features and object features, the final scene expression feature is obtained as Where n represents the instance ID of the object, so the geometry decoder outputs the predicted SDF value and feature vector h:

[0068] D S (f sdf (x))→(sdf,h) (19)

[0069] The color and instance decoded output is:

[0070] D c (f sdf (x),h)→c D i (f sdf (x),h)→i (20)

[0071] like Figure 3 As shown in the figure, the experiment reconstructs the scene by inputting scene features and object features into the decoder separately and by jointly encoding them. It is obvious that joint encoding can effectively improve the reconstruction effect of the object part in the scene and achieve high-fidelity reconstruction accuracy.

[0072] Rendering is achieved by minimizing the mean squared error between the ground truth and the predicted values ​​and updating the camera pose via gradient backpropagation. For a valid depth sample point r, the rendering loss for color and depth is defined as follows:

[0073]

[0074] In order to make more accurate instance prediction, the one-hot encoding network H is used to predict the instance. The rendering loss of instance rendering is calculated as follows:

[0075]

[0076] To improve the accuracy of scene geometry reconstruction, we use a truncated signed distance error for sampling depths close to the surface, with a truncation region of where |t r -t p | <t r We use the distance between sample points and true values ​​as the SDF ground truth for supervision.

[0077]

[0078] For Region in Stay away from |t r -t p |>t r For surfaces, we use the free space loss to force the SDF prediction to truncate the distance tr:

[0079]

[0080] Therefore, the global loss function is defined as:

[0081] L=λ fs L fs +λ sdf L sdf +λ t L t +λ rgb L rgb +λ i L i (25)

[0082] where {λ fs ,λ sdf ,λ t ,λ rgb ,λ i} is the weight factor. It is worth noting that the loss calculation requires the ray to have actual depth.

[0083] Step 3: The estimation of the camera pose is a process of minimizing the loss function optimized by the global Adam optimizer. The current estimate of the camera pose is calculated using the transformation matrix from the camera coordinate system to the world coordinate system We initialize the pose of the current frame t by assuming constant velocity, which can be designed as:

[0084]

[0085] In the selection of tracking sample points, selection is done by using bounding box guidance. It tracks the target better than depth-guided sampling. In addition, keyframes are selected for pose optimization. Each keyframe is represented by storing only a subset of pixels (approximately 5%). This enables inserting new keyframes more frequently and maintaining a larger keyframe database. For the joint optimization, a total of 2048 rays are randomly sampled from the global keyframe list to optimize the scene representation and camera pose. The joint optimization is performed in an alternating manner. Specifically, k are first optimized. m The scene representation of steps θ is used, and the camera parameters {x t} is used to update the camera pose using the accumulated gradients on the scene. Since only 6 parameters are used for each camera pose, this approach can improve the robustness of camera pose optimization, while the additional computational cost of gradient accumulation is negligible. Selection of a global keyframe strategy. By selecting 2048 pixels from M frames, including the current frame, the first two keyframes, and M-3 frames randomly selected from the keyframe list. We jointly optimize the scene features, target features, and MLP decoder of the M frames by minimizing the overall loss function. The algorithm does not require a phase optimization strategy and can optimize the entire scene at the same time. On this basis, a method based on joint encoding of coordinates and planes is used to fill in the unobserved viewpoints of the scene, which improves the completeness of the scene and achieves high-fidelity reconstruction effects.

[0086] In order to verify the improvement of the algorithm in tracking accuracy, the tracking accuracy of the algorithm was tested in multiple scenes on the Scannet dataset. As shown in Table 1, compared with the current neural implicit SLAM system, the algorithm has improved the accuracy by 2 cm on the basis of the current tracking accuracy.

[0087] Table 1 Data comparison of the tracking accuracy of the present invention under real data sets

[0088]

[0089] Furthermore, step four: using the generated SDF network, Color network and Instance decoding network to complete the reconstruction of the scene Mesh and instance Mesh.

[0090] By using the scale information of the scene to perform high-resolution segmentation of the space, the spatial sampling points are input into the trained SDF network, Color network and Instance network for encoding and decoding to generate scene Mesh and instance Mesh. Figure 4As shown in Figure 1, in order to verify the effectiveness of the joint coding of coordinates and planes in the algorithm, experiments were conducted using plane coding and joint coding of coordinates and planes. The comparison showed that the latter can achieve a high-fidelity scene reconstruction effect. The overall design flow chart of the system is shown in Figure 1. Figure 5 shown.

[0091] This algorithm is suitable for processing the robot's self-positioning and high-fidelity scene reconstruction during the application process, and realizing 1:1 reconstruction of the scene in the Metaverse. When the algorithm is running, it can provide relatively good instance semantic perception and scene repositioning functions while positioning itself. The algorithm can achieve high-fidelity reconstruction of the scene and generate instance maps to facilitate subsequent application to the Metaverse for downstream operations.

Claims

1. A method for modeling dense instances of a metaverse based on neural implicit representation, characterized in that: The specific steps are as follows: Step 1: Obtain the object point cloud containing instance attributes and the background scene point cloud x through the camera intrinsic parameters, instance image and depth image w ; The PCA principal component analysis method is used to generate the object bounding box for the object point cloud, and the object point cloud is obtained according to the object bounding box. Coordinates in the object coordinate system Step 2: The background scene point cloud and the object point cloud are jointly input into three encoding networks based on feature fusion to obtain feature vectors, and the feature vectors are decoded by decoding networks respectively; the decoding networks are SDF network, Color network and Instance decoding network, and a global loss function is designed to optimize the encoding network and decoding network based on feature fusion; Step 3: Set the image keyframes through the common view relationship of the scene, use the network prediction results in step 2 and the true value error to predict the current camera pose, and use the BA algorithm to optimize the global pose; Step 4: Reconstruct the scene Mesh and instance Mesh through the optimized feature fusion-based encoding network and decoding network.

2. The method for modeling a dense instance of a metaverse based on neural implicit representation according to claim 1, characterized in that: The specific process of step 1 is as follows: The scene point cloud is obtained based on the camera intrinsic parameters and depth image, and then the instance attributes are assigned to the scene point cloud through the instance image to obtain the object point cloud and the background scene point cloud containing the object ID information; for each type of object point cloud in the space, the PCA principal component analysis method is used to obtain the three main directions of the object point cloud and obtain the center of mass Calculate the covariance and obtain the covariance matrix Find the eigenvalue ε of the covariance matrix n and eigenvectors, the eigenvectors are used as the main directions; according to the calculated main directions and centroids The object point cloud is transformed into the object coordinate system, and the main direction coincides with the direction of the object coordinate system, and the object bounding box of the point cloud transformed to the origin of the object coordinate system is established; by giving the object point cloud Set the main direction and object bounding box, and transform the object point cloud Convert to the object coordinate system; 3. The dense-level instance-based SLAM reconstruction method based on neural implicit representation according to claim 1, characterized in that: The specific process of step 2 is as follows: In each frame of the image, the same number of background pixels and object pixels are randomly selected through the object mask, and all rays x(t k )=o+t k d, select N sampling rays, select K points on each sampling ray as sampling points, and predict the color by integrating the predicted values ​​along the sampling rays depth and examples The weight of each sampling point on the sampling ray is parameterized as: Among them, tr is the cutoff distance; s k is the predicted SDF; The depth value of each sampling ray is {t1…t K }, the predicted color is {c1…c K }, the prediction instance is {i1…i K }, predict the color, depth and instance of each pixel as:

4. The method for modeling a dense instance of a metaverse based on neural implicit representation according to claim 3, characterized in that: The encoding network based on feature fusion is specifically: First, the 3D space is decomposed into three planes using orthogonal projection, and then each plane is represented by Mipmap to simulate the pre-filtered feature space; for each Mipmap, the base level Is a feature map of shape H×W×C, where H, W, C are the height, width, and number of channels respectively; basic level For random initialization, other levels i=1,2,...,N,from the previous layer It is derived by reducing the height and width by a factor of 2; To query the feature vector f(x) corresponding to the sampling point S(x,r), first orthogonally project S on three planes to obtain three disks D = {D XY ,D XZ ,D YZ }; For each disk, query the feature vector from the corresponding Mipmap; Based on the properties of orthogonal projection, disk D XY has the same radius r as the sampling sphere, D XY center The two-dimensional coordinates are the partial coordinates (x, y) of the sampling sphere center x(x, y, z); for Mipmap M XY Disc D XY Query level l and assign it to: Among them, r is Mipmap The radius of the base feature element, B max and B min are the maximum and minimum angles of the axis-aligned bounding box of the 3D space of interest, respectively; after obtaining the query coordinates (x, y, 1), trilinear interpolation is used to obtain the coordinates from Mipmap M XY Get the eigenvector f XY ; Repeat the operation to obtain the eigenvectors f of the other two projection planes XZ and f YZ ; The result f obtained by encoding the spatial coordinates through Oneblod encoding x , the eigenvector is: f=f x +(f XY +f XZ +f YZ ) (5) The final scene expression feature obtained by fusing the scene feature vector and the object feature vector is: Where n represents the instance ID of the object.

5. The method for modeling a dense instance of a metaverse based on neural implicit representation according to claim 4, characterized in that: The SDF network outputs the predicted SDF value and feature vector h: D S (f sdf (x))→(sdf,h) (6) Among them, f sdf (x) is the feature vector input from the encoding network to the SDF network based on feature fusion, D S It is a SDF network based on two-layer MLP; The decoded outputs of the Color network and Instance decoding network are: D c (f c (x),h)→c D i (f i (x),h)→i (7) Among them, f c (x) is the feature vector input to the Color network by the encoding network based on feature fusion, D c is a Color network based on two layers of MLP; f i (x) is the feature vector input from the encoding network to the instance decoding network based on feature fusion; D i It is an Instance decoding network based on two-layer MLP; By minimizing the mean square error between the true value and the predicted value and updating the camera pose and image rendering through gradient backpropagation; For a valid real depth sampling point r, its color value is c r , the depth value is t r ; The rendering loss function for color and depth is defined as follows: Since instance prediction is a classification problem, the one-hot encoding network H is used to process the decoding results to obtain the real instance results. The rendering loss function of instance rendering is calculated as follows: To improve the accuracy of scene geometry reconstruction, a truncated signed distance error is used for the sampling depth near the surface, and the truncation region is P r tr , where |t r -t p | < tr; where t p is the ground truth depth of the distance from the camera center to the object surface, and the distance between the sample point and the ground truth is used as the SDF ground truth for supervision; Among them, s p The sdf value of the sampling point is decoded from the SDF feature. r fs , where P r fs Stay away from |t r -t p |>tr, using a free space loss to force the SDF to predict a cutoff distance tr: Therefore, the global loss function is defined as: L=λ fs L fs +λ sdf L sdf +λ t L t +λ rgb L rgb +λ i L i (12) where {λ fs ,λ sdf ,λ t ,λ rgb ,λ i } is the weight factor.

6. The method for modeling a dense instance of a metaverse based on neural implicit representation according to claim 1, characterized in that: The specific process of step three is: The current estimate of the camera pose is the transformation matrix from the camera coordinate system to the world coordinate system Indicates that the prediction of the camera's pose at time t is initialized by assuming the pose through the constant velocity model, and a relative pose deviation is calculated through the poses of the first two frames. Therefore, the camera pose at time t is initialized as: A total of K rays are randomly sampled from the global keyframe list to optimize the encoding network, decoding network and camera pose. First, the encoding network and decoding network of the current frame are optimized to obtain the rendering effect that meets the requirements of the current frame. Then, the camera parameters {x t } based on the accumulated gradient to update the camera pose T t ; By selecting K pixels from M frames, including the current frame, the first two key frames, and M-3 frames randomly selected from the key frame list, the scene features, object features and parameters of the three decoding networks of M frames are jointly optimized by minimizing the global loss function.

7. The method for modeling a dense instance of a metaverse based on neural implicit representation according to claim 1, characterized in that: The specific process of step 4 is as follows: By using the scale information of the scene to perform high-resolution space segmentation, the spatial points are input into the optimized encoding network, and the features are decoded through the SDF network, Color network and Instance network to obtain the SDF value, color value and instance value of each scene spatial point, and the scene Mesh and instance Mesh are generated according to the predicted values.

Images