An implicit three-dimensional scene representation method based on multi-layer dynamic feature point cloud

Abstract

The application discloses an implicit three-dimensional scene representation method based on a multi-layer dynamic feature point cloud and belongs to the field of computer vision. The method comprises the following steps: constructing a three-dimensional scene representation by using a multi-layer implicit feature point cloud; measuring the local feature point cloud complexity of a query point coordinate by using an average nearest neighbor distance; performing feature interpolation based on an adaptive inverse distance weight; converting the interpolated feature into a characteristic distance by using a deep neural network; and further generating a three-dimensional scene triangular mesh by using the characteristic distance. The application uses an implicit multi-layer feature point cloud and can efficiently and accurately represent a three-dimensional scene. The method can be used not only for representing a single three-dimensional object or scene, but also for representing a large-scale three-dimensional data set.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular, the present application relates to an implicit three-dimensional scene representation method based on multi-layer dynamic feature point cloud. BACKGROUND

[0002] Representation of three-dimensional scenes and objects is a heavily researched topic in the field of computer vision, and has fundamental value for downstream applications such as three-dimensional reconstruction, game design, computer modeling, animation and visualization. Traditional three-dimensional representations usually use meshes, point clouds or voxels as media, however these traditional representations are difficult to combine with deep learning frameworks. In recent years, great progress has been made in using neural implicit representation methods to model three-dimensional scenes. This novel expression method parameterizes the entire scene or object as one or more latent variables, and achieves modeling by predicting the zero level surface of the object. Compared with traditional representations, neural implicit representations exhibit better properties such as continuity, generalization, efficiency and flexibility, and achieve better performance in the fields of neural rendering, AR / VR modeling, virtual digital people, etc. However, obtaining a high-fidelity and efficient neural implicit representation is still a great challenge.

[0003] Current neural implicit representation methods for three-dimensional scenes and objects are mainly divided into two categories.

[0004] The first category is global neural implicit representation methods. This category of methods usually parameterizes the entire three-dimensional shape as a single latent code. For example, the DeepSDF method proposed by Jeong Joon Park and Peter Florence (Jeong Joon Park, et al. Deepsdf: Learning continuous signed distance functions for shape representation. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 165-174.) represents a three-dimensional object as a single latent code, and uses a multi-layer perceptron to learn the zero level surface of the object. For the input query point, it is determined whether the point is inside the object, on the object surface or outside the object to obtain its signed distance. They finally use the Marching Cubes algorithm to convert the obtained signed distance geometry into a triangular mesh of the represented three-dimensional object.

[0005] The second type is a local neural implicit representation method. This type of method usually parameterizes the entire three-dimensional shape as a plurality of implicit encodings. Rohan Chabra1 and Jan E. Lenssen et al. (Rohan Chabra1, et al. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. European Conference on Computer Vision. Springer, Cham, 2020: 608-625.) proposed a Deep Local Shapes method, which represents the entire three-dimensional object as a plurality of implicit encodings according to an explicit voxel division, wherein each implicit encoding expresses the geometric information of a part of the local area. By collecting all the local implicit encodings, the overall three-dimensional scene shape can be obtained.

[0006] However, the above two methods cannot meet the current requirements for high-fidelity and high-efficiency neural implicit representation. The first type of method cannot restore fine geometric details due to the limitations of single implicit variable and the expression ability of neural network. The second type of method requires memory in a cubic manner to express geometric details, thereby limiting the highest expression quality. SUMMARY

[0007] In view of the defects of the prior art, in order to obtain a high-fidelity and high-efficiency neural implicit representation method, the present application provides a method based on multi-layer dynamic feature point cloud, which aims to represent complex three-dimensional scenes using a small amount of multi-layer dynamic point cloud.

[0008] In order to achieve the above-mentioned application purposes, the technical solutions adopted by the present application are as follows:

[0009] An implicit three-dimensional scene representation method based on multi-layer dynamic feature point cloud, comprising the following steps:

[0010] S1: constructing a multi-level implicit feature point cloud to represent a three-dimensional scene, wherein the multi-level implicit feature point cloud has a receptive field from large to small, and can represent object geometric details from coarse to fine in multiple levels;

[0011] S2: giving the coordinates of the query point in the unit cube of the point cloud in each direction in turn, and determining the local feature point cloud complexity in the implicit feature point cloud constructed in step S1 by calculating the average nearest neighbor distance;

[0012] S3: adaptively interpolating features at the coordinates of the query point according to the calculated local feature point cloud complexity;

[0013] S4: using the interpolation features obtained in step S3 as the input of the decoder network, generating the indicative distance of the query point coordinate;

[0014] S5: using the indicative distance of all query points, generating the triangular mesh of the three-dimensional scene.

[0015] Further, in step S1, the multi-level implicit feature point cloud is constructed by training a deep learning network by minimizing the loss of the objective function, so that the multi-level implicit feature point cloud can represent the three-dimensional scene geometry from coarse to fine.

[0016] Further, in step S2, the query point coordinates are input in a certain sequence, and the local feature point cloud complexity of the multi-level implicit feature point cloud learned in step S1 is determined, specifically including:

[0017] The random average nearest neighbor metric of the multi-level implicit feature point cloud is calculated;

[0018] According to the input query point coordinates, the nearest neighbor feature point distance is calculated and the average nearest neighbor metric is calculated;

[0019] The ratio of the local average nearest neighbor metric to the random average nearest neighbor metric of the query point coordinates is calculated to obtain the local complexity.

[0020] Further, in step S3, the interpolation weight of the candidate point is adaptively calculated according to the local point cloud complexity obtained in step S2, and the interpolation feature is calculated, specifically including:

[0021] For each query point, a candidate feature point set for calculating the interpolation feature is selected, which is composed of the feature points within the receptive field of the feature point cloud at this layer and their nearest neighbor feature points;

[0022] A piecewise linear function based on trigonometric function is used to construct a nonlinear mapping, so as to construct a nonlinear mapping from the local point cloud complexity to the inverse distance interpolation distance decay parameter, and the piecewise linear function is specifically:

[0023]

[0024] Here α min ,α max is the minimum and maximum value of the distance decay parameter α, and α mean = 0.5×(α min +α max ),α amp = 0.5×(α max -α min );R min ,R maxare user-specified hyper-parameters, representing the lower and upper bound of the local complexity R, respectively.

[0025] Further, in the step S4, the interpolation features obtained in the step S3 are decoded into the indicative distance using a decoder network.

[0026] Further, in the step S5, the indicative distance of all query points is used to generate a triangular mesh of the three-dimensional scene by a marching cubes algorithm.

[0027] The method of the present application can efficiently use multi-level dynamic feature point cloud to represent complex and large-scale three-dimensional object scenes or large-scale three-dimensional data sets. The specific effects include: (1) by multi-level dynamic feature point cloud representation, the traditional fixed static feature point representation method is abandoned, and the feature points can be freely and flexibly distributed around the complex geometry, thereby improving the reconstruction accuracy; (2) an adaptive interpolation weight calculation method is adopted, which simultaneously considers the geometric complexity and feature point complexity, improves the utilization efficiency of the feature points, and makes the overall method more efficient. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flowchart of the method of the present application.

[0029] Figure 2 is an example diagram of three-level feature point cloud in the embodiment of the present application.

[0030] Figure 3 is a schematic diagram of adaptive inverse distance feature interpolation in the embodiment of the present application. Wherein (a) is an overall intuitive schematic diagram of adaptive inverse distance feature interpolation, (b) is a comparison diagram of adaptive inverse distance feature interpolation and standard inverse distance feature interpolation, and (c) is a relationship diagram of distance attenuation coefficient a and local feature point cloud complexity R in adaptive inverse distance feature interpolation.

[0031] Figure 4 is an effect schematic diagram of multi-level modeling from coarse to fine in the embodiment of the present application.

[0032] Figure 5 is a schematic diagram of the reconstruction effect of the three-dimensional scene object in the embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the following will clearly and completely describe the method of the present application with reference to the drawings of the embodiments of the present application. Obviously, the drawings in the following description are part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0034] As Figure 1 shown, a multi-layer dynamic feature point cloud-based implicit three-dimensional scene representation method of the present application, the specific process is as follows:

[0035] (1) Construct a multi-level implicit feature point cloud to represent a three-dimensional scene, wherein the multi-level implicit feature point cloud has a receptive field from large to small, and can represent the geometric details of an object from coarse to fine (as Figure 4 shown).

[0036] An indicative distance function is used to represent a three-dimensional scene and an object, and its representation process can be expressed by the formula: At the same time, d = f(x) is the shortest indicative distance from any point x to the target three-dimensional shape. This indicative distance determines whether the arbitrary point is inside or outside the target object. Therefore, the target three-dimensional shape can be implicitly represented by the zero surface of the indicative function equation as:

[0037]

[0038] Here, the self-decoder network f is used to represent the indicative distance function. At the same time, the multi-level implicit feature point cloud C is used as a conditional input to adapt to different target shapes. Each level of implicit feature point cloud is composed of a series of implicit encodings and its corresponding encoding coordinates . Wherein, n is the number of implicit encodings of each feature point cloud, and m is the dimension of each feature point. Each encoding coordinate records the (x, y, z) spatial coordinates of each implicit encoding. Figure 1 An example schematic diagram of a three-level feature point cloud is shown.

[0039] (2) In the unit cube, the local feature point cloud complexity of the implicit feature point cloud constructed in the previous step is calculated by sequentially inputting the query point coordinates in each dimension direction. First, a standard complexity metric is needed to measure the local point cloud complexity. Here, the Average Nearest Neighbor Statistic is used as the standard measurement standard, which calculates the average nearest neighbor distance of random distribution. If the local average nearest neighbor distance of the query point is greater than the standard metric, it means that the point cloud distribution is relatively complex, otherwise it is relatively simple. Specifically, given a feature point cloud and an input query point x, the standard metric and the observed local metric can be expressed as:

[0040]

[0041] Here V is the volume of the study area. In the standard cube, the value is 1. r exp is the standard metric. r obsis the average nearest neighbor distance of the local region of the query point, which is the average of the distances between the query point and its k nearest neighbors. Thus, the local spatial complexity of the query point is:

[0042]

[0043] (3) The local point cloud complexity obtained by step (2) is used to adaptively calculate the interpolation weight of the candidate point and calculate the interpolated feature. For each query point, a candidate feature point set for calculating the interpolated feature is selected, which is composed of two parts: the feature points within the receptive field of the feature point cloud of the layer and the nearest neighbor feature points. After obtaining the local spatial complexity R(x), the feature interpolation is performed on the query point. Given a query point x, the standard inverse distance interpolation method is:

[0044]

[0045] Here, the distance decay parameter α is a user-specified hyperparameter that remains the same for all query points. The selected ith feature point, w i is the weight corresponding to the feature point. The standard inverse distance interpolation method assumes that the feature interpolation Z(x) will give more weight to nearby feature points and less weight to distant feature points. The present application adaptively adjusts the distance decay parameter α according to the local spatial complexity of the query point. Specifically, when the local complexity R(x0 is high, a smaller α value is used, and vice versa. Therefore, the distance adjustment parameter α should rise or fall synchronously with R(x). This embodiment uses a piecewise linear function based on a trigonometric function to construct the flexible nonlinear mapping, thereby constructing a nonlinear mapping from the local point cloud complexity to the inverse distance interpolation distance decay parameter, and the piecewise linear function is specifically:

[0046]

[0047] Here α min ,α max is the minimum and maximum value of the distance decay parameter α, and α mean = 0.5×(α min +α max ),α amp = 0.5×(α max -α min ). R min ,R max are user-specified hyperparameters, representing the lower and upper limits of the local complexity R, respectively. Figure 3 (c) describes the curve of the function. Figure 3(b) describes the comparison of adaptive inverse distance interpolation with standard methods, using the adaptive inverse distance interpolation method can avoid excessive dependence on the nearest feature points, so as to consider more feature points. After obtaining the distance attenuation parameter a(x), the feature interpolation can be calculated for each layer of dynamic feature point cloud:

[0048]

[0049] Here Indicates n c The candidate latent variables and their corresponding spatial positions. In each layer of dynamic feature point cloud, the candidate feature point set is selected according to two rules. Each feature point has its fixed receptive field size F i (x), which gradually decreases with the increase of the layer number. Therefore, in the i-th layer point cloud, its candidate feature point set is In order to avoid the case that there is no feature point in the receptive field, while forcibly selecting the nearest k = 8 nearest neighbor feature point set Therefore, in the i-th layer point cloud, its final candidate point set is This design can well improve the utilization efficiency and reconstruction accuracy of the feature point cloud.

[0050] (4) Using a self-decoder network f composed of multiple layers of perception to decode the interpolated features obtained in step (3) into indicative distances. The decoder structure is shown in Figure 1 The lower right part. The decoder and the multi-layer dynamic feature point cloud are trained using the L2 Euclidean distance loss function and the encoding position loss function. Specifically, the L2 Euclidean distance loss function is:

[0051]

[0052] Here d, Respectively, the indicative distance predicted by the network and its true value. The learnable multi-level feature point cloud position is updated in position by optimizing the encoding position loss function, while optimizing the decoder parameters:

[0053]

[0054] Here Is the mean square distance matrix between the query point coordinate x and the feature point point cloud P, Composed of normalized weights w i (x). Is the attraction matrix, which describes the mutual relationship between the current query point and the feature point cloud. The larger the value, the higher the importance between the two. At the same time, the gradient backpropagated to the attraction matrix Is truncated, so only The optimization is performed so as to dynamically optimize the positions of the latent variables. By optimizing the encoding position loss function, more latent variables can be guided to distribute on complex geometries while less latent variables on simple geometries, achieving a reasonable and efficient allocation of latent variables.

[0055] (5) Using the characteristic distance d obtained in step (4), finally use the Marching Cubes algorithm to generate a triangular mesh of the three-dimensional scene. Specifically, use the Marching Cubes algorithm in the Scipy library to realize the generation of the mesh from the characteristic distance. Figure 4 An example of a rough-to-fine reconstruction effect diagram is shown. Figure 5 A final reconstruction accuracy effect diagram is shown.

[0056] Although the present application is disclosed as above, it is not limited thereto. Any person skilled in the art, without departing from the spirit and scope of the present application, can make various changes and modifications, therefore the protection scope of the present application should be limited by the scope defined in the claims.

Claims

1. An implicit three-dimensional scene representation method based on multi-layer dynamic feature point cloud, characterized in that, The method comprises the following steps: S1: constructing a multi-level implicit feature point cloud to represent a three-dimensional scene, wherein the multi-level implicit feature point cloud has a receptive field from large to small and can represent object geometric details from coarse to fine in multiple levels; S2: in the unit cube of the point cloud, the coordinates of the query points are given in each direction in turn, and the local feature point cloud complexity is determined by calculating the average nearest neighbor distance in the implicit feature point cloud constructed in step S1; S3: according to the calculated local feature point cloud complexity, the interpolation weight of the candidate point is adaptively calculated and the interpolation feature is calculated, specifically including: For each query point, a candidate feature point set for calculating the interpolation feature is selected, which is composed of the feature points within the receptive field of the feature point cloud at this level and their nearest neighbor feature points; A trigonometric function-based piecewise linear function is used to construct a nonlinear mapping, so as to construct a nonlinear mapping from the local point cloud complexity to the inverse distance interpolation distance decay parameter, and the piecewise linear function is specifically: Here α min ,α max are the minimum and maximum values of the distance decay parameter α, while α mean = 0.5 x (α min + α max ), α amp = 0.5 x (α max - α min ); R min , R max are user-specified hyperparameters that represent the lower and upper values of the local complexity R, respectively; S4: using the feature interpolation obtained in step S3 as the input of the decoder network, the indicative distance of the query point coordinates is generated; S5: using the indicative distances of all query points, the triangular mesh of the three-dimensional scene is generated.

2. The method of claim 1, wherein, In step S1, the multi-level implicit feature point cloud is constructed by minimizing the loss of the objective function to train the deep learning network, so that the multi-level implicit feature point cloud can represent the three-dimensional scene geometry from coarse to fine.

3. The method of claim 1, wherein, In step S2, the query point coordinates are input in a certain sequence, and the local feature point cloud complexity is determined in the multi-level implicit feature point cloud learned in step S1, specifically including: The random average nearest neighbor metric is calculated for the multi-level implicit feature point cloud; According to the input query point coordinates, the nearest neighbor feature point distance is calculated and the average nearest neighbor metric is calculated; The ratio of the local average nearest neighbor metric to the random average nearest neighbor metric of the query point coordinates is calculated to obtain the local complexity.

4. The method of claim 1, wherein, In step S4, the decoder network is used to decode the interpolation feature obtained in step S3 into the indicative distance.

5. The method of claim 1, wherein, In step S5, the indicative distances of all query points are used to generate the triangular mesh of the three-dimensional scene by matching the cube algorithm.

Images