Graph reconstruction method based on nearest neighbor prior and low-frequency component fitting and related device

Abstract

The invention discloses a graph reconstruction method based on nearest neighbor prior and low-frequency component fitting and a related device, and the method comprises the steps: obtaining the point cloud data of a to-be-reconstructed graph, and constructing a nearest neighbor prior network and a symbol distance network; initializing the symbol distance network, and repeatedly sampling query points from the periphery of the input point cloud; optimizing the weight of the symbolic distance network by using a projection prediction result given by the priori network to obtain a symbolic distance field representing the graph to be reconstructed; and according to the symbol distance field of the to-be-reconstructed graph, using a Marching Cubes algorithm to extract the surface of the to-be-reconstructed graph to complete graph reconstruction. According to the method, data-driven nearest neighbor priori is used for replacing a symbol distance field accurate value to serve as optimized supervision information. And finally, position coding in a trigonometric function form is used to enhance the tendency of a neural network fitting symbol to a low-frequency component of a field so as to obtain a smoother reconstruction result.

Description

Technical Field

[0001] The present application belongs to the field of graphics reconstruction technology, and relates to a graphics reconstruction method based on nearest neighbor prior and low-frequency component fitting and a related device. Background Art

[0002] Point cloud is a relatively simple 3D shape representation format. Users usually need to reconstruct point clouds into polygon meshes for further editing or rendering, and restore continuous surfaces (represented as polygon meshes) from discrete point clouds, which is called surface reconstruction.

[0003] In the classic surface reconstruction problem, the algorithm relies on the sparsity of the point cloud and the ability to provide normal information, otherwise the stability will be seriously affected. Deep neural networks have excellent generalization and fitting capabilities, so in recent years, a large number of exploratory works using deep neural networks to reconstruct surfaces have been produced. Deep neural networks with specific structures themselves have a tendency to fit continuous or smooth functions. In addition, deep learning technology can learn features that are difficult to describe using analytical methods from a large amount of data. Such features can also be used to ensure the rationality of the appearance of the reconstruction results. These methods learn global shape features, which work better when the object to be reconstructed belongs to a category already in the database, and have poor results when reconstructing objects that do not belong to a known category or are difficult to classify into a specific category. Therefore, some algorithms for learning local geometric features have also been derived.

[0004] In summary, most of the existing methods take dense point clouds as input, and the results of surface reconstruction based on sparse point clouds are relatively rare. In addition, the existing methods of surface reconstruction based on sparse point clouds still have problems such as low accuracy, local fragmentation and holes in the results. Summary of the invention

[0005] The purpose of this application is to solve the problems in the prior art and provide a method and related device for image reconstruction based on nearest neighbor prior and low frequency component fitting. The surface reconstruction algorithm described in this application uses the signed distance field as an intermediate form to improve shape accuracy and reconstruction stability as much as possible while ensuring good topological properties of the reconstruction result.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions: In a first aspect, the present application proposes a graphics reconstruction method based on nearest neighbor prior and low-frequency component fitting, comprising the following steps: Obtain the point cloud data of the graphics to be reconstructed, and construct the nearest neighbor prior network and the signed distance network; Initialize the signed distance network and repeatedly sample query points around the input point cloud; The projection prediction results given by the prior network are used to optimize the weights of the signed distance network to obtain a signed distance field representing the graphics to be reconstructed; According to the signed distance field of the image to be reconstructed, the Marching Cubes algorithm is used to extract the surface of the image to be reconstructed to complete the image reconstruction.

[0007] Furthermore, before initializing the signed distance network, the nearest neighbor prior network is trained, and the data for training the nearest neighbor prior network are query points, K-nearest neighbor sets and nearest neighbor points collected from polygonal meshes and their corresponding point clouds, and the loss function used for training is Euclidean distance.

[0008] Furthermore, the neighbor prior network It is a neural network with an encoding-decoding structure, which is used to K-nearest neighbors in the input point cloud Predict the nearest neighbors of a query point on the surface to be reconstructed .

[0009] Further, all the K-nearest neighbors The points in the query point are transformed to In the local coordinate system with the origin as the origin, we get the transformed K-nearest neighbor As The input of the nearest neighbor prior network makes the output of the nearest neighbor prior network independent of the global coordinates of the query point; Represents the query point coordinates.

[0010] Furthermore, the use of the projection prediction results given by the prior network to optimize the weights of the signed distance network is to use the nearest neighbor prior network The given result replaces the accurate value of the signed distance field as supervision. The specific method is as follows: 1) Randomly sample several query points near the input point cloud ; 2) Input the query point into the signed distance network and calculate the approximate signed distance value at the point and the gradient vector , substitute Find the sampling point of the query point on the zero isosurface of the current signed distance function; where, Represents the projection result calculated based on the value and gradient of the signed distance network; 3) Find the K-nearest neighbors of the query point in the input point cloud , and transform the K-nearest neighbors to the local coordinate system with the query point as the origin to obtain the transformed K-nearest neighbors ; 4) The transformed K-nearest neighbors Input the nearest neighbor prior network to get the nearest neighbor point predicted by the prior network ; 5) Calculate the average Euclidean distance between all query points and their corresponding nearest neighbor points As projection loss , plus the Eikonal regularization term that constrains the gradient of the signed distance function Get the total loss function value , according to the total loss function value Compute the back-propagation optimized signed distance network; where, represents the weight of the projection loss, Represents the weight of the Eikonal regularization term.

[0011] Furthermore, the step of extracting the surface of the graphic to be reconstructed using the Marching Cubes algorithm according to the signed distance field of the graphic to be reconstructed includes: Add a position encoding layer before MLP to enhance the signed distance function The tendency of fitting the low-frequency components of the object's signed distance field; input the query point coordinates When , the position encoding layer first maps the coordinates to a low-frequency basis function space as follows, and takes the encoding result and the original coordinates as input:

[0012] in, represents the input vector obtained after position encoding, represents the weight of the lowest frequency component, Represents the encoding vector of the lowest frequency component, " indicates vector concatenation operation, represents the weight of the highest frequency component, represents the encoding vector of the highest frequency component, Indicates The encoding vector of frequency components, Indicates the number of the frequency component; The position encoding is performed in the form of trigonometric functions in the order of low to high frequency, and gradually added during the optimization process , and the weight of each frequency component control:

[0013] in, Indicates the optimization progress. Indicates the initial value of the optimization progress, Indicates the number of rounds of the current optimization iteration. represents the upper limit of the number of optimization iterations, Indicates The weight control parameters of the frequency components, Indicates truncation to the specified lower limit (0) and upper limit (1). Indicates the number of the highest frequency component, Indicates the number of the lowest frequency component, Indicates The weight of the frequency components; like is 0, then at the end of the signed distance function optimization, the weight of the highest frequency component just increases to 1; if If it is greater than 0, the weights of all frequency components will increase to 1 and then the optimization will continue for several rounds before ending. As the number of optimization rounds increases, the signed distance function gradually converges to the object's signed distance field, and its isosurface will also converge to the surface of the object accordingly.

[0014] In the second aspect, the present application proposes a graphics reconstruction system based on nearest neighbor prior and low-frequency component fitting, comprising: A network construction module is used to obtain point cloud data of the graphics to be reconstructed and to construct a nearest neighbor prior network and a signed distance network; The network initialization module is used to initialize the signed distance network and repeatedly sample query points from around the input point cloud; A graphics optimization module, used to optimize the weights of the signed distance network using the projection prediction results given by the prior network, to obtain a signed distance field representing the graphics to be reconstructed; The graphics reconstruction module is used to extract the surface of the graphics to be reconstructed using the Marching Cubes algorithm according to the signed distance field of the graphics to be reconstructed, and complete the graphics reconstruction.

[0015] In a third aspect, the present application proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0016] In a fourth aspect, the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and wherein the computer program implements the steps of the above method when executed by a processor.

[0017] In a fifth aspect, the present application proposes a computer program product, characterized in that the computer program product includes computer instructions, a processor of a computer device reads the computer instructions, and the processor of the computer device executes the computer instructions to implement the steps of the above method.

[0018] Compared with the prior art, this application has the following beneficial effects: This application uses local nearest neighbor prior information to provide supervision for the optimization process, so that it can process sparse point clouds without line information. The position encoding technology is used to enhance the tendency of the neural network to fit the low-frequency components of the signed distance field, so that the reconstructed surface is smoother. While improving the reconstruction effect, the performance requirements are maintained at a low level, and the algorithm can be run on a mid-end consumer-grade GPU. This application uses data-driven nearest neighbor priors instead of the exact value of the signed distance field as the supervisory information for optimization. Finally, position encoding in the form of trigonometric functions is used to enhance the tendency of the neural network to fit the low-frequency components of the signed distance field to obtain a smoother reconstruction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A flowchart of the method of the present application.

[0021] Figure 2 This is the schematic diagram of the system of this application.

[0022] Figure 3 This is a neural network structure diagram of the nearest neighbor prior network in this application.

[0023] Figure 4 Schematic diagram of the process of reconstructing a surface from a point cloud in this application.

[0024] Figure 5 This is a diagram of the gradual convergence process during reconstruction of this application.

[0025] Figure 6 This is a comparison chart of the reconstruction results of this application and existing similar algorithms. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0027] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0028] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0029] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invented product is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0030] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0031] In the description of the embodiments of the present application, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0032] Point cloud is a relatively simple three-dimensional shape representation format. Users usually need to reconstruct point clouds into polygonal meshes (Polygon Mesh) for further editing or rendering. Restoring continuous surfaces (represented as polygonal meshes) from discrete point clouds is called surface reconstruction. The simplest point cloud uses several points in three-dimensional space to roughly represent a shape without any other information. It is one of the easiest three-dimensional shape formats to obtain. However, such point clouds cannot provide continuous surface shape and normal vector information, and are not suitable as input for various downstream tasks in computer graphics, and are relatively lacking in practical value. Users usually need to reconstruct point clouds into polygonal meshes (Polygon Mesh, sometimes also called "discrete surfaces" or "piecewise linear surfaces" in the field of computational geometry) for further editing or rendering. The problem of restoring continuous surfaces (represented as polygonal meshes) from discrete point clouds is called surface reconstruction. It is one of the classic research topics in the field of computer graphics, and there are many effective reconstruction algorithms in academia and industry.

[0033] In the classic surface reconstruction problem, the input of the algorithm is mostly dense (the point cloud representing a single object contains tens of thousands to millions of points) and point clouds with normal (normal vector) information. When the point cloud to be reconstructed is quite sparse (the point cloud of a single object contains only hundreds to thousands of points) and no normal information is given, the reconstruction algorithm designed for the classic problem cannot work stably, and may even reconstruct a completely wrong shape. In order to solve the problem of reconstructing a highly sparse point cloud without normals into a surface, this application specifically designs a reconstruction algorithm based on the characteristics of such point clouds, in order to obtain stable and reasonable reconstruction results.

[0034] To use a reconstruction algorithm that takes a dense point cloud as input, you must first obtain dense point cloud data with normals. To obtain this point cloud data from the real world, you can only use high-precision 3D scanning equipment to scan the object for a long time and in all directions. This scanning process is difficult to apply to large-sized objects (such as vehicles, ships, aircraft, etc.), and the scanning equipment used is also quite precise and expensive. The high cost of data acquisition limits the practical application scope of these reconstruction algorithms (such as the digitization of a few high-value cultural relics and high-precision reverse engineering). In recent years, the development of hardware equipment and software algorithms has spawned new methods for obtaining point clouds, two of which are low-cost and large in quantity, and have the potential to become a new source of data for surface reconstruction problems.

[0035] One type is the various depth cameras that are gradually maturing and becoming popular. They have fast shooting speeds (up to dozens of frames per second) and large scanning ranges (up to several meters or even more than ten meters). The depth images they capture can also be easily converted into point cloud data, greatly reducing the cost of acquiring point clouds. The defects of depth cameras are that their resolution and accuracy are much lower than those of high-precision scanning devices, and objects can only be photographed from one perspective at a time, so the resulting point cloud is sparse, slightly inaccurate, and incomplete. In order to solve the problem of incomplete point clouds, researchers have designed various point cloud registration algorithms (combining multiple incomplete point clouds into a complete object shape) and point cloud completion algorithms (directly completing the missing parts of the point cloud). However, the complete point clouds given by these algorithms are usually still sparse and do not carry normal information, and the surface cannot be directly reconstructed using traditional surface reconstruction algorithms.

[0036] The other type is point cloud generation algorithms driven by deep learning technology. This type of algorithm uses the generalization ability of deep learning technology to learn the general shape features of various objects from a large number of artificial object data sets, and then generates objects that do not exist in the data set (represented by point clouds) through random sampling. Compared with obtaining point clouds from the real world, directly generating point clouds is cheaper and has more diverse results, which has considerable potential reference value for designers. However, most generation algorithms are affected by the instability of the results of the neural network itself and the characteristics of the data. The output point clouds are also sparse and have no normals, so they are rarely used effectively. The number of such point clouds is huge (hundreds or even thousands can be easily obtained), and even if only a small part of them can be well reconstructed, a considerable number of 3D model assets will be obtained.

[0037] Designing a surface reconstruction algorithm that takes sparse point clouds as input is helpful for downstream applications such as point cloud completion and point cloud generation. On the other hand, it also explores methods for situations different from classic surface reconstruction problems, which can accumulate more theoretical and experimental results for surface reconstruction algorithms to combat the problem of bad input.

[0038] Deep neural networks have excellent generalization and fitting capabilities, so in recent years, a large number of exploratory works using deep neural networks to reconstruct surfaces have been produced. Deep neural networks with specific structures have a tendency to fit continuous or smooth functions. For example, Atzmon et al. proved that multi-layer perceptrons (MLP) can fit piecewise linear surfaces (i.e., polygonal meshes), and Gadelha et al. further proved that convolutional neural networks (CNN) also have the same ability. Therefore, some reconstruction algorithms directly use deep neural networks themselves instead of data-driven deep learning techniques for reconstruction. For example, Deep Geometric Prior divides the entire point cloud into several overlapping parts, each of which is fitted by an MLP, and constrains the values ​​of the overlapping parts to ensure that the local surfaces fitted by each MLP can be well stitched together. Point2Mesh and SAIL-S3 algorithms both use multiple networks to fit the field functions of different parts separately, but use shared weights to maintain global consistency. On the other hand, deep learning technology can learn features that are difficult to describe using analytical methods from a large amount of data, and such features can also be used to ensure the rationality of the appearance of the reconstruction results. This type of data-driven reconstruction algorithm usually first uses a large amount of data to train a priori network for regularization, and then uses the priori network to constrain the fitting network of the output surface during the reconstruction process. Based on large-scale object databases represented by ShapeNet, algorithms such as Deep Marching Cubes, OccNet, and DeepSDF constrain the reconstruction results by learning category-level object shape features. Since these algorithms learn global shape features, they work better when the object to be reconstructed belongs to a category already in the database, and they work worse when reconstructing objects that do not belong to a known category or are difficult to classify into a specific category. According to the tests of Badki et al., some algorithms cannot obtain correct reconstruction results after simply rotating the point cloud to change the posture of the object to be reconstructed, while algorithms that learn local geometric features can overcome this difficulty well. According to the different methods used to divide the object to be reconstructed, algorithms that learn local geometric features can be divided into two categories: spatial division and surface division. Spatial partitioning is to divide the space where the entire object is located into several blocks. For example, IF-Nets and ConvOccNet divide the space evenly into cubic grids and learn the general shape features at the scale of a single grid, while SSRNet uses octrees for uneven partitioning to adapt to the details of different scales of each part of the object. Surface partitioning is to divide the point cloud into several pieces, each piece corresponds to a connected area on the surface. For example, Meshlet and PatchNets use several smaller local surfaces to splice into a complete reconstruction result.

[0039] Most of the above-mentioned reconstruction algorithms require dense point clouds as input. Although Meshlet has preliminarily explored the problem of reconstructing surfaces from relatively sparse point clouds (about 5,000 to 20,000 points), there is still a large gap with the data targeted by this application (about 2,000 points). At present, there are relatively few results on how to reconstruct surfaces from sparse point clouds. More successful examples include reconstruction algorithms based on surface decision function (ODF) prior knowledge, GeoUDF based on local weighted average prediction of unsigned distance, and SuperUDF based on local optimal projection. The reconstruction algorithm with surface decision prior as the core adopts the optimization strategy of Neural Pull, and uses the data-driven surface decision loss function to reconstruct the signed distance field from the point cloud without line, and then extracts the surface from the signed distance field; GeoUDF and SuperUDF start from local differential properties and optimal projection respectively, use data-driven prediction methods to predict the unsigned distance field in the entire space, and then extract the surface from the unsigned distance field. The method using signed distance field as an intermediate form can ensure that the obtained result has good topological properties, but the accuracy is slightly lower; the method using unsigned distance field as an intermediate form has a faster reconstruction speed and more accurate shape details, but the result often has defects such as local fragmentation and holes.

[0040] The present application is further described in detail below with reference to the accompanying drawings: See also Figure 1 The present application embodiment discloses a graphics reconstruction method based on nearest neighbor prior and low-frequency component fitting, comprising the following steps: S1, obtain the point cloud data of the graphics to be reconstructed, and build the nearest neighbor prior network and the signed distance network.

[0041] In practical applications, this application neighbor prior network It is a neural network with an encoding-decoding structure, which is used to K-nearest neighbors in the input point cloud Predict the nearest neighbors of a query point on the surface to be reconstructed .

[0042] All K-nearest neighbors The points in the query point are transformed to In the local coordinate system with the origin as the origin, we get the transformed K-nearest neighbor As The input of the nearest neighbor prior network makes the output of the nearest neighbor prior network independent of the global coordinates of the query point; Represents the query point coordinates.

[0043] S2, initialize the signed distance network and repeatedly sample query points from around the input point cloud.

[0044] It should be noted that before initializing the signed distance network, the present application trains the nearest neighbor prior network. The data for training the nearest neighbor prior network are query points, K-nearest neighbor sets, and nearest neighbor points collected from the polygonal mesh and its corresponding point cloud. The loss function used for training is the Euclidean distance.

[0045] S3, uses the projection prediction results given by the prior network to optimize the weights of the signed distance network to obtain the SDF (Signed Distance Field) representing the graphics to be reconstructed.

[0046] In practical applications, this application uses the projection prediction results given by the prior network to optimize the weights of the signed distance network, which is to use the nearest neighbor prior network The given result replaces the accurate value of SDF as supervision. The specific method is as follows: 1) Randomly sample several query points near the input point cloud ; 2) Input the query point into the signed distance network and calculate the approximate signed distance value at the point and the gradient vector , substitute Find the sampling point of the query point on the zero isosurface of the current signed distance function; where, Represents the projection result calculated based on the value and gradient of the signed distance network; 3) Find the K-nearest neighbors of the query point in the input point cloud , and transform the K-nearest neighbors to the local coordinate system with the query point as the origin to obtain the transformed K-nearest neighbors ; 4) The transformed K-nearest neighbors Input the nearest neighbor prior network to get the nearest neighbor point predicted by the prior network ; 5) Calculate the average Euclidean distance between all query points and their corresponding nearest neighbor points As projection loss , plus the Eikonal regularization term that constrains the gradient of the signed distance function Get the total loss function value , according to the total loss function value Compute the back-propagation optimized signed distance network; where, represents the weight of the projection loss, Represents the weight of the Eikonal regularization term.

[0047] S4, according to the SDF of the graphics to be reconstructed, the Marching Cubes algorithm is used to extract the surface of the graphics to be reconstructed, and the graphics reconstruction is completed.

[0048] It should be noted that this application adds a position encoding layer before MLP to enhance the signed distance function Fit the tendency of the low-frequency components of the object SDF; input the query point coordinates When , the position encoding layer first maps the coordinates to a low-frequency basis function space as follows, and takes the encoding result and the original coordinates as input:

[0049] in, represents the input vector obtained after position encoding, represents the weight of the lowest frequency component, Represents the encoding vector of the lowest frequency component, " indicates vector concatenation operation, represents the weight of the highest frequency component, represents the encoding vector of the highest frequency component, Indicates The encoding vector of frequency components, Indicates the number of the frequency component; The position encoding is performed in the form of trigonometric functions in the order of low to high frequency, and gradually added during the optimization process , and the weight of each frequency component control:

[0050] in, Indicates the optimization progress. Indicates the initial value of the optimization progress, Indicates the number of rounds of the current optimization iteration. represents the upper limit of the number of optimization iterations, Indicates The weight control parameters of the frequency components, Indicates truncation to the specified lower limit (0) and upper limit (1). Indicates the number of the highest frequency component, Indicates the number of the lowest frequency component, Indicates The weight of the frequency components; like is 0, then at the end of the signed distance function optimization, the weight of the highest frequency component just increases to 1; if If it is greater than 0, the weights of all frequency components will increase to 1 and the optimization will continue for several rounds before ending. As the number of optimization rounds increases, the signed distance function gradually converges to the object SDF, and its isosurface will also converge to the surface of the object accordingly.

[0051] like Figure 2As shown, another embodiment of the present application provides a graphics reconstruction system based on nearest neighbor prior and low frequency component fitting, including: A network construction module is used to obtain point cloud data of the graphics to be reconstructed and to construct a nearest neighbor prior network and a signed distance network; The network initialization module is used to initialize the signed distance network and repeatedly sample query points from around the input point cloud; A graphics optimization module is used to optimize the weights of the signed distance network using the projection prediction results given by the prior network to obtain the SDF representing the graphics to be reconstructed; The graphics reconstruction module is used to extract the surface of the graphics to be reconstructed using the Marching Cubes algorithm according to the SDF of the graphics to be reconstructed, and complete the graphics reconstruction.

[0052] like Figure 3 As shown, the embodiment of the present application discloses a surface reconstruction method based on local nearest neighbor prior and low-frequency component fitting, explains its reconstruction process and operation mechanism, and then introduces its implementation method.

[0053] This application uses a signed distance function represented by a multi-layer perceptron (MLP) to fit the signed distance field (SDF) of the object to be reconstructed under the supervision of the nearest neighbor prior, thereby completing the reconstruction. The main differences between this application and existing similar algorithms are: 1) The neural network that has learned the nearest neighbor prior knowledge provides the supervision information required to optimize the signed distance function, thereby improving the accuracy of the fitting; 2) By adding an additional position encoding layer to the MLP, its tendency to fit the low-frequency components of the SDF is enhanced, thereby enhancing the stability of the reconstruction and making the reconstructed surface smoother.

[0054] This application mainly consists of two parts: the accurate value of the SDF of the surrogate object and the local nearest neighbor prior network used to calculate the loss. The signed distance network that approximates the SDF of the object to be reconstructed Before reconstruction, a larger dataset is needed to train the nearest neighbor prior network so that it can accurately predict the nearest neighbor of a point in space on the surface to be reconstructed. When reconstructing, the signed distance network is first initialized to a fixed value, and then query points are repeatedly sampled from around the input point cloud, and then the prior network is used to The projected prediction results given by the Optimized Signed Distance Network The weights are used to obtain the SDF representing the object to be reconstructed, and finally the Marching Cubes algorithm is executed to extract the object surface.

[0055] Nearest Neighbor Prior Network It is a neural network with an encoding-decoding structure, which is used to K-nearest neighbors in the input point cloud Predict the nearest neighbors of a query point on the potential surface (surface to be reconstructed) To ensure that the output of the nearest neighbor prior network is independent of the global coordinates of the query point, all All points in the query point In the local coordinate system with the origin as the origin, we get the transformed K-nearest neighbor As The data for training the nearest neighbor prior are the query points, K-nearest neighbor sets, and nearest neighbor points collected from the polygonal mesh and its corresponding point cloud, and the loss function used for training is the Euclidean distance.

[0056] The complete process of surface reconstruction is as follows Figure 4 As shown in the figure (excluding the process of training the nearest neighbor prior network), it is divided into an optimization stage and a surface extraction stage. The optimization stage completely determines the shape of the reconstruction result and is the key part of the entire process. In the optimization stage, the signed distance network is repeatedly optimized to fit the shape of the object represented by the point cloud. This process usually requires providing the accurate value of the SDF at the sampling point as supervision, and comparing it with the evaluation result of the current signed distance function, and using the difference between the two as the loss function. This application uses the nearest neighbor prior network The given result replaces the accurate SDF value as supervision, thus avoiding the dependence on the SDF value and not requiring additional information such as the normal of the point cloud. The specific optimization process in each round is: 1) Randomly sample several query points near the input point cloud ; 2) Input the query point into the signed distance network and calculate the approximate signed distance value at the point and the gradient vector , substitute Find the sampling point of the query point on the zero isosurface of the current signed distance function; 3) Find the K-nearest neighbors of the query point in the input point cloud , and transform the K-nearest neighbors to the local coordinate system with the query point as the origin to obtain ; 4) The transformed K-nearest neighbors Input the nearest neighbor prior network to get the nearest neighbor point predicted by the prior network ; 5) Calculate the average Euclidean distance between all query points and their corresponding nearest neighbor points As projection loss , plus the Eikonal regularization term that constrains the gradient of the signed distance function Get the total loss function value ,according to Compute a backpropagation optimized signed distance network.

[0057] See also Figure 4 , this application adds an additional position encoding layer before the conventional MLP to enhance the signed distance function The tendency of fitting the low-frequency components of the object SDF. Input the query point coordinates When , the position encoding layer first maps the coordinates to a low-frequency basis function space as follows, and takes the encoding result and the original coordinates as input:

[0058] Parameters in the spectrum range and are small, this mapping is equivalent to explicitly providing a set of low-frequency input , when the query point coordinates change The change of is usually smaller, making it easier for the signed distance network to fit the low-frequency components of the object SDF. According to Park et al. Nerfies : Deformable neural radiance fields The research results in the literature show that when using trigonometric function position encoding, the frequency should be gradually increased from low to high, and the optimization process should be gradually added. , this process consists of the weight of each frequency component control:

[0059] As the optimization process progresses, the parameters From the initial value Start to increase linearly, each frequency component Weight The number of rounds required for the weight of each frequency component to increase from 0 to 1 is the same, but only when After the weight of increases to 1, The weight of starts to grow. is 0, then at the end of the signed distance function optimization, the weight of the highest frequency component just increases to 1; if If it is greater than 0, the weights of all frequency components will be increased to 1, and then the optimization will be continued for several rounds before ending. Figure 5 ,As the number of optimization rounds increases, the signed distance function gradually converges to the object SDF, and its isosurface will also converge to the surface of the object accordingly.

[0060] like Figure 6 As shown, Figure 6This is a comparison chart of the reconstruction results of this application and existing similar algorithms, where: GT represents the reference result (accurate result), NP represents the result of the Neural Pull algorithm (for the algorithm, see Neural-Pull: Learning SignedDistance Functions from Point Clouds by Learning to Pull Space ontoSurfaces), OSP represents the reconstruction result using surface judgment prior (for the algorithm, see Reconstructing Surfacesfor Sparse Point Clouds With On-Surface Priors), and GeoUDF represents the result of the GeoUDF algorithm (for the algorithm, see GeoUDF: Surface Reconstruction from 3D Point Clouds via Geometry-guided Distance Representation).

[0061] A computer device is provided in one embodiment of the present application. The computer device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0062] The computer program may be divided into one or more modules / units, one or more modules / units are stored in a memory and executed by a processor to complete the present application.

[0063] The computer device may be a desktop computer, a notebook computer, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a processor and a memory.

[0064] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0065] The memory can be used to store computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0066] If the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0067] The present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes Figure 1 The methods provided in the various optional methods are therefore not described in detail here.

[0068] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0069] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0070] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0071] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 The computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device, or are transmitted through a computer-readable storage medium. Computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The instruction device is implemented in the process Figure 1 A process or multiple processes and / or structures Figure 1These computer program instructions can also be loaded onto a computer or other programmable device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0072] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0073] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0074] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A graphics reconstruction method based on nearest neighbor prior and low-frequency component fitting, characterized in that: The following steps are involved: Obtain the point cloud data of the graphics to be reconstructed, and construct the nearest neighbor prior network and the signed distance network; Initialize the signed distance network and repeatedly sample query points around the input point cloud; The projection prediction results given by the prior network are used to optimize the weights of the signed distance network to obtain a signed distance field representing the graphics to be reconstructed; According to the signed distance field of the image to be reconstructed, the Marching Cubes algorithm is used to extract the surface of the image to be reconstructed to complete the image reconstruction.

2. The image reconstruction method based on nearest neighbor prior and low frequency component fitting according to claim 1, characterized in that: Before initializing the signed distance network, the nearest neighbor prior network is trained. The data for training the nearest neighbor prior network are query points, K-nearest neighbor sets and nearest neighbor points collected from polygonal grids and their corresponding point clouds. The loss function used for training is Euclidean distance.

3. The image reconstruction method based on nearest neighbor prior and low frequency component fitting according to claim 1 or 2, characterized in that: The neighbor prior network It is a neural network with an encoding-decoding structure, which is used to K-nearest neighbors in the input point cloud Predict the nearest neighbors of a query point on the surface to be reconstructed .

4. The method for image reconstruction based on nearest neighbor prior and low frequency component fitting according to claim 1, characterized in that: All the K-nearest neighbors The points in the query point are transformed to In the local coordinate system with the origin as the origin, we get the transformed K-nearest neighbor As The input of the nearest neighbor prior network makes the output of the nearest neighbor prior network independent of the global coordinates of the query point; Represents the query point coordinates.

5. The method for image reconstruction based on nearest neighbor prior and low frequency component fitting according to claim 1, characterized in that: The use of the projection prediction results given by the prior network to optimize the weights of the signed distance network is to use the nearest neighbor prior network The given result replaces the accurate value of the signed distance field as supervision. The specific method is as follows: 1) Randomly sample several query points near the input point cloud ; 2) Input the query point into the signed distance network and calculate the approximate signed distance value at the point and the gradient vector , substitute Find the sampling point of the query point on the zero isosurface of the current signed distance function; where, Represents the projection result calculated based on the value and gradient of the signed distance network; 3) Find the K-nearest neighbors of the query point in the input point cloud , and transform the K-nearest neighbors to the local coordinate system with the query point as the origin to obtain the transformed K-nearest neighbors ; 4) The transformed K-nearest neighbors Input the nearest neighbor prior network to get the nearest neighbor point predicted by the prior network ; 5) Calculate the average Euclidean distance between all query points and their corresponding nearest neighbor points As projection loss , plus the Eikonal regularization term that constrains the gradient of the signed distance function Get the total loss function value , according to the total loss function value Compute the back-propagation optimized signed distance network; where, represents the weight of the projection loss, Represents the weight of the Eikonal regularization term.

6. The method for image reconstruction based on nearest neighbor prior and low frequency component fitting according to claim 1, characterized in that: The method of extracting the surface of the image to be reconstructed using the Marching Cubes algorithm according to the signed distance field of the image to be reconstructed comprises: Add a position encoding layer before MLP to enhance the signed distance function The tendency of fitting the low-frequency components of the object's signed distance field; input the query point coordinates When , the position encoding layer first maps the coordinates to a low-frequency basis function space as follows, and takes the encoding result and the original coordinates as input: in, represents the input vector obtained after position encoding, represents the weight of the lowest frequency component, Represents the encoding vector of the lowest frequency component, " " indicates vector concatenation operation, represents the weight of the highest frequency component, represents the encoding vector of the highest frequency component, Indicates The encoding vector of frequency components, Indicates the number of the frequency component; The position encoding is performed in the form of trigonometric functions in the order of low to high frequency, and gradually added during the optimization process , and the weight of each frequency component control: in, Indicates the optimization progress. Indicates the initial value of the optimization progress, Indicates the number of rounds of the current optimization iteration. represents the upper limit of the number of optimization iterations, Indicates The weight control parameters of the frequency components, Indicates truncation to the specified lower limit (0) and upper limit (1). Indicates the number of the highest frequency component, Indicates the number of the lowest frequency component, Indicates The weight of the frequency components; like is 0, then at the end of the signed distance function optimization, the weight of the highest frequency component just increases to 1; if If it is greater than 0, the weights of all frequency components will increase to 1 and then the optimization will continue for several rounds before ending. As the number of optimization rounds increases, the signed distance function gradually converges to the object's signed distance field, and its isosurface will also converge to the surface of the object accordingly.

7. A graphics reconstruction system based on nearest neighbor prior and low frequency component fitting, characterized in that: include: A network construction module is used to obtain point cloud data of the graphics to be reconstructed and to construct a nearest neighbor prior network and a signed distance network; The network initialization module is used to initialize the signed distance network and repeatedly sample query points from around the input point cloud; A graphics optimization module, used to optimize the weights of the signed distance network using the projection prediction results given by the prior network, to obtain a signed distance field representing the graphics to be reconstructed; The graphics reconstruction module is used to extract the surface of the graphics to be reconstructed using the Marching Cubes algorithm according to the signed distance field of the graphics to be reconstructed, and complete the graphics reconstruction.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product includes computer instructions. A processor of a computer device reads the computer instructions, and the processor of the computer device executes the computer instructions to implement the steps of the method according to any one of claims 1 to 6.

Images