Reconstruction method, system and equipment of three-dimensional grid model and medium

Abstract

The invention relates to a reconstruction method, system and device of a three-dimensional grid model and a medium. The method comprises the steps of obtaining point cloud data; sampling the point cloud data to obtain a plurality of feature points; dividing the point cloud data into a plurality of local point clouds according to the plurality of feature points, and constructing a corresponding neural surface element according to each local point cloud; inputting each neural surface element into a symbol distance field prediction model to obtain a corresponding local symbol distance field; fusing the local symbol distance fields to obtain a global symbol distance field; and extracting point cloud points of which the symbol distance is a preset symbol first distance threshold value from the global symbol distance field, and constructing a three-dimensional network model. And the reconstruction efficiency and precision of the three-dimensional grid model are improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction, and in particular to a method, system, device and medium for reconstructing a three-dimensional grid model. Background Art

[0002] In the field of 3D reconstruction, both traditional geometry-based methods and deep learning-based methods are widely used. Traditional geometry-based methods mainly involve real-time data acquisition using RGB-D cameras and fusing depth information into 3D models through volume fusion techniques. These methods reconstruct surfaces by averaging truncated signed distance functions over voxel grid representations of space. Another approach is based on surfels, which avoids the memory overhead of maintaining volume grids by maintaining surfels in a global model. In addition, for large-scale map modeling, LiDAR scanners directly provide sparse depth measurements, usually representing maps in the form of point clouds.

[0003] In recent years, deep learning methods have been introduced to 3D reconstruction for improving robustness and exploiting shape priors. Some methods use neural networks to regularize RGB-D fusion, combining traditional geometry and learning components for visual SLAM. In addition, some methods directly output intermediate 3D representations of point clouds or voxels through neural networks, or directly output mesh representations. Neural implicit representation methods such as DeepSDF and IM-NET are able to reconstruct surfaces and shapes.

[0004] However, traditional geometry-based methods have limitations when dealing with complex surfaces and structures, such as limited ability to capture details and textures; deep learning methods are highly dependent on data and require a large amount of training data. In addition, some methods may have problems such as high computational complexity and insufficient model generalization ability. Therefore, it is necessary to provide a reconstruction method, system, device and medium for a three-dimensional mesh model. Summary of the invention

[0005] The present invention provides a method for reconstructing a three-dimensional grid model, so as to solve the problem that the accuracy of the reconstructed three-dimensional grid model is not high in the prior art.

[0006] The present invention provides a method for reconstructing a three-dimensional mesh model, comprising: acquiring point cloud data; sampling the point cloud data to obtain a plurality of feature points; dividing the point cloud data into a plurality of local point clouds according to the plurality of feature points, and constructing a corresponding neural facet according to each local point cloud; inputting each neural facet into a signed distance field prediction model to obtain a corresponding local signed distance field; fusing the local signed distance fields to obtain a global signed distance field; extracting point cloud points whose signed distance is a preset first distance threshold of a sign from the global signed distance field to construct a three-dimensional network model.

[0007] In one embodiment of the present invention, the point cloud data is divided into multiple local point clouds according to multiple feature points, and a corresponding neural surface is constructed according to each local point cloud, including: extracting all point cloud points within a preset range of each feature point from the point cloud data, forming a local point cloud corresponding to the feature point together with the corresponding feature point, and obtaining the normal of each local point cloud; for each local point cloud: calculating the translation vector of the local point cloud based on the principal component analysis method; constructing the rotation matrix of the local point cloud according to the normal of the local point cloud; obtaining the bounding box size according to the coordinates of the outermost point cloud point and the innermost point cloud point of the local point cloud; constructing a neural surface including the translation vector, the rotation matrix, the bounding box size, the pre-generated initialized latent code and the local point cloud; performing an affine transformation on each neural surface, updating the translation vector and the rotation matrix therein, and obtaining an updated neural surface.

[0008] In one embodiment of the present invention, all point cloud points within a preset range of each feature point are extracted from the point cloud data, and together with the corresponding feature points, a local point cloud corresponding to the feature point is formed, and the normal of each local point cloud is obtained, including: for each feature point: multiple point cloud points within a preset range of the feature point are extracted from the point cloud data, and together with the initial feature point cloud points, a candidate local point cloud is formed; the normal of the candidate local point cloud is calculated based on the principal component analysis method; the distance between the feature point and each other feature point and the normal angle are calculated, if the distance is less than a preset first distance threshold, and the normal angle is greater than a preset angle threshold, then a feature point with a larger index is selected and marked as a redundant point; wherein the index is a code generated in the sampling order when the feature point is obtained by sampling the point cloud data; all redundant points are deleted from all feature points, and the candidate local point cloud corresponding to each remaining feature point is used as the final local point cloud.

[0009] In one embodiment of the present invention, the affine transformation is performed on each neural facet, and the translation vector and rotation matrix therein are updated to obtain an updated neural facet, including: for each neural facet: selecting a feature point whose distance from the current neural facet is less than a preset first distance threshold, and taking the corresponding neural facet as an adjacent neural facet; performing an affine transformation on the current neural facet based on the adjacent neural facets, and using the rotation matrix and the translation vector as parameters to calculate the energy of the current neural facet after the affine transformation; counting the energy of each neural facet to obtain the total energy; iteratively optimizing the total energy to obtain the minimized total energy, and updating the rotation vector and translation vector of all neural faces according to the minimized total energy to obtain an updated neural facet.

[0010] In one embodiment of the present invention, the fusing of the local signed distance fields to obtain a global signed distance field includes: transforming each local signed distance field into a global coordinate system using a rotation matrix and a translation vector within each neural surface element; and merging each transformed local signed distance field in the global coordinate system to obtain a global signed distance field.

[0011] In one embodiment of the present invention, the transformed local signed distance field is Among them, x' is a point cloud point in the transformed local signed distance field, r i is the preset scaling factor, is the inverse matrix of the rotation matrix, is the inverse vector of the translation vector, and x is the point cloud point before the transformation corresponding to x'.

[0012] In one embodiment of the present invention, extracting point cloud points whose signed distance is a preset first signed distance threshold from the global signed distance field to construct a three-dimensional network model includes: reading each signed distance in the global signed distance field, and based on the marching cubes algorithm, selecting point cloud points whose signed distance is a preset signed distance value, and combining them into a three-dimensional mesh model.

[0013] In one embodiment of the present invention, a reconstruction system for a three-dimensional mesh model is also provided, the system comprising: a data acquisition module for acquiring point cloud data; a sampling module for sampling the point cloud data to obtain a plurality of feature points; a neural surfel generation module for dividing the point cloud data into a plurality of local point clouds according to the plurality of feature points, and constructing a corresponding neural surfel according to each local point cloud; a local signed distance field generation module for inputting each neural surfel into a signed distance field prediction model to obtain a corresponding local signed distance field; a global signed distance field generation module for fusing the local signed distance fields to obtain a global signed distance field; and a reconstruction module for extracting isosurfaces from the global signed distance field to construct a three-dimensional network model.

[0014] In one embodiment of the present invention, an electronic device is also provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements any one of the above-described methods for reconstructing a three-dimensional mesh model.

[0015] In one embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes any of the above-mentioned methods for reconstructing a three-dimensional mesh model.

[0016] The present invention proposes a method, system, device and medium for reconstructing a three-dimensional mesh model. By dividing the point cloud data into multiple local point clouds, each local point cloud can be independently processed, and the corresponding neural facets are obtained according to each local point cloud, so that the entire three-dimensional reconstruction process is more efficient and faster, and the obtained neural facets provide strong data support for the subsequent signed distance field prediction. Each neural facet data is converted into a signed distance field prediction model to achieve accurate prediction of the local signed distance field. Then each local signed distance field is fused, and the isosurface is extracted from the fused global signed distance field to construct a three-dimensional mesh model. The high-quality conversion from the original point cloud data to the precise mesh model is completed, and the constructed three-dimensional mesh model has high resolution and high precision, which solves the problem of poor precision of the traditional three-dimensional reconstruction method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic flow chart of a method for reconstructing a three-dimensional mesh model provided by an embodiment of the present invention;

[0018] Figure 2a Shown is a representation diagram of a single neural surface in three-dimensional space in one embodiment of the present invention;

[0019] Figure 2b for Figure 2a Schematic diagram of the local signed distance field corresponding to the neural surface element;

[0020] Figure 3 Displayed as a global point cloud under traditional noise interference;

[0021] Figure 4 Shown is a three-dimensional mesh reconstructed using neural surfels in one embodiment of the present invention;

[0022] Figure 5 Shown is a structural block diagram of a three-dimensional mesh model reconstruction system provided by an embodiment of the present invention;

[0023] Figure 6 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and functions of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0025] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0026] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0027] The present invention provides a method for reconstructing a three-dimensional mesh model. By dividing point cloud data into multiple local point clouds, each local point cloud can be independently processed, and the corresponding neural facet is obtained according to each local point cloud, so that the entire three-dimensional reconstruction process is more efficient and faster, and the obtained neural facet provides a strong data support for the subsequent signed distance field prediction. Each neural facet data is converted into a signed distance field prediction model to achieve accurate prediction of the local signed distance field. Then each local signed distance field is fused, and an isosurface is extracted from the fused global signed distance field to construct a three-dimensional mesh model. The high-quality conversion from the original point cloud data to the precise mesh model is completed, and the constructed three-dimensional mesh model has high resolution, high precision and high robustness, and can accurately express the geometric structure of the point cloud data, solving the problems of poor precision, low efficiency and poor noise resistance of the traditional three-dimensional reconstruction method.

[0028] See also Figure 1 , the reconstruction method of the three-dimensional mesh model includes the following steps:

[0029] S1. Obtain point cloud data.

[0030] Point cloud data is a data structure that represents the geometric shape of an object or scene surface in three-dimensional space in the form of a collection of point cloud points. Each point cloud point in the point cloud data contains three-dimensional spatial coordinates (x, y, z), and sometimes also includes attribute information such as color information or intensity information. The point cloud data of the scene to be reconstructed can be obtained through point cloud acquisition equipment, including but not limited to laser scanners, structured light sensors, etc. The point cloud data can also be denoised to remove noisy point cloud points, thereby improving the quality of the point cloud data and the speed and quality of subsequent data processing.

[0031] S2. Sampling the point cloud data to obtain a plurality of feature points.

[0032] Based on the dense random sampling strategy, a preset number (such as N) of point cloud points are randomly collected from the acquired point cloud data, and these point cloud points are used as feature points. An index is generated for each feature point according to the sampling order. In addition, in order to ensure the quality of the sampled point cloud points, the distribution and quality of the sampled point cloud points can also be checked by visualizing the sampling results to obtain better sampling effects. For example, the N feature points obtained need to cover the entire point cloud data well to ensure that the subsequently divided local point cloud can fully and accurately represent the relevant information of the original point cloud data.

[0033] S3. Divide the point cloud data into multiple local point clouds according to multiple feature points, and construct a corresponding neural surface element based on each local point cloud.

[0034] According to the feature points, the point cloud data is divided into multiple local point clouds. Each local point cloud consists of a feature point and the point cloud points within the preset range around it. Then, according to each local point cloud, the corresponding neural surface element is constructed. Among them, the neural surface element is used to represent the features and local geometric information of the point cloud data, and supports subsequent complex 3D modeling and surface reconstruction by describing the local surface characteristics.

[0035] Specifically, in one embodiment of the present invention, the point cloud data is divided into a plurality of local point clouds according to a plurality of feature points, and a corresponding neural surface is constructed according to each local point cloud, including:

[0036] Extracting all point cloud points within a preset range of each feature point from the point cloud data, forming a local point cloud corresponding to the feature point together with the corresponding feature point, and obtaining a normal of each local point cloud;

[0037] For each local point cloud:

[0038] Calculate the translation vector of the local point cloud based on principal component analysis;

[0039] Construct the rotation matrix of the local point cloud according to the normal of the local point cloud;

[0040] According to the outermost point cloud point and the innermost point cloud point of the local point cloud, the bounding box size is obtained;

[0041] Construct a neural surfel consisting of a translation vector, rotation matrix, bounding box dimensions, pre-generated initialized latent code, and local point cloud;

[0042] Perform affine transformation on each neural surface element, update the translation vector and rotation matrix therein, and obtain the updated neural surface element.

[0043] Extract all point cloud points within the preset range of each feature point from the point cloud data to form a local point cloud, and calculate the normal of each local point cloud based on the principal component analysis method. Traverse each local point cloud obtained, and for the i-th local point cloud obtained, process it in the following way: (1) Calculate the centroid of the local point cloud based on the principal component analysis method, and use the centroid as the translation vector t of the current i-th local point cloud i (2) Construct the rotation matrix R of the current i-th local point cloud according to the normal direction i , to align the local coordinate system of the i-th local point cloud to the global coordinate system. (3) Traverse each point of the local point cloud and obtain the maximum and minimum values ​​of these point cloud points on each coordinate axis. The difference between the maximum and the corresponding minimum value constitutes the bounding box size r of the current i-th local point cloud. i , to represent the spatial range of the i-th local point cloud. (4) In addition, for the current i-th local point cloud, an initialization potential code z is pre-generated i For the N feature points obtained by sampling, N neural surfels are obtained through the above four steps (1) to (4), and the obtained neural surfels are used to represent the features and geometric information in the point cloud data.

[0044] Specifically, as shown in formula (1), for the i-th neural surface element s i : It includes the translation vector t i , rotation matrix R i , bounding box size r i , pre-generate and initialize the latent code z i and the local point cloud P i :

[0045] s i = {R i ,t i ,r i ,z i ,P i} (1)

[0046] Among them, R i ∈SO(3) is the i-th neural surface element s i The rotation matrix relative to the global coordinate system, SO(3) represents the special orthogonal group, which is a set describing rotations in three-dimensional space; is the i-th neural surface element s i The translation vector relative to the global coordinate system, Represents three-dimensional Euclidean space, used to describe the position in space; r i is the i-th neural surface element s i The size of the bounding box of the cube in three-dimensional space, z i is the i-th neural surface element s iThe pre-generated initialized latent code of is the i-th neural surface element s i The local point cloud within the bounding box, p m is the local point cloud P i Then, the current neural surface element is affine transformed according to the rotation matrix and translation vector of the adjacent neural surface element, and the translation vector and rotation matrix of the current neural surface element are updated to obtain the updated neural surface element.

[0047] Specifically, in one embodiment of the present invention, extracting all point cloud points within a preset range of each feature point from the point cloud data, forming a local point cloud corresponding to the feature point together with the corresponding feature point, and obtaining the normal of each local point cloud includes:

[0048] For each feature point:

[0049] Extracting multiple point cloud points within a preset range of the feature points from the point cloud data, and forming a candidate local point cloud together with the feature points;

[0050] Calculate the normals of candidate local point clouds based on principal component analysis;

[0051] Calculate the distance and normal angle between the feature point and each other feature point. If the distance is less than the preset first distance threshold and the normal angle is greater than the preset angle threshold, select the feature point with a larger index and mark it as a redundant point; where the index is the feature point obtained by sampling the point cloud data.

[0052] The codes are generated in the sampling order;

[0053] All redundant points are deleted from all feature points, and the candidate local point cloud corresponding to each remaining feature point is used as the final local point cloud.

[0054] The search radius of each feature point is pre-set, and the search radius determines the size and coverage of the corresponding local point cloud. It is understandable that those skilled in the art can adaptively set the size of the search radius based on the spatial density of the original point cloud data and the actual reconstruction needs, which is not limited here. After determining the search radius, for each feature point: in the point cloud data, all point cloud points within the search radius are determined, and these point cloud points and the corresponding feature points are collectively used as a candidate local point cloud. It is understandable that the number of feature points is the same as the number of candidate local point clouds. Then, the principal component analysis method is used to calculate the normal of each candidate local point cloud, and the distance and normal angle between each feature point and other feature points are calculated. If the distance between the current feature point and another feature point is less than the first distance threshold, and the normal angle between the two is greater than the angle threshold, it means that the distance between the two is close and the direction is similar. At this time, a feature point with a larger index is marked as a redundant point. After all feature points are processed according to the above process, all feature points marked as redundant points are deleted, thereby pruning the neural surface elements that are too close to avoid redundant information. The candidate local point cloud corresponding to each remaining feature point is the final local point cloud. Therefore, the number of the final local point cloud is less than or equal to the number of feature points generated when sampling the point cloud data, such as the final local point cloud is L, and L is less than N. Through the above method, neural surface elements can be extracted from the original point cloud data to achieve a deeper understanding and representation of the point cloud data, providing more accurate and comprehensive information for subsequent overall shape reconstruction and analysis.

[0055] In one embodiment of the present invention, performing an affine transformation on each neural surface element, updating the translation vector and the rotation matrix therein, and obtaining an updated neural surface element includes:

[0056] For each neural facet:

[0057] Select a feature point whose distance from the current neural surface is less than a preset second distance threshold, and use the corresponding neural surface as an adjacent neural surface;

[0058] Perform affine transformation on the current neural surface element according to the adjacent neural surface elements, and use the rotation matrix and translation vector as parameters to calculate the energy of the current neural surface element after affine transformation;

[0059] Count the energy of each neural facet to get the total energy;

[0060] The total energy is iteratively optimized to obtain the minimized total energy, and the rotation vectors and translation vectors of all neural surfels are updated according to the minimized total energy to obtain the updated neural surfels.

[0061] Through K-nearest neighbor (KNN) search, for each neural surface: select feature points whose distance to the current neural surface is less than the preset second distance threshold (such as 4), and use the neural surfaces corresponding to these feature points as adjacent neural surfaces. Then, the current neural surface and each adjacent neural surface form a deformation graph. In this deformation graph, each node corresponds to a neural surface, and the relationship between adjacent neural surfaces is represented by edges. By constructing a deformation graph, the geometric consistency of local and global data can be maintained during subsequent data processing. For any point cloud point v of the current neural surface, an affine transformation is performed according to formula (2):

[0062]

[0063] Among them, A i (v) is the new position after the affine transformation applied to the point cloud point v, is the rotation matrix of node i (i.e., the i-th neural surface element), is the translation vector of node i, Tt i is the translation transformation vector of node i. It should be noted that for the i-th neural surface element, With the rotation matrix R i The values ​​are the same, but Used to "transform" the point cloud point v in the global coordinate system. Therefore, As a rotation matrix, it is used to rotate the point in the global coordinate system so that the rotated point can represent the rotation of the neural surface element where the point is located. At the beginning of the deformation graph optimization, it is initialized to a zero vector. As the deformation graph is continuously optimized, the corresponding adjustment t i , and until the optimal value is reached.

[0064] When a neural surface is deformed, it not only needs to consider its own shape and position, but also the shapes and positions of the surrounding adjacent neural surfaces. In order to ensure that the entire 3D structure to be reconstructed is continuous and smooth during deformation without abrupt breaks or uncoordinated deformation, it is necessary to calculate the global deformation position of the neural surface. In the global case, according to the neighboring neural surfaces' response to the current neural surface s i Affine transformation is performed to obtain the deformation position of any point v in the current neural surface element as shown in formula (3):

[0065] φ(v)=∑ j∈∏(i) w j (v)A j (v) (3)

[0066] Where φ(v) is the neural surface element s i(i.e., node i) is the global position of any point v after deformation, and ∏(i) is the position of the neural surface element s i The set of adjacent neural pixels, w j (v) is the neural surface element s j The weights of adjacent neural elements s j The influence on point v, the weight changes with the i-th neural surface element s i To the jth neural surface s j The distance decays linearly, and the sum of all weights is 1, A j (v) is the neural surface element s j The affine transformation result of point v. For example, if neural cells s2 and s3 are adjacent neural cells of neural cell s1, for any point v in neural cell s1, if the distance from s2 to s1 is d 12 , the distance from s3 to s1 is d 13 , and d 12 Greater than d 13 , then the corresponding weight w2(v) is less than w3(v), and the sum of the two weights is 1. If v also exists in s2 and s3, then the global deformation position of point v φ(v)=w2(v)A2(v)+w3(v)A3(v). It should be noted that if point v is not in s2, the affine transformation of s2 has no effect on point v, that is, w2(v) is zero, thereby ensuring the accuracy of deformation calculation.

[0067] When optimizing the deformation graph, the rotation matrix and translation vector of the neural surface element are first used as the initial affine transformation parameters. The value of the rotation change matrix is ​​the same as the rotation matrix, and the value of the translation transformation vector is zero. The total energy is calculated. Then the optimization is iterated continuously to minimize the total energy by adjusting the position and posture of each node (i.e., neural surface element) to update the translation transformation vector and rotation change matrix of each node. Among them, the total energy includes rotation energy, position energy, and regularization energy. The rotation energy is used to minimize the difference between the affine transformation of each node and the true rotation matrix. Specifically, for all neural surfaces, the rotation energy can be obtained by calculating the Frobenius norm between each rotation change matrix. The rotation energy is shown in formula (4):

[0068]

[0069] Among them, E rot is the rotation energy, L is the number of neural panels, is the rotation matrix of the jth neural element, I is the unit matrix, ‖·‖ F is the Frobenius norm, which is used to calculate the difference between matrices.

[0070] The position energy incorporates the position constraint of the neural surfel into the objective function and is calculated by summing the squared errors between the predicted transformed positions of all points in each neural surfel and the specified target position. The position energy is shown in formula (5):

[0071]

[0072] Among them, E rot is the position energy, L is the number of neural surfels, φ(V i ) is the neural surface element s i The global position of the set V of all points after deformation, q i Neural facets i The target position of the set V of all points v in contains the position information provided by the sensor data, and ‖·‖2 is the L2 norm, which is used to calculate the Euclidean distance between two position vectors. By taking all points in each neural surfel into consideration and summing the deformed position of each point with the corresponding target position, the position energy finally obtained can fully reflect the difference between the neural surfel and the target position. It can be understood that the deformed position of each point is as shown in the above formula (4).

[0073] Regularization energy is used to minimize the transformation difference between adjacent nodes to ensure smoothness and avoid discontinuous and inconsistent deformations. The regularization energy is shown in formula (6):

[0074]

[0075] Among them, E reg is the regularization energy, L is the number of neural surfels, ∏(i) is the i The set of adjacent neural facets (i.e., the set of nodes connected to node i in the deformation graph), v k Neural facets i and j The common point in A i (v k ) is applied to point v k The affine transformation result of the i-th neural surface element, A i is the i-th neural surface element s i Affine transformation of point v k is the target of the affine transformation, A j (v k ) is applied to point v k The affine transformation result of the jth neural surface element, A j is the jth neural surface element s j The regularization energy is obtained by applying the affine transformation of each neural surface element s i and each of its adjacent neural facets jThe difference of the affine transformation results of is summed to ensure that the deformation of adjacent neural elements is smooth and consistent. By performing inner summation, we can calculate the same point v k In different neural elements i 、s j Then, the outer layer sums up by traversing all neural surfels, thus ensuring good global consistency and smoothness after affine transformation.

[0076] The total energy is obtained by weighted summing the rotation energy, position energy and regularization energy obtained above, as shown in formula (7):

[0077] E=w rot E rot +w pos E pos +w reg E reg (7)

[0078] Among them, w rot 、w pos 、w reg are weight factors of rotation energy, position energy and regularization energy respectively. The total energy is iteratively optimized by an iterative optimization algorithm (such as LM algorithm, Gauss-Newton method, etc.), and the affine transformation parameters (translation vector, rotation change matrix and translation transformation vector) of each neural surfel are adjusted to minimize the total energy E. The specific optimization steps include: initializing the node position of the deformation graph and starting the optimization process. Through iterative optimization, the translation vector, rotation change matrix and translation transformation vector of each node (i.e. each neural surfel) are adjusted so that each point in the deformation graph is consistent locally and globally to avoid the situation where local optimization leads to overall unreasonableness. When the total energy E converges to the minimum value, the optimized translation vector, rotation change matrix and translation transformation vector of each neural surfel are obtained. Then, the updated translation vector, rotation change matrix and translation transformation vector are used to transform the point cloud in the global coordinate system to the optimized position according to formula (3) to update the neural surfel and ensure the consistency and accuracy of the entire three-dimensional model.

[0079] Through the above steps, the optimized deformation map finally obtained represents the non-rigid registration and global consistency alignment relationship between the two objects, realizing shape matching and consistency optimization between the objects. Deformation map optimization can achieve global consistency optimization, providing an effective method for shape matching and alignment. This technology has broad application prospects in computer vision, image processing and 3D reconstruction.

[0080] S4. Input each neural face into the signed distance field prediction model to obtain the corresponding local signed distance field.

[0081] Signed distance field (SDF) means calculating the distance from each point cloud point to the nearest surface, and using a sign to distinguish whether the point cloud point is inside, outside, or on the surface. Specifically, a positive value indicates that the point cloud point is outside the surface, a negative value indicates that the point cloud point is inside the surface, and a zero value indicates that the point cloud point is on the surface. The geometric structure of a three-dimensional object or scene can be effectively represented by a signed distance field. For each neural surface element obtained, define any point cloud point v in its local space such that v∈[-1,1] 3 . The potential code z of the neural face unit i And the point cloud point v is input to the trained signed distance field prediction model f θ In the example, the signed distance value of the point cloud point v is f θ (z i ,v) to describe the geometric properties of the local shape. By optimizing the latent code z of the neural surface unit i , in order to accurately capture and represent the geometric structure details of the target object. After all the point cloud points on the neural surface element are passed through the signed distance field prediction model to obtain the corresponding signed distance values, the set of these signed distance values ​​is used as the local signed distance field of the neural surface element. Among them, the signed distance field prediction model is a multi-layer perceptron. See Figure 2a and Figure 2b , Figure 2a It shows the representation of a single neural surface in three-dimensional space. The blue surface represents the position and shape of the neural surface in three-dimensional space. Figure 2b yes Figure 2a The local signed distance field corresponding to the neural surfel consists of a series of local signed distance values ​​of different colors, where red indicates that the signed distance value is positive, that is, these points are located outside the neural surfel, and blue indicates that the signed distance value is negative, that is, these points are located inside the neural surfel.

[0082] S5. Fusing the local signed distance fields to obtain a global signed distance field.

[0083] The global signed distance field is a three-dimensional scalar field, in which each voxel corresponds to a signed distance value, which represents the distance from the point cloud point to the nearest surface. A positive value indicates that the point cloud point is outside the surface, a negative value indicates that the point cloud point is inside the surface, and a zero value indicates that the point cloud point is on the surface. Specifically, in one embodiment of the present invention, the fusion of the local signed distance fields to obtain the global signed distance field includes:

[0084] Each local signed distance field is transformed into the global coordinate system using the rotation matrix and translation vector within each neural surfel;

[0085] In the global coordinate system, each transformed local signed distance field is merged to obtain a global signed distance field.

[0086] For each local signed distance field: traverse all the point cloud points v and the corresponding signed distance values ​​contained in it, and transform the point cloud point x in the local coordinate system to the global coordinate system according to formula (8) to obtain the transformed local signed distance field:

[0087]

[0088] Among them, v' is a point cloud point in the transformed local signed distance field, r i is the preset scaling factor, is the inverse matrix of the rotation matrix (i.e. the updated rotation change matrix), is the inverse vector of the translation vector, and v is the point cloud point before the transformation corresponding to v'. After all local signed distance fields are converted to the global coordinate system, these values ​​need to be fused to obtain the global signed distance field. Specifically, first initialize a three-dimensional grid of a signed distance field, and assign a weight value to each grid point cloud point. For each point cloud point in each local signed distance field, multiply the weight value corresponding to its signed distance value v', accumulate it to the corresponding position of the global signed distance field grid, and update the weight value of the grid point cloud point. Then calculate the weighted average of each point cloud point in the global signed distance field grid to obtain the final global signed distance field.

[0089] S6. Extracting point cloud points whose signed distance is a preset signed distance threshold from the global signed distance field, and constructing a three-dimensional network model.

[0090] In one embodiment of the present invention, extracting point cloud points whose signed distance is a preset signed distance threshold from the global signed distance field to construct a three-dimensional network model includes: reading each signed distance in the global signed distance field, and selecting point cloud points whose signed distance is a preset signed distance value based on the marching cubes algorithm, and combining them into a three-dimensional mesh model. Specifically, reading each signed distance value in the global signed distance field, and selecting point cloud points whose signed distance value is a preset signed distance threshold (such as 0) based on the marching cubes algorithm. Traversing each voxel of the global signed distance field, determining its corresponding signed distance value, and extracting point cloud points whose signed distance value is equal to the preset threshold as surface point cloud points to form a three-dimensional mesh model. By fusing the local global signed distance field into a global three-dimensional surface representation, high-quality three-dimensional reconstruction of the target scene can be achieved.

[0091] See also Figure 3 and Figure 4 , Figure 3 is the global point cloud under traditional noise interference, Figure 4is a three-dimensional mesh reconstructed using neural surfels in one embodiment of the present invention. It can be seen that Figure 3 There is a lot of noise in the original point cloud data, and the boundaries of objects are not clear. After reconstruction using the neural facet method of the present invention, the noise of the obtained three-dimensional mesh model is significantly reduced, and the reconstructed structure is clearer and more accurate. In addition, this method of the present invention can also more accurately capture the details in the point cloud data. By more accurately characterizing the local point cloud, the reconstructed three-dimensional mesh model can retain more detailed features, effectively improving the quality of the reconstruction of the three-dimensional mesh model. It should be noted that the present invention mainly focuses on the offline reconstruction task of the scene. For real-time reconstruction tasks, it is only necessary to change the global optimization to local optimization.

[0092] See also Figure 5 , the reconstruction system 100 of the three-dimensional mesh model includes: a data acquisition module 110, a sampling module 120, a neural facet generation module 130, a local signed distance field generation module 140, a global signed distance field generation module 150 and a reconstruction module 160. Among them, the above-mentioned data acquisition module 110 is used to obtain point cloud data. The sampling module 120 is used to sample the point cloud data to obtain multiple feature points. The neural facet generation module 130 is used to divide the point cloud data into multiple local point clouds according to multiple feature points, and construct a corresponding neural facet according to each local point cloud. The local signed distance field generation module 140 is used to input each neural facet into a signed distance field prediction model to obtain a corresponding local signed distance field. The global signed distance field generation module 150 is used to fuse the local signed distance fields to obtain a global signed distance field. The reconstruction module 160 is used to extract point cloud points whose signed distance is a preset first distance threshold of the sign from the global signed distance field to construct a three-dimensional network model.

[0093] The specific definition of the reconstruction system of the three-dimensional mesh model can be found in the definition of the reconstruction method of the three-dimensional mesh model in the above text, which will not be repeated here. Each module in the above-mentioned reconstruction system of the three-dimensional mesh model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware format, or can be stored in the memory of the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0094] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0095] See also Figure 6The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a three-dimensional mesh model reconstruction program.

[0096] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code for the reconstruction of the three-dimensional mesh model, but also can be used to temporarily store data that has been output or is to be output.

[0097] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect various components of the entire electronic device 1, and executes or executes programs or modules (such as a reconstruction program for a three-dimensional mesh model, etc.) stored in the memory 12, and calls data stored in the memory 12 to execute various functions of the electronic device 1 and process data.

[0098] The processor 13 executes the operating system and various installed application programs of the electronic device 1. The processor 13 executes the application programs to implement the steps in the above-mentioned three-dimensional mesh model reconstruction method.

[0099] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a data acquisition module 110, a sampling module 120, a neural surface element generation module 130, a local signed distance field generation module 140, a global signed distance field generation module 150, and a reconstruction module 160.

[0100] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the three-dimensional mesh model reconstruction method described in each embodiment of the present application.

[0101] In summary, the present invention discloses a method, system, device and medium for reconstructing a three-dimensional mesh model. The present invention combines neural network technology with the concept of traditional surface elements (i.e., neural facets), which not only optimizes loop detection and global consistency processing, but also significantly improves the accuracy and detail level of three-dimensional reconstruction while maintaining efficient calculation, thereby achieving high-precision and real-time three-dimensional scene reconstruction. This method has broad application prospects in the fields of autonomous driving, augmented reality, and cultural heritage protection. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0102] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for reconstructing a three-dimensional mesh model, characterized in that: The method comprises: Get point cloud data; Sampling the point cloud data to obtain a plurality of feature points; According to the plurality of feature points, the point cloud data is divided into a plurality of local point clouds, and a corresponding neural surface element is constructed according to each local point cloud; Input each neural surface into the signed distance field prediction model to obtain the corresponding local signed distance field; The local signed distance fields are fused to obtain the global signed distance field; Point cloud points whose signed distance is a preset first distance threshold of the signed distance are extracted from the global signed distance field to construct a three-dimensional network model.

2. The method for reconstructing a three-dimensional mesh model according to claim 1, characterized in that: The step of dividing the point cloud data into a plurality of local point clouds according to the plurality of feature points, and constructing a corresponding neural surface element according to each local point cloud includes: Extracting all point cloud points within a preset range of each feature point from the point cloud data, forming a local point cloud corresponding to the feature point together with the corresponding feature point, and obtaining a normal of each local point cloud; For each local point cloud: Calculate the translation vector of the local point cloud based on principal component analysis; Construct the rotation matrix of the local point cloud according to the normal of the local point cloud; According to the coordinates of the outermost point cloud point and the innermost point cloud point of the local point cloud, the size of the bounding box is obtained; Construct a neural surfel consisting of a translation vector, rotation matrix, bounding box dimensions, pre-generated initialized latent code, and local point cloud; Perform affine transformation on each neural surface element, update the translation vector and rotation matrix therein, and obtain the updated neural surface element.

3. The method for reconstructing a three-dimensional mesh model according to claim 2, characterized in that: The step of extracting all point cloud points within a preset range of each feature point from the point cloud data, forming a local point cloud corresponding to the feature point together with the corresponding feature point, and obtaining a normal of each local point cloud includes: For each feature point: Extracting multiple point cloud points within a preset range of the feature points from the point cloud data, and forming a candidate local point cloud together with the feature points; Calculate the normals of candidate local point clouds based on principal component analysis; Calculate the distance and normal angle between the feature point and each other feature point. If the distance is less than the preset first distance threshold and the normal angle is greater than the preset angle threshold, select the feature point with a larger index and mark it as a redundant point; where the index is the code generated in the sampling order when the feature point is obtained by sampling the point cloud data; All redundant points are deleted from all feature points, and the candidate local point cloud corresponding to each remaining feature point is used as the final local point cloud.

4. The method for reconstructing a three-dimensional mesh model according to claim 2, characterized in that: The step of performing an affine transformation on each neural surface element, updating the translation vector and the rotation matrix therein, and obtaining an updated neural surface element includes: For each neural facet: Select a feature point whose distance from the current neural surface is less than a preset first distance threshold, and use the corresponding neural surface as an adjacent neural surface; Perform affine transformation on the current neural surface element according to the adjacent neural surface elements, and use the rotation matrix and translation vector as parameters to calculate the energy of the current neural surface element after affine transformation; Count the energy of each neural facet to get the total energy; The total energy is iteratively optimized to obtain the minimized total energy, and the rotation vectors and translation vectors of all neural surfels are updated according to the minimized total energy to obtain the updated neural surfels.

5. The method for reconstructing a three-dimensional mesh model according to claim 4, characterized in that: The fusion of the local signed distance fields to obtain the global signed distance field includes: Each local signed distance field is transformed into the global coordinate system using the rotation matrix and translation vector within each neural surfel; In the global coordinate system, each transformed local signed distance field is merged to obtain a global signed distance field.

6. The method for reconstructing a three-dimensional mesh model according to claim 5, characterized in that: The transformed local signed distance field is Among them, v' is a point cloud point in the transformed local signed distance field, r i is the preset scaling factor, is the inverse matrix of the rotation matrix, is the inverse vector of the translation vector, and v is the point cloud point before the transformation corresponding to v'.

7. The method for reconstructing a three-dimensional mesh model according to claim 1, characterized in that: The method extracts point cloud points whose signed distance is a preset signed distance threshold from the global signed distance field to construct a three-dimensional network model, including: reading each signed distance in the global signed distance field, and selecting point cloud points whose signed distance is a preset signed distance value based on a marching cubes algorithm to combine them into a three-dimensional mesh model.

8. A three-dimensional mesh model reconstruction system, characterized in that: The system comprises: Data acquisition module, used to acquire point cloud data; A sampling module, used for sampling the point cloud data to obtain a plurality of feature points; A neural surfel generation module, used for dividing the point cloud data into a plurality of local point clouds according to a plurality of feature points, and constructing a corresponding neural surfel according to each local point cloud; A local signed distance field generation module is used to input each neural face into the signed distance field prediction model to obtain the corresponding local signed distance field; A global signed distance field generation module is used to fuse the local signed distance fields to obtain a global signed distance field; The reconstruction module is used to extract point cloud points whose signed distance is a preset first distance threshold of the signed distance from the global signed distance field, and construct a three-dimensional network model.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the three-dimensional mesh model reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method for reconstructing a three-dimensional mesh model as described in any one of claims 1 to 7.

Images