Three-dimensional reconstruction method, device, equipment and storage medium

Abstract

The present disclosure relates to the field of machine vision, specifically to a 3D reconstruction method, apparatus, device, and storage medium. The invention comprises: obtaining an initial input point cloud of an object to be reconstructed; dividing the point cloud space into regional grids at a set resolution; traversing the regional grids, creating a dataset using a continuous stereo 27-grid differential local space, and computing a global model in parallel or distributed fashion. Based on xy-plane sampling points, the predicted z-axis coordinate components and colors are predicted to generate a predicted and patched point cloud, which is then merged with the input point cloud to improve resolution; converting the merged point cloud into a mesh model, and performing gap filling, smoothing filtering, and simplification to optimize resolution and computational efficiency.

Description

Technical Field

[0001] The present disclosure relates to the fields of point cloud, differential grid computing, deep learning and image processing technology, and in particular to a three-dimensional reconstruction method, apparatus, device and storage medium. Background Art

[0002] Three-dimensional (3D) reconstruction is the process of creating mathematical models of three-dimensional objects that are represented and processed by computers. 3D reconstruction is the basis for processing, manipulating, and analyzing the properties of 3D objects in a computer environment, and is also the key to technologies such as virtual reality and proto-universe.

[0003] In related technologies, 3D reconstruction primarily involves building a 3D digital model of the scene based on an input image sequence. For example, the input image sequence is processed using the Structure From Motion (SFM) algorithm to generate a sparse point cloud model. This sparse point cloud model is then processed using the Multi-view Stereo (MVS) algorithm to generate a dense point cloud model. The dense point cloud model is then patched and reconstructed to create a high-resolution 3D mesh model that approximates the physical object.

[0004] However, when the number of images in the input image sequence is large, generating a dense point cloud model takes a long time and is slow. If the number of images in the input image sequence is reduced, many holes will appear in a small number of areas such as the edges of the 3D model. Summary of the Invention

[0005] In order to solve all or at least one of the above technical problems, the embodiments of the present disclosure provide a three-dimensional reconstruction method, apparatus, device and storage medium to achieve rapid reconstruction of point cloud models and mesh models, and reduce or even eliminate holes in three-dimensional models.

[0006] In a first aspect, an embodiment of the present disclosure provides a three-dimensional reconstruction method, including: obtaining an initial input point cloud of an object to be reconstructed; dividing the input point cloud space into grids according to a set resolution; traversing the grid, creating a training data set and a prediction data set using a continuous stereo 27-grid differential local space, and computing a global model in parallel or in a distributed manner; training a prediction network model based on the created training data set, and using the prediction network model to predict the z-axis coordinate components and color information of the sampling points based on the predicted input feature vectors of the xy plane sampling points; generating a predicted patch point cloud based on the xy plane sampling points, the predicted z-axis coordinate components and the color information, and merging it with the input point cloud to obtain a predicted point cloud to improve the resolution; converting the predicted point cloud into a grid model, and performing smoothing filtering and simplification processing to optimize the resolution and computational efficiency. In a second aspect, an embodiment of the present disclosure provides a three-dimensional reconstruction device, including: an initial point cloud acquisition module, used to obtain the initial input point cloud of the physical object to be reconstructed; a grid division module, used to divide the input point cloud space into grids according to a set resolution; a data set creation module, traversing the grid, using continuous stereo 27-grid differential local space to create a training data set and a prediction data set, and parallel or distributed computing a global model; a sampling point information generation module, training a prediction network model based on the created training data set, and using the prediction network model to predict the z-axis coordinate component and color information of the sampling point and an optional normal vector based on the predicted input feature vector of the xy plane sampling point; a predicted point cloud determination module, generating a predicted patching point cloud based on the xy plane sampling point, the predicted z-axis coordinate component and the color information, and merging it with the existing input point cloud to obtain a predicted point cloud to improve the resolution; a point cloud generation grid model module, using 27-grid differential and parallel or distributed computing to convert the three-dimensional point cloud model into a three-dimensional grid model, and then performing gap patching, smoothing filtering and simplification processing to optimize the resolution and computational efficiency.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device including:

[0008] One or more processing machines;

[0009] a storage device for storing one or more programs;

[0010] When one or more programs are executed by one or more processing machines, the one or more processing machines implement the three-dimensional reconstruction method provided in the first aspect above.

[0011] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional reconstruction method provided in the first aspect above.

[0012] In a fifth aspect, an embodiment of the present disclosure provides a program product, which includes a computer program. When the computer program is executed by a processor, it implements the three-dimensional reconstruction method provided in the first aspect above.

[0013] The disclosed embodiments provide a three-dimensional reconstruction method, apparatus, and computing facility, the method comprising: obtaining an initial input point cloud of an object to be reconstructed; dividing the input point cloud space into a grid according to a set resolution; traversing the grid, creating a training dataset and a prediction dataset using a continuous stereo 27-grid differential local space, and computing a global model in parallel or in a distributed manner; training a prediction network model based on the created training dataset, and using the prediction network model to predict the z-axis coordinate components and color information of the sampling points based on the predicted input feature vectors of the xy-plane sampling points; generating a predicted patch point cloud based on the xy-plane sampling points, the predicted z-axis coordinate components, and the color information, and merging it with the input point cloud to obtain a predicted point cloud to improve resolution; converting the predicted point cloud into a grid model, and performing gap patching, smoothing filtering, and simplification processing to optimize resolution and computational efficiency. Using parallel or distributed cluster computing shortens computing time and enables rapid reconstruction of point clouds and grid models. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0015] Figure 1 Schematic diagram of a process of a point cloud 3D reconstruction method in an embodiment of the present disclosure;

[0016] Figure 2 Schematic diagram of a basic computing unit 27 grid cube in an embodiment of the present disclosure;

[0017] Figure 3 A schematic diagram of a spatial region where gaps exist in a point cloud according to an embodiment of the present disclosure;

[0018] Figure 4 This is a schematic diagram of a use case of data point distribution of a training data set in an embodiment of the present disclosure;

[0019] Figure 5 Schematic diagram of the structure of the depth prediction network in the embodiment of the present disclosure;

[0020] Figure 6 A schematic diagram of a flow chart of a grid-optimized 3D reconstruction method in an embodiment of the present disclosure;

[0021] Figure 7is the intersection of the grid and the grid point fitting plane in the embodiment of the present disclosure;

[0022] Figure 8 A schematic diagram of the triangular surface composition of a mesh cross-section plane in an embodiment of the present disclosure;

[0023] Figure 9 A schematic diagram of a plane gap grid obtained in an embodiment of the present disclosure;

[0024] Figure 10 is a schematic diagram of an edge connecting adjacent grid cutting planes in an embodiment of the present disclosure;

[0025] Figure 11 Schematic diagram of the structure of a three-dimensional reconstruction device in an embodiment of the present disclosure;

[0026] Figure 12 A schematic diagram of the structure of a computing facility in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0030] It should be noted that the concepts mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] 3D reconstruction involves creating mathematical models of 3D objects that can be represented and processed by computers. It is the foundation for processing, manipulating, and analyzing 3D objects in a computer environment and is crucial for technologies such as virtual reality and the proto-universe. Currently, commonly used representations of 3D models include depth maps, voxel models, point cloud models, and mesh models.

[0034] The depth map consists of a 2D image of the real scene and a 2D depth map. The disadvantage of the depth map is that it cannot represent the features of the interior of objects or occluded objects, and can only represent the vertical depth of the viewpoint plane.

[0035] Voxel models represent three-dimensional objects using three-dimensional pixels. The amount of computation required increases with the cube of the resolution on a single axis. High resolutions lead to a significant computational effort.

[0036] A point cloud model is a three-dimensional model that uses three-dimensional point features to represent an object. It is a series of spatial feature point groups with coordinates. An image itself cannot directly generate a point cloud, but the input image can be processed using the SFM algorithm to obtain sparse point cloud data. This sparse point cloud data can then be processed using the MVS algorithm to obtain dense point cloud data. However, the above point cloud data has the following disadvantages: poor robustness, discontinuity between points, and an uneven surface of the reconstructed object. Alternatively, depth information of the point cloud can be directly obtained using lidar and depth cameras, but this increases equipment costs. Although it saves a lot of computation, it also suffers from problems such as narrow measurement range, high noise, small field of view, uneven data, sunlight interference, and inability to determine the projected material.

[0037] A mesh model approximates a three-dimensional object using a series of vertices, edges, and polygons (usually triangles). This representation saves points and computational memory; however, like voxel models, mesh models suffer from resolution issues. One mesh is equivalent to one voxel, and the computational effort still increases as the cube of the resolution on a single axis. This computational effort is substantial when modeling large, high-resolution scenes. Generating high-resolution orthophoto 3D reconstructions of ground-based landscapes directly using voxels or meshes can significantly burden computer memory and the CPU or GPU.

[0038] In order to solve all or at least one of the above technical problems, the embodiments of the present disclosure provide a three-dimensional object reconstruction method, device, equipment and storage medium, the method including: obtaining an initial point cloud model of the physical object to be reconstructed; dividing the initial point cloud space into regional grids according to a set resolution; traversing the regional grids, creating a data set using a continuous stereo 27-grid differential local space, and parallel or distributed computing a global model; predicting the z-axis coordinate component and color information of the sampling point based on the input feature vector of the sampling point; generating a patched color point cloud model using the z-axis coordinate component and color information of the sampling point; adding the patched color point cloud model to the initial point cloud model to obtain a predicted patched point cloud model, thereby improving the resolution of the point cloud model. At the same time, due to the small number of input image sequences, combined with distributed computing, the model reconstruction time is shortened, and fast point cloud model reconstruction is achieved.

[0039] The three-dimensional reconstruction method provided by the present disclosure is described in detail below with reference to the embodiments and drawings.

[0040] Figure 1 This is a flow chart of a three-dimensional reconstruction method in an embodiment of the present disclosure. This embodiment is applicable to the situation of three-dimensional reconstruction of a physical scene. The method can be executed by a three-dimensional reconstruction device, which can be implemented in software and / or hardware. The three-dimensional reconstruction device can be configured with a terminal device or a server.

[0041] like Figure 1 As shown, the three-dimensional reconstruction method provided by the embodiment of the present disclosure mainly includes steps S101-S106.

[0042] S101: Obtain an initial input point cloud of an object to be reconstructed.

[0043] Physical objects refer to real objects in real scenes, while objects to be reconstructed refer to objects that need to be represented using a 3D model in a computer. The initial input point cloud is a point cloud model generated from an image. A point cloud model is a 3D representation of an object using spatial point feature data. Each point feature in a point cloud model contains 3D coordinates and may include information such as color, reflectance, and intensity.

[0044] The point cloud model is stored in a computer using a polygonal mesh file format (ply). The ply mesh file stores point cloud data, which includes the three-dimensional coordinates, color information, and normal vector of each point. The color information can be represented in red, green, and blue (RGB) format.

[0045] The initial input point cloud is a dense point cloud model obtained by processing an image sequence or monocular video using the SFM and MVS algorithms. The input image sequence or the image sequence converted from the monocular video includes two-dimensional images of the object to be reconstructed, taken from different angles. Feature points are extracted from this image sequence, matched, and the camera pose and coordinate transformation matrix are estimated to calculate a sparse color point cloud. The image is then dedistorted, three-dimensional surfaces are matched, stereo fusion is performed, and point cloud noise is removed. Finally, a dense color point cloud is calculated and stored in a ply file.

[0046] S102: Divide the input point cloud space into grids according to the set resolution.

[0047] The resolution setting is limited by the computing power of the device. A higher resolution means a higher computational load and cost. This should be determined based on the specific task. In the embodiments of the present invention, the resolution is an adjustable parameter and is not limited.

[0048] The point cloud space is defined as a 3D hexahedron space limited by the length, width, and depth of the point cloud. The length of the point cloud refers to the maximum value H on the y-axis of the point cloud model. max and the y-axis minimum value H min The length of the area between the points is the maximum value W of the x-axis in the point cloud model. max and the minimum value W of the x-axis min The area between is long, and the depth of the point cloud refers to the maximum value D of the z axis in the point cloud model. max and the z-axis minimum value D min The area between is long.

[0049] A grid is a collection of spatial intervals divided at a set resolution. In certain cases, it can refer to a single spatial interval, where each cell can be of the same or different size. For ease of programming, the space is divided equally. However, in areas with high point density, more spatial divisions can be used to improve accuracy. In these cases, the set resolution is not uniform, and the space is not equally divided.

[0050] In one possible implementation, the input point cloud space is divided into grids according to a set resolution, including: dividing the input point cloud coordinate axis interval into equal or unequal parts according to the set resolution to obtain a grid; assigning a unique grid number to each spatial interval grid; using the grid number as a key and the point data as a value to form a key-value tuple; storing the key-value tuple into a grid number file with one point per line, each line of data in the grid number file includes parameters of a point, and the point parameters include the following: point coordinates, point color, normal vector and the number of the grid where the point is located.

[0051] The coordinate axes of the input point cloud are divided equally or unequally according to the set resolution to generate a grid. Each grid is assigned a region number. All points of the point cloud and their grid numbers are stored in a CSV file, one line per point. Each line in the CSV grid number file contains the vertex parameters of the point. The vertex parameters include the following: point coordinates, point color, normal vector, and the grid number. The normal vector is an optional feature parameter.

[0052] Specifically, set the x-axis resolution of the point cloud to Xp, the y-axis resolution to Yp, and the z-axis resolution to Zp. Calculate the maximum y-axis length H, the maximum x-axis width W, and the maximum z-axis depth D of the point cloud. Use the maximum value of length H, width W, and depth D to normalize each point so that the coordinate components of each point are in the range [0.0, 1.0]. Among them, the maximum y-axis length H is the maximum y-axis value H. max and the y-axis minimum value H min The maximum length of the x-axis W is the maximum value of the x-axis W max and the x-axis minimum value W min The maximum length D of the z-axis is the maximum value D of the z-axis max and the z-axis minimum value D min The normalized point cloud space x-axis, y-axis and z-axis coordinate intervals are divided into integer equal parts according to the set resolution, and the side length of the point cloud space x-axis interval grid is calculated as x step =W / Xp, the side length of the y-axis interval grid is y step =H / Yp, the side length of the z-axis interval is z step =D / Zp. The length of each grid on the x-axis is x step , the side length on the y-axis is y step , the side length on the z axis is z step .

[0053] For subsequent calculations, each spatial interval is numbered. The specific method is as follows.

[0054] The x-axis region numbers Xi are integers in the interval [0, Xp), the y-axis region numbers Yi are integers in the interval [0, Yp), and the z-axis region numbers Zi are integers in the interval [0, Zp). Each spatial region grid is assigned a unique point coordinate number Pi = (xi,yi,zi), where xi∈[0,Xp),yi∈[0,Yp),zi∈[0,ZXp). The grid number can be set as a string consisting of fixed-bit coordinate interval numbers, for example: xiyizi. For example, when the resolution is 4 bits, if xi = 0000, yi = 0128, and zi = 1080, then the spatial region number = "000001281080". In actual calculations, coordinate number triplets (xi,yi,zi) or strings can be used directly, depending on the characteristics of the programming language used.

[0055] Based on the scale of the dense point cloud obtained in S101, use single-machine, parallel, or distributed computing to traverse the input point cloud points P = (x, y, z), calculate the point feature x, y, and z axis coordinates, x, y, and z axis colors, x, y, and z axis normal vectors, and x, y, and z axis interval numbers, point by point, and store them in a point cloud mesh number file. At the same time, the mesh side length, resolution, and normalization parameters are stored in the mesh number configuration file for subsequent calculations and model scale calculation.

[0056] S103, traverse the regional grid, use the continuous three-dimensional 27-grid differential local model space to create a training data set and a prediction data set, and calculate the global model in parallel or in a distributed manner.

[0057] First, read the point cloud grid number file and grid number configuration file. Use single-machine, parallel or distributed computing based on the input point cloud obtained in S101. Use the groupby function to group all the points of the point cloud into grid point groups with the grid number as the key, and store the grid point groups in a container with the grid number tuple as the key for query at any time. Storing the grid point group in the container includes but is not limited to storing the three-axis coordinate values, color information, normal vectors and grid numbers of the grid points in the container. Using hash table and other technologies to directly access the key-value container to obtain neighboring points can save memory, be intuitive to analyze, and do not require tedious preprocessing compared to using an octree. It can also be quickly calculated in parallel, concurrent, and distributed ways.

[0058] In one possible implementation, the grid is traversed, the local space is differentiated using a continuous stereo 27-square grid, a training data set and a prediction data set are created, and a global model is calculated in parallel or distributed, including: taking the central grid of the stereo 27-square grid as the current grid; obtaining the neighboring grids of the current grid and the points within the grid; taking each point within the central grid of the 27-square grid as a label point and the other points within the 27-square grid as a neighbor point group to create a training data set; constructing a training input feature vector and a label vector based on each label point and the ordered neighbor point group; judging whether the number of points in the current calculation grid is less than a preset value; if the number of points in the current calculation grid is less than the preset value, the current calculation grid is used as a prediction patch grid, and the xy plane coordinates of the prediction points need to be sampled; N plane input sampling points are selected from the prediction patch grid, where N is at least the difference between the preset value and the number of points in the prediction patch grid, and each non-sampling point within the 27-square grid other than the sampling point of the current calculation grid generates a neighbor point group of the sampling point to create a prediction data set; and constructing a prediction input feature vector based on each sampling point and the ordered neighbor point group.

[0059] Point cloud grouping grid is similar to voxel, but instead of calculating all points in the voxel grid into a 3D pixel point, it takes a grid as the center and considers all points in up to 26 adjacent grids of the center grid, and each grid is a local precise search calculation. Figure 4 As shown, the five-pointed star and sun symbols in the center grid represent the points of the current grid, where the sun symbol represents the current point, and the five-pointed star point and the circles in the adjacent grid represent neighboring points. The distance between the current point and the neighboring points is calculated in the 27-square grid to sort and extract the top nearest neighbor points.

[0060] Create a training dataset and traverse the grid, setting the currently traversed grid as the center grid. The feature vector for each point in the current grid contains the point's x- and y-coordinate components, the 3-axis coordinates of its top nearest neighbor, the 3-axis color, and an optional 3-axis normal. Neighbor point features are arranged in the shape of (Top, 6) or (Top, 9), with the former excluding the normal and the latter including it. The current point's x- and y-coordinates can be alternately arranged to form a 6- or 9-dimensional vector, which precedes the neighbor point features to form a training input feature vector. For example, the Top=T features excluding the normal are: [[x,y,x,y,x,y],[x1,y1,z1,r1,g1,b1],...,[xT,yT,zT,rT,gT,bT]], where r, g, and b represent color values ​​between 0 and 1, and 1 to T represent the point number. Each point's label contains the point's z-axis coordinate component, color information, and an optional normal, such as [z,r,g,b] excluding the normal.

[0061] Extract the point group data for the current grid's 26 neighboring grids. If the neighboring grid does not exist, the spatial region and grid at that location have no points. The number of extracted neighboring grids is limited to 26. Then, for each current point, generate a list containing only the current neighboring points. Calculate the distance between the current point and each neighbor in the list, sorting them from closest to farthest. Extract the top nearest points, and fill any points missing with zero vectors. Finally, use the current point's depth and color information to compose a training label vector.

[0062] Creating a prediction dataset requires determining the input feature vectors of sampling points within a spatial region grid. Only grids that meet a predefined condition are used to calculate the prediction data features. This condition can mean that the number of points within the grid is less than a predefined value. This value is set based on actual conditions. For example, the predefined value is 8. If sufficient memory and CPU are available, a larger value such as 16 can also be used.

[0063] Sampling points are random sampling points on the xy plane that are used to enhance or patch the mesh to be predicted. A prediction input feature vector consists of the xy-2D coordinates of the sampling point, the 3D coordinates of the neighboring points, color information, and an optional normal vector. Neighboring feature points are calculated in the same way as when training input feature vectors are calculated, except that the xy coordinate components of the current point before the neighboring feature points are replaced by the xy coordinate components of the sampling point. The xy-2D coordinates of the sampling point are the x- and y-axis components of the 3D point coordinates. The sampling points are used for prediction and only the 2D coordinate components of the xy plane are sampled. The z-axis component of the depth is unknown and is predicted and estimated using a deep learning model.

[0064] In one possible implementation, traversing the regional grid to determine the predicted input feature vector for the sample points in the prediction grid includes: selecting the central grid of the 27-square grid as the current grid; obtaining the neighboring grids of the current grid; determining whether the number of points Nr in the current grid is less than a preset value N; if the number of points Nr in the current grid is less than the preset value N, then using the current grid as the prediction grid; sampling Ns points from the prediction grid in a uniform distribution, where Ns is the difference between the preset value and the actual number of points in the prediction grid, i.e., Ns = N - Nr; constructing the predicted input feature vector for the sample point using the xy-2D coordinates of the sample point and its top nearest neighbor points; and if the number of points Nr in the current grid is greater than or equal to the preset threshold N, indicating that the number of points in the current grid is sufficient and no sample points need to be generated. The next grid is then directly read for calculation, and each grid is traversed to enhance information in sparse areas of the point cloud.

[0065] Take a grid as a calculation unit. Figure 2 As shown in the figure, the cube group on the left (a) is a schematic diagram of the actual calculation unit cube 27 square grid (b) on the right. In the actual calculation, there are no gaps between the grids, but some grids do not contain vertices and are empty, so there is no grid number for this part of the space in the input point cloud. Figure 3 As shown in the figure, gaps numbered 130, 230, and 330 are located at locations where there are no points in the point cloud. These gaps are ignored when creating the prediction dataset. Only the grids containing points are collected, and the gaps are subsequently filled. The prediction dataset is the dataset consisting of the input feature vectors of all sampled points.

[0066] Take the current grid as the center grid, obtain all possible adjacent grids, and calculate the neighbor point list of each point in the center grid. The number of adjacent grids is up to 26. To calculate the top neighbor points of a point in the center grid, add the points in the center grid and adjacent grids to the neighbor point list except for this point. Specifically, according to the area number of the current grid, determine one by one whether the key value container constructed above contains the following: Figure 2The 26 adjacent grids of the current grid are shown; if an adjacent grid is not in the key-value container, it means that the grid in the adjacent spatial area does not include any points; if the adjacent grid is in the key-value container, the points in the adjacent grid are added to the neighbor point list.

[0067] Calculate the number of points Nr in the current grid. If the number of points Nr is less than the preset value N, generate at least N-Nr sampling points. Sample at least N-Nr x-axis and y-axis data (xs, ys) in the current grid in a uniform distribution as prediction sampling points. Among them, xs is within the x-axis value range of the current grid; ys is within the y-axis value range of the current grid. Calculate the distance between the sampling point and the xy coordinates of each neighbor point, sort them from near to far according to the distance, and extract 1 to Top points in sequence. Those less than Top are then padded with 0 vectors. Top is the number of selected neighbor points.

[0068] The prediction input feature vector includes the xy coordinates of the sampling point and the 3D coordinates of the neighboring points, color information, and an optional normal vector. The prediction dataset can have no label vector, or the label vector can be composed of the z-axis coordinate mean, color, and normal vector of the points in the current grid for subsequent prediction comparison.

[0069] The following describes the method for constructing the input feature vector.

[0070] The input feature vector can be determined by the x-axis and y-axis coordinates of the current point or sampling point, the 3-axis coordinates of the neighboring points, and the 3-axis color information of the neighboring points. Furthermore, the input feature vector can be determined by the x-axis and y-axis coordinates of the current point or sampling point, the 3-axis coordinates of the neighboring points, the 3-axis color of the neighboring points, and the 3-axis normal vector of the neighboring points. The training input feature vector generated by the current point is used as the training data input, and the prediction input feature vector generated by the sampling point is used as the prediction data input.

[0071] If the number of neighbor points is Top, the feature form of the neighbor points is (Top, 6) or (Top, 9). The former does not include the normal vector, while the latter does. (Top, 6) means there are Top neighbor points, and each neighbor point has 6 values: the neighbor point's 3-axis coordinates and the neighbor point's 3-axis color. (Top, 9) means there are Top neighbor points, and each neighbor point has 9 values: the neighbor point's 3-axis coordinates, the neighbor point's 3-axis color, and the neighbor point's 3-axis normal vector.

[0072] The x- and y-coordinates of the current point can be alternately arranged to form a 6- or 9-dimensional vector, which is placed before the neighboring point features to form a feature point vector. For example, the Top=M feature, excluding the normal vector, is in the form of: [[x,y,x,y,x,y], [x1,y1,z1,r1,g1,b1,], ..., [xM,yM,zM,rM,gM,bM]], where r, g, and b represent color components ranging from 0 to 1, and 1 to M represent the neighbor point numbers. The distances from point (x,y,z) to points (x1,y1,z1) and (xM,yM,zM) are arranged from smallest to largest. The label vector consists of the z-coordinate component of the current point and color information, in the following format: [z,r,g,b].

[0073] Expand the training input feature vector and convert it into a CSV string by separating it with ","; expand the label vector and convert it into a CSV string by separating it with ","; use ":" to connect the two into a row of feature label data and save them into a file as a feature label vector. The format of each row is as follows:

[0074] x,y,x,y,x,y,x1,y1,z1,r1,g1,b1,...,xM,yM,zM,rM,gM,bM:z,r,g,b

[0075] The prediction input feature vector is generated using the xy coordinates of the sampling point and the top=M neighboring points. Its format is as follows: [[xs,ys,xs,ys,xs,ys],[x1,y1,z1,r1,g1,b1],...,[xM,yM,zM,rM,gM,bM]]. The label vector can be constructed using the z-axis coordinate component and color value of a point in the current grid, or using the mean z-axis coordinate and color value of the points in the current grid. For example, the point closest to the sampling point is selected as the label of the sampling point: [z_n,r_n,g_n,b_n].

[0076] Expand the predicted input feature vector of the sampling point and convert it into a CSV string by separating it with ","; if there is a label vector, expand it and convert it into a CSV string by separating it with ","; use ":" to connect the two into a row of feature label data, and then save it to the file row by row as the predicted feature label vector. The format of each row is as follows:

[0077] xs,ys,xs,ys,xs,ys,x1,y1,z1,r1,g1,b1,...,xM,yM,zM,rM,gM,bM:z_n,r_n,g_n,b_n

[0078] S104 , training a prediction network model based on the created training data set, and using the prediction network model to predict and generate z-axis coordinate components and color information of the sampling points based on the prediction input feature vectors of the sampling points.

[0079] Feeding the predicted input feature vector of the sampling point into a pre-trained prediction network model, so that the prediction network model outputs the z-axis coordinate component and color information of the sampling point, or outputs the z-axis coordinate component, color information and normal vector of the sampling point;

[0080] After obtaining the z-axis coordinate component of the sampling point, determine whether the z-axis coordinate component exceeds the z-axis coordinate boundary of the grid where the sampling point is located. If the z-axis coordinate component exceeds the z-axis coordinate boundary of the grid, a random z-axis component is assigned or discarded. The randomly assigned z-axis component is within the z-axis coordinate range of the grid.

[0081] The prediction network model is introduced in the embodiments of the present disclosure.

[0082] The prediction network model is trained by a training data set, which includes an input feature vector column and a label vector column created from an initial input point cloud; the input feature vector is determined by the two-dimensional plane xy coordinates of the current calculation point of the central grid and the three-dimensional xyz coordinates, color and normal vector (optional) of its top neighbor points; the label vector is determined by the z-axis coordinate component, color information and normal vector (optional) of the current calculation point of the central grid.

[0083] like Figure 5 The figure shows the deep prediction network used in the prediction network model. The input is the feature vectors of the current point and neighboring points. The vertex dimension Dim_v = 6 (without normal vectors) or 9 (with normal vectors), the input dimension Dim_i = (Top+1)*Dim_v, and the output dimension Dim_o = 4. The prediction network model consists of three parts: a parallel splicing MLP group 401, a serial Conv2D group 402, and an output splicing MLP group 403. The first layer of the parallel splicing MLP group 401 consists of n MLPs in parallel. The spliced ​​hidden layer vector is connected to the output of 401 using one MLP. The MLP units in each layer can set and adjust the hidden layer neurons. The serial Conv2D group 402 consists of m convolutions with the activation function Conv2d+tanh and h linear layers with the activation function Linear+tanh. The hidden layer neurons, kernel window, stride, and padding can be set and adjusted in each convolution layer. The outputs of the parallel splicing MLP group 401 and the serial Conv2D group 402 are spliced ​​and then output by the MLP network 403. The hidden layers of the output splicing MLP group 403 can be set and adjusted.

[0084] When calculating the predicted output, the xy coordinates of the current point of the input feature vector are replaced by the xy coordinates of the sampling point. For details, please refer to the description in the above embodiment, and no specific limitation is given in the embodiment of the present invention.

[0085] Set the training dataset read buffer size (read_buff) and batch size (train_batch). Because point cloud data can be large, read feature labels from the training dataset file in rows (read_buff) at a time. Set the input scale (scale) to adjust the input data to the lower 1 / 3 of the range 0 to 1; set the output gain (gain) to adjust the output data to the lower 1 / 3 of the range 0 to 1. Set the number of training epochs, the learning rule (lr), and the MSE loss function. Feed the input data into the model and perform a forward pass to calculate the loss and error. Update the weights using the RMSprop algorithm. Repeat the training dataset loop for epochs before terminating.

[0086] S105 , generating a predicted patching point cloud based on the xy plane sampling points, the predicted z-axis coordinate component and the color information, and merging the predicted patching point cloud with the input point cloud to obtain a predicted point cloud to improve the resolution.

[0087] The input point cloud is patched using the sampling points and their predicted z-axis coordinate components and color information to obtain the predicted point cloud.

[0088] Specifically, the plane xy coordinates of the sampling point are used instead of the plane xy coordinates of the current point in the training input, and the z-axis coordinate component and RGB color information are predicted in combination with the neighbor point group. The sampled xy coordinate component, the predicted z-axis coordinate component and the RGB color value are used to form a patched color point cloud. The patched color point cloud is merged with the input point cloud to obtain the predicted point cloud, thereby enhancing the point cloud quality.

[0089] S106: Convert the predicted point cloud into a mesh model, and perform smoothing, filtering, and simplification to optimize resolution and computational efficiency.

[0090] The disclosed embodiments provide a three-dimensional point cloud reconstruction method, comprising: obtaining an initial input point cloud of an object to be reconstructed; dividing the input point cloud space into a grid according to a set resolution; traversing the grid, creating a training dataset and a prediction dataset using a continuous stereo 27-grid differential local space, and computing a global model in parallel or in a distributed manner; training a prediction network model based on the created training dataset, and using the prediction network model to predict the z-axis coordinate components and color information of the sampling points based on the predicted input feature vectors of the xy-plane sampling points; generating a predicted patch point cloud based on the xy-plane sampling points, the predicted z-axis coordinate components, and the color information, and merging the predicted patch point cloud with the input point cloud to obtain a predicted point cloud to improve the resolution; converting the predicted point cloud into a grid model, and performing smoothing, filtering, and simplification to optimize the resolution and computational efficiency. Using parallel or distributed cluster computing shortens the computation time and enables rapid reconstruction of point clouds and grid models.

[0091] Based on the point cloud prediction step S105 in the above embodiment, the present embodiment converts and optimizes the three-dimensional point cloud reconstruction method into a grid model, such as Figure 6 As shown, step S106 of the present invention is a three-dimensional optimization reconstruction method for converting the predicted point cloud into a mesh model, and the detailed steps include the following.

[0092] S201 , obtaining the predicted point cloud of the object to be reconstructed generated in step S105 .

[0093] Set the resolution of each axis of the grid. This resolution is used to divide the grid spacing of each axis to perform differential calculations. The resolution can be set to equal or unequal divisions. In key areas or areas with high density, it can be set to high resolution, and in non-key areas or areas with low density, it can be set to low resolution. When the point cloud is dense enough, a larger resolution can improve the 3D smoothness. For the convenience of calculation, this embodiment is set to equal division, and the predicted point cloud model space is divided into regional grids according to the set resolution. The steps are basically the same as those in the above S102. Please refer to the description in the above embodiment for details and will not be repeated here.

[0094] S202 : Generate a key-value container for querying and extracting grid data in the same way as generating a data set in step S103 .

[0095] S203: Generate a grid model of the predicted point cloud by using the predicted point cloud through distributed continuous stereo 27-grid differential calculation.

[0096] Among them, generating a grid model using the predicted point cloud is to traverse the spatial grid divided by the predicted point cloud, differentiate the local grid points using a continuous three-dimensional 27-square grid, and generate a global three-dimensional grid model by parallel or distributed cluster computing.

[0097] A possible implementation method includes: setting the grid resolution to divide the space where the predicted point cloud model is located into spatial grids; traversing the spatial grids, calculating the average normal vector and center point coordinates of each grid; constructing the grid section with the center point coordinates and the average normal vector; calculating the intersection of the grid section and the grid boundary edge; using the intersection as the vertex of the initial polygonal surface, and using the surface formed by the intersection as the grid polygonal surface, the vertices and grid polygonal surface constitute the three-dimensional grid model of the predicted point cloud.

[0098] Specifically, the predicted point cloud space is divided into grids according to the set grid resolution; the grid is traversed with the central grid of the 27-grid algorithm, and the average normal vector and center point coordinates of all points in the central grid are calculated; the point normal section plane of the central grid is constructed with the center point coordinates and the average normal vector; the intersection of the section plane and the boundary edge of the central grid is calculated; a point outside the central grid edge in the δ neighborhood of each intersection point is used as a vertex of the central grid polygonal face, and the plane enclosed by the vertex lines generated by the central grid is used as a polygonal face of the grid model, as described below.

[0099] If the predicted point cloud is large in scale and consumes a lot of computing memory, you can use persistent distributed computing to temporarily store intermediate data on disk, traverse the point cloud spatial grid generated by S201 and S202, and calculate the number of points num in the current center grid.

[0100] If the number of points num > 2, the current central spatial region grid contains curved or polygonal surfaces with more than three points, and the surface of the 3D object can be represented by at least one triangle. Calculate the average normal vector, center point coordinates, and average color of the current central grid. Use the average normal vector and the point-normal plane equation of the center point to approximate the surface of the 3D object within the central grid. Calculate the edge intersections of this approximate plane with the current central grid. If sufficient memory is available, add the edge intersections to the point group of the current central grid. If insufficient memory is available, replace the points within the current central grid with the edge intersections, reducing the number of points within the current central grid to fewer than six, saving memory and CPU time.

[0101] Furthermore, when calculating the edge intersection, a random additive value δ>0 is set, such as 1.0e-8, and the edge intersection is extended to the outside of the current center grid where the section is located, such as Figure 7 As shown, the edge intersection point A is extended to A', replacing A with A' and D with D'. This reduces the probability of meshes in adjacent spatial regions on the edge having the same intersection point, avoiding errors when importing files in some mesh software. Replacing AD with A'D' ensures that the mesh sections of adjacent spatial regions intersect at AD, automatically creating a stitching effect when displayed. Subsequent fitting steps can also merge points within the δ neighborhood of point A into a single point as needed to save memory.

[0102] If the number of points num = 2, it means that there is a straight line with 2 points in the current center grid. The points in the center grid have no surface representation, but the average normal vector and the center point can still be calculated. The δ neighborhood intersection of the current center grid edge is calculated based on this point normal plane, and the actual extended intersection point is calculated according to the method in the above steps.

[0103] If the number of points num = 1, it means that there is only one point in the current grid. The points in the grid have no surface representation. However, the normal vector of the point and the point normal plane of the point coordinates can still be used to calculate the δ neighborhood intersection of the current center grid edge. The actual extended intersection point used is calculated according to the method in the above steps.

[0104] Split the polygonal cross section into triangular face groups and store them in the grid file. Traverse the grid, calculate the number of points in the grid, and calculate the cumulative number of points in the current center grid noff according to the order of grid storage. Let the point number in the current center grid be pid. When it only contains the intersection points of the δ neighborhood, pid∈[0,6), then the vertex number of this point in the grid file is vid=noff+pid. Figure 8 As shown in the triangle diagram of the mesh cross-section plane, the triangles of the current center grid are stored in the mesh file in the order of OAB, OBC, OCD, and ODA. Using vertex numbers as row numbers, the mesh numbers are stored in the vertex mesh number file in the following format: "mesh x-axis number, mesh y-axis number, mesh z-axis number." Each time a row of mesh numbers is read, the mesh number x,y,z triplet for that row of vertices is generated.

[0105] S204, determining whether there are gaps in the mesh model of the predicted point cloud. If there are gaps in the mesh model of the predicted point cloud, the mesh model of the predicted point cloud is repaired, and deduplication, merging, smoothing, filtering, and simplification are performed. Otherwise, the process ends.

[0106] Use the mesh tool to open the mesh file generated in the above steps and observe whether there are gaps or holes in the mesh. If there are gaps or holes, perform mesh gap repair.

[0107] Grid gap repair, the specific steps are as follows:

[0108] In one possible embodiment, all points in the predicted point cloud are projected onto a gap identification plane; the gap identification plane is divided into multiple plane grids according to a set resolution, the point cloud includes at least one projection point in the plane grid, and the observation gap has no projection point in the plane grid; sampling points are obtained in the plane grid, and the known edge grid closest to the sampling point is used as a neighbor grid; a patch input data set is generated based on the neighbor grid; a patch prediction point cloud is generated based on the patch input data set; a patch prediction grid model is generated based on the patch prediction point cloud; and the patch prediction grid model is sequentially deduplicated, vertex merged, estimated, smoothed, filtered, and simplified.

[0109] Step 301: Generate a repair gap identification plane.

[0110] Read the grid number file generated in the above step S201, traverse the input image sequence in S101, select a current image, read the transformation matrix from world coordinates to camera coordinates from the coordinate system transformation matrix, use the predicted point cloud model generated in step S105 as the input point cloud, transform all points to the camera coordinate system, the far end of the camera is the positive z-axis, the right side is the positive x-axis, and the bottom side is the positive y-axis. Project all camera coordinate system 3D points (x, y, z) to the xy plane of the camera, or set z to 0 to generate xy plane coordinates (x, y), divide the xy plane area of ​​the projection point into a plane grid according to the resolution of x and y, calculate the average color of the plane grid, and the color value of the gap part seen by the camera end is 0. Save the plane projection image to the file for traversing the xy axis number of the gap grid.

[0111] Step 302: Generate a grid gap repair prediction dataset.

[0112] a. Traverse and read the xy-plane gap identification pixel grid generated in step 301, and use the gap color threshold to filter out the gap grid numbers, i.e., the xy pixel coordinates of the image. Convert the color identification plane image to a grayscale image, and set the gap grayscale threshold to Cth. Pixels with values ​​less than Cth are considered to be gaps or holes in the xy-plane. The actual pixel RGB color values ​​of these gaps can be read from the selected image.

[0113] b. Transform the input point cloud to the current camera coordinate system, calculate the grid number of the input point cloud according to the method in S102, and group and calculate all possible grids between the xy identification plane grid and the camera lens observation plane using the x and y axis grid number tuples (xp, yp) as keys and the z axis number and grid point list tuples (zp, V) as values. Find a grid with the minimum depth distance to the lens observation plane, and store it in a key-value container Cp using (xp, yp) as the key and (zp, V) as the value.

[0114] c. Use (xp0, yp0) of the plane gap grid as the key to extract the edge grid near the gap, such as Figure 9 As shown, 11, 12, 21, 22, and 31 are gap grids, and the edge grid near gap 11 is as follows Figure 9 The 1st row grid, 4th row grid, 1st column grid, 4th column grid and grid 32 are shown; and the optional edge grids of each grid contained in the same gap are the same. Traverse the edge grids of the plane, and use the current edge grid number (xp, yp) as the key to extract the depth number zp and grid point list V in the key-value container Cp. This is the edge grid near the gap on the object surface that can be seen from the lens observation plane. Set a gap edge search level, then use each gap grid as the current center point, and search along the normal line in 26 directions for a maximum of level times. Record the first grid found on each line (including the points in the point cloud), then jump out of the search in this line direction and continue searching in the next direction. After transforming the grid found this time to the world coordinate system, store it in the current edge grid list as the neighbor space area for the calculation of the 27-square grid of the cube.

[0115] d. Sample k (x, y) coordinates within the coordinate interval of the current gap grid. Each sampling point (xs, ys) and the top = T edge grid neighbor points closest to the center point of the current gap grid form a feature point vector. The format excluding the normal vector is as follows:

[0116] [[xs,ys,xs,ys,xs,ys],[x1,y1,z1,r1,g1,b1],...,[xT,yT,zT,rT,gT,bT]]

[0117] The format containing the normal vector is as follows:

[0118] [[xs,ys,xs,ys,xs,ys,xs,ys,0],[x1,y1,z1,r1,g1,b1,nx1,ny1,nz1],...,[xT,yT,zT,rT,gT,bT,nxT,nyT,nzT]]

[0119] e. Transform the point coordinates to the world coordinate system, set the label vector of the gap grid to the zero vector, and combine the feature point vector and label vector. If using color-corrected prediction of the gap edge grid, set the label vector to the z-axis average of the nearest gap edge grid and the color value of the nearest edge grid. Expand the feature point vector and convert it into a CSV string by separating it with a ","; expand the label vector and convert it into a CSV string by separating it with a ","; concatenate the two with a ":" to form a line of feature label data, and save it row by row into the gap grid dataset file.

[0120] Step 303: Patch the gap point cloud. Calculate the output prediction of the gap grid input dataset according to step S104, generate gap patch points, and store them in the patch prediction point cloud file along with the predicted point cloud. When calculating the patch points, limit the depth to within the predicted gap grid. If the depth value exceeds the predicted gap grid, a random depth value is assigned or discarded.

[0121] Step 304 : Output a mesh number file and a configuration file of the repaired predicted point cloud according to the color point cloud mesh number generation method of step S102 .

[0122] Step 305: Generate a mesh model of the repair prediction point cloud according to step S203 and save it into a repair prediction mesh file.

[0123] Step 306: Optimize and repair mesh gaps.

[0124] a. Read the patch prediction grid file and the patch prediction grid number configuration file. Read the patch prediction grid vertex coordinate number file generated in step 305.

[0125] b. Group the vertices of the patch prediction mesh by vertex coordinate number, generating a key-value container Gp with the mesh number as the key and (mesh vertex number, point list) as the value. Read the triangle set Ts of the patch prediction mesh. Generate an empty set of supplementary triangles T0.

[0126] c. Traverse the key value container Gp. Calculate the intersection points of the edge δ neighborhood of the center grid of the cube 27 grids. Assume that the grid side numbers are: 0 = top (negative y axis), 1 = bottom (positive y axis), 2 = left (negative x axis), 3 = right (positive x axis), 4 = front (negative z axis), 5 = back (positive z axis). Calculate the edge points of each side and store them in the grid side edge point key value container Pl according to the side number key. Traverse the faces of the container Pl. If the number of the previous points is greater than 2, merge the points on the face whose distance is less than the set value eps>0 (slightly greater than the random positive value δ in step 203). Determine again if the number of the previous points is greater than or equal to 2, then sequentially add two connecting points (when equal to 2, there is only one edge connecting) to the fan triangle face of the nearest two vertices of the adjacent face, for example Figure 10Connect the edges of the tangent planes of adjacent meshes. Points A and D on the top of the central mesh form a three-dimensional quadrilateral with points A' and D' on the bottom of the adjacent mesh. In this case, add two triangles, A'AD and A'DD', to set T0. If the top of the central mesh only has point A, add one triangle, A'AD', to set T0. Similarly, connect the triangles of the bottom, left, right, front, and back faces. If the top adjacent grid of the aforementioned central grid is the central grid, and the midpoint of points A' and B' in front of the grid is less than the midpoint coordinate of the x-axis edge of the central grid, connect A'B' and the left side of the front face A"B". At this time, two triangles, A'B'B" and A'B"A" are added to the set T0. If there is only point A' in front of the central grid, add a triangle, A'B"A" to the set T0. Similarly, connect the triangles of the other front and side faces. If a front face of the central grid has no adjacent faces, it is a hole. Set a connection level L (experienced value ≤ 3, too large a value will introduce noise) to calculate the intervening polygonal faces of the surrounding grid. Query the edge points on the faces that are at most L grids apart in the straight line direction of the front face of the central grid one by one for connection. If a positive side face of the central grid is a hole and has no adjacent faces, set a connection level L (experienced value ≤ 3, too large a value will introduce noise) to calculate the intervening polygonal faces of the surrounding grid. Query the edge points on the faces that are at most L grids apart in the direction of the side face one by one for connection. For example Figure 10 For the three meshes 015, 115, and 215, if 115 does not exist, when calculating the side surface of the bottom surface of 005, if L=1, the side surface of 215 is used for calculation. If all meshes within level L are empty, no triangles are added.

[0127] d. Filter the patched mesh generated by steps a through c. This removes duplicate vertices and triangles, merges vertices smaller than a set distance, and removes degenerate triangles.

[0128] e. Use the QEMD algorithm to simplify the mesh as needed, or use the Taubin algorithm or Laplacian algorithm to smooth the mesh, remove degenerate triangles to generate an optimized patch mesh, or use the Poisson or Ball-Pivoting algorithm to reconstruct the optimized patch mesh.

[0129] f. The vertex number file storing the optimized patched mesh has the same format as the vertex number file in step S203.

[0130] g. Set the hole integer tolerance Tor, and calculate the number of holes in the xy plane according to step 301. If it is less than Tor, stop the calculation, otherwise jump to step 301 to continue optimizing the gap repair grid.

[0131] The present invention uses SFM and MVS to extract a 3D point cloud model of a scene from an image sequence; uses a depth prediction network to enhance the point cloud expression and improve the resolution; uses a distributed differential cube 27-grid calculation to convert the point cloud into a grid model, and further fills grid gaps in the 27-grid, filters, smoothes, simplifies and optimizes the grid, achieving a grid model that is better than the MVS output.

[0132] This paper provides a 3D reconstruction method that uses the SceauxCastle dataset and the SFM and MVS software tools to obtain a preliminary 3D dense point cloud. The point cloud is then numbered according to resolution accuracy, gridded, and calculated using the 27-square cube algorithm to generate training and prediction datasets. A deep network model is trained, and then the enhanced point cloud is predicted. Finally, a 3D mesh model is generated by gap filling and optimization. The specific steps are as follows:

[0133] Step 1: Calculate the dense point cloud. Use colmap's feature extraction, matching, and bundle adjustment (BA) algorithms to calculate parameters such as the sparse point cloud and camera pose. Perform image dedistortion to reconstruct the dense point cloud and fuse the scene to generate a preliminary point cloud model.

[0134] Step 2: Set the x, y, and z axes resolution to 1080. Generate a grid file with 1080 grid cells per axis in the preliminary point cloud model. The maximum number of grid cells is 1080*1080*1080, but the actual point cloud scene cube is mostly empty, so the file only contains grid cells with points.

[0135] Step 3. Select the input features as [[current point x-axis coordinate, current point y-axis coordinate, current point x-axis coordinate, current point y-axis coordinate, current point x-axis coordinate, current point y-axis coordinate], [neighbor point x-axis coordinate, neighbor point y-axis coordinate, neighbor point z-axis coordinate, neighbor point r color component, neighbor point g color component, neighbor point b color component] * 8], and the output features as [current point z-axis coordinate, current point r color component, current point g color component, current point b color component]. Set the number of neighbor vertices for the input features to 8, and use the mesh number file to generate training data. During calculation, the current point is in the center mesh and has a maximum of 26 adjacent meshes around it. If there are no adjacent meshes outside the point cloud boundary or in the gaps, only the points in the center mesh are calculated. If the center mesh has only one point, it is considered a gap and the data of that mesh is discarded. Insufficient feature points are padded with zero vectors into the input feature vector.

[0136] Step 4: Generate prediction data using the grid number file. Set the grid prediction threshold to 8. That is, when the number of points in a grid is less than 8, use that grid as the center grid and generate s input data for the depth prediction model. s is the number of (x, y) samples in the center grid. Each (x, y) sample generates an input data for predicted depth z and color.

[0137] Step 5: Set the learning law to 0.0001, the number of training cycles to 10, the scaling to 0.3, and the gain to 10 / 3, and use the training data from step 3 to train the prediction network model. The prediction network model has 15 parallel concatenation MLPs, 2 serial concatenation Conv2d+tanh layers, and 2 linear concatenation Linear+tanh layers, and a 7-layer output MLP.

[0138] Step 6: Use the trained prediction network to predict the depth z and color of the generated xy sampling prediction points to generate a point cloud of the sampling points, and add the predicted sampling points to the dense point cloud generated in step 1 to generate a predicted point cloud.

[0139] Step 7: Calculate the grid number of the predicted point cloud.

[0140] Step 8: Due to the high computational complexity and memory usage, the predicted point cloud grid number is used to calculate the reconstructed grid model of the predicted point cloud according to the 27-square grid of the cube on the Spark distributed computing system and save it into the preliminary grid model file. The neighborhood of the edge intersection is set to δ = 1.0e-5.

[0141] Step 9: Generate a grayscale image of the point cloud grid projected on the xy plane in the depth direction of the camera coordinate system using the grid number. Set the grayscale color threshold of the gap grid to 10, that is, a grid with a grayscale value less than 10 is considered to be a gap grid.

[0142] Step 10. Generate a gap patch prediction dataset. Use the xy plane to project the grayscale image to determine the xy axis number and coordinates of the gap, traverse the gap plane grid, and estimate the z axis position of the center grid of the gap. Read the xy number and coordinates of the nearest non-empty grid near the gap, and extract the z axis number and coordinates closest to the camera lens observation plane as the neighboring grid data. Generate s input data for the depth prediction model at the current gap center grid; where s is the number of (x, y) samples in the center grid. Each (x, y) sample generates an input data for predicted depth z and color. After transforming the point coordinates to the world coordinate system, store them in the gap patch dataset file.

[0143] Step 11: Use the gap patching dataset to predict points in the gap grid, where the number of sampling points is s = 8. Generate a point cloud of gap patching points, and add the patching points to the predicted point cloud generated in step 6 to generate a gap patching predicted point cloud.

[0144] Step 12: Calculate the grid number of the gap repair prediction point cloud.

[0145] Step 13: Calculate the reconstructed mesh model of the gap-filling prediction point cloud using a 27-square cube grid on the Spark distributed computing system and save it into a gap-filling mesh model file. The neighborhood of the edge intersection is set to δ = 1.0e-5.

[0146] Step 14: Optimize the gap repair mesh, wherein the search level is set to 1, the smoothing method is Laplacian smoothing, λ=0.1, and filtering and simplification operations such as deduplication and merging adjacent vertices are performed, and the mesh is stored in the optimized repair mesh model file.

[0147] Step 15: Calculate the number of gaps in the optimized patching mesh according to the grayscale image projected on the xy plane. If the design requirements are met, jump to the next step to stop the calculation; otherwise, jump to step 10 to continue patching the optimized mesh.

[0148] Step 16: End the calculation and output the optimized gap repair mesh model file.

[0149] Figure 11 FIG. 1 is a schematic structural diagram of a three-dimensional reconstruction device according to an embodiment of the present disclosure. Figure 11 As shown, the three-dimensional reconstruction device 80 provided by the embodiment of the present disclosure mainly includes: a point cloud acquisition module 81, a grid division module 82, a data set creation module 8383, a sampling point information generation module 84, a predicted point cloud determination module 85, a point cloud generation grid model module 86 and an optimization grid model module 87.

[0150] The point cloud acquisition module 81 is used to obtain the initial input point cloud of the physical object to be reconstructed; the grid division module 82 is used to divide the input point cloud space into grids according to the set resolution; the data set creation module 83 is used to traverse the grid, use the continuous stereo 27-grid differential local space to create a training data set and a prediction data set, and parallel or distributedly calculate the global model; the sampling point information generation module 84 is used to train the prediction network model based on the created training data set, and use the prediction network model to predict the z-axis coordinate component and color information of the sampling point based on the predicted input feature vector of the xy plane sampling point; the predicted point cloud determination module 85 is used to generate a predicted patching point cloud based on the xy plane sampling point, the predicted z-axis coordinate component and the color information, and merge it with the input point cloud to obtain a predicted point cloud to improve the resolution; the point cloud generation grid model module 86 is used to convert the predicted point cloud into a grid model; the grid model optimization module 87 is used to determine whether the grid model performs gap repair, and perform smoothing filtering and simplification and other processing to optimize the resolution and computational efficiency.

[0151] This embodiment provides a 3D reconstruction device for performing the following process: obtaining an initial input point cloud of an object to be reconstructed; dividing the input point cloud space into a grid according to a set resolution; traversing the grid, creating a training dataset and a prediction dataset using a continuous stereo 27-grid differential local space, and computing a global model in parallel or distributed manner; training a prediction network model based on the created training dataset, and using the prediction network model to predict the z-axis coordinate components and color information of the sampling points based on the predicted input feature vectors of the xy-plane sampling points; generating a predicted patch point cloud based on the xy-plane sampling points, the predicted z-axis coordinate components, and the color information, and merging it with the input point cloud to obtain a predicted point cloud to improve resolution; converting the predicted point cloud into a grid model, and performing smoothing, filtering, and simplification processing to optimize resolution and computational efficiency. Using parallel or distributed cluster computing shortens computation time, enabling rapid reconstruction using point cloud and grid models.

[0152] In one possible embodiment, the grid division module 82 is used to divide the coordinate axis interval of the input point cloud space into equal or unequal parts according to the set resolution to obtain a grid; assign a unique grid number to each of the grids; use the grid number as a key and the point data as a value to form a key-value tuple; store the key-value tuple into a grid number file with one point per line, and each line of data in the grid number file includes the parameters of a point, and the point parameters include the following: point coordinates, point color, normal vector and the grid number where the point is located.

[0153] In one possible embodiment, the data set creation module 83 is used to use the central grid of the three-dimensional 27-square grid as the current grid when traversing the grid; obtain the neighbor grids of the current grid and the points therein; use each point in the central grid of the 27-square grid as a feature label point and the other points in the 27-square grid as a neighbor point group to create a training data set; construct a training input feature vector and a training label vector based on each of the feature label points and the ordered neighbor point group; determine whether the number of points in the current calculation grid is less than a preset value; if the number of points in the current calculation grid is less than the preset value, use the current calculation grid as a prediction patch grid, and sample the xy plane coordinates of the prediction points; select N plane input sampling points from the prediction patch grid, where N is at least the difference between the preset value and the number of points in the prediction patch grid, and generate a neighbor point group for each non-sampling point within the 27-square grid other than the sampling point of the current calculation grid to create a prediction data set; construct a prediction input feature vector based on each of the sampling points and the ordered neighbor point group.

[0154] In one possible embodiment, the sampling point information generation module 84 is configured to feed the predicted input feature vector of the sampling point into a pre-trained prediction network model, so that the prediction network model outputs the z-axis coordinate component and color information of the sampling point, or outputs the z-axis coordinate component, color information, and normal vector of the sampling point; the prediction network model is trained using a training dataset, the training dataset including a training input feature vector and a training label vector constructed from an initial input point cloud; the input feature vector is determined by the two-dimensional plane xy coordinates of the current calculation point in the central grid and the three-dimensional coordinate vectors of the ordered group of neighboring points; the label vector is determined by the z-axis coordinate component, color information, and normal vector of the calculation point in the central grid, or the label vector is determined by the z-axis coordinate component and color information of the calculation point in the central grid.

[0155] In one possible embodiment, the predicted point cloud determination module 85 is used to obtain an enhanced predicted point cloud; generate a predicted patch point cloud using the xy-axis coordinate components of the plane sampling points, the predicted z-axis coordinate components and the color values, and merge it with the input point cloud to generate a predicted point cloud.

[0156] In one possible embodiment, the point cloud generates a mesh model module 86, which is used to convert the predicted input point cloud into a mesh model; set the mesh resolution to divide the predicted input point cloud space into grids; traverse the grids and calculate the average normal vector and center point coordinates of each grid; construct the mesh section with the center point coordinates and the average normal vector; calculate the intersection of the mesh section and the mesh boundary edge; use the intersection as the polygonal face vertices of the mesh model, and use the face formed by the intersection as the polygonal face of the mesh model, and the vertices and polygonal faces are stored in the mesh model file of the predicted point cloud.

[0157] In one possible embodiment, the mesh model module 87 is optimized to perform gap repair of the point cloud model and the mesh model; all points in the input point cloud model are projected onto the gap identification plane; the gap identification plane is divided into a plane grid according to the spatial resolution, the point cloud includes at least one projection point in the plane grid, and the gap has no projection point in the plane grid; sampling points are obtained in the plane grid, and the known edge grid closest to the sampling point is used as a neighbor grid; a repair input data set is generated based on the neighbor grid; a repair prediction point cloud is generated based on the repair input data set; a repair prediction mesh model is generated based on the repair prediction point cloud; and the repair prediction mesh model is sequentially deduplicated, vertex merged, estimated, smoothed, filtered, and simplified.

[0158] The three-dimensional reconstruction device provided in the embodiment of the present disclosure can execute the steps executed in the three-dimensional reconstruction method provided in the embodiment of the method of the present disclosure. The execution steps and beneficial effects are not repeated here.

[0159] Figure 12 This is a schematic diagram of the structure of a computing facility in an embodiment of the present disclosure. The computing facility may be one or more three-dimensional reconstruction facilities. Figure 12 , showing a structural diagram of a computing facility 900 suitable for implementing the embodiments of the present disclosure.

[0160] like Figure 12 As shown, computing facility 900 may include computing nodes (nodes 1 to n) 920, which can access node data and data on storage devices 930 (storage 1 to storage m) via high-speed interconnect devices 910, such as gigabit networks. The facility is a distributed cluster consisting of n computing nodes and m storage nodes, each of which can execute relevant programs to implement the 3D reconstruction method according to the embodiments of the present disclosure. Each storage node can store input, intermediate, and output data.

[0161] Typically, the following devices can be connected to the storage node to obtain input video and image sequences and store output computational results: for example, cameras, aerial drones, mobile phones, computers, other computing clusters, and cloud storage devices.

[0162] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowcharts, thereby implementing the three-dimensional reconstruction method described above. When the computer program is executed by the distributed cluster 900, the functions defined in the method of the embodiments of the present disclosure are performed.

[0163] The computer storage device may be included in the device; or it may exist independently and not be installed in the facility.

[0164] In a possible implementation of the present disclosure, the distributed cluster carries one or more programs. When the one or more programs are executed by a node device, the node device can implement all or part of the three-dimensional reconstruction method described in any of the above embodiments.

[0165] In a possible implementation of the present disclosure, when the above one or more programs are executed by a node device, the node device may also execute other steps described in the above embodiment.

[0166] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to C / C++, Java, Python, and Scala; computing and storage cluster frameworks include but are not limited to Spark, Flink, Ray, Hadoop, and Milvus. The program code may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote server or computing cluster. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0167] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, patching point cloud gaps and patching grid model gaps, patching grid model gaps itself includes patching point cloud gaps, and the execution process is parallel. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.

[0168] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0169] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0170] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0171] In addition, although each operation is described in a specific order, this should not be required to require these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0172] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Obtain the initial input point cloud of the object to be reconstructed; Divide the input point cloud space into grids according to the set resolution; Traverse the grid and use the continuous 27-grid differential local space to create training and prediction datasets, and calculate the global model in parallel or distributed manner; Training a prediction network model based on the created training data set, and using the prediction network model to predict the z-axis coordinate component and color information of the sampling point based on the prediction input feature vector of the xy plane sampling point; Generate a predicted patch point cloud based on the xy plane sampling points, the predicted z-axis coordinate component and the color information, and merge it with the input point cloud to obtain a predicted point cloud to improve the resolution; Convert the predicted point cloud into a mesh model and perform smoothing, filtering and simplification to optimize resolution and computational efficiency; The traversal of the grid, differentiating the local space using a continuous three-dimensional 27-square grid, creating a training data set and a prediction data set, and parallel or distributed computing of the global model include: The central grid of the 27-square grid is used as the current calculation grid; Obtaining neighboring grids of the current calculation grid and points within them; Each point in the center grid of the 27-square grid is used as a label point, and the other points in the 27-square grid are used as neighbor point groups to create a training data set; Constructing a training input feature vector and a training label vector based on each of the label points and the ordered neighbor point group; Determine whether the number of points in the current calculation grid is less than a preset value; If the number of points in the current calculation grid is less than the preset value, the current calculation grid is used as a prediction patching grid, and the xy plane coordinates of the prediction points need to be sampled; Selecting N planar input sampling points from the prediction patching grid, where N is at least the difference between the preset value and the number of points in the prediction patching grid, and generating a neighbor point group for each non-sampling point within the 27-square grid outside the sampling point of the current calculation grid to create a prediction data set; A prediction input feature vector is constructed based on each of the sampling points and the ordered neighbor point group.

2. The method according to claim 1, characterized in that The step of dividing the input point cloud space into grids according to the set resolution includes: Divide the coordinate axis interval of the input point cloud space into equal or unequal parts according to the set resolution to obtain a grid; assigning a unique grid number to each of the grids; The grid number is used as a key and the point data is used as a value to form a key-value tuple; The key-value tuple is stored in a grid number file with one point per line. Each line of data in the grid number file includes parameters of a point. The parameters of the point include the following: point coordinates, point color, normal vector and the grid number where the point is located.

3. The method according to claim 1, characterized in that Predicting and generating a z-axis coordinate component and color information of the sampling point based on the predicted input feature vector of the sampling point, or generating the z-axis coordinate component, color information, and normal vector of the sampling point, including: Feeding the predicted input feature vector of the sampling point into a pre-trained prediction network model, so that the prediction network model outputs the z-axis coordinate component and color information of the sampling point, or outputs the z-axis coordinate component, color information and normal vector of the sampling point; The prediction network model is trained by a training data set, and the training data set includes a training input feature vector and a training label vector constructed from an initial input point cloud; the input feature vector is determined by the two-dimensional plane xy coordinates of the calculation point in the central grid and the three-dimensional coordinate features of the neighboring points; the label vector is determined by the z-axis coordinate component, color information and normal vector of the calculation point in the central grid, or the label vector is determined by the z-axis coordinate component and color information of the calculation point in the central grid.

4. The method according to claim 1, wherein The predicted point cloud is converted into a mesh model and smoothed, filtered, and simplified to optimize resolution and computational efficiency, including: generating a grid model of the predicted point cloud using the predicted point cloud; Determining whether there are gaps in the grid model of the predicted point cloud; If there are gaps in the grid model of the predicted point cloud, the grid model of the predicted point cloud is processed to fill the gaps, merge, smooth, filter, and simplify.

5. The method according to claim 4, characterized in that Generating a grid model of the predicted point cloud using the predicted point cloud includes: Setting a grid resolution to divide the space where the predicted point cloud is located into spatial grids; Traverse the spatial grid and calculate the average normal vector and center point coordinates of each grid; Constructing the mesh section using the center point coordinates and the average normal vector; Calculating the intersection of the mesh cross section and the mesh boundary edge; The intersection points are used as vertices of a mesh polygonal surface, and a surface formed by the intersection points is used as the mesh polygonal surface. The mesh polygonal surface is included in the mesh model of the predicted point cloud.

6. The method according to claim 4, characterized in that The mesh model of the predicted point cloud is patched, merged, smoothed, filtered, and simplified, including: Projecting each point in the point cloud model onto a gap identification plane; Dividing the gap identification plane into a plane grid according to the spatial resolution, wherein the point cloud includes at least one projection point in the plane grid, and the gap has no projection point in the plane grid; Obtaining a sampling point in the plane grid, and taking a known edge grid closest to the sampling point as a neighboring grid; generating a patch input dataset based on the neighbor grid; generating a patch prediction point cloud based on the patch input data set; generating a repair prediction grid model based on the repair prediction point cloud; The repair prediction mesh model is sequentially subjected to deduplication, vertex merging, estimation, smoothing, filtering and simplification processes.

7. A three-dimensional reconstruction device, characterized in that: include: The initial point cloud acquisition module is used to obtain the motion structure recovery point cloud of the object to be reconstructed; The grid division module is used to divide the input point cloud space into grids according to the set resolution; The dataset creation module is used to traverse the grid, create training datasets and prediction datasets using the continuous 27-grid differential local space, and perform parallel or distributed computing of the global model. A sampling point information generation module is used to train a prediction network model based on the created training data set, and use the prediction network model to predict the z-axis coordinate component and color information of the sampling point based on the prediction input feature vector of the xy plane sampling point; A predicted point cloud determination module is used to generate a predicted patch point cloud based on the xy plane sampling points, the predicted z-axis coordinate component and the color information, and merge it with the input point cloud to obtain a predicted point cloud to improve the resolution; The point cloud generation mesh model module uses parallel or distributed computing to convert the predicted point cloud into a mesh model, and performs smoothing, filtering, and simplification processing to optimize resolution and computational efficiency; The dataset creation module is used to: The central grid of the 27-square grid is used as the current calculation grid; Obtaining neighboring grids of the current calculation grid and points within them; Each point in the center grid of the 27-square grid is used as a label point, and the other points in the 27-square grid are used as neighbor point groups to create a training data set; Constructing a training input feature vector and a training label vector based on each of the label points and the ordered neighbor point group; Determine whether the number of points in the current calculation grid is less than a preset value; If the number of points in the current calculation grid is less than the preset value, the current calculation grid is used as a prediction patching grid, and the xy plane coordinates of the prediction points need to be sampled; Selecting N planar input sampling points from the prediction patching grid, where N is at least the difference between the preset value and the number of points in the prediction patching grid, and generating a neighbor point group for each non-sampling point within the 27-square grid outside the sampling point of the current calculation grid to create a prediction data set; A prediction input feature vector is constructed based on each of the sampling points and the ordered neighbor point group.

8. An electronic device, characterized in that: The electronic device comprises: One or more processing machines; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processing machines, the one or more processing machines implement the three-dimensional reconstruction method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the three-dimensional reconstruction method according to any one of claims 1 to 6 is implemented.

Images