Three-dimensional reconstruction method and device, equipment and storage medium
By using continuous three-dimensional 27 grid differential local space and deep learning model to predict z-axis coordinates and color information in the three-dimensional reconstruction method, the long and hollow problems of three-dimensional reconstruction in the existing technology are solved, and fast and efficient three-dimensional model reconstruction and optimization.
Patent Information
- Application Number
- CN202410984638.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-22
AI Technical Summary
Existing 3D reconstruction methods work with large number of image sequences, and generate dense point cloud models for a long time and have hollow problems, especially at the edges and a small part of the 3D model.
The training and prediction data sets are created using continuous stereo 27 grids, and the deep learning model is used to predict the z-axis coordinates and color information based on the x-y plane sampling points, to generate the predicted point cloud and merge it with the initial point cloud. Convert to mesh models and perform smooth filtering and simplified processing to optimize resolution and computational efficiency.
Fast reconstruction of point cloud and mesh models is realized, reducing or eliminating holes in the three-dimensional model, and improving computing efficiency and resolution.
Smart Images

Figure CN119027478A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical fields of point cloud, differential grid computing, deep learning and image processing, and in particular to a three-dimensional reconstruction method, device, equipment and storage medium. Background Art
[0002] Three-dimensional (3D) reconstruction is the establishment of a mathematical model of a three-dimensional object represented and processed by a computer. 3D reconstruction is the basis for processing, operating and analyzing the properties of 3D objects in a computer environment, and is also the key to technologies such as virtual reality and proto-universe.
[0003] In the related art, the 3D reconstruction method is mainly to establish a 3D digital model of the input image scene based on the input image sequence. For example, the input image sequence is processed by the Structure From Motion (SFM) algorithm to obtain a sparse point cloud model, and then the sparse point cloud model is processed by the Multi-view Stereo (MVS) algorithm to obtain a dense point cloud model, and then the dense point cloud model is repaired and the surface is reconstructed to obtain a high-resolution 3D mesh model close to the real object.
[0004] However, when the number of images in the input image sequence is large, it takes a long time and is slow to generate a dense point cloud model. If the number of images in the input image sequence is reduced, there will be many holes in a few areas such as the edges of the 3D model. Summary of the invention
[0005] In order to solve all or at least one of the above technical problems, the embodiments of the present disclosure provide a three-dimensional reconstruction method, device, equipment and storage medium to achieve rapid reconstruction of point cloud models and mesh models, and reduce or even eliminate holes in three-dimensional models.
[0006] In a first aspect, an embodiment of the present disclosure provides a three-dimensional reconstruction method, including: obtaining an initial input point cloud of a physical object to be reconstructed; dividing the input point cloud space into grids according to a set resolution; traversing the grids, using a continuous stereoscopic 27-grid differential local space to create a training data set and a prediction data set, and computing a global model in parallel or in a distributed manner; training a prediction network model based on the created training data set, and using the prediction network model to predict the z-axis coordinate component and color information of the sampling point based on the predicted input feature vector of the xy plane sampling point; generating a predicted patch point cloud based on the xy plane sampling point, the predicted z-axis coordinate component and the color information, and merging it with the input point cloud to obtain a predicted point cloud to improve the resolution; converting the predicted point cloud into a grid model, and performing smoothing filtering and simplification processing to optimize the resolution and computational efficiency. In a second aspect, an embodiment of the present disclosure provides a three-dimensional reconstruction device, including: an initial point cloud acquisition module, used to acquire an initial input point cloud of a physical object to be reconstructed; a grid division module, used to divide the input point cloud space into grids according to a set resolution; a data set creation module, traversing the grid, using continuous stereoscopic 27-grid differential local space to create a training data set and a prediction data set, and parallel or distributed computing a global model; a sampling point information generation module, training a prediction network model based on the created training data set, and using the prediction network model to predict and generate the z-axis coordinate component and color information of the sampling point and an optional normal vector based on the predicted input feature vector of the xy plane sampling point; a predicted point cloud determination module, generating a predicted patching point cloud based on the xy plane sampling point, the predicted z-axis coordinate component and the color information, and merging it with the existing input point cloud to obtain a predicted point cloud to improve the resolution; a point cloud generation grid model module, using 27-grid differential and parallel or distributed computing to convert the three-dimensional point cloud model into a three-dimensional grid model, and then performing gap patching, smoothing filtering and simplification processing to optimize the resolution and computing efficiency.
[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising:
[0008] one or more processing machines;
[0009] A storage device for storing one or more programs;
[0010] When one or more programs are executed by one or more processing machines, the one or more processing machines implement the three-dimensional reconstruction method provided in the first aspect above.
[0011] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional reconstruction method provided in the first aspect above.
[0012] In a fifth aspect, an embodiment of the present disclosure provides a program product, which includes a computer program, and when the computer program is executed by a processor, it implements the three-dimensional reconstruction method provided in the first aspect above.
[0013] The disclosed embodiment provides a three-dimensional reconstruction method, device and computing facility, the method comprising: obtaining an initial input point cloud of a physical object to be reconstructed; dividing the input point cloud space into grids according to a set resolution; traversing the grids, using a continuous stereoscopic 27-grid differential local space to create a training data set and a prediction data set, and computing a global model in parallel or in a distributed manner; training a prediction network model based on the created training data set, and using the prediction network model to predict the z-axis coordinate component and color information of the generated sampling point based on the predicted input feature vector of the xy plane sampling point; generating a predicted patch point cloud based on the xy plane sampling point, the predicted z-axis coordinate component and the color information, merging it with the input point cloud to obtain a predicted point cloud to improve the resolution; converting the predicted point cloud into a grid model, and performing gap patching, smooth filtering and simplification processing to optimize the resolution and computing efficiency. The use of parallel or distributed cluster computing shortens the computing time and enables rapid reconstruction of point clouds and grid models. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0015] Figure 1 A schematic diagram of a process of a point cloud 3D reconstruction method in an embodiment of the present disclosure;
[0016] Figure 2 It is a schematic diagram of a grid cube of a basic computing unit 27 in an embodiment of the present disclosure;
[0017] Figure 3 A schematic diagram of a spatial region where gaps exist in a point cloud in an embodiment of the present disclosure;
[0018] Figure 4 A schematic diagram of a use case of data point distribution of a training data set in an embodiment of the present disclosure;
[0019] Figure 5 is a schematic diagram of the structure of a depth prediction network in an embodiment of the present disclosure;
[0020] Figure 6 A schematic diagram of a flow chart of a mesh optimization 3D reconstruction method in an embodiment of the present disclosure;
[0021] Figure 7is the intersection of the grid and the grid point fitting plane in the embodiment of the present disclosure;
[0022] Figure 8 A schematic diagram of the triangular surface composition of the mesh cross-section plane in the embodiment of the present disclosure;
[0023] Fig. 9 A schematic diagram of a plane gap grid obtained in an embodiment of the present disclosure;
[0024] Fig.10 is a schematic diagram of an edge connecting adjacent mesh cutting planes in an embodiment of the present disclosure;
[0025] Fig.11 is a schematic structural diagram of a three-dimensional reconstruction device in an embodiment of the present disclosure;
[0026] Fig.12 A schematic diagram of the structure of a computing facility in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0029] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0030] It should be noted that the concepts mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0033] 3D reconstruction is the establishment of a mathematical model of a 3D object represented and processed by a computer. It is the basis for processing, operating and analyzing 3D objects in a computer environment, and is also the key to technologies such as virtual reality and proto-universe. Currently, the commonly used representation forms of 3D models mainly include depth maps, voxel models, point cloud models and mesh models.
[0034] The depth map includes a 2D map of the real scene and a 2D depth map. The disadvantage of the depth map is that it cannot represent the features of the interior of the object and the obscured objects, and can only represent the vertical depth of the viewpoint plane.
[0035] The voxel model is a model that uses three-dimensional pixels to represent a three-dimensional object. The amount of calculation increases with the cube of the single-axis resolution. If the resolution is very high, the amount of calculation is very large.
[0036] The point cloud model is a three-dimensional model that uses three-dimensional point features to represent an object. It is a series of spatial feature point groups with coordinates. The image itself cannot directly generate a point cloud, but the SFM algorithm can be used to process the input image to obtain sparse point cloud data, and then the MVS algorithm can be used to process the sparse point cloud data to obtain dense point cloud data. However, the disadvantages of the above point cloud data include poor robustness, discontinuity between points, and rough surface of the reconstructed object. In addition, laser radar and depth cameras can be used to directly obtain the depth information of the point cloud, but the equipment cost is increased. Although a lot of calculations are saved, there are also problems such as narrow measurement range, high noise, small field of view, uneven data, sunlight interference, and inability to determine the projection material.
[0037] A mesh model is a model that approximates a three-dimensional object using a series of vertices, edges, and polygons (usually triangles). The mesh model representation method saves the number of points and computing memory; however, the mesh model has the same resolution problem as the voxel model. One mesh is equivalent to one voxel, and the amount of calculation still increases at the cube of the single-axis resolution. When modeling large scenes with high resolution, the amount of calculation is very large. If a high-resolution orthophoto 3D reconstruction model is directly generated using voxels or meshes for ground image landscapes, the computer's memory and CPU or GPU load will be very large.
[0038] In order to solve all or at least one of the above-mentioned technical problems, the embodiments of the present invention provide a three-dimensional object reconstruction method, apparatus, device and storage medium, the method comprising: obtaining an initial point cloud model of the physical object to be reconstructed; dividing the initial point cloud space into regional grids according to a set resolution; traversing the regional grids, creating a data set using a continuous stereoscopic 27-grid differential local space, and computing a global model in parallel or in a distributed manner; predicting the z-axis coordinate component and color information of the sampling point based on the input feature vector of the sampling point; generating a patched color point cloud model using the z-axis coordinate component and color information of the sampling point; adding the patched color point cloud model to the initial point cloud model to obtain a predicted patched point cloud model, thereby improving the resolution of the point cloud model. At the same time, due to the small number of input image sequences and the combination of distributed computing, the model reconstruction time is shortened, thereby realizing rapid reconstruction of the point cloud model.
[0039] The three-dimensional reconstruction method provided by the present invention is described in detail below in conjunction with the embodiments and drawings.
[0040] Figure 1 This is a flow chart of a three-dimensional reconstruction method in an embodiment of the present disclosure. This embodiment is applicable to the case of three-dimensional reconstruction of a physical scene. The method can be executed by a three-dimensional reconstruction device. The three-dimensional reconstruction device can be implemented in software and / or hardware. The three-dimensional reconstruction device can be configured with a terminal device or a server.
[0041] like Figure 1 As shown, the 3D reconstruction method provided by the embodiment of the present disclosure mainly includes steps S101-S106.
[0042] S101, obtaining an initial input point cloud of a physical object to be reconstructed.
[0043] The real object refers to the real object existing in the real scene, and the real object to be reconstructed refers to the real object that needs to be represented by a three-dimensional model in a computer device. The initial input point cloud is a point cloud model generated by an image. The point cloud model is a three-dimensional model of an object represented by spatial point feature data. Each point feature of the point cloud model contains three-dimensional coordinates, which may contain color information, reflectivity information, intensity information, etc.
[0044] The point cloud model is stored in a computer in a polygonal mesh file format (ply). The ply mesh file stores point cloud data, which includes the three-dimensional coordinates, color information and normal vector of each point. The color information can be expressed in red green blue (RGB) form.
[0045] The initial input point cloud refers to the dense point cloud model obtained by processing the image sequence or monocular video using the SFM algorithm and the MVS algorithm. The above-mentioned input image sequence or the image sequence converted from the monocular video includes two-dimensional images of the object to be reconstructed taken from different angles, and the image feature points are extracted from the image sequence, the feature points are matched, the camera pose and the coordinate system transformation matrix are estimated, and the sparse color point cloud is calculated, and then the image distortion is removed, the three-dimensional surface is matched, the stereo fusion is performed, the point cloud noise is removed, and the dense color point cloud is calculated and stored in the ply file.
[0046] S102: Divide the input point cloud space into grids according to the set resolution.
[0047] The resolution setting is limited by the computing power of the device. The higher the resolution, the greater the computing workload and cost. It needs to be determined according to the specific task. In the embodiment of the present invention, the resolution is an adjustable parameter and is not limited.
[0048] The point cloud space is defined as a 3D hexahedral space limited by the length, width, and depth of the point cloud. The length of the point cloud refers to the maximum value of the y-axis in the point cloud model, H. max and the y-axis minimum value H min The width of the point cloud refers to the maximum value W of the x-axis in the point cloud model. max and the minimum value W of the x-axis min The depth of the point cloud refers to the maximum value D of the z-axis in the point cloud model. max and the z-axis minimum value D min The area between is long.
[0049] A grid is a set of space interval grids divided according to a set resolution. In certain cases, it can refer to a space interval grid, where the size of each grid can be the same or different. In order to facilitate programming, the space is divided equally, but in some special areas with high point density, more space divisions can be used to improve accuracy. At this time, the set resolution is not uniformly achieved, and the space is not equally divided.
[0050] In one possible implementation, the input point cloud space is divided into grids according to a set resolution, including: dividing the input point cloud coordinate axis interval equally or unequally according to the set resolution to obtain a grid; assigning a unique grid number to each spatial interval grid; using the grid number as a key and the point data as a value to form a key-value tuple; storing the key-value tuple into a grid number file with one point per line, each line of data in the grid number file includes parameters of a point, and the point parameters include the following: point coordinates, point color, normal vector and the number of the grid where the point is located.
[0051] The coordinate axis of the input point cloud is divided equally or unequally according to the set resolution to obtain a grid. Each grid is compiled with an area number, and all points of the point cloud and the grid number of the point are stored in a csv format file according to one line for each point. Each line of features in the csv grid number file includes the vertex parameters of the point, and the vertex parameters include the following: point coordinates, point color, normal vector and the grid number, where the normal vector is an optional feature parameter.
[0052] Specifically, set the x-axis resolution of the point cloud to Xp, the y-axis resolution to Yp, and the z-axis resolution to Zp. Calculate the maximum y-axis length H, the maximum x-axis width W, and the maximum z-axis depth D of the point cloud, and use the maximum value of the length H, width W, and depth D to normalize each point so that the coordinate components of each point are in the range [0.0,1.0]. Among them, the maximum y-axis length H is the maximum y-axis value H max and the y-axis minimum value H min The maximum length W of the x-axis is the maximum value W of the x-axis max and the x-axis minimum value W min The maximum length D of the z-axis is the maximum value D of the z-axis max and the z-axis minimum value D min The normalized point cloud space x-axis, y-axis, and z-axis coordinate intervals are divided into integer equal parts according to the set resolution, and the side length of the point cloud space x-axis interval grid is calculated as x step =W / Xp, the side length of the y-axis interval grid is y step =H / Yp, the side length of the z-axis interval grid is z step =D / Zp. The length of each grid on the x-axis is x step , the side length on the y-axis is y step , the length of the side on the z-axis is z step .
[0053] For subsequent calculations, each spatial interval grid is numbered. The specific method is as follows.
[0054] The integer interval of the region number Xi of the x-axis is [0, Xp), the integer interval of the region number Yi of the y-axis is [0, Yp), and the integer interval of the region number Zi of the z-axis is [0, Zp). According to the interval numbers of the three coordinate axes, each spatial region grid is assigned a unique point coordinate number Pi = (xi, yi, zi), where xi∈[0, Xp), yi∈[0, Yp), zi∈[0, ZXp). The grid number can be set to a string connected by a fixed number of coordinate interval numbers, for example: xiyizi. For example, when the number of bits of resolution is 4, if xi=0000, yi=0128, zi=1080, then the spatial region number = "000001281080". In actual calculations, the coordinate number triple (xi, yi, zi) or string can be directly used for calculation according to the characteristics of the programming language used.
[0055] According to the scale of the dense point cloud obtained in S101, use single machine, parallel or distributed computing to traverse the points P = (x, y, z) of the input point cloud, calculate point by point and store the point feature x, y and z axis coordinates, x, y and z axis colors, x, y and z axis normal vectors, x, y and z axis interval numbers in the point cloud grid number file by row. At the same time, the grid side length, resolution, normalization parameters, etc. are stored in the grid number configuration file for subsequent calculations and model scale calculation.
[0056] S103, traversing the regional grid, using the continuous three-dimensional 27-grid differential local model space to create a training data set and a prediction data set, and computing the global model in parallel or in a distributed manner.
[0057] First, read the point cloud grid number file and grid number configuration file. Use single-machine, parallel or distributed computing according to the input point cloud obtained in S101. Use the groupby function to group all the points of the point cloud into grid point groups with the grid number as the key, and store the grid point groups in a container with the grid number tuple as the key for query at any time. Storing the grid point group in the container includes but is not limited to storing the three-axis coordinate values, color information, normal vectors and grid numbers of the grid points in the container. Using hash table and other technologies to directly access the key-value container to obtain neighboring points can save memory, make the analysis intuitive, and do not require cumbersome preprocessing. It can also be quickly calculated in parallel, concurrent, and distributed ways.
[0058] In a possible implementation, the grid is traversed, the local space is differentiated using a continuous stereoscopic 27-grid, a training data set and a prediction data set are created, and a global model is calculated in parallel or in a distributed manner, including: taking the central grid of the stereoscopic 27-grid as the current grid; obtaining the neighbor grids of the current grid and the points therein; taking each point in the central grid of the 27-grid as a label point and the other points in the 27-grid as a neighbor point group to create a training data set; constructing a training input feature vector and a label vector based on each label point and the ordered neighbor point group; judging whether the number of points in the current calculation grid is less than a preset value; if the number of points in the current calculation grid is less than the preset value, the current calculation grid is used as a prediction patch grid, and the xy plane coordinates of the prediction points need to be sampled; N plane input sampling points are selected from the prediction patch grid, N being at least the difference between the preset value and the number of points in the prediction patch grid, and each non-sampling point within the 27-grid other than the sampling point of the current calculation grid generates a neighbor point group of the sampling point to create a prediction data set; and constructing a prediction input feature vector based on each sampling point and the ordered neighbor point group.
[0059] Point cloud grouping grid is similar to voxel, but instead of calculating all points in the voxel grid into a 3D pixel point, it takes a grid as the center and considers all points in up to 26 adjacent grids of the center grid, and each grid is calculated for local precise search. Figure 4 As shown, the five-pointed star and sun symbol in the central grid represent the points of the current grid, where the sun symbol represents the current point, the five-pointed star point and the circle in the adjacent grid represent the neighbor point, and the distance between the current point and the neighbor point is calculated in the three-dimensional 27-square grid to sort and extract the top nearest neighbor points.
[0060] Create a training data set, traverse the grid, set the currently traversed grid as the central grid, and the feature vector of each point in the current grid contains the xy coordinate component of the point, the 3-axis coordinates of the top nearest neighbor point of the point, the 3-axis color, and the optional 3-axis normal vector. The shape of the neighbor point feature is (Top, 6) or (Top, 9), the former does not include the normal vector, and the latter includes the normal vector. The x and y coordinates of the current point can be alternately arranged to form a 6-dimensional or 9-dimensional vector, which is arranged in front of the neighbor point feature to form a training input feature vector. For example, the Top=T feature excluding the normal vector is: [[x,y,x,y,x,y],[x1,y1,z1,r1,g1,b1],...,[xT,yT,zT,rT,gT,bT]], where r, g, and b represent color value components between 0 and 1, and 1 to T represent the sequential point number. The label of each point contains the z-axis coordinate component of the point, color information, and an optional normal vector, such as [z,r,g,b] excluding the normal vector.
[0061] Extract the point group data of the 26 adjacent grids of the current grid. If the adjacent grid does not exist, the spatial area and grid at that location do not have any points. The number of adjacent grids actually extracted is no more than 26. Then, each current point generates a list containing only the current neighbor points, calculates the distance between the current point and each neighbor point in the list, sorts them from near to far according to the distance, extracts the nearest top points, and fills the points that are less than the top with 0 vectors. Finally, the depth and color information of the current point are used to form a training label vector.
[0062] To create a prediction data set, it is necessary to determine the input feature vector of the sampling points in the spatial region grid, where the grids that meet the preset conditions are used to calculate the prediction data features. The preset condition can mean that the number of points in the grid is less than the preset value, which is set according to the actual situation. For example, the preset value is 8. If the memory and CPU are sufficient, it can also be set to a larger value such as 16.
[0063] Sampling points refer to random sampling points in the xy plane for enhancing or patching the mesh to be predicted. A prediction input feature vector consists of the xy-2D coordinates of the sampling point, the 3D coordinates of the neighbor point, color information, and an optional normal vector. Among them, the neighbor feature points are calculated in the same way as the training input feature vector, and the xy coordinate components of the current point before the neighbor feature point are replaced by the xy coordinate components of the sampling point. The xy-2D coordinates of the sampling point are the x-axis components and y-axis components of the 3D point coordinates. The sampling points are used for prediction, and only the 2D coordinate components of the xy plane are sampled. The depth z-axis component is unknown and is predicted and estimated using a deep learning model.
[0064] In a possible implementation, the regional grid is traversed to determine the predicted input feature vector of the sampling point in the predicted grid, including: taking the central grid of the 27-square grid as the current grid; obtaining the neighboring grid of the current grid; judging whether the number of points Nr in the current grid is less than the preset value N; if the number of points Nr in the current grid is less than the preset value N, the current grid is taken as the predicted grid; sampling Ns points from the predicted grid in a uniform distribution, where Ns is the difference between the preset value and the actual number of points in the predicted grid, that is, Ns=N-Nr; constructing the predicted input feature vector of the sampling point using the xy-2D coordinates of the sampling point and the top nearest neighbor points; if the number of points Nr in the current grid is greater than or equal to the preset threshold N, it means that the number of points in the current grid is sufficient and there is no need to generate sampling points. Then directly read the next grid calculation, so as to traverse and calculate each grid and enhance the information of the sparse part of the point cloud.
[0065] A grid is used as a calculation unit. Figure 2 As shown in the figure, the cube group on the left (a) is a schematic diagram of the actual calculation unit cube 27 grid (b) on the right after being split. In the actual calculation, there are no gaps between the grids, but some grids do not contain vertices and are empty, so there is no grid number for this part of the space in the input point cloud. Figure 3 As shown in the figure, there are no points in the point cloud at the positions of the gaps numbered 130, 230, and 330, so they are ignored when creating the prediction dataset. Only the grids containing the points are collected, and then the gaps are filled. The prediction dataset refers to the dataset composed of the input feature vectors of all sampling points.
[0066] Take the current grid as the center grid, obtain all possible adjacent grids, and calculate the neighbor point list of each point in the center grid. The number of adjacent grids is up to 26. To calculate the top neighbor points of a point in the center grid, in addition to the point, add the points in the center grid and adjacent grids to the neighbor point list. Specifically, determine whether the key value container constructed above contains the following according to the area number of the current grid: Figure 2The 26 adjacent grids of the current grid are shown; if an adjacent grid is not in the key-value container, it means that the grid in the adjacent space area does not include any points; if the adjacent grid is in the key-value container, the points in the adjacent grid are added to the neighbor point list.
[0067] Calculate the number of points Nr in the current grid. If the number of points Nr is less than the preset value N, generate at least N-Nr sampling points. Sample at least N-Nr x-axis and y-axis data (xs, ys) in the current grid in a uniform distribution as prediction sampling points. Among them, xs is within the x-axis value range of the current grid; ys is within the y-axis value range of the current grid. Calculate the distance between the sampling point and the xy coordinates of each neighbor point, sort them from near to far according to the distance, and extract 1 to Top points in sequence. Those that are less than Top will be filled with 0 vectors. Top is the number of selected neighbor points.
[0068] The prediction input feature vector includes the xy coordinates of the sampling point and the 3D coordinates of the neighboring points, color information and an optional normal vector. The prediction data set can have no label vector, or the label vector can be composed of the z-axis coordinate mean, color and normal vector of the points in the current grid for subsequent prediction comparison.
[0069] The following is an explanation of the method for constructing the input feature vector.
[0070] The input feature vector can be determined by the x-axis coordinate and y-axis coordinate of the current point or sampling point, the 3-axis coordinate of the neighbor point, and the 3-axis color information of the neighbor point. Further, the input feature vector can be determined by the x-axis coordinate and y-axis coordinate of the current point or sampling point, the 3-axis coordinate of the neighbor point, the 3-axis color of the neighbor point, and the 3-axis normal vector of the neighbor point. The training input feature vector generated by the current point is used as the training data input, and the prediction input feature vector generated by the sampling point is used as the prediction data input.
[0071] If the number of neighbor points is Top, the feature form of the neighbor points is (Top, 6) or (Top, 9). The former does not include the normal vector, while the latter includes the normal vector. (Top, 6) means that there are Top neighbor points, and each neighbor point consists of 6 values, namely the 3-axis coordinates of the neighbor point and the 3-axis color of the neighbor point. (Top, 9) means that there are Top neighbor points, and each neighbor point consists of 9 values, namely the 3-axis coordinates of the neighbor point, the 3-axis color of the neighbor point, and the 3-axis normal vector of the neighbor point.
[0072] The x-axis coordinate and y-axis coordinate of the current point can be arranged alternately to form a 6-dimensional or 9-dimensional vector, and arranged in front of the neighbor point feature to form a feature point vector. For example, the Top=M feature format excluding the normal vector is: [[x,y,x,y,x,y], [x1,y1,z1,r1,g1,b1,], ..., [xM,yM,zM,rM,gM,bM]], where r, g, and b represent color value components with values of 0 to 1, and 1 to M represent neighbor point numbers. The distances from point (x,y,z) to point (x1,y1,z1) to point (xM,yM,zM) are arranged from small to large. The label vector is composed of the z-axis coordinate component of the current point and color information, and the format is as follows: [z,r,g,b].
[0073] Expand the training input feature vector, separate it with "," and convert it into a csv string; expand the label vector, separate it with "," and convert it into a csv string; connect the two with ":" to form a line of feature label data, and save it to the file line by line as a feature label vector. The format of each line is as follows:
[0074] x,y,x,y,x,y,x1,y1,z1,r1,g1,b1,...,xM,yM,zM,rM,gM,bM:z,r,g,b
[0075] The prediction input feature vector is generated using the sampling point xy coordinates and Top=M neighboring points, and its format is as follows: [[xs,ys,xs,ys,xs,ys],[x1,y1,z1,r1,g1,b1],...,[xM,yM,zM,rM,gM,bM]]. The label vector can be constructed using the z-axis coordinate component and color value of a point in the current grid, or using the z-axis coordinate mean and color value mean of the points in the current grid. For example, the point closest to the sampling point is selected as the label of the sampling point: [z_n,r_n,g_n,b_n].
[0076] Expand the predicted input feature vector of the sampling point, separate it with "," and convert it into a csv string; if there is a label vector, expand it, separate it with "," and convert it into a csv string; connect the two with ":" to form a row of feature label data, and then save it to the file row by row as the predicted feature label vector. The format of each row is as follows:
[0077] xs,ys,xs,ys,xs,ys,x1,y1,z1,r1,g1,b1,...,xM,yM,zM,rM,gM,bM:z_n,r_n,g_n,b_n
[0078] S104, training a prediction network model based on the created training data set, and using the prediction network model to predict and generate z-axis coordinate components and color information of the sampling points based on the prediction input feature vectors of the sampling points.
[0079] Feeding the predicted input feature vector of the sampling point into a pre-trained prediction network model, so that the prediction network model outputs the z-axis coordinate component and color information of the sampling point, or outputs the z-axis coordinate component, color information and normal vector of the sampling point;
[0080] After obtaining the z-axis coordinate component of the sampling point, determine whether the z-axis coordinate component exceeds the z-axis coordinate boundary of the grid where the sampling point is located. If the z-axis coordinate component exceeds the z-axis coordinate boundary of the grid, a z-axis component is randomly assigned or the value is discarded. The randomly assigned z-axis component is within the z-axis coordinate range of the grid.
[0081] The prediction network model is introduced in the embodiments of the present disclosure.
[0082] The prediction network model is trained by a training data set, which includes an input feature vector column and a label vector column created from an initial input point cloud; the input feature vector is determined by the two-dimensional plane xy coordinates of the current calculation point of the central grid and the three-dimensional xyz coordinates, color and normal vector (optional) of its Top neighbor points; the label vector is determined by the z-axis coordinate component, color information and normal vector (optional) of the current calculation point of the central grid.
[0083] like Figure 5 As shown, it is the deep prediction network used by the prediction network model. The input is the feature vector of the current point and the neighboring point. The vertex dimension Dim_v = 6 (no normal vector) or 9 (normal vector), the input dimension Dim_i = (Top+1)*Dim_v, and the output dimension Dim_o = 4. The prediction network model consists of three parts: a parallel splicing MLP group 401, a serial Conv2D group 402, and an output splicing MLP group 403. Among them, the previous layer of the parallel splicing MLP group 401 is composed of n MLPs in parallel, and the spliced hidden layer vector is connected to the output of 401 with 1 MLP. The MLP units of each layer can set and adjust the hidden layer neurons. The serial Conv2D group 402 is composed of m convolutions plus activation function Conv2d+tanh and h linear layers plus activation function Linear+tanh in series. Each convolution layer can set and adjust the hidden layer neurons, kernel window, stride and padding. The outputs of the parallel splicing MLP group 401 and the serial Conv2D group 402 are spliced and then output by the MLP network 403. The hidden layers of the output splicing MLP group 403 can be set and adjusted.
[0084] When calculating the predicted output, the xy coordinates of the current point of the input feature vector are replaced by the xy coordinates of the sampling point. For details, please refer to the description in the above embodiment, which is not specifically limited in the embodiment of the present invention.
[0085] Set the training data set read buffer size read_buff and batch size train_batch. Since the amount of point cloud data may be large, read the feature labels of read_buff rows from the training data set file at a time. Set the input scaling scale to adjust the input data to the lower 1 / 3 range of 0 to 1; set the output gain gain to adjust the output data to the lower 1 / 3 range of 0 to 1. Set the number of training rounds epoch, the learning law lr, select the MSE loss function, send the input data to the model to perform forward calculations, calculate the loss and error, and update the weight coefficients with the RMSprop algorithm. The training data set is cycled for epochs and then ends.
[0086] S105 , generating a predicted patching point cloud based on the xy plane sampling points, the predicted z-axis coordinate components and the color information, and merging the predicted point cloud with the input point cloud to obtain a predicted point cloud to improve the resolution.
[0087] The input point cloud is patched using the sampling points and their predicted z-axis coordinate components and color information to obtain the predicted point cloud.
[0088] Specifically, the plane xy coordinates of the sampling point are used instead of the plane xy coordinates of the current point input in training, and the z-axis coordinate component and RGB color information are predicted by combining the neighbor point group. The sampled xy coordinate component, the predicted z-axis coordinate component and the RGB color value are used to form a patched color point cloud. The patched color point cloud is merged with the input point cloud to obtain the predicted point cloud, thereby enhancing the point cloud quality.
[0089] S106: Convert the predicted point cloud into a mesh model, and perform smoothing, filtering, simplification and other processing to optimize resolution and computational efficiency.
[0090] The disclosed embodiment provides a three-dimensional point cloud reconstruction method, which includes: obtaining an initial input point cloud of a physical object to be reconstructed; dividing the input point cloud space into grids according to a set resolution; traversing the grids, using a continuous stereoscopic 27-grid differential local space to create a training data set and a prediction data set, and computing a global model in parallel or in a distributed manner; training a prediction network model based on the created training data set, and using the prediction network model to predict the z-axis coordinate component and color information of the sampling point based on the predicted input feature vector of the xy plane sampling point; generating a predicted patch point cloud based on the xy plane sampling point, the predicted z-axis coordinate component and the color information, and merging it with the input point cloud to obtain a predicted point cloud to improve the resolution; converting the predicted point cloud into a grid model, and performing smoothing, filtering and simplification and other processing to optimize the resolution and computing efficiency. The use of parallel or distributed cluster computing shortens the computing time and can achieve rapid reconstruction of point clouds and grid models.
[0091] Based on the point cloud prediction step S105 in the above embodiment, the present disclosure embodiment converts and optimizes the grid model of the three-dimensional point cloud reconstruction method, such as Figure 6 As shown, step S106 of the present invention is a three-dimensional optimization reconstruction method for predicting point cloud to mesh model, and the detailed steps include the following.
[0092] S201 , obtaining the predicted point cloud of the object to be reconstructed generated in step S105 .
[0093] Set the resolution of each axis of the grid. The resolution is used to divide the grid spacing of each axis to perform differential calculations. The resolution can be set to equal or unequal divisions. It can be set to high resolution in key areas or areas with high density, and set to low resolution in areas that are not key areas or areas with low density. When the point cloud is dense enough, a larger resolution can improve the 3D smoothness. For the convenience of calculation, this embodiment is set to equal division, and the predicted point cloud model space is divided into regional grids according to the set resolution. It is basically the same as the steps in S102 above, and specific reference is made to the description in the above embodiment, which will not be repeated.
[0094] S202 , generating a key-value container for querying and extracting grid data in the same way as generating a data set in step S103 .
[0095] S203, using the predicted point cloud to generate a grid model of the predicted point cloud by distributed continuous stereo 27-grid differential calculation.
[0096] Among them, generating a grid model using the predicted point cloud is to traverse the spatial grid divided by the predicted point cloud, differentiate the local grid points using a continuous three-dimensional 27-grid, and generate a global three-dimensional grid model by parallel or distributed cluster computing.
[0097] A possible implementation method includes: setting the grid resolution to divide the space where the predicted point cloud model is located into spatial grids; traversing the spatial grids, calculating the average normal vector and center point coordinates of each grid; constructing the grid section with the center point coordinates and the average normal vector; calculating the intersection of the grid section and the grid boundary edge; using the intersection as the vertex of the initial polygonal surface, and using the surface formed by the intersection as the grid polygonal surface, the vertices and the grid polygonal surface constitute the three-dimensional grid model of the predicted point cloud.
[0098] Specifically, the predicted point cloud space is divided into grids according to the set grid resolution; the grid is traversed with the central grid of the 27-grid algorithm, and the average normal vector and the center point coordinates of all points in the central grid are calculated; the point normal section plane of the central grid is constructed with the center point coordinates and the average normal vector; the intersection of the section plane and the boundary edge of the central grid is calculated; a point outside the central grid edge in the δ neighborhood of each intersection point is taken as a vertex of the central grid polygonal face, and the plane enclosed by the vertex lines generated by the central grid is taken as a polygonal face of the grid model, as described below for details.
[0099] If the predicted point cloud is large in scale and the computational memory consumption is large, the intermediate data can be temporarily stored in the disk based on persistent distributed computing, and the point cloud spatial grid generated by S201 and S202 can be traversed to calculate the number of points num in the current central grid.
[0100] If the number of points num>2, it means that there are more than 3 points of curved surface or polygonal surface in the current center space area grid, and at least one triangle surface can be used to represent the surface of 3D object. Calculate the average normal vector, center point coordinates, and average color of the current center grid. Use the average normal vector and the point normal plane equation of the center point to approximate the surface of 3D object in the center grid, calculate the edge intersection points of the approximate plane and the current center grid, if the computing memory is sufficient, add the edge intersection points to the point group of the current center grid, if the computing memory is insufficient, use the edge intersection points to replace the points in the current center grid, so that the points in the current center grid are less than 6, saving memory and CPU computing time.
[0101] Furthermore, when calculating the edge intersection, a random additive value δ>0 is set, such as 1.0e-8, and the edge intersection is extended to the outside of the current center grid where the section is located, such as Figure 7 As shown, the edge intersection A is extended to A', and A' is used to replace A, and D' is used to replace D. The advantage of this is to reduce the probability of the same intersection of several adjacent spatial area meshes on the edge, and avoid errors when importing files in some mesh software. A'D' replaces AD, so that the adjacent spatial area mesh fitting section uses AD as the intersection line, and automatically produces a stitching effect when displayed. The subsequent fitting step can also merge the points in the δ neighborhood of point A into one point as required to save memory.
[0102] If the number of points num=2, it means that there are two straight lines in the current center grid. The points in the center grid have no surface expression, but the average normal vector and the center point can still be calculated. The δ neighborhood intersection of the current center grid edge is calculated using this point normal plane, and the actual extended intersection is calculated according to the method in the above steps.
[0103] If the number of points num=1, it means that there is only one point in the current grid, and the points in the grid have no surface expression. However, the normal vector of the point and the point normal plane of the point coordinates can still be used to calculate the δ neighborhood intersection of the current central grid edge, and the actual extended intersection point can be calculated according to the method in the above steps.
[0104] Split the polygonal cross section into triangular face groups and store them in the grid file. Traverse the grid, calculate the number of points in the grid, and calculate the cumulative number of points in the current center grid noff according to the order of grid storage. Let the point number in the current center grid be pid. When it only contains the intersection of the δ neighborhood, pid∈[0,6), then the vertex number of the point in the grid file is vid=noff+pid. Figure 8 As shown in the schematic diagram of the triangle face of the mesh section plane, the triangle face of the current center grid is stored in the mesh file in the order of OAB, OBC, OCD and ODA. With the vertex number as the row number, the mesh number is stored in the vertex mesh number file in the following format: "mesh x-axis number, mesh y-axis number, mesh z-axis number". Each time a row of mesh numbers is read, it is the mesh number xyz triplet of the vertex in that row.
[0105] S204, determining whether there are gaps in the mesh model of the predicted point cloud. If there are gaps in the mesh model of the predicted point cloud, the mesh model of the predicted point cloud is repaired to fill the gaps, and deduplication, merging, smoothing, filtering and simplification are performed. Otherwise, the process ends.
[0106] Use the mesh tool to open the mesh file generated in the above steps and observe whether there are gaps or holes in the mesh. If there are gaps or holes, perform mesh gap repair.
[0107] Grid gap repair, the specific steps are as follows:
[0108] In one possible embodiment, all points in the predicted point cloud are projected onto a gap identification plane; the gap identification plane is divided into multiple plane grids according to a set resolution, the point cloud includes at least one projection point in the plane grid, and the observation gap has no projection point in the plane grid; sampling points are obtained in the plane grid, and the known edge grid closest to the sampling point is used as a neighbor grid; a patch input data set is generated based on the neighbor grid; a patch prediction point cloud is generated based on the patch input data set; a patch prediction grid model is generated based on the patch prediction point cloud; and the patch prediction grid model is sequentially deduplicated, vertex merged, estimated, smoothed, filtered, and simplified.
[0109] Step 301: Generate a repair gap identification plane.
[0110] Read the grid number file generated in the above step S201, traverse the input image sequence in S101, select a current image, read the transformation matrix from world coordinates to camera coordinates from the coordinate system transformation matrix, use the predicted point cloud model generated in step S105 as the input point cloud, transform all points to the camera coordinate system, the far end of the camera is the positive z-axis, the right side is the positive x-axis, and the bottom side is the positive y-axis. Project all camera coordinate system 3D points (x, y, z) to the xy plane of the camera, or set z to 0 to generate xy plane coordinates (x, y), divide the xy plane area of the projection point into plane grids according to the resolution of x, y, calculate the average color of the plane grid, the color value of the gap part seen by the camera end is 0, and save the plane projection image into a file for traversing the xy axis number of the gap grid.
[0111] Step 302: Generate a grid gap repair prediction data set.
[0112] a. Traverse and read the xy plane gap identification pixel grid generated in step 301, and use the gap color threshold to filter out the gap grid number, that is, the xy pixel coordinates of the image. Convert the color identification plane image into a grayscale image, set the gap grayscale threshold to Cth, then the pixels with a value less than Cth are considered to be the plane xy pixels of the gap or hole, and the real pixel RGB color value of the gap can be read from the selected image.
[0113] b. Transform the input point cloud to the current camera coordinate system, calculate the grid number of the input point cloud according to the method in S102, use the x- and y-axis grid number tuples (xp, yp) as keys, and the z-axis number and grid point list tuples (zp, V) as values to group and calculate all possible grids between the xy identification plane grid and the camera lens observation plane, find a grid with the smallest depth distance to the lens observation surface, and store it in a key-value container Cp with (xp, yp) as the key and (zp, V) as the value.
[0114] c. Use (xp0, yp0) of the plane gap grid as the key to extract the edge grid near the gap, such as Fig. 9 As shown, 11, 12, 21, 22, 31 are gap grids, and the edge grid near gap 11 is as follows Fig. 9 The 1st row grid, 4th row grid, 1st column grid, 4th column grid and grid 32 are shown; and the optional edge grids of each grid contained in the same gap are the same. Traverse the edge grids of the plane, and use the current edge grid number (xp, yp) as the key to extract the depth number zp and grid point list V in the key-value container Cp. This is the edge grid near the gap on the surface of the object that can be seen from the lens observation plane. Set a gap edge search level level, then use each gap grid as the current center point, search up to level times along the normal line in 26 directions, record the first grid found on each line (including the points in the point cloud), then jump out of the search in this line direction and continue the search in the next direction, transform the grid found this time to the world coordinate system and store it in the current edge grid list as the neighbor space area for the cube 27 grid calculation.
[0115] d. Sample k (x, y) coordinates within the coordinate interval of the current gap grid, then each sampling point (xs, ys) and the Top = T edge grid neighbor points closest to the center point of the current gap grid form a feature point vector. The format excluding the normal vector is as follows:
[0116] [[xs,ys,xs,ys,xs,ys],[x1,y1,z1,r1,g1,b1],...,[xT,yT,zT,rT,gT,bT]]
[0117] The format containing the normal vector is as follows:
[0118] [[xs,ys,xs,ys,xs,ys,xs,ys,0],[x1,y1,z1,r1,g1,b1,nx1,ny1,nz1],...,[xT,yT,zT,rT,gT,bT,nxT,nyT,nzT]]
[0119] e. Transform the coordinates of the point to the world coordinate system, set the label vector of the gap grid to the 0 vector, and combine the feature point vector and the label vector. If the color correction prediction of the gap edge grid is used, set the label vector to the z-axis average value of the nearest gap edge grid and the color value of the nearest edge grid. Expand the feature point vector and convert it into a csv string by separating it with ","; expand the label vector and convert it into a csv string by separating it with ","; connect the two into a line of feature label data with ":", and save them into the gap grid dataset file line by line.
[0120] Step 303: Repair the gap point cloud. According to step S104, the output prediction of the gap grid input data set is calculated to generate gap repair points, which are stored in the repair prediction point cloud file together with the predicted point cloud. When calculating the repair point, the depth is limited to the predicted gap grid. If the depth value exceeds the predicted gap grid, a random depth is assigned or the value is discarded.
[0121] Step 304: According to the color point cloud grid number generation method of step S102, a grid number file and a configuration file of the repair prediction point cloud are output.
[0122] Step 305: Generate a mesh model of the repair prediction point cloud according to step S203 and store it in a repair prediction mesh file.
[0123] Step 306: Optimize and repair mesh gaps.
[0124] a. Read the patch prediction grid file and the patch prediction grid number configuration file. Read the patch prediction grid vertex coordinate number file generated in step 305.
[0125] b. Calculate the vertices of the patch prediction mesh by grouping according to the mesh vertex coordinate numbers, and generate a key-value container Gp with the mesh number as the key and (mesh vertex number, point list) as the value. Read the triangle face set Ts of the patch prediction mesh. Generate an empty set of supplementary triangle faces T0.
[0126] c. Traverse the key value container Gp. Calculate the intersection points of the edge δ neighborhood of the central grid of the cube 27 grids. Assume that the grid side faces are numbered: 0 = top face (negative y axis), 1 = bottom face (positive y axis), 2 = left face (negative x axis), 3 = right face (positive x axis), 4 = front face (negative z axis), 5 = back face (positive z axis). Calculate the edge points of each side face and store them into the grid side edge point key value container Pl according to the side face number key. Traverse the faces of the container Pl. If the number of the previous points is greater than 2, merge the points on the face whose distance is less than the set value eps>0 (slightly greater than the random positive value δ in step 203). Determine again if the number of the previous points is greater than or equal to 2, then sequentially add two connecting points (when it is equal to 2, there is only one edge connecting) to the fan-shaped triangle face of the nearest two vertices of the adjacent face, for example Fig.10Connect the edges of the tangent planes of adjacent meshes. The top points A and D of the central mesh and the bottom points A' and D' of the adjacent meshes form a three-dimensional quadrilateral. At this time, add two triangles A'AD and A'DD' to the set T0. If the top surface of the central mesh only has point A, add a triangle A'AD' to the set T0. And so on for the bottom, left, right, front, and back connected triangles. If the top adjacent grid of the aforementioned central grid is taken as the central grid, the midpoint of the front points A' and B' of the grid is smaller than the midpoint coordinate of the x-axis edge of the central grid, connect A'B' and A"B" on the front left side, and add two triangular faces A'B'B" and A'B"A" to the set T0; if there is only point A' in front of the central grid, add a triangular face A'B"A" to the set T0. Similarly, connect the triangular faces of the side faces of other front faces. If there is no adjacent face on a front face of the central grid, there is a hole. Set a connection level L (empirical value ≤3, too large a value introduces noise) to calculate the interval of the outer mesh polygonal faces; query the edge points on the faces of the most L meshes in the straight line direction of the front face of the central grid one by one for connection. If a positive side face of the central grid is a hole and has no adjacent face, set a connection level L (empirical value ≤3, too large a value introduces noise) to calculate the interval of the outer mesh polygonal faces; query the edge points on the faces of the most L meshes in the lateral direction one by one along the direction of the side face for connection; for example Fig.10 If 115 does not exist, when calculating the side of the bottom surface of 005, if L = 1, the side surface of 215 is used for calculation. If all the grids within level L are empty, no triangles are added.
[0127] d. Filter the patch mesh generated by a to c. This performs vertex deduplication, triangle deduplication, merges vertices smaller than a set distance, and removes degenerate triangles.
[0128] e. Use the QEMD algorithm to simplify the mesh as needed, or use the Taubin algorithm or Laplacian algorithm to smooth the mesh, remove degenerate triangles to generate an optimized patch mesh, or use the Poisson or Ball-Pivoting algorithm to reconstruct the optimized patched mesh.
[0129] f. The vertex number file storing the optimized patched mesh has the same format as the vertex number file in step S203.
[0130] g. Set the hole integer tolerance Tor, and calculate the number of holes in the xy plane according to step 301. If it is less than Tor, stop the calculation, otherwise jump to step 301 to continue optimizing and repairing the gap grid.
[0131] The present invention uses SFM and MVS to extract a 3D point cloud model of a scene from an image sequence; uses a deep prediction network to enhance the point cloud expression and improve the resolution; uses a distributed differential cube 27-grid calculation to convert the point cloud into a grid model, further fills grid gaps in the 27-grid, filters, smoothes, simplifies and optimizes the grid, and achieves a grid model that is better than the MVS output.
[0132] The present invention provides a 3D reconstruction method, which selects the SceauxCastle dataset, uses the SFM and MVS software tools to obtain a preliminary 3D dense point cloud, numbers the point cloud according to the resolution accuracy, calculates the grid, generates training and prediction data sets using the cube 27 grid algorithm, trains a deep network model, and then predicts the enhanced point cloud, and finally generates a 3D mesh model by filling gaps and optimizing. The specific steps are as follows:
[0133] Step 1: Calculate the dense point cloud. Use colmap's feature extraction, matching and BA algorithm to calculate the sparse point cloud and camera pose parameters, perform image dedistortion and reconstruct the dense point cloud, and fuse the scene to generate a preliminary point cloud model.
[0134] Step 2: Set the xyz axis resolution to 1080, and number the point cloud 3D space in the preliminary point cloud model according to 1080 grids per axis to generate a grid number file. There are at most 1080*1080*1080 grids, but most of the scene cubes in the actual point cloud are empty, and the file only includes grids with points.
[0135] Step 3, select input features including [[current point x-axis coordinate, current point y-axis coordinate, current point x-axis coordinate, current point y-axis coordinate, current point x-axis coordinate, current point y-axis coordinate], [neighbor point x-axis coordinate, neighbor point y-axis coordinate, neighbor point z-axis coordinate, neighbor point r color component, neighbor point g color component, neighbor point b color component]*8], output features including [current point z-axis coordinate, current point r color component, current point g color component, current point b color component]. Set the number of neighbor vertices of the input feature to 8, and use the grid number file to generate training data. When calculating, the current point is in the center grid, and there are at most 26 adjacent grids around it. There is no grid outside the boundary of the point cloud or in the gap. If there is no adjacent grid, only the points of the center grid are calculated. If the center grid has only one point, it is regarded as a gap and the data of the grid is eliminated. If the number of feature points is insufficient, the input feature vector is padded with a 0 vector.
[0136] Step 4: Generate prediction data using the grid number file. Set the grid prediction threshold to 8, that is, when the number of points in the grid is less than 8, use the grid as the center grid to generate s input data for the depth prediction model; where s is the number of (x, y) samples in the center grid, and each (x, y) sample generates an input data for predicting depth z and color.
[0137] Step 5: Set the learning law to 0.0001, the number of cyclic training to 10, the amount of expansion to 0.3, and the gain to 10 / 3, and use the training data from step 3 to train the prediction network model. The prediction network model has 15 parallel splicing MLPs, 2 serial splicing Conv2d+tanhs, and 2 Linear+tanhs; the output MLP has 7 layers.
[0138] Step 6: Use the trained prediction network to predict the depth z and color of the generated xy sampling prediction points to generate a point cloud of the sampling points, and add the predicted sampling points to the dense point cloud generated in step 1 to generate a predicted point cloud.
[0139] Step 7: Calculate the grid number of the predicted point cloud.
[0140] Step 8: Because the amount of calculation and memory usage are relatively large, the predicted point cloud grid number is used to calculate the reconstructed grid model of the predicted point cloud according to the 27-square grid of the cube on the Spark distributed computing system and save it into the preliminary grid model file. The neighborhood of the edge intersection is set to δ = 1.0e-5.
[0141] Step 9: Generate a grayscale image of the point cloud grid projected on the xy plane in the depth direction of the camera coordinate system using the grid number, and set the grayscale color threshold of the gap grid to 10, that is, the grayscale value less than 10 is considered to be a gap grid.
[0142] Step 10, generate a gap patch prediction dataset. Use the xy plane to project the grayscale image to determine the xy axis number and coordinates of the gap, traverse the gap plane grid, and estimate the z axis position of the center grid of the gap. Read the xy number and coordinates of the nearest non-empty grid near the gap, and extract the z axis number and coordinates closest to the camera lens observation plane as the neighbor grid data. Generate s input data of the depth prediction model in the current gap center grid; where s is the number of (x, y) samples in the center grid, and each (x, y) sample generates a predicted depth z and color input data, and the point coordinates are transformed to the world coordinate system and stored in the gap patch dataset file.
[0143] Step 11: Use the gap repair data set to predict the points in the gap grid, where the number of sampling points s = 8. Generate a point cloud of gap repair points, and add the repair points to the predicted point cloud generated in step 6 to generate a gap repair predicted point cloud.
[0144] Step 12: Calculate the grid number of the gap repair prediction point cloud.
[0145] Step 13: Calculate the reconstructed grid model of the gap repair prediction point cloud using the cube 27 grid on the Spark distributed computing system and save it into the gap repair grid model file, wherein the neighborhood of the edge intersection δ is set to 1.0e-5.
[0146] Step 14: Optimize the gap repair mesh, wherein the search level is set to 1, the smoothing method is Laplacian smoothing, λ=0.1, perform filtering and simplification operations such as deduplication and merging adjacent vertices, and store the mesh into an optimized repair mesh model file.
[0147] Step 15: Calculate the number of gaps in the optimized patching grid according to the grayscale image projected on the xy plane. If it meets the design requirements, jump to the next step to stop the calculation; otherwise, jump to step 10 to continue patching the optimized grid.
[0148] Step 16: End the calculation and output the optimized gap repair mesh model file.
[0149] Fig.11 is a schematic diagram of the structure of a three-dimensional reconstruction device in an embodiment of the present disclosure, such as Fig.11 As shown, the three-dimensional reconstruction device 80 provided by the embodiment of the present disclosure mainly includes: a point cloud acquisition module 81, a grid division module 82, a data set creation module 8383, a sampling point information generation module 84, a predicted point cloud determination module 85, a point cloud generation grid model module 86 and an optimization grid model module 87.
[0150] The point cloud acquisition module 81 is used to obtain the initial input point cloud of the object to be reconstructed; the grid division module 82 is used to divide the input point cloud space into grids according to the set resolution; the data set creation module 83 is used to traverse the grid, use the continuous stereo 27 grid differential local space to create a training data set and a prediction data set, and parallel or distributedly calculate the global model; the sampling point information generation module 84 is used to train the prediction network model based on the created training data set, and use the prediction network model to predict the z-axis coordinate component and color information of the sampling point based on the predicted input feature vector of the xy plane sampling point; the predicted point cloud determination module 85 is used to generate a predicted repair point cloud based on the xy plane sampling point, the predicted z-axis coordinate component and the color information, and merge it with the input point cloud to obtain a predicted point cloud to improve the resolution; the point cloud generation grid model module 86 is used to convert the predicted point cloud into a grid model; the grid model optimization module 87 is used to determine whether the grid model performs gap repair, and perform smoothing filtering and simplification and other processing to optimize the resolution and calculation efficiency.
[0151] In this embodiment, a 3D reconstruction device is provided, which is used to perform the following process: obtaining the initial input point cloud of the object to be reconstructed; dividing the input point cloud space into grids according to the set resolution; traversing the grid, using the continuous stereoscopic 27-grid differential local space to create a training data set and a prediction data set, and computing the global model in parallel or distributed manner; training the prediction network model based on the created training data set, and using the prediction network model to predict the z-axis coordinate component and color information of the sampling point based on the predicted input feature vector of the xy plane sampling point; generating a predicted patch point cloud based on the xy plane sampling point, the predicted z-axis coordinate component and the color information, and merging it with the input point cloud to obtain a predicted point cloud to improve the resolution; converting the predicted point cloud into a grid model, and performing smooth filtering and simplification processing to optimize the resolution and computing efficiency. Using parallel or distributed cluster computing, the computing time is shortened, and fast reconstruction can be achieved with point cloud and grid models.
[0152] In one possible implementation, the grid division module 82 is used to divide the coordinate axis interval of the input point cloud space into equal or unequal parts according to a set resolution to obtain a grid; assign a unique grid number to each of the grids; use the grid number as a key and the point data as a value to form a key-value tuple; store the key-value tuple into a grid number file with one point per line, and each line of data in the grid number file includes the parameters of a point, and the point parameters include the following: point coordinates, point color, normal vector and the grid number where the point is located.
[0153] In one possible implementation, the data set creation module 83 is used to use the central grid of the three-dimensional 27-grid as the current grid when traversing the grid; obtain the neighbor grids of the current grid and the points therein; use each point in the central grid of the 27-grid as a feature label point and other points in the 27-grid as a neighbor point group to create a training data set; construct a training input feature vector and a training label vector based on each of the feature label points and the ordered neighbor point group; determine whether the number of points in the current calculation grid is less than a preset value; if the number of points in the current calculation grid is less than the preset value, use the current calculation grid as a prediction patch grid, and sample the xy plane coordinates of the prediction points; select N plane input sampling points from the prediction patch grid, N is at least the difference between the preset value and the number of points in the prediction patch grid, and generate a neighbor point group of the sampling point for each non-sampling point within the 27-grid outside the sampling point of the current calculation grid to create a prediction data set; construct a prediction input feature vector based on each of the sampling points and the ordered neighbor point group.
[0154] In a possible implementation, the sampling point information generation module 84 is used to feed the predicted input feature vector of the sampling point into a pre-trained prediction network model, so that the prediction network model outputs the z-axis coordinate component and color information of the sampling point, or outputs the z-axis coordinate component, color information and normal vector of the sampling point; the prediction network model is trained by a training data set, and the training data set includes a training input feature vector and a training label vector constructed from an initial input point cloud; the input feature vector is determined by the two-dimensional plane xy coordinate of the current calculation point in the central grid and the three-dimensional coordinate vector of the ordered neighbor point group; the label vector is determined by the z-axis coordinate component, color information and normal vector of the calculation point in the central grid, or the label vector is determined by the z-axis coordinate component and color information of the calculation point in the central grid.
[0155] In one possible implementation, the predicted point cloud determination module 85 is used to obtain an enhanced predicted point cloud; generate a predicted patch point cloud using the xy axis coordinate components of the plane sampling points, the predicted z axis coordinate components and the color values, and merge them with the input point cloud to generate a predicted point cloud.
[0156] In one possible implementation, a point cloud generates a mesh model module 86, which is used to convert the predicted input point cloud into a mesh model; set the mesh resolution to divide the predicted input point cloud space into meshes; traverse the meshes to calculate the average normal vector and center point coordinates of each mesh; construct the mesh section with the center point coordinates and the average normal vector; calculate the intersection of the mesh section and the mesh boundary edge; use the intersection as the polygonal face vertices of the mesh model, and use the face formed by the intersection as the polygonal face of the mesh model, and the vertices and polygonal faces are stored in the mesh model file of the predicted point cloud.
[0157] In one possible implementation, the mesh model optimization module 87 is used to perform gap repair of the point cloud model and the mesh model; project all points in the input point cloud model to the gap identification plane; divide the gap identification plane into a plane grid according to the spatial resolution, the point cloud includes at least one projection point in the plane grid, and the gap has no projection point in the plane grid; obtain sampling points in the plane grid, and use the known edge grid closest to the sampling point as a neighbor grid; generate a patch input data set based on the neighbor grid; generate a patch prediction point cloud based on the patch input data set; generate a patch prediction mesh model based on the patch prediction point cloud; and perform deduplication, vertex merging, estimation, smoothing, filtering and simplification on the patch prediction mesh model in sequence.
[0158] The three-dimensional reconstruction device provided in the embodiment of the present disclosure can execute the steps executed in the three-dimensional reconstruction method provided in the embodiment of the method of the present disclosure, and the execution steps and beneficial effects are no longer repeated here.
[0159] Fig.12 Schematic diagram of a computing facility in an embodiment of the present disclosure, the computing facility may be one or more three-dimensional reconstruction facilities. Fig.12 , showing a schematic diagram of the structure of a computing facility 900 suitable for implementing the embodiments of the present disclosure.
[0160] like Fig.12 As shown, the computing facility 900 may include computing nodes (node 1 to node n) 920, which can access node data and storage 930 (storage 1 to storage m) device data through a high-speed interconnection device 910 such as a gigabit network. The facility is a distributed cluster consisting of n computing nodes and m storage nodes, and each node can execute relevant programs to implement the three-dimensional reconstruction method of the embodiment described in the present disclosure. Each storage node can store input, intermediate data and output data.
[0161] Typically, the following devices can be connected to the storage node to obtain input video and image sequences and store output computational results: including, for example, cameras, aerial drones, mobile phones, computers, other computing clusters, and cloud storage devices.
[0162] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart, thereby implementing the three-dimensional reconstruction method as described above. When the computer program is executed by the distributed cluster 900, the above functions defined in the method of the embodiment of the present disclosure are executed.
[0163] The computer storage device may be included in the device; or it may exist independently and not be installed in the facility.
[0164] In a possible implementation of the present disclosure, the above-mentioned distributed cluster carries one or more programs. When the above-mentioned one or more programs are executed by a node device, the node device can implement all or part of the three-dimensional reconstruction method described in any of the above-mentioned embodiments.
[0165] In a possible implementation manner of the present disclosure, when the above one or more programs are executed by a node device, the node device may also execute other steps described in the above embodiment.
[0166] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including but not limited to C / C++, Java, Python, and Scala, etc.; computing and storage cluster frameworks include but are not limited to Spark, Flink, Ray, Hadoop, and Milvus, etc. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote server or computing cluster. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0167] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a portion of a code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, patching point cloud gaps and patching grid model gaps, patching grid model gaps itself includes patching point cloud gaps, and the execution process is parallel. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified functions or operations, or can be implemented with a combination of dedicated hardware and computer instructions.
[0168] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.
[0169] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0170] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.
[0171] In addition, although each operation is described in a specific order, this should not be required to perform these operations in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0172] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A three-dimensional reconstruction method, characterized in that: include: Obtaining the initial input point cloud of the object to be reconstructed; Divide the input point cloud space into grids according to the set resolution; Traverse the grid, use the continuous stereo 27-grid differential local space to create training data sets and prediction data sets, and calculate the global model in parallel or distributed manner; Training a prediction network model based on the created training data set, and using the prediction network model to predict the z-axis coordinate component and color information of the sampling point based on the prediction input feature vector of the xy plane sampling point; Generate a predicted patch point cloud based on the xy plane sampling points, the predicted z-axis coordinate components and the color information, and merge it with the input point cloud to obtain a predicted point cloud to improve the resolution; The predicted point cloud is converted into a mesh model and smoothed, filtered and simplified to optimize resolution and computational efficiency.
2. The method according to claim 1, characterized in that The step of dividing the input point cloud space into grids according to the set resolution includes: Divide the coordinate axis interval of the input point cloud space into equal or unequal parts according to the set resolution to obtain a grid; assigning a unique grid number to each of said grids; The grid number is used as a key and the point data is used as a value to form a key-value tuple; The key-value tuple is stored in a grid number file with one point per line. Each line of data in the grid number file includes parameters of a point. The parameters of the point include the following: point coordinates, point color, normal vector and the grid number where the point is located.
3. The method according to claim 1, characterized in that The traversal of the grid, using the continuous stereoscopic 27-grid to differentiate the local space, creating a training data set and a prediction data set, and parallel or distributed computing of the global model include: Take the central grid of the 27-square grid as the current calculation grid; Obtaining neighboring grids of the current calculation grid and points within the neighboring grids; Each point in the central grid of the 27-square grid is used as a label point, and the other points in the 27-square grid are used as neighbor point groups to create a training data set; Constructing a training input feature vector and a training label vector based on each of the label points and the ordered neighbor point group; Determine whether the number of points in the current calculation grid is less than a preset value; If the number of points in the current calculation grid is less than the preset value, the current calculation grid is used as a prediction patching grid, and the xy plane coordinates of the prediction points need to be sampled; Selecting N planar input sampling points from the prediction patching grid, where N is at least the difference between the preset value and the number of points in the prediction patching grid, and generating a neighbor point group of the sampling point for each non-sampling point within the 27-square grid outside the sampling point of the current calculation grid to create a prediction data set; A prediction input feature vector is constructed based on each of the sampling points and the ordered neighbor point groups.
4. The method according to claim 1, characterized in that: Predicting and generating a z-axis coordinate component and color information of the sampling point based on the predicted input feature vector of the sampling point, or generating a z-axis coordinate component, color information and a normal vector of the sampling point, including: Feeding the predicted input feature vector of the sampling point into a pre-trained prediction network model, so that the prediction network model outputs the z-axis coordinate component and color information of the sampling point, or outputs the z-axis coordinate component, color information and normal vector of the sampling point; The prediction network model is trained by a training data set, and the training data set includes a training input feature vector and a training label vector constructed from an initial input point cloud; the input feature vector is determined by features such as the two-dimensional plane xy coordinates of the calculation point in the central grid and the three-dimensional coordinates of the neighboring points; the label vector is determined by the z-axis coordinate component, color information and normal vector of the calculation point in the central grid, or the label vector is determined by the z-axis coordinate component and color information of the calculation point in the central grid.
5. The method according to claim 1, characterized in that The predicted point cloud is converted into a mesh model and smoothed, filtered and simplified to optimize resolution and computational efficiency, including: Generating a grid model of the predicted point cloud using the predicted point cloud; Determining whether there are gaps in the grid model of the predicted point cloud; If there are gaps in the grid model of the predicted point cloud, the grid model of the predicted point cloud is repaired, merged, smoothed, filtered, and simplified.
6. The method according to claim 5, characterized in that Generating a grid model of the predicted point cloud using the predicted point cloud includes: Setting a grid resolution to divide the space where the predicted point cloud is located into spatial grids; Traverse the spatial grid and calculate the average normal vector and center point coordinates of each grid; Constructing the mesh section with the center point coordinates and the average normal vector; Calculating the intersection of the mesh cross section and the mesh boundary edge; The intersection points are used as vertices of a mesh polygonal surface, and the surface formed by the intersection points is used as the mesh polygonal surface. The mesh polygonal surface is included in the mesh model of the predicted point cloud.
7. The method according to claim 5, characterized in that The mesh model of the predicted point cloud is processed by gap filling, merging, smoothing, filtering, simplifying, etc., including: Projecting each point in the point cloud model onto a gap identification plane; Dividing the gap identification plane into plane grids according to the spatial resolution, the point cloud includes at least one projection point in the plane grid, and the gap has no projection point in the plane grid; Acquire a sampling point in the plane grid, and use a known edge grid closest to the sampling point as a neighboring grid; generating a patch input data set based on the neighbor grid; Generate a patch prediction point cloud based on the patch input data set; generating a repair prediction grid model based on the repair prediction point cloud; The repair prediction mesh model is sequentially subjected to deduplication, vertex merging, estimation, smoothing, filtering and simplification processing.
8. A three-dimensional reconstruction device, characterized in that: include: An initial point cloud acquisition module is used to obtain the motion structure recovery point cloud of the object to be reconstructed; A meshing module is used to divide the input point cloud space into grids according to the set resolution; The data set creation module is used to traverse the grid, create training data sets and prediction data sets using the continuous stereoscopic 27-grid differential local space, and calculate the global model in parallel or distributed manner; A sampling point information generation module, used to train a prediction network model based on the created training data set, and use the prediction network model to predict and generate z-axis coordinate components and color information of the sampling points based on the prediction input feature vectors of the sampling points in the xy plane; A predicted point cloud determination module is used to generate a predicted patch point cloud based on the xy plane sampling points, the predicted z-axis coordinate components and the color information, and merge it with the input point cloud to obtain a predicted point cloud to improve the resolution; The point cloud generation mesh model module converts the predicted point cloud into a mesh model using parallel or distributed computing, and performs smooth filtering and simplification processing to optimize resolution and computing efficiency.
9. An electronic device, characterized in that: The electronic device comprises: one or more processing machines; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processing machines, the one or more processing machines implement the three-dimensional reconstruction method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processing machine, the three-dimensional reconstruction method as described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Large-scale image point cloud parallel distributed gridding reconstruction method, system and device
CN110533778A
Collection resource allocation method and device based on multivariate linear combination, and storage medium
CN117786496A
Single building three-dimensional reconstruction method based on point cloud semantic segmentation and structure fitting
WO2024077812A1
Cited By
Robust three-dimensional reconstruction method based on weighted local curved surface approximation
CN120014204A
