Method and device for generating textured three-dimensional mesh model based on color point cloud

Abstract

The present invention provides a method for generating a textured three-dimensional mesh model based on a color point cloud, which is characterized by comprising the following steps: step S1, processing the color three-dimensional point cloud data using a preprocessing algorithm; step S2, inputting a three-dimensional geometric prior deep network for optimization; step S3, processing using a three-dimensional mesh model expansion algorithm to obtain a UV mapping relationship; step S4, projecting based on the UV mapping relationship to obtain a sparse point cloud coordinate two-dimensional image and a sparse point cloud color two-dimensional image; step S5, inputting a two-dimensional geometric prior deep network for optimization; step S6, inputting a two-dimensional texture prior deep network to obtain the texture of an initial three-dimensional mesh model; step S7, inputting the output of step S5 into a three-dimensional geometric prior deep network for optimization; step S8, judging whether the number of three-dimensional mesh iterations has been reached, repeating steps S3 to S8 when judging no, and entering step S9 when judging yes; step S9, combining the current three-dimensional mesh model with the texture to obtain a textured three-dimensional mesh model.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional computer vision, and in particular relates to a method and a device for generating a textured three-dimensional grid model based on a color point cloud. Background Art

[0002] Digital geometry processing technology based on 3D modeling has become the fourth wave of digital media technology, and is widely used in industrial design, digital entertainment and other fields. 3D mesh model is an important working method in 3D modeling, and how to generate high-quality 3D mesh models with textures has become a research hotspot in recent years.

[0003] The general generation process of a 3D mesh model is to first use a scanning device or a multi-view stereo matching algorithm to collect color 3D point cloud data of the object to be modeled, and then use some 3D mesh model generation algorithms based on the color 3D point cloud data to obtain the 3D mesh model of the object.

[0004] Traditional 3D mesh model generation algorithms can combine a variety of prior knowledge and point cloud data to generate 3D mesh models, such as surface reconstruction based on Voronoi diagrams [1] and Poisson surface reconstruction [2]. The above two methods have high requirements on the density of point cloud data, and the generated 3D mesh models are sensitive to the noise in the point cloud data and have poor generalization performance. Therefore, traditional 3D mesh model generation algorithms cannot effectively model sparse and noisy point cloud data, nor can they generate high-quality 3D mesh texture maps based on color 3D point cloud data.

[0005] In recent years, some scholars have tried to use deep neural networks to generate three-dimensional mesh models. For example, the paper [3] provides a method for generating three-dimensional mesh models from multiple images based on deformation inference. This method constructs an image feature extraction network that extracts geometric features and semantic features of two-dimensional images, and samples the potential moving positions of the mesh vertices of the three-dimensional mesh model based on the above features. The feature consistency of the image perception features is used for weighted scoring to deform the three-dimensional mesh model into a refined three-dimensional mesh model. This method needs to be trained on a large amount of point cloud data, and in practical applications, if the input data deviates greatly from the training data, the performance of this method will be unsatisfactory.

[0006] In addition, based on the 3D prior of the deep graph convolutional neural network itself, the paper [4] proposed a method to transform the convex hull into a 3D mesh model based on the 3D point cloud input. The 3D graph convolutional deep network in this method has a weak prior, which makes its performance overly dependent on the accuracy and density of the input 3D point cloud data.

[0007] Most of the above-mentioned methods for generating three-dimensional models based on graph convolutional neural networks generate three-dimensional mesh models from two-dimensional color images. They are overly dependent on the prior obtained by training on a large amount of point cloud data, and therefore usually have serious generalization performance problems and cannot use the actual input point cloud data to generate high-quality three-dimensional mesh models.

[0008] References

[0009] [1]AmentaN,BernM.,KamyysselisM.,et al.A newvoronoi-basedsurfacereconstruction algorithm[C] / / Proceedings of the 25th annual conference onComputer graphics and interactivetechniques.1998:415–421.

[0010] [2]KazhdanM,BolithoM,HoppeH,et al.Poisson surfacereconstruction[C] / / In Proceedings of the fourth Eurographicssymposium on Geometryprocessing.2006:volume 7.

[0011] [3]WenC, ZhangY, Li Z, and Fu Y.Pixel2mesh++:Multiview3d mesh generationvia deformation[C] / / IEEE International Conference on Computer Vision(ICCV).2019.

[0012] [4]Hanocka R, Metzer G, Giryes R, et al.Point2mesh: A self-prior fordeformable meshes[J] / / Special Interest Group on Computer Graphics andInteractive Techniques.2020:volume 39. Summary of the invention

[0013] To solve the above problems, the present invention provides a method and device for generating a textured 3D mesh model based on color 3D point cloud data with low density or noise. The present invention adopts the following technical solutions:

[0014] The present invention provides a method and device for generating a textured 3D mesh model based on a color point cloud, which is used for processing color 3D point cloud data to obtain a textured 3D mesh model so that modeling users can view and apply it, and is characterized in that it comprises the following steps: step S1, using a predetermined preprocessing algorithm to preprocess the color 3D point cloud data to obtain an initial 3D convex hull mesh model; step S2, building a 3D geometric prior deep network and inputting the initial 3D convex hull mesh model into the 3D geometric prior deep network for optimization to obtain a primary 3D mesh model as the current 3D mesh model; step S3, using a predetermined 3D mesh model expansion algorithm to perform a 2D expansion process on the current 3D mesh model to obtain a corresponding 3D to 2D UV mapping relationship; step S4, projecting the color 3D point cloud data based on the 3D to 2D UV mapping relationship to obtain a sparse point cloud coordinate 2D image. and a sparse point cloud color two-dimensional image; step S5, building a two-dimensional geometric prior deep network and inputting the sparse point cloud coordinate two-dimensional image and the current three-dimensional mesh model into the two-dimensional geometric prior deep network for optimization to obtain a second-generation three-dimensional mesh model; step S6, building a two-dimensional texture prior deep network and inputting the sparse point cloud color two-dimensional image and the current three-dimensional mesh model into the two-dimensional texture prior deep network to obtain the texture of the first-generation three-dimensional mesh model; step S7, inputting the second-generation three-dimensional mesh model into the three-dimensional geometric prior deep network for optimization to obtain a third-generation three-dimensional mesh model as a new current three-dimensional mesh model; step S8, judging whether the predetermined number of three-dimensional mesh iterations has been reached, repeating steps S3 to S8 if it is not, and entering step S9 if it is yes; step S9, combining the current three-dimensional mesh model and the texture finally obtained to obtain a textured three-dimensional mesh model and output it.

[0015] The method and device for generating a textured three-dimensional mesh model based on a color point cloud provided by the present invention may also have such technical features, wherein step S2 includes the following sub-steps: step S2-1, building a three-dimensional geometric prior deep network; step S2-2, using a predetermined three-dimensional surface processing algorithm to process the initial three-dimensional mesh model to obtain a uniformly distributed initial three-dimensional mesh model, and setting it as a uniform three-dimensional mesh model; step S2-3, inputting the uniform three-dimensional mesh model into the three-dimensional geometric prior deep network to obtain the three-dimensional mesh vertex coordinates; step S2-4, based on the three-dimensional mesh vertex coordinates and the color three-dimensional point cloud The data constructs a loss function of the three-dimensional geometric prior deep network and sets it as the three-dimensional geometric loss function, and based on the three-dimensional geometric loss function, trains and updates the three-dimensional geometric prior deep network to obtain a converged three-dimensional geometric prior deep network; step S2-5, determines whether the predetermined number of three-dimensional geometric iterations is reached, and repeats steps S2-2 to S2-5 when it is no, and enters step S2-6 when it is yes; step S2-6, updates the three-dimensional mesh vertex coordinates output by the last converged three-dimensional geometric prior deep network to the uniform three-dimensional mesh model to obtain an initial three-dimensional mesh model as the current three-dimensional mesh model.

[0016] The method and device for generating a textured three-dimensional mesh model based on a color point cloud provided by the present invention may also have such a technical feature, wherein the three-dimensional geometric loss function includes chamfer loss and edge length loss.

[0017] The method and device for generating a textured three-dimensional mesh model based on a color point cloud provided by the present invention may also have such a technical feature, wherein the three-dimensional geometric prior deep network is a three-dimensional graph convolutional neural network composed of an encoder-decoder structure in which nodes are first reduced and then increased, and the three-dimensional graph convolutional neural network is composed of a stack of convolution blocks consisting of a graph convolution layer, a graph pooling layer, a batch normalization and a linear rectification unit, and the convolution blocks are connected using a residual connection method.

[0018] The method and device for generating a textured three-dimensional mesh model based on a color point cloud provided by the present invention may also have such technical features, wherein step S5 includes the following sub-steps: step S5-1, building a two-dimensional geometric prior deep network; step S5-2, generating noise using a predetermined noise generation function; step S5-3, inputting the sparse point cloud coordinate two-dimensional image and the noise into the two-dimensional geometric prior deep network to obtain a dense point cloud coordinate two-dimensional image; step S5-4, constructing a loss function of the two-dimensional geometric prior deep network based on the sparse point cloud coordinate two-dimensional image and the dense point cloud coordinate two-dimensional image and setting it as a two-dimensional geometric loss function; step S5-5, iteratively update the two-dimensional geometric prior deep network based on the two-dimensional geometric loss function, until the predetermined number of two-dimensional geometric iterations is reached to obtain the final dense point cloud coordinate two-dimensional image, and set the final dense point cloud coordinate two-dimensional image as the target two-dimensional geometric image; step S5-6, map the target two-dimensional geometric image to the three-dimensional space to obtain the corresponding three-dimensional vertex coordinate values, and update the three-dimensional vertex coordinate values ​​to the initial three-dimensional mesh model to obtain the second-generation three-dimensional mesh model, wherein the two-dimensional geometric loss function is the average of the second-order distance losses between the same point values ​​in the sparse point cloud coordinate two-dimensional image and the dense point cloud coordinate two-dimensional image.

[0019] The method and device for generating a textured three-dimensional mesh model based on a color point cloud provided by the present invention may also have such technical features, wherein step S6 includes the following sub-steps: step S6-1, building a two-dimensional texture prior deep network; step S6-2, generating noise using a noise generation function; step S6-3, inputting a sparse point cloud color two-dimensional image and noise into a two-dimensional texture prior deep network to obtain a dense point cloud color two-dimensional image; step S6-4, constructing a loss function of a two-dimensional texture prior deep network based on the sparse point cloud color two-dimensional image and the dense point cloud color two-dimensional image and setting it as a two-dimensional texture loss function; step S6-5, iteratively updating the two-dimensional texture prior deep network based on the two-dimensional texture loss function until a predetermined number of two-dimensional texture iterations is reached to obtain a final dense point cloud color two-dimensional image, and using the final dense point cloud color two-dimensional image as a texture, wherein the two-dimensional texture loss function is the average of the second-order distance losses between the same point values ​​in the sparse point cloud color two-dimensional image and the dense point cloud color two-dimensional image.

[0020] The method and device for generating a textured three-dimensional mesh model based on a color point cloud provided by the present invention may also have such technical features, wherein the two-dimensional geometric prior deep network and the two-dimensional texture prior deep network are two-dimensional convolutional neural networks composed of an encoder-decoder structure in which the feature size is first reduced and then increased, and the two-dimensional convolutional neural network is composed of a stack of convolution blocks consisting of a convolution layer, an average pooling layer, a batch normalization and a linear rectification unit, and the convolution blocks are connected using a residual connection method.

[0021] The method and device for generating a textured three-dimensional grid model based on a color point cloud provided by the present invention may also have such a technical feature, wherein the preprocessing algorithm is a convex hull algorithm.

[0022] The method and device for generating a textured three-dimensional mesh model based on a color point cloud provided by the present invention may also have such technical features, including: a mesh model generation unit, which generates a textured three-dimensional mesh model using the method for generating a textured three-dimensional mesh model based on a color point cloud; and a model output unit, which is used to output the textured three-dimensional mesh model.

[0023] Function and Effect of the Invention

[0024] According to a method and device for generating a textured 3D mesh model based on a color point cloud of the present invention, the color 3D point cloud data is first preprocessed to obtain an initial 3D convex hull mesh model, and then the final textured 3D mesh model is continuously optimized through a 3D geometric prior deep network, a 2D geometric prior deep network and a 2D texture prior deep network. In this process, the normal vector direction information of each point in the color 3D point cloud data is not required, and training on pre-collected training data is not required. Only the prior information of the deep network itself and the constraints of the actual input color 3D point cloud data are relied upon as training supervision signals in the optimization process. Therefore, the method can still have a relatively stable performance in the absence of accurate point cloud data normal vectors, and does not require the pre-collection of a large amount of training data.

[0025] At the same time, because the 3D geometry prior deep network, the 2D geometry prior deep network and the 2D texture prior deep network are all deep neural networks, and the deep neural network itself has the characteristics of noise resistance and continuous smoothness, even if the input color 3D point cloud data is relatively sparse or noisy, the 3D mesh model generated by this method is also more stable.

[0026] This method can have relatively stable performance, good robustness and generalization when processing color three-dimensional point cloud data of different types, densities and scales. At the same time, the stable performance is not affected by the density or noise of the input. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A structural block diagram of a device for generating a textured three-dimensional mesh model based on a color point cloud according to an embodiment of the present invention;

[0028] Figure 2 A flowchart of a method for generating a textured three-dimensional mesh model based on a color point cloud according to an embodiment of the present invention;

[0029] Figure 3A schematic diagram of a process of generating a textured three-dimensional mesh model based on a color point cloud according to an embodiment of the present invention;

[0030] Figure 4 is a flowchart of the sub-steps of step S2 of an embodiment of the present invention;

[0031] Figure 5 A schematic diagram of the structure of a three-dimensional geometric prior deep network according to an embodiment of the present invention;

[0032] Figure 6 is a flowchart of the sub-steps of step S5 of an embodiment of the present invention;

[0033] Figure 7 A schematic diagram of the structure of a two-dimensional geometric prior deep network and a two-dimensional texture prior deep network according to an embodiment of the present invention; and

[0034] Figure 8 Flow chart of sub-steps of step S6 of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following is a detailed description of a method and device for generating a textured three-dimensional mesh model based on a color point cloud in combination with embodiments and drawings.

[0036] <Example>

[0037] The device for generating a textured three-dimensional mesh model based on a color point cloud in this embodiment can process the object to be modeled to obtain a high-quality textured three-dimensional mesh model, so that the modeling user can view it or directly apply it in the fields of industrial design and digital entertainment.

[0038] Figure 1 It is a structural block diagram of an apparatus for generating a textured three-dimensional mesh model based on a color point cloud according to an embodiment of the present invention.

[0039] like Figure 1 As shown, the device 10 for generating a textured three-dimensional mesh model based on a color point cloud includes a point cloud data acquisition unit 101 , a mesh model generation unit 102 , a model output unit 103 and a control unit 104 .

[0040] The point cloud data acquisition unit 101 can acquire color three-dimensional point cloud data of an object to be modeled.

[0041] The mesh model generating unit 102 generates a textured three-dimensional mesh model by using a method for generating a textured three-dimensional mesh model based on a color point cloud.

[0042] Among them, the method of generating a textured three-dimensional mesh model based on a color point cloud is described in detail below.

[0043] The model output unit 103 can output a textured three-dimensional mesh model for modeling users to view or directly apply in the fields of industrial design and digital entertainment.

[0044] The control unit 104 controls the above-mentioned units to realize corresponding functions.

[0045] Figure 2 A flowchart of a method for generating a textured three-dimensional mesh model based on a color point cloud according to an embodiment of the present invention; and

[0046] Figure 3 The present invention is a flowchart of a method for generating a textured three-dimensional mesh model based on a color point cloud according to an embodiment of the present invention.

[0047] like Figure 2 as well as Figure 3 As shown, the method for generating a textured three-dimensional mesh model based on a color point cloud comprises the following steps:

[0048] Step S1, using a predetermined preprocessing algorithm to preprocess the color three-dimensional point cloud data to obtain an initial three-dimensional convex hull mesh model.

[0049] The preprocessing algorithm is a convex hull algorithm, which is provided by the open source software MeshLab.

[0050] The color 3D point cloud data is uniformly sampled from the surface of a high-quality 3D grid model with texture, including categories such as animals, humans, and airplanes. The color 3D point cloud data collected from each 3D grid model surface has a total of 25,000 color points. In this embodiment, a dog is taken as an example (e.g. Figure 3 shown).

[0051] Step S2, building a three-dimensional geometric prior deep network and inputting the initial three-dimensional convex hull mesh model into the three-dimensional geometric prior deep network for optimization to obtain an initial three-dimensional mesh model as the current three-dimensional mesh model.

[0052] Figure 4 Flow chart of sub-steps of step S2 of an embodiment of the present invention.

[0053] like Figure 4 As shown, step S2 includes the following sub-steps:

[0054] Step S2-1, building a three-dimensional geometric prior deep network.

[0055] Among them, the three-dimensional geometric prior deep network is a three-dimensional graph convolutional neural network composed of an encoder-decoder structure in which the nodes are first reduced and then increased. The three-dimensional graph convolutional neural network is composed of a stack of convolution blocks consisting of a graph convolution layer, a graph pooling layer, batch normalization and a linear rectification function (ReLU for short), and the convolution blocks are jump-connected using residual connections.

[0056] Figure 5 Schematic diagram of the structure of a three-dimensional geometric prior deep network according to an embodiment of the present invention.

[0057] like Figure 5 As shown in Figure 1, the 3D geometric prior deep network is an encoder-decoder structure consisting of 6 graph convolution blocks. Each graph convolution block consists of three stacked graph convolution units. Each graph convolution unit includes a 1×5 graph convolution layer, batch standard normalization (i.e. Figure 5 A residual connection is added between each two adjacent graph convolution units. A skip connection is added between the graph convolution blocks corresponding to the encoder and decoder.

[0058] Step S2-2: Process the initial three-dimensional mesh model using a predetermined three-dimensional surface processing algorithm to obtain a uniformly distributed initial three-dimensional mesh model, and set it as a uniform three-dimensional mesh model.

[0059] Among them, the three-dimensional surface processing algorithm is a dense manifold surface generation algorithm. This algorithm evenly distributes the surface triangular mesh and controls the number of vertices to a specific value while allowing the colored three-dimensional point cloud data to maintain the surface shape. The specific value is 2000 in the first three-dimensional geometric iteration, twice the first in the second three-dimensional geometric iteration, twice the second in the third, and so on. The specific value in each three-dimensional geometric iteration is twice the previous one.

[0060] Step S2-3, input the uniform three-dimensional mesh model into the three-dimensional geometric prior deep network to obtain the three-dimensional mesh vertex coordinates.

[0061] Step S2-4, constructing the loss function of the 3D geometric prior deep network based on the 3D mesh vertex coordinates and the color 3D point cloud data and setting it as the 3D geometric loss function, and training and updating the 3D geometric prior deep network based on the 3D geometric loss function to obtain a converged 3D geometric prior deep network.

[0062] Among them, the 3D geometric loss function includes chamfer loss and edge length loss. The chamfer loss is calculated based on a number of random points on the triangle formed by the coordinates of the 3D mesh vertices and the color 3D point cloud data, and the edge length loss is the average value of the triangle edge length.

[0063] In this embodiment, there are 25,000 random points in total.

[0064] The 3D geometric prior deep network iteratively updates the weights in the 3D geometric prior deep network through the back propagation algorithm and the gradient descent algorithm according to the 3D geometric loss function, thereby obtaining a converged 3D geometric prior deep network.

[0065] The specific parameters in the 3D geometric prior deep network training process are set as follows: the gradient descent algorithm is the Adam optimization algorithm, the coefficient betas = (0.9, 0.999), the weight decay coefficient is 1e-5, the learning rate is set to 1e-4, and the number of 3D geometric prior deep network parameter training rounds is 2000.

[0066] Step S2-5, determine whether the predetermined number of three-dimensional geometric iterations has been reached, and if it is determined to be no, repeat steps S2-2 to S2-5, and if it is determined to be yes, proceed to step S2-6.

[0067] In this example, the number of 3D geometry iterations is 4.

[0068] Step S2-6, updating the three-dimensional mesh vertex coordinates output by the last converged three-dimensional geometric prior deep network into the uniform three-dimensional mesh model to obtain an initial three-dimensional mesh model as the current three-dimensional mesh model.

[0069] Among them, after the color 3D point cloud data is input into the final converged 3D geometric prior deep network, the final 3D mesh vertex coordinates will be output, and the 3D mesh vertex coordinates will be updated to the uniform 3D mesh model to obtain the initial 3D mesh model as the current 3D mesh model.

[0070] Based on the 3D geometry loss function, the 3D geometry prior deep network is iteratively updated until the predetermined number of 3D geometry iterations is reached to obtain the final 3D mesh vertex coordinates, and the 3D mesh vertex coordinates are updated to the uniform 3D mesh model to obtain the initial 3D mesh model as the current 3D mesh model.

[0071] Step S3, using a predetermined 3D mesh model expansion algorithm to perform 2D expansion processing on the current 3D mesh model to obtain a corresponding 3D to 2D UV mapping relationship.

[0072] Among them, the 3D mesh model unfolding algorithm is provided by the open source project OptCuts, and the 3D to 2D UV mapping relationship is a bidirectional mapping of any point on the surface of the previous 3D mesh model to the point on a 1024×1024 sized 2D plane.

[0073] During the two-dimensional unfolding process, the distortion critical value is taken as 4.3. When the distortion value is higher than 4.3, the three-dimensional mesh model unfolding algorithm continues to run; when the distortion value is lower than 4.3, the three-dimensional mesh model unfolding algorithm is stopped to obtain the UV mapping relationship from the current three-dimensional mesh model surface to the two-dimensional plane.

[0074] Step S4, projecting the colored three-dimensional point cloud data based on the three-dimensional to two-dimensional UV mapping relationship to obtain a sparse point cloud coordinate two-dimensional image and a sparse point cloud color two-dimensional image.

[0075] Specifically, based on the 3D to 2D UV mapping relationship, the coordinate values ​​of each point in the color 3D point cloud data are projected into a 2D space of fixed size, thereby obtaining a 1024×1024 sparse point cloud coordinate 2D image containing 25,000 valid values ​​(such as Figure 3 sparse graph in ).

[0076] Based on the 3D to 2D UV mapping relationship, the color value of each point in the color 3D point cloud data is projected into a fixed-size 2D space to obtain a 1024×1024 sparse point cloud color 2D image containing 25,000 valid values ​​(such as Figure 3 sparse graph in ).

[0077] Step S5, building a two-dimensional geometric prior deep network and inputting the sparse point cloud coordinate two-dimensional image and the current three-dimensional mesh model into the two-dimensional geometric prior deep network for optimization to obtain a second-generation three-dimensional mesh model.

[0078] Figure 6 Flow chart of sub-steps of step S5 of an embodiment of the present invention.

[0079] like Figure 6 As shown, step S5 includes the following sub-steps:

[0080] Step S5-1, building a two-dimensional geometric prior deep network.

[0081] Among them, the two-dimensional geometric prior deep network is a two-dimensional convolutional neural network composed of an encoder-decoder structure in which the feature size is first reduced and then increased. The two-dimensional convolutional neural network is composed of a stack of convolutional blocks consisting of convolutional layers, average pooling layers, batch normalization and linear rectification units, and the convolutional blocks are connected using residual connections.

[0082] Figure 7 It is a schematic diagram of the structure of a two-dimensional geometric prior deep network and a two-dimensional texture prior deep network according to an embodiment of the present invention.

[0083] like Figure 7 As shown in the figure, the two-dimensional geometric prior deep network is an encoder-decoder structure composed of 10 two-dimensional convolutional blocks. Each two-dimensional convolutional block consists of two stacked convolutional units, and each convolutional unit includes a convolutional layer, batch normalization, and ReLU. Figure 7 The five light-colored 2D convolutional blocks (i.e. Figure 7The rectangles numbered 1-1, 1-2, 1-3, 1-4, and 1-5 in the figure are encoders, and the five two-dimensional convolutional blocks with dark colors (i.e. Figure 7 The rectangles numbered 2-1, 2-2, 2-3, 2-4 and 2-5 in the figure are decoders. Each two-dimensional convolution block consists of two stacked convolution units, each of which includes a convolution layer, batch normalization and ReLU.

[0084] The convolution unit structure of the light-colored two-dimensional convolution block is different from that of the dark-colored two-dimensional convolution block. Specifically, the first convolution unit of the light-colored two-dimensional convolution block is a 3×3 convolution layer, with 72 channels, a step size of 1, a downsampling layer, a batch normalization layer, and a ReLU; the second convolution unit is a 3×3 convolution layer, with 72 channels, a step size of 1, a batch normalization layer, and a ReLU (e.g. Figure 7 The first convolution unit of the dark-colored two-dimensional convolution block is a batch normalization layer, a 3×3 convolution layer, 72 channels, a stride of 1, a batch normalization layer, and a ReLU; the second convolution unit is a 1×1 convolution layer, 72 channels, a stride of 1, a batch normalization layer, a ReLU, and an upsampling layer (as shown in Figure 7 2 in the figure).

[0085] The entire two-dimensional geometric prior deep network is an encoder-decoder structure, and a jump connection is added between the convolution blocks corresponding to the encoder and the decoder. Figure 7 The convolutional unit implementation in includes a 1×1 convolutional layer, 4 channels, a stride of 1, a batch normalization layer, and a ReLU.

[0086] Step S5-2: Generate noise using a predetermined noise generation function.

[0087] The noise generating function is a Gaussian function, and the random noise is generated by the Gaussian function.

[0088] Step S5-3, input the sparse point cloud coordinate two-dimensional image and noise into the two-dimensional geometric prior deep network to obtain a dense point cloud coordinate two-dimensional image (such as Figure 3 ).

[0089] Specifically, the noise is random noise with a mean of 0 and a variance of 0.1, and random noise with a mean of 0 and a variance of 0.02 is added for perturbation in each subsequent two-dimensional geometric iteration.

[0090] Step S5-4, constructing a loss function of a two-dimensional geometric prior deep network based on the sparse point cloud coordinate two-dimensional image and the dense point cloud coordinate two-dimensional image and setting it as a two-dimensional geometric loss function.

[0091] Among them, the two-dimensional geometric loss function is constructed by averaging the second-order distance loss between the points at the same position in the two-dimensional image of dense point cloud coordinates and the two-dimensional image of sparse point cloud coordinates.

[0092] Step S5-5, iteratively update the two-dimensional geometric prior deep network based on the two-dimensional geometric loss function until the predetermined number of two-dimensional geometric iterations is reached to obtain the final dense point cloud coordinate two-dimensional image, and set the final dense point cloud coordinate two-dimensional image as the target two-dimensional geometric image.

[0093] In this embodiment, the two-dimensional geometric prior deep network iteratively updates the weights in the two-dimensional geometric prior deep network according to the two-dimensional geometric loss function through the back propagation algorithm and the gradient descent algorithm, thereby obtaining a dense point cloud coordinate two-dimensional image, and setting the dense point cloud coordinate two-dimensional image as the target two-dimensional geometric image.

[0094] Specifically, the gradient descent algorithm is the Adam optimization algorithm, the coefficient betas = (0.9, 0.999), the weight decay coefficient is 1e-5, the learning rate is set to 1e-2, the number of training rounds of the two-dimensional geometric prior deep network parameters is 3000, and the number of two-dimensional geometric iterations is 1.

[0095] Step S5-6, mapping the target two-dimensional geometric image to three-dimensional space to obtain corresponding three-dimensional vertex coordinate values, and updating the three-dimensional vertex coordinate values ​​to the first-generation three-dimensional mesh model to obtain a second-generation three-dimensional mesh model.

[0096] Step S6, building a two-dimensional texture prior deep network and inputting the sparse point cloud color two-dimensional image and the current three-dimensional mesh model into the two-dimensional texture prior deep network to obtain the texture of the initial three-dimensional mesh model.

[0097] Figure 8 Flow chart of sub-steps of step S6 of an embodiment of the present invention

[0098] like Figure 8 As shown, step S6 includes the following sub-steps:

[0099] Step S6-1, building a two-dimensional texture prior deep network.

[0100] Among them, the two-dimensional texture prior deep network is a two-dimensional convolutional neural network composed of an encoder-decoder structure in which the feature size is first reduced and then increased. The two-dimensional convolutional neural network is composed of a stack of convolutional blocks consisting of convolutional layers, average pooling layers, batch normalization and linear rectification units, and the convolutional blocks are connected using residual connections.

[0101] like Figure 7As shown in the figure, the two-dimensional texture prior deep network has the same network structure as the two-dimensional geometric prior deep network, and also includes 10 two-dimensional convolutional blocks. The entire two-dimensional geometric prior deep network is an encoder-decoder structure, and jump connections are added between the convolutional blocks corresponding to the encoder and decoder.

[0102] Step S6-2: Generate noise using a noise generation function.

[0103] The noise generating function is a Gaussian function, and the random noise is generated by the Gaussian function.

[0104] Step S6-3, input the sparse point cloud color two-dimensional image and noise into the two-dimensional texture prior deep network to obtain a dense point cloud color two-dimensional image (such as Figure 3 ).

[0105] Specifically, the noise is random noise with a mean of 0 and a variance of 0.1, and random noise with a mean of 0 and a variance of 0.02 is added for perturbation in each subsequent two-dimensional geometric iteration.

[0106] Step S6-4, constructing a loss function of a two-dimensional texture prior deep network based on the sparse point cloud color two-dimensional image and the dense point cloud color two-dimensional image and setting it as a two-dimensional texture loss function.

[0107] Among them, the two-dimensional texture loss function is constructed by averaging the second-order distance loss between the same-position point values ​​of the sparse point cloud color two-dimensional image and the dense point cloud color two-dimensional image.

[0108] Step S6-5, iteratively update the two-dimensional texture prior deep network based on the two-dimensional texture loss function until a predetermined number of two-dimensional texture iterations is reached to obtain a final dense point cloud color two-dimensional image, and use the final dense point cloud color two-dimensional image as the texture.

[0109] In this embodiment, the two-dimensional texture prior deep network iteratively updates the weights in the two-dimensional texture prior deep network according to the two-dimensional texture loss function through the back propagation algorithm and the gradient descent algorithm, thereby obtaining a dense point cloud color two-dimensional image, and using the dense point cloud color two-dimensional image as texture.

[0110] Specifically, the gradient descent algorithm is the Adam optimization algorithm, the coefficient betas = (0.9, 0.999), the weight decay coefficient is 1e-5, the learning rate is set to 1e-2, the number of training rounds of the two-dimensional texture prior deep network parameters is 2000, and the number of two-dimensional texture iterations is 1.

[0111] Step S7, input the second-generation three-dimensional mesh model into the three-dimensional geometric prior deep network for optimization to obtain a third-generation three-dimensional mesh model as a new current three-dimensional mesh model.

[0112] The three generations of three-dimensional mesh models optimized in each step S7 are further optimized as new current three-dimensional mesh models in the next iteration.

[0113] Step S8, determining whether the predetermined number of three-dimensional mesh iterations has been reached, and if it is determined to be no, repeating steps S3 to S8, and if it is determined to be yes, proceeding to step S9.

[0114] Step S9, combining the current three-dimensional mesh model and the texture finally obtained to obtain a textured three-dimensional mesh model, and then outputting it to the modeling user for viewing and application.

[0115] In this embodiment, the number of three-dimensional grid iterations is 2.

[0116] In order to verify the effectiveness and accuracy of the method for generating a textured 3D mesh model based on a color point cloud according to an embodiment of the present invention, an evaluation is performed from three aspects: chamfering error, F-score, and texture perception quality loss. The smaller the chamfering error and texture perception quality loss, the better the output textured 3D mesh model, and the larger the F-score, the better the output textured 3D mesh model.

[0117] The performance of the method for generating a textured three-dimensional mesh model based on a color point cloud of the present invention is compared with that of a Poisson surface reconstruction method and a Point2Mesh method.

[0118] Specifically, the chamfer error, F-score and texture perception quality loss of the Poisson surface reconstruction method are 0.0422, 97.8 and 20.65 respectively; the chamfer error, F-score and texture perception quality loss of the Point2Mesh method are 0.0308, 98.5 and 20.39 respectively; the chamfer error, F-score and texture perception quality loss of the method of generating a textured three-dimensional mesh model based on a color point cloud of the present invention are 0.0287, 98.7 and 19.35 respectively.

[0119] As can be seen from the above, the chamfer error value and texture perception quality loss value of the method for generating a textured 3D mesh model based on a color point cloud of the present invention are both the smallest, and the F-score value is the largest. Therefore, the method for generating a textured 3D mesh model based on a color point cloud of the present invention is superior to the Poisson surface reconstruction method and the Point2Mesh method, and has a better performance.

[0120] Example Function and Effect

[0121] According to a method and device 10 for generating a textured three-dimensional mesh model based on a color point cloud provided by the above embodiment, the color three-dimensional point cloud data is first preprocessed to obtain an initial three-dimensional convex hull mesh model, and then the final textured three-dimensional mesh model is continuously optimized through a three-dimensional geometric prior deep network, a two-dimensional geometric prior deep network, and a two-dimensional texture prior deep network. In this process, no normal vector direction information of each point in the color three-dimensional point cloud data is required, and no training is required on pre-collected training data. Only the prior information of the deep network itself and the constraints of the actual input color three-dimensional point cloud data are relied upon as training supervision signals in the optimization process. Therefore, the method can still have a relatively stable performance in the absence of accurate point cloud data normal vectors, and does not require the pre-collection of a large amount of training data.

[0122] At the same time, because the 3D geometry prior deep network, the 2D geometry prior deep network and the 2D texture prior deep network are all deep neural networks, and the deep neural network itself has the characteristics of noise resistance and continuous smoothness, even if the input color 3D point cloud data is relatively sparse or noisy, the 3D mesh model generated by this method is also more stable.

[0123] This method can have relatively stable performance, good robustness and generalization when processing color three-dimensional point cloud data of different types, densities and scales. At the same time, the stable performance is not affected by the density or noise of the input.

[0124] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

[0125] In the above embodiment, the noise is random noise generated by a Gaussian function. In other embodiments of the present invention, the noise may also be generated by other functions.

[0126] In the above embodiment, the preprocessing algorithm in step S1 is a convex hull algorithm, which is provided by the open source software MeshLab. In other solutions of the present invention, other existing convex hull algorithms may also be used.

[0127] In the above embodiment, the three-dimensional mesh model expansion algorithm in step S3 uses the three-dimensional mesh model expansion algorithm provided by the open source project OptCuts. In other schemes of the present invention, other existing three-dimensional mesh model expansion algorithms may also be used.

Claims

1. A method for generating a textured three-dimensional mesh model based on a color point cloud, which is used to process the color three-dimensional point cloud data to obtain a textured three-dimensional mesh model so that the modeling user can view and apply it, characterized in that: The steps include: Step S1, preprocessing the color three-dimensional point cloud data using a predetermined preprocessing algorithm to obtain an initial three-dimensional convex hull mesh model; Step S2, building a three-dimensional geometric prior deep network and inputting the initial three-dimensional convex hull mesh model into the three-dimensional geometric prior deep network for optimization to obtain an initial three-dimensional mesh model as the current three-dimensional mesh model; Step S3, using a predetermined 3D mesh model expansion algorithm to perform 2D expansion processing on the current 3D mesh model to obtain a corresponding 3D to 2D UV mapping relationship; Step S4, projecting the colored three-dimensional point cloud data based on the three-dimensional to two-dimensional UV mapping relationship to obtain a sparse point cloud coordinate two-dimensional image and a sparse point cloud color two-dimensional image; Step S5, building a two-dimensional geometric prior deep network and inputting the sparse point cloud coordinate two-dimensional image and the current three-dimensional mesh model into the two-dimensional geometric prior deep network for optimization to obtain a second-generation three-dimensional mesh model; Step S6, building a two-dimensional texture prior deep network and inputting the sparse point cloud color two-dimensional image and noise into the two-dimensional texture prior deep network to obtain the texture of the initial three-dimensional mesh model; Step S7, inputting the second-generation three-dimensional mesh model into the three-dimensional geometric prior deep network for optimization to obtain a third-generation three-dimensional mesh model as a new current three-dimensional mesh model; Step S8, judging whether the predetermined number of three-dimensional grid iterations is reached, and if the judgment is no, repeating the steps S3 to S8, and if the judgment is yes, proceeding to step S9; Step S9, combining the current three-dimensional mesh model and the texture finally obtained to obtain the textured three-dimensional mesh model and output it; Wherein, the step S5 includes the following sub-steps: Step S5-1, building the two-dimensional geometric prior deep network, Step S5-2, generating noise using a predetermined noise generating function, Step S5-3, inputting the sparse point cloud coordinate two-dimensional image and the noise into the two-dimensional geometric prior deep network to obtain a dense point cloud coordinate two-dimensional image, Step S5-4, constructing a loss function of a two-dimensional geometric prior deep network based on the sparse point cloud coordinate two-dimensional image and the dense point cloud coordinate two-dimensional image and setting it as a two-dimensional geometric loss function, Step S5-5, iteratively updating the two-dimensional geometric prior deep network based on the two-dimensional geometric loss function until a predetermined number of two-dimensional geometric iterations is reached to obtain the final dense point cloud coordinate two-dimensional image, and setting the final dense point cloud coordinate two-dimensional image as the target two-dimensional geometric image, Step S5-6, mapping the target two-dimensional geometric image to a three-dimensional space to obtain corresponding three-dimensional vertex coordinate values, and updating the three-dimensional vertex coordinate values ​​to the first-generation three-dimensional mesh model to obtain the second-generation three-dimensional mesh model, The two-dimensional geometric loss function is the average of the second-order distance losses between the same point values ​​in the sparse point cloud coordinate two-dimensional image and the dense point cloud coordinate two-dimensional image; Wherein, the step S6 includes the following sub-steps: Step S6-1, building the two-dimensional texture prior depth network, Step S6-2, generating the noise using the noise generating function, Step S6-3, inputting the sparse point cloud color two-dimensional image and the noise into the two-dimensional texture prior deep network to obtain a dense point cloud color two-dimensional image, Step S6-4, constructing a loss function of a two-dimensional texture priori deep network based on the sparse point cloud color two-dimensional image and the dense point cloud color two-dimensional image and setting it as a two-dimensional texture loss function, Step S6-5, iteratively updating the two-dimensional texture prior deep network based on the two-dimensional texture loss function until a predetermined number of two-dimensional texture iterations is reached to obtain the final dense point cloud color two-dimensional image, and using the final dense point cloud color two-dimensional image as the texture, The two-dimensional texture loss function is the average of the second-order distance losses between the same point values ​​in the sparse point cloud color two-dimensional image and the dense point cloud color two-dimensional image.

2. The method for generating a textured three-dimensional mesh model based on a color point cloud according to claim 1, Features: Wherein, the step S2 includes the following sub-steps: Step S2-1, building the three-dimensional geometric prior deep network; Step S2-2, using a predetermined three-dimensional surface processing algorithm to process the initial three-dimensional grid model to obtain a uniformly distributed initial three-dimensional grid model, and setting it as a uniform three-dimensional grid model; Step S2-3, inputting the uniform three-dimensional mesh model into the three-dimensional geometric prior deep network to obtain the three-dimensional mesh vertex coordinates; Step S2-4, constructing a loss function of a three-dimensional geometric prior deep network based on the three-dimensional mesh vertex coordinates and the color three-dimensional point cloud data and setting it as a three-dimensional geometric loss function, and training and updating the three-dimensional geometric prior deep network based on the three-dimensional geometric loss function to obtain a converged three-dimensional geometric prior deep network; Step S2-5, judging whether the predetermined number of three-dimensional geometric iterations is reached, repeating the steps S2-2 to S2-5 if it is judged as no, and proceeding to step S2-6 if it is judged as yes; Step S2-6, updating the three-dimensional mesh vertex coordinates output by the final converged three-dimensional geometric prior deep network into the uniform three-dimensional mesh model to obtain the initial three-dimensional mesh model as the current three-dimensional mesh model.

3. The method for generating a textured three-dimensional mesh model based on a color point cloud according to claim 2, characterized in that: in, The three-dimensional geometric loss function includes chamfer loss and edge length loss.

4. The method for generating a textured three-dimensional mesh model based on a color point cloud according to claim 1, characterized in that: in, The three-dimensional geometric prior deep network is a three-dimensional graph convolutional neural network composed of an encoder-decoder structure in which nodes are first reduced and then increased. The three-dimensional graph convolutional neural network is composed of a stack of convolution blocks consisting of a graph convolution layer, a graph pooling layer, a batch normalization and a linear rectification unit, and the convolution blocks are connected using a residual connection.

5. The method for generating a textured three-dimensional mesh model based on a color point cloud according to claim 1, characterized in that: in, The two-dimensional geometric prior deep network and the two-dimensional texture prior deep network are two-dimensional convolutional neural networks composed of an encoder-decoder structure in which the feature size is first reduced and then increased. The two-dimensional convolutional neural network is composed of a stack of convolutional blocks consisting of convolutional layers, average pooling layers, batch normalization and linear rectification units, and the convolutional blocks are connected using residual connections.

6. The method for generating a textured three-dimensional mesh model based on a color point cloud according to claim 1, characterized in that: in, The preprocessing algorithm is a convex hull algorithm.

7. A device for generating a textured three-dimensional mesh model based on a color point cloud, characterized in that: include: A point cloud data acquisition unit, used for acquiring color three-dimensional point cloud data; A mesh model generating unit, which generates a textured three-dimensional mesh model using the method for generating a textured three-dimensional mesh model based on a color point cloud according to any one of claims 1 to 6; and The model output unit is used to output the textured three-dimensional mesh model.

Images