A synthesis method and system of three-dimensional foot mesh data based on text guidance

Abstract

The present invention belongs to the field of three-dimensional data synthesis, and discloses a synthesis method and system of three-dimensional foot mesh data guided by text, the method comprising: designing rules based on measurement results, and combining the measurement results of the three-dimensional foot mesh to generate corresponding natural text data describing the three-dimensional foot mesh data; extracting text features from natural text data based on a pre-trained multimodal model; training to obtain a three-dimensional foot model encoder and a three-dimensional foot model decoder; using the three-dimensional foot model encoder to extract structural information from the three-dimensional foot mesh data; constructing a diffusion model based on artificial neural network technology, and training the diffusion model to obtain a three-dimensional generation module; using the pre-trained multimodal model, the three-dimensional generation module and the three-dimensional foot model decoder to generate the three-dimensional foot mesh data corresponding to the desired natural text data. The present invention realizes the synthesis of high-quality three-dimensional mesh data of human feet under text guidance, and reduces the difficulty of training.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional data synthesis, and in particular to a synthesis method and system of three-dimensional foot mesh data based on text guidance. Background Art

[0002] In the prior art, there are many implementation forms of foot model synthesis methods, for example, a linear foot model synthesis method based on PCA (principal component analysis): that is, when characterizing the human body, a 3D representation method of the human foot is constructed through a linear scheme. Since SUPR-FOOT (a part of the SUPR model, focusing on accurately simulating the biomechanics of the foot, the SUPR model is a comprehensive model that includes not only the body model but also extends to the hands, head and feet) synthesizes new 3D foot data in a linear space, the synthesized 3D foot data set lacks diversity and is difficult to characterize the characteristics of real-world human feet. In addition, SUPR-FOOT only uses more than 200 three-dimensional vertices to represent the foot model, which cannot express three-dimensional detail information. For example, using implicit methods to represent 3D foot models, such methods can obtain more realistic foot rendering images, but it is difficult to obtain good quality three-dimensional mesh data. In addition, there are technologies that use intelligent algorithms to synthesize three-dimensional data, but such technical solutions do not involve three-dimensional foot mesh data, and human feet have their own inherent unique characteristics. Therefore, how to provide a text-guided synthesis method and system for three-dimensional foot mesh data is a problem that needs to be solved urgently. Summary of the invention

[0003] The embodiment of the present invention provides a method and system for synthesizing three-dimensional foot mesh data based on text guidance, so as to solve the problem in the prior art that it is difficult to form a large-scale 3D foot model data set due to high acquisition cost and insufficient acquisition accuracy. The present invention uses a diffusion generation model that has been proven to be effective on 2D data and combines the inherent prior knowledge of 3D feet to disclose a method for synthesizing three-dimensional foot mesh data based on text guidance.

[0004] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not a general review, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0005] According to a first aspect of an embodiment of the present invention, a method for synthesizing three-dimensional foot mesh data based on text guidance is provided.

[0006] In one embodiment, a method for synthesizing three-dimensional foot mesh data based on text guidance includes:

[0007] Configure a 3D foot mesh dataset, design rules based on the measurement results, and generate corresponding natural text data describing the 3D foot mesh data in combination with the measurement results of the 3D foot mesh; extract text features from the natural text data based on a pre-trained multimodal model;

[0008] Using deep learning technology and three-dimensional foot mesh data, a three-dimensional foot model encoder and a three-dimensional foot model decoder are trained; the three-dimensional foot model encoder is used to extract structural information from the three-dimensional foot mesh data;

[0009] A diffusion model is constructed based on artificial neural network technology, and text features, time steps and random vectors are used as inputs, and structural information is used as output. The diffusion model is trained to obtain a three-dimensional generation module.

[0010] The pre-trained multimodal model, the 3D generation module and the 3D foot model decoder are used to generate the 3D foot mesh data corresponding to the expected natural text data.

[0011] In one embodiment, configuring a three-dimensional foot mesh data set, designing rules based on measurement results, and combining the measurement results of the three-dimensional foot mesh to generate corresponding natural text data describing the three-dimensional foot mesh data includes:

[0012] Collect existing 3D foot mesh data sets with the same topology; use the 3D plane in the 3D space to adjust the posture of the 3D foot mesh data;

[0013] Based on the known topological information, the three-dimensional positions of the key points of the three-dimensional foot mesh are obtained, and the three-dimensional positions of the key points include the position information of the toe point, the heel point, the ankle point, the foot height point, the foot width point and the arch point;

[0014] Based on the three-dimensional positions of the key points of the three-dimensional foot mesh, the biomechanical characteristics of the foot are calculated, and the biomechanical characteristics of the foot include the values ​​of foot length, foot width, foot height, arch and heel inversion and inversion; natural text data is generated according to the biomechanical characteristics of the foot.

[0015] In one embodiment, extracting text features from natural text data based on a pre-trained multimodal model includes:

[0016] Prepare pre-trained multimodal pre-trained neural network and corresponding model parameters;

[0017] The natural text data is input into the text encoder in the multimodal pre-trained neural network, and the text feature information of the natural text is output.

[0018] In one embodiment, using deep learning technology and three-dimensional foot mesh data, training a three-dimensional foot model encoder and a three-dimensional foot model decoder includes:

[0019] The three-dimensional foot mesh data is converted into point cloud by point cloud technology to obtain foot point cloud data;

[0020] Construct a 3D foot model encoder with a pointnet++ network structure and a 3D foot model decoder with a multi-layer MLP network structure;

[0021] Processing the foot point cloud data through a three-dimensional foot model encoder to obtain a point cloud feature vector, and reconstructing the foot point cloud data through a three-dimensional foot model decoder to obtain reconstructed data;

[0022] Calculate the absolute value of the difference between the point cloud coordinate values ​​of the foot point cloud data and the reconstructed data, and use the gradient descent algorithm to optimize the parameters of the 3D foot model encoder and the 3D foot model decoder;

[0023] The training process of the 3D foot model encoder and the 3D foot model decoder is repeated until the loss value meets the preset requirements.

[0024] In one embodiment, extracting structural information from three-dimensional foot mesh data using a three-dimensional foot model encoder includes:

[0025] Point cloud sampling: obtain foot point cloud data, randomly select initial points in the foot point cloud data, and combine the farthest point sampling algorithm to obtain a subset of the point cloud data;

[0026] Grouping: Each point in the subset is used as a sampling point, and the sampling points are divided into point cloud clusters based on the spherical structure. The number of point cloud clusters is the same as the number of sampling points in the subset.

[0027] Extract local features: Input the point cloud cluster into a neural network consisting of one-dimensional convolution, pooling, and activation functions, and output point cloud data consisting of features;

[0028] Repeat the steps of point cloud sampling, grouping, and extracting local features, and in the final network layer, use the maximum pooling method to convert the point cloud data composed of features into a one-dimensional vector.

[0029] In one embodiment, the three-dimensional foot mesh data is converted into a point cloud using a point cloud technology to obtain foot point cloud data including:

[0030] Using a mesh subdivision algorithm to process the three-dimensional foot mesh to obtain dense mesh data; removing the triangle face information in the dense mesh data to obtain dense vertex information;

[0031] The farthest point sampling algorithm is used to sample dense vertex information to obtain a fixed number of foot point cloud data.

[0032] In one embodiment, the data of the foot point cloud is reconstructed by a 3D foot model decoder to obtain the reconstructed data including:

[0033] Construct a 3D foot model decoder using a multi-layer perceptron, and use the Gaussian layer of the 3D foot model decoder to convert the point cloud feature vector into the mean and standard deviation describing the Gaussian distribution using a fully connected network structure;

[0034] The reparameterization technique is used to generate a latent vector that conforms to the mean and standard deviation distribution, and the cascaded reconstruction layer is used to reconstruct the data of the latent vector.

[0035] In one embodiment, a diffusion model is constructed based on artificial neural network technology, and text features, time steps and random vectors are used as inputs, and structural information is used as output. The diffusion model is trained to obtain a three-dimensional generation module, which includes:

[0036] Construct a diffusion model using multi-layer perceptron and design a noise prediction module in the diffusion model. The input of the noise prediction module includes text features, time steps and random vectors, and the output includes structural information.

[0037] Using the matched text features and structural information data pairs, the diffusion model is trained to obtain a three-dimensional generation module;

[0038] The random vector includes adding random Gaussian noise of different degrees twice to the structural information according to the time step, so as to obtain a first random vector and a second random vector as input.

[0039] In one embodiment, generating three-dimensional foot mesh data corresponding to expected natural text data using a pre-trained multimodal model, a three-dimensional generation module, and a three-dimensional foot model decoder includes:

[0040] Using natural text to describe the three-dimensional foot mesh information that is expected to be generated; inputting the three-dimensional foot mesh information that is expected to be generated into a pre-trained multimodal model to obtain the text features that are expected to be generated;

[0041] Through random sampling and combining with the expected text features, the three-dimensional generation module is input to obtain the expected structural information.

[0042] The desired structural information is input into the 3D foot model decoder to obtain the 3D foot mesh data that the natural text is expected to generate.

[0043] According to a second aspect of an embodiment of the present invention, a text-guided three-dimensional foot mesh data synthesis system is provided.

[0044] In one embodiment, the system comprises:

[0045] The foot text feature acquisition module is used to configure the three-dimensional foot mesh data set, design rules based on the measurement results, and generate corresponding natural text data describing the three-dimensional foot mesh data in combination with the measurement results of the three-dimensional foot mesh; extract text features from the natural text data based on the pre-trained multimodal model;

[0046] A foot structure information acquisition module is used to use deep learning technology and three-dimensional foot mesh data to train a three-dimensional foot model encoder and a three-dimensional foot model decoder; and use the three-dimensional foot model encoder to extract structural information from the three-dimensional foot mesh data;

[0047] The model training module is used to build a diffusion model based on artificial neural network technology, and takes text features, time steps and random vectors as inputs and structural information as outputs to train the diffusion model to obtain a three-dimensional generation module;

[0048] The three-dimensional foot mesh generation module is used to generate three-dimensional foot mesh data corresponding to the expected natural text data using a pre-trained multimodal model, a three-dimensional generation module and a three-dimensional foot model decoder.

[0049] According to a third aspect of an embodiment of the present invention, a computer device is provided.

[0050] In some embodiments, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0051] According to a fourth aspect of embodiments of the present invention, a computer-readable storage medium is provided.

[0052] In one embodiment, the computer-readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0053] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0054] (1) It has the advantages of being fast and convenient: the present invention uses an intelligent algorithm to automatically synthesize 3D models of different foot shapes in batches by calling a trained generation model.

[0055] (2) Advantage of consistent structural topology: The 3D foot model output by the present invention is convenient for unified processing in subsequent programs.

[0056] (3) It has the advantage of being more realistic: the 3D foot model synthesized by the present invention is not interpolated between existing models, but after fully exploring the rules inside the 3D foot model, a new sample that conforms to the data distribution of the 3D foot model is synthesized.

[0057] (4) The present invention realizes the synthesis of high-quality human foot three-dimensional mesh data under text guidance through the generation capability of the diffusion model. It also combines the VAE (Variational Autoencoder) algorithm to extract the structural information of the three-dimensional data and reduce the difficulty of training.

[0058] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0060] Figure 1 is a flow chart of a method for synthesizing three-dimensional foot mesh data based on text guidance according to an exemplary embodiment;

[0061] Figure 2 is a structural block diagram of a text-guided 3D foot mesh data synthesis system according to an exemplary embodiment;

[0062] Figure 3 is a structural schematic diagram of a computer device according to an exemplary embodiment;

[0063] Figure 4 is a flowchart of a training phase A according to an exemplary embodiment;

[0064] Figure 5 is a flowchart of a training phase B according to an exemplary embodiment;

[0065] Figure 6 is a flowchart of the reasoning phase according to an exemplary embodiment. DETAILED DESCRIPTION

[0066] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0067] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0068] As used herein, the term "plurality" means two or more than two, unless otherwise specified.

[0069] In this article, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, D / E means: D or E.

[0070] In this article, the term "and / or" is a description of the association relationship of objects, indicating that three relationships may exist. For example, D and / or E means: D or E, or, D and E.

[0071] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0072] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0073] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0074] Figure 1 An embodiment of the text-guided 3D foot mesh data synthesis method of the present invention is shown.

[0075] In this optional embodiment, the text-guided three-dimensional foot mesh data synthesis method includes:

[0076] Step S101, configuring a three-dimensional foot mesh data set, designing rules based on measurement results, and combining the measurement results of the three-dimensional foot mesh to generate corresponding natural text data describing the three-dimensional foot mesh data; extracting text features from the natural text data based on a pre-trained multimodal model;

[0077] Step S103, using deep learning technology and the three-dimensional foot mesh data, training a three-dimensional foot model encoder and a three-dimensional foot model decoder; using the three-dimensional foot model encoder to extract structural information from the three-dimensional foot mesh data;

[0078] Step S105, constructing a diffusion model based on artificial neural network technology, taking text features, time steps and random vectors as inputs and structural information as outputs, training the diffusion model to obtain a three-dimensional generation module;

[0079] Step S107, using the pre-trained multimodal model, the three-dimensional generation module and the three-dimensional foot model decoder to generate the three-dimensional foot mesh data corresponding to the expected natural text data.

[0080] Figure 2An embodiment of a text-guided 3D foot mesh data synthesis system of the present invention is shown.

[0081] In this optional embodiment, the text-guided 3D foot mesh data synthesis system comprises:

[0082] The foot text feature acquisition module 201 is used to configure a three-dimensional foot mesh data set, design rules based on measurement results, and generate corresponding natural text data describing the three-dimensional foot mesh data in combination with the measurement results of the three-dimensional foot mesh; extract text features from the natural text data based on a pre-trained multimodal model;

[0083] The foot structure information acquisition module 203 is used to train a 3D foot model encoder and a 3D foot model decoder using deep learning technology and 3D foot mesh data; and extract structure information from the 3D foot mesh data using the 3D foot model encoder;

[0084] The model training module 205 is used to construct a diffusion model based on artificial neural network technology, and take text features, time steps and random vectors as inputs and structural information as outputs to train the diffusion model to obtain a three-dimensional generation module;

[0085] The three-dimensional foot mesh generation module 207 is used to generate three-dimensional foot mesh data corresponding to the expected natural text data using the pre-trained multimodal model, the three-dimensional generation module and the three-dimensional foot model decoder.

[0086] In order to facilitate understanding of the above technical solutions of the present invention, the above technical solutions of the present invention are further described below from the perspective of architecture and principle, as follows:

[0087] 1. Preparation phase-A: Collect existing foot models (3D mesh datasets of the same topology) as training sets.

[0088] 2. Preparation phase-B: Measure the 3D foot mesh and design rules based on the measurement results to obtain the natural text description corresponding to the 3D mesh. So far, the matching data pair of 3D foot mesh-natural text description is obtained. The specific steps are as follows:

[0089] First, the 3D foot mesh is adjusted in 3D space so that: the sole is located in the XY plane, the instep faces the positive direction of the Z axis; the toes face the positive direction of the Y axis; the heel is tangent to the XZ plane; and the YZ plane divides the 3D foot mesh equally;

[0090] Secondly, based on the known topological information, the three-dimensional positions of the key points of the three-dimensional foot mesh are directly obtained: including the position information of the toe point, heel point, ankle point, foot height point, foot width point and arch point.

[0091] Then, calculate the foot length based on the toe point and the heel point, calculate the heel inversion and inversion based on the ankle point, calculate the foot height based on the foot high point, calculate the foot width based on the foot width point, and calculate the arch height based on the arch point.

[0092] Finally, natural text is generated based on the values ​​of foot length, foot width, foot height, arch, and heel inversion and inversion. For example, the length of this foot is 250 mm, the width is 60 mm, the arch height is 15 mm, and other descriptive information.

[0093] 3. Preparation phase-C: Based on the existing CLIP model (a multimodal pre-trained neural network), extract the text features of the natural text description. So far, the matching data pairs of 3D foot mesh-natural text description-text features are obtained. The specific steps are as follows:

[0094] First, download the pre-trained CLIP model parameters from the Internet;

[0095] Then, the script is constructed to directly input the natural text into the text encoder part of the CLIP model, and the text feature information of the natural text is obtained after calculation;

[0096] 4. Training phase-A: Based on the 3D foot mesh dataset, the VAE (variational autoencoder) algorithm is used to train the 3D foot model encoder part and the 3D foot model decoder part. The 3D foot mesh is processed based on the trained 3D foot model encoder to obtain the structural information corresponding to the 3D foot mesh. Thus, the matching data pair of 3D foot mesh-natural text description-text feature-structural information is obtained.

[0097] like Figure 4 As shown, the foot mesh data is input into a three-dimensional foot model encoder to obtain structural information corresponding to the foot mesh data, and the structural information is input into a three-dimensional foot model decoder to be converted into foot mesh data.

[0098] The details that need to be paid attention to at this stage are:

[0099] 4.1. Convert the 3D foot mesh into a point cloud (remove topological information) so that the encoder can process the 3D foot data.

[0100] First, mesh upsampling is performed: the Loop mesh subdivision algorithm is used to process the three-dimensional foot mesh to obtain denser mesh data; then the triangle face information in the dense mesh data is directly discarded, and only the vertex information in the dense mesh data is retained; finally, the farthest point sampling algorithm is used to sample the dense vertex information to obtain a fixed number of point cloud data, that is, the point cloud data representing the three-dimensional foot, completing the point cloud process.

[0101] 4.2. The 3D foot model encoder uses the pointnet++ network structure (a deep learning model specially designed to process disordered 3D point cloud data). This network can extract features from the foot point cloud data;

[0102] The feature extraction part of the pointnet++ network consists of four layers of the same structure, each of which consists of point cloud sampling, grouping, and extraction of local features. The details are as follows:

[0103] Point cloud sampling: input point cloud data (multiple points), randomly select an initial point, and then use the farthest point sampling algorithm to obtain a subset of the point cloud data (the number of points is less than the input point cloud data);

[0104] Grouping: Take each point in the above subset as a sampling point and do the following processing in a loop: For a certain sampling point, draw a sphere with a radius of R in space, and take the point cloud inside the sphere as the point cloud cluster of the sampling point. The number of point cloud clusters obtained is the number of sampling points in this step.

[0105] Extract local features: Input the above point cloud cluster into a neural network composed of one-dimensional convolution, pooling, and activation function to obtain the point cloud data composed of features output by the network layer.

[0106] Repeat the above three steps, that is, overlap multiple network layers. In the last network layer, the maximum pooling method is used to convert the point cloud features into a one-dimensional vector, which is the point cloud feature obtained using the pointnet++ network structure.

[0107] 4.3. The 3D foot model decoder uses a multi-layer MLP network structure (multi-layer perceptron) to directly map point cloud features into a fixed number of point cloud data;

[0108] The multi-layer MLP network structure consists of 1 Gaussian layer and 8 cascaded reconstruction layers;

[0109] The Gaussian layer uses a fully connected network structure to convert point cloud features into the mean and standard deviation describing the Gaussian distribution;

[0110] Use the reparameterization technique to generate latent vectors that conform to the above mean and standard deviation distribution;

[0111] 8 groups of cascaded reconstruction layers: Each group consists of a fully connected layer, a batch normalization layer, and an activation function, which directly transforms the above potential vector into a fixed number (implicit 3D topology) of 3D point clouds to represent the 3D foot.

[0112] 4.4. Calculate L1-Loss for the vertex data of the input mesh and the vertex data obtained by the decoder to train the 3D foot model encoder and 3D foot model decoder. The specific process is as follows:

[0113] The training data required for this process is the 3D foot data after point clouding. First, the 3D data of point clouding is passed through the encoder to obtain the point cloud feature vector, and then through the decoder to restore the point cloud data with the same number of vertices and the same point cloud order. Then, the L1-Loss is calculated for the point cloud coordinates in the input data and the reconstructed data, that is, the absolute value of the difference in coordinate values. Finally, the gradient descent algorithm is used to update and optimize the parameters in the encoder and decoder. Repeat the above process until the entire training process converges (the loss value is reduced to a very small part).

[0114] 5. Training phase-B: Based on the text features and structural information parts in the matching data pairs, the diffusion model-based 3D generation module is trained.

[0115] like Figure 5 As shown, the text features and random information are substituted into the 3D generation module based on the diffusion model to obtain the corresponding structural information.

[0116] For a single sample, both text features and structural information are one-dimensional feature vectors. The training data at this time is paired text features and structural information. The specific process of the training generation module is as follows:

[0117] First, the noise prediction module in the diffusion model is designed. Its input is text features, time steps, random vectors (with the same dimension as the structural information), and its output is structural information. The network structure of the noise prediction module is:

[0118] There are 18 basic layers. The output of the first layer is skip-connected to the input of the last layer, the second layer is skip-connected to the second-to-last layer, and so on, forming an overall structure similar to U-Net.

[0119] Each base layer is composed of multiple fully connected layers, batch normalization layers, activation layers, and residual blocks are added to them to form a resnet-like structure.

[0120] Second, specify that the total number of time steps is 1000.

[0121] Then, based on the matched text feature-structure information data pair, the diffusion model is trained: in one iteration cycle, a time step is randomly selected, and two different degrees of random Gaussian noise are added to the structure information according to the time step to obtain the random vector A and random vector B as input. The random vector A is input into the noise prediction module together with the time step and text feature, and the predicted noise is output. The random vector is subtracted from the noise to obtain the output vector of this generation module. After the loss calculation of the output vector and the random vector B, the optimization algorithm is used to reversely update the noise prediction module. Repeat the above iteration process until convergence.

[0122] Finally, after the noise prediction module is trained, forward inference is performed. The following process is repeated from 1 to 1000 according to the time step, and the structural information is gradually recovered from the noise according to the guidance of the text features: input the text feature, time step, and the random vector obtained in the previous step to generate a new random vector.

[0123] The details that need to be paid attention to at this stage are:

[0124] The input of the diffusion model is random sampling based on Gaussian distribution, text features, and time steps;

[0125] The network structure of the diffusion model no longer uses the U-Net network in the 2D diffusion model structure, but directly uses a multi-layer MLP structure to adapt the feature information.

[0126] 6. Reasoning stage: After the training of each module is completed, the modules are assembled to obtain the overall flow chart.

[0127] Use natural text to describe the 3D foot mesh information you want to generate;

[0128] Use the CLIP model as a text encoder to obtain text features corresponding to natural text;

[0129] After random sampling from the Gaussian distribution, combined with text features, the data is input into a 3D generation module based on a diffusion model to obtain structural information that characterizes the three-dimensional grid structure.

[0130] The structural information is input into the 3D foot model decoder to obtain the 3D foot mesh data corresponding to the natural text.

[0131] like Figure 6 As shown in the figure, natural text is converted into text features through a text encoder, and then combined with random information, a 3D generation module based on a diffusion model is used to generate structural information, and finally a foot mesh is generated through a three-dimensional foot model decoder.

[0132] The present invention uses VAE to extract the structural information of three-dimensional mesh data as a representation method of three-dimensional data, and can also be applied to the representation of three-dimensional data of human bodies, people, single objects, etc. The generation capability of the diffusion model is introduced to achieve high-quality synthesis of three-dimensional mesh data of human feet. This solution improves the diversity of three-dimensional mesh data of human feet and improves the quality of synthetic foot data. Use text-guided methods to generate foot mesh data of specific shapes, specific sizes, etc.

[0133] The training process of VAE is the process of training phase-A. (The relationship between data and feature vectors is analogous to the relationship between people and DNA. DNA can be extracted from a person, and the DNA contains information about the person. In theory, the person can be reconstructed from the DNA.) The VAE algorithm is to find the mapping relationship from data → features, and features → data.

[0134] The encoding part of the VAE algorithm: converts a data of any type (an image, a text, here is a 3D point cloud data) into a feature vector, which contains the information to restore this set of data.

[0135] The decoding part of the VAE algorithm: restore the original data from the feature vector. At the same time, the latent vector that conforms to the Gaussian distribution is hidden inside the decoder.

[0136] Therefore, for the present invention, the role of the VAE algorithm is to train an encoder to convert three-dimensional point cloud data into a feature vector, and train a decoder to recover the three-dimensional point cloud data from the feature vector.

[0137] In the present invention:

[0138] 3D model reconstruction: Obtain a 3D representation of a specific object in the real world.

[0139] 3D model synthesis: A 3D representation of an object that does not exist in the real world is randomly generated by an algorithm after it has discovered the inherent laws of the object.

[0140] Text-generated 3D: Input descriptive natural language text, and the intelligent algorithm / system will generate a 3D model based on the text content.

[0141] 3D mesh: Mesh data consists of 3D vertices and 3D faces.

[0142] 3D point cloud: Point cloud data consists of 3D vertices.

[0143] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, Figure 3In the embodiment, the computer device includes a processor, a system bus, a non-volatile storage medium, an internal memory and a network interface, and the non-volatile storage medium includes an operating system, a computer program and a database. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0144] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] In addition, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.

[0146] In addition, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0148] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for synthesizing three-dimensional foot mesh data based on text guidance, characterized in that: include: Configure a 3D foot mesh dataset, design rules based on the measurement results, and generate corresponding natural text data describing the 3D foot mesh data in combination with the measurement results of the 3D foot mesh; extract text features from the natural text data based on a pre-trained multimodal model; Using deep learning technology and three-dimensional foot mesh data, a three-dimensional foot model encoder and a three-dimensional foot model decoder are trained; the three-dimensional foot model encoder is used to extract structural information from the three-dimensional foot mesh data; A diffusion model is constructed based on artificial neural network technology, and text features, time steps and random vectors are used as inputs, and structural information is used as output. The diffusion model is trained to obtain a three-dimensional generation module. Generate 3D foot mesh data corresponding to the desired natural text data using the pre-trained multimodal model, 3D generation module and 3D foot model decoder; The method of generating three-dimensional foot mesh data corresponding to the expected natural text data by using the pre-trained multimodal model, the three-dimensional generation module and the three-dimensional foot model decoder includes: Using natural text to describe the three-dimensional foot mesh information that is expected to be generated; inputting the three-dimensional foot mesh information that is expected to be generated into a pre-trained multimodal model to obtain the text features that are expected to be generated; Through random sampling and combining with the expected text features, the three-dimensional generation module is input to obtain the expected structural information. The desired structural information is input into the 3D foot model decoder to obtain the 3D foot mesh data that the natural text is expected to generate.

2. The text-guided 3D foot mesh data synthesis method according to claim 1, characterized in that: The configuring of the three-dimensional foot mesh data set, designing rules based on the measurement results, and generating corresponding natural text data describing the three-dimensional foot mesh data in combination with the measurement results of the three-dimensional foot mesh include: Collect existing 3D foot mesh data sets with the same topology; use the 3D plane in the 3D space to adjust the posture of the 3D foot mesh data; Based on the known topological information, the three-dimensional positions of the key points of the three-dimensional foot mesh are obtained, and the three-dimensional positions of the key points include the position information of the toe point, the heel point, the ankle point, the foot height point, the foot width point and the arch point; Based on the three-dimensional positions of the key points of the three-dimensional foot mesh, the biomechanical characteristics of the foot are calculated, and the biomechanical characteristics of the foot include the values ​​of foot length, foot width, foot height, arch and heel inversion and inversion; natural text data is generated according to the biomechanical characteristics of the foot.

3. The text-guided 3D foot mesh data synthesis method according to claim 1, characterized in that: The method of extracting text features from natural text data based on a pre-trained multimodal model includes: Prepare pre-trained multimodal pre-trained neural network and corresponding model parameters; The natural text data is input into the text encoder in the multimodal pre-trained neural network, and the text feature information of the natural text is output.

4. The text-guided 3D foot mesh data synthesis method according to claim 1, characterized in that: The method of using deep learning technology and three-dimensional foot mesh data to train a three-dimensional foot model encoder and a three-dimensional foot model decoder includes: The three-dimensional foot mesh data is converted into point cloud by point cloud technology to obtain foot point cloud data; Construct a 3D foot model encoder with a pointnet++ network structure and a 3D foot model decoder with a multi-layer MLP network structure; Processing the foot point cloud data through a three-dimensional foot model encoder to obtain a point cloud feature vector, and reconstructing the foot point cloud data through a three-dimensional foot model decoder to obtain reconstructed data; Calculate the absolute value of the difference between the point cloud coordinate values ​​of the foot point cloud data and the reconstructed data, and use the gradient descent algorithm to optimize the parameters of the 3D foot model encoder and the 3D foot model decoder; The training process of the 3D foot model encoder and the 3D foot model decoder is repeated until the loss value meets the preset requirements.

5. The method for synthesizing three-dimensional foot mesh data based on text guidance according to claim 4, characterized in that: The extracting of structural information from the three-dimensional foot mesh data using the three-dimensional foot model encoder includes: Point cloud sampling: obtain foot point cloud data, randomly select initial points in the foot point cloud data, and combine the farthest point sampling algorithm to obtain a subset of the point cloud data; Grouping: Each point in the subset is used as a sampling point, and the sampling points are divided into point cloud clusters based on the spherical structure. The number of point cloud clusters is the same as the number of sampling points in the subset. Extract local features: Input the point cloud cluster into a neural network consisting of one-dimensional convolution, pooling, and activation functions, and output point cloud data consisting of features; Repeat the steps of point cloud sampling, grouping, and extracting local features, and in the final network layer, use the maximum pooling method to convert the point cloud data composed of features into a one-dimensional vector.

6. The text-guided 3D foot mesh data synthesis method according to claim 4, characterized in that: The point cloud conversion process is performed on the three-dimensional foot mesh data by the point cloud technology to obtain the foot point cloud data, including: Using a mesh subdivision algorithm to process the three-dimensional foot mesh to obtain dense mesh data; removing the triangle face information in the dense mesh data to obtain dense vertex information; The farthest point sampling algorithm is used to sample dense vertex information to obtain a fixed number of foot point cloud data.

7. The method for synthesizing three-dimensional foot mesh data based on text guidance according to claim 4, characterized in that: The foot point cloud data is reconstructed through the 3D foot model decoder, and the reconstructed data includes: Construct a 3D foot model decoder using a multi-layer perceptron, and use the Gaussian layer of the 3D foot model decoder to convert the point cloud feature vector into the mean and standard deviation describing the Gaussian distribution using a fully connected network structure; The reparameterization technique is used to generate a latent vector that conforms to the mean and standard deviation distribution, and the cascaded reconstruction layer is used to reconstruct the data of the latent vector.

8. The text-guided 3D foot mesh data synthesis method according to claim 1, characterized in that: The diffusion model is constructed based on artificial neural network technology, and text features, time steps and random vectors are used as inputs, and structural information is used as output. The three-dimensional generation module obtained by training the diffusion model includes: Construct a diffusion model using multi-layer perceptron and design a noise prediction module in the diffusion model. The input of the noise prediction module includes text features, time steps and random vectors, and the output includes structural information. Using the matched text features and structural information data pairs, the diffusion model is trained to obtain a three-dimensional generation module; The random vector includes adding random Gaussian noise of different degrees twice to the structural information according to the time step, so as to obtain a first random vector and a second random vector as input.

9. A text-guided synthesis system for three-dimensional foot mesh data, characterized in that: include: The foot text feature acquisition module is used to configure the three-dimensional foot mesh data set, design rules based on the measurement results, and generate corresponding natural text data describing the three-dimensional foot mesh data in combination with the measurement results of the three-dimensional foot mesh; extract text features from the natural text data based on the pre-trained multimodal model; A foot structure information acquisition module is used to use deep learning technology and three-dimensional foot mesh data to train a three-dimensional foot model encoder and a three-dimensional foot model decoder; and use the three-dimensional foot model encoder to extract structural information from the three-dimensional foot mesh data; The model training module is used to build a diffusion model based on artificial neural network technology, and takes text features, time steps and random vectors as inputs and structural information as outputs to train the diffusion model to obtain a three-dimensional generation module; A three-dimensional foot mesh generation module is used to generate three-dimensional foot mesh data corresponding to expected natural text data using a pre-trained multimodal model, a three-dimensional generation module and a three-dimensional foot model decoder; wherein, the use of the pre-trained multimodal model, the three-dimensional generation module and the three-dimensional foot model decoder to generate three-dimensional foot mesh data corresponding to expected natural text data includes: using natural text to describe the three-dimensional foot mesh information expected to be generated; inputting the three-dimensional foot mesh information expected to be generated into the pre-trained multimodal model to obtain the text features expected to be generated; through random sampling, and in combination with the text features expected to be generated, inputting the three-dimensional generation module to obtain the expected generated structural information; inputting the expected generated structural information into the three-dimensional foot model decoder to obtain the three-dimensional foot mesh data expected to be generated by the natural text.

Images