A progressive high-precision human foot shape reconstruction method

Abstract

The invention discloses a progressive high-precision human foot reconstruction method, the steps of which are as follows: constructing a visible shell of a foot from three foot contour images and corresponding camera internal and external parameters as a rough foot shape; establishing a human foot visual shell feature learning model to learn the visual shell shape features of the foot; establishing a human foot mesh feature learning model to learn the mesh shape features of the foot; using a deep learning method to establish a feature conversion model between the human foot visual shell features and the mesh features. With the foot contour image and the camera internal and external parameters as input, first construct a visible shell as a rough foot, extract the visible shell shape features, convert them into foot mesh shape features, and thereby reconstruct a high-precision three-dimensional foot. The invention reduces the equipment cost and labor cost of human foot model reconstruction, improves the efficiency of foot reconstruction, and can quickly and accurately reconstruct the three-dimensional shape of human feet using the progressive high-precision human foot reconstruction method.

Description

Technical Field

[0001] The invention belongs to the field of combining computer with three-dimensional foot shape reconstruction, and in particular relates to a progressive high-precision human foot shape reconstruction method. Background Art

[0002] With the advancement of science and technology and the improvement of living standards, using the power of technology to customize one's own exclusive products has gradually become a popular demand among users, and the personalized shoemaking industry has also flourished. In this industry, practitioners collect the user's foot shape 3D model and foot shape measurement parameters, and then adjust the shoe last size according to the foot shape 3D model and foot shape measurement parameters to design customized footwear styles that match the user's foot shape. This personalized shoemaking method usually requires high professional knowledge from practitioners and requires expensive 3D scanner equipment to obtain the foot shape model. Therefore, the entire shoemaking process is inefficient and expensive, making it difficult to promote.

[0003] Three-dimensional object reconstruction is a hot research issue in the field of computer vision. Three-dimensional object reconstruction refers to reconstructing a complete three-dimensional model of an object in an image from one or more images. This technology has broad application prospects in many scenarios such as human-computer interaction, autonomous driving, and game design. Traditional three-dimensional object reconstruction technology mainly restores the three-dimensional information of the object by finding the correspondence between feature points in multiple calibrated images and using multi-view stereo geometry. This method has strict requirements on the scene's lighting, the texture of the object, etc., and it is difficult to further promote it to actual application scenarios. And due to reasons such as object occlusion, this method usually cannot reconstruct the complete three-dimensional object shape.

[0004] Deep learning technology has developed rapidly in recent years and has made significant progress in a series of computer vision-related tasks. Among the 3D object reconstruction methods based on deep learning, they can be divided into two categories according to the image type: one is 3D object reconstruction based on RGB images. This type of method takes RGB images as input, extracts image features through deep convolutional neural networks and reconstructs the 3D shape of the object; the other is 3D object reconstruction based on depth images. This type of method takes depth images as input, represents them as voxel models or 3D point clouds, and uses deep convolutional neural networks to extract features and reconstruct the 3D object shape.

[0005] The visual hull is a 3D shape generated by the intersection of the projection cones of multiple images. This shape can be regarded as a rough 3D model of the object in the image, which integrates the image and the corresponding camera internal and external parameter information. Using the visual hull as input and reconstructing the 3D object based on it can better integrate the object information contained in multiple images and reconstruct the 3D object shape with higher accuracy. Summary of the invention

[0006] The object of the present invention is to provide a three-dimensional reconstruction method for a specific class of objects (i.e., human foot shape) in view of the insufficiency of the prior art that high-precision human foot shape cannot be reconstructed. Specifically, the present invention provides a progressive high-precision human foot shape reconstruction method, which constructs a visual shell as a rough three-dimensional shape of a human foot shape according to an image and corresponding camera parameters. Based on deep learning, a foot shape visual shell feature learning model, a foot shape mesh feature learning model, and a feature conversion model are established for the human foot shape visual shell and the human foot shape mesh shape. With the image and camera parameters as input, a visual shell is constructed, the visual shell features are extracted, converted into mesh features, and the foot shape three-dimensional shape is reconstructed. The present invention simplifies the complex process of the traditional foot shape reconstruction method, reduces the requirements for scene illumination and object texture during data acquisition, and uses the progressive human foot shape reconstruction method to quickly and accurately reconstruct a high-precision three-dimensional foot shape.

[0007] The object of the present invention is achieved through the following technical scheme: a progressive high-precision human foot shape reconstruction method, comprising the following steps:

[0008] (1) Construct a rough foot shape. On a public human foot 3D model dataset, specify the camera intrinsic parameters and use random camera extrinsic parameters to generate foot contours from different camera perspectives. Use three foot contours and the camera intrinsic and extrinsic parameters corresponding to the images to calculate the viewing cone space of each image in the world coordinate system by back projection. Intersect the viewing cone spaces of all images to obtain the visible shell of the foot, which is voxelized and represented as a 3D voxel model. The visible shell represented by the 3D voxel model is the rough foot shape. After obtaining the visible shell, perform post-processing such as coordinate alignment on the center point of the bounding box of the visible shell.

[0009] (2) Establish a learning model for the visual shell features of human foot shape. The learning model for the visual shell features of human foot shape is an auto encoder structure. It takes the visual shell of human foot shape constructed in step (1) as input, extracts the features of the visual shell of foot shape and reconstructs the visual shell of foot shape. The auto encoder consists of an encoder and a decoder. The encoder extracts features from the visual shell of foot shape represented by the three-dimensional voxel model. The decoder decodes the features of the visual shell of foot shape and reconstructs the visual shell of foot shape. The learning model for the visual shell features of human foot shape is trained by the self-supervised learning method, calculates the loss function between the input visual shell and the output visual shell, and optimizes the model parameters.

[0010] (3) Establish a human foot mesh feature learning model. The human foot mesh feature learning model is an auto encoder structure. It takes the human foot 3D mesh as input, extracts the foot mesh features and reconstructs the foot mesh model. The auto encoder consists of two parts: an encoder and a decoder. The encoder extracts features from the foot mesh. The decoder extracts the PCA shape parameters of the foot mesh model from the foot mesh features, and reconstructs the foot 3D mesh from the shape parameters using the inverse PCA method. The human foot mesh feature learning model is trained using a self-supervised learning method, calculates the loss function between the input foot 3D mesh and the output foot 3D mesh, and optimizes the model parameters.

[0011] (4) Establish a feature conversion model between the visual shell features and mesh features of the human foot shape. The feature conversion model is a multi-layer perception structure. The feature conversion model connects the foot shape visual shell feature learning model and the foot shape mesh feature learning model, takes the visual shell features extracted by the foot shape visual shell feature learning model as input, converts them into foot shape mesh features, and uses the decoder of the foot shape mesh feature learning model to reconstruct the foot shape three-dimensional mesh. The feature conversion model is trained through the supervised learning method, calculates the loss function between the reconstructed foot shape three-dimensional mesh and the real foot shape three-dimensional mesh, and optimizes the model parameters.

[0012] (5) When reconstructing the high-precision body foot shape, the encoder of the foot shape visual shell feature learning model, the decoder of the foot shape mesh feature learning model and the feature conversion model are used to reconstruct the foot shape. With three foot contour images and the corresponding camera internal and external parameters as input, the visual shell is first constructed as a rough foot shape, and then the encoder network of the visual shell feature learning model is used to extract the visual shell features of the foot shape. Subsequently, the feature conversion model is used to convert the visual shell features into the three-dimensional mesh features of the foot shape. Finally, the decoder network of the foot shape mesh feature learning model is used to reconstruct the high-precision three-dimensional foot shape and output a high-precision three-dimensional foot shape.

[0013] Furthermore, in the step (1), the camera external parameters are randomly generated in the front, left and right directions of the foot, respectively, to ensure that the generated foot contour map is evenly distributed in the three directions. When the foot visual shell is voxelized, the 320mm*160mm*160mm three-dimensional space around the origin in the world coordinate system is discretized according to the 2:1:1 ratio of the length, width and height of the foot, and the visual shell is represented as a 256*128*128 three-dimensional voxel model, wherein each voxel size is 1.25mm.

[0014] Furthermore, in the step (2), the human foot type visual shell feature learning model is an autoencoder structure, which consists of an encoder and a decoder. The encoder uses a 3D convolutional network and a fully connected network to extract features from the foot type visual shell. The decoder uses a fully connected network and a 3D deconvolutional network to decode the features extracted by the encoder and reconstruct the foot type visual shell shape. The loss function during model training is a binary cross-entropy loss (binary cross-entropy) between the input foot type visual shell and the reconstructed foot type visual shell voxel by voxel.

[0015] Furthermore, in the step (3), the human foot mesh feature learning model is an autoencoder structure, which consists of two parts: an encoder and a decoder. The encoder uses a graph convolutional network to extract features from the foot three-dimensional mesh. The decoder uses a fully connected network to predict the PCA shape parameters of the foot from the foot mesh features, and reconstructs the foot three-dimensional mesh from the PCA shape parameters through the inverse PCA method. The loss function during model training is the L2 distance between the input foot three-dimensional mesh and the reconstructed foot three-dimensional mesh vertex by vertex. The PCA shape parameters of the foot mesh model and the orthogonal vectors corresponding to each parameter are obtained by PCA decomposition of a public human foot three-dimensional model dataset.

[0016] Furthermore, in the step (4), the feature conversion model between the foot type visual shell feature and the foot type mesh feature is a multi-layer perceptron structure, and a bottleneck layer is used to reduce the feature dimension to reduce the overfitting problem of the model. The loss function during model training is the L2 distance between the reconstructed foot type three-dimensional mesh and the real foot type three-dimensional mesh vertex by vertex. The feature conversion model is jointly trained with the human foot type visual shell feature learning model and the human foot type mesh feature learning model.

[0017] The beneficial effects of the present invention are as follows: the progressive high-precision human foot shape reconstruction method can be used to collect foot photos through mobile devices such as cameras / mobile phones to perform high-precision foot shape reconstruction, which reduces the complex process of traditional foot shape reconstruction methods and reduces the requirements for scene lighting and object texture during data collection, and can quickly and accurately reconstruct high-precision three-dimensional foot shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart example of a specific implementation method of the present invention is provided;

[0019] Figure 2 This is an example of a detailed network structure diagram of the network model of the present invention. DETAILED DESCRIPTION

[0020] The specific embodiments of the present invention are as follows Figure 1 The progressive high-precision human foot shape reconstruction method provided by the present invention has the following specific implementation steps:

[0021] (1) First, collect multiple chessboard photos from different angles, and calibrate the camera internal parameters based on these photos to obtain the camera's internal parameters such as focal length, optical center, and distortion. Then, combine external calibration reference objects (A4 paper, chessboard calibration paper, scattered point calibration paper, etc.) to collect three photos of the user's feet in different directions (front, left, and right), and calculate the camera external parameters of each photo based on the external calibration reference objects. Finally, manually process the collected foot photos and extract the contour map of the foot area.

[0022] (2) Using the three foot region contour images and the camera internal and external parameters corresponding to each image as input, the visual shell of the foot shape is constructed and represented as a 3D voxel model as a rough model of the human foot shape. The bounding box of the visual shell is calculated, and the center alignment of the visual shell is performed according to the center position of the bounding box.

[0023] (3) Taking the processed visual shell as input, the features of the foot visual shell are extracted through the human foot visual shell feature learning model, and the visual shell features are converted into foot mesh features using the feature conversion model. Finally, the human foot mesh feature learning model predicts the foot PCA shape parameters from the foot mesh features, and reconstructs the three-dimensional mesh model of the human foot through the inverse PCA method.

[0024] Among them, the specific establishment method for the process of establishing the visual shell feature learning model in step (3) is as follows:

[0025] The visual shell feature learning model is based on the autoencoder structure, which consists of the encoder E vh and decoder D vh The visual shell feature learning model uses an encoder to learn visual shell features, and uses a decoder to reconstruct the foot shape visual shell from the learned visual shell features. The model is trained through self-supervised learning to better obtain the visual shell features of the human foot shape.

[0026] Encoder E vh 3D voxel model of the visible shell in the shape of a human foot vh As input. Encoder E vh It consists of 6 3D convolutional layers and 1 fully connected layer. Each 3D convolutional layer is followed by a maximum pooling layer and a leaky ReLU nonlinear activation unit. The fully connected layer is followed by a leaky ReLU nonlinear activation unit to extract the foot shape visual shell feature y vh .

[0027] Decoder D vh The shell features can be seen by the foot shape vh As input. Decoder D vhIt consists of 1 fully connected layer and 6 3D deconvolution layers. Except for the last 3D deconvolution layer, all other layers are followed by ReLU nonlinear activation units. The last 3D deconvolution layer is followed by a sigmoid activation unit, which outputs the probability value of each voxel of the reconstructed visual shell and outputs the reconstructed visual shell voxel model through a threshold function.

[0028] The loss function of the foot shape visual shell feature learning model l vh The calculation formula is:

[0029]

[0030] Where N is the number of voxels in the visible shell of the foot, and are the i-th voxel of the input visual hull and the reconstructed visual hull, respectively.

[0031] Among them, the specific establishment method for the foot type mesh feature learning model in step (3) is as follows:

[0032] The foot shape grid feature learning model is based on the autoencoder structure, which consists of the encoder E mesh and decoder D mesh The foot mesh feature learning model uses an encoder to learn the mesh features of the human foot, and uses a decoder to reconstruct the three-dimensional mesh model of the foot from the learned mesh features. The model is trained through self-supervised learning to better obtain the mesh features of the human foot.

[0033] Encoder E mesh 3D mesh model of human foot mesh As input. Encoder E mesh It consists of 4 layers of graph convolutional layers. Each graph convolutional layer is followed by 1 fast ReLU nonlinear activation unit. Finally, the maximum pooling layer is used to fuse the features of all vertices to extract the foot mesh feature y mesh .

[0034] Decoder D mesh Foot mesh feature y mesh As input. Decoder D mesh It consists of 5 layers of fully connected layers. Except for the last fully connected layer, all other layers are followed by ReLU nonlinear activation units. The last fully connected layer directly outputs the predicted foot PCA shape parameters and outputs the reconstructed foot 3D mesh model through the inverse PCA method.

[0035] The loss function of the foot mesh feature learning model is l mesh The calculation formula is:

[0036]

[0037] Where M is the number of vertices of the foot 3D mesh model, and are the i-th vertex of the input foot 3D mesh and the reconstructed foot 3D mesh respectively.

[0038] The specific establishment method for the feature conversion model in step (3) is as follows:

[0039] Foot shape feature conversion model T vh2mesh Based on the multi-layer perceptron structure, the foot-shaped visible shell feature y vh As input, it consists of 2 layers of fully connected layers, each of which is followed by a leaky ReLU nonlinear activation unit, and outputs the foot shape grid feature y mesh The first fully connected layer is used as the bottleneck layer, and its number of channels is half of the number of input / output feature channels to alleviate the overfitting problem of the model. The output foot mesh features are then decoded by the foot mesh feature learning model. mesh To reconstruct the foot mesh model x′ mesh , used to match the real foot mesh model x mesh Calculate the loss and train the feature conversion model.

[0040] The loss function l of the feature conversion model vh2mesh The calculation formula is:

[0041]

[0042] Where M is the number of vertices of the foot 3D mesh model, and are the i-th vertex of the real foot 3D mesh and the reconstructed foot 3D mesh respectively.

[0043] Example

[0044] (1) First, the camera is internally calibrated and the calibrated camera is used to collect photos of the human foot from three perspectives (front, left, and right) with external calibration reference objects. When collecting photos, an A4 paper with a fixed number of scattered points is placed on the ground as an external calibration reference object. The user's foot is placed at the center of the long side of the A4 paper and remains motionless during photo collection. When collecting photos, it is necessary to ensure that the entire foot area and the A4 paper appear in the photo. After the photo collection is completed, the contour map of the foot area in the photo is extracted, and the camera external parameters corresponding to each photo are calculated.

[0045] (2) Using the three foot contour images and the corresponding camera internal and external parameters, the foot-shaped visible shell is calculated and represented as a 3D voxel model. The obtained foot-shaped visible shell is then post-processed by center alignment according to its bounding box.

[0046] (3) The processed foot shape visual shell is taken as input, and the encoder of the visual shell feature learning model is used to extract the foot shape visual shell features. The feature conversion model converts the visual shell features into foot shape mesh features. The decoder of the foot shape mesh feature learning model is used to reconstruct the foot shape three-dimensional mesh model as output.

[0047] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A progressive high-precision human foot shape reconstruction method, characterized in that: The steps include: (1) Constructing a rough foot shape: Using the three foot contour images and the camera intrinsic and extrinsic parameters corresponding to the images, construct the viewing cone space of each contour image and intersect all the viewing cone spaces to obtain the visible shell of the foot as the rough foot shape; (2) Establishing a human foot shape visual shell feature learning model: The human foot shape visual shell feature learning model is an autoencoder structure, which takes the human foot shape visual shell as input, and learns the foot shape visual shell features by extracting features from the foot shape visual shell and reconstructing the foot shape visual shell; (3) Establishing a human foot mesh feature learning model: The human foot mesh feature learning model is an autoencoder structure that takes a human foot 3D mesh as input and learns the foot mesh features by extracting features from the 3D foot mesh and reconstructing the 3D foot mesh. (4) Establishing a feature conversion model between the visual shell features and mesh features of human foot shape: The feature conversion model between the visual shell features and mesh features of foot shape is a multi-layer perceptron structure, which learns the feature conversion between the visual shell features and mesh features; (5) When performing high-precision reconstruction of the human foot shape, three foot contour images and the corresponding camera internal and external parameters are used as input. First, a visual shell is constructed as a rough foot shape. Then, the visual shell features of the foot shape are extracted through the visual shell feature learning model. Subsequently, the feature conversion model is used to convert the visual shell features into three-dimensional mesh features of the foot shape. Finally, the foot shape mesh feature learning model is used to perform high-precision three-dimensional foot shape reconstruction and output a high-precision three-dimensional foot shape.

2. A progressive high-precision human foot shape reconstruction method according to claim 1, characterized in that: In the step (1), the rough foot shape is constructed in combination with the human foot shape law, and the cone space is discretized based on the length, width and height ratio of the human foot shape, thereby reducing unnecessary space occupation and using a higher-precision voxel model to represent the visual shell. The generated visual shell can better reflect the rough foot shape.

3. A progressive high-precision human foot shape reconstruction method according to claim 1, characterized in that: In the step (2), the human foot type visual shell feature learning model is based on an autoencoder structure, uses a 3D convolutional layer and a fully connected layer to extract features from the foot type visual shell, and reconstructs the foot type visual shell from the extracted visual shell features through a fully connected layer and a 3D deconvolutional layer. The model uses a self-supervised learning method for model training; the reconstruction loss of the foot type visual shell is used as the loss function, specifically the binary cross entropy between the reconstructed foot type visual shell and the input foot type visual shell.

4. A progressive high-precision human foot shape reconstruction method according to claim 1, characterized in that: In the step (3), the human foot mesh feature learning model is based on an autoencoder structure, uses a graph convolution layer to extract features from the foot three-dimensional mesh, and calculates the foot shape parameters from the foot mesh features through a fully connected layer, and uses an inverse principal component analysis method to reconstruct the foot three-dimensional mesh. The model uses a self-supervised learning method for model training; the reconstruction loss of the foot three-dimensional mesh is used as the loss function, specifically the L2 point cloud distance between the reconstructed foot three-dimensional mesh and the input foot three-dimensional mesh.

5. The method for progressive high-precision human foot shape reconstruction according to claim 1, characterized in that: In the step (4), the feature conversion model of the foot type visual shell feature and the mesh feature is based on a multi-layer perceptron structure, and a fully connected layer is used to convert between the visual shell feature and the mesh feature, so as to convert the foot type visual shell feature into the foot type mesh feature; the converted foot type mesh feature is then used to reconstruct the foot type three-dimensional mesh by the autoencoder in step (3); the model is trained using a supervised learning method; and the reconstruction loss of the foot type three-dimensional mesh is used as the loss function, specifically, the L2 point cloud distance between the reconstructed foot type three-dimensional mesh and the real foot type three-dimensional mesh.

6. The method for progressive high-precision human foot shape reconstruction according to claim 1, characterized in that: In the step (5), a progressive reconstruction method is used for foot shape reconstruction. First, a rough foot shape is reconstructed from three foot shape contour images and corresponding camera internal and external parameters. Then, a trained visual shell feature learning model, feature conversion model, and mesh feature learning model are used to extract visual shell features, convert them into mesh features, and reconstruct a high-precision three-dimensional foot shape.

Images