Three-dimensional human body shape estimation method for realizing surface alignment based on random curvature structure learning
Through the graph convolution network based on random curvature structure learning, the surface patch structure of the human body model is reconstructed and aligned, which solves the problem of low accuracy of three-dimensional human body shape estimation in the existing technology and achieves more accurate and detailed three-dimensional human body shape reconstruction. .
Patent Information
- Application Number
- CN202411374163.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2025-01-21
AI Technical Summary
In the prior art, when estimating the shape of a three-dimensional human body from monocular images or videos, it is difficult to accurately reconstruct the complex topological structure of the human body, resulting in a decrease in accuracy.
Using the method of learning based on random curvature structure, through the constructing graphic convolution U -shaped codec facade network, reconstruct the surface facial structure of the human model, and integrate the curvature supervision and training side to the end graphic convolution U -shaped codec decoder Rebuild the network and enhance the estimation of the shape of the three -dimensional human body.
Through the integration of grid alignment and curvature information, the accuracy and detailed performance of 3D human body shape estimation are improved, and the detailed features of the 3D human body shape can be reconstructed more accurately.
Smart Images
Figure CN119339179A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a three-dimensional human body shape estimation method based on random curvature structure learning to achieve surface alignment. Background Art
[0002] 3D human shape estimation has long been considered a fundamental task in computer vision and is essential in various applications such as augmented virtual reality, somatosensory gaming, and human-computer interaction. Unlike methods that rely on multi-view or depth cameras, reconstructing 3D shapes from monocular images or videos provides a more convenient and cost-effective solution. However, this task poses significant challenges due to the complex structure of the human body and the uncertainty of mapping from 2D to 3D. Therefore, researchers have worked hard to overcome these difficulties and improve the accuracy of estimating 3D human shape from monocular images or videos.
[0003] Traditionally, optimization-based methods have been a reliable approach for estimating human pose and shape in monocular images. These methods aim to estimate the 3D pose and shape of a human body by iteratively fitting a given parametric model. However, these methods also have limitations, including slow inference speed and sensitivity to initialization. Recent work has turned to solving the problem of 3D human pose and shape estimation through deep learning. Existing algorithms typically rely on deep learning techniques to construct a mapping between input images and output shapes in Euclidean space. However, this technique limits the output 3D human shape from the perspective of Euclidean space. For example, using the vertex information of the 3D human shape or the 2D / 3D joint coordinate information as supervision ignores the fact that the human shape is a complex topological structure, resulting in a decrease in algorithm accuracy. Summary of the invention
[0004] The object of the present invention is to provide a three-dimensional human body shape estimation method based on random curvature structure learning to achieve surface alignment, which is conducive to more accurate reconstruction of the three-dimensional human body shape.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is: a three-dimensional human body shape estimation method based on random curvature structure learning to achieve surface alignment, comprising the following steps:
[0006] Step S1: Obtain a human body posture data set and divide it into a training data set and a test data set;
[0007] Step S2: reconstructing the surface patch structure of the human body model;
[0008] Step S3: construct a patch alignment network based on graph convolution U-type encoder-decoder;
[0009] Step S4: pre-training a graph convolutional U-coder-decoder patch alignment network;
[0010] Step S5: Fusing the trained network with the curvature supervised training end-to-end graph convolutional U-coder-decoder to reconstruct the network;
[0011] Step S6: input the test data set into the trained end-to-end graph convolutional U-coder / decoder reconstruction network for reconstruction, obtain the reconstruction result, and obtain the weighted optimal model by comparing it with the true value;
[0012] Step S7: load the optimal model weights, input the monocular image, and obtain the reconstruction result.
[0013] Furthermore, in step S1, the human body posture datasets are Human3.6M and UP-3D.
[0014] Furthermore, in step S2, the vertex coordinates of the human body model are processed to reconstruct the surface patch structure of the human body model, thereby affecting the curvature value of each vertex, so as to train a patch alignment network based on a graph convolutional U-type codec.
[0015] Furthermore, in step S2, a two-dimensional Gaussian kernel function is used to generate a two-dimensional Gaussian matrix G, which is added element by element to the input original human body model M to obtain a reconstructed human body model M'.
[0016] Furthermore, in step S3, a patch alignment network based on graph convolution U-type codec is constructed, which is composed of an encoder and a decoder and is designed as a U-shaped architecture; the reconstructed human body model is input into the patch alignment network based on graph convolution U-type codec, so that the codec learns the curvature information and outputs a human body model that is as realistic as possible;
[0017] The encoder consists of three encoding modules, each of which consists of graph convolution, graph normalization, a fully connected layer and a ReLU activation function, and a downsampling layer is added between the encoding modules to downsample the features; the reconstructed human body model M' is input into the encoder Encoder1 to obtain M1 E ; Then the first downsampling is performed After the first downsampling, it enters encoder Encoder2 and obtains M2 E ; Then perform the second downsampling After the second downsampling, it enters encoder Encoder3 and obtains M3 E ;
[0018] The decoder consists of three decoding modules, each of which consists of graph convolution, graph normalization, a fully connected layer and a ReLU activation function. At the same time, an upsampling layer is added between the decoding modules to upsample the features. The obtained M3 E Input decoder Decoder3, get M3D ; Then the first upsampling is performed Then the first up-sampled feature is combined with M2 E Add And send it to the decoder Dncoder2 to get M2 D ; Then perform the second upsampling Then the second up-sampled features are combined with M1 E Add And send it to the decoder Dncoder1 to get M1 D ; Finally, the output M is obtained through the fully connected layer out .
[0019] Furthermore, in step S4, the loss function is used to train and optimize the network parameters, and the training is performed under the Pytorch deep learning framework; the optimization target expression is as follows:
[0020] argmin(M,D(E(M')))
[0021] Among them, M' is the reconstructed human body model, and D(E(M')) is the output of the encoder and decoder.
[0022] Further, in step S5, the trained network is fused with the curvature supervised training end-to-end graph convolutional U-coder-decoder reconstruction network, specifically:
[0023] The input image is passed through the Resnet network to extract features, and then the extracted features are processed using the GraphCMR algorithm to obtain the initial human body model; then the model is input into the encoder and decoder to obtain the output;
[0024] The training was performed under the Pytorch deep learning framework. During the training, the curvature information was integrated for supervision, and the encoder and decoder parameters were optimized. The GraphCMR parameters remained unchanged.
[0025] The loss function is used to continuously optimize and update the weight parameters of each layer. The loss function expression is as follows:
[0026] L mesh =L shape +L j2d +L j3c +λL GC
[0027] Among them, L shape is the shape loss function, L j2d is the joint point loss in two-dimensional space, L j3d is the joint point loss in three-dimensional space, L GC is the curvature loss;
[0028] L shape=||MM GT ||1
[0029] Among them, M is the real labeled node, M GT are nodes in the estimated human body shape;
[0030]
[0031] In addition, a joint-level loss is set to further align the mesh representation with the skeleton joints; the 3D human skeleton joints are regressed from the estimated mesh representation, where J 3D is the 3D joint point obtained by regression, The 3D joint points of the human body are truly marked;
[0032]
[0033] The 2D joint points are obtained by projecting the 3D joint points of the human body onto the image plane, where J 2D is the 2D human joint point obtained by projection, It is the real annotated 2D human joint points;
[0034] L GC =L PC +L NC
[0035] Among them, L PC represents the positive curvature loss, and L NC represents negative curvature loss.
[0036] Furthermore, in step S6, the test data set is input into the trained end-graph convolutional U-type codec reconstruction network for reconstruction to obtain the reconstruction result, and then the error between the reconstruction result and the true value is calculated, and the training round with the smallest error is taken as the model with the optimal model weight and the model is saved.
[0037] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a three-dimensional human body shape estimation method based on random curvature structure learning to achieve surface alignment, which enhances the detailed representation of the local position of the human body model under non-parametric regression through grid alignment; at the same time, the Gaussian curvature information of the grid is considered and integrated into the loss function, so that the grid can be successfully refined during the reconstruction process, thereby more accurately reconstructing the detailed features of the three-dimensional human body shape. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A flowchart of a method implementation of an embodiment of the present invention;
[0039] Figure 2This is an example diagram of training a patch alignment network based on a graph convolutional U-coder / decoder in an embodiment of the present invention;
[0040] Figure 3 This is an example diagram of an end-to-end graph convolutional U-coder / decoder reconstruction network fused with curvature supervision training in an embodiment of the present invention;
[0041] Figure 4 This is the reconstruction result of the test data set in the embodiment of the present invention. DETAILED DESCRIPTION
[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0043] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0044] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0045] like Figure 1 As shown, this embodiment provides a method for estimating a 3D human body shape based on random curvature structure learning to achieve surface alignment, comprising the following steps:
[0046] Step S1: Obtain a human body posture data set and divide it into a training data set and a test data set;
[0047] Step S2: reconstructing the surface patch structure of the human body model;
[0048] Step S3: construct a patch alignment network based on graph convolution U-type encoder-decoder;
[0049] Step S4: pre-training a graph convolutional U-coder-decoder patch alignment network;
[0050] Step S5: Fusing the trained network with the curvature supervised training end-to-end graph convolutional U-coder-decoder to reconstruct the network;
[0051] Step S6: input the test data set into the trained end-to-end graph convolutional U-coder / decoder reconstruction network for reconstruction, obtain the reconstruction result, and obtain the weighted optimal model by comparing it with the true value;
[0052] Step S7: load the optimal model weights, input the monocular image, and obtain the reconstruction result.
[0053] In step S1, the human posture datasets used are Human3.6M and UP-3D. Human3.6M is one of the important datasets in the current 3D human shape reconstruction task; the dataset contains more than 1,000 video sequences, and the 2D and 3D joint annotations of the human body are collected using an accurate motion capture system in a controlled indoor scene; UP-3D is a 3D human posture dataset obtained by fitting using the SMPLify method on some existing 2D datasets, which contains about 7,000 manually selected natural images with high-quality annotation information, including 2D joint annotations, 3D joint annotations, and 3D human shape annotations.
[0054] In step S2, the human body surface network is reconstructed; the surface patch structure of the human body model is reconstructed by processing the vertex coordinates of the human body model, thereby affecting the curvature value of each vertex, so as to train the graph convolutional U-type codec patch alignment network.
[0055] Specifically, the original human body model M∈(V,N) is input, and a two-dimensional Gaussian kernel function is used to generate a two-dimensional Gaussian matrix G, which is added element by element to the input original human body model M to obtain the reconstructed human body model M'. At this time, the curvature distribution of the human body model has changed compared to the original model. The specific formula is as follows:
[0056]
[0057] Among them, the two-dimensional Gaussian matrix is obtained by the following formula:
[0058]
[0059] In step S3, the constructed patch alignment network based on graph convolution U-type encoder-decoder is composed of an encoder and a decoder. In order to reduce the cost of computing resources, it is designed as a U-shaped architecture. The reconstructed human body model is input into the patch alignment network based on graph convolution U-type encoder-decoder, so that the encoder-decoder learns the curvature information and outputs a human body model that is as realistic as possible.
[0060] The encoder consists of three encoding modules, each of which consists of graph convolution, graph normalization, a fully connected layer and a ReLU activation function, and a downsampling layer is added between the encoding modules to downsample the features; the reconstructed human body model M' is input into the encoder Encoder1 to obtain M1 E ; Then the first downsampling is performed After the first downsampling, it enters encoder Encoder2 and obtains M2 E ; Then perform the second downsampling After the second downsampling, it enters encoder Encoder3 and obtains M3 E .
[0061] The decoder consists of three decoding modules, each of which consists of graph convolution, graph normalization, a fully connected layer and a ReLU activation function. At the same time, an upsampling layer is added between the decoding modules to upsample the features. The obtained M3 E Input decoder Decoder3, get M3 D ; Then the first upsampling is performed Then the first up-sampled feature is combined with M2 E Add And send it to the decoder Dncoder2 to get M2 D ; Then perform the second upsampling Then the second up-sampled features are combined with M1 E Add And send it to the decoder Dncoder1 to get M1 D ; Finally, the output M is obtained through the fully connected layer out .
[0062] The process of training the graph convolutional U-coder-decoder patch alignment network in this embodiment is as follows: Figure 2 shown.
[0063] In step S4, the loss function is used to train and optimize the network parameters, and the training is performed under the Pytorch deep learning framework; the optimization target expression is as follows:
[0064] argmin(M,D(E(M')))
[0065] Among them, M' is the reconstructed human body model, and D(E(M')) is the output of the encoder and decoder.
[0066] In this embodiment, the process of integrating curvature supervision to train the end-to-end graph convolutional U-coder and decoder to reconstruct the network is as follows: Figure 3 shown.
[0067] In step S5, the trained network is fused with the curvature supervised training end-to-end graph convolutional U-coder and decoder to reconstruct the network, specifically:
[0068] S51, extract features from the input image through the Resnet network, and then process the extracted features using the GraphCMR algorithm to obtain an initial human body model.
[0069] S52, input encoder and decoder, and get output. Train under the Pytorch deep learning framework, integrate curvature information for supervision during training, optimize encoder and decoder parameters, and keep GraphCMR parameters unchanged.
[0070] First, we calculate the curvature of each vertex of the human body model in the training data set, and obtain the distribution of positive and negative curvatures in the human body model. The following formula is used for calculation:
[0071]
[0072] Among them, θ k is the angle of the kth face, The triangle neighborhood shown by the vertex, is the local area approximating vertex i. For each vertex V of the human body model in the training set G (N), N∈{1,2,3...6890}, (here we use 6890 vertices in the SMPL model), use the formula to calculate and get the vertex curvature value, where the vertex set with positive curvature is N PC , the set of vertices with negative curvature N PC We use this as a reference to make the curvature value of the vertex of the human body shape reconstructed from the image positive at the vertex of positive curvature, and the curvature value of the vertex of the negative curvature negative.
[0073] Then we calculate the curvature of each vertex of the reconstructed human body model, and use the above formula to calculate and obtain the curvature value of each vertex:
[0074] The specific curvature supervision loss is as follows:
[0075] L GC =L PC +L NC
[0076] Among them, L PC represents the positive curvature loss, and L NC represents negative curvature loss.
[0077] Specific:
[0078] L PC =∑argmin||k GC (V Y (N PC ))-k GC (V G (N PC ))||1
[0079] Among them, k GC (V Y (N PC)) represents the curvature value of the positive curvature vertex set of the reconstructed 3D human body model, k GC (V G (N PC )) represents the curvature value of the set of positive curvature vertices in the training data set.
[0080] L NC =∑argmin||k GC (V Y (N NC ))-k GC (V G 9N NC ))||1
[0081] Among them, k GC (V Y (N NC )) represents the curvature value of the negative curvature vertex set of the reconstructed 3D human body model, k GC (V G (N NC )) represents the curvature value of the set of negative curvature vertices in the training data set.
[0082] S53, use the loss function to continuously optimize and update the weight parameters of each layer. The loss function expression is as follows:
[0083] L mesh =L shape +L j2d +L j3c +λL GC
[0084] Among them, L shape is the shape loss function, L j2d is the joint point loss in two-dimensional space, L j3d is the joint point loss in three-dimensional space, L GC is the curvature loss.
[0085] L shape =||MM GT ||1
[0086] Among them, M is the real labeled node, M GT are nodes in the estimated human body shape.
[0087]
[0088] In addition, a joint-level loss is set to further align the mesh representation with the skeleton joints; the 3D human skeleton joints are regressed from the estimated mesh representation, where J 3D is the 3D joint point obtained by regression, These are the real labeled 3D joint points of the human body.
[0089]
[0090] The 2D joint points are obtained by projecting the 3D joint points of the human body onto the image plane, where J 2D is the 2D human joint point obtained by projection, These are the real annotated 2D human joint points.
[0091] L GC =L PC +L NC
[0092] Among them, L PC represents the positive curvature loss, and L NC represents negative curvature loss.
[0093] In step S6, the test data set is input into the trained end-graph convolutional U-coder / decoder reconstruction network for reconstruction to obtain the reconstruction result, and then the error between the reconstruction result and the true value is calculated. The training round with the smallest error is taken as the model with the optimal model weight and the model is saved. The reconstruction result of the test data set in this embodiment is as follows: Figure 4 shown.
[0094] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0095] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0096] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0098] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
Claims
1. A three-dimensional human body shape estimation method based on random curvature structure learning to achieve surface alignment, characterized in that: The following steps are involved: Step S1: Obtain a human body posture data set and divide it into a training data set and a test data set; Step S2: reconstructing the surface patch structure of the human body model; Step S3: construct a patch alignment network based on graph convolution U-type encoder-decoder; Step S4: pre-training a graph convolutional U-coder-decoder patch alignment network; Step S5: Fusing the trained network with the curvature supervised training end-to-end graph convolutional U-coder-decoder to reconstruct the network; Step S6: input the test data set into the trained end-to-end graph convolutional U-coder / decoder reconstruction network for reconstruction, obtain the reconstruction result, and obtain the weighted optimal model by comparing it with the true value; Step S7: load the optimal model weights, input the monocular image, and obtain the reconstruction result.
2. The method for estimating three-dimensional human body shape by realizing surface alignment based on random curvature structure learning according to claim 1, characterized in that: In step S1, the human body posture datasets are Human3.6M and UP-3D.
3. The method for estimating three-dimensional human body shape by realizing surface alignment based on random curvature structure learning according to claim 1, characterized in that: In step S2, the vertex coordinates of the human body model are processed to reconstruct the surface patch structure of the human body model, thereby affecting the curvature value of each vertex to train a patch alignment network based on a graph convolutional U-type codec.
4. The method for estimating three-dimensional human body shape by realizing surface alignment based on random curvature structure learning according to claim 1, characterized in that: In step S2, a two-dimensional Gaussian kernel function is used to generate a two-dimensional Gaussian matrix G, which is added element by element to the input original human body model M to obtain a reconstructed human body model M'.
5. The method for estimating three-dimensional human body shape by realizing surface alignment based on random curvature structure learning according to claim 1, characterized in that: In step S3, a patch alignment network based on graph convolution U-type codec is constructed, which is composed of an encoder and a decoder and is designed as a U-shaped architecture; the reconstructed human body model is input into the patch alignment network based on graph convolution U-type codec, so that the codec learns the curvature information and outputs a human body model that is as realistic as possible; The encoder consists of three encoding modules, each of which consists of graph convolution, graph normalization, a fully connected layer and a ReLU activation function, and a downsampling layer is added between the encoding modules to downsample the features; Input the reconstructed human body model M' into encoder Encoder1 to get M1 E ; Then the first downsampling is performed After the first downsampling, it enters the encoder Enocoder2 and obtains M2 E ; Then perform the second downsampling After the second downsampling, it enters encoder Encoder3 and obtains M3 E ; The decoder consists of three decoding modules, each of which consists of graph convolution, graph normalization, a fully connected layer and a ReLU activation function. At the same time, an upsampling layer is added between the decoding modules to upsample the features. The obtained M3 E Input decoder Encoder3, get M3 D ; Then the first upsampling is performed Then the first up-sampled feature is combined with M2 E Add And send it to decoder Encoder2 to get M2 D ; Then perform the second upsampling Then the second up-sampled features are combined with M1 E Add And send it to the decoder Dncoder1 to get M1 D ; Finally, the output M is obtained through the fully connected layer out .
6. The method for estimating three-dimensional human body shape by realizing surface alignment based on random curvature structure learning according to claim 1, characterized in that: In step S4, the loss function is used to train and optimize the network parameters, and the training is performed under the Pytorch deep learning framework; the optimization target expression is as follows: argmin(M,D(E(M'))) Among them, M' is the reconstructed human body model, and D(E(M')) is the output of the encoder and decoder.
7. The method for estimating three-dimensional human body shape by realizing surface alignment based on random curvature structure learning according to claim 1, characterized in that: In step S5, the trained network is fused with the curvature supervised training end-to-end graph convolutional U-coder and decoder to reconstruct the network, specifically: The input image is passed through the Resnet network to extract features, and then the extracted features are processed using the GraphCMR algorithm to obtain the initial human body model; then the model is input into the encoder and decoder to obtain the output; The training was performed under the Pytorch deep learning framework. During the training, the curvature information was integrated for supervision, and the encoder and decoder parameters were optimized. The GraphCMR parameters remained unchanged. The loss function is used to continuously optimize and update the weight parameters of each layer. The loss function expression is as follows: THE mesh =L shape +L j2d +L j3c +λL GC Among them, L shape is the shape loss function, L j2d is the joint point loss in two-dimensional space, L j3d is the joint point loss in three-dimensional space, L GC is the curvature loss; L shape =||M-M GT ||1 Among them, M is the real labeled node, M GT are nodes in the estimated human body shape; In addition, a joint-level loss is set to further align the mesh representation with the skeleton joints; the 3D human skeleton joints are regressed from the estimated mesh representation, where J 3D is the 3D joint point obtained by regression, The 3D joint points of the human body are truly marked; The 2D joint points are obtained by projecting the 3D joint points of the human body onto the image plane, where J 2D is the 2D human joint point obtained by projection, It is the real annotated 2D human joint points; L GC =L PC +L NC Among them, L PC represents the positive curvature loss, and L NC represents negative curvature loss.
8. The method for estimating three-dimensional human body shape by realizing surface alignment based on random curvature structure learning according to claim 1, characterized in that: In step S6, the test data set is input into the trained end-graph convolutional U-type codec reconstruction network for reconstruction to obtain the reconstruction result, and then the error between the reconstruction result and the true value is calculated. The training round with the smallest error is taken as the model with the optimal model weight and the model is saved.