Human body three-dimensional model reconstruction method and device, equipment and storage medium
Through the method based on feature extraction model and attention calculation network, feature extraction and aggregation of sample videos is solved, and the accuracy and stability of human body three-dimensional model reconstruction in the existing technology is achieved, and efficient and accurate three-dimensional model reconstruction is achieved. Effect.
Patent Information
- Application Number
- CN202411149503.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-31
AI Technical Summary
The existing human body three-dimensional model reconstruction methods have harsh conditions, complex processes, and unstable continuous frame reconstruction jitter, resulting in insufficient accuracy of reconstruction results and low output video quality.
Using a method based on feature extraction model and attention computing network, the sample video is characterized and feature aggregation processed, and predicted frame images are generated to realize the reconstruction of the human body's three-dimensional model, which improves the reconstruction accuracy and reduces the computational complexity.
The reconstruction efficiency and accuracy of the human body three-dimensional model are improved, the jitter situation of the human body three-dimensional model in the target video is reduced, and the reconstruction effect is improved.
Smart Images
Figure CN119228990A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of machine learning, and in particular to a method, device, equipment and computer-readable storage medium for reconstructing a three-dimensional human body model. Background Art
[0002] The construction of human 3D models is used in various scenarios, such as character modeling, clothing design, and virtual fitting in movies and games. However, the current methods for constructing human 3D models have strict conditions for model construction, complex processes, and jitter and instability in the reconstruction of continuous frames, which leads to insufficient accuracy of the reconstruction results of the human 3D model and low quality of the output human 3D model reconstruction video. Summary of the invention
[0003] The present application provides a method, device, equipment and computer-readable storage medium for reconstructing a three-dimensional human body model, aiming to improve the reconstruction efficiency and accuracy of the three-dimensional human body model so as to enhance the reconstruction effect of the three-dimensional human body model.
[0004] In a first aspect, the present application provides a method for reconstructing a three-dimensional human body model, the method comprising the following steps:
[0005] Acquire a sample video, wherein the sample video includes a human body image;
[0006] Based on the trained feature extraction model, feature extraction processing is performed on each frame image in the sample video to obtain human feature data corresponding to each frame image;
[0007] Based on the attention calculation network of the human body 3D reconstruction model, the human body feature data corresponding to the input image is fully connected and normalized to obtain the feature weight value corresponding to the input image;
[0008] Based on the feature aggregation network of the human body 3D reconstruction model, feature aggregation processing is performed according to the feature weight values and the corresponding human body feature data to obtain a predicted frame image, wherein the human body features in the predicted frame image are different from the human body features of the corresponding frame image in the sample video;
[0009] Generate a target video according to the predicted frame image.
[0010] In a second aspect, the present application further provides a device for reconstructing a three-dimensional human body model, the device for reconstructing a three-dimensional human body model comprising:
[0011] A video acquisition module, used to acquire a sample video, wherein the sample video includes a human body image;
[0012] A feature data extraction module is used to perform feature extraction processing on each frame of the sample video based on the trained feature extraction model to obtain human feature data corresponding to each frame of the image;
[0013] An attention calculation module is used to perform full connection and normalization processing on human feature data corresponding to an input image based on an attention calculation network of a human body three-dimensional reconstruction model to obtain a feature weight value corresponding to the input image;
[0014] An image prediction module, configured to perform feature aggregation processing based on a feature aggregation network of the human body three-dimensional reconstruction model according to the feature weight values and corresponding human body feature data to obtain a predicted frame image, wherein the human body features in the predicted frame image are different from the human body features of the corresponding frame image in the sample video;
[0015] The video generation module is used to generate a target video according to the predicted frame image.
[0016] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the method for reconstructing a three-dimensional human body model as described above are implemented.
[0017] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method for reconstructing a three-dimensional human body model as described above are implemented.
[0018] The present application provides a method, apparatus, device and computer-readable storage medium for reconstructing a three-dimensional model of a human body. The present application obtains human feature data by extracting features from images in a sample video, and uses an attention calculation network based on the three-dimensional reconstruction model of the human body to fully connect and normalize the human feature data corresponding to the image input into the attention calculation network to obtain a feature weight value corresponding to the image, and determines a predicted frame image based on the feature weight value and the human feature data, so that a target video can be generated using the obtained predicted frame image to complete the reconstruction of the three-dimensional model of the human body, thereby improving the accuracy of the reconstruction of the three-dimensional model of the human body and reducing the jitter of the three-dimensional model of the human body in the target video; and in the process of reconstructing the three-dimensional model of the human body, the data calculation complexity is reduced, thereby improving the reconstruction efficiency of the three-dimensional model of the human body. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0020] Figure 1 A schematic flow chart of a method for reconstructing a three-dimensional human body model provided in an embodiment of the present application;
[0021] Figure 2 A schematic diagram of a use scenario of a three-dimensional human body model provided in an embodiment of the present application;
[0022] Figure 3 A schematic block diagram of a device for reconstructing a three-dimensional human body model provided in an embodiment of the present application;
[0023] Figure 4 The present invention is a block diagram showing the structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0026] The embodiments of the present application provide a method, device, computer equipment and computer-readable storage medium for reconstructing a three-dimensional model of a human body. The method for reconstructing a three-dimensional model of a human body can be applied to a computer device, which can be an electronic device such as a tablet computer, a laptop computer, or a desktop computer. It can also be applied to a server, which can be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0028] Please refer to Figure 1 , Figure 1 A schematic flow chart of a method for reconstructing a three-dimensional human body model provided in an embodiment of the present application.
[0029] like Figure 1 As shown, the method for reconstructing the three-dimensional human body model includes steps S101 to S104.
[0030] Step S101: Obtain a sample video, where the sample video includes a human body image.
[0031] Exemplarily, in the process of reconstructing a three-dimensional model of the human body, it is necessary to capture the human body features in the sample video to achieve the reconstruction of the three-dimensional model of the human body. It should be understood that at least two frames of images in the sample video include human body images. After feature extraction of the human body features in the video, the extracted human body feature data is used to achieve the reconstruction of the three-dimensional model of the human body.
[0032] In a specific implementation, a video of a target person is captured by a shooting device to obtain a sample video, or a video containing a human body image in a video image frame is obtained as a sample video, which is not limited in this application.
[0033] Step S102: Based on the trained feature extraction model, feature extraction processing is performed on each frame image in the sample video to obtain human body feature data corresponding to each frame image.
[0034] Exemplarily, before performing feature extraction, each frame image in the video is determined, so that feature extraction processing is performed on each frame image in the sample video through a trained feature extraction model, thereby obtaining human body feature data corresponding to each frame image.
[0035] In the specific implementation process, there may be several frames of images in the sample video that do not contain human images. In this case, the human feature data obtained for these images is 0, or the presence of human images is identified for each frame of the image, and then feature extraction is performed on the images including the human body, thereby improving the efficiency of feature extraction.
[0036] Exemplarily, the feature extraction model is a model composed of a ResNet-50 network. After the model is pre-trained by SPIN, it is used as a feature extraction model to extract human feature data from an image, and there is no need to retrain the trained feature extraction model and update the parameters, so as to shorten the time of the human body 3D model reconstruction process, thereby improving the efficiency of human body 3D model reconstruction. It should be understood that human feature data can also be extracted from an image through other deep learning models and / or networks with feature extraction functions, and this application does not limit the specific feature extraction model.
[0037] Step S103: Based on the attention calculation network of the human body three-dimensional reconstruction model, the human body feature data corresponding to the input image is fully connected and normalized to obtain the feature weight value corresponding to the input image.
[0038] Exemplarily, after the human body feature data is extracted, each frame of the sample video corresponding to the human body 3D reconstruction model is input into the human body 3D reconstruction model, wherein the human body 3D reconstruction model is a trained model.
[0039] Specifically, the image is input into the attention calculation network in the three-dimensional reconstruction model of the human body, so that the human body feature data corresponding to the input image is fully connected and normalized in the attention calculation network of the three-dimensional reconstruction model of the human body, so as to determine the feature weight value corresponding to the input image, and then reconstruct the three-dimensional model of the human body according to the feature weight value and the human body feature data.
[0040] In some embodiments, the attention computing network based on the three-dimensional reconstruction model of the human body performs full connection and normalization processing on the human feature data corresponding to the input image to obtain the feature weight value corresponding to the input image, including: based on the first fully connected layer of the attention computing network, performing data dimension conversion processing on the human feature data of the input image to obtain converted human feature data, wherein the data dimension of the converted human feature data is smaller than the data dimension of the human feature data before conversion; based on the data splicing layer of the attention computing network, performing feature fusion processing on the converted human feature data corresponding to each input image to obtain a human feature data set; based on the fully connected and normalized layer of the attention computing network, performing data dimension conversion on the human feature data set, and normalizing the converted human feature data set to obtain the feature weight value of the input image.
[0041] Exemplarily, the attention calculation network includes a first fully connected layer, a data splicing layer, a fully connected and normalized layer, and performs attention calculation on the human feature data corresponding to the image in the attention calculation network to obtain the corresponding feature weight value.
[0042] In the specific implementation process, in the first fully connected layer, the data dimension of the human feature data is converted. Specifically, when the feature extraction model is ResNet-50, the human feature data corresponding to each frame image extracted includes 2048-dimensional features. In the first fully connected layer, the 2048-dimensional features can be converted to obtain converted human feature data. In this embodiment, the converted human feature data contains 256-dimensional features.
[0043] It should be noted that the feature dimensions of the human feature data provided above are for illustration only. In actual application, the dimensions of the extracted feature data can be adjusted according to actual needs, and this application does not limit this.
[0044] After the dimension conversion, the human feature data corresponding to each input image are subjected to feature fusion processing in the data splicing layer to obtain a human feature data set, and in the full connection and normalization layer, the feature weight value corresponding to the input image is determined according to the human feature data set.
[0045] In some embodiments, the input image includes a first image, a second image for indicating adjacent frames of the first image, and a third image for indicating adjacent frames of the second image, and the first fully connected layer based on the attention computing network performs data dimension conversion processing on the human feature data of the input image to obtain converted human feature data, wherein the data dimension of the converted human feature data is smaller than the data dimension of the human feature data before conversion, including: based on the first fully connected layer of the attention computing network, respectively performing data dimension conversion processing on the human feature data corresponding to the first image, the second image and the third image; the data splicing layer based on the attention computing network performs feature fusion processing on the converted human feature data corresponding to each input image to obtain a human feature data set, including: based on the attention computing network The data splicing layer of the attention computing network performs feature fusion connection processing on the converted human feature data corresponding to the first image, the converted human feature data corresponding to the second image, and the converted human feature data corresponding to the third image to obtain a human feature data set; the fully connected and normalized layer based on the attention computing network performs data dimension conversion on the human feature data set, and normalizes the converted human feature data set to obtain the feature weight value of the input image, including: based on the fully connected and normalized layer of the attention computing network, performs data dimension conversion on the human feature data set, and normalizes the converted human feature data set to obtain a first feature weight value corresponding to the first image, a second feature weight value corresponding to the second image, and a third feature weight value corresponding to the third image.
[0046] In the specific implementation process, when the attention calculation network inputs the human feature data corresponding to each frame of image, the human feature data corresponding to each three adjacent frames of image are processed to determine the feature weight values corresponding to the three frames of image.
[0047] Specifically, for example, a sample video contains five frames of images, which are represented by A, B, C, D, and E respectively; after feature extraction processing is performed on these images, human feature data corresponding to each frame of the image is obtained, and after the human feature data corresponding to each frame of the image is input into the attention calculation network, the attention calculation network processes based on every three frames of images, for example, image A, image B, and image C are processed, and at the same time or after the aforementioned processing is completed, image B, image C, and image D are processed, and so on, until all images are processed.
[0048] Exemplarily, during the processing, the first image is used to indicate image A, the second image is used to indicate image B, and the third image is used to indicate image C; in the first fully connected layer, the human body feature data corresponding to image A, image B, and image C are respectively dimensionally converted, such as converting 2048-dimensional human body feature data into 256-dimensional human body feature data, and in the data splicing layer, the converted human body feature data corresponding to the three images are feature fused to obtain a 758-dimensional human body feature data set.
[0049] The fully connected and normalized layers include a second fully connected layer, a first activation function layer, a third fully connected layer, a second activation function layer, a fourth fully connected layer, a third activation function layer and a normalized layer; wherein the second fully connected layer performs data dimension conversion processing on a 758-dimensional human feature data set to obtain a 256-dimensional human feature data set, and the first activation function layer is used to perform activation calculation processing on the 256-dimensional human feature data set output by the second fully connected layer based on the activation function. It should be understood that after the activation function processing, the dimension of the human feature data set will not change; the third fully connected layer is used to keep the human feature data set in 256 dimensions; the second activation function layer is used to perform activation calculation processing on the human feature data set of the third fully connected layer data based on the activation function; the fourth fully connected layer is used to convert the 256-dimensional human feature data set output from the second activation function layer into a 3-dimensional human feature data set; the third activation function layer is used to perform activation calculation processing on the 3-dimensional human feature data set based on the function; the normalization layer is used to normalize the 3-dimensional data output from the third activation function layer, thereby obtaining a total of 3-dimensional data including a first feature weight value, a second feature weight value and a third feature weight value. It should be understood that the first feature weight value is used to characterize the importance of the human feature data of image A, the second feature weight value characterizes the importance of the human feature data of image B, and the third feature weight value characterizes the importance of the human feature data of image C. Thus, in the process of determining the predicted frame image, the aggregation of features in the previous frame image and the subsequent frame image is achieved to improve the accuracy of the reconstruction of the human body three-dimensional model.
[0050] In a specific implementation process, the activation function includes but is not limited to a Tanh function, a Sigmoid function, a ReLU function, etc. A softmax function is provided in the normalization layer to normalize the input data.
[0051] It should be understood that after image A, image B and image C are processed, or during the process of processing image A, image B and image C, the first image can also be used to indicate image B, the second image indicates image C, and the third image indicates image D, so as to perform the same steps provided in the above-mentioned image A, B, and C embodiments to obtain the feature weight values corresponding to each input image.
[0052] Step S104, based on the feature aggregation network of the human body 3D reconstruction model, feature aggregation processing is performed according to the feature weight values and the corresponding human body feature data to obtain a predicted frame image, wherein the human body features in the predicted frame image are different from the human body features of the corresponding frame image in the sample video.
[0053] Exemplarily, after the feature weight value is calculated, feature aggregation processing is performed on the feature weight value and the human body feature data to achieve prediction of the human body features, so that the human body three-dimensional model can be reconstructed according to the predicted results of the human body features. The human body features can be used to indicate the actions and human body shapes displayed by the human body three-dimensional model, thereby completing the reconstruction of the human body three-dimensional model and obtaining a predicted frame image containing the reconstructed human body three-dimensional model.
[0054] It should be noted that, in the process of predicting the image, the prediction is made based on the input image, and the input image is a frame image in the sample video. Therefore, the predicted frame image has a corresponding image in the sample video, and the human body features in the predicted frame image are different from the human body features of the corresponding frame image in the sample video, thereby realizing the reconstruction of the three-dimensional model of the human body.
[0055] In some embodiments, the feature aggregation network based on the three-dimensional reconstruction model of the human body performs feature aggregation processing according to the feature weight value and the corresponding human feature data to obtain a predicted frame image, including: based on the feature aggregation network of the three-dimensional reconstruction model of the human body, determining first feature calculation data according to the first feature weight value, the second feature weight value and the third feature weight value; determining second feature calculation data according to the human feature data corresponding to the first image, the human feature image corresponding to the second image and the human feature data corresponding to the third image; performing feature aggregation processing on the first feature calculation data and the second feature calculation data to obtain the predicted frame image.
[0056] Exemplarily, in the feature aggregation network, the first feature calculation data is determined according to the first, second and third feature weight values, and the second feature calculation data is determined according to the human body feature data corresponding to the first, second and third images, so that feature aggregation processing is performed on the first feature calculation data and the second feature calculation data to reconstruct the human body features, thereby obtaining a predicted frame image.
[0057] In the specific implementation process, the human body feature data corresponding to the first, second and third images are spliced to obtain second feature calculation data, that is, in the feature aggregation process, the first, second and third feature weight values are used as a calculation factor, and the human body feature data corresponding to the first, second and third images are spliced as another calculation factor, so as to calculate these two calculation factors in the feature aggregation network, thereby obtaining the reconstructed human body features, and determining the reconstructed human body three-dimensional model based on the reconstructed human body features.
[0058] For example, taking the case where human feature data is represented by feature vectors, the splicing of the human feature data corresponding to the first, second and third images can be achieved by vector addition or vector splicing, which is not limited in this application.
[0059] In some embodiments, the performing feature aggregation processing on the first feature calculation data and the second feature calculation data to obtain the predicted frame image includes: performing convolution processing on the first feature calculation data and the second feature calculation data to obtain the predicted frame image.
[0060] Exemplarily, the first feature calculation data and the second feature calculation data are convolved to reconstruct a three-dimensional model of a human body according to the convolution result, thereby obtaining a predicted frame image.
[0061] In other embodiments, if the first feature calculation data and the second feature calculation data are both represented in the form of vectors, the first feature calculation data and the second feature calculation data can be vector multiplied, and the three-dimensional human body model can be reconstructed according to the result of the vector multiplication to obtain a predicted frame image. It should be understood that the convolution result or the result of the vector multiplication is an updated human body feature, which can represent the human body movement or form corresponding to the three-dimensional human body model. Therefore, the three-dimensional human body model can be reconstructed according to the convolution result or the result of the vector multiplication, and the reconstructed three-dimensional human body model can be converted into an image form to obtain a predicted frame image.
[0062] See also Figure 2 , Figure 2 A schematic diagram of a usage scenario of a three-dimensional human body model provided in one embodiment of the present application.
[0063] like Figure 2As shown, the human feature data of the first image, the human feature data of the second image, and the human feature data of the third image are input into the first fully connected layer of the attention calculation network, and each 2048-dimensional human feature data is converted into 256-dimensional human feature data in the first fully connected layer, and the three 256-dimensional human feature data input are connected in the data splicing layer to obtain a 768-dimensional human feature data set; in the second fully connected layer, the 768-dimensional human feature data set is converted into a 256-dimensional human feature data set, and output to the first activation function layer to perform activation calculation in the first activation function layer, after the activation calculation of the first activation function layer, it is input to the third fully connected layer, and the human feature data set is kept at 256 dimensions, and then input to the second activation function layer for activation calculation again, and then input to the fourth fully connected layer, the 256-dimensional human feature data set is converted into a 3-dimensional human feature data set, and the 3-dimensional human feature data set output from the fourth fully connected layer is sequentially input to the third activation function layer and the normalization layer to obtain the first feature weight value, the second feature weight value, and the third feature weight value. The first feature weight value, the second feature weight value and the third feature weight value are input into the feature aggregation network, and the human body feature data corresponding to the first image, the second image and the third image are respectively spliced and input into the feature aggregation network, so that in the feature aggregation network, the first feature calculation data determined according to the first feature weight value, the second feature weight value and the third feature weight value and the second feature calculation data determined according to the human body feature data corresponding to the first image, the second image and the third image are subjected to feature aggregation processing to obtain a predicted frame image, wherein the human body feature data in the predicted frame image is different from the human body feature data of the corresponding frame image in the sample video.
[0064] It should be understood that the overall network structure of the human body 3D reconstruction model is completely composed of the attention mechanism, and the human body 3D reconstruction model can improve the continuity of the time series, achieve better human body 3D model reconstruction effect, and reduce the amount of data calculation, thereby improving the reconstruction efficiency of the human body 3D model.
[0065] Exemplarily, in addition to the above-mentioned network, the human body 3D reconstruction model may also include, for example, a network in the SMPL model and / or the SMPLX model, or other networks, which are not limited in this application. In some specific embodiments, based on the image human body network regression method, the human body 3D reconstruction model is pre-trained using images extracted from the sample video to improve the accuracy of the human body 3D model reconstruction.
[0066] Step S105: Generate a target video according to the predicted frame image.
[0067] Exemplarily, after the predicted frame images are obtained, the predicted frame images may be sorted, and a target video may be generated based on the predicted frame image sequence, thereby completing the reconstruction of the three-dimensional model of the human body.
[0068] In some embodiments, generating the target video according to the predicted frame images includes: sorting the multiple predicted frame images in the order of image generation timing, so as to generate the target video according to the sorted predicted frame image sequence.
[0069] Exemplarily, when the human body 3D reconstruction model predicts each frame of the sample video in sequence, it will output the corresponding predicted frame images in sequence, and sort the predicted frame images according to the output timing of the predicted frame images of the human body 3D reconstruction model to obtain a predicted frame image sequence, and then generate the target video according to the image sequence. It should be understood that the corresponding image playback order in the target video is the same as the order in which the predicted frames are output in the human body 3D reconstruction model.
[0070] In some embodiments, the sample video includes at least a first image and a second image, and the timing of the second image in the sample video is located after the timing of the first image in the sample video; the predicted frame image includes at least a predicted frame image corresponding to the first image and a predicted frame image corresponding to the second image; and sorting the multiple predicted frame images in the order of image generation timing includes: inserting the predicted frame image corresponding to the second image into the predicted frame image sequence after the predicted frame image corresponding to the first image, so as to complete the sorting of the predicted frame images.
[0071] In a specific implementation process, the human body 3D reconstruction model may process multiple frame images in a sample video at the same time and output multiple predicted frame images at the same time. In this case, the predicted frame image is determined to correspond to the frame image in the sample video. For example, the sample video includes five frame images A, B, C, D, and E. It can be seen from the above embodiments that the human body 3D reconstruction model predicts the predicted frame image corresponding to image B based on the three frame images A, B, and C, and predicts the predicted frame image corresponding to image C based on the three frame images B, C, and D, and predicts the predicted frame image corresponding to image D based on the three frame images C, D, and E. Then, according to the image corresponding to the predicted frame image in the sample video, the order of each predicted frame image is determined. For example, the predicted frame image corresponding to image C is located after the predicted frame image corresponding to image B in the predicted frame image sequence. The predicted frame images are sorted according to the above rules to obtain the corresponding predicted frame image sequence, thereby generating a target video based on the predicted frame images.
[0072] The method for reconstructing a three-dimensional human body model provided in the above embodiment obtains human body feature data by extracting features from images in a sample video, and uses an attention calculation network based on the three-dimensional reconstruction model of the human body to fully connect and normalize the human body feature data corresponding to the image input into the attention calculation network to obtain a feature weight value corresponding to the image, and determines a predicted frame image based on the feature weight value and the human body feature data, so that the obtained predicted frame image can be used to generate a target video to complete the reconstruction of the three-dimensional model of the human body, thereby improving the accuracy of the reconstruction of the three-dimensional model of the human body and reducing the jitter of the three-dimensional model of the human body in the target video; and in the process of reconstructing the three-dimensional model of the human body, the data calculation complexity is reduced, thereby improving the reconstruction efficiency of the three-dimensional model of the human body.
[0073] See also Figure 3 , Figure 3 1 is a schematic diagram of a human body three-dimensional model reconstruction device 100 provided in an embodiment of the present application. The human body three-dimensional model reconstruction device 100 can be configured in a server or a computer device to execute the aforementioned human body three-dimensional model reconstruction method.
[0074] like Figure 3 As shown, the human body three-dimensional model reconstruction device 100 includes: a video acquisition module 110, a feature data extraction module 120, an attention calculation module 130, an image prediction module 140, and a video generation module 150.
[0075] The video acquisition module 110 is used to acquire sample videos, where the sample videos include human body images.
[0076] The feature data extraction module 120 is used to perform feature extraction processing on each frame image in the sample video based on the trained feature extraction model to obtain human feature data corresponding to each frame image.
[0077] The attention calculation module 130 is used to perform full connection and normalization processing on the human feature data corresponding to the input image based on the attention calculation network of the human body three-dimensional reconstruction model to obtain the feature weight value corresponding to the input image.
[0078] The image prediction module 140 is used to perform feature aggregation processing based on the feature aggregation network of the human body three-dimensional reconstruction model according to the feature weight value and the corresponding human body feature data to obtain a predicted frame image, and the human body features in the predicted frame image are different from the human body features of the corresponding frame image in the sample video.
[0079] The video generation module 150 is used to generate a target video according to the predicted frame image.
[0080] Exemplarily, the attention calculation module 130 includes a data dimension conversion submodule, a data splicing submodule, and a normalization processing submodule.
[0081] The data dimension conversion submodule is used to perform data dimension conversion processing on the human feature data of the input image based on the first fully connected layer of the attention calculation network to obtain converted human feature data, wherein the data dimension of the converted human feature data is smaller than the data dimension of the human feature data before the conversion.
[0082] The data splicing submodule is used to perform feature fusion processing on the converted human feature data corresponding to each input image based on the data splicing layer of the attention calculation network to obtain a human feature data set.
[0083] The normalization processing submodule is used to perform data dimension conversion on the human feature data set based on the full connection and normalization layer of the attention calculation network, and to perform normalization processing on the converted human feature data set to obtain the feature weight value of the input image.
[0084] Exemplarily, the data dimension conversion submodule is also used to perform data dimension conversion processing on the human feature data corresponding to each of the first image, the second image and the third image based on the first fully connected layer of the attention calculation network.
[0085] The data splicing submodule is also used to perform feature fusion and connection processing on the converted human feature data corresponding to the first image, the converted human feature data corresponding to the second image, and the converted human feature data corresponding to the third image based on the data splicing layer of the attention calculation network to obtain a human feature data set.
[0086] The normalization processing submodule is also used to perform data dimension conversion on the human feature data set based on the fully connected and normalized layers of the attention calculation network, and to perform normalization processing on the converted human feature data set to obtain a first feature weight value corresponding to the first image, a second feature weight value corresponding to the second image, and a third feature weight value corresponding to the third image.
[0087] Exemplarily, the image prediction module 140 includes a first feature calculation data determination submodule, a second feature calculation data determination submodule, and a feature aggregation submodule.
[0088] The first feature calculation data determination submodule is used to determine the first feature calculation data based on the feature aggregation network of the human body three-dimensional reconstruction model according to the first feature weight value, the second feature weight value and the third feature weight value.
[0089] The second feature calculation data determination submodule is used to determine second feature calculation data according to the human body feature data corresponding to the first image, the human body feature image corresponding to the second image and the human body feature data corresponding to the third image.
[0090] The feature aggregation submodule is used to perform feature aggregation processing on the first feature calculation data and the second feature calculation data to obtain the predicted frame image.
[0091] Exemplarily, the feature aggregation submodule is further used to perform convolution processing on the first feature calculation data and the second feature calculation data to obtain the predicted frame image.
[0092] Exemplarily, the video generation module 150 is further configured to sort the multiple prediction frame images in the order of image generation timing, so as to generate a target video according to the sorted prediction frame image sequence.
[0093] Exemplarily, the video generation module 150 is further configured to insert the predicted frame image corresponding to the second image into the predicted frame image sequence after the predicted frame image corresponding to the first image.
[0094] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0095] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0096] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Figure 4 Runs on the computer device shown.
[0097] See also Figure 4 , Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal.
[0098] like Figure 4 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.
[0099] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for reconstructing a three-dimensional human body model.
[0100] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0101] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any method for reconstructing a three-dimensional human body model.
[0102] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0103] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0104] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0105] Acquire a sample video, wherein the sample video includes a human body image;
[0106] Based on the trained feature extraction model, feature extraction processing is performed on each frame image in the sample video to obtain human feature data corresponding to each frame image;
[0107] Based on the attention calculation network of the human body 3D reconstruction model, the human body feature data corresponding to the input image is fully connected and normalized to obtain the feature weight value corresponding to the input image;
[0108] Based on the feature aggregation network of the human body 3D reconstruction model, feature aggregation processing is performed according to the feature weight values and the corresponding human body feature data to obtain a predicted frame image, wherein the human body features in the predicted frame image are different from the human body features of the corresponding frame image in the sample video;
[0109] Generate a target video according to the predicted frame image.
[0110] In one embodiment, the processor implements an attention calculation network based on a three-dimensional reconstruction model of a human body, performs full connection and normalization processing on human feature data corresponding to an input image, and obtains a feature weight value corresponding to the input image, and is used to implement:
[0111] Based on the first fully connected layer of the attention calculation network, performing data dimension conversion processing on the human feature data of the input image to obtain converted human feature data, wherein the data dimension of the converted human feature data is smaller than the data dimension of the human feature data before the conversion;
[0112] Based on the data concatenation layer of the attention computing network, feature fusion processing is performed on the converted human feature data corresponding to each input image to obtain a human feature data set;
[0113] Based on the fully connected and normalized layers of the attention calculation network, the human feature data set is converted in data dimension, and the converted human feature data set is normalized to obtain the feature weight value of the input image.
[0114] In one embodiment, the processor, when implementing the first fully connected layer based on the attention computing network, performs data dimension conversion processing on the human feature data of the input image to obtain the converted human feature data, is used to implement:
[0115] Based on the first fully connected layer of the attention computing network, respectively perform data dimension conversion processing on the human feature data corresponding to the first image, the second image, and the third image;
[0116] The processor, when implementing the data splicing layer based on the attention computing network, performs feature fusion processing on the converted human feature data corresponding to each input image to obtain a human feature data set, is used to implement:
[0117] Based on the data splicing layer of the attention computing network, feature fusion connection processing is performed on the converted human feature data corresponding to the first image, the converted human feature data corresponding to the second image, and the converted human feature data corresponding to the third image to obtain a human feature data set;
[0118] The processor is used to implement the following when implementing the fully connected and normalized layers based on the attention computing network, performing data dimension conversion on the human feature data set, and performing normalization processing on the converted human feature data set to obtain the feature weight value of the input image:
[0119] Based on the fully connected and normalized layers of the attention calculation network, the human feature data set is converted in data dimension, and the converted human feature data set is normalized to obtain a first feature weight value corresponding to the first image, a second feature weight value corresponding to the second image, and a third feature weight value corresponding to the third image.
[0120] In one embodiment, the processor implements a feature aggregation network based on the three-dimensional reconstruction model of the human body, performs feature aggregation processing according to the feature weight value and the corresponding human feature data, and obtains a predicted frame image, and is used to implement:
[0121] Based on the feature aggregation network of the three-dimensional reconstruction model of the human body, determining first feature calculation data according to the first feature weight value, the second feature weight value and the third feature weight value;
[0122] Determine second feature calculation data according to the human body feature data corresponding to the first image, the human body feature image corresponding to the second image, and the human body feature data corresponding to the third image;
[0123] Feature aggregation processing is performed on the first feature calculation data and the second feature calculation data to obtain the predicted frame image.
[0124] In one embodiment, when the processor performs feature aggregation processing on the first feature calculation data and the second feature calculation data to obtain the predicted frame image, it is used to implement:
[0125] Convolution processing is performed on the first feature calculation data and the second feature calculation data to obtain the predicted frame image.
[0126] In one embodiment, when the processor generates a target video according to the predicted frame image, it is used to implement:
[0127] The plurality of prediction frame images are sorted in the order of image generation timing, so as to generate a target video according to the sorted prediction frame image sequence.
[0128] In one embodiment, when the processor implements sorting the multiple predicted frame images according to the order of image generation timing, it is used to implement:
[0129] The predicted frame image corresponding to the second image is inserted after the predicted frame image corresponding to the first image in the predicted frame image sequence to complete the sorting of the predicted frame images.
[0130] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the method for reconstructing a three-dimensional human body model of the present application.
[0131] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0132] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0133] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.
[0134] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for reconstructing a three-dimensional human body model, characterized in that: The method for reconstructing the three-dimensional human body model comprises: Acquire a sample video, wherein the sample video includes a human body image; Based on the trained feature extraction model, feature extraction processing is performed on each frame image in the sample video to obtain human feature data corresponding to each frame image; Based on the attention calculation network of the human body 3D reconstruction model, the human body feature data corresponding to the input image is fully connected and normalized to obtain the feature weight value corresponding to the input image; Based on the feature aggregation network of the human body 3D reconstruction model, feature aggregation processing is performed according to the feature weight values and the corresponding human body feature data to obtain a predicted frame image, wherein the human body features in the predicted frame image are different from the human body features of the corresponding frame image in the sample video; Generate a target video according to the predicted frame image.
2. The method for reconstructing a three-dimensional human body model as claimed in claim 1, characterized in that: The attention calculation network based on the human body three-dimensional reconstruction model performs full connection and normalization processing on the human body feature data corresponding to the input image to obtain the feature weight value corresponding to the input image, including: Based on the first fully connected layer of the attention calculation network, performing data dimension conversion processing on the human feature data of the input image to obtain converted human feature data, wherein the data dimension of the converted human feature data is smaller than the data dimension of the human feature data before the conversion; Based on the data concatenation layer of the attention computing network, feature fusion processing is performed on the converted human feature data corresponding to each input image to obtain a human feature data set; Based on the fully connected and normalized layers of the attention calculation network, the human feature data set is converted in data dimension, and the converted human feature data set is normalized to obtain the feature weight value of the input image.
3. The method for reconstructing a three-dimensional human body model as claimed in claim 2, characterized in that: The input image includes a first image, a second image for indicating adjacent frames of the first image, and a third image for indicating adjacent frames of the second image. The first fully connected layer based on the attention calculation network performs data dimension conversion processing on the human feature data of the input image to obtain the converted human feature data, including: Based on the first fully connected layer of the attention computing network, respectively perform data dimension conversion processing on the human feature data corresponding to the first image, the second image, and the third image; The data concatenation layer based on the attention computing network performs feature fusion processing on the converted human feature data corresponding to each input image to obtain a human feature data set, including: Based on the data splicing layer of the attention computing network, feature fusion connection processing is performed on the converted human feature data corresponding to the first image, the converted human feature data corresponding to the second image, and the converted human feature data corresponding to the third image to obtain a human feature data set; The fully connected and normalized layer based on the attention computing network performs data dimension conversion on the human feature data set, and normalizes the converted human feature data set to obtain the feature weight value of the input image, including: Based on the fully connected and normalized layers of the attention calculation network, the human feature data set is converted in data dimension, and the converted human feature data set is normalized to obtain a first feature weight value corresponding to the first image, a second feature weight value corresponding to the second image, and a third feature weight value corresponding to the third image.
4. The method for reconstructing a three-dimensional human body model as claimed in claim 3, characterized in that: The feature aggregation network based on the human body 3D reconstruction model performs feature aggregation processing according to the feature weight value and the corresponding human body feature data to obtain a predicted frame image, including: Based on the feature aggregation network of the three-dimensional reconstruction model of the human body, determining first feature calculation data according to the first feature weight value, the second feature weight value and the third feature weight value; Determine second feature calculation data according to the human body feature data corresponding to the first image, the human body feature image corresponding to the second image, and the human body feature data corresponding to the third image; Feature aggregation processing is performed on the first feature calculation data and the second feature calculation data to obtain the predicted frame image.
5. The method for reconstructing a three-dimensional human body model as claimed in claim 4, characterized in that: The performing feature aggregation processing on the first feature calculation data and the second feature calculation data to obtain the predicted frame image includes: Convolution processing is performed on the first feature calculation data and the second feature calculation data to obtain the predicted frame image.
6. The method for reconstructing a three-dimensional human body model according to any one of claims 1 to 5, characterized in that: The generating a target video according to the predicted frame image comprises: The plurality of prediction frame images are sorted in the order of image generation timing, so as to generate a target video according to the sorted prediction frame image sequence.
7. The method for reconstructing a three-dimensional human body model according to claim 6, wherein: The sample video at least includes a first image and a second image, and the timing of the second image in the sample video is later than the timing of the first image in the sample video; the predicted frame image at least includes a predicted frame image corresponding to the first image and a predicted frame image corresponding to the second image; The step of sorting the plurality of predicted frame images according to the order of image generation timing includes: The predicted frame image corresponding to the second image is inserted after the predicted frame image corresponding to the first image in the predicted frame image sequence to complete the sorting of the predicted frame images.
8. A device for reconstructing a three-dimensional human body model, characterized in that: The reconstruction device of the human body three-dimensional model comprises: A video acquisition module, used to acquire a sample video, wherein the sample video includes a human body image; A feature data extraction module is used to perform feature extraction processing on each frame image in the sample video based on the trained feature extraction model to obtain human body feature data corresponding to each frame image; An attention calculation module is used to perform full connection and normalization processing on human feature data corresponding to an input image based on an attention calculation network of a human body three-dimensional reconstruction model to obtain a feature weight value corresponding to the input image; An image prediction module, configured to perform feature aggregation processing based on a feature aggregation network of the human body three-dimensional reconstruction model according to the feature weight values and corresponding human body feature data to obtain a predicted frame image, wherein the human body features in the predicted frame image are different from the human body features of the corresponding frame image in the sample video; The video generation module is used to generate a target video according to the predicted frame image.
9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the method for reconstructing a three-dimensional human body model as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for reconstructing a three-dimensional human body model as claimed in any one of claims 1 to 7 are implemented.