Virtual human three-dimensional model generation method and device, electronic equipment and storage medium

Abstract

The invention provides a virtual human three-dimensional model generation method and device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the following steps: acquiring a plurality of continuous video frames extracted from an initial video; respectively extracting static features in each video frame, and sequentially inputting each continuous static feature into a time convolution network for convolution processing to obtain each corresponding potential feature; performing combination processing on the static feature under each video frame and the corresponding potential feature to obtain each corresponding time sequence enhancement feature; performing nonlinear conversion on each time sequence enhancement feature to obtain a skin multi-person linear parameter corresponding to the target object in each video frame; and inputting each skin multi-person linear parameter into the skin multi-person linear model to obtain a virtual human three-dimensional image corresponding to each video frame, and generating a virtual human three-dimensional model based on a plurality of continuous virtual human three-dimensional models. The quality of the generated virtual human three-dimensional model can be improved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a method and device for generating a three-dimensional model of a virtual human, an electronic device, and a storage medium. Background Art

[0002] With the development of information technology, the application of virtual human 3D models is becoming more and more widespread, especially in the financial field. At present, virtual human 3D models are needed in many places in the financial field. For example, in banks or financial securities units, it is necessary to generate corresponding animations or videos based on virtual human 3D models to simulate online customer service personnel and improve user experience.

[0003] In the related art, the 3D model of a virtual human is generated from a video frame containing a target object by identifying the body parameters of the target object. However, the current process of reconstructing a 3D model of a virtual human from a video frame relies more on static features in a single video frame and is easily blocked by other objects in the video, resulting in poor temporal consistency and smoothness of the 3D model of the virtual human, and poor quality of the generated 3D model of the virtual human. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a method and device for generating a virtual human three-dimensional model, an electronic device and a storage medium, which can improve the quality of the generated virtual human three-dimensional model.

[0005] To achieve the above-mentioned purpose, the first aspect of an embodiment of the present application proposes a method for generating a three-dimensional model of a virtual human, comprising: obtaining a plurality of continuous video frames extracted from an initial video; extracting static features from each of the video frames respectively, and inputting the continuous static features into a preset temporal convolutional network in turn, performing convolution processing on the continuous static features to obtain corresponding potential features; combining the static features under each of the video frames with the corresponding potential features to obtain corresponding time-series enhancement features; performing nonlinear transformation on each of the time-series enhancement features to obtain skinned multi-person linear parameters corresponding to the target object in each of the video frames; inputting each of the skinned multi-person linear parameters into a pre-trained skinned multi-person linear model to obtain a three-dimensional image of a virtual human corresponding to each of the video frames, and generating a three-dimensional model of a virtual human based on the continuous plurality of the three-dimensional models of the virtual human.

[0006] In some embodiments, the static features in each of the video frames are combined with the corresponding latent features to obtain corresponding timing enhancement features, including: inputting the static features in each of the video frames and the corresponding latent features into a preset residual block; performing residual connection on the static features in each of the video frames and the corresponding latent features through the residual block to obtain corresponding timing enhancement features after combination.

[0007] In some embodiments, the nonlinear transformation of each of the timing enhancement features to obtain the skinned multi-person linear parameters corresponding to the target object in each of the video frames includes: inputting each of the timing enhancement features into a preset multi-layer perceptron, wherein the multi-layer perceptron is provided with a plurality of fully connected layers and a nonlinear activation function; performing full-connection processing on each of the timing enhancement features through the plurality of fully connected layers to obtain the corresponding fully connected features, and performing nonlinear transformation on the fully connected features through the nonlinear activation function to obtain the skinned multi-person linear parameters corresponding to the target object in each of the video frames.

[0008] In some embodiments, the skinned multi-person linear model is trained by the following steps, including: obtaining a plurality of continuous sample video frames; extracting sample static features from each of the sample video frames respectively, and inputting the continuous sample static features into the temporal convolutional network in turn, performing convolution processing on the continuous sample static features to obtain corresponding sample potential features; combining the sample static features under each of the sample video frames with the corresponding sample potential features to obtain corresponding sample temporal enhancement features; performing nonlinear transformation on each of the sample temporal enhancement features to obtain sample skinned multi-person linear parameters corresponding to the sample objects in each of the sample video frames; inputting each of the sample skinned multi-person linear parameters into the initial skinned multi-person linear model to obtain sample virtual human three-dimensional images corresponding to each of the video frames, and inputting the sample virtual human three-dimensional images into a preset discriminator to output a discrimination result; performing generative adversarial training on the initial skinned multi-person linear model based on the discrimination result until a training stop condition is met to obtain the trained skinned multi-person linear model.

[0009] In some embodiments, the generative adversarial training is performed on the initial skinned multi-person linear model based on the discrimination result until a training stop condition is met to obtain the trained skinned multi-person linear model, including: determining a first loss value according to the discrimination result; extracting the first visual geometry group information of each of the sample video frames, and extracting the second visual geometry group information of the corresponding sample virtual human three-dimensional image, and determining a second loss value based on the first visual geometry group information and the second visual geometry group information; obtaining the labeled skinned multi-person linear parameters corresponding to the sample objects in each of the sample video frames, and determining a third loss value based on the sample skinned multi-person linear parameters and the labeled skinned multi-person linear parameters; determining a target loss value based on the first loss value, the second loss value, and the third loss value, and performing generative adversarial training on the initial skinned multi-person linear model based on the target loss value until the training stop condition is met to obtain the trained skinned multi-person linear model.

[0010] In some embodiments, the discriminator is trained through the following steps, including: obtaining the label skinned multi-person linear parameters corresponding to the sample objects in each of the sample video frames; using the sample skinned multi-person linear parameters as negative samples and the label skinned multi-person linear parameters as positive samples, training the discriminator until the training stop condition is met, and obtaining the trained skinned multi-person linear model.

[0011] In some embodiments, generating a virtual human three-dimensional model based on a plurality of continuous virtual human three-dimensional models includes: determining scene information in the initial video; generating models for a plurality of continuous virtual human three-dimensional images based on the scene information to obtain a virtual human three-dimensional model wearing clothing corresponding to the scene information.

[0012] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes a device for generating a three-dimensional model of a virtual human, comprising: a video frame acquisition module, used to acquire multiple continuous video frames extracted from an initial video; a feature processing module, used to extract static features in each of the video frames respectively, and input each of the continuous static features into a preset temporal convolutional network in turn, and perform convolution processing on each of the continuous static features to obtain corresponding potential features; a feature combination module, used to combine the static features under each of the video frames with the corresponding potential features to obtain corresponding time-series enhancement features; a parameter generation module, used to perform nonlinear transformation on each of the time-series enhancement features to obtain skinned multi-person linear parameters corresponding to the target object in each of the video frames; a three-dimensional model generation module, used to input each of the skinned multi-person linear parameters into a pre-trained skinned multi-person linear model to obtain a three-dimensional image of a virtual human corresponding to each of the video frames, and generate a three-dimensional model of a virtual human based on the continuous multiple three-dimensional models of the virtual human.

[0013] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the method for generating a three-dimensional model of a virtual human as described in the first aspect above.

[0014] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method for generating a virtual human three-dimensional model as described in the first aspect above.

[0015] The method and device for generating a virtual human three-dimensional model proposed in the present application, the electronic device and the storage medium, obtain multiple continuous video frames extracted from the initial video, extract the static features in each video frame respectively, and input the continuous static features into the preset time convolution network in sequence, and perform convolution processing on the continuous static features. The time convolution network can effectively integrate the information of the previous frame, extract useful information from the previous static features, and generate smooth predictions through convolution, so as to obtain the corresponding potential features. Then, the static features under each video frame are combined with the corresponding potential features, and the information of the two dimensions of space and time is integrated to obtain the corresponding time-series enhancement features. Finally, the time-series enhancement features are nonlinearly transformed to obtain the skinned multi-person linear parameters corresponding to the target object in each video frame, and the skinned multi-person linear parameters are input into the pre-trained skinned multi-person linear model to obtain the virtual human three-dimensional image corresponding to each video frame, and the virtual human three-dimensional model is generated based on the continuous multiple virtual human three-dimensional models, so as to improve the quality of the generated virtual human three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a method for generating a virtual human three-dimensional model provided in an embodiment of the present application;

[0017] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0018] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0019] Figure 4 is a flowchart of a skinned multi-person linear model training process provided in an embodiment of the present application;

[0020] Figure 5 yes Figure 4 Flow chart of step S406 in FIG.

[0021] Figure 6 is a flowchart of the discriminator training process provided in an embodiment of the present application;

[0022] Figure 7 yes Figure 1 Flow chart of step S105 in FIG.

[0023] Figure 8 is a schematic diagram of a process for generating a three-dimensional model of a virtual human provided in an embodiment of the present application;

[0024] Fig. 9 It is a structural schematic diagram of a device for generating a three-dimensional model of a virtual human provided in an embodiment of the present application;

[0025] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0029] First, some nouns involved in this application are analyzed:

[0030] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0031] Motion capture, also known as motion capture, refers to recording the movement of an object in three-dimensional space and simulating its motion trajectory into a digital model.

[0032] Visual Geometry Group Network (VGGNet) is a deep convolutional neural network. Its core idea is to build a deep network structure by stacking multiple convolutional layers and pooling layers to improve the performance of image recognition.

[0033] Transparent Content Negotiation (TCN) is a deep learning model specifically designed for processing time series data. It is an efficient, stable and flexible time series data processing model. It extracts features from time series data through convolution operations, and uses structures such as dilated convolution and causal convolution to capture long-term dependencies, providing a new solution for the analysis and prediction of time series data.

[0034] Skinned Multi-Person Linear Model (SMPL) is a skinned, vertex-based 3D human model that can accurately represent different shapes and poses of the human body. The SMPL model can define the shape of the human body, such as fatness, thinness, height, and posture of human body movements. In order to define the movement of a human body, each movable joint of the human body needs to be parameterized. When the parameters of a joint point are changed, the posture of the human body will change accordingly.

[0035] The SMPL model is a statistical model that uses the average human body mesh calculated from a large number of scanned human body shapes. The model describes the human body through two types of statistical parameters:

[0036] 1. Shape parameters: A set of shape parameters has 10 dimensions of values ​​to describe a person's shape. The value of each dimension can be interpreted as an indicator of the human body shape, such as height, fatness, etc.

[0037] 2. Pose parameters: A set of pose parameters has 24×3 dimensional numbers to describe the posture of the human body at a certain moment. The 24 represents the 24 defined human joints, and the 3 refers to the axis-angle representation of the rotation angle of the node relative to its parent node.

[0038] In the related art, the 3D model of a virtual human is generated by identifying the body parameters of the target object from the video frame containing the target object, or by collecting the data of the actual person through the motion capture device. However, the scheme of collecting through the motion capture device requires a lot of manpower and material resources, and requires professional equipment and teams, which is very costly. At present, the process of reconstructing the 3D model of a virtual human from the video frame is more dependent on the static features in a single video frame, and is easily blocked by other objects in the video, resulting in the poor consistency and smoothness of the action time of the 3D model of the virtual human, making the quality of the generated 3D model of the virtual human poor.

[0039] Based on this, the embodiments of the present application provide a method and device for generating a virtual human three-dimensional model, an electronic device and a storage medium, aiming to improve the quality of the generation of the virtual human three-dimensional model.

[0040] The method and device for generating a three-dimensional model of a virtual human, an electronic device, and a storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the method for generating a three-dimensional model of a virtual human in the embodiments of the present application is described.

[0041] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0042] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0043] The method for generating a three-dimensional model of a virtual person provided in an embodiment of the present application relates to the field of artificial intelligence technology. The method for generating a three-dimensional model of a virtual person provided in an embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for generating a three-dimensional model of a virtual person, etc., but is not limited to the above forms.

[0044] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0045] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. For example, when obtaining the initial video frame, it is necessary to obtain the user's permission or consent first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0046] Figure 1 is an optional flowchart of the method for generating a virtual human three-dimensional model provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0047] Step S101, obtaining a plurality of continuous video frames extracted from an initial video;

[0048] Step S102, extracting static features from each video frame respectively, and inputting each continuous static feature into a preset temporal convolutional network in sequence, performing convolution processing on each continuous static feature to obtain corresponding potential features;

[0049] Step S103, combining the static features of each video frame with the corresponding latent features to obtain corresponding temporal enhancement features;

[0050] Step S104, performing nonlinear transformation on each temporal enhancement feature to obtain skinned multi-person linear parameters corresponding to the target object in each video frame;

[0051] Step S105, inputting each skinned multi-person linear parameter into the pre-trained skinned multi-person linear model to obtain a virtual human 3D image corresponding to each video frame, and generating a virtual human 3D model based on a plurality of continuous virtual human 3D models.

[0052] In the above-mentioned step S101, the initial video is a video containing the target object, and the video is the original data source for generating the virtual human three-dimensional model. Exemplarily, the initial video can be a video of any format, and the embodiment of the present application is not specifically limited to this. Since the video is composed of a series of continuous pictures (frames), each frame represents a static image of the video at a certain moment, so the application can decompose the initial video into multiple continuous separate static pictures, and obtain multiple continuous video frames, which are continuous in time, and provide the position and action information of the target object at different time points, which is the basis for subsequent processing and analysis. Therefore, by obtaining multiple continuous video frames, the embodiment of the present application can capture the continuous action and posture changes of the target object over a period of time.

[0053] In the above step S102, in the embodiment of the present application, for each video frame obtained from the initial video, it is first necessary to extract static features therein. Static features are features that can describe information such as the posture, shape, texture, etc. of the target object at a certain moment. Exemplarily, static features can be extracted by various computer vision technologies, such as edge detection, corner detection, feature point matching, deep learning models (such as convolutional neural networks CNN), etc.

[0054] The temporal convolutional network is a deep learning model specifically used to process time series data. It can capture the temporal dependency in time series data, that is, the correlation between different time points, through convolution operations. In an embodiment of the present application, the static features extracted from each video frame can be arranged in their time order in the video to form a static feature sequence, and then this static feature sequence is used as input and sequentially input into the preset temporal convolutional network. The input layer of the temporal convolutional network receives this sequence and processes it through the convolutional layer to extract potential temporal dependency features.

[0055] Furthermore, each convolution kernel in the temporal convolutional network generates a new feature map, which contains feature representations of different levels in the original static feature sequence, and also captures temporal dependencies, so that the temporal convolutional network can obtain useful information from the static features of the previous frame, and when the posture of the target object is unclear or there are occluders that partially occlude the body, the previous information can help solve this problem, thereby generating a smooth prediction. After the convolution processing of the temporal convolutional network, a series of potential features will be output, which not only contain the static information of the target object, but also incorporate the continuity information in time.

[0056] In the above step S103, after obtaining the static features and the corresponding latent features in each video frame, the embodiment of the present application can combine the static features and the corresponding latent features in each video frame, for example, splicing, weighted summation or other forms of fusion operations can be performed, the purpose of which is to integrate information in both spatial and temporal dimensions to improve the accuracy of parameter estimation in subsequent steps. The combined features are called time-series enhanced features, so the obtained time-series enhanced features contain both spatial information of the target object, such as shape, posture, etc., and temporal information, such as motion continuity, smoothness, etc.

[0057] In the above step S104, after obtaining the temporal enhancement features of the target object in each video frame, in an embodiment of the present application, each temporal enhancement feature can be nonlinearly transformed. Exemplarily, in an embodiment of the present application, nonlinear transformation can be performed in a variety of ways, for example, a deep learning model (such as a multi-layer perceptron, a neural network, etc.) can be used to perform nonlinear mapping on the temporal enhancement features, with the purpose of converting high-dimensional temporal enhancement features into points in a low-dimensional SMPL parameter space.

[0058] Finally, the SMPL parameters of the target object in each video frame, namely the skinned multi-person linear parameters, can be obtained. Further, the skinned multi-person linear parameters can include posture parameters and shape parameters, which can be used to describe the three-dimensional shape and posture of the target object.

[0059] In the above step S105, the embodiment of the present application can input the SMPL parameters corresponding to each video frame, that is, the skinned multi-person linear parameters, into the pre-trained skinned multi-person linear model (SMPL model) to generate a virtual human three-dimensional image corresponding to each video frame. The skinned multi-person linear model is a pre-trained three-dimensional human model that can generate a corresponding three-dimensional human image based on the input SMPL parameters. In the embodiment of the present application, the SMPL parameters corresponding to each video frame are input into the SMPL model to generate a virtual human three-dimensional image corresponding to the video frame.

[0060] Finally, since the embodiment of the present application processes multiple continuous video frames, a series of continuous virtual human three-dimensional images can be obtained, and the final virtual human three-dimensional model can be generated based on the continuous multiple virtual human three-dimensional images. The embodiment of the present application not only improves the quality of the generated virtual human three-dimensional model, but also enhances the temporal consistency and smoothness of its movements.

[0061] In summary, through the above steps S101 to S105, the embodiment of the present application obtains multiple continuous video frames extracted from the initial video, extracts static features in each video frame respectively, and inputs the continuous static features into the preset time convolution network in sequence, and performs convolution processing on the continuous static features. The time convolution network can effectively integrate the information of the previous frame, extract useful information from the previous static features, and generate smooth predictions through convolution, thereby obtaining the corresponding potential features. Then, the static features under each video frame are combined with the corresponding potential features, and the information of the two dimensions of space and time is integrated to obtain the corresponding time-series enhancement features. Finally, each time-series enhancement feature is nonlinearly transformed to obtain the skinned multi-person linear parameters corresponding to the target object in each video frame, and each skinned multi-person linear parameter is input into the pre-trained skinned multi-person linear model to obtain the virtual human three-dimensional image corresponding to each video frame, and a virtual human three-dimensional model is generated based on the continuous multiple virtual human three-dimensional models, so as to improve the quality of the generated virtual human three-dimensional model.

[0062] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S202:

[0063] Step S201, inputting the static features and corresponding latent features of each video frame into a preset residual block;

[0064] Step S202, performing residual connection on the static features and corresponding latent features in each video frame through the residual block to obtain the corresponding temporal enhancement features after combination.

[0065] In the above steps, the residual block is a network structure in deep learning, which adds the input directly to the output through a "skip connection" (or "residual connection"). This structure helps to solve the gradient vanishing or gradient exploding problems that may occur in the training process of deep neural networks, thereby allowing the training of deeper networks. In an embodiment of the present application, the static features and corresponding potential features of each video frame can be used as input and sent to a preset residual block. Inside the residual block, the input features will be processed through a series of convolutional layers, activation functions, etc. to extract higher-level features. At the same time, due to the existence of the residual block, the input features will also be directly added to the output features through a skip connection, which helps to retain useful information in the input features and prevent information from being lost in the deep network.

[0066] Next, in the embodiment of the present application, the static features of each video frame are effectively combined with the corresponding potential features through the residual connection inside the residual block. This combination method not only retains the useful information in the static features and potential features, but also introduces additional information through the residual connection, making the generated time-series enhancement features more accurate and comprehensive. Finally, after the processing of the residual block and the combination of the residual connection, the corresponding time-series enhancement features after the combination are obtained. Therefore, the time-series enhancement features integrate the information of the two dimensions of space (static features) and time (potential features), which provides strong support for the subsequent nonlinear transformation and the generation of virtual human three-dimensional models.

[0067] See also Figure 3 In some embodiments, step S104 may include but is not limited to steps S301 to S302:

[0068] Step S301, inputting each time series enhancement feature into a preset multi-layer perceptron, wherein the multi-layer perceptron is provided with a plurality of fully connected layers and a nonlinear activation function;

[0069] Step S302, fully connect each temporal enhancement feature through multiple fully connected layers to obtain corresponding fully connected features, and nonlinearly transform the fully connected features through a nonlinear activation function to obtain skinned multi-person linear parameters corresponding to the target object in each video frame.

[0070] In the above steps, the multilayer perceptron is a feedforward neural network, which is composed of multiple fully connected layers and nonlinear activation functions, and can learn the complex mapping relationship between input and output. In the embodiment of the present application, each time series enhancement feature is used as input and sent to a preset multilayer perceptron, which processes the input time series enhancement features layer by layer, and each layer contains multiple neurons, which are connected to the neurons of the previous layer in a fully connected manner.

[0071] In each layer of the multilayer perceptron, the input features are connected to the neurons of the previous layer in a fully connected manner, so that the neurons of each layer can receive all the information of the previous layer. Through the layer-by-layer processing of multiple fully connected layers, the multilayer perceptron can extract high-level information from the input features. The nonlinear activation function is a key component in the neural network. It can introduce nonlinear factors so that the neural network can learn the complex mapping relationship between input and output. Nonlinear activation functions include ReLU, Sigmoid, and Tanh. In an embodiment of the present application, a nonlinear activation function (such as ReLU) suitable for the current task can be selected to perform nonlinear transformation on the features output by the fully connected layer. After layer-by-layer processing and nonlinear transformation of the multilayer perceptron, the skinned multi-person linear parameters (SMPL parameters) corresponding to the target object in each video frame are obtained.

[0072] Therefore, the embodiment of the present application introduces a multi-layer perceptron and utilizes its powerful nonlinear mapping capability to convert the temporal enhancement features into skinned multi-person linear parameters, so that the skinned multi-person linear parameters not only retain the useful information in the input features, but also introduce additional nonlinear factors through nonlinear activation functions, making the generated SMPL parameters more accurate and comprehensive.

[0073] See also Figure 4 In some embodiments, the skinned multi-person linear model is trained by the following steps, including but not limited to steps S401 to S406:

[0074] Step S401, obtaining a plurality of continuous sample video frames;

[0075] Step S402, extracting sample static features from each sample video frame respectively, and inputting each continuous sample static feature into the temporal convolutional network in sequence, performing convolution processing on each continuous sample static feature, and obtaining corresponding each sample potential feature;

[0076] Step S403, combining the sample static features of each sample video frame with the corresponding sample latent features to obtain corresponding temporal enhancement features of each sample;

[0077] Step S404, performing nonlinear transformation on each sample temporal enhancement feature to obtain sample skinning multi-person linear parameters corresponding to the sample object in each sample video frame;

[0078] Step S405, inputting each sample skinned multi-person linear parameter into the initial skinned multi-person linear model to obtain a sample virtual human 3D image corresponding to each video frame, and inputting the sample virtual human 3D image into a preset discriminator to output a discrimination result;

[0079] Step S406, performing generative adversarial training on the initial skinned multi-person linear model based on the discrimination result until a training stop condition is met, thereby obtaining a trained skinned multi-person linear model.

[0080] In the above steps, the sample video is a video in the training process, similar to the initial video in the above embodiment, and similarly, the sample video contains a sample object, and the sample object is also similar to the target object in the above embodiment. Exemplarily, similar to the initial video, the sample video can also be a video of any format, and the embodiments of the present application are not specifically limited to this. Since the video is composed of a series of continuous pictures (frames), each frame represents a static image of the video at a certain moment, so the present application can decompose the sample video frame into a plurality of continuous separate static pictures, and obtain a plurality of continuous sample video frames, which are continuous in time, and provide the position and action information of the sample object at different time points, which is the basis for subsequent processing and analysis. Therefore, by obtaining a plurality of continuous sample video frames, the embodiments of the present application can capture the continuous action and posture changes of the sample object over a period of time.

[0081] During the training process, for each sample video frame obtained from the sample video in the embodiment of the present application, it is first necessary to extract the sample static features therein. The sample static features are similar to the static features in the above embodiment, and are features that can describe the posture, shape, texture and other information of the sample object at a certain moment. Exemplarily, the sample static features can also be extracted by various computer vision technologies, which will not be described in detail in the embodiment of the present application.

[0082] During the training process, the sample static features extracted from each sample video frame can be arranged in their chronological order in the video to form a static feature sequence, and then this static feature sequence is used as input and input into the temporal convolutional network in sequence. The input layer of the temporal convolutional network receives this sequence and processes it through the convolutional layer to extract potential time-dependent features.

[0083] Furthermore, similar to the processing of static features by the temporal convolutional network in the above embodiment, the temporal convolutional network in the training process will output a series of sample potential features, which not only contain the static information of the sample objects, but also incorporate the temporal continuity information.

[0084] During the training process, after obtaining the sample static features and the corresponding sample latent features under each sample video frame, the embodiment of the present application combines the sample static features and the corresponding sample latent features under each sample video frame, similar to the way of combining the static features and the latent features in the above embodiment. The combined features are called sample timing enhancement features. Therefore, the obtained sample timing enhancement features include both the spatial information of the sample object, such as shape, posture, etc., and the temporal information, such as motion continuity and smoothness, etc.

[0085] After obtaining the sample temporal enhancement features of the sample objects in each sample video frame, in the embodiment of the present application, each sample temporal enhancement feature can be nonlinearly transformed. Finally, the SMPL parameters of the sample objects in each sample video frame, that is, the sample skinned multi-person linear parameters, can be obtained. Furthermore, the sample skinned multi-person linear parameters can also include posture parameters and shape parameters, which can be used to describe the three-dimensional shape and posture of the sample object.

[0086] Next, the embodiment of the present application can input the SMPL parameters corresponding to each sample video frame, that is, the sample skinned multi-person linear parameters into the initial skinned multi-person linear model (SMPL model), that is, the skinned multi-person linear model that has not yet been trained, to generate a sample virtual human three-dimensional image corresponding to each sample video frame.

[0087] In order to evaluate the quality of the generated sample virtual human three-dimensional image, an adversarial training method is introduced in the embodiment of the present application to train the skinned multi-person linear model. Specifically, the embodiment of the present application uses the skinned multi-person linear model as a generator and introduces a preset discriminator. The discriminator is a trained neural network that can distinguish between real human images and images generated by the model. The task of the discriminator is to judge as accurately as possible whether the input image is real or generated. Based on this, in the embodiment of the present application, the sample virtual human three-dimensional image can be input into the preset discriminator, and the discriminator will output a discrimination result, for example, a probability value or a classification label, indicating the probability that the input sample virtual human three-dimensional image is a real image or the degree to which it is considered to be a real image.

[0088] In each training iteration, the embodiment of the present application updates the parameters of the skinned multi-person linear model according to the discrimination result of the discriminator. If the discriminator can easily distinguish between the generated image and the real image, then it is known that the generated image is not realistic enough and the parameters of the model need to be adjusted to generate a better image. This update is usually achieved through a back propagation algorithm, which calculates the loss of the discrimination result and adjusts the weights and biases of the model accordingly. The training process will continue until a certain stop condition is met. This condition may be that a preset number of iterations has been reached, the quality of the generated image has reached a certain threshold, or the discriminator can no longer significantly improve the ability to distinguish between real images and generated images. Once the training stop condition is met, the embodiment of the present application obtains a trained skinned multi-person linear model.

[0089] See also Figure 5 In some embodiments, step S406 may include but is not limited to steps S501 to S504:

[0090] Step S501, determining a first loss value according to the discrimination result;

[0091] Step S502, extracting first visual geometry group information of each sample video frame, and extracting second visual geometry group information of the corresponding sample virtual human 3D image, and determining a second loss value based on the first visual geometry group information and the second visual geometry group information;

[0092] Step S503, obtaining the label skinned multi-person linear parameters corresponding to the sample objects in each sample video frame, and determining a third loss value based on the sample skinned multi-person linear parameters and the label skinned multi-person linear parameters;

[0093] Step S504, determining a target loss value based on the first loss value, the second loss value, and the third loss value, and performing generative adversarial training on the initial skinned multi-person linear model based on the target loss value until a training stop condition is met, thereby obtaining a trained skinned multi-person linear model.

[0094] In the above steps, the first loss value is the adversarial loss value, which is generated after the discrimination result is input into the adversarial loss function. It is used to measure the difference between the generated virtual human three-dimensional image and the real image, as well as the ability of the discriminator to distinguish these images. It can be calculated by comparing the difference between the discrimination result output by the discriminator and the real label.

[0095] The second loss value is a geometric loss or a visual consistency loss, which is used to ensure that the generated virtual human three-dimensional image is geometrically consistent with the target object in the original video frame. The second loss value is calculated by comparing the difference between the first visual geometry group information extracted from the sample video frame and the second visual geometry group information extracted from the generated sample virtual human three-dimensional image. Exemplarily, VGG16 can be introduced in an embodiment of the present application to perceive the visual geometry group information, including inputting each sample video frame into VGG16 to obtain the corresponding first visual geometry group information, and also inputting the sample virtual human three-dimensional image into VGG16 to obtain the corresponding second visual geometry group information.

[0096] The third loss value is a regression loss or a parameter loss, which is used to ensure that the generated skinned multi-person linear parameters are consistent with the preset label skinned multi-person linear parameters, that is, the real or expected parameters. The third loss value is calculated by comparing the difference between the sample skinned multi-person linear parameters and the label skinned multi-person linear parameters. Among them, the label skinned multi-person linear parameters are parameters that are manually marked for the shape and posture of the sample object in the sample video frame, which can be obtained through a motion capture device.

[0097] The target loss value is the total loss value after combining the first loss value, the second loss value and the third loss value, and is used to guide the training process of the skinned multi-person linear model. Exemplarily, the target loss value can be calculated by adding or weighted adding the first loss value, the second loss value and the third loss value, and the choice of weight depends on the specific application scenario and training objectives. During the training process, the target loss value is propagated to each layer of the model through the back propagation algorithm, and the weights and biases of the model are updated to minimize the target loss value. This process will continue until the training stop condition is met, such as reaching a preset number of iterations, the target loss value converges, etc.

[0098] For example, L represents the target loss value, and L adv Indicates the first loss value, L v Indicates the second loss value, L smpl represents the third loss value. Therefore, the target loss value is as follows:

[0099] L=L adv +L smpl +L v

[0100] Among them, use x i Represents the first visual geometry group information, using y i represents the second visual geometry group information, n is the number of video frames, so the second loss value can be expressed by |oss(x, y), so the second loss value is as follows:

[0101] loss(x,y)=1 / n∑|x i -y i |

[0102] The sample skinned multi-person linear parameters include (β, θ), where β is the morphological parameter and θ is the posture parameter. The corresponding label skinned multi-person linear parameters are: T is the number of postures, so the third mode loss value is as follows:

[0103]

[0104] See also Figure 6 In some embodiments, the discriminator is trained by the following steps, including but not limited to steps S601 to S602:

[0105] Step S601, obtaining label-skinned multi-person linear parameters corresponding to sample objects in each sample video frame;

[0106] Step S602, using the sample skinned multi-person linear parameters as negative samples and the label skinned multi-person linear parameters as positive samples, training the discriminator until the training stop condition is met, and obtaining a trained skinned multi-person linear model.

[0107] In the above steps, the purpose of training the discriminator is to enable it to accurately distinguish the generated skinned multi-person linear parameters from the real labeled skinned multi-person linear parameters. Through training, the discriminator will learn to identify which parameters are real and which are generated by the generator. Therefore, in the process of training the discriminator, the embodiment of the present application first needs to obtain the labeled skinned multi-person linear parameters corresponding to the sample objects in each sample video frame. The labeled skinned multi-person linear parameters are parameters that are manually marked for the shape and posture of the sample objects in the sample video frames, which can be obtained through a motion capture device.

[0108] Next, in the embodiment of the present application, the sample-skinned multi-person linear parameters are used as negative samples, and the label-skinned multi-person linear parameters are used as positive samples. The positive and negative samples are input into the discriminator, and the output of the discriminator is obtained. During the training process, a loss function (such as cross entropy loss) is used to calculate the difference between the output of the discriminator and the true label, and the weight of the discriminator is updated through the back propagation algorithm to minimize this difference, until the performance of the discriminator on the validation set reaches a certain threshold or the training reaches a preset number of iterations, and the trained discriminator is obtained.

[0109] Eventually, the discriminator will learn to accurately distinguish the generated skinned multi-person linear parameters from the real labeled skinned multi-person linear parameters. In subsequent generative adversarial training, the output of the discriminator will be used as part of the skinned multi-person linear model training to guide the generator to generate more and more realistic skinned multi-person linear parameters, thereby improving the performance of the entire system.

[0110] See also Figure 7 In some embodiments, step S105 may include but is not limited to steps S701 to S702:

[0111] Step S701, determining scene information in an initial video;

[0112] Step S702: Generate a model of a plurality of continuous three-dimensional images of a virtual human based on the scene information to obtain a three-dimensional model of a virtual human wearing clothing corresponding to the scene information.

[0113] In the above steps, in order to further enhance the realism and adaptability of the virtual human model, especially for applications in specific scenarios, the embodiments of the present application can introduce scene information so that the generated three-dimensional model of the virtual human can be more closely integrated into and reflect the characteristics of its environment.

[0114] Specifically, the embodiments of the present application first need to determine the scene information in the initial video. Scene information includes, but is not limited to, background environment, lighting conditions, decoration style, furniture layout, and other possible environmental factors, such as weather, time (day or night), etc. Exemplarily, the embodiments of the present application can obtain scene information in a variety of ways, such as using image recognition and machine learning algorithms to analyze video frames frame by frame to identify key scene elements; or, segment the video frames into different semantic areas, such as the sky, the ground, buildings, people, etc., in order to more accurately extract scene information; or, it can also combine multiple frames of information and use deep learning models to understand the overall layout and dynamic changes of the scene, such as the flow of people and vehicle movement.

[0115] Then, after obtaining a preliminary three-dimensional image of the virtual person, the appearance of the virtual person can be further customized and optimized according to the extracted scene information, especially in terms of clothing and accessories, to ensure that the virtual person can perfectly integrate into the scene in which it is located. Among them, the embodiment of the present application can select suitable clothing for the virtual person according to the style, era background and possible occupational characteristics of the scene; it can also adjust the texture and material of the virtual person's clothing according to the lighting conditions and overall atmosphere of the scene to make it more realistic; it can also add appropriate accessories such as hats, glasses, watches, etc. to the virtual person according to the details of the scene, such as the social status of the character, the type of activity, etc.; it can also simulate the interaction between the virtual person and the scene environment, such as holding objects, sitting on a chair, etc., to enhance the realism and interactivity of the virtual person; in addition, if the scene information contains dynamic changes (such as weather changes, time lapse, etc.), the appearance of the virtual person needs to be dynamically adjusted to maintain consistency with the scene.

[0116] In summary, in the embodiments of the present application, the virtual human three-dimensional model is intelligently generated by comprehensive scene information, so that the generated virtual human three-dimensional model can not only accurately reflect the movements and postures of the target object, but also more naturally integrate into the scene in which it is located, thereby improving the user experience and the practicality of the model.

[0117] The method for generating the above-mentioned virtual human three-dimensional model is described below by way of an example.

[0118] See also Figure 8 , Figure 8 Schematic diagram of the virtual human 3D model generation process provided by the embodiment of the present application. The embodiment of the present application can save the initial video frame by frame in sequence as I0, I1...In, use the resnet50 pre-trained model to extract features, input each video frame into resnet50, and output the static features corresponding to each video frame. Furthermore, the embodiment of the present application can record the static features extracted to the tth frame as ft (2048 dimensions).

[0119] Subsequently, each continuous static feature is input into the preset temporal convolutional network in turn, and convolution processing is performed on each continuous static feature. The temporal convolutional network can effectively integrate the information of the previous frame, extract useful information from the previous static features, and generate smooth predictions through convolution, thereby obtaining the corresponding potential features.

[0120] Next, the static features of each video frame are combined with the corresponding potential features in the residual block for processing, so that the information of the two dimensions of space and time can be integrated to obtain the corresponding temporal enhancement features. Finally, each temporal enhancement feature is input into the multi-layer perceptron for nonlinear transformation to obtain the skinned multi-person linear parameters (SMPL parameters) corresponding to the target object in each video frame.

[0121] Finally, each skinned multi-person linear parameter is input into the pre-trained skinned multi-person linear model to obtain a virtual human three-dimensional image corresponding to each video frame, and a virtual human three-dimensional model is generated based on multiple continuous virtual human three-dimensional models, thereby improving the quality of the generated virtual human three-dimensional model.

[0122] See also Fig. 9 The embodiment of the present application further provides a device for generating a virtual human three-dimensional model, which can implement the above-mentioned method for generating a virtual human three-dimensional model. The device includes:

[0123] The video frame acquisition module 901 is used to acquire a plurality of continuous video frames extracted from the initial video;

[0124] The feature processing module 902 is used to extract static features from each video frame respectively, and input each continuous static feature into a preset temporal convolution network in sequence, perform convolution processing on each continuous static feature, and obtain corresponding potential features;

[0125] The feature combining module 903 is used to combine the static features of each video frame with the corresponding latent features to obtain corresponding temporal enhancement features;

[0126] The parameter generation module 904 is used to perform nonlinear transformation on each temporal enhancement feature to obtain the skinned multi-person linear parameters corresponding to the target object in each video frame;

[0127] The 3D model generation module 905 is used to input each skinned multi-person linear parameter into the pre-trained skinned multi-person linear model to obtain a virtual human 3D image corresponding to each video frame, and generate a virtual human 3D model based on multiple continuous virtual human 3D models.

[0128] The device for generating a virtual human three-dimensional model can implement the method for generating a virtual human three-dimensional model in the above-mentioned embodiment, by acquiring multiple continuous video frames extracted from the initial video, extracting the static features in each video frame respectively, and inputting the continuous static features into a preset temporal convolutional network in sequence, and performing convolution processing on the continuous static features. The temporal convolutional network can effectively integrate the information of the previous frame, extract useful information from the previous static features, and generate a smooth prediction through convolution, thereby obtaining the corresponding potential features. Then, the static features under each video frame are combined with the corresponding potential features, and the information of the two dimensions of space and time is integrated, thereby obtaining the corresponding temporal enhancement features, and finally, the temporal enhancement features are nonlinearly transformed to obtain the skinned multi-person linear parameters corresponding to the target object in each video frame, and the skinned multi-person linear parameters are input into the pre-trained skinned multi-person linear model to obtain the virtual human three-dimensional image corresponding to each video frame, and the virtual human three-dimensional model is generated based on the continuous multiple virtual human three-dimensional models, so as to improve the quality of the generated virtual human three-dimensional model.

[0129] The specific implementation of the virtual human three-dimensional model generation device is basically the same as the specific implementation of the virtual human three-dimensional model generation method described above, and will not be repeated here. On the premise of meeting the requirements of the embodiment of the present application, the virtual human three-dimensional model generation device can also be provided with other functional modules to implement the virtual human three-dimensional model generation method in the above embodiment.

[0130] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned method for generating a virtual human three-dimensional model when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0131] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0132] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0133] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the method for generating a virtual human three-dimensional model in the embodiments of this application;

[0134] Input / output interface 1003, used to implement information input and output;

[0135] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0136] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0137] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0138] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for generating a three-dimensional model of a virtual human.

[0139] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0140] The method and device for generating a virtual human three-dimensional model provided by the embodiment of the present application, the electronic device and the storage medium, obtain multiple continuous video frames extracted from the initial video, extract the static features in each video frame respectively, and input the continuous static features into the preset time convolution network in sequence, and perform convolution processing on the continuous static features. The time convolution network can effectively integrate the information of the previous frame, extract useful information from the previous static features, and generate smooth predictions through convolution, so as to obtain the corresponding potential features. Then, the static features under each video frame are combined with the corresponding potential features, and the information of the two dimensions of space and time is integrated to obtain the corresponding time-series enhancement features. Finally, the time-series enhancement features are nonlinearly transformed to obtain the skinned multi-person linear parameters corresponding to the target object in each video frame, and the skinned multi-person linear parameters are input into the pre-trained skinned multi-person linear model to obtain the virtual human three-dimensional image corresponding to each video frame, and the virtual human three-dimensional model is generated based on the continuous multiple virtual human three-dimensional models, so as to improve the quality of the generated virtual human three-dimensional model.

[0141] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0142] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0143] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0145] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0146] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0151] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for generating a virtual human three-dimensional model, characterized in that: include: Obtain multiple consecutive video frames extracted from the initial video; Extracting static features from each of the video frames respectively, and inputting each of the continuous static features into a preset temporal convolutional network in sequence, performing convolution processing on each of the continuous static features, and obtaining corresponding potential features; Combining the static features of each video frame with the corresponding latent features to obtain corresponding temporal enhancement features; Performing nonlinear transformation on each of the temporal enhancement features to obtain skinned multi-person linear parameters corresponding to the target object in each of the video frames; Input each of the skinned multi-person linear parameters into a pre-trained skinned multi-person linear model to obtain a virtual human three-dimensional image corresponding to each of the video frames, and generate a virtual human three-dimensional model based on a plurality of continuous virtual human three-dimensional models.

2. The method for generating a virtual human three-dimensional model according to claim 1, characterized in that: The combining and processing the static features in each video frame with the corresponding potential features to obtain corresponding temporal enhancement features includes: Inputting the static features and the corresponding potential features in each video frame into a preset residual block; The static features and the corresponding latent features in each video frame are residually connected through the residual block to obtain the corresponding temporal enhancement features after combination.

3. The method for generating a virtual human three-dimensional model according to claim 1, characterized in that: The nonlinear transformation of each of the temporal enhancement features to obtain the skinned multi-person linear parameters corresponding to the target object in each of the video frames includes: Inputting each of the time series enhancement features into a preset multi-layer perceptron, wherein the multi-layer perceptron is provided with a plurality of fully connected layers and a nonlinear activation function; Each of the temporal enhancement features is fully connected through multiple fully connected layers to obtain corresponding fully connected features, and the fully connected features are nonlinearly transformed through the nonlinear activation function to obtain skinned multi-person linear parameters corresponding to the target objects in each of the video frames.

4. The method for generating a virtual human three-dimensional model according to claim 1, characterized in that: The skinned multi-person linear model is trained by the following steps, including: Acquire multiple consecutive sample video frames; Extracting sample static features from each of the sample video frames respectively, and inputting each of the continuous sample static features into the temporal convolutional network in sequence, performing convolution processing on each of the continuous sample static features, and obtaining corresponding sample potential features; Combining the sample static features under each sample video frame with the corresponding sample potential features to obtain corresponding temporal enhancement features of each sample; Performing nonlinear transformation on each of the sample temporal enhancement features to obtain sample mask multi-person linear parameters corresponding to the sample objects in each of the sample video frames; Inputting each of the sample skinned multi-person linear parameters into the initial skinned multi-person linear model to obtain a sample virtual human 3D image corresponding to each of the video frames, and inputting the sample virtual human 3D image into a preset discriminator to output a discrimination result; Generative adversarial training is performed on the initial skinned multi-person linear model based on the discrimination result until a training stop condition is met, thereby obtaining the trained skinned multi-person linear model.

5. The method for generating a virtual human three-dimensional model according to claim 4, characterized in that: The generating adversarial training is performed on the initial skinned multi-person linear model based on the discrimination result until a training stop condition is met to obtain the trained skinned multi-person linear model, including: Determine a first loss value according to the discrimination result; Extracting the first visual geometry group information of each of the sample video frames, and extracting the second visual geometry group information of the corresponding sample virtual human three-dimensional image, and determining a second loss value based on the first visual geometry group information and the second visual geometry group information; Obtaining label skinned multi-person linear parameters corresponding to the sample objects in each of the sample video frames, and determining a third loss value based on the sample skinned multi-person linear parameters and the label skinned multi-person linear parameters; A target loss value is determined based on the first loss value, the second loss value, and the third loss value, and generative adversarial training is performed on the initial skinned multi-person linear model based on the target loss value until a training stop condition is met, thereby obtaining the trained skinned multi-person linear model.

6. The method for generating a virtual human three-dimensional model according to claim 4, characterized in that: The discriminator is trained by the following steps, including: Obtaining label masked multi-person linear parameters corresponding to the sample objects in each of the sample video frames; The sample skinned multi-person linear parameters are used as negative samples, and the label skinned multi-person linear parameters are used as positive samples to train the discriminator until a training stop condition is met, thereby obtaining the trained skinned multi-person linear model.

7. The method for generating a virtual human three-dimensional model according to claim 1, characterized in that: The generating of a virtual human three-dimensional model based on a plurality of continuous virtual human three-dimensional models comprises: Determining scene information in the initial video; Model generation is performed on a plurality of continuous three-dimensional images of the virtual human based on the scene information to obtain a three-dimensional model of the virtual human wearing clothing corresponding to the scene information.

8. A device for generating a three-dimensional model of a virtual human, characterized in that: include: A video frame acquisition module, used to acquire a plurality of continuous video frames extracted from an initial video; A feature processing module, used to extract static features from each of the video frames, and input each of the continuous static features into a preset temporal convolutional network in sequence, perform convolution processing on each of the continuous static features, and obtain corresponding potential features; A feature combining module, used for combining the static feature of each video frame with the corresponding latent feature to obtain corresponding temporal enhancement features; A parameter generation module, used for performing nonlinear transformation on each of the temporal enhancement features to obtain a skinned multi-person linear parameter corresponding to the target object in each of the video frames; The three-dimensional model generation module is used to input each of the skinned multi-person linear parameters into the pre-trained skinned multi-person linear model to obtain a virtual human three-dimensional image corresponding to each of the video frames, and generate a virtual human three-dimensional model based on a plurality of continuous virtual human three-dimensional models.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method for generating a virtual human three-dimensional model as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a virtual human three-dimensional model according to any one of claims 1 to 7 is implemented.

Images