Virtual digital human processing method, model training method and device
By obtaining the key point image sequence of reference roles and training the variational self-coding model, the problem of high cost of virtual digital human driving in the existing technology is solved, and a low-cost and efficient virtual digital human driving solution is realized, reducing equipment investment and Editing cost.
Patent Information
- Application Number
- CN202210547459.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-19
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-05-19
AI Technical Summary
In the prior art, virtual digital human-driven solutions based on optical motion capture or inertial motion capture require expensive acquisition equipment and complex hardware, making it difficult to achieve universality and reduce costs.
By obtaining the key point image sequence of the reference character, the key point data of the virtual digital person in the two-dimensional space is determined, and the action prior sub-model and action coding sub-model are trained based on the variational auto-coding model, and the virtual digital person is driven to perform actions. Achieve low-cost virtual digital human-driven.
It provides a low-cost and friendly virtual digital human driver solution, which reduces the initial investment cost of equipment, realizes the real-time synchronous posture action driving, and greatly reduces the cost of virtual digital human posture editing.
Smart Images

Figure CN114862992B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, in particular to the field of artificial intelligence such as augmented reality technology and deep learning technology, and in particular to a virtual digital human processing method, a virtual digital human model training method and a device thereof. Background Art
[0002] Real-life three-dimensional (3D) digital human driving has always been a research hotspot in academia and has a wide range of industrial applications. The most common application areas include virtual anchors, virtual customer service, virtual assistants, virtual classrooms, virtual idols, and other interactive games and entertainment.
[0003] In the related technology, a motion capture studio is built through commercial software to use optical motion capture, or wearable sensor equipment is used to use inertial motion capture software to capture human body movements to drive the limbs of digital humans in real time.
[0004] However, solutions based on optical motion capture or inertial motion capture rely on expensive acquisition equipment, high-end computer hardware, and complex and arduous solution processes, making them difficult to be widely used. Summary of the invention
[0005] The present application provides a virtual digital human processing method, a training method, a device, an electronic device and a storage medium for generating a virtual digital human model.
[0006] According to the first aspect of the present application, a method for processing a virtual digital human is provided, comprising:
[0007] Obtain a key point image sequence of a reference character;
[0008] Determining, according to each key point data in the key point image sequence, virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space;
[0009] According to the key point data of the virtual digital human, a skeleton rotation coefficient sequence of the virtual digital human is obtained;
[0010] According to the skeleton rotation coefficient sequence, the virtual digital human is driven to perform corresponding actions.
[0011] According to a second aspect of the present application, a training method for a virtual digital human generation model is provided, wherein the virtual digital human generation model includes an action encoding sub-model and an action priori sub-model; the training method includes:
[0012] Acquire motion capture data, and parse the motion capture data to obtain a first bone rotation coefficient of the virtual digital human, and train a variational autoencoder model based on the first bone rotation coefficient; the variational autoencoder model includes an encoder, an intermediate encoding vector and a decoder;
[0013] Using the intermediate encoding vector and the decoder in the trained variational autoencoder model as the action prior sub-model;
[0014] Based on the key point image sequence of the reference character sample, the action prior sub-model is trained until the trained action prior sub-model meets the preset conditions, and the model parameters of the action prior sub-model are fixed;
[0015] Acquire training data; the training data includes key point data of the virtual digital human and a second bone rotation coefficient;
[0016] The virtual digital human generation model is trained according to the key point data of the virtual digital human and the second bone rotation coefficient until a training termination condition is met.
[0017] According to a third aspect of the present application, a virtual digital human processing device is provided, comprising:
[0018] A first acquisition module is used to acquire a key point image sequence of a reference character;
[0019] A determination module, used to determine, based on each key point data in the key point image sequence, virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space;
[0020] A second acquisition module is used to acquire a skeleton rotation coefficient sequence of the virtual digital human according to the key point data of the virtual digital human;
[0021] The driving module is used to drive the virtual digital human to perform corresponding actions according to the skeleton rotation coefficient sequence.
[0022] According to a fourth aspect of the present application, a training device for a virtual digital human generation model is provided, wherein the virtual digital human generation model includes an action encoding sub-model and an action priori sub-model; the training device includes:
[0023] A first training module is used to obtain motion capture data, analyze the motion capture data to obtain a first bone rotation coefficient of the virtual digital human, and train a variational autoencoder model based on the first bone rotation coefficient; the variational autoencoder model includes an encoder, an intermediate encoding vector and a decoder;
[0024] A determination module, used to determine the intermediate encoding vector and the decoder in the trained variational autoencoder model as the action prior sub-model;
[0025] A second training module is used to train the action priori sub-model based on the key point image sequence of the reference character sample until the trained action priori sub-model meets the preset conditions and fix the model parameters of the action prior sub-model;
[0026] An acquisition module, used for acquiring training data, wherein the training data includes key point data of the virtual digital human and a second bone rotation coefficient;
[0027] The third training module is used to train the virtual digital human generation model according to the key point data of the virtual digital human and the second bone rotation coefficient until the training termination condition is met.
[0028] According to a fifth aspect of the present application, there is provided an electronic device, including:
[0029] at least one processor; and
[0030] a memory communicatively connected to the at least one processor; wherein,
[0031] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in the first aspect above, and / or execute the method as described in the second aspect above.
[0032] According to the sixth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method as described in the first aspect above, and / or execute the method as described in the second aspect above.
[0033] According to the seventh aspect of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect and / or implements the steps of the method described in the second aspect.
[0034] According to the technical solution of the present application, a sequence of key point images of a reference character can be used to drive a virtual digital human to perform synchronized gestures in real time. Compared with related technologies, the present application can provide users with a low-cost and user-friendly driving solution, greatly reducing the initial equipment investment cost.
[0035] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.
[0037] Figure 1a A flowchart of a virtual digital human processing method provided in an embodiment of the present application;
[0038] Figure 1b An example diagram of key point data in a key point image sequence provided in an embodiment of the present application;
[0039] Figure 2 It is a flow chart of a method for visually editing a key point image provided by an embodiment of the present application;
[0040] Figure 3 A flowchart of another virtual digital human processing method provided in an embodiment of the present application;
[0041] Figure 4 A flowchart of another virtual digital human processing method provided in an embodiment of the present application;
[0042] Figure 5 An example flowchart of a virtual digital human processing method provided in an embodiment of the present application;
[0043] Figure 6 A flowchart of a training method for a virtual digital human generation model provided in an embodiment of the present application;
[0044] Figure 7 A schematic diagram of the structure of a variational autoencoder model provided in an embodiment of the present application;
[0045] Figure 8 A structural block diagram of a virtual digital human processing device provided in an embodiment of the present application;
[0046] Fig. 9 A structural block diagram of another virtual digital human processing device provided in an embodiment of the present application;
[0047] Fig.10 A structural block diagram of another virtual digital human processing device provided in an embodiment of the present application;
[0048] Fig.11 A structural block diagram of another virtual digital human processing device provided in an embodiment of the present application;
[0049] Fig.12A structural block diagram of a training device for generating a virtual digital human model provided in an embodiment of the present application;
[0050] Fig.13 A structural block diagram of another training device for generating a virtual digital human model provided in an embodiment of the present application;
[0051] Fig.14 It is a block diagram of an electronic device used to implement the virtual digital human processing method or model training method of the embodiment of the present application. DETAILED DESCRIPTION
[0052] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0053] The following describes the virtual digital human processing method, the virtual digital human generation model training method and the device thereof in the embodiments of the present application with reference to the accompanying drawings.
[0054] Figure 1a This is a flowchart of a virtual digital human processing method provided in an embodiment of the present application. It should be noted that the virtual digital human processing method in an embodiment of the present application can be applied to a virtual digital human processing device in an embodiment of the present application, and the virtual digital human processing device can be configured on an electronic device. Figure 1a As shown, the virtual digital human processing method may include but is not limited to the following steps.
[0055] In step 101, a key point image sequence of a reference character is obtained.
[0056] In the embodiment of the present application, the key point image sequence includes at least one key point image. It can be understood that the key point image can be a two-dimensional image. The key point image can be understood as a key point data image for the limbs of the reference character. For example, Figure 1b As shown, it is a key point image in the key point image sequence of the reference character, and each image in the sequence has key point data of the limbs of the reference character. Optionally, the key point image sequence can also be called a key point animation sequence.
[0057] In an embodiment of the present application, the key point image sequence may be predetermined. Alternatively, the key point image sequence may be a reference character video captured in real time by an image acquisition device (such as a camera), and 2D (two-dimensional) key point data of the limbs of the video is detected to obtain the key point image sequence of the reference character.
[0058] In step 102, based on each key point data in the key point image sequence, the key point data of the virtual digital human corresponding to each key point data when the virtual digital human is projected into the two-dimensional space is determined.
[0059] Optionally, the key point image sequence includes a T-shaped posture image. As an example, the T-shaped posture image can be the first image in the key point image sequence. The T-shaped posture image is a human posture with both arms stretched horizontally, which can clearly express the body proportions of the reference character in the key point data in the image. Since the body proportions of the reference character may be different from the proportions of each joint of the virtual digital human, the body proportions of the reference character can be used to update the key point data of the virtual digital human, that is, to update the key point data of the virtual digital human's limbs, so as to ensure that the proportions of each joint of the virtual digital human are consistent with the body proportions of the reference character.
[0060] It can be understood that since the key point data of the virtual digital human is obtained by projecting the key point data in each key point image of the reference character, the number of images of the key point data of the virtual digital human should be the same as the number of key point images of the reference character, and the total number of key point data of the virtual digital human should also be the same as the total number of key point data of the reference character. For example, taking the key point image sequence of the reference character containing two key point images, such as the first key point image and the second key point image as an example, the key point data of the virtual digital human corresponding to the key point data in the first key point image when the virtual digital human is projected into the two-dimensional space can be determined based on the key point data in the first key point image; the key point data of the virtual digital human corresponding to the key point data in the second key point image when the virtual digital human is projected into the two-dimensional space can be determined based on the key point data in the second key point image. The key point data of the virtual digital human obtained based on the two key point images are combined as the final key point data of the virtual digital human. Among them, the key point data combination of the virtual digital human can be arranged and combined according to the order of the key point images in the key point image sequence, such as the key point data combination of the virtual digital human can be {the key point data of the virtual digital human corresponding to the key point data in the first key point image, the key point data of the virtual digital human corresponding to the key point data in the second key point image}.
[0061] In step 103, a skeleton rotation coefficient sequence of the virtual digital human is obtained according to the key point data of the virtual digital human.
[0062] In a possible implementation, the mapping relationship between the key point data of the virtual digital human and the bone rotation coefficient of the virtual digital human can be predetermined. In this way, the bone rotation coefficient having a mapping relationship with the key point data of the virtual digital human can be determined based on the mapping relationship and the key point data of the virtual digital human determined in step 102. Since the key point data of the virtual digital human can be a combination of multiple key point data, when the bone rotation coefficient is determined using the above mapping relationship, a bone rotation coefficient sequence of the virtual digital human can be obtained, that is, a series of bone rotation coefficients can be obtained.
[0063] In another possible implementation, a model can be used to obtain a sequence of bone rotation coefficients of a virtual digital human. For example, a virtual digital human generation model is pre-trained so that the virtual digital human generation model has learned the mapping relationship between key point data and bone rotation coefficients. In this way, the virtual digital human generation model can be used to predict the key point data of the virtual digital human to obtain a sequence of bone rotation coefficients of the virtual digital human output by the virtual digital human generation model.
[0064] In step 104, the virtual digital human is driven to perform corresponding actions according to the skeleton rotation coefficient sequence.
[0065] In one implementation, a virtual digital human driving module may be used to drive the virtual digital human to perform corresponding actions according to the skeleton rotation coefficient sequence.
[0066] For example, taking the reference character waving as an example, a key point image sequence of the reference character waving is obtained, and based on the key point data in the key point image sequence, the key point data of the virtual digital human corresponding to the key point data of the reference character when the virtual digital human is projected into the two-dimensional space is determined, and based on the key point data of the virtual digital human, the bone rotation coefficient sequence of the virtual digital human is obtained, and then the virtual digital human driving module can be used to drive the virtual digital human to perform the corresponding waving action according to the bone rotation coefficient sequence.
[0067] Optionally, the virtual digital human processing method in the present application can be applied to online real-time scenes, that is, the virtual digital human processing method in the present application can be used to drive the virtual digital human to perform corresponding actions in real time using the key points of the reference character. For example, the virtual digital human processing method in the present application can be applied to live broadcast scenes, and the virtual digital human is a virtual anchor as an example. The anchor turns on the camera of the live broadcast room, and the real-time video stream of the anchor collected by the camera is used to perform 2D (two-dimensional) key point data detection of the limbs frame by frame (or key frame) on the real-time video stream to obtain the key point image sequence of the anchor, and the virtual anchor key point data corresponding to each key point data of the anchor when the virtual anchor is projected into the two-dimensional space is determined according to each key point data in the key point image sequence, and the virtual anchor's bone rotation coefficient sequence is obtained according to the virtual anchor key point data, and then the virtual anchor can be driven to perform corresponding waving actions according to the bone rotation coefficient sequence on the playback interface of the live broadcast room by using the driving module.
[0068] According to the virtual digital human processing method of the embodiment of the present application, the key point image sequence of the reference character can be used to drive the virtual digital human to perform synchronized posture movements in real time. Compared with the related technology, the present application provides users with a low-cost and friendly driving solution, greatly reducing the initial equipment investment cost.
[0069] In order to reduce the editing cost of the virtual digital human posture, the present application may provide a visual editing function for key point images. Figure 2 As shown, the visual editing method of the key point image may include but is not limited to the following steps.
[0070] In step 201, a visual editing interface is displayed.
[0071] In one implementation, a visual editing interface for key point images may be provided, and when the editing object clicks on the interface, the visual editing interface may be displayed so that the editing object can perform editing operations on the key point images in the key point image sequence on the visual editing interface. The editing object can use the visual editing interface to edit one or more key point images in the key point image sequence.
[0072] In step 202, based on the visual editing interface, an editing operation on at least part of the key point images in the key point image sequence is received.
[0073] In step 203, the corresponding key point images are processed according to the editing operation to obtain an edited key point image sequence, so that the virtual digital human processing method in this application can be used to process the visually edited key point image sequence to generate a corresponding virtual digital human action sequence.
[0074] It can be seen that the present application can provide a visual editing function for key point images, so that the key point images of the reference character can be visually edited, and then the key point image sequence after visual editing can be used to synchronously update the posture of the virtual digital human, avoiding direct editing of the virtual digital human's skeleton, which can greatly reduce the editing cost of the virtual digital human's posture.
[0075] It should be noted that, since the body proportions of the reference character and the proportions of each joint of the virtual digital human may be different, the body proportions of the reference character can be used to update the key point data of the virtual digital human, that is, to update the key point data of the limbs of the virtual digital human, so as to ensure that the proportions of each joint of the virtual digital human are consistent with the body proportions of the reference character. Figure 3 As shown, the virtual digital human processing method may include but is not limited to the following steps.
[0076] In step 301, a key point image sequence of a reference character is obtained.
[0077] In the embodiments of the present application, step 301 may be implemented in any of the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.
[0078] In step 302, body proportion information of a reference character in the T-shaped posture graph is determined according to the T-shaped posture graph in the key point image sequence.
[0079] Optionally, the key point image sequence includes a T-shaped posture image, and as an example, the T-shaped posture image may be the first image in the key point image sequence. The T-shaped posture image is a human posture with arms horizontally extended, which can clearly express the body proportions of the reference character in the key point data in the image.
[0080] In step 303, the proportion information of each joint of the virtual digital human is determined.
[0081] In one implementation, the proportion information of each joint of a virtual digital human can be given in advance.
[0082] In step 304, based on each key point data in the key point image sequence, the key point data of the virtual digital human corresponding to each key point data when the virtual digital human is projected into the two-dimensional space is determined.
[0083] It can be understood that since the key point data of the virtual digital human is obtained by projecting the key point data in each key point image of the reference character, the number of images of the key point data of the virtual digital human should be the same as the number of key point images of the reference character, and the total number of key point data of the virtual digital human should also be the same as the total number of key point data of the reference character. For example, taking the key point image sequence of the reference character containing two key point images, such as the first key point image and the second key point image as an example, the key point data of the virtual digital human corresponding to the key point data in the first key point image when the virtual digital human is projected into the two-dimensional space can be determined based on the key point data in the first key point image; the key point data of the virtual digital human corresponding to the key point data in the second key point image when the virtual digital human is projected into the two-dimensional space can be determined based on the key point data in the second key point image. The key point data of the virtual digital human obtained based on the two key point images are combined as the final key point data of the virtual digital human. Among them, the key point data combination of the virtual digital human can be arranged and combined according to the order of the key point images in the key point image sequence, such as the key point data combination of the virtual digital human can be {the key point data of the virtual digital human corresponding to the key point data in the first key point image, the key point data of the virtual digital human corresponding to the key point data in the second key point image}.
[0084] In step 305, the key point data of the virtual digital human is updated according to the body proportion information of the reference character and the proportion information of each joint of the virtual digital human.
[0085] In one implementation, the updating of the key point data of the virtual digital human includes: updating the vector length ratio of each key point connection line in the key point data of the virtual digital human. That is, according to the body proportion information of the reference character and the proportion information of each joint of the human body of the virtual digital human, the vector length ratio of each key point connection line in the key point data of the virtual digital human is updated, and the vector angle of each key point connection line in the key point data of the virtual digital human is kept unchanged.
[0086] In step 306, a skeleton rotation coefficient sequence of the virtual digital human is obtained according to the key point data of the virtual digital human.
[0087] In the embodiments of the present application, step 306 may be implemented in any of the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.
[0088] In step 307, the virtual digital human is driven to perform corresponding actions according to the skeleton rotation coefficient sequence.
[0089] In the embodiments of the present application, step 307 may be implemented in any of the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.
[0090] According to the virtual digital human processing method of the embodiment of the present application, the body proportions of the reference character can be used to update the key point data of the virtual digital human, that is, to update the key point data of the virtual digital human's limbs to ensure that the proportions of each joint of the virtual digital human are consistent with the body proportions of the reference character.
[0091] In one implementation, the skeleton rotation coefficient of the virtual digital human can be determined based on the key point data of the virtual digital human using the action coding vector. Figure 4 As shown, the virtual digital human processing method may include but is not limited to the following steps.
[0092] In step 401, a key point image sequence of a reference character is obtained.
[0093] In the embodiments of the present application, step 401 may be implemented in any of the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.
[0094] In step 402, based on each key point data in the key point image sequence, the key point data of the virtual digital human corresponding to each key point data when the virtual digital human is projected into the two-dimensional space is determined.
[0095] In the embodiments of the present application, step 402 may be implemented in any of the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.
[0096] In step 403, based on the key point data of the virtual digital human, the action coding vector corresponding to the key point data of the virtual digital human is determined.
[0097] In one implementation, the key point data of the virtual digital human is input into a preset virtual digital human generation model; wherein the virtual digital human generation model has learned the mapping relationship between the key point data and the bone rotation coefficient, and the virtual digital human generation model includes an action coding sub-model and an action prior sub-model; and the action coding vector corresponding to the key point data of the virtual digital human output by the action coding sub-model is obtained.
[0098] For example, a virtual digital human generation model can be pre-trained so that the virtual digital human generation model has learned the mapping relationship between key point data and bone rotation coefficients. The virtual digital human generation model includes an action coding sub-model and an action prior sub-model. The action coding sub-model includes an action coding vector module, which can learn the prior distribution of human body movements. As an example, the length of the action coding vector output by the action coding vector module can be 32, that is, each action can be represented by a vector with a length of 32. The implementation method of training the action coding vector module can refer to the description of the subsequent embodiments, which will not be repeated here.
[0099] In step 404, a skeleton rotation coefficient sequence of the virtual digital human is obtained according to the action coding vector corresponding to the key point data of the virtual digital human.
[0100] In one implementation, the motion coding vector corresponding to the key point data of the virtual digital human is input into the motion priori sub-model to obtain the bone rotation coefficient sequence of the virtual digital human. The implementation of training the motion priori sub-model can be found in the subsequent embodiments and will not be repeated here.
[0101] In step 405, the virtual digital human is driven to perform corresponding actions according to the skeleton rotation coefficient sequence.
[0102] In the embodiments of the present application, step 405 can be implemented in any of the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.
[0103] For example, Figure 5 As shown, when obtaining the key point image sequence of the reference character sample, a visual editing interface can be displayed, and an editing operation on at least part of the key point images in the key point image sequence can be received based on the visual editing interface, so that the corresponding key point images can be processed accordingly according to the editing operation to obtain the edited key point image sequence. According to each key point data in the edited key point image sequence, the virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into the two-dimensional space is determined, and the virtual digital human key point data is input into the action coding sub-model 501 in the virtual digital human generation model 500 to obtain the action coding vector corresponding to the virtual digital human key point data. The action coding vector is input into the action prior sub-model 502 to obtain the virtual digital human bone rotation coefficient sequence.
[0104] According to the virtual digital human processing method of the embodiment of the present application, the virtual digital human's bone rotation coefficient can be determined based on the virtual digital human's key point data using an action coding vector, wherein the length of the action coding vector can be 32, that is, each action can be represented by a vector with a length of 32, and then the corresponding virtual digital human's bone rotation coefficient sequence can be determined using the action coding vector with a length of 32, so that the generation of the virtual digital human's bone rotation coefficient can be achieved using a simple model.
[0105] In order to implement the above embodiment, the present application also proposes a training method for generating a virtual digital human model.
[0106] Figure 6 A flowchart of a training method for a virtual digital human generation model provided in an embodiment of the present application. It should be noted that the training method for a virtual digital human generation model in an embodiment of the present application can be applied to a training device for a virtual digital human generation model in an embodiment of the present application, and the training device can be configured in an electronic device.
[0107] It should also be noted that the virtual digital human generation module of the embodiment of the present application may include an action coding sub-model and an action priori sub-model. Figure 6 As shown, the training method of the virtual digital human generation model may include but is not limited to the following steps.
[0108] In step 601, motion capture data is acquired, and the motion capture data is parsed to obtain the first bone rotation coefficient of the virtual digital human, and a variational autoencoder model is trained based on the first bone rotation coefficient. Figure 7 As shown, the variational auto-encoder (VAE) model includes an encoder 701, an intermediate coding vector 702 and a decoder 703.
[0109] In one implementation, the motion capture data may be existing motion capture data, for example, motion capture data of related technologies may be obtained. That is, the present application may not lay out equipment for capturing motion capture data, but directly use motion capture data of related technologies.
[0110] In an embodiment of the present application, the motion capture data may be parsed to obtain the first bone rotation coefficient of the virtual digital human. Figure 7As shown, the first bone rotation coefficient is used as the input of the variational autoencoder model, and also as the output supervised training of the decoder result in the VAE model. The length of the intermediate coding vector 702 of the VAE model can be 32, that is, each action can be represented by a vector of length 32. Under the training of a large amount of motion capture data, the intermediate coding vector 702 of the VAE model can learn the prior distribution of human body movements. The intermediate coding vector 702 and the decoder 703 of the VAE model are used as a priori sub-model of the action, that is, the intermediate coding vector 702 is used to generate a sequence of bone rotation coefficients of a virtual digital person based on the decoder 703 by sampling.
[0111] In step 602, the intermediate encoding vector and decoder in the trained variational autoencoder model are used as the motion prior sub-model.
[0112] In step 603, based on the key point image sequence of the reference character sample, the action prior sub-model is trained until the trained action prior sub-model meets the preset conditions, and the model parameters of the action prior sub-model are fixed.
[0113] In one implementation, the key point data of the virtual digital human can be obtained according to the key point image sequence and the proportion information of each joint of the virtual digital human, and the key point data of the virtual digital human can be input into the action prior sub-model to obtain the first bone rotation coefficient prediction value output by the action prior sub-model; the first bone rotation coefficient prediction value is projected into a two-dimensional space to obtain the key point data prediction value of the virtual digital human; a first loss value is generated according to the key point data of the virtual digital human and the key point data prediction value; and the action prior sub-model is trained according to the first loss value. Among them, the loss function for calculating the first loss value can be a cross loss function, or other loss functions, which is not specifically limited in this application.
[0114] In a possible implementation, the body proportion information of the reference character in the T-shaped posture diagram can be determined according to the T-shaped posture diagram in the key point image sequence; the proportion information of each joint of the human body of the virtual digital person can be determined; according to each key point data in the key point image sequence, the key point data of the virtual digital person corresponding to each key point data when the virtual digital person is projected into a two-dimensional space can be determined; according to the body proportion information of the reference character and the proportion information of each joint of the human body of the virtual digital person, the key point data of the virtual digital person is updated. Among them, the updated content is the vector length ratio of each key point connection line in the key point data of the virtual digital person. In addition, the implementation method of updating the key point data of the virtual digital person can be implemented by any of the methods in the various embodiments of the present application, and the embodiments of the present application do not limit this and will not be repeated.
[0115] In step 604, training data is obtained; the training data includes key point data of the virtual digital human and the second bone rotation coefficient.
[0116] In an embodiment of the present application, the second bone rotation coefficient may be the same as the first bone rotation coefficient; or, the second bone rotation coefficient may be different from the first bone rotation coefficient.
[0117] In an embodiment of the present application, the key point data in the training data may correspond to the second bone rotation coefficient, or the key point data in the training data may not correspond to the second bone rotation coefficient.
[0118] In step 605, the virtual digital human generation model is trained according to the key point data of the virtual digital human and the second bone rotation coefficient until the training termination condition is met.
[0119] In one implementation, the virtual digital human generation model may be a real-time reasoning network based on LSTM (Long Short-Term Memory) as the main architecture. For example, the virtual digital human generation model may include an action encoding sub-model and an action prior sub-model constructed based on LSTM. It can be understood that when the virtual digital human generation model is trained in step 605, the model parameters of the action prior sub-model in the virtual digital human generation model are fixed and no longer participate in training and learning.
[0120] In one possible implementation, the key point data of the virtual digital human is input into the action coding sub-model to obtain the action coding vector output by the action coding sub-model; the action coding vector is input into the action prior sub-model to obtain the second bone rotation coefficient prediction value output by the action prior sub-model; a second loss value is generated based on the second bone rotation coefficient prediction value and the second bone rotation coefficient; and the model parameters of the action coding sub-model are adjusted based on the second loss value.
[0121] That is to say, when training the virtual digital human generation model, the model parameters of the action coding sub-model in the virtual digital human generation model need to be adjusted so that the action coding sub-model learns the mapping relationship between the key point data of the virtual digital human and the action coding vector. The loss function for calculating the second loss value may be a cross loss function or other loss functions, which is not specifically limited in this application.
[0122] In some embodiments of the present application, a visual editing interface may be displayed, and based on the visual editing interface, an editing operation on at least part of the key point images in the key point image sequence is received, and the corresponding key point images are processed accordingly according to the editing operation to obtain an edited key point image sequence. In this way, more key point images of postures can be obtained through the visual editing interface to enrich the training data.
[0123] According to the training method of the virtual digital human generation model of the present application, an action prior sub-model is constructed based on the existing motion capture data, and the action prior sub-model is trained based on the key point image sequence of the reference character sample until the trained action prior sub-model meets the preset conditions, and the model parameters of the action prior sub-model are fixed. Then, the action encoding sub-model in the virtual digital human generation model can be trained according to the key point data and bone rotation coefficients of the virtual digital human until the training termination conditions are met. So that when it is applied online, the key point image sequence of the reference character can be used to drive the virtual digital human to perform synchronized posture movements in real time. Compared with related technologies, the present application provides users with a low-cost and friendly driving solution, which greatly reduces the initial equipment investment cost.
[0124] In order to implement the above embodiments, the present application also proposes a virtual digital human processing device.
[0125] Figure 8 This is a structural block diagram of a virtual digital human processing device provided in an embodiment of the present application. Figure 8 As shown, the virtual digital human processing device may include: a first acquisition module 810, a determination module 820, a second acquisition module 830 and a driving module 840.
[0126] The first acquisition module 810 is used to acquire a key point image sequence of a reference character.
[0127] The determination module 820 is used to determine the key point data of the virtual digital human corresponding to each key point data when the virtual digital human is projected into the two-dimensional space according to each key point data in the key point image sequence.
[0128] The second acquisition module 830 is used to acquire the skeleton rotation coefficient sequence of the virtual digital human according to the key point data of the virtual digital human.
[0129] The driving module 840 is used to drive the virtual digital human to perform corresponding actions according to the skeleton rotation coefficient sequence.
[0130] According to the virtual digital human processing device of the embodiment of the present application, the key point image sequence of the reference character can be used to drive the virtual digital human to perform synchronized posture movements in real time. Compared with the related technology, the present application provides users with a low-cost and friendly driving solution, which greatly reduces the initial equipment investment cost.
[0131] In order to reduce the editing cost of the virtual digital human posture, the present application may provide a visual editing function for key point images. Fig. 9 As shown, the virtual digital human processing device may also include: a display module 950 , a receiving module 960 and a processing module 970 .
[0132] The display module 950 is used to display a visual editing interface. The receiving module 960 is used to receive an editing operation on at least part of the key point images in the key point image sequence based on the visual editing interface. The processing module 970 is used to perform corresponding processing on the corresponding key point images according to the editing operation to obtain the edited key point image sequence. Fig. 9 910-940 and Figure 8 810-840 have the same function and structure.
[0133] Optionally, in some embodiments of the present application, Fig.10 As shown, the determination module 1020 may include a first determination unit 1021, a second determination unit 1022, a third determination unit 1023 and an updating unit 1024. Among them, the first determination unit 1021 is used to determine the body proportion information of the reference character in the T-shaped posture diagram according to the T-shaped posture diagram in the key point image sequence; the second determination unit 1022 is used to determine the proportion information of each joint of the human body of the virtual digital human; the third determination unit 1023 is used to determine the virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space according to each key point data in the key point image sequence; the updating unit 1024 is used to update the virtual digital human key point data according to the body proportion information of the reference character and the proportion information of each joint of the human body of the virtual digital human. As an example, the updated content is the vector length ratio of each key point connection line in the virtual digital human key point data. Among them, Fig.10 Medium 1010-1070 and Fig. 9 910-970 have the same function and structure.
[0134] Optionally, in some embodiments of the present application, Fig.11 As shown, the second acquisition module 1130 may include: a determination unit 1131 and an acquisition unit 1132. The determination unit 1131 is used to determine the action coding vector corresponding to the key point data of the virtual digital human according to the key point data of the virtual digital human; the acquisition unit 1132 is used to acquire the bone rotation coefficient sequence of the virtual digital human according to the action coding vector corresponding to the key point data of the virtual digital human. Fig.11 Medium 1110-1170 and Fig.10 1010-1070 have the same function and structure.
[0135] Optionally, in some embodiments of the present application, the determination unit 1131 is specifically used to: input the key point data of the virtual digital human into a preset virtual digital human generation model; wherein the virtual digital human generation model has learned the mapping relationship between the key point data and the bone rotation coefficient, and the virtual digital human generation model includes an action coding sub-model and an action prior sub-model; obtain the action coding vector corresponding to the key point data of the virtual digital human output by the action coding sub-model; the acquisition unit 1132 is specifically used to: input the action coding vector corresponding to the key point data of the virtual digital human into the action prior sub-model to obtain the bone rotation coefficient sequence of the virtual digital human.
[0136] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0137] In order to implement the above-mentioned embodiment, the present application also proposes a training device for a virtual digital human generation model, wherein the virtual digital human generation model includes an action encoding sub-model and an action prior sub-model. Fig.12 This is a structural block diagram of a training device for generating a virtual digital human model provided in an embodiment of the present application. Fig.12 As shown, the training device for generating a virtual digital human model may include: a first training module 1201 , a determination module 1202 , a second training module 1203 , an acquisition module 1204 and a third training module 1205 .
[0138] Among them, the first training module 1201 is used to obtain motion capture data, and parse the motion capture data to obtain the first bone rotation coefficient of the virtual digital human, and train the variational autoencoder model based on the first bone rotation coefficient; the variational autoencoder model includes an encoder, an intermediate encoding vector and a decoder.
[0139] The determination module 1202 is used to determine the intermediate coding vector and the decoder in the trained variational autoencoder model as the action prior sub-model.
[0140] The second training module 1203 is used to train the action priori sub-model based on the key point image sequence of the reference character sample until the trained action priori sub-model meets the preset conditions and fix the model parameters of the action prior sub-model.
[0141] In one implementation, the second training module 1203 is specifically used to: obtain the key point data of the virtual digital human according to the key point image sequence and the proportion information of each joint of the virtual digital human; input the key point data of the virtual digital human into the action prior sub-model to obtain the first bone rotation coefficient prediction value output by the action prior sub-model; project the first bone rotation coefficient prediction value into a two-dimensional space to obtain the key point data prediction value of the virtual digital human; generate a first loss value according to the key point data of the virtual digital human and the key point data prediction value; and train the action prior sub-model according to the first loss value.
[0142] In a possible implementation, the second training module 1203 is specifically used to: determine the body proportion information of the reference character in the T-shaped posture diagram according to the T-shaped posture diagram in the key point image sequence; determine the proportion information of each joint of the human body of the virtual digital human; determine the virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space according to each key point data in the key point image sequence; update the virtual digital human key point data according to the body proportion information of the reference character and the proportion information of each joint of the human body of the virtual digital human.
[0143] The acquisition module 1204 is used to acquire training data, where the training data includes key point data of the virtual digital human and a second bone rotation coefficient.
[0144] The third training module 1205 is used to train the virtual digital human generation model according to the key point data of the virtual digital human and the second bone rotation coefficient until the training termination condition is met.
[0145] In one implementation, the third training module 1205 is specifically used to: input the key point data of the virtual digital human into the action coding sub-model to obtain the action coding vector output by the action coding sub-model; input the action coding vector into the action priori sub-model to obtain the second bone rotation coefficient prediction value output by the action priori sub-model; generate a second loss value based on the second bone rotation coefficient prediction value and the second bone rotation coefficient; and adjust the model parameters of the action coding sub-model based on the second loss value.
[0146] Optionally, in some embodiments of the present application, Fig.13As shown, the training device may also include: a display module 1306, a receiving module 1307 and a processing module 1308. The display module 1306 is used to display a visual editing interface; the receiving module 1307 is used to receive an editing operation on at least part of the key point images in the key point image sequence based on the visual editing interface; the processing module 1308 is used to perform corresponding processing on the corresponding key point images according to the editing operation to obtain the edited key point image sequence. Fig.13 Medium 1301-1305 and Fig.12 1201-1205 have the same function and structure.
[0147] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0148] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.
[0149] like Fig.14 , is a block diagram of an electronic device according to a virtual digital human processing method or training method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0150] like Fig.14 As shown, the electronic device includes: one or more processors 1401, memory 1402, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.14 A processor 1401 is taken as an example.
[0151] The memory 1402 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to execute the virtual digital human processing method or the training method for generating a virtual digital human model provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to execute the virtual digital human processing method or the training method for generating a virtual digital human model provided in the present application.
[0152] The memory 1402 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to any method in the embodiments of the present application. The processor 1401 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1402, that is, the virtual digital human processing method or the virtual digital human generation model training method in the above method embodiments is implemented.
[0153] The memory 1402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1402 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1402 may optionally include a memory remotely arranged relative to the processor 1401, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0154] The electronic device may further include: an input device 1403 and an output device 1404. The processor 1401, the memory 1402, the input device 1403 and the output device 1404 may be connected via a bus or other means. Fig.14 The example of connecting through bus is taken in the following.
[0155] The input device 1403 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1404 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.
[0156] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0157] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0159] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0160] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0161] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0162] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A method for processing a virtual digital human, comprising: Obtain a key point image sequence of a reference character; Determining, according to each key point data in the key point image sequence, virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space; According to the key point data of the virtual digital human, a skeleton rotation coefficient sequence of the virtual digital human is obtained; According to the skeleton rotation coefficient sequence, driving the virtual digital human to perform corresponding actions; Wherein, obtaining the skeleton rotation coefficient sequence of the virtual digital human according to the key point data of the virtual digital human includes: According to the key point data of the virtual digital human, determining the action coding vector corresponding to the key point data of the virtual digital human; Acquire a skeleton rotation coefficient sequence of the virtual digital human according to the action coding vector corresponding to the key point data of the virtual digital human; Wherein, determining the action coding vector corresponding to the key point data of the virtual digital human according to the key point data of the virtual digital human includes: Inputting the key point data of the virtual digital human into a preset virtual digital human generation model; wherein the virtual digital human generation model has learned the mapping relationship between the key point data and the bone rotation coefficient, and the virtual digital human generation model includes an action encoding sub-model and an action prior sub-model; Obtaining the action coding vector output by the action coding sub-model corresponding to the key point data of the virtual digital human; The step of obtaining the skeleton rotation coefficient sequence of the virtual digital human according to the action coding vector corresponding to the key point data of the virtual digital human comprises: The action coding vector corresponding to the key point data of the virtual digital human is input into the action priori sub-model to obtain the skeleton rotation coefficient sequence of the virtual digital human.
2. The method of claim 1, wherein: Also includes: Display the visual editing interface; Based on the visual editing interface, receiving an editing operation on at least part of the key point images in the key point image sequence; The corresponding key point images are processed accordingly according to the editing operation to obtain the edited key point image sequence.
3. The method according to claim 1 or 2, wherein: The step of determining the virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space according to each key point data in the key point image sequence includes: Determining body proportion information of the reference character in the T-shaped posture graph according to the T-shaped posture graph in the key point image sequence; Determining the proportion information of each joint of the virtual digital human; Determining, according to each key point data in the key point image sequence, virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space; The key point data of the virtual digital human are updated according to the body proportion information of the reference character and the proportion information of each joint of the human body of the virtual digital human.
4. The method of claim 3, wherein: The updated content is the vector length ratio of each key point connection line in the key point data of the virtual digital human.
5. A training method for generating a virtual digital human model, wherein: The virtual digital human generation model includes an action coding sub-model and an action priori sub-model; the training method includes: Acquire motion capture data, and parse the motion capture data to obtain a first bone rotation coefficient of the virtual digital human, and train a variational autoencoder model based on the first bone rotation coefficient; the variational autoencoder model includes an encoder, an intermediate encoding vector and a decoder; Using the intermediate encoding vector and the decoder in the trained variational autoencoder model as the action prior sub-model; Based on the key point image sequence of the reference character sample, the action prior sub-model is trained until the trained action prior sub-model meets the preset conditions, and the model parameters of the action prior sub-model are fixed; Acquire training data; the training data includes key point data of the virtual digital human and a second bone rotation coefficient; Training the virtual digital human generation model according to the key point data of the virtual digital human and the second bone rotation coefficient until a training termination condition is met; The step of training the virtual digital human generation model according to the key point data of the virtual digital human and the second bone rotation coefficient includes: Inputting the key point data of the virtual digital human into the action coding sub-model to obtain the action coding vector output by the action coding sub-model; Inputting the motion encoding vector into the motion prior sub-model to obtain a predicted value of a second bone rotation coefficient output by the motion prior sub-model; Generate a second loss value according to the second bone rotation coefficient prediction value and the second bone rotation coefficient; According to the second loss value, the model parameters of the action encoding sub-model are adjusted.
6. The method of claim 5, wherein: The step of training the action prior sub-model based on the key point image sequence of the reference character sample comprises: Acquire key point data of the virtual digital human according to the key point image sequence and the proportion information of each joint of the virtual digital human; Inputting the key point data of the virtual digital human into the action priori sub-model to obtain a predicted value of the first bone rotation coefficient output by the action priori sub-model; Projecting the first bone rotation coefficient prediction value into a two-dimensional space to obtain a key point data prediction value of the virtual digital human; Generate a first loss value according to the key point data of the virtual digital human and the predicted value of the key point data; The action prior sub-model is trained according to the first loss value.
7. The method of claim 6, wherein: The step of obtaining key point data of the virtual digital human according to the key point image sequence and the proportion information of each joint of the human body of the virtual digital human comprises: Determining body proportion information of the reference character in the T-shaped posture graph according to the T-shaped posture graph in the key point image sequence; Determining the proportion information of each joint of the virtual digital human; Determining, according to each key point data in the key point image sequence, virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space; The key point data of the virtual digital human are updated according to the body proportion information of the reference character and the proportion information of each joint of the human body of the virtual digital human.
8. The method of claim 5, wherein: Also includes: Display the visual editing interface; Based on the visual editing interface, receiving an editing operation on at least part of the key point images in the key point image sequence; The corresponding key point images are processed accordingly according to the editing operation to obtain the edited key point image sequence.
9. A virtual digital human processing device, comprising: A first acquisition module is used to acquire a key point image sequence of a reference character; A determination module, used to determine, based on each key point data in the key point image sequence, virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space; A second acquisition module is used to acquire a skeleton rotation coefficient sequence of the virtual digital human according to the key point data of the virtual digital human; A driving module, used to drive the virtual digital human to perform corresponding actions according to the skeleton rotation coefficient sequence; Wherein, the second acquisition module includes: A determination unit, configured to determine, based on the key point data of the virtual digital human, an action coding vector corresponding to the key point data of the virtual digital human; An acquisition unit, used for acquiring a skeleton rotation coefficient sequence of the virtual digital human according to an action coding vector corresponding to key point data of the virtual digital human; The determining unit is specifically used for: Inputting the key point data of the virtual digital human into a preset virtual digital human generation model; wherein the virtual digital human generation model has learned the mapping relationship between the key point data and the bone rotation coefficient, and the virtual digital human generation model includes an action encoding sub-model and an action prior sub-model; Obtaining the action coding vector output by the action coding sub-model corresponding to the key point data of the virtual digital human; The acquisition unit is specifically used for: The action coding vector corresponding to the key point data of the virtual digital human is input into the action priori sub-model to obtain the skeleton rotation coefficient sequence of the virtual digital human.
10. The device of claim 9, wherein: Also includes: Display module, used to display the visual editing interface; A receiving module, configured to receive, based on the visual editing interface, an editing operation on at least part of the key point images in the key point image sequence; The processing module is used to perform corresponding processing on the corresponding key point image according to the editing operation to obtain the edited key point image sequence.
11. The device according to claim 9 or 10, wherein: The determination module comprises: A first determining unit is used to determine the body proportion information of the reference character in the T-shaped posture diagram according to the T-shaped posture diagram in the key point image sequence; A second determining unit is used to determine the proportion information of each joint of the virtual digital human; A third determining unit is used to determine the virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space according to each key point data in the key point image sequence; An updating unit is used to update the key point data of the virtual digital human according to the body proportion information of the reference character and the proportion information of each joint of the human body of the virtual digital human.
12. The device of claim 11, wherein: The updated content is the vector length ratio of each key point connection line in the key point data of the virtual digital human.
13. A training device for generating a virtual digital human model, wherein: The virtual digital human generation model includes an action coding sub-model and an action priori sub-model; the training device includes: A first training module is used to obtain motion capture data, analyze the motion capture data to obtain a first bone rotation coefficient of the virtual digital human, and train a variational autoencoder model based on the first bone rotation coefficient; the variational autoencoder model includes an encoder, an intermediate encoding vector and a decoder; A determination module, used to determine the intermediate encoding vector and the decoder in the trained variational autoencoder model as the action prior sub-model; A second training module is used to train the action priori sub-model based on the key point image sequence of the reference character sample until the trained action priori sub-model meets the preset conditions and fix the model parameters of the action prior sub-model; An acquisition module, used for acquiring training data, wherein the training data includes key point data of the virtual digital human and a second bone rotation coefficient; A third training module is used to train the virtual digital human generation model according to the key point data of the virtual digital human and the second bone rotation coefficient until a training termination condition is met; Wherein, the third training module is specifically used for: Inputting the key point data of the virtual digital human into the action coding sub-model to obtain the action coding vector output by the action coding sub-model; Inputting the motion encoding vector into the motion prior sub-model to obtain a predicted value of a second bone rotation coefficient output by the motion prior sub-model; Generate a second loss value according to the second bone rotation coefficient prediction value and the second bone rotation coefficient; According to the second loss value, the model parameters of the action encoding sub-model are adjusted.
14. The device of claim 13, wherein: The second training module is specifically used for: Acquire key point data of the virtual digital human according to the key point image sequence and the proportion information of each joint of the virtual digital human; Inputting the key point data of the virtual digital human into the action priori sub-model to obtain a predicted value of the first bone rotation coefficient output by the action priori sub-model; Projecting the first bone rotation coefficient prediction value into a two-dimensional space to obtain a key point data prediction value of the virtual digital human; Generate a first loss value according to the key point data of the virtual digital human and the predicted value of the key point data; The action prior sub-model is trained according to the first loss value.
15. The device of claim 14, wherein: The second training module is specifically used for: Determining body proportion information of the reference character in the T-shaped posture graph according to the T-shaped posture graph in the key point image sequence; Determining the proportion information of each joint of the virtual digital human; Determining, according to each key point data in the key point image sequence, virtual digital human key point data corresponding to each key point data when the virtual digital human is projected into a two-dimensional space; The key point data of the virtual digital human are updated according to the body proportion information of the reference character and the proportion information of each joint of the human body of the virtual digital human.
16. The device of claim 13, wherein: Also includes: Display module, used to display the visual editing interface; A receiving module, configured to receive, based on the visual editing interface, an editing operation on at least part of the key point images in the key point image sequence; The processing module is used to perform corresponding processing on the corresponding key point image according to the editing operation to obtain the edited key point image sequence.
17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 4 and / or execute the method of any one of claims 5 to 8.
18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4 and / or to execute the method according to any one of claims 5 to 8.
19. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 4 and / or implements the steps of the method according to any one of claims 5 to 8.
