A motion choreography method for sign language digital humans

Abstract

This invention discloses a motion choreography method for sign language digital humans, aiming to fill the gap in research on motion choreography for sign language digital humans by constructing a Chinese sign language motion dataset. Firstly, it directly retrieves motion data from a motion database based on user input, eliminating the motion graph construction process. Secondly, it introduces an online intelligent motion redirection model, which supports multi-role motion choreography by converting multi-role skeletons to a unified skeleton. Finally, it employs a transition motion generation model to address the shortcomings of traditional transition motion generation methods, ensuring the realism of the generated transition motions.

Description

Technical Field

[0001] This invention belongs to the field of animation production technology, specifically relating to a motion choreography method for sign language digital humans. Background Technology

[0002] Computer character animation production methods are mainly divided into two categories: one is based on physical simulation and character control, and the other is based on skeletal animation. Physical simulation is the simulation of objective physical phenomena, while character control is a subjective motion signal. The fusion of the two constitutes the digital human driving technology that supports environmental interaction. The skeletal animation-based driving method updates the digital human's posture in real time by rotating the joints of the digital human. The advantage of this method is that it can perform low-dimensional control, allowing the movements to present different styles, such as the exaggerated artistic style of Disney. However, the skeletal animation-based method blocks interaction with the physical world, making it unable to handle problems such as clipping and sliding.

[0003] Skeletal animation-driven animation production methods are mainly divided into keyframe interpolation, motion capture, motion graphs, and deep learning-based methods. Keyframe interpolation relies on animators manually creating keyframe movements, then generating transition frames using linear interpolation and other methods. However, this method produces uneven results with short transition frames and lacks realism with longer transition frames. Motion capture methods use motion capture technology to capture the movements of real people, then use motion redirection technology to redirect the captured movements onto a digital human. However, this method is costly and easily limited by physical space. Motion graph methods are prone to needing to regenerate motion graphs as the motion library expands. As the motion library grows larger, the computational complexity of motion graph node differences increases exponentially, slowing down or even preventing motion graph generation.

[0004] Currently, common deep learning-based transition animation generation methods mainly fall into three technical schools: Recurrent Neural Network (RNN)-based methods, Convolutional Neural Network (CNN)-based methods, and Transformer-based methods. While RNN-based methods support inputs and outputs of variable length, their training and inference efficiency is relatively low. CNN-based methods, although not suffering from the difficulty of parallelizing RNN computation, are still limited by the size of the receptive field when processing long-term temporal data. Transformer-based methods effectively address the problems of the aforementioned two technical schools. The global attention mechanism in Transformers is also better suited for handling long-term dependencies, thus finding widespread application in human motion synthesis. However, they are still limited by the fixed size of the input and output.

[0005] Furthermore, since motion choreography involves multiple sets of skeleton standards, and transition motion generation tasks require a unified skeleton standard data, motion retargeting technology is needed to redirect multiple sets of skeleton standard data to a unified skeleton standard. Motion retargeting is defined as motion transfer between skeletons with different topologies or between two skeletons with the same topology but different bone lengths. However, the baseline retargeting method may cause distortion or stretching when applying motion due to differences in bone structure, thus affecting the realism and smoothness of the animation. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a motion choreography method for sign language digital humans, which improves the accuracy and efficiency of motion choreography by constructing a motion library and generating transitional motions based on neural networks.

[0007] To achieve the above-mentioned objectives, an embodiment provides a motion choreography method for sign language digital humans, comprising the following steps:

[0008] Construct a Chinese sign language action dataset based on an initial digital human skeleton;

[0009] Obtain the Chinese sign language vocabulary, transition frames, and target digital human required for choreography;

[0010] The action sequences corresponding to Chinese sign language terms are retrieved from the Chinese sign language action dataset, and the action sequences are parsed and preprocessed to obtain the processed action sequences;

[0011] The preprocessed motion sequence is split into a hand rotation sequence and a torso motion sequence under the initial digital human skeleton. Spherical linear interpolation is performed on the hand rotation sequence to generate a hand transition motion sequence that meets the number of transition frames.

[0012] The first action redirection model based on generative adversarial network is used to generate a torso action sequence under the intermediate skeleton based on the torso action sequence under the initial digital human skeleton, the initial skeleton information corresponding to the torso action sequence, and the intermediate skeleton information.

[0013] The transition motion generation model based on the Transformer encoder is used to generate a torso transition motion sequence that meets the number of transition frames based on keyframes extracted from the torso motion sequence under the intermediate skeleton.

[0014] The second action redirection model based on generative adversarial network is used to generate the torso action sequence under the target digital human skeleton from the torso action sequence under the intermediate skeleton, the torso transition action sequence, the intermediate skeleton information, and the target digital human skeleton information.

[0015] The motion choreography result is obtained by fusing the hand rotation sequence, the hand transition motion sequence, and the torso motion sequence under the target digital human skeleton.

[0016] Preferably, the first action redirection model and the second action redirection model are constructed based on generative adversarial networks (GANs), wherein the GAN framework includes Skel. A Skeleton encoder, M A Motion encoder, M B Motion encoder, Skel B Skeleton encoder, M A Motion decoder, M B Action decoder, discriminator A and discriminator B, skeleton information of discriminator A. A After Skel A The skeleton encoder obtains the skel A The encoding results are input into M respectively. A Motion encoder and M A Action decoder, action sequence M under skeleton A A and Skel A The encoding result is processed by M A The motion encoder encodes the motion encoding result Z corresponding to skeleton A. A The action encoding result Z A With Skel A The encoding result in M A The motion decoder obtains the motion decoding result corresponding to skeleton A after decoding. B-skeletal information skel B After Skel B The skeleton encoder obtains the skel B Encode the result and input it into M BAction decoder, action encoding result Z A and Skel B The encoding result in M B The motion decoder decodes the motion corresponding to the B skeleton. The decoding result of the action In M B The motion encoder obtains the motion encoding result Z corresponding to the B skeleton after encoding. B Discriminator A is used to determine M A With Skel A The authenticity, and With Skel A The discriminator A is used to determine the authenticity of M. B With skel B Authenticity and counterfeiting, and With skel B Authenticity or counterfeit.

[0017] Preferably, based on the generative adversarial network framework, when constructing the first action redirection model, the torso action sequence under the initial digital human skeleton is used as M. A The initial skeletal information corresponding to the torso movement sequence is used as the skel A The torso motion sequence under the intermediate skeleton (e.g., SMPL) is used as M B The intermediate skeleton information is used as a skel B The parameters of the generative adversarial network framework are optimized using the aforementioned loss function L. After parameter optimization, Skel... A Skeleton encoder, M A Motion encoder, Skel B Skeleton encoder and M B The action decoder constitutes the first action redirection model.

[0018] Preferably, when generating the torso action sequence under the intermediate skeleton using the first motion retargeting model based on the torso action sequence under the initial digital human skeleton, the initial skeleton information corresponding to the torso action sequence, and the intermediate skeleton information, the specific steps include: inputting the initial skeleton information corresponding to the torso action sequence into the skel... A The skeleton encoder obtains the encoding result and inputs it into M. A The motion encoder inputs the torso motion sequence from the initial digital human skeleton to M. A Motion encoder, M A The motion encoder encodes the two inputs to obtain the encoded vector Z under the initial digital human skeleton. A The intermediate skeleton information is input into Skel. B The skeleton encoder obtains the encoding result and inputs it into M. B Action decoder, simultaneously encoding vector ZA Also enter into M B Motion decoder, M B The motion decoder decodes the two inputs to obtain the torso motion sequence under the intermediate skeleton.

[0019] Preferably, based on the generative adversarial network framework, when constructing the second action redirection model, the torso action sequence under the intermediate skeleton is used as M. A The intermediate skeleton information corresponding to the torso movement sequence is used as skel A The torso movement sequence under the target digital human skeleton is used as M B Using the target digital human skeleton information as a skel B The parameters of the generative adversarial network framework are optimized using the aforementioned loss function L. After parameter optimization, Skel... A Skeleton encoder, M A Motion encoder, Skel B Skeleton encoder and M B The action decoder constitutes the second action redirection model.

[0020] Preferably, when generating the torso action sequence under the target digital human skeleton using the second motion retargeting model based on the torso action sequence and torso transition action sequence under the intermediate skeleton, the intermediate skeleton information, and the target digital human skeleton information, the specific steps include: inputting the intermediate skeleton information corresponding to the torso action sequence into the skel. A The skeleton encoder obtains the encoding result and inputs it into M. A The motion encoder inputs the torso motion sequence under the intermediate skeleton to M. A Motion encoder, M A The motion encoder encodes the two inputs to obtain the encoded vector Z under the intermediate skeleton. A The target digital human skeleton information is input into Skel. B The skeleton encoder obtains the encoding result and inputs it into M. B Action decoder, simultaneously encoding vector Z A Also enter into M B Motion decoder, M B The motion decoder decodes the two inputs to obtain the torso motion sequence under the intermediate skeleton.

[0021] Preferably, the transition motion generation model based on the Transformer encoder includes two branches. Each branch includes a first linear layer, a Transformer Encoder, and a second linear layer. Spherical linear interpolation is performed on the keyframe motion to obtain the interpolated frame motion. The keyframe temporal index and the required number of transition frames are initially encoded and linearly mapped, then concatenated with the difference between the keyframe motion and the reference frame motion. The concatenated result is linearly mapped through the first linear layer of the first branch. The first linear mapping result is then input to the Transformer Encoder as the K, Q, and V channels of the attention mechanism to obtain the first branch encoding result. This first branch encoding result is mapped through the second linear layer, which acts as the decoder, to obtain the first branch decoding result. This first branch decoding result is added to the reference frame motion to obtain the reconstructed keyframe motion. The transition frame temporal index and the required number of transition frames are initially encoded and linearly mapped, then input to the second branch. The second branch's first linear layer linearly maps this result to obtain the second mapping result, which serves as the Q channel of the attention mechanism. Simultaneously, the first branch encoding result is used as the K and V channels of the attention mechanism and input to the Transformer Encoder. The Encoder obtains the second branch encoding result, which is then mapped through the second linear layer that acts as the decoder to obtain the second decoding result. This second decoding result is added to the interpolated frame action to obtain the predicted transition action. The TransformerEncoder weights of the two branches are shared.

[0022] Preferably, both the predicted transition action sequence and the reconstructed keyframe action sequence output by the transition action generation model include two parts: a global 3D coordinate sequence and a global quaternion sequence. The L1 loss function is used to calculate the loss of each part separately, and the two parts are added together to obtain the final model loss L. total .

[0023] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0024] To fill the gap in research on digital sign language human motion choreography, this study constructs a Chinese sign language motion dataset. First, it directly retrieves motion data from a motion database based on user input, eliminating the motion graph construction process. Second, it introduces an online intelligent motion redirection model, which supports multi-role motion choreography by converting multi-role skeletons to a unified skeleton. Finally, it employs a transition motion generation model to address the shortcomings of traditional methods in generating transition motions, ensuring the realism of the generated transition motions. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the motion choreography method for sign language digital humans provided in the embodiment;

[0027] Figure 2 This is a flowchart illustrating the construction process of the Chinese sign language dataset provided in the embodiment;

[0028] Figure 3 This is a flowchart of motion choreography for a sign language digital human provided in the embodiment;

[0029] Figure 4 This is a schematic diagram of the action redirection framework provided in the embodiment;

[0030] Figure 5 This is a simplified comparison diagram of the joint points provided in the embodiment;

[0031] Figure 6 This is an example diagram of the adjusted skeletal pooling operator provided in the embodiment;

[0032] Figure 7 This is a flowchart of the invocation of the first action redirection model provided in the embodiment;

[0033] Figure 8 This is a schematic diagram of the transition action generation model provided in the embodiment generating transition actions;

[0034] Figure 9 This is an example diagram of sign language digital human motion choreography provided in the embodiment. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0036] Figure 1 This is a flowchart of a motion choreography method for sign language digital humans provided in an embodiment. For example... Figure 1 The motion choreography method for sign language digital humans provided in the embodiment includes the following steps:

[0037] Step 1: Construct a Chinese sign language action dataset based on the initial digital human skeleton.

[0038] In this embodiment, the corpus standard is based on the *National Common Sign Language Dictionary*, using its 8609 Chinese sign language entries. For example... Figure 2 As shown, a motion capture system was used at a frame rate of 120 FPS to capture the actions of 6664 Chinese sign language words, including the QingTong Avatar motion capture system and the Glove Feeler acquisition software. The following points should be noted during data acquisition: 1) Start the software and hardware: Open the QingTong Avatar motion capture system and the Glove Feeler acquisition software, turn on the glove receiver switch, and wait for the glove indicator light to stay on; 2) Check the equipment: Confirm that the number of cameras loaded by the Avatar system is correct, check the connection status of the 12 motion capture cameras and 1 RGB camera with the motion capture software, and troubleshoot any wiring or camera malfunctions; 3) Clear the scene: Ensure there are no people or reflective objects in the scene, and turn on the Avatar system to shield ambient light sources to eliminate interference from ambient light; 4) Camera calibration: Use a calibration stick to calibrate the camera by waving it throughout the motion capture scene; place a flexible calibration plate in the scene to set the origin of the world coordinate system. 5) Clothing donning: The sign language interpreter should wear the motion capture clothing (excluding gloves) and be checked. 6) Glove calibration: Place the gloves on a horizontal table, lay them flat, and perform static calibration. The sign language interpreter should wear the gloves, ensuring a proper fit, and then perform dynamic calibration by flipping their hands in a figure-eight motion. The sign language interpreter should extend their hands, with their thumbs perpendicular to their four fingers and their fingers straight, and perform a figure-seven calibration. Verify the correctness after calibration. 7) Motion capture sticker verification: The sign language interpreter should walk into the motion capture area, naturally extend their arms into a t-pose, and confirm that all 53 stickers on the body are in the correct and complete positions. 8) Character creation: Establish the correspondence between sparse stickers and bones. After creation, bind the created character skeleton to the digital human and verify the correctness of the rotation of each joint.

[0039] Based on the above collection precautions, the data collection process includes: 1) The collector switches to a PPT page, and the text of the sign language words to be collected is displayed on the screen; 2) The collector raises their left hand to signal the sign language teacher to prepare; 3) After the collector clicks "Start Recording," they lower their left hand, and the sign language teacher performs the sign language according to the text on the screen; 4) After the sign language teacher finishes performing, the collector clicks the "End Recording" button to save the sign language motion capture data and prepare for the collection of the next sign language word.

[0040] After data collection, the collected data undergoes cleaning and repair, specifically including the following processes: 1) Motion data inspection: The data collection personnel inspect the collected data to check for unnatural or distorted movements, and record these findings. Incorrect motion data resulting from the initial data collection is re-collected; 2) Motion redirection: The collected data is redirected to the initial digital human skeleton. For example, if the initial digital human skeleton is the MetaHuman digital human skeleton, artists use MotionBuilder 3D software to redirect the character created from the QingTong system based on the collected sign language motion capture data to the MetaHuman digital human skeleton; 3) Motion refinement: For motion data that needs to be improved in realism... According to reports, the art staff used MotionBuilder 3D software to refine the collected sign language motion capture data; 4) Skeleton simplification: The MetaHuman digital human skeleton contains 1157 joints, 828 of which are facial joints, accounting for about 71.6% of the total number of joints, and there are a large number of joints used to represent the human body's weight; while the digital human drive only needs to use 61 joints; therefore, it is necessary to further simplify the skeleton of the motion data, retaining only the joints used for drive; 5) Lower body fixation: Sign language movements do not include lower body movements, so it is necessary to fix the global 3D coordinates of the root and the local Euler angles of each joint of the lower body to eliminate the shaking problem of the lower body.

[0041] In this embodiment, an experience-based automated annotation tool is also designed to annotate the end frame of the starting posture and the start frame of the ending posture, respectively, for use in the motion processing stage before the generation of subsequent sign language transition movements. Specifically, the motion data of the first 10 frames and the last 10 frames are first cropped to prevent the process of the arm transitioning from the T-Pose state to the relaxed state from causing misjudgment in subsequent automated annotation. Then, forward kinematics (FK) is calculated to obtain the global three-dimensional coordinates of each joint, and a plane is "cut" from the Z-axis component of the "Spine1" joint. Traversing from the start frame backward, when any joint of the two wrists moves onto the plane, the current frame is taken as the end frame of the starting posture; similarly, traversing from the end frame backward, when any joint of the two wrists moves onto the plane, the current frame is taken as the start frame of the ending posture.

[0042] Step 2: Obtain the Chinese sign language vocabulary, transition frames, and digital human required for choreography.

[0043] In this embodiment, the user-inputted text of choreography requirements is obtained, and the Chinese sign language vocabulary is obtained by splitting the text. At the same time, the specified number of transition frames and the digital human presenting the actions are also obtained. This information is used to guide the generation of choreography.

[0044] For example, when the input text is "The weather is nice today", through the word segmentation algorithm and the corresponding processing of the standard sign language words in the lexicon, the Chinese sign language word sequence obtained is "The weather is good today".

[0045] Step 3: Retrieve the action sequence corresponding to the Chinese sign language word entry from the Chinese sign language action dataset, and parse and preprocess the action sequence to obtain the processed action sequence.

[0046] In the embodiment, the action sequence corresponding to the Chinese sign language word entry is retrieved from the Chinese sign language action dataset. This action sequence is stored in a BVH file, simply referred to as the BVH action sequence. For example, the BVH action sequences corresponding to the three sign language word entries "today", "weather", and "good" in the sign language word sequence are retrieved. By parsing the BVH action sequence, the coordinates and rotation information of each joint relative to the parent joint are extracted, and the rotation information is converted into the required 6D rotation representation to obtain the processed action sequence.

[0047] Step 4: Split the preprocessed action sequence into a hand rotation sequence and a torso action sequence under the initial digital human skeleton, and perform spherical linear interpolation on the hand rotation sequence to generate a hand transition action sequence that meets the transition frame number.

[0048] In the embodiment, as Figure 3 shown, the preprocessed action sequence obtained in Step 3 is split into a hand rotation sequence and a torso action sequence. For the hand rotation sequence, the spherical linear interpolation method is called according to the specified transition frame number to generate transition actions, and a hand transition action sequence that meets the transition frame number is obtained.

[0049] Step 5: Use the first action redirection model constructed based on the generative adversarial network to generate the torso action sequence under the intermediate skeleton according to the torso action sequence under the initial digital human skeleton, the initial skeleton information corresponding to the torso action sequence, and the intermediate skeleton information.

[0050] In the embodiment, adopting the CycleGAN idea, the features of the same action in the general skeleton latent space should be similar enough or the distance should be close enough, so as to force the encoders of different skeletons to learn similar features, improve the inductive ability of the encoder, save a large amount of time and computing resources, and at the same time ensure the effect of subsequent online action redirection, and further ensure the input action quality of the transition action generation based on key frames.

[0051] Action redirection is defined as: Given the action sequence M A under skeleton A, the skeleton information skel A of skeleton A, and the target skeleton information skel B of skeleton B, use the action redirection method to redirect the action sequence M A under skeleton A to the action sequence M BThis ensures that the semantics of the action remain unchanged and the action is natural after redirection.

[0052] Wherein, the skeleton information of A is defined as skel A ={O A E A}, Among them O A E represents the offset of each joint point under skeleton A. A J is the parent joint index of each joint in skeleton A. A The number of joints in skeleton A; similarly, the information for skeleton B is defined as skel. B =O B ,

[0053] The action sequence under skeleton A is defined as M. A ={P A ,R A}, where P A ∈R t*3 This is a sequence of three-dimensional coordinates of the root joints of the human skeleton in the global coordinate system. This provides rotation information for all joints; t represents the length of the motion sequence.

[0054] The action sequence redirected to the skeleton of target B is defined as follows: Among them, D rot The dimension representing rotation information, if represented using quaternions, is D. rot =4. A quaternion is a complex extension used to describe rotations. It consists of one real part and three imaginary parts. Quaternions can be used for rotation and interpolation operations. If 6D rotation is used, then D rot =6, 6D rotation representation refers to the method of representing rotation in six-dimensional space, including: rotation matrix, Euler angle, quaternion, rotation vector, rotation group, and Lie algebra.

[0055] The rotation matrix is ​​a 6x6 matrix used to describe the rotation transformation. The rotation matrix must satisfy the conditions of orthogonality and a determinant of 1. Euler angles are a method of describing rotation; they consist of three angles, representing the rotation angles around the three coordinate axes. The rotation vector is a three-dimensional vector that describes the angle and direction of rotation around an axis. In the 6D rotation representation, the rotation vector is extended to a six-dimensional vector, describing the angle and direction of rotation around any plane. A rotation group is a set of rotation transformations with a specific algebraic structure. In 6D space, the most commonly used rotation group is the SO(6) group, which consists of all 6x6 matrices that satisfy orthogonality and a determinant of 1. The Lie algebra is the algebraic structure used to describe the Lie group, containing the transformation relationships between group elements. In the 6D rotation representation, the Lie algebra of the SO(6) group is a vector space consisting of 6x6 antisymmetric matrices, describing the generating elements of the rotation transformation.

[0056] like Figure 4 As shown, the embodiment provides a generative adversarial network framework for implementing the above action redirection definition, including skel A Skeleton encoder, M A Motion encoder, M B Motion encoder, Skel B Skeleton encoder, M A Motion decoder, M B Action decoder, discriminator A and discriminator B, skeleton information of discriminator A. A After Skel A The skeleton encoder obtains the skel A The encoding results are input into M respectively. A Motion encoder and M A Action decoder, action sequence M under skeleton A A and Skel A The encoding result is processed by M A The motion encoder encodes the motion encoding result Z corresponding to skeleton A. A The action encoding result Z A With Skel A The encoding result in M A The motion decoder obtains the motion decoding result corresponding to skeleton A after decoding. B-skeletal information skel B After Skel B The skeleton encoder obtains the skel B Encode the result and input it into M B Action decoder, action encoding result Z A and Skel B The encoding result in M BThe motion decoder decodes the motion corresponding to the B skeleton. The decoding result of the action In M B The motion encoder obtains the motion encoding result Z corresponding to the B skeleton after encoding. B Discriminator A is used to determine M A With Skel A The authenticity, and With Skel A The discriminator B is used to determine the authenticity of M. B With skel B Authenticity and counterfeiting, and With skel B Authenticity or counterfeit.

[0057] The loss function L of the above generative adversarial network framework includes, based on M A and The constructed M A Reconstruction loss L rec According to Z A and Z B Constructed cyclic loss L cycle Based on M A and The constructed end node loss L ee and discriminator loss L adv This can be expressed as a formula:

[0058] L = w rec *L rec +w cycle *L cycle +w ee *L ee +L adv

[0059] Among them, w rec w cycle and w ee The weights of the loss function are 5, 2.5, and 50, respectively, and the reconstruction loss L is... rec End-node loss L ee The MSE loss is used, and the MSE loss is as follows:

[0060]

[0061] Where, p t Let be the global 3D coordinates of each joint in frame t. Let be the global 3D coordinates of each joint in the predicted t-th frame, D be the evaluation sample set, T be the number of action frames, and h be the character's height.

[0062] In this embodiment, the end node can be the shoulder joint. After performing FK on the shoulder joint, the normalized shoulder joint velocity is obtained based on the global coordinate difference between the previous and next frames and the character's height. The MSE is then calculated and used as a penalty for the distance between the shoulder joints of the two skeletons, so that the redirected shoulder distance is as close as possible, thus solving the problem of sloping shoulders.

[0063] Cyclic loss function L cycle Use L1 loss.

[0064] Discriminator loss in,

[0065]

[0066]

[0067] Where i and j represent frame numbers, T fake T represents the number of sample frames generated. real D represents the number of real sample frames. A and D B Let A and B represent discriminators, and s represent the evaluation sample. This represents the set of generated samples to be evaluated by discriminator A. Skel represents the set of real samples to be evaluated by discriminator A. A,s This represents the skeleton information of the evaluation sample for discriminator A. This represents the action information of the i-th frame of the generated sample input to discriminator A. This represents the action information of the j-th frame of the real sample input to discriminator A. This represents the set of generated samples to be evaluated by discriminator B. Skel represents the set of real samples to be evaluated by discriminator B. B,s This represents the skeleton information of the evaluation sample for discriminator B. This represents the action information of the i-th frame of the generated sample input to discriminator B. This represents the action information of the j-th frame of the real sample input to discriminator B.

[0068] In this embodiment, the inconsistency in motion distribution between the AMASS dataset and the Chinese sign language dataset of the SMPL skeleton is addressed by redirecting the AMASS data to the latter using 3D software to augment the Chinese sign language dataset and to solve the problem of the inability to quantify and evaluate unpaired motion datasets.

[0069] In this embodiment, to improve the accuracy of redirection, skeleton normalization and skeletal pooling operator adjustments are performed. Skeleton normalization uses the distance from the end node of the leg to the end node of the head as the normalization coefficient; while in the upper body dataset, the normalization coefficient is adjusted to the distance from the root joint to the end node of the head. Each node is normalized according to this normalization coefficient, and the result is as follows: Figure 5 As shown.

[0070] The original skeletal pooling operator performs a depth-first search (DFS) from the root, traversing each kinematic chain from the root to each terminal joint. It checks if a point with a degree of 2 exists in each kinematic chain; if so, the two edges connected to it are merged. The process ends when a terminal joint is reached, resulting in the pooled skeleton. However, in the SMPL and MetaHuman skeletons, which only retain the upper body joints, the root joints are terminal joints, making proper pooling impossible. Therefore, this invention adjusts the skeletal pooling operator, starting from the vertebra with the highest degree as the center point and performing a DFS. It traverses each kinematic chain, checking if a point with a degree of 2 exists in each kinematic chain. If so, the two edges connected to it are merged. The process ends when a terminal joint is reached, resulting in the pooled skeleton. An example of the adjusted skeletal pooling operator pooling on the upper body skeleton is shown in the figure below. Figure 6 As shown.

[0071] Based on the aforementioned generative adversarial network framework, when constructing the first action redirection model, the torso action sequence under the initial digital human skeleton is used as M. A The initial skeletal information corresponding to the torso movement sequence is used as the skel A The torso motion sequence under the intermediate skeleton (e.g., SMPL) is used as M B The intermediate skeleton information is used as a skel B The parameters of the generative adversarial network framework are optimized using the aforementioned loss function L. After parameter optimization, Skel... A Skeleton encoder, M A Motion encoder, Skel B Skeleton encoder and M B The action decoder constitutes the first action retargeting model. To ensure the smoothness and realism of the retargeted actions, the data input to the generative adversarial network framework is represented by 6D rotations.

[0072] like Figure 7As shown, the process based on the first motion retargeting model is as follows: the torso motion sequence under the initial digital human skeleton is parsed using BVH to obtain Euler angles, and a 6D rotation representation is obtained by rotation transformation based on the Euler angles. After normalizing the 6D rotation representation, model inference is performed when loading the first motion retargeting model to obtain the inference result. The inference result is then inversely normalized to obtain the 6D rotation representation. The 6D rotation representation is then rotated to obtain Euler angles, and the torso motion sequence under the intermediate skeleton is obtained by saving the Euler angles.

[0073] When using the first motion retargeting model to generate a torso motion sequence under an intermediate skeleton based on the torso motion sequence under the initial digital human skeleton, the initial skeleton information corresponding to the torso motion sequence, and the intermediate skeleton information, the specific steps include: inputting the initial skeleton information corresponding to the torso motion sequence into the skel... A The skeleton encoder obtains the encoding result and inputs it into M. A The motion encoder inputs the torso motion sequence from the initial digital human skeleton to M. A Motion encoder, M A The motion encoder encodes the two inputs to obtain the encoded vector Z under the initial digital human skeleton. A The intermediate skeleton information is input into Skel. B The skeleton encoder obtains the encoding result and inputs it into M. B Action decoder, simultaneously encoding vector Z A Also enter into M B Motion decoder, M B The motion decoder decodes the two inputs to obtain the torso motion sequence under the intermediate skeleton.

[0074] Step 6: Using the transition motion generation model built on the Transformer encoder, generate a torso transition motion sequence that meets the number of transition frames based on the keyframes extracted from the torso motion sequence under the intermediate skeleton.

[0075] In the embodiment, the transition motion generation model built based on the Transformer encoder is as follows: Figure 8As shown, the algorithm comprises two branches, each including a first linear layer, a Transformer Encoder, and a second linear layer. The first branch performs spherical linear interpolation (SLERP & LERP) on the keyframe action to obtain the interpolated frame action. The keyframe temporal index and the required number of transition frames, after initial encoding and linear mapping, are concatenated with the difference between the keyframe action and the reference frame action. This concatenated result is then linearly mapped through the first linear layer of the first branch. The first linear mapping result is then used as the K, Q, and V channels of the attention mechanism and input to the Transformer Encoder to obtain the first branch encoding result. This first branch encoding result is then mapped through the second linear layer, which acts as the decoder, to obtain the first branch decoding result. This first branch decoding result is added to the reference frame action to obtain the reconstructed keyframe action. The transition frame temporal index and the required number of transition frames, after initial encoding and linear mapping, are input to the second branch. The second branch's first linear layer then performs linear mapping to obtain the second mapping result, which serves as the Q channel of the attention mechanism. Simultaneously, the first branch encoding result is used as the K and V channels of the attention mechanism and input to the Transformer Encoder. The Encoder obtains the second branch encoding result, which is then mapped through a second linear layer that acts as the decoder to obtain the second decoding result. This second decoding result is added to the interpolated frame action to obtain the predicted transition action. It should be noted that the Transformer Encoder weights are shared between the two branches.

[0076] Before application, the transition motion generation model requires parameter optimization. During optimization, expanded LaFAN1 and AMASS datasets are used. The LaFAN1 dataset is expanded by flipping the data horizontally. To reduce the difference in motion distribution between the AMASS and Chinese sign language datasets, MotionBuilder 3D character animation software is used to augment the Chinese sign language dataset, helping the model better establish the mapping between the SMPL skeleton data and the MetaHuman skeleton data in the latent space of motion. Specifically, AMASS data is redirected to the latter using 3D software to augment the Chinese sign language dataset. When optimizing the transition motion generation model on the expanded LaFAN1 and AMASS datasets, since sign language actions only include upper body movements and do not involve displacement, the offsets of the lower body joints are removed, along with the rotation information of the corresponding joints, and the root joints are fixed at the origin of the world coordinate system. The predicted transition motion sequence and reconstructed keyframe motion sequence output by the model both include a global 3D coordinate sequence and a global quaternion sequence. The L1 loss function is used to calculate the losses of the two parts separately and then summed to obtain the final model loss L. total, is represented as:

[0077]

[0078]

[0079] L total =L position +L rotation

[0080] Among them, L position L represents the loss function constructed based on the global 3D coordinate sequence. rotation p represents the loss function constructed based on the global quaternion sequence. t and q represents the global 3D coordinates of each joint in the Ground Truth and the global 3D coordinates of each joint in the generated result, respectively. t and These represent the global quaternions of each joint in the Ground Truth and the global quaternions of each joint in the generated result, respectively.

[0081] In this embodiment, keyframe actions are extracted from the torso action sequence under the intermediate skeleton. The keyframe timing index, transition frame timing index, and number of transition frames are all known.

[0082] Step 7: Using the second action redirection model built on generative adversarial network, generate the torso action sequence under the target digital human skeleton from the torso action sequence and torso transition action sequence under the intermediate skeleton, the intermediate skeleton information and the target digital human skeleton information.

[0083] In this embodiment, based on the aforementioned generative adversarial network framework, when constructing the second action redirection model, the torso action sequence under the intermediate skeleton is used as M. A The intermediate skeleton information corresponding to the torso movement sequence is used as skel A The torso movement sequence under the target digital human skeleton is used as M B Using the target digital human skeleton information as a skel B The parameters of the generative adversarial network framework are optimized using the aforementioned loss function L. After parameter optimization, Skel... A Skeleton encoder, M A Motion encoder, Skel B Skeleton encoder and M B The action decoder constitutes the second action redirection model. To ensure the smoothness and realism of the redirected actions, the data input to the generative adversarial network framework is represented by 6D rotations.

[0084] When using the second motion retargeting model to generate the torso motion sequence under the target digital human skeleton based on the torso motion sequence and torso transition motion sequence under the intermediate skeleton, intermediate skeleton information, and target digital human skeleton information, the specific steps include: inputting the intermediate skeleton information corresponding to the torso motion sequence into skel. A The skeleton encoder obtains the encoding result and inputs it into M. A The motion encoder inputs the torso motion sequence under the intermediate skeleton to M. A Motion encoder, M A The motion encoder encodes the two inputs to obtain the encoded vector Z under the intermediate skeleton. A The target digital human skeleton information is input into Skel. B The skeleton encoder obtains the encoding result and inputs it into M. B Action decoder, simultaneously encoding vector Z A Also enter into M B Motion decoder, M B The motion decoder decodes the two inputs to obtain the torso motion sequence under the intermediate skeleton.

[0085] Step 8: Perform skeleton fusion of the hand rotation sequence, hand transition sequence, and torso movement sequence under the target digital human skeleton to obtain the motion choreography result.

[0086] In this embodiment, after obtaining the hand rotation sequence, hand transition motion sequence, and torso motion sequence under the target digital human skeleton, skeleton fusion is performed on all the obtained sequences, and then the overall sign language motion generation result is saved as a BVH motion sequence or exported as video data format. Figure 9 The animation choreography for "The weather is nice today" is shown.

[0087] In this embodiment, to meet the needs of sign language animation generation, the performance of each scheme in upper body and full-body movements was evaluated. Different transition movement generation strategies were adopted for different application scenarios, thereby improving the coherence and naturalness of transition movements in motion choreography. Secondly, an intelligent motion redirection solution for sign language digital humans was implemented. The baseline model was improved from multiple aspects, such as rotation representation and shoulder joint loss. Through upper body movement constraints and expansion of the Chinese sign language dataset, a higher-quality motion redirection method between the SMPL skeleton and the MetaHuman sign language digital human skeleton was implemented, enabling motion choreography that supports multiple characters. Finally, to realize the application of Chinese sign language animation generation, a Chinese sign language motion library was constructed.

[0088] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for choreographing movements for a sign language digital human, characterized in that, Includes the following steps: Construct a Chinese sign language action dataset based on an initial digital human skeleton; Obtain the Chinese sign language vocabulary, transition frames, and target digital human required for choreography; The action sequences corresponding to Chinese sign language terms are retrieved from the Chinese sign language action dataset, and the action sequences are parsed and preprocessed to obtain the processed action sequences; The preprocessed motion sequence is split into a hand rotation sequence and a torso motion sequence under the initial digital human skeleton. Spherical linear interpolation is performed on the hand rotation sequence to generate a hand transition motion sequence that meets the number of transition frames. The first action redirection model based on generative adversarial network is used to generate a torso action sequence under the intermediate skeleton based on the torso action sequence under the initial digital human skeleton, the initial skeleton information corresponding to the torso action sequence, and the intermediate skeleton information. The transition motion generation model based on the Transformer encoder is used to generate a torso transition motion sequence that meets the number of transition frames based on keyframes extracted from the torso motion sequence under the intermediate skeleton. The transition motion generation model based on the Transformer encoder comprises two branches, each including a first linear layer, a Transformer Encoder, and a second linear layer. Spherical linear interpolation is performed on keyframe actions to obtain interpolated frame actions. The keyframe temporal index and the required number of transition frames are initially encoded and linearly mapped, then concatenated with the difference between the keyframe action and the reference frame action. This concatenated result is linearly mapped through the first linear layer of the first branch. The first linear mapping result is then input to the Transformer Encoder as the K, Q, and V channels of the attention mechanism to obtain the first branch encoding result. This first branch encoding result is mapped through the second linear layer, which acts as the decoder, to obtain the first branch decoding result. This first branch decoding result is added to the reference frame action to obtain the reconstructed keyframe action. The transition frame temporal index and the required number of transition frames are initially encoded and linearly mapped, then input to the second branch. The second branch's first linear layer linearly maps this result to obtain the second mapping result, which serves as the Q channel of the attention mechanism. Simultaneously, the first branch encoding result is used as the K and V channels of the attention mechanism and input to the Transformer Encoder. The Encoder obtains the second branch encoding result, which is then mapped through the second linear layer that acts as the decoder to obtain the second decoding result. This second decoding result is added to the interpolated frame action to obtain the predicted transition action. The Transformer Encoder weights of the two branches are shared. The second action redirection model based on generative adversarial network is used to generate the torso action sequence under the target digital human skeleton from the torso action sequence under the intermediate skeleton, the torso transition action sequence, the intermediate skeleton information, and the target digital human skeleton information. The motion choreography result is obtained by fusing the hand rotation sequence, the hand transition motion sequence, and the torso motion sequence under the target digital human skeleton.

2. The motion choreography method for sign language digital humans according to claim 1, characterized in that, The first action redirection model and the second action redirection model are built based on generative adversarial networks (GANs), wherein the GAN framework includes Skel. A Skeleton encoder, M A Motion encoder, M B Motion encoder, Skel B Skeleton encoder, M A Motion decoder, M B Action decoder, discriminator A and discriminator B, skeleton information of discriminator A. A After Skel A The skeleton encoder obtains the skel A The encoding results are input into M respectively. A Motion encoder and M A Action decoder, action sequence M under skeleton A A and Skel A The encoding result is processed by M A The motion encoder encodes the motion encoding result Z corresponding to skeleton A. A The action encoding result Z A With Skel A The encoding result in M A The motion decoder obtains the motion decoding result corresponding to skeleton A after decoding. B-skeletal information skel B After Skel B The skeleton encoder obtains the skel B Encode the result and input it into M B Action decoder, action encoding result Z A and Skel B The encoding result in M B The motion decoder decodes the motion corresponding to the B skeleton. The decoding result of the action In M B The motion encoder obtains the motion encoding result Z corresponding to the B skeleton after encoding. B Discriminator A is used to determine M A With Skel A The authenticity, and With Skel A The discriminator A is used to determine the authenticity of M. B With skel B Authenticity and counterfeiting, and With skel B Authenticity or counterfeit; Based on the generative adversarial network framework, when constructing the first action redirection model, the torso action sequence under the initial digital human skeleton is used as M. A The initial skeletal information corresponding to the torso movement sequence is used as the skel A The torso movement sequence under the intermediate skeleton is used as M B The intermediate skeleton information is used as a skel B The parameters of the generative adversarial network framework are optimized using the loss function L. After parameter optimization, Skel... A Skeleton encoder, M A Motion encoder, Skel B Skeleton encoder and M B The action decoder constitutes the first action redirection model; Based on the generative adversarial network framework, when constructing the second action redirection model, the torso action sequence under the intermediate skeleton is used as M. A The intermediate skeleton information corresponding to the torso movement sequence is used as skel A The torso movement sequence under the target digital human skeleton is used as M B Using the target digital human skeleton information as a skel B The parameters of the generative adversarial network framework are optimized using the aforementioned loss function L. After parameter optimization, Skel... A Skeleton encoder, M A Motion encoder, Skel B Skeleton encoder and M B The action decoder constitutes the second action redirection model.

3. The motion choreography method for sign language digital humans according to claim 2, characterized in that, The loss function L of the generative adversarial network includes, based on M A and The constructed M A Reconstruction loss L rec According to Z A and Z B Constructed cyclic loss L cycle Based on M A and The constructed end node loss L ee and discriminator loss L adv This can be expressed as a formula: L=w rec *L rec +w cycle *L cycle +w ee *L ee +L adv Among them, w rec w cycle and w ee The weights of the loss function are used to calculate the reconstruction loss L. rec End-node loss L ee Using MSE loss, the cyclic loss function L cycle Use L1 loss; Discriminator loss in, Where i and j represent frame numbers, T fake T represents the number of sample frames generated. real D represents the number of real sample frames. A and D B Let A and B represent discriminators, and s represent the evaluation sample. This represents the set of generated samples to be evaluated by discriminator A. Skel represents the set of real samples to be evaluated by discriminator A. A,s This represents the skeleton information of the evaluation sample for discriminator A. This represents the action information of the i-th frame of the generated sample input to discriminator A. This represents the action information of the j-th frame of the real sample input to discriminator A. This represents the set of generated samples to be evaluated by discriminator B. Skel represents the set of real samples to be evaluated by discriminator B. B,s This represents the skeleton information of the evaluation sample for discriminator B. This represents the action information of the i-th frame of the generated sample input to discriminator B. This represents the action information of the j-th frame of the real sample input to discriminator B.

4. The motion choreography method for sign language digital humans according to claim 2, characterized in that, When using the first motion retargeting model to generate a torso motion sequence under an intermediate skeleton based on the torso motion sequence under the initial digital human skeleton, the initial skeleton information corresponding to the torso motion sequence, and the intermediate skeleton information, the specific steps include: inputting the initial skeleton information corresponding to the torso motion sequence into the skel... A The skeleton encoder obtains the encoding result and inputs it into M. A The motion encoder inputs the torso motion sequence from the initial digital human skeleton to M. A Motion encoder, M A The motion encoder encodes the two inputs to obtain the encoded vector Z under the initial digital human skeleton. A The intermediate skeleton information is input into Skel. B The skeleton encoder obtains the encoding result and inputs it into M. B Action decoder, simultaneously encoding vector Z A Also enter into M B Motion decoder, M B The motion decoder decodes the two inputs to obtain the torso motion sequence under the intermediate skeleton.

5. The motion choreography method for sign language digital humans according to claim 2, characterized in that, When using the second motion retargeting model to generate the torso motion sequence under the target digital human skeleton based on the torso motion sequence and torso transition motion sequence under the intermediate skeleton, intermediate skeleton information, and target digital human skeleton information, the specific steps include: inputting the intermediate skeleton information corresponding to the torso motion sequence into skel. A The skeleton encoder obtains the encoding result and inputs it into M. A The motion encoder inputs the torso motion sequence under the intermediate skeleton to M. A Motion encoder, M A The motion encoder encodes the two inputs to obtain the encoded vector Z under the intermediate skeleton. A The target digital human skeleton information is input into Skel. B The skeleton encoder obtains the encoding result and inputs it into M. B Action decoder, simultaneously encoding vector Z A Also enter into M B Motion decoder, M B The motion decoder decodes the two inputs to obtain the torso motion sequence under the intermediate skeleton.

6. The motion choreography method for sign language digital humans according to claim 1, characterized in that, The predicted transition action sequence and the reconstructed keyframe action sequence output by the transition action generation model both include two parts: a global 3D coordinate sequence and a global quaternion sequence. The L1 loss function is used to calculate the loss of each part separately, and the sum is used as the final model loss L. total .

Images