Motion redirection method driven by body part of virtual character and skin maintaining method thereof

Abstract

The invention relates to the technical field of virtual reality, and discloses a virtual character body part driven action redirection method and a skin maintaining method, and the method comprises the following specific steps: extracting joint features of a virtual character, and marking body parts of the virtual character; inputting the body part marks and the joint level features into an adaptive attention network, and extracting body part level features; inputting the body part level features into an action encoder to obtain a shared action code; inputting the skeleton offset data of the virtual character into a skeleton encoder to obtain a skeleton code; the shared action code and the skeleton code are integrated and input into an action decoder, and a redirection action is generated; processing the redirection action by adopting an action discriminator; and alternately updating the redirection action for adversarial training until a final redirection action is generated. According to the method and the device, the problem of poor redirection effect between virtual characters in the prior art is solved, and the method and the device have the characteristic that the accuracy and the flexibility of action redirection can be improved at the same time.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and more specifically, to a method for redirecting actions driven by a body part of a virtual character and a method for maintaining skin. Background Art

[0002] In recent years, with the booming development of digital twins and metaverse, motion redirection plays a key role in virtual-to-virtual and real-to-virtual character motion mapping. The motion redirection algorithm can remap the motion data of the source character to the skeleton of the target character by inputting the motion sequence of the source character and the skeleton topology of the source character and the target character. How to ensure that the redirection result has a satisfactory sense of reality and credibility has always been a key issue in research and application. Traditional methods usually directly convert the complete motion data of one character into the motion data of another character, where the input and output are the position or rotation angle of the character joints. Common public datasets in the field of motion redirection, such as Mixamo, are often used for experiments. However, it is found that the redirected animation obtained by directly training on the existing dataset is very different from the real action, and the character model in the dataset is accompanied by skin penetration problems. Therefore, it is difficult to accurately reflect the quality of redirection by relying solely on evaluation indicators such as joint position error. Even if the evaluation value is good, it cannot guarantee the excellent redirection result.

[0003] In motion retargeting research, two main challenges are usually encountered. First, the skeleton topology or number of joints of the source and target characters must be the same. This strict requirement reduces the flexibility of the motion retargeting model. Second, previous methods mainly rely on joint-level features to evaluate the motion retargeting results, ignoring geometry-level features. Most people are less interested in animated skeletons and more concerned with full character animation with skin.

[0004] In order to bind the target character's skin to its skeleton animation, skinning technology is needed, which is a method used in computer graphics and animation to associate skeleton animation with mesh models. Its main purpose is to drive the mesh model through the action of the skeleton to achieve realistic animation. For this reason, linear blending skin (LBS) is generally used in actual work. The main reason for choosing linear blending skin is the simplicity and efficiency of calculation. However, an obvious disadvantage of linear blending skin is that skin penetration problems are prone to occur when dealing with complex joint deformations, which makes the skin deformation effect unnatural.

[0005] The method first proposed by Gleicher in 1998 regards redirection as a spatiotemporal optimization problem with kinematic constraints. After that, some optimization-based motion redirection methods emerged. For example, inverse dynamics and dynamic constraints are used to optimize the motion redirection process, but these methods require a lot of human intervention. In recent years, research on neural network redirection based on deep learning has become popular. For example, a convolutional encoder-decoder architecture is used to simulate source motion and limb proportions. By integrating DC-IGN and U-Net, motion relocation between different bone lengths is effectively achieved. There is also an unsupervised motion redirection model based on RNN in the prior art to achieve motion redirection between different mesh shapes. There is also an iterative method in the prior art to generate redirected motion based on a trained motion autoencoder. Although deep learning-based methods can achieve automatic motion redirection, the lack of spatial modeling of bone structure limits the redirection accuracy. ET modifies the intermediate redirection result based on the character shape, retains the motion semantics, and realizes the redirection of bones and skin. However, this method has poor effect in redirecting between characters with different numbers of joints, and is limited to the premise that the number of skeleton joints of the source character and the target character is the same.

[0006] In summary, in view of the problem of poor redirection effect between virtual characters in the prior art, how to invent a method that can improve the action redirection effect driven by the body part of the virtual character is a technical problem that urgently needs to be solved in this technical field. Summary of the invention

[0007] In order to solve the problem of poor redirection effect between virtual characters in the prior art, the present invention provides a motion redirection method and a skin preservation method driven by a virtual character body part, which have the characteristics of being able to simultaneously improve the accuracy and flexibility of motion redirection.

[0008] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows: The method for redirecting the action driven by the body part of a virtual character includes the following specific steps: Extract the joint features of the virtual character and label its body parts; Input body part labels and joint-level features into the adaptive attention network to extract body part-level features; Input the body part level features into the action encoder to obtain the shared action encoding; input the skeleton offset data of the virtual character into the skeleton encoder to obtain the skeleton encoding; Integrate shared action encoding and skeleton encoding and input them into action decoder to generate redirection action; use action discriminator to process redirection action; The action discriminator and the action generator are alternately updated to perform adversarial training until the final redirection action is generated.

[0009] Preferably, the joint features of the virtual character are extracted and the body parts are marked, and the specific steps are as follows: Extract the motion sequence of A of the virtual character ,Will Input into the joint embedding layer and map it to the hidden embedding space of each joint to extract joint features :

[0010] in are all learnable parameters of the network. is the number of hidden units in the nonlinear layer, is the dimension of the joint embedding layer; Based on the position information encoding method in Transformer, the joints are indexed as positions and the position encoding of each joint is calculated. :

[0011] in, represents the index of the joint in the skeleton, ; Used to control the sine frequency; The joint embedding is combined with the position encoding in an additive manner to generate the final joint feature representation:

[0012] Introduce a vector with the same dimensions as the joint embedding layer As a body part marker; Describe the number of body parts.

[0013] Furthermore, the body part labels and joint-level features are input into the adaptive attention network to extract body part-level features. The specific steps are as follows: Feed body part landmarks and joint-level features into an adaptive attention network; At every moment , the adaptive attention network combines body part labels and joint-level features As input, and output query vector , key vector Sum value vector : Introducing the mask matrix , perform attention calculation:

[0014]

[0015] In order to ensure that the features of each body part are only aggregated through its internal joints, a mask matrix is ​​introduced It is used to isolate the attention calculation between different body parts, thereby improving the accuracy of the attention mechanism for extracting action features at the body part level. is symmetrical, when When and belonging to the same body part; and It means that they do not belong to the same body part and are not considered in the attention calculation; yes The transpose of Used to keep the gradient within a reasonable range and prevent the gradient from disappearing; Get the final attention calculation result ;Will Expressed as ,in is a joint-level feature, It is a body part level feature.

[0016] Furthermore, the action encoder includes an adaptive attention network and a temporal convolution module; the skeleton encoder includes a multi-layer perceptron; Input the body part level features into the action encoder to obtain the shared action encoding; input the skeleton offset data of the virtual character into the skeleton encoder to obtain the skeleton encoding. The specific steps are as follows: The body part level features are fed into the body part adaptive attention network of the action encoder; Generating hidden features with adaptive attention networks for body parts ; The hidden features are fed into the temporal convolutional block of the action encoder:

[0017] The * indicates the convolution operation. is the weight matrix, is the bias of the convolutional layer, is the number of hidden units, is the width of the filter; Get shared action code ; Offset the avatar's target bone The input skeleton encoder is converted into a skeleton code in the multi-perceptron ;in, Included joints, Indicates the number of joints, Represents the dimension of the feature space; Connect the offsets of the corresponding joints to form a vector ,in The range is 1 to ;based on Obtain the latent codes for different body parts:

[0018] in, , is the number of linear hidden units; Concatenate the latent codes of different body parts to get the final skeleton code: .

[0019] Furthermore, the action decoder includes a deconvolution block; the action discriminator consists of multiple convolutional layers and an average pooling layer; Integrate shared action encoding and skeleton encoding to generate redirection actions; use action discriminator to process redirection actions. The specific steps are as follows: Copy the target skeleton code along the timeline , so that it is have the same shape dimensions; Encode the copied target skeleton Add to source action encoding In the new action code ; Will Input Action Decoder , using deconvolution blocks to generate redirection actions ; The deconvolution block includes upsampling and deconvolution, expressed as: ; Redirect action Input Action Discriminator , each convolution layer compresses the feature size, the compressed features pass through the average pooling layer, and finally output a one-dimensional feature between 0 and 1 through the sigmoid function:

[0020] This one-dimensional feature represents the naturalness of the redirection action sequence.

[0021] Going further, adversarial training is carried out. The specific steps are: First action encoder Actions for source character skeleton A Encode to get action code ; It is divided into two branches; in the first branch, the action decoder Combined with action coding and skeleton bias encoding Reconstruction Action ; In branch 2, the action decoder and target skeleton encoding Synthetic redirect action , and through the action discriminator Ability to evaluate the authenticity of redirection actions based on the actual value and the redirection result; to redirect the action Input to the action encoder , through the loop redirect action to re Position to Above; enforce the consistency of the original action space during the loop redirection process to ensure the action Action with source resemblance.

[0022] Furthermore, when performing the adversarial training, the following loss function is used; Reconstruction loss:

[0023] Cycle consistency loss:

[0024] Fighting Loss:

[0025] Kinematic loss:

[0026] This gives the final loss function: .

[0027] The skin preservation method during action redirection, based on the action redirection method, includes the following specific steps: Decouple the final redirection action to obtain local joint rotations; Obtaining the body shape characteristics of the virtual character; Use multiple multi-layer perceptrons to guide the rotation adjustment of limbs and torso based on local joint rotation and body shape features; The correction parameters are obtained and applied to the local joint rotations to obtain the adjusted joint rotations that can preserve the skin.

[0028] Preferably, multiple multi-layer perceptrons are used to guide the rotation adjustment of the limbs and trunk based on local joint rotation and body shape features, and the specific steps are as follows: Rotate the local joint , body shape characteristics and Input consists of 4 multi-layer perceptron trimmers , and get the quaternion correction ; Specifically:

[0029] in is a learnable parameter, is the number of hidden units in the nonlinear layer, is the dimension of the joint embedding layer.

[0030] Further, the correction parameters are obtained and applied to the local joint rotation to obtain the adjusted joint rotation that can maintain the skin. The specific steps are: Hamilton price Applied to local joint rotations , thus obtaining the adjusted joint rotation :

[0031] Use local joint rotation right Perform linear interpolation to obtain smoothed quaternion rotation ; By smoothing the quaternion rotation , apply linear blending skinning to the mesh vertices of the target character Perform deformation to obtain several pairs of geometric shapes reflecting the target person's adjusted skeleton posture; At each time step, a collection of geometric shapes are detected on the deforming mesh using a bounding volume hierarchy ; A local 3D distance field is defined for each geometric shape. ; If the vertex Lower than , it means that penetration has occurred and a repulsive force needs to be applied to push it to above; if the vertex is above , then no penetration occurs and no operation is required; For all pairs of geometric shapes, define the loss function:

[0032] in, are the vertices of the triangle, is the corresponding normal of each vertex, and It is a geometric shape and The distance field is positive for interpenetrating triangles and zero otherwise. During training, the weight factor Adjust each pair For the loss function Contribution in is a normalization factor and is the maximum geodesic distance between all pairs of vertices in the mesh , and get the final loss function:

[0033] The corrected quaternion is trained by the final loss function , get the adjusted joint rotation that can maintain the skin.

[0034] The beneficial effects of the present invention are as follows: The present invention discloses a method for action redirection driven by a virtual character's body part. By introducing an action redirection strategy based on body parts, the limitation of existing methods that the skeletal structures of the source character and the target character must be the same is broken, thereby realizing flexible action redirection between different skeletal structures. The present invention also improves the spatial modeling capability of the action encoder by introducing an adaptive attention network. The joint weights of the body parts can be dynamically predicted according to the input action posture, and a shared potential space of each body part is constructed through feature pooling technology, thereby improving the accuracy of action redirection. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of a method for redirecting motion driven by a body part of a virtual character.

[0036] Figure 2 It is a concrete framework flow chart of body part driven skeleton redirection.

[0037] Figure 3 It is a schematic diagram of the training process in the motion redirection method driven by the body part of the virtual character.

[0038] Figure 4 This is a flowchart of the skin preservation method during action redirection. Figure 5 This is a diagram of the distance field detection penetration mesh for the skin preservation method during motion redirection. DETAILED DESCRIPTION

[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Example 1 like Figure 1 As shown, the action redirection method driven by the body part of the virtual character includes the following specific steps: Extract the joint features of the virtual character and label its body parts; Input body part labels and joint-level features into the adaptive attention network to extract body part-level features; Input the body part level features into the action encoder to obtain the shared action encoding; input the skeleton offset data of the virtual character into the skeleton encoder to obtain the skeleton encoding; Integrate shared action encoding and skeleton encoding and input them into action decoder to generate redirection action; use action discriminator to process redirection action; The action discriminator and the action generator are alternately updated to perform adversarial training until the final redirection action is generated.

[0041] In this embodiment, in order to achieve flexible action redirection, a body part redirection strategy with body parts as redirection units is introduced. In skeletal actions, the importance of each joint is different, and its importance changes with the change of posture. In order to be able to aggregate the action features at the body part level, an attention network is used to extract the spatial features of the joints. It allows the network to pay attention to different body parts according to their importance, which changes with posture. This is crucial for the body part-driven redirection strategy, which enables the model to dynamically capture the movement patterns of different joints according to the context.

[0042] Example 2 More specifically, in a specific embodiment, the joint features of the virtual character are extracted and the body parts are marked, and the specific steps are as follows: Extract the motion sequence of A of the virtual character ,Will Input into the joint embedding layer and map it to the hidden embedding space of each joint to extract joint features :

[0043] in are all learnable parameters of the network. is the number of hidden units in the nonlinear layer, is the dimension of the joint embedding layer; Based on the position information encoding method in Transformer, the joints are indexed as positions and the position encoding of each joint is calculated. :

[0044] in, represents the index of the joint in the skeleton, ; The frequency used to control the sine wave is usually set to 10,000. This 10,000 is an empirical value that can effectively change the frequency from a smaller value to a larger value to cover different scales; The joint embedding is combined with the position encoding in an additive manner to generate the final joint feature representation:

[0045] Introduce a vector with the same dimensions as the joint embedding layer As a body part marker; To express the number of body parts, in this embodiment, the value N is set to 6, which are the head, spine, left and right arms, and left and right legs.

[0046] In a specific embodiment, the body part labels and joint level features are input into the adaptive attention network to extract the body part level features. The specific steps are as follows: Feed body part landmarks and joint-level features into an adaptive attention network; At every moment , the adaptive attention network combines body part labels and joint-level features As input, and output query vector , key vector Sum value vector : Introducing the mask matrix , perform attention calculation:

[0047]

[0048] in, , is the learnable parameter matrix, ; In order to ensure that the features of each body part are only aggregated through its internal joints, a mask matrix is ​​introduced It is used to isolate the attention calculation between different body parts, thereby improving the accuracy of the attention mechanism for extracting action features at the body part level. is symmetrical, when When and belonging to the same body part; and It means that they do not belong to the same body part and are not considered in the attention calculation; yes The transpose of Used to keep the gradient within a reasonable range and prevent the gradient from disappearing; Get the final attention calculation result ;Will Expressed as ,in is a joint-level feature, is a body part level feature. and Related, and because of The calculated attention weights depend on the variables , so the adaptive attention network proposed based on this point can dynamically extract the spatial features of joint movements.

[0049] In a specific embodiment, Figure 2 As shown in the green part, the action encoder includes an adaptive attention network and a temporal convolution module; the skeleton encoder includes a multi-layer perceptron; Input the body part level features into the action encoder to obtain the shared action encoding; input the skeleton offset data of the virtual character into the skeleton encoder to obtain the skeleton encoding. The specific steps are as follows: The body part level features are fed into the body part adaptive attention network of the action encoder; Generating hidden features with adaptive attention networks for body parts ; The hidden features are fed into the temporal convolutional block of the action encoder:

[0050] The * indicates the convolution operation. is the weight matrix, is the bias of the convolutional layer, is the number of hidden units, which is set to 32 in the experiment. is the width of the filter, which is set to 15 in the experiment, which allows the receptive field to cover half a second of action sequences. For the purpose of time compression, the stride of all convolution kernels is set to 2. , and get the shared action encoding .

[0051] In this embodiment, two layers of temporal convolution blocks are used, and the features of body parts are taken as input; Get shared action code ; like Figure 2 As shown in the yellow part, the target bone offset of the virtual character The input skeleton encoder is converted into a skeleton code in the multi-perceptron ;in, Included joints, Indicates the number of joints, Represents the dimension of the feature space; In this embodiment, the multi-layer sensor is specifically a three-layer sensor; Connect the offsets of the corresponding joints to form a vector ,in The range is 1 to ;based on Obtain the latent codes for different body parts:

[0052] in, , is the number of linear hidden units; Concatenate the latent codes of different body parts to get the final skeleton code: .

[0053] In a specific embodiment, the action decoder includes a deconvolution block; the action discriminator consists of multiple convolutional layers and an average pooling layer; Integrate shared action encoding and skeleton encoding to generate redirection actions; use action discriminator to process redirection actions. The specific steps are as follows: Copy the target skeleton code along the timeline , so that it is have the same shape dimensions; Encode the copied target skeleton Add to source action encoding In the new action code ; like Figure 2 As shown in the blue part, Input Action Decoder , using deconvolution blocks to generate redirection actions ; The deconvolution block includes upsampling and deconvolution, expressed as: ; In order to restore the time dimension layer by layer, the upsampling factor is set to 2, and the spatial dimension of the target action is restored in the last layer, that is, .

[0054] Redirect action Input Action Discriminator , each convolution layer compresses the feature size, the compressed features pass through the average pooling layer, and finally output a one-dimensional feature between 0 and 1 through the sigmoid function:

[0055] In this embodiment, the action discriminator It consists of three convolutional layers and an average pooling layer. The average score of the action discriminator represents the naturalness of the redirected action sequence. During training, the action discriminator and the action generator are updated alternately to promote adversarial training and encourage the action decoder to generate more realistic actions.

[0056] This one-dimensional feature represents the naturalness of the redirection action sequence.

[0057] In a specific embodiment, a cyclic adversarial learning method is used to train encoder-decoder pairs with different structures to achieve self-supervision. This strategy can overcome the problem of insufficient data. The training process is as follows Figure 3 As shown, first the action encoder Actions for source character skeleton A Encode to get action code , followed by two branches, the red branch uses the action decoder Combined with action coding and skeleton bias encoding Reconstruction Action , which helps in training the encoder-decoder pair.

[0058] Conduct adversarial training. The specific steps are: First action encoder Actions for source character skeleton A Encode to get action code ; There are two branches; in the red branch 1, the action decoder Combined with action coding and skeleton bias encoding Reconstruction Action ; In the second branch of black, action decoder and target skeleton encoding Synthetic redirect action , and through the action discriminator The authenticity of the redirection action can be evaluated based on the true value and the redirection result; since the potential action encoding It is based on body parts and can be shared between different skeleton structures. Therefore, to redirect the action Input to the action encoder , through the loop redirect action to re Position to Above; enforce the consistency of the original action space during the loop redirection process to ensure the action Action with source resemblance.

[0059] In a specific embodiment, when performing the adversarial training, the following loss function is used: Reconstruction loss: By minimizing the reconstruction loss, the network can be encouraged to reconstruct the input action more accurately, ensuring that the reconstructed action retains the main features of the original action;

[0060] from Figure 3 The horizontal black line shows that The action encoder , Skeleton Encoder And the motion decoder get.

[0061] Cycle consistency loss: Cycle consistency loss is used to ensure shared action encoding Be consistent during the retargeting cycle. This means that if you retarget the retargeted motion back to the original skeleton again, the result should be similar to the original motion data;

[0062] Adversarial loss: Since the training process is unsupervised, using adversarial loss ensures that the redirected motion looks more natural and matches the motion of the target skeleton. This is achieved by training a discriminator that tries to distinguish between real motion and synthesized motion;

[0063] Kinematic Loss: Motion Sequence Offset from skeleton Combined, they are sufficient to describe a skeleton motion. However, people are usually more interested in the correspondence in Cartesian space, especially the position of the end effector, such as the foot contact point. Therefore, forward kinematics is used Convert the local rotation of each joint to 3D coordinates in Cartesian space. Encourage the network to minimize the joint position during reconstruction and loop reorientation:

[0064] This gives the final loss function:

[0065] in .

[0066] Example 3 like Figure 4 As shown, the skin preservation method during action redirection, based on the action redirection method, includes the following specific steps: Decouple the final redirection action to obtain local joint rotations; Obtaining the body shape characteristics of the virtual character; Use multiple multi-layer perceptrons to guide the rotation adjustment of limbs and torso based on local joint rotation and body shape features; The correction parameters are obtained and applied to the local joint rotations to obtain the adjusted joint rotations that can preserve the skin.

[0067] In a specific embodiment, multiple multi-layer perceptrons are used to guide the rotation adjustment of limbs and torso based on local joint rotation and body shape features, and the specific steps are as follows: Rotate the local joint , body shape characteristics and Input consists of 4 multi-layer perceptron trimmers , and get the quaternion correction ; Specifically:

[0068] in is a learnable parameter, is the number of hidden units in the nonlinear layer, which is set to 256 in the embodiment. is the dimension of the joint embedding layer.

[0069] In this example, while the quaternion correction can adjust the joint rotation to minimize interpenetration due to shape mismatch, this is only an adjustment to the rotation. More obvious interpenetration may still occur at the geometric level (left hand passing through the torso to touch the right arm).

[0070] Therefore, the distance field is introduced to detect the mesh vertices that penetrate, and then the vertices that penetrate are separated from the mesh that is penetrated. Specifically: Get the correction parameters, apply them to the local joint rotation, and get the adjusted joint rotation that can maintain the skin. The specific steps are: Hamilton price Applied to local joint rotations , thus obtaining the adjusted joint rotation :

[0071] Use local joint rotation right Perform linear interpolation to obtain smoothed quaternion rotation ; By smoothing the quaternion rotation , apply linear blending skinning to the mesh vertices of the target character Perform deformation to obtain several pairs of geometric shapes reflecting the target person's adjusted skeleton posture; In this embodiment, these new vertex positions reflect the geometry of the target character in the adjusted skeleton pose. At each time step, a bounding volume hierarchy is used to detect the set of collision triangles on the deformed mesh. .

[0072] At each time step, a collection of geometric shapes are detected on the deforming mesh using a bounding volume hierarchy ; In this embodiment, Figure 5 As shown, a local 3D distance field in the form of a cone is defined for each triangle ; If the vertex Lower than , it means that penetration has occurred and a repulsive force needs to be applied to push it to above; if the vertex is above , then no penetration occurs and no operation is required; For all pairs of geometric shapes, define the loss function:

[0073] in, are the vertices of the triangle, is the corresponding normal of each vertex, and It is a geometric shape and The distance field is positive for interpenetrating triangles and zero otherwise. During training, the weight factor Adjust each pair For the loss function Contribution in is a normalization factor and is the maximum geodesic distance between all pairs of vertices in the mesh , and get the final loss function:

[0074] In this embodiment, the weight factor and Each pair of collision triangles can be adjusted For the loss function These weights help balance the influence of different pairs of triangles on the mesh surface based on their geodesic distance. This can prevent unnatural motion due to slight penetration near joints such as underarms or crotch. The larger the geodesic distance, the larger the weight, reducing the influence of slight penetration in less important areas and giving more weight to penetration in important areas.

[0075] In this embodiment, Set to 5.

[0076] The corrected quaternion is trained by the final loss function , get the adjusted joint rotation that can maintain the skin.

[0077] The present invention proposes a body part driven motion redirection technology, which can achieve redirection of different numbers of joints between source and target characters. In order to improve the accuracy of skeleton motion relocation, the present invention proposes a body part adaptive attention network, which can dynamically capture spatial information and focus on feature extraction of various body parts. The present invention also proposes a skin protection module, which can perform targeted optimization based on the character mesh vertex set data, and introduces a distance field to further solve the penetration problem during training.

[0078] ET modifies the intermediate redirection result based on the character shape, retains the motion semantics, and realizes the redirection of bones and skin. However, this method has poor effect between characters with different numbers of joints, that is, cross-skeleton redirection, and is limited to the premise that the number of skeleton joints of the source character and the target character is the same.

[0079] The method proposed in the present invention uses body parts as redirection units to achieve cross-skeleton action redirection with good effect. At the same time, combining distance field technology on this basis has better performance in reducing skin mesh penetration.

[0080] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for redirecting actions driven by a virtual character's body part, characterized in that: The specific steps include: Extract the joint features of the virtual character and label its body parts; Input body part labels and joint-level features into the adaptive attention network to extract body part-level features; Input the body part level features into the action encoder to obtain the shared action encoding; input the skeleton offset data of the virtual character into the skeleton encoder to obtain the skeleton encoding; Integrate shared action encoding and skeleton encoding and input them into action decoder to generate redirection action; use action discriminator to process redirection action; The action discriminator and the action generator are alternately updated to perform adversarial training until the final redirection action is generated.

2. The method for redirecting actions driven by a virtual character's body part according to claim 1, characterized in that: Extract the joint features of the virtual character and mark its body parts. The specific steps are: Extract the motion sequence of A of the virtual character ,Will Input into the joint embedding layer and map it to the hidden embedding space of each joint to extract joint features : in are all learnable parameters of the network. is the number of hidden units in the nonlinear layer, is the dimension of the joint embedding layer; Based on the position information encoding method in Transformer, the joints are indexed as positions and the position encoding of each joint is calculated. : in, represents the index of the joint in the skeleton, ; Used to control the sine frequency; The joint embedding is combined with the position encoding in an additive manner to generate the final joint feature representation: Introduce a vector with the same dimensions as the joint embedding layer As a body part marker; Describe the number of body parts.

3. The method for redirecting the action driven by the virtual character body part according to claim 2, characterized in that: The body part labels and joint-level features are fed into the adaptive attention network to extract body part-level features. The specific steps are as follows: Feed body part landmarks and joint-level features into an adaptive attention network; At every moment , the adaptive attention network combines body part labels and joint-level features As input, and output query vector , key vector Sum value vector : Introducing the mask matrix , perform attention calculation: In order to ensure that the features of each body part are only aggregated through its internal joints, a mask matrix is ​​introduced It is used to isolate the attention calculation between different body parts, thereby improving the accuracy of the attention mechanism for extracting action features at the body part level. is symmetrical, when When and belonging to the same body part; and It means that they do not belong to the same body part and are not considered in the attention calculation; yes The transpose of Used to keep the gradient within a reasonable range and prevent the gradient from disappearing; Get the final attention calculation result ;Will Expressed as ,in is a joint-level feature, It is a body part level feature.

4. The method for redirecting the action driven by the body part of a virtual character according to claim 3, characterized in that: The action encoder includes an adaptive attention network and a temporal convolution module; the skeleton encoder includes a multi-layer perceptron; Input the body part level features into the action encoder to obtain the shared action encoding; input the skeleton offset data of the virtual character into the skeleton encoder to obtain the skeleton encoding. The specific steps are as follows: The body part level features are fed into the body part adaptive attention network of the action encoder; Generating hidden features with adaptive attention networks for body parts ; The hidden features are fed into the temporal convolutional block of the action encoder: The * indicates the convolution operation. is the weight matrix, is the bias of the convolutional layer, is the number of hidden units, is the width of the filter; Get shared action code ; Offset the avatar's target bone The input skeleton encoder is converted into a skeleton code in the multi-perceptron ;in, Included joints, Indicates the number of joints, Represents the dimension of the feature space; Connect the offsets of the corresponding joints to form a vector ,in The range is 1 to ;based on Obtain the latent codes for different body parts: in, , is the number of linear hidden units; Concatenate the latent codes of different body parts to get the final skeleton code: 。 5. The method for redirecting the action driven by the body part of a virtual character according to claim 4, characterized in that: The action decoder includes a deconvolution block; the action discriminator consists of multiple convolution layers and an average pooling layer; Integrate shared action encoding and skeleton encoding to generate redirection actions; use action discriminator to process redirection actions. The specific steps are as follows: Copy the target skeleton code along the timeline , so that it is have the same shape dimensions; Encode the copied target skeleton Add to source action encoding In the new action code ; Will Input Action Decoder , using deconvolution blocks to generate redirection actions ; The deconvolution block includes upsampling and deconvolution, expressed as: ; Redirect action Input Action Discriminator , each convolution layer compresses the feature size, the compressed features pass through the average pooling layer, and finally output a one-dimensional feature between 0 and 1 through the sigmoid function: This one-dimensional feature represents the naturalness of the redirection action sequence.

6. The method for redirecting the action driven by the body part of a virtual character according to claim 5, characterized in that: Conduct adversarial training. The specific steps are: First action encoder Actions for source character skeleton A Encode to get action code ; Divided into 2 branches; In branch 1, the action decoder Combined with action coding and skeleton bias encoding Reconstruction Action ; In branch 2, the action decoder and target skeleton encoding Synthetic redirect action , and through the action discriminator Ability to evaluate the authenticity of redirection actions based on the actual value and the redirection result; to redirect the action Input to the action encoder , through the loop redirect action to re Position to Above; enforce the consistency of the original action space during the loop redirection process to ensure the action Action with source resemblance.

7. The method for redirecting the action driven by the body part of a virtual character according to claim 6, characterized in that: When performing the adversarial training described above, the following loss function is used; Reconstruction loss: Cycle consistency loss: Fighting Loss: Kinematic loss: This gives the final loss function: 。 8. A skin preservation method during action redirection, characterized in that: The action redirection method according to any one of claims 1 to 7 comprises the following specific steps: Decouple the final redirection action to obtain local joint rotations; Obtaining the body shape characteristics of the virtual character; Use multiple multi-layer perceptrons to guide the rotation adjustment of limbs and torso based on local joint rotation and body shape features; The correction parameters are obtained and applied to the local joint rotations to obtain the adjusted joint rotations that can preserve the skin.

9. The method for redirecting the action driven by the body part of a virtual character according to claim 8, characterized in that: Use multiple multi-layer perceptrons to guide the rotation adjustment of the limbs and torso based on local joint rotation and body shape features. The specific steps are as follows: Rotate the local joint , body shape characteristics and Input consists of 4 multi-layer perceptron trimmers , and get the quaternion correction ; Specifically: in is a learnable parameter, is the number of hidden units in the nonlinear layer, is the dimension of the joint embedding layer.

10. The method for redirecting the action driven by the body part of a virtual character according to claim 9, characterized in that: Get the correction parameters, apply them to the local joint rotation, and get the adjusted joint rotation that can maintain the skin. The specific steps are: Hamilton price Applied to local joint rotations , thus obtaining the adjusted joint rotation : Use local joint rotation right Perform linear interpolation to obtain smoothed quaternion rotation ; By smoothing the quaternion rotation , apply linear blending skinning to the mesh vertices of the target character Perform deformation to obtain several pairs of geometric shapes reflecting the target person's adjusted skeleton posture; At each time step, a collection of geometric shapes are detected on the deforming mesh using a bounding volume hierarchy ; A local 3D distance field is defined for each geometric shape. ; If the vertex Lower than , it means that penetration has occurred and a repulsive force needs to be applied to push it to above; if the vertex is above , then no penetration occurs and no operation is required; For all pairs of geometric shapes, define the loss function: in, are the vertices of the triangle, is the corresponding normal of each vertex, and It is a geometric shape and The distance field is positive for interpenetrating triangles and zero otherwise. During training, the weight factor Adjust each pair For the loss function Contribution in is a normalization factor and is the maximum geodesic distance between all pairs of vertices in the mesh , and get the final loss function: The corrected quaternion is trained by the final loss function , get the adjusted joint rotation that can maintain the skin.

Images