Digital human motion redirection method and device based on skeleton part feature driving

Abstract

The invention provides a digital human motion redirection method and device based on skeleton part feature driving, and the method comprises the steps: obtaining the motion video information of a user through monocular camera equipment, obtaining rough three-dimensional human body key point information through a three-dimensional human body posture estimation model, and preprocessing the rough three-dimensional human body key point information into three-dimensional human body motion capture data; and performing feature extraction on the preprocessed data to generate skeleton part features of the source model. In the marking process, a motion encoder is utilized to extract attitude features of a source model and a target model, and the precision of skeleton part features is optimized through a cyclic generative adversarial network. In the motion redirection process, through dynamic matching and adjustment of skeleton part features, it is ensured that redirected motion data is highly matched with the structure and posture of a target model. And fusing the dynamic features of the source model and the static features of the target model by using a motion decoder to generate a high-quality motion sequence. According to the method, the accuracy and reliability of the motion redirection structure are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and graphics, and in particular to a method and device for redirecting digital human motion based on skeletal feature driving. Background Art

[0002] Skeletal motion retargeting is a technology with important background significance and plays a key role in fields such as computer animation and virtual reality. By remapping the motion data of one character onto the skeletal structure of another character, this technology provides people with many potential applications and benefits. Skeletal motion retargeting technology is of great significance to the fields of animation production and film special effects. By capturing the actions of actors and retargeting them to virtual characters, producers can achieve realistic animation effects. This not only saves a lot of time for manual animation, but also improves the realism and detail performance of animation, making the movements of virtual characters more natural and smooth. Secondly, skeletal motion retargeting has a wide range of applications in game development. The movements of game characters often require a high degree of flexibility and diversity. Through skeletal motion retargeting technology, developers can apply pre-captured movements to different game characters, thereby enriching the movement performance in the game. This technology can also be used for real-time character control, allowing players to interact with the game world in a more natural and intuitive way. In addition, skeletal motion retargeting also plays an important role in the fields of virtual reality and augmented reality. By redirecting the user's actual movements to the virtual character, the user can experience a more immersive and realistic interaction in the virtual environment. This technology can be applied to scenes such as virtual meetings, virtual training, and virtual performances to enhance the user's experience and participation. In addition, skeletal motion redirection also has important application value in digital entertainment and creative industries. By redirecting motion data from different sources, unique and innovative visual effects can be created, providing new creative possibilities for movies, TV shows, advertisements, and art installations. In summary, skeletal motion redirection technology has important applications in many fields. It provides unlimited possibilities for animation production, game development, virtual reality, and digital entertainment, bringing more convenience and innovation to our lives.

[0003] Specifically, skeletal motion retargeting refers to remapping the skeletal motion data of one character to the skeletal structure of another character, so that different characters can share the same motion data. Compared with traditional character animation production methods, skeletal motion retargeting has some obvious differences and difficulties. First, skeletal motion retargeting needs to deal with the differences in skeletal structure between different characters. The bone length, joint position and proportion of each character may be different, which increases the difficulty of accurately mapping the motion data of one character to another. The algorithm needs to be able to flexibly adapt to different skeletal structures to ensure that the retargeted motion is natural and smooth. Secondly, skeletal motion retargeting needs to consider the physical properties and motion characteristics of the character. For example, the weight, center of gravity and range of motion of different characters may be different, and these factors will affect the character's motion performance. The algorithm needs to be able to maintain the physical consistency of the character during the retargeting process to avoid unnatural movements or postures. In addition, skeletal motion retargeting also needs to solve the problems of motion details and expressiveness. High-quality animation usually requires fine motion details and rich expressiveness, and these details are often easily lost during the retargeting process. Therefore, how to accurately convey details and expressiveness while maintaining the original motion style is an important challenge. Finally, skeletal motion redirection also needs to consider the issues of computational efficiency and real-time performance. In many application scenarios, real-time performance is crucial, especially in games and virtual reality, where users expect to see the character's motion effects in real time. Therefore, the algorithm needs to be efficient in time, be able to quickly redirect motion and generate visual results. In summary, skeletal motion redirection has obvious differences and difficulties compared to traditional character animation production methods. It needs to deal with differences in bone structure, physical properties, and motion characteristics, and solve problems of motion details and expressiveness. At the same time, computational efficiency and real-time performance are also important factors to consider. Overcoming these challenges requires combining technologies and algorithms in fields such as computer graphics, physical simulation, and machine learning to continuously improve the accuracy, efficiency, and usability of skeletal motion redirection.

[0004] In view of this, this application is filed. Summary of the invention

[0005] The present invention provides a method and device for digital human motion redirection based on skeletal feature drive, which can at least partially improve the above problems.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A digital human motion redirection method based on skeletal feature-driven, comprising:

[0008] Obtaining the user's motion video information collected by a monocular camera device, detecting the human body area of ​​the user's motion video information using a BlazePose posture estimation network model, generating three-dimensional joint point position sequence information of a source model, converting the three-dimensional joint point position sequence information of the source model into a BVH motion capture data format, and performing simplified processing to retain the main joint motion data, thereby obtaining joint motion data in the form of BVH skeletal motion data;

[0009] Use a preset motion feature extraction network to perform semantic feature deep feature extraction on the joint motion data in the form of the BVH bone motion data, process the static postures of the source model and the target model respectively, extract their respective bone static features, and obtain corresponding source model bone features and target model bone features;

[0010] Based on the source model skeleton features and the target model skeleton features, a preset motion feature extraction network is used to further extract the joint motion data in the form of BVH skeleton motion data, and local and global features of the motion are extracted in parallel, wherein the local branch extracts local motion features through the intra-part joint attention mechanism, and the global branch extracts global motion features through full-body joint convolution;

[0011] The source model skeleton features and the target model skeleton features, as well as the local and global features of the motion are used as input, and a motion decoder is used for decoding processing to obtain a corresponding motion sequence, wherein the motion decoder includes a source model motion decoder and a target model motion decoder;

[0012] The motion sequence is optimized according to a preset final loss function to ensure the integrity and consistency of information in the motion data conversion process between the source model and the target model, and generate motion redirection data, wherein the final loss function includes a reconstruction loss function, a cycle consistency loss function, an adversarial loss function and a forward kinematics loss.

[0013] The present invention also provides a digital human motion redirection device based on skeletal feature-driven, which comprises:

[0014] A data acquisition unit is used to acquire the motion video information of the user collected by a monocular camera device, detect the human body area of ​​the motion video information of the user using a BlazePose posture estimation network model, generate three-dimensional joint point position sequence information of a source model, convert the three-dimensional joint point position sequence information of the source model into a BVH motion capture data format, perform simplified processing, retain the main joint motion data, and obtain the joint motion data in the form of BVH skeletal motion data;

[0015] A static feature extraction unit is used to perform semantic feature deep feature extraction on the joint motion data in the form of the BVH bone motion data using a preset motion feature extraction network, process the static postures of the source model and the target model respectively, extract their respective bone static features, and obtain corresponding source model bone features and target model bone features;

[0016] A local-global feature extraction unit is used to further extract joint motion data in the form of BVH skeleton motion data based on the source model skeleton features and the target model skeleton features using a preset motion feature extraction network, and extract local and global features of the motion in parallel, wherein the local branch extracts local motion features through an intra-part joint attention mechanism, and the global branch extracts global motion features through full-body joint convolution;

[0017] A motion sequence generating unit, used for taking the source model skeleton features and the target model skeleton features, as well as the local and global features of the motion as input, and performing decoding processing by using a motion decoder to obtain a corresponding motion sequence, wherein the motion decoder includes a source model motion decoder and a target model motion decoder;

[0018] An optimization unit is used to optimize the motion sequence according to a preset final loss function to ensure the integrity and consistency of information in the process of motion data conversion between the source model and the target model, and generate motion redirection data, wherein the final loss function includes a reconstruction loss function, a cycle consistency loss function, an adversarial loss function and a forward kinematics loss.

[0019] In summary, the digital human motion redirection method based on skeletal feature drive first uses a monocular camera device to obtain the user's motion video information, obtains rough 3D human key point information through a 3D human posture estimation model, and preprocesses it into 3D human motion capture data. Next, feature extraction is performed on the preprocessed data to generate the skeletal features of the source model. The labeling process uses a motion encoder to extract the posture features of the source model and the target model, and optimizes the accuracy of the skeletal features through a cyclic adversarial generation network. In the motion redirection process, the skeletal features are dynamically matched and adjusted to ensure that the redirected motion data is highly consistent with the structure and posture of the target model. Finally, a motion decoder is used to fuse the dynamic features of the source model and the static features of the target model to generate a high-quality motion sequence. Compared with the prior art, the beneficial effects of the present invention are embodied in:

[0020] 1. High-quality motion redirection: Compared with traditional methods, the present invention can more accurately capture and reproduce complex motion details, avoiding visually too smooth or rigid redirection effects. By using a motion generation model driven by part features, the accuracy and authenticity of the geometric structure of the motion are maintained.

[0021] 2. High scalability of monocular camera application scope: The present invention only uses the monocular camera device commonly equipped in daily electronic devices to obtain the user's motion video information, without relying on complex multi-view or depth sensor devices. This makes the method of the present invention more convenient and practical, and can be widely used in various smart devices and application scenarios, providing users with a better human motion redirection experience.

[0022] 3. High applicability of unsupervised learning methods: The present invention adopts an unsupervised motion redirection method, which does not require a large-scale labeled 3D motion data set as training samples. This reduces the cost of data collection and annotation, making the method more flexible and scalable, and can be applied to a variety of different skeletal models and motion types.

[0023] 4. Automation and efficiency: The method of the present invention utilizes the motion generation model driven by part features and deep network structure optimization technology to automatically redirect and optimize motion data. By encoding and adjusting the dynamic features of the source model and the target model, high-quality motion redirection results can be quickly generated, reducing the need for manual intervention and improving the efficiency and accuracy of motion redirection.

[0024] 5. Maintaining the smoothness of motion: The method of the present invention can maintain the smoothness of motion of the source model and the target model during the motion redirection and data conversion process. This means that the generated motion data will remain consistent in shape and topology without distortion or distortion, providing more reliable motion redirection results and facilitating further animation production or virtual reality applications.

[0025] 6. Application prospects and commercial value: The part feature-driven motion redirection method of the present invention has broad application prospects and commercial value in the fields of game development, virtual reality, animation production, and digital entertainment. It provides an effective solution for the technological development in these fields, can improve the quality and accuracy of motion redirection, meet users' needs for high-quality animation and virtual experience, and promote the development and innovation of related industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a digital human motion redirection method based on skeletal feature-driven provided by the first embodiment of the present invention;

[0027] Figure 2 It is a model framework diagram of a digital human motion redirection method based on skeletal feature-driven provided by the first embodiment of the present invention;

[0028] Figure 3 It is a module schematic diagram of a digital human motion redirection device driven by skeletal features provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0030] refer to Figure 1 , Figure 2 As shown, the first embodiment of the present invention discloses a digital human motion redirection method based on skeletal feature-driven, which can be executed by a digital human motion redirection device based on skeletal feature-driven (hereinafter referred to as redirection device), and in particular, executed by one or more processors in the redirection device to implement the following method:

[0031] S1, obtaining the motion video information of the user collected by the monocular camera device, using the BlazePose posture estimation network model to detect the human body area of ​​the motion video information of the user, generating the three-dimensional joint point position sequence information of the source model, converting the three-dimensional joint point position sequence information of the source model into the BVH motion capture data format, and simplifying it, retaining the main joint motion data, and obtaining the joint motion data in the form of BVH skeletal motion data;

[0032] Specifically, step S1 includes: obtaining motion video information I of the user collected by a monocular camera device;

[0033] The BlazePose posture estimation network model is used to detect the key points of the human body in the user's motion video information I, and generate a video joint point set V of the source model. o , the formula is: V o = BlazePose(I), where represents the set of human joint points in the i-th video frame, represents the set of human joint points in the Fth video frame, F is the number of frames of the motion video, J is the number of bones in the model, represents the Jth human joint point in the i-th video frame;

[0034] The video joint point set V of the source model o and the joint point set V of the digital human target model t o Perform unified simplified matching processing to simplify the matching process and calculation complexity. The simplified matching formula of the joint point is: Among them, finrm represents the finger removal operation of the joint point, V t ′ is the 3D joint position sequence information of the source model, V tThe three-dimensional joint position sequence information of the digital human target model;

[0035] The three-dimensional joint point position sequence information V of the source model t ′ and the three-dimensional joint position sequence information V of the digital human target model t Convert the three-dimensional coordinates into BVH bone motion data to obtain joint motion data in the form of BVH bone motion data. The conversion formula is: o ,M o}=convert(V t ′),{S t ,M t}=convert(V t ), where S o is the target bone joint offset of the source model, S t is the target bone joint offset of the digital human. represents the motion data of the source model, represents the motion data of the i-th source model, The unit quaternion representing the local joint rotation for each joint, Represents the global motion speed of the root joint in the x-axis, y-axis, and z-axis directions, M t Represents the motion data of the digital human target model.

[0036] In this embodiment, motion data collection and preprocessing are performed. A monocular camera device is used to obtain the user's motion video information, and a posture estimation network is used to detect the human body area to generate the three-dimensional joint point position sequence information of the source model. Subsequently, these sequence data are converted into BVH (BiovisionHierarchy) motion capture data format, and the data is simplified to retain only the main joint motion data.

[0037] Specifically, in this embodiment, for the human body motion image information obtained by the monocular camera commonly equipped in daily electronic devices, a posture detection network is used to obtain the three-dimensional coordinate information of the human body joints in the original image. The BlazePose posture estimation network model is used to obtain the three-dimensional coordinate information of the human body joints in the original video. (Bazarevsky V, Grishchenko I, Raveendran K, et al. Blazepose: On-device real-time body pose tracking [J]. arXiv preprint arXiv: 2006.10204, 2020.). The posture estimation model is used to detect the key points of the human body in the original video to generate a video joint point set V of the source model. o , the above process can be described as: V o= BlazePose(I), where Represents the set of human joint points in the i-th video frame, F is the number of frames of the motion video, J is the number of bones in the model, and I is the original human motion video data.

[0038] In order to reduce the complexity of motion redirection, the set V of video joint source models is o and the joint point set of the digital human target model Perform unified simplified matching, mainly removing the finger parts, thereby simplifying the matching process and computational complexity. The simplified matching process of joint points is expressed as: V t ′=finrm(V o ), Wherein, finrm represents the joint point finger part removal operation. After completing the simplified matching of the joint point set, it is necessary to convert the joint point set of the source model from the three-dimensional coordinates into the skeletal motion data form of the present invention for further processing and application. The conversion operation is expressed as follows: {S o ,M o}=convert(V t ′),{S t ,M t}=convert(V t ), where S o , S t Represents the offset of the source model target and the digital human target skeleton joints. The offset of each joint in the skeleton is a three-dimensional vector relative to the coordinate system of its parent joint. Represents the motion data of the source model, where the motion data of the i-th frame The unit quaternion representing the local joint rotation for each joint, Represents the global movement speed of the root joint in the x, y and z directions, as well as the additional angular velocity of the rotation speed around the axis perpendicular to the ground (set to the y axis in the present invention).

[0039] S2, using a preset motion feature extraction network to perform semantic feature deep feature extraction on the joint motion data in the form of the BVH skeleton motion data, respectively processing the static postures of the source model and the target model, extracting their respective skeleton static features, and obtaining corresponding source model skeleton features and target model skeleton features;

[0040] Specifically, step S2 includes: using a preset motion feature extraction network to perform semantic feature deep feature extraction on the joint motion data in the form of the BVH skeleton motion data, wherein the motion feature extraction network is composed of a skeleton encoder Enc s and action encoder Enc m Composition, skeleton encoder Enc s Include source model skeleton encoder and target model skeleton encoder Action Encoder Enc m Includes source model action encoder and target model action encoder

[0041] The source model and the target model use the source model skeleton encoder and target model skeleton encoder Encode to obtain the corresponding source model bone features and target model bone features The formula is:

[0042] Among them, the source model skeleton encoder and target model skeleton encoder Both use a two-layer perceptron MLP, whose output is a 1×P×d1 feature, where P=6 represents the six main body parts and d1=32 represents the dimension of the feature.

[0043] In the present embodiment, the motion generation model driven by part feature is redirected. In this step, the motion generation model based on part feature is utilized to carry out motion redirection. First, the static posture of source model and target model is processed respectively by motion encoder to extract its respective skeleton static features. Because source model and target model may have differences in the topological structure of skeleton, different encoders are used to encode their static features, and its static posture is mapped to corresponding skeleton static feature representation. These feature representations will provide necessary basis for ensuring the accuracy and consistency of data in subsequent dynamic feature encoding and motion redirection process.

[0044] Specifically, the preprocessed skeletal motion data is input into the motion feature extraction network for deep feature extraction of semantic features. The motion feature extraction network consists of two modules: the skeletal encoder and the action encoder. s and Enc m Indicates that since the skeletal topology of the source model may be different from that of the target model, based on the skeletal topology, the encoder is divided into two categories: source model encoder and target model encoder. The source model encoder includes the skeletal encoder With action encoder The target model decoder includes a skeleton encoder With action encoder The two skeleton models are encoded using corresponding encoders, and the corresponding skeleton features are obtained and action features The representation forms are:

[0045] Specific source model skeleton encoder and target model skeleton encoder Both are implemented using a two-layer perceptron (MLP) and the output is a 1×P×d1 feature, where P=6 represents the six main body parts and d1=32 represents the dimension of the feature.

[0046] S3, based on the source model skeleton features and the target model skeleton features, using a preset motion feature extraction network to further extract the joint motion data in the form of BVH skeleton motion data, and extract local and global features of the motion in parallel, wherein the local branch extracts local motion features through the intra-part joint attention mechanism, and the global branch extracts global motion features through full-body joint convolution;

[0047] Specifically, step S3 includes: source model action encoder and target model action encoder Both use two branches to extract local and global features of motion in parallel;

[0048] The motion input at a given time frame i is the motion data of the i-th source model The motion data of the i-th source model is input through the joint embedding layer Mapped to the hidden embedding space, the formula is: where W0∈R 4×l ,W1∈R l×d ,W2∈R l×d ,b0∈R l ,b1∈R l ,b2∈R d are all network mapping parameters, l is the number of hidden units used in nonlinear mapping, and d is the dimension of the joint embedding space;

[0049] Use joint-level position encoding to add the position of each joint and use the joint index in the skeleton as the position. The formula is: Where j represents the index of the joint in the skeleton, k∈[1,2,3,...,d] is the dimension index of the position code, and mod is the modulus operation;

[0050] After the position encoding of each joint is completed, the joint embedding and the position encoding are combined in an additive manner to obtain the enhanced joint motion representation, as follows: Among them, e i i represents the enhanced joint motion representation of the i-th frame;

[0051] The joint features are integrated and aggregated into the body part feature encoding to obtain the body part label j ii ;

[0052] For each video frame, body part labels and joints are embedded into features The combination of is taken as input and the query vector is output through the self-attention model With key-value pairs and The formula is: Among them, ATT is the self-attention function;

[0053] The body parts are separated by a mask matrix U, which is symmetric, where U k,j = 1 means that joints k and j are in the same body part, and the final body joint feature is expressed as: in, represents the features of the part mark of the i-th frame, represents the joint features of the i-th frame, is the video joint point of the i-th source model, and N is the number of body parts.

[0054] According to the motion data M of the source model o , perform masked two-dimensional convolution processing on bones and time joint by joint to obtain the spatiotemporal features of each joint. The formula is: Among them, d j represents the jth joint feature, N o (j) represents the set of adjacent joint points with joint point j as the vertex, T j represents the adjacency mask matrix of the j-th joint, Indicates that the kth joint is adjacent to the jth joint, W c is the convolution kernel parameter matrix, b c ∈R d is the bias vector, * represents a two-dimensional convolution operation;

[0055] Get all joint features All joint features

[0056] Perform global two-dimensional convolution feature enhancement operation on all joint features, the formula is: Conc(D o )=Relu(D o *W conv +b conv ), where Conc is a two-dimensional convolution function, W conv is the convolution kernel, b conv is the bias vector;

[0057] The local features are combined with the global features in an additive manner, and the final spatiotemporal motion features are obtained through convolution operations. The formula is:

[0058] In this embodiment, the static features of the source model are used to further analyze and process the dynamic features of the source model so that it can match the skeletal structure of the target model, thereby achieving accurate motion data redirection. This process ensures smooth conversion and high-quality redirection of motion data between different models through refined processing and feature mapping of joint points. and target model action encoder Both branches are used to process the local and global features of the motion in parallel. The local branch extracts local motion features through the intra-part joint attention mechanism, while the global branch extracts global motion features through full-body joint convolution.

[0059] First, local feature extraction is performed. The local branch will process motion gestures frame by frame. Given the motion input at time frame i The step of mapping the input to the hidden embedding space through the joint embedding layer can be expressed as follows: Among them, {W0∈R 4×l ,W1∈R l×d ,W2∈R l×d ,b0∈R l ,b1∈R l ,b2∈R d}Network mapping parameters, where l is the number of hidden units used in the nonlinear mapping, and d is the dimension of the joint embedding space. In this embodiment, l = 256 and d = 32.

[0060] Inspired by the position encoding in the Transformer network, joint-level position encoding is used to mark the position of each joint in an additive manner. The original mathematical formula of position encoding is modified, and the joint index in the skeleton is used as the position. The formula is as follows: Where j is the index of the joint in the skeleton, k is the dimension index of the position encoding, k∈[1,2,3,...,d], d is the dimension of the joint embedding space, and mod is the modulo operation used to determine the parity of i. This is the same as in most Transformer implementations. Joint position encoding is more bounded, smoother, and denser than one-hot encoding, and the vector dimension of joint position encoding is independent of the number of skeletal joints, which makes it very suitable for encoding when the skeletal structure changes.

[0061] After completing the position encoding of each joint, the joint embedding is further combined with the position encoding in an additive manner to obtain an enhanced joint motion representation: in, Represents the joint motion representation enhanced in the i-th frame. Since the redirection strategy of this method is to use body parts as basic units for redirection, it is necessary to integrate the joint part features and aggregate them into the body part feature encoding. Therefore, a learnable part tag is added to each body part before joint embedding, and the part tag aggregates the joint features of the entire part using attention output. Specifically, the present invention introduces a learnable parameter called "Joint Token" for each body part, which has the same dimension d as the joint embedding, and the parameter is denoted as j o ∈R N×d , wherein N=6 represents the number of body parts defined in the present invention.

[0062] For each video frame i, we first embed the body part labels and joints into features as input and output the query vector through the self-attention model and key-value pairs and The formula is as follows: ATT is the self-attention function. Before calculating the final body joint features, a mask matrix U is used to separate the body parts to ensure that the body part features are completely generated by aggregating the features of the internal joints of the parts. The mask matrix U is symmetrical, where U k,j = 1 means joints k and j are in the same body part, otherwise U k,j =0, so the final body joint features can be expressed as: in represents the features of the part mark of the i-th frame, Indicates the joint features of the i-th frame. In subsequent operations, only the value of the part tag "Joint Token" is used Because in the process of motion redirection, using part features as redirection units is more advantageous than directly using joint features.

[0063] Specifically, part features can better abstract and generalize the common topological structures of different models. Each part feature represents the overall characteristics of a body part, rather than the local characteristics of a single joint. This can effectively reduce the inconsistency problems caused by different bone topological structures, simplify the computational complexity, make the redirection process more efficient, and ensure consistent feature expression when processing different source and target models.

[0064] Next, global feature extraction is performed. The global branch performs convolutions on each joint in time and on the skeleton, using convolution kernels that have local scope on both the skeleton and time axis. It is important to note that although the convolution kernels share weights on the time axis, they do not share weights between different skeletal joints. This is because different body parts have different movement patterns, so the features extracted in each part may be unique. For example, the features extracted by a convolution kernel centered on an arm joint may be different from the features extracted by a convolution kernel centered on a knee joint.

[0065] Specifically, the motion data M of the source model is obtained o Finally, perform skeletal and temporal masked 2D convolution joint by joint to obtain the features of each joint: Among them, d j represents the jth joint feature, N o (j) represents the set of adjacent joint points with joint point j as the vertex, T j represents the adjacency mask matrix of the j-th joint, Indicates that the kth joint is adjacent to the jth joint, and vice versa W c is the convolution kernel parameter matrix, b c ∈R d is the bias vector, and * represents a two-dimensional convolution operation.

[0066] After obtaining the spatiotemporal features of each joint, the features of all joints can be obtained Furthermore, a global two-dimensional convolution is performed on all joint features to enhance the features: Conc(D o )=Relu(D o *W conv +b conv ), where Conv is a two-dimensional convolution function, W conv Convolution kernel, b conv is the bias vector, and * represents a two-dimensional convolution operation. After completing the feature extraction of the two branches, the local features are further combined with the global features in an additive manner, and the final spatiotemporal motion features are further obtained through convolution operations.

[0067] S4, taking the source model skeleton features and the target model skeleton features, as well as the local and global features of the motion as input, and performing decoding processing using a motion decoder to obtain a corresponding motion sequence, wherein the motion decoder includes a source model motion decoder and a target model motion decoder;

[0068] Specifically, step S4 includes: the source model motion decoder and target model motion decoder Both use double-layer temporal deconvolution Deconv;

[0069] Using source model motion decoder and target model motion decoder Decode the source model skeleton features and the target model skeleton features, as well as the local and global features of the motion, to obtain the corresponding motion sequence, the formula is: Among them, Upsampling represents the upsampling operation, W dec and b dec are the weight matrix and bias vector of the trainable convolution kernel respectively. Deconv1 and Deconv2 are two double-layer temporal deconvolution layers respectively.

[0070] In this embodiment, the motion decoder takes the dynamic features of the source model and the static features of the target model as input, and decodes the features into a motion sequence suitable for the target model. The role of the decoder is to fuse and transform the complex motion features and static features extracted by the encoder to generate a high-quality, realistic and smooth target model motion sequence. By learning the complex motion relationship between the part features of the source model and the target model, it is ensured that the generated motion data maintains consistency and high fidelity with the source model in appearance and action. This process provides reliable data support and guarantee for subsequent steps, meeting the requirements of smooth conversion and high-quality redirection of motion data.

[0071] Specifically, the motion decoder is responsible for converting the encoded motion feature vector into a motion sequence suitable for the target model. Similar to the encoder of the motion feature extraction network, the decoders are divided into two categories based on the skeleton topology. The motion decoder of the source model is represented as The motion decoder of the target model is represented as The specific motion decoders all use a double-layer temporal deconvolution Deconv, and the two skeleton models are decoded using corresponding decoders. The corresponding motion sequence obtained is in the form of: Where Upsampling represents the upsampling operation, W dec and b dec are the weight matrix and bias vector of the trainable convolution kernel. Deconv1 and Deconv2 are two double-layer temporal deconvolution layers. The convolution kernel width and hidden unit size are the same as those of the above temporal convolution layer. The stride of the convolution kernel is set to 1 and the upsampling factor is set to 2.

[0072] S5, optimizing the motion sequence according to a preset final loss function to ensure the integrity and consistency of information in the motion data conversion process between the source model and the target model, and generating motion redirection data, wherein the final loss function includes a reconstruction loss function, a cycle consistency loss function, an adversarial loss function and a forward kinematics loss.

[0073] Specifically, step S5 includes: using the bone features of the source model and the final space-time motion characteristics Generate reconstructed motion data

[0074] Construct a reconstruction loss function to represent the error between the reconstructed motion and the source motion. The formula of the loss term is: Among them, ‖.‖ represents the second normal form symbol, is the error between the reconstructed motion and the source motion, To reconstruct motion data;

[0075] Construct a cycle consistency loss function, whose loss term formula is: in, is the loss term of the cycle consistency loss function, Redirects the motion for the source model loop.

[0076] Construct an adversarial loss function to ensure that the redirected target motion is reasonable. The adversarial loss term formula is: in, is the true motion of the target model, and C is the discriminator consisting of two convolutional layers;

[0077] Construct a forward kinematics loss function, the formula is:

[0078] Get the final loss function Among them, λ1=2.5, λ2=100.

[0079] In this embodiment, part feature driven motion generation model optimization is performed. To further improve the quality and consistency of motion redirection, two key loss functions, cycle consistency loss and posture consistency loss, are introduced. Cycle consistency loss ensures that the integrity and consistency of information are maintained during the motion data conversion process between the source model and the target model. Posture consistency loss further enhances the accuracy of motion redirection, especially in complex movements and detail processing. The comprehensive application of these loss functions effectively improves the system's ability to handle complex motion sequences and ensures the realism and quality of the final generated animation.

[0080] The model structure of this method is based on a cycle consistency adversarial network. A pair of generators and discriminators are used to convert data between the source domain and the target domain, and another pair of generators and discriminators are used to ensure that the converted data can maintain consistency between the two domains. Specifically, the motion encoder, skeleton encoder, and motion decoder are put together as a set, called the generator G = {Enc s,Enc m ,Dec m The discriminator C consists of two convolutional layers and receives the redirected motion M t As input, the parameters between the generator G and the discriminator C are updated alternately, while the parameters of the other network are fixed. The loss function is explained in detail below:

[0081] The first one is the reconstruction loss function. During the training process, the skeleton features and spatiotemporal features of the source model are first used to generate reconstructed motion data. To train the source model encoder-decoder pair. Specifically, a reconstruction loss function is constructed to represent the error between the reconstructed motion and the source motion. The loss term can be expressed by the following formula: Here, ‖.‖ represents the second normal form symbol.

[0082] The second one is the cycle consistency loss function, due to the part motion characteristics of the source model It is composed of part-level features. The part features of the same motion should be similar between models with different bone structures, which can maintain the consistency of the part features during cyclic redirection. In addition to the consistency of part motion features, the cyclic redirection also requires consistency in the original motion space, that is, the cyclic redirection motion of the source model. Should be with source motion M o Based on this idea, a cycle consistency loss function is constructed, and the loss term is expressed as follows: in is the motion decoder representation of the source model, and the motion decoder of the target model The structure is the same.

[0083] The third, adversarial loss function, requires a discriminator C to ensure the redirected target motion M since the model is trained in an unsupervised manner. t is reasonable and conforms to the motion pattern of the target skeleton structure. The discriminator C consists of two convolutional layers, so the following adversarial loss term is used: in, is the true motion of the target model. When the discriminator is trained alone, the generated redirected target motion M t Separate from the gradient calculation graph and maximize the loss function To encourage the discriminator to distinguish between real data and synthetic motion. While training the generator, minimize the function and other loss terms to deceive the discriminator and thus obtain more realistic redirection results.

[0084] Fourth, the forward kinematics loss function, in order to more accurately capture and redirect motion, the learning framework uses the standard forward kinematics function (FK) to convert the local rotation of each joint into a three-dimensional coordinate in Cartesian space. During the reconstruction and loop redirection process, the network is encouraged to generate more accurate and natural motion sequences by minimizing the three-dimensional space position error: in To reconstruct motion data, is the cyclic redirection motion of the source model. The final loss function can be summarized as follows: In this embodiment, λ1 = 2.5, λ2 = 100. The entire network learns the generator and discriminator of motion redirection in an end-to-end manner, and the minimax strategy can be effectively implemented using the stochastic gradient descent optimization strategy until the network converges.

[0085] In this embodiment, the effectiveness of this method in motion redirection between humanoid skeletons is mainly evaluated. According to the difference in the skeletal structure of the source model and the target model, motion redirection is divided into two cases. One is intra-structure redirection, that is, the target skeleton has the same joint structure, but different bone proportions. The other is cross-structure redirection, in which the target skeleton has a similar topology to the source skeleton, but a different number of joints. In order to take these two cases into account, this method divides the characters into two groups, A and B, according to the skeletal structure, and each group of characters has the same skeletal structure but different bone proportions. Compared with structure B, the skeleton of structure A contains additional joints on each limb (left arm / right arm, left leg / right leg, spine and neck).

[0086] Mixamo is a 3D motion dataset with rich motion types, containing more than 2,000 motion sequences of 29 different humanoid characters, of which structure A includes 24 characters and structure B includes 5 characters. In this experiment, 20 characters in structure A and 4 characters in structure B were selected for training, and the remaining characters were used for testing.

[0087] The comparison methods used in this embodiment are NKN, PMnet, and SAN. NKN is a pioneering work in motion redirection technology based on deep learning. It uses a recurrent neural network (RNN) to model the temporal relationship of motion and combines skeleton offsets for intra-structure redirection. PMnet learns frame-by-frame poses and overall motions separately to improve redirection accuracy. The most similar to this work is SAN, which extracts skeleton features through skeleton-based graph convolutions, which can be applied to skeletons of different structures. In the following sections, it will be demonstrated that attention-based spatial modeling at the body part level is more conductive and flexible for motion redirection tasks.

[0088]

Mixamo

[0089] [NKN] R.Villegas, J.Yang, D.Ceylan, and H.Lee, "Neural kinematic networks for unsupervised motion retargeting," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp.8639–8648.

[0090] [PMnet] J.Lim, HJChang, and JYChoi, "Pmnet: Learning of disentangledpose and movement for unsupervised motion retargeting." in BMVC, 2019, p.136.

[0091] [SAN] K. Aberman, P. Li, D. Lischinski, O. Sorkine-Hornung, D. Cohen-Or, and B. Chen, "Skeleton-aware networks for deep motion retargeting," ACM Transactionson Graphics (TOG), vol.39, no.4, pp.62–1, 2020.

[0092] For quantitative evaluation, the mean position error of skeleton (MPJPE) from the [PMnet] paper is adopted as the metric, and 106 generated results are compared with the real motion sequences.

[0093] The quantitative results are shown in Table 1. Since the target skeleton has a one-to-one joint correspondence with the source skeleton when redirecting the target motion within the structure, the result of the joint rotation copy is used as the benchmark, that is, each joint rotation is directly copied to the target skeleton. As can be seen from the table, the performance of NKN and PMnet lags far behind SAN and the method of the present invention, because their networks simply concatenate the whole body joint features together instead of modeling the spatial and temporal characteristics of the skeleton. The present method outperforms all competing methods in both intra-structure and cross-structure redirection, thanks to the part-based redirection strategy.

[0094] Table 1 Quantitative evaluation of redirection results

[0095]

[0096]

[0097] Experimental verification shows that this method has a high degree of automation, significant practical application effects, and meets the needs. This method is widely used in animation redirection, virtual live broadcast, virtual reality and other fields, providing an effective solution, improving the accuracy and reliability of motion redirection structure, and has broad application prospects and commercial value.

[0098] In summary, the digital human motion redirection method driven by skeletal part features is used to redirect smooth animation sequences from digital human models with different topological structures. This method uses part feature adversarial learning to learn posture feature consistency, which can effectively learn the same posture features between different skeletal models and improve the accuracy of the motion redirection process. The innovation of this method lies in the proposal of a powerful motion redirection framework, which uses skeletal parts as redirection units and improves the flexibility of processing different source-target skeletal structure pairs. This method and technology have broad commercial value and application prospects, can promote the development of face-related fields, and provide users with better experience and services.

[0099] See also Figure 3 The second embodiment of the present invention provides a digital human motion redirection device based on skeletal feature-driven, which includes:

[0100] The data acquisition unit 201 is used to acquire the motion video information of the user collected by the monocular camera device, detect the human body area of ​​the motion video information of the user by using the BlazePose posture estimation network model, generate the three-dimensional joint point position sequence information of the source model, convert the three-dimensional joint point position sequence information of the source model into the BVH motion capture data format, and simplify it, retain the main joint motion data, and obtain the joint motion data in the form of BVH skeleton motion data;

[0101] The static feature extraction unit 202 is used to perform semantic feature deep feature extraction on the joint motion data in the form of the BVH bone motion data using a preset motion feature extraction network, process the static postures of the source model and the target model respectively, extract their respective bone static features, and obtain corresponding source model bone features and target model bone features;

[0102] A local-global feature extraction unit 203 is used to further extract the joint motion data in the form of BVH skeleton motion data based on the source model skeleton features and the target model skeleton features using a preset motion feature extraction network, and extract the local and global features of the motion in parallel, wherein the local branch extracts the local motion features through the intra-part joint attention mechanism, and the global branch extracts the global motion features through the whole body joint convolution;

[0103] A motion sequence generating unit 204 is used to take the source model skeleton features and the target model skeleton features, as well as the local and global features of the motion as input, and use a motion decoder to perform decoding processing to obtain a corresponding motion sequence, wherein the motion decoder includes a source model motion decoder and a target model motion decoder;

[0104] The optimization unit 205 is used to optimize the motion sequence according to a preset final loss function to ensure the integrity and consistency of information in the motion data conversion process between the source model and the target model, and generate motion redirection data, wherein the final loss function includes a reconstruction loss function, a cycle consistency loss function, an adversarial loss function and a forward kinematics loss.

[0105] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A digital human motion redirection method based on skeletal feature-driven, characterized in that: include: Obtaining the user's motion video information collected by a monocular camera device, detecting the human body area of ​​the user's motion video information using a BlazePose posture estimation network model, generating three-dimensional joint point position sequence information of a source model, converting the three-dimensional joint point position sequence information of the source model into a BVH motion capture data format, and performing simplified processing to retain the main joint motion data, thereby obtaining joint motion data in the form of BVH skeletal motion data; Use a preset motion feature extraction network to perform semantic feature deep feature extraction on the joint motion data in the form of the BVH bone motion data, process the static postures of the source model and the target model respectively, extract their respective bone static features, and obtain corresponding source model bone features and target model bone features; Based on the source model skeleton features and the target model skeleton features, a preset motion feature extraction network is used to further extract the joint motion data in the form of BVH skeleton motion data, and local and global features of the motion are extracted in parallel, wherein the local branch extracts local motion features through the intra-part joint attention mechanism, and the global branch extracts global motion features through full-body joint convolution; The source model skeleton features and the target model skeleton features, as well as the local and global features of the motion are used as input, and a motion decoder is used for decoding processing to obtain a corresponding motion sequence, wherein the motion decoder includes a source model motion decoder and a target model motion decoder; The motion sequence is optimized according to a preset final loss function to ensure the integrity and consistency of information in the motion data conversion process between the source model and the target model, and generate motion redirection data, wherein the final loss function includes a reconstruction loss function, a cycle consistency loss function, an adversarial loss function and a forward kinematics loss.

2. The method for digital human motion redirection based on skeletal feature-driven according to claim 1, characterized in that: The user's motion video information collected by the monocular camera device is obtained, and the human body area of ​​the user's motion video information is detected using the BlazePose posture estimation network model to generate the three-dimensional joint point position sequence information of the source model, and the three-dimensional joint point position sequence information of the source model is converted into the BVH motion capture data format, and simplified, and the main joint motion data is retained to obtain the joint motion data in the form of BVH skeletal motion data, specifically: Obtain the user's motion video information I collected by the monocular camera device; The BlazePose posture estimation network model is used to detect the key points of the human body in the user's motion video information I, and generate a video joint point set V of the source model. o , the formula is: V o = BlazePose(I), where represents the set of human joint points in the i-th video frame, represents the set of human joint points in the Fth video frame, where F is the number of frames in the motion video, and J is the number of bones in the model. represents the Jth human joint point in the i-th video frame; The video joint point set V of the source model o and the joint point set of the digital human target model Perform unified simplified matching processing to simplify the matching process and calculation complexity. The simplified matching formula of the joint point is: V′ t =finrm(V o ), Among them, finrm represents the finger removal operation of the joint point, V t ′ is the 3D joint position sequence information of the source model, V t The three-dimensional joint position sequence information of the digital human target model; The three-dimensional joint point position sequence information V of the source model t ′ and the three-dimensional joint position sequence information V of the digital human target model t Convert the three-dimensional coordinates into BVH bone motion data to obtain joint motion data in the form of BVH bone motion data. The conversion formula is: o ,M o }=convert(V′ t ),{S t ,M t }=convert(V t ), where S o is the target bone joint offset of the source model, S t is the target bone joint offset of the digital human. represents the motion data of the source model, represents the motion data of the i-th source model, The unit quaternion representing the local joint rotation for each joint, Represents the global motion speed of the root joint in the x-axis, y-axis, and z-axis directions, M t Represents the motion data of the digital human target model.

3. The method for digital human motion redirection based on skeletal feature-driven according to claim 2, characterized in that: The static postures of the source model and the target model are processed separately, and their respective skeletal static features are extracted to obtain the corresponding source model skeletal features and target model skeletal features, specifically: A preset motion feature extraction network is used to perform semantic feature deep feature extraction on the joint motion data in the form of the BVH skeleton motion data, wherein the motion feature extraction network is composed of a skeleton encoder Enc s and action encoder Enc m Composition, skeleton encoder Enc s Include source model skeleton encoder and target model skeleton encoder Action Encoder Enc m Includes source model action encoder and target model action encoder The source model and the target model use the source model skeleton encoder respectively and target model skeleton encoder Encode to obtain the corresponding source model bone features and target model bone features The formula is: Among them, the source model skeleton encoder and target model skeleton encoder Both use a two-layer perceptron MLP, whose output is a 1×P×d1 feature, where P=6 represents the six main body parts and d1=32 represents the dimension of the feature.

4. The method for digital human motion redirection based on skeletal feature-driven according to claim 3, characterized in that: Based on the source model bone features and the target model bone features, a preset motion feature extraction network is used to further extract the joint motion data in the form of BVH bone motion data, and extract the local and global features of the motion in parallel, specifically: Source Model Action Encoder and target model action encoder Both use two branches to extract local and global features of motion in parallel; The motion input at a given time frame i is the motion data of the i-th source model The motion data of the i-th source model is input through the joint embedding layer Mapped to the hidden embedding space, the formula is: where W0∈R 4×l ,W1∈R l×d ,W2∈R l×d ,b0∈R l ,b1∈R l ,b2∈R d are all network mapping parameters, l is the number of hidden units used in nonlinear mapping, and d is the dimension of the joint embedding space; Use joint-level position encoding to add the position of each joint and use the joint index in the skeleton as the position. The formula is: Where j represents the index of the joint in the skeleton, k∈[1,2,3,...,d] is the dimension index of the position code, and mod is the modulus operation; After the position encoding of each joint is completed, the joint embedding and the position encoding are combined in an additive manner to obtain the enhanced joint motion representation, as follows: in, represents the enhanced joint motion representation of the i-th frame; Integrate the joint features and aggregate them into the body part feature encoding to obtain the body part labeling For each video frame, body part labels and joints are embedded into features The combination of is taken as input and the query vector is output through the self-attention model With key-value pairs and The formula is: Among them, ATT is the self-attention function; The body parts are separated by a mask matrix U, which is symmetric, where U k,j = 1 means that joints k and j are in the same body part, and the final body joint feature is expressed as: in, represents the features of the part mark of the i-th frame, represents the joint features of the i-th frame, is the video joint point of the i-th source model, and N is the number of body parts.

5. The method for digital human motion redirection based on skeletal feature-driven according to claim 4, characterized in that: Also includes: According to the motion data M of the source model o , perform masked two-dimensional convolution processing on bones and time joint by joint to obtain the spatiotemporal features of each joint. The formula is: Among them, d j represents the jth joint feature, N o (j) represents the set of adjacent joint points with joint point j as the vertex, T j represents the adjacency mask matrix of the j-th joint, Indicates that the kth joint is adjacent to the jth joint, W c is the convolution kernel parameter matrix, b c ∈R d is the bias vector, * represents a two-dimensional convolution operation; Get all joint features All joint features Perform global two-dimensional convolution feature enhancement operation on all joint features, the formula is: Conc(D o )=Relu(D o *W conv +b conv ), where Conc is a two-dimensional convolution function, W conv is the convolution kernel, b conv is the bias vector; The local features are combined with the global features in an additive manner, and the final spatiotemporal motion features are obtained through convolution operations. The formula is:

6. The method for digital human motion redirection based on skeletal feature-driven according to claim 5, characterized in that: The source model skeleton features and the target model skeleton features, as well as the local and global features of the motion are used as input, and a motion decoder is used for decoding processing to obtain a corresponding motion sequence, specifically: The source model motion decoder and target model motion decoder Both use double-layer temporal deconvolution Deconv; Using source model motion decoder and target model motion decoder Decode the source model skeleton features and the target model skeleton features, as well as the local and global features of the motion, to obtain the corresponding motion sequence, the formula is: Among them, Upsampling represents the upsampling operation, V dec and b dec are the weight matrix and bias vector of the trainable convolution kernel respectively. Deconv1 and Deconv2 are two double-layer temporal deconvolution layers respectively.

7. The method for digital human motion redirection based on skeletal feature-driven according to claim 6, characterized in that: The motion sequence is optimized according to a preset final loss function to ensure the integrity and consistency of information during the motion data conversion process between the source model and the target model, and motion redirection data is generated, specifically: Using the source model bone features and the final space-time motion characteristics Generate reconstructed motion data Construct a reconstruction loss function to represent the error between the reconstructed motion and the source motion. The formula of the loss term is: Among them, ‖.‖ represents the second normal form symbol, is the error between the reconstructed motion and the source motion, To reconstruct motion data; Construct a cycle consistency loss function, whose loss term formula is: in, is the loss term of the cycle consistency loss function, Redirects the motion for the source model loop.

8. The method for digital human motion redirection based on skeletal feature-driven according to claim 7, characterized in that: Also includes: Construct an adversarial loss function to ensure that the redirected target motion is reasonable. The adversarial loss term formula is: in, is the true motion of the target model, and C is the discriminator consisting of two convolutional layers; Construct a forward kinematics loss function, the formula is: Get the final loss function Among them, λ1=2.5, λ2=100.

9. A digital human motion redirection device based on skeletal feature-driven, characterized in that: include A data acquisition unit is used to acquire the motion video information of the user collected by a monocular camera device, detect the human body area of ​​the motion video information of the user using a BlazePose posture estimation network model, generate three-dimensional joint point position sequence information of a source model, convert the three-dimensional joint point position sequence information of the source model into a BVH motion capture data format, perform simplified processing, retain the main joint motion data, and obtain the joint motion data in the form of BVH skeletal motion data; A static feature extraction unit is used to perform semantic feature deep feature extraction on the joint motion data in the form of the BVH bone motion data using a preset motion feature extraction network, process the static postures of the source model and the target model respectively, extract their respective bone static features, and obtain corresponding source model bone features and target model bone features; A local-global feature extraction unit is used to further extract joint motion data in the form of BVH skeleton motion data based on the source model skeleton features and the target model skeleton features using a preset motion feature extraction network, and extract local and global features of the motion in parallel, wherein the local branch extracts local motion features through an intra-part joint attention mechanism, and the global branch extracts global motion features through full-body joint convolution; A motion sequence generating unit, used for taking the source model skeleton features and the target model skeleton features, as well as the local and global features of the motion as input, and performing decoding processing by using a motion decoder to obtain a corresponding motion sequence, wherein the motion decoder includes a source model motion decoder and a target model motion decoder; An optimization unit is used to optimize the motion sequence according to a preset final loss function to ensure the integrity and consistency of information in the motion data conversion process between the source model and the target model, and generate motion redirection data, wherein the final loss function includes a reconstruction loss function, a cycle consistency loss function, an adversarial loss function and a forward kinematics loss.

Images