Animation style migration method and electronic equipment

Abstract

The invention discloses an animation style migration method and electronic equipment, and the method comprises the steps: extracting the content features of a first animation segment through a content encoder, and obtaining a content feature vector; extracting a style feature of the second animation segment by using a style encoder to obtain a style feature vector; a decoder is used for decoding the content feature vector and the style feature vector to obtain a third animation segment, and the third animation segment is an animation segment obtained by migrating the style of the second animation segment to the first animation segment; the content encoder and the style encoder are time convolutional neural networks.

Description

Technical Field

[0001] The embodiments of the present application relate to the fields of image processing and computer vision, and relate to but are not limited to an animation style transfer method and electronic device. Background Art

[0002] Animation style transfer is a key research area in computer vision and graphics. It aims to transfer the visual style of one animation clip to the content of another, thereby generating new animation content with a new style. This technology is widely used in film and television production, game development, and digital art creation, relying on the effective extraction and fusion of content and style features.

[0003] Most existing motion style transfer methods rely on paired data: pairs of animation clips with the same motion content but different styles. However, obtaining paired data requires extensive manual labeling and recording, which is costly and inefficient. They typically only transfer styles seen in the training data and struggle to handle unseen styles. Using fixed feature vectors to represent style makes it difficult to capture complex style characteristics. Summary of the Invention

[0004] In view of this, embodiments of the present application provide an animation style transfer method and an electronic device.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an animation style transfer method, comprising: extracting content features of a first animation clip using a content encoder to obtain a content feature vector; extracting style features of a second animation clip using a style encoder to obtain a style feature vector; decoding the content feature vector and the style feature vector using a decoder to obtain a third animation clip, wherein the third animation clip is an animation clip in which the style of the second animation clip is transferred to the first animation clip; the content encoder and the style encoder are temporal convolutional neural networks.

[0007] In a second aspect, an embodiment of the present application provides an animation style transfer apparatus, comprising:

[0008] A content extraction module, configured to extract content features of the first animation clip using a content encoder to obtain a content feature vector;

[0009] A style extraction module, configured to extract style features of the second animation clip using a style encoder to obtain a style feature vector;

[0010] A decoding module is used to decode the content feature vector and the style feature vector using a decoder to obtain a third animation clip, wherein the third animation clip is an animation clip in which the style of the second animation clip is transferred to the first animation clip; the content encoder and the style encoder are temporal convolutional neural networks.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0012] A content encoder, configured to extract content features of the first animation clip to obtain a content feature vector;

[0013] A style encoder, configured to extract style features of the second animation clip and obtain a style feature vector;

[0014] The decoder is used to decode the content feature vector and the style feature vector to obtain a third animation clip, wherein the third animation clip is an animation clip in which the style of the second animation clip is transferred to the first animation clip; the content encoder and the style encoder are temporal convolutional neural networks.

[0015] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions, which, when executed by a processor, implements the steps of extracting content features of a first animation clip using a content encoder to obtain a content feature vector; extracting style features of a second animation clip using a style encoder to obtain a style feature vector; and decoding the content feature vector and the style feature vector using a decoder to obtain a third animation clip, wherein the third animation clip is an animation clip in which the style of the second animation clip is transferred to the first animation clip; and the content encoder and the style encoder are temporal convolutional neural networks.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program or instructions. When the computer program or instructions are executed by a processor, the content features of a first animation clip are extracted using a content encoder to obtain a content feature vector; the style features of a second animation clip are extracted using a style encoder to obtain a style feature vector; and the content feature vector and the style feature vector are decoded using a decoder to obtain a third animation clip, wherein the third animation clip is an animation clip in which the style of the second animation clip is transferred to the first animation clip; the content encoder and the style encoder are temporal convolutional neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of an implementation flow of an animation style transfer method provided in an embodiment of the present application;

[0018] Figure 2A A schematic diagram of an implementation flow for obtaining a content feature vector provided in an embodiment of the present application;

[0019] Figure 2B A schematic diagram of an implementation flow for obtaining a style feature vector provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of an implementation flow for decoding a content feature vector and a style feature vector provided in an embodiment of the present application;

[0021] Figure 4A A schematic diagram of an electronic device performing animation style transfer provided by an embodiment of the present application;

[0022] Figure 4B A schematic diagram of the implementation flow of an animation style transfer method provided in an embodiment of the present application.

[0023] Figure 5 A schematic diagram of the structure of an animation style transfer device provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] To make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the specific technical solutions of the embodiments of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0026] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0029] The present application embodiment provides an animation style transfer method, such as Figure 1 As shown, the method includes:

[0030] Step S110: extracting content features of the first animation clip using a content encoder to obtain a content feature vector; the content encoder is a temporal convolutional neural network;

[0031] The content encoder, a model based on a temporal convolutional neural network (TCN), extracts features describing the motion content from the first input animation clip. These features include action type (such as walking, running, jumping), keyframes, joint positions and rotations, body posture, and motion trajectory. By extracting these features, the underlying semantics and motion structure of the original animation are preserved during the subsequent style transfer process.

[0032] A temporal convolutional neural network (TCN) is a deep learning architecture specifically designed for processing time series data. It captures local and global dependencies in time series by stacking multiple convolutional layers. Compared to traditional recurrent neural networks (RNNs), TCNs offer greater parallel computing capabilities and improved stability, making them particularly well-suited for modeling time-sensitive data such as joint rotation sequences in human animation. The content encoder uses a TCN structure to better extract the temporal features of the first animation segment.

[0033] The content feature vector is a high-dimensional vector that contains the key content information of the first animation clip. The dimensionality of this vector is determined by the encoder's structural design and is composed of the features output by multiple convolutional layers. The content feature vector accurately describes the essence of the first animation's movements, providing foundational content support for subsequent style transfer.

[0034] In its implementation, the content encoder first receives as input a first animation clip, which can be any form of human motion data, such as a sequence of joint rotations of a 3D model. The content encoder then extracts higher-level content features through layer-by-layer convolution operations, ultimately generating a content feature vector that represents the core content of the entire animation clip.

[0035] Step S120: extracting style features of the second animation clip using a style encoder to obtain a style feature vector; the style encoder is a temporal convolutional neural network;

[0036] Here, the style encoder is also based on a temporal convolutional neural network, but its goal is to extract features describing the motion style of the second animation clip. Stylistic features primarily include rhythm, speed, force, dynamic range, emotional expression, and pose preferences. These features determine the animation's presentation, rather than its underlying motion content.

[0037] The style feature vector is also a high-dimensional vector that captures the stylistic properties of the second animation clip. Unlike the content feature vector, the style feature vector does not include specific content information such as action type or keyframes, but instead focuses on the abstract features that express the style. This separation mechanism makes the style transfer process more flexible, allowing a single style to be applied to a variety of different action contents.

[0038] In implementation, the style encoder receives the second animation clip as input and extracts its stylistic features through a temporal convolutional neural network. For example, if the second animation clip depicts a comedian running with exaggerated movements, the style feature vector might highlight features such as rapid gait, large body movements, and strong emotional expressions.

[0039] Here, the second animation clip may be an animation clip different from the first animation clip.

[0040] During implementation, the execution order of step S110 and step S120 is not limited, and they may be executed simultaneously.

[0041] Step S130: Decode the content feature vector and the style feature vector using a decoder to obtain a third animation clip, wherein the third animation clip is an animation clip in which the style of the second animation clip is transferred to the first animation clip;

[0042] Here, the decoder's role is to fuse the content feature vector and the style feature vector to generate a new animation clip, the third animation clip. This animation clip retains the content features of the first animation clip while incorporating the style features of the second animation clip, thus achieving style transfer.

[0043] The decoder can use Adaptive Instance Normalization (AdaIN) technology. AdaIN effectively injects style features by normalizing the mean and variance of content features and matching them with the statistical properties of style features. Specifically, the decoder first performs instance normalization on the content features to have a mean of 0 and a variance of 1. It then applies the mean and variance of the style features to the normalized content features to generate a new animation clip with the target style.

[0044] In real time, the decoder receives content and style feature vectors as input and fuses these features using AdaIN technology. The decoder then adjusts the temporal resolution of the generated animation clips using an interpolation algorithm to ensure they are temporally coherent and natural, without abrupt jumps or unnatural transitions.

[0045] In this application, by introducing an unpaired data training mechanism and a content-style separation structure based on a temporal convolutional neural network, we not only reduce data acquisition costs and improve model generalization, but also achieve high-quality style transfer while preserving the action content. This approach is suitable for practical applications requiring extensive style changes and lacking paired data, such as game character animation design and virtual human motion simulation.

[0046] In some embodiments, the content encoder includes a temporal convolution layer, a residual block, and an average pooling layer; the above step S110 "using the content encoder to extract the content features of the first animation clip to obtain a content feature vector" is as follows: Figure 2A As shown, this can be achieved by following the steps below:

[0047] Step S201: Analyze the first animation clip to obtain joint rotation sequence data corresponding to the first animation clip;

[0048] Here, joint rotation sequence data refers to a data set that quantifies the rotational state of each joint of a human model in the time dimension within an animation clip. Each joint's rotation can be described using quaternions, which consist of four elements that accurately represent the angle and direction of the joint's rotation in three-dimensional space. By extracting joint rotation information frame by frame, a time-varying joint rotation sequence can be constructed as a mathematical representation of the motion content. This data can reflect the essential characteristics of movements, such as basic motion forms like walking, running, and jumping.

[0049] During implementation, computer vision or motion capture can be used to analyze the first animation clip to obtain key rotation sequence data. Motion capture technology uses sensors placed at human joints to capture the joint's motion trajectory and rotation angle, thereby obtaining the joint rotation sequence.

[0050] In this way, complex motion processes can be converted into computable numerical forms, making it easier for subsequent deep learning models to process and analyze them.

[0051] Step S202: performing convolution processing on the joint rotation sequence data using at least one temporal convolution layer to obtain at least one temporal feature graph;

[0052] The temporal convolution layer is a neural network layer designed for time series data. It extracts local temporal dependencies by sliding the convolution kernel along the time axis, thereby capturing dynamic patterns in joint rotation sequences. Temporal feature maps are the intermediate outputs of the temporal convolution operation, reflecting the kinematic characteristics of joints at different time points, such as rotation trends, velocity changes, and periodic fluctuations. Each temporal feature map can be viewed as an abstract representation of the original joint rotation sequence, helping the model identify key features of the motion content.

[0053] In the implementation, multiple one-dimensional temporal convolutional layers are used to process the input 3D joint rotation sequence to capture the local temporal features of the input data. The convolutional layer uses unit quaternions as input and outputs a set of temporal feature maps.

[0054] In this way, the temporal features of joint motion can be effectively extracted, providing a basis for subsequent content feature extraction; thereby improving the model's ability to understand motion content and, in turn, improving the accuracy of style transfer.

[0055] Step S203: setting the residual block after each of the temporal convolutional layers, and using the residual block to normalize the temporal feature map to obtain a content feature map that removes style information and retains content information;

[0056] Here, the residual block is a deep learning structure that alleviates the gradient vanishing problem and enhances the learning ability of the model by introducing skip connections.

[0057] During implementation, the extracted temporal features are further processed using residual blocks to map content motion to a temporal content latent code. Residual blocks are used to normalize the temporal feature map, adding the input feature map to the feature map after convolution and nonlinear transformation to maintain the consistency of information flow. Normalization involves statistically adjusting the distribution of the feature map to a mean close to zero and a variance close to one, thereby eliminating individual style-related differences and retaining only the common characteristics of the motion content. The final output content feature map is an abstract representation that removes the influence of style and accurately reflects the core content of the action.

[0058] In this way, motion content and style information can be separated to avoid style information interfering with content modeling; thereby improving the generalization ability and stability of content features, and further improving the effect of content retention during style transfer.

[0059] Step S204: Use the average pooling layer to map the content feature map into the content feature vector.

[0060] Here, the average pooling layer is a dimensionality reduction operation that compresses high-dimensional feature maps into low-dimensional feature vectors by averaging the spatial dimensions of the feature map. In other words, average pooling converts the content feature map into a fixed-length vector. It can effectively preserve the local information in the content feature map.

[0061] The content feature vector is a fixed-length vector obtained by integrating the content features of multiple time steps. It is used to represent the core content of the entire animation clip. This vector summarizes the main motion content of the animation clip, such as action type and key poses, without including specific temporal details or stylistic characteristics.

[0062] In the implementation, the content encoder encodes the input 3D joint rotation sequence into a content feature vector Zc after processing through multiple temporal convolutional layers and residual blocks.

[0063] In this way, high-dimensional temporal features can be converted into compact and representative content feature vectors; thus, they can be more conveniently used in subsequent style transfer tasks, thereby improving the efficiency and robustness of the overall system.

[0064] In some embodiments, the content feature vector includes at least one of the following information corresponding to the first animation clip: action type, motion trajectory, and body posture.

[0065] Here, action type refers to the specific type of movement performed in an animation clip, such as walking, running, jumping, waving, and so on. By identifying action types, we can distinguish different basic motion behaviors, providing a basic classification basis for subsequent style transfer. This information helps ensure that the transferred animation retains the core semantics of the original action, preventing the essential attributes of the action from being altered during the style transfer process.

[0066] A motion trajectory is the path of a joint or body part's position in three-dimensional space over time. By recording the movement of key joints (such as the hands, feet, and torso) between consecutive frames, the system can capture the spatial dynamics of the movement, such as stride length and arm swing. This information helps the model understand the spatial distribution of movement, making the transferred movements more physically plausible and coherent.

[0067] Body pose refers to the position of various parts of the human body relative to a reference point (such as the center of the spine) at a given moment. It is determined by joint angles, bone length ratios, and relative positions. Body pose information helps describe the morphological characteristics of an animated character at a specific moment, such as standing, bending, or stretching. This information is crucial for maintaining the realism and naturalness of movement, especially when complex pose transitions are involved.

[0068] Combining these three aspects of information, the content feature vector can more comprehensively characterize the motion content of an animation clip, providing a solid data foundation for subsequent separation and combination with style features. This fine-grained feature representation not only improves the model's understanding of content but also enhances the controllability and accuracy of the style transfer process.

[0069] In this embodiment, joint rotation sequence data is obtained by analyzing the first animation clip, and a temporal convolution layer is used to extract a temporal feature map. This is then normalized using a residual block to remove style information, and finally, a content feature vector is generated using an average pooling layer. This allows for efficient modeling of motion content and style decoupling, significantly improving the quality of motion style transfer, reducing reliance on paired data, and achieving high-quality style transfer using unsupervised training.

[0070] In some embodiments, the residual block includes a convolutional layer, a normalization layer, an activation layer, and a connection layer. In step S203 above, “normalizing the temporal feature map using the residual block” can be implemented by the following steps:

[0071] Step 2031: Perform convolution processing on the temporal feature map using the convolution layer to obtain a first feature map;

[0072] Here, convolutional layers are used to extract spatial features of the input data.

[0073] In some embodiments, the convolutional layer uses a sliding window approach to perform local perception and parameter sharing on the temporal feature map, thereby extracting a hierarchical feature representation. This operation can effectively reduce the number of model parameters while preserving key temporal and spatial information.

[0074] By using the convolutional layer, local features related to motion content in the temporal feature map can be extracted without increasing too much computational burden, thereby improving the efficiency and accuracy of subsequent processing.

[0075] Step 2032: Use the normalization layer to normalize the first feature map, remove the style information in the first feature map, retain the content information, and obtain a second feature map;

[0076] Here, the normalization layer can be used to standardize the feature map after convolution to eliminate the scale differences between different samples.

[0077] During implementation, the normalization layer calculates the mean and variance of each channel and normalizes the feature map to a standard distribution with a mean of 0 and a variance of 1, thereby removing style-related information from the feature map and retaining the core features of the motion content.

[0078] During the encoding process, the first feature map can be processed using an instance normalization layer to remove the style information from the input data and retain only the content information. The instance normalization layer normalizes the data of each time step of each feature map independently and learns a scaling factor and an offset to adjust the mean and variance of the feature map.

[0079] Through normalization, content and style information can be effectively separated, allowing the model to focus more on learning the essential characteristics of motion content without being disturbed by style, thereby improving the ability to retain content during the migration process.

[0080] Step 2033: Using the activation layer, perform nonlinear activation function processing on the second feature map to obtain a third feature map;

[0081] Here, the activation layer introduces nonlinear transformation, giving the neural network stronger expressive power.

[0082] During implementation, the activation layer can use common nonlinear functions (such as ReLU) to process the normalized feature map to enhance its nonlinear expression ability so that it can better fit complex motion patterns.

[0083] The introduction of nonlinear activation functions enables the model to capture more complex and detailed motion features, improving the generalization ability and expressiveness of the overall model.

[0084] Step 2034: Use the connection layer to connect the temporal feature map and the third feature map to obtain the content feature map.

[0085] Here, the role of the connection layer is to splice or fuse the original time feature map with the third feature map after convolution, normalization and activation to generate the final content feature map.

[0086] During the implementation process, the integrity of the content features is ensured by retaining all the information of the original input, and the processed new features are combined to form a richer feature representation.

[0087] Fusing the original input with the processed features through the connection layer helps to introduce new feature dimensions while retaining the original motion content, thereby improving the richness and diversity of feature expression.

[0088] In this embodiment, a residual block consisting of a convolutional layer, a normalization layer, an activation layer, and a connection layer is provided to perform multi-stage processing on the temporal feature map. This allows for the gradual extraction and optimization of motion content features, effectively separating content and style information, and thus achieving high-quality motion style transfer.

[0089] In some embodiments, the content encoder includes a temporal convolution layer and a global pooling layer; the above step S120 "using the style encoder to extract the style features of the second animation clip to obtain a style feature vector" is as follows Figure 2B As shown, this can be achieved by following the steps below:

[0090] Step S211: Analyze the second animation clip to obtain joint position sequence data corresponding to the second animation clip;

[0091] Joint position sequence data refers to a sequence of information extracted from the target animation clip, describing the position of each joint in three-dimensional space over time. This data, indexed by timestamps, records the coordinates of each joint at each moment, representing changes in body posture. Joint position sequence data is the foundation for building human motion models and an important input for extracting motion style features.

[0092] By extracting joint position sequence data, we can accurately capture the spatial variation patterns of human motion, providing high-quality input for subsequent style feature extraction and helping to achieve a more natural and coherent style transfer effect. Joint position sequence data can be obtained using motion capture devices or skeleton tracking algorithms.

[0093] During implementation, the system extracts the 3D coordinates of all joints from each frame of the animation clip and arranges them in chronological order to form a continuous sequence. This process ensures the integrity of the motion content, allowing subsequent style transfer to be performed while maintaining the semantics of the original motion.

[0094] Step S212: performing convolution processing on the joint position sequence data using at least one temporal convolution layer to obtain at least one temporal feature graph;

[0095] Here, the temporal convolutional layer is a neural network layer specifically designed for processing time series data, capable of capturing local temporal dependencies in motion sequences. This layer extracts features from joint position information frame by frame through a one-dimensional convolution operation, generating a feature representation with temporal correlation.

[0096] A temporal feature map is an intermediate feature representation generated by applying a temporal convolution operation to joint position sequence data. Each temporal feature map corresponds to the activation of a specific type of motion feature, such as speed, direction, and rhythm. By stacking multiple temporal convolution layers, higher-level temporal features can be gradually extracted, enhancing the model's understanding of motion style.

[0097] Generating temporal feature maps through temporal convolution operations can effectively capture key timing patterns in motion sequences, laying the foundation for subsequent style feature extraction, thereby improving the accuracy and expressiveness of style transfer.

[0098] Step S213: Use the global pooling layer to map the last temporal feature map output by the last temporal convolutional layer into the style feature vector.

[0099] Here, the global pooling layer is an operation layer used to compress high-dimensional feature maps into low-dimensional feature vectors. Its function is to statistically aggregate the spatial dimensions of the feature map (such as height and width) using the maximum or average value. The role of the global pooling layer is to eliminate spatial redundancy and extract information that best represents the overall features. For example, when processing the feature maps output by multiple temporal convolutional layers, the global pooling layer can map the output of the last temporal convolutional layer to a fixed-length vector as the final style feature representation.

[0100] The style feature vector is a fixed-length vector that characterizes the overall motion style of the target animation clip. It is generated by a global pooling layer that performs a dimensionality reduction operation on the last temporal feature map, removing specific positional information while retaining the key attributes of the motion style. This feature vector serves as input to the subsequent style transfer model, guiding the generation of new animation clips with the target style.

[0101] By generating style feature vectors through global pooling operations, complex motion sequences can be converted into concise numerical representations, which are convenient for the style transfer model to process and improve the efficiency and consistency of style transfer.

[0102] In some embodiments, the style feature vector includes at least one of the following information corresponding to the second animation clip: motion rhythm, speed, strength, or emotion.

[0103] Here, motion rhythm refers to the overall sense of rhythm, as reflected by the time intervals and speed changes in the joint rotation sequence within an animation clip. For example, in running, motion rhythm can be expressed as the speed and evenness of the steps; in dance, it can be expressed as the pauses and continuity between steps. By capturing these rhythmic features, the system can better understand the dynamic characteristics of the animation and transfer them to the target animation, ensuring that it retains the original content while reflecting a similar rhythmic sense.

[0104] Speed ​​is the rate at which joint rotation angles change over time, measured as the rate of change of joint position per unit time. Speed ​​features can help distinguish different types of movement, such as fast running and slow walking. During style transfer, preserving or adjusting speed features helps control the dynamic performance of the target animation, enhancing the realism and expressiveness of the movements.

[0105] Dynamics refers to the force expressed during movement, determined by the speed and acceleration of joint rotation. High-dynamics movements convey a strong sense of explosiveness and impact, while low-dynamics movements appear softer and more restrained. By extracting dynamics features, the system can impart different levels of dynamics to the target animation during style transfer, achieving more expressive animation effects.

[0106] Emotions are derived from a comprehensive analysis of factors such as movement rhythm, speed, amplitude, and posture. For example, joy might be expressed as a brisk pace and wide arm swings, while sadness might be expressed as slow movements and a lowered head. Extracting emotional features allows style transfer to go beyond physical imitation and further convey a character's psychological state and emotional tone.

[0107] By taking the above-mentioned motion rhythm, speed, strength or emotion as part of the style feature vector and using a richer and more multi-dimensional style feature representation method, the style essence of the animation clip can be captured more accurately.

[0108] In the embodiments of this application, by combining temporal convolutional layers with global pooling layers to extract motion style features, we can effectively separate motion content from style information, thereby achieving high-quality style transfer without the use of paired data. We can also effectively extract the temporal features of motion sequences and compress them into feature vectors of uniform dimensions, facilitating subsequent style transfer processing while reducing model complexity. This reduces reliance on labeled data, significantly lowering training costs and enabling style transfer in a wider range of application scenarios.

[0109] In the embodiment of the present application, the decoder has at least one residual block and an output layer including at least one linear layer, wherein each residual block includes a convolutional layer and an adaptive instance normalization layer provided after the convolutional layer; in the above step S130, "using the decoder to decode the content feature vector and the style feature vector to obtain a third animation clip", such as Figure 3 As shown, this can be achieved by following the steps below:

[0110] Step S310: using the convolutional layer in the residual block, decoding the content feature vector into a time vector in a preset dimensional feature space in the time dimension;

[0111] Here, the residual block includes multiple convolutional layers and introduces skip connections to retain input information. The convolutional layer is used to extract local features and achieve multi-scale feature extraction through stacking. The time dimension refers to the distribution characteristics of the data on the time axis, such as the pattern of changes in the state of each joint in a motion sequence over time. The content feature vector is a vector representing the content of the motion extracted from the original action and does not contain style information. The preset dimensional feature space is a designed fixed-dimensional space for easy subsequent processing. The time vector refers to the vector that represents the time evolution relationship in this feature space.

[0112] During the implementation process, by decoding the content feature vector in the time dimension, the abstract content information can be transformed into a representation with temporal continuity, thus providing a basis for subsequent style transfer.

[0113] Step S320: Using the adaptive instance normalization layer in the residual block, perform style injection on the time vector based on the style feature vector to obtain a target time vector after completing the style injection;

[0114] Here, the Adaptive Instance Normalization layer (AdaIN) is an improved instance normalization method that not only normalizes the feature map but also dynamically adjusts the mean and variance based on externally provided style features. The style feature vector is a feature extracted from the target style sample and is used to describe the motion performance of a specific style. Style injection refers to the application of style features to content features, so that the generated motion sequence has the visual effect of the target style. The target time vector is the time vector after style injection, which combines content and style information.

[0115] In the implementation process, by combining style features with content features, the target style can be effectively transferred to the original action while keeping the core content of the original action unchanged.

[0116] Step S330: Utilize the output layer to map the target time vector into a target motion sequence through linear transformation, so as to obtain the third animation clip based on the target motion sequence.

[0117] Here, the output layer includes a fully connected layer, which converts high-dimensional features into low-dimensional outputs. Linear transformations project the input vector into another vector space, achieved through matrix multiplication. The target motion sequence is the decoded representation of the action, typically a time series containing joint positions or rotation angles at each point in time. The third animation clip is the final generated animation, which contains the motion representation of the target style.

[0118] During the implementation process, the input data is mapped to the output space through linear transformation, that is, the target time vector is converted into specific motion parameters, which can generate animation clips that conform to the target style and achieve the purpose of style transfer.

[0119] In this embodiment, residual blocks are used in conjunction with convolutional layers and adaptive instance normalization layers to decode and fuse content and style features. This effectively separates content and style information, enabling high-quality style transfer without relying on paired data, and ultimately generating natural, smooth, and expressive animation clips.

[0120] In some embodiments, the decoder further includes an upsampling layer. Before the above step S330 of "using the output layer to map the target time vector into a target motion sequence through a linear transformation to obtain the third animation clip based on the target motion sequence", the method further includes the following steps:

[0121] Step S340: upsampling the target time vector using the upsampling layer to obtain a joint rotation feature vector corresponding to a preset resolution;

[0122] Here, an upsampling layer is a neural network module that increases the temporal resolution or spatial dimension of the input data, thereby generating a more accurate feature representation. This layer, implemented through deconvolution or interpolation, gradually expands the low-resolution temporal vector into a high-resolution joint rotation feature vector. This design helps preserve motion details and improves the quality of the resulting animation.

[0123] The target time vector refers to the intermediate representation before being passed to the output layer during the decoding process. It is usually a compressed form of time series features that contains key information about motion but lacks sufficient spatial resolution.

[0124] During implementation, the upsampling layer expands this temporal vector into a more refined joint rotation feature vector capable of describing specific joint movements. This significantly improves the decoder's output accuracy, resulting in more natural and smooth animations. Furthermore, this step ensures that the features used in the subsequent linear transformation have sufficient temporal resolution, enhancing the effectiveness of style transfer.

[0125] Correspondingly, the above step S330 of "using the output layer to map the target time vector into a target motion sequence through linear transformation, so as to obtain the third animation clip based on the target motion sequence" can be implemented by the following process:

[0126] The output layer is used to map the joint rotation feature vector into a target motion sequence through linear transformation, so as to obtain the third animation clip based on the target motion sequence.

[0127] In implementation, the joint rotation feature vectors are fed into the output layer and linearly transformed to generate an accurate target motion sequence. This step ensures that the generated animation clips are consistent in time and conform to the physical constraints of human motion in space. Ultimately, this motion sequence can be used to reconstruct the complete animation clip, achieving style transfer.

[0128] In an embodiment of the present application, an upsampling layer is introduced to upsample the target time vector to obtain a joint rotation feature vector, which is then mapped into a target motion sequence through an output layer, thereby improving the temporal resolution and motion detail expression of the generated animation, thereby achieving a high-quality motion style transfer effect.

[0129] In some embodiments, in the process of training the style transfer network including the content encoder, the style encoder and the decoder, at least an adversarial loss function is used, wherein,

[0130] The adversarial loss function includes a generator loss function and a discriminator loss function. Using the adversarial loss function during the training process can make the third animation clip output by the decoder close to the preset real animation clip.

[0131] Here, the adversarial loss function is an optimization objective used to enhance the model's generative capabilities, including both the generator loss function and the discriminator loss function. By constructing an adversarial process, this function enables the model to generate outputs that are more realistic and consistent with the real-world distribution. The adversarial loss function is applied to the training of the style transfer network to improve the quality and realism of the decoder output animation clips. The adversarial loss helps the network learn more generalizable style features, enabling it to transfer styles not seen during training.

[0132] The adversarial loss function L can be calculated using the following formula (1): adv Function value of :

[0133]

[0134] Among them, M represents the motion sequence dataset, n t represents the style motion sequence, m s represents the content motion sequence, and D represents the discriminator;

[0135] Indicates the expectation of all style inputs in the dataset M (expectation operator);

[0136] D t (n t ) means to set n t Input into the discriminator D, and the output corresponding to the target style t is obtained;

[0137] ||D t (n t )|| is the calculation of D t (n t ), that is, the L2 norm of the vector D t (n t )'s length;

[0138] The discriminator D should be able to correctly identify the true style input, D t (n t ) is closer to 1, indicating a more realistic result.

[0139] Represents the expectation of all content inputs and style input combinations in the dataset;

[0140] E c (m s ) means to enter the content into m s Input to content encoder E c Get the content code Z c ;

[0141] E s (n t ) means inputting style into n t Input to content encoder E s Get the content code Zs;

[0142] F(E c (m s )|E s (n t )) means to encode the content Z c And the style code Zs is input into the decoder F to obtain the generated style transfer results m~t.

[0143] D t (F(E c (m s )|E s (n t )) indicates that the generated style transfer result m~t is input into the multi-style discriminator D to obtain the output corresponding to the target style t.

[0144] ||D t (F(E c (m s )|E s (n t ))-1||2 , calculate the L2 norm, and by minimizing the norm, encourage the generator G to learn to generate more realistic style transfer results.

[0145] In some embodiments, the loss function also includes a content consistency loss function, which is used to characterize the distance between a first motion sequence and a second motion sequence, wherein the first motion sequence is determined based on the content feature vector, and the second motion sequence is determined based on the decoder decoding the content feature vector and the style feature vector.

[0146] Here, the transformation network acts as an “identity mapping”, i.e. the content motion of the input should be completely preserved in the output motion.

[0147] Randomly select two motion sequences (ms, ns) with the same style label from the dataset M.

[0148] Using Content Encoder E c and style encoder E s , encode the content input into content code Z c , encode the style input into the style code Z s . Use decoder F content code Z c and style code Z s Decoded into the output motion sequence m~t. Calculating the L1 distance between the output motion sequence m~t and the content input ms can obtain the loss value of the content consistency loss function.

[0149] In some embodiments, the loss function further includes a feature matching loss function for characterizing the difference between the third animation segment output by the decoder and a preset real segment.

[0150] Here, the goal of the feature matching loss is to make the fake samples generated by the generator as close as possible to the real samples in the feature space of the discriminator. The feature matching loss is used as a regularization term, that is, it is added to the total loss of the generator to help control the generator learning process, prevent overfitting, and promote the generator to generate higher quality samples. The loss value of the feature matching loss function can be obtained by calculating the distance between the samples generated by the generator and the real samples in the feature space of the discriminator.

[0151] The feature matching loss function L can be calculated using the following formula (2): reg The loss value is:

[0152]

[0153] Among them, M represents the motion sequence dataset, n t represents the style motion sequence, m s Represents content motion sequence;

[0154] Represents the expectation of all content inputs and style input combinations in the dataset;

[0155] D f It is a subnetwork of the discriminator D, used to extract high-level features. Indicates sample extraction characteristics; middle is a real sample, which is the real sample feature extracted by the discriminator D;

[0156] |M t | is the number of true samples with a specific style t;

[0157] calculate The purpose is to normalize the extracted features so that the feature distributions of different styles are comparable;

[0158] It is used to calculate the L1 norm between the generated sample features and the real sample features, and is used to measure the sum of the absolute differences between the two feature vectors.

[0159] In the embodiment of the present application, by introducing an adversarial loss function during the training process, the authenticity and diversity of the generated animation clips can be effectively improved, thereby improving the style transfer effect, reducing the dependence on paired data and improving the generalization ability of the model.

[0160] In the field of computer animation, migrating one motion style to another while preserving the original motion is a challenging problem. For example, transferring a cartoon character's exaggerated running style to an ordinary person makes their running more expressive and humorous; transferring a child's lively and brisk walk to an adult to make their walk appear relaxed; and transferring a comedian's antics to an ordinary person to make their movements more interesting and humorous.

[0161] Among existing solutions, handcrafted feature-based methods utilize manually designed features such as physical parameters or spectral features to represent motion style and perform style transfer. These methods struggle to capture complex style characteristics and require customization for different styles. Data-driven methods, on the other hand, utilize machine learning models to learn motion style from data and perform style transfer. However, these methods require large amounts of paired data and struggle to handle unseen styles.

[0162] Existing solutions have the following disadvantages: high data acquisition costs, as the acquisition of paired data requires a large amount of manual labeling and recording work, which is costly and inefficient; difficulty in handling unseen styles, as existing methods can only transfer styles seen in the training data; in addition, style feature representation is limited, as existing methods use fixed feature vectors to represent style, which makes it difficult to capture complex style features.

[0163] To address the above issues, this application provides a motion style transfer method based on unpaired data. This method does not require paired data and only requires style labeling of animation clips. The key points of this method include: using an unpaired data training method to enable the model to learn style features from unpaired data; using a deep learning model to extract the content features and style features of the animation clips respectively. The content features are used to describe the motion content of the animation clips, while the style features are used to describe the motion style of the animation clips; and using a style transfer model to combine the content features and style features of the target animation clip to generate an animation clip with the target style.

[0164] Figure 4A A schematic diagram of an electronic device performing animation style transfer provided in an embodiment of the present application is shown as follows: Figure 4A As shown, the electronic device for performing animation style transfer includes two encoders (content encoder E C and style encoder E s ) and decoder F, where the style encoder E C For extracting style features from target actions, content encoder E s It is used to extract content features from the original action. The decoder F receives the content features and style features for decoding and generates an animation clip with the target style.

[0165] During the implementation process, if Figure 4B As shown, animation style transfer can be achieved through the following steps:

[0166] Step S410: extracting a temporal feature graph of the motion;

[0167] Here, complex human motion (including any form of physical activity such as walking, running, jumping, waving, etc.) can be represented as a sequence of joint rotations that changes over time. Each time point contains the rotation angles of J joints, and each angle is represented by a unit quaternion with 4 elements. By processing the time series, the following main temporal feature graphs are extracted:

[0168] Joint relative position feature map: contains information about the relative positions of different joints, such as the relative position of the arm and torso;

[0169] Posture feature map: represents the posture of the body at a specific point in time, such as walking, running, jumping, standing, bending, etc.

[0170] Motion direction feature map: represents the change in the joint's motion direction, which is the angular change relative to a reference direction. It describes the joint's motion direction in three-dimensional space, such as forward and backward, left and right, up and down, and rotational directions.

[0171] Movement amplitude feature map: contains information about the movement amplitude of different joints, such as the amplitude of arm swing when walking happily or sadly;

[0172] Motion speed feature map: Indicates the speed of a moving object at a specific point in time, providing information about the speed of the movement, such as the change in speed during walking, running, jumping, etc.

[0173] Motion acceleration characteristic graph: represents the acceleration of a moving object at a specific time point, such as changes in acceleration, deceleration, or maintaining a constant speed;

[0174] Movement rhythm feature map: contains information about the movement rhythm of different joints, such as brisk or heavy steps;

[0175] Periodic motion feature maps: Capturing periodic changes in motion, such as repetitive motion patterns in running or jumping;

[0176] Dynamic balance characteristic diagram: represents the body's balance state during exercise, capturing the changes and adjustments of the center of gravity.

[0177] Step S420: Content encoder E c Extract motion content features, style encoder E s Extract motion style features;

[0178] 1) Content Encoder E s Extract motion content features:

[0179] Here, motion content features are a further abstraction and generalization of the temporal feature graph. Motion content features (such as joint positions and rotation angles, key postures, and motion trajectories) are extracted based on the temporal feature graph by identifying significant change points and tracking joint position changes. They more directly describe the specific posture and motion path of the human body and are more convenient for subsequent style transfer operations. For example, in walking motion, by analyzing the joint relative position feature graph and the motion direction feature graph, it can be determined that the person is walking; by extracting key postures, the starting, middle, and ending states of the walking process can be identified; by tracking joint position changes, the walking trajectory information can be obtained.

[0180] The sports content characteristics include at least one of the following characteristics:

[0181] Joint position and rotation angle: Extract the rotation state and position of each joint at each moment to define the specific posture of the human body.

[0182] Key poses: By identifying significant change points in the joint rotation sequence to define keyframes, we can extract iconic or transitional poses in the motion, ensuring that the style-transferred motion still conveys the meaning of the original action.

[0183] Motion trajectory: This is extracted by tracking the changes in joint position over time, describing the path of a body part's motion. The frame rate of motion sequence data is typically around 60 frames per second; motion trajectories are extracted by tracking the spatial position of joints at each moment, describing the path of a body part's motion over time. The position data of each joint at different moments is connected to form a continuous curve, which is the motion trajectory. By analyzing motion trajectories, we can understand the motion path, velocity, acceleration, and other information of body parts, thereby better understanding the content and style of movement.

[0184] Content encoder processing flow: joint rotation sequence data -> 1D temporal convolution layer group -> temporal feature map -> residual block layer (normalization) -> feature map -> feature vector Zc.

[0185] Among them, the residual block layer processing flow is: temporal feature map -> convolution layer inside the residual block -> feature map normalization -> nonlinear activation function processing of the residual block -> residual connection -> output residual.

[0186] Instance Normalization (IN) processing to extract motion content features:

[0187] Instance normalization is a technique used in deep learning to normalize feature maps. It is particularly suitable for style transfer tasks. The feature map of each sample is normalized to a mean of 0 and a standard deviation of 1. This process eliminates sample-specific style information and effectively preserves content information. This can be achieved by following the following steps:

[0188] 1) Calculate the mean μ of the c-channel of sample x using the following formulas (3) and (4): c and variance

[0189]

[0190] Among them, H is the height of the feature map and W is the width of the feature map.

[0191] 2) Use the following formula (5) to normalize the elements of each channel using the calculated mean and variance:

[0192]

[0193] Here, ε is a small constant used for numerical stability.

[0194] Residual connections add the input feature map directly to the output of the convolutional layer. This effectively transfers information, prevents gradient vanishing, and accelerates training.

[0195] The content encoder input is a sequence of joint rotations, because joint rotation information more accurately describes the character's movements and ensures the fidelity of the output motion.

[0196] 2) Style encoder E S Extract motion style features;

[0197] The sports style features include at least one of the following features:

[0198] Movement rhythm: Analyze the time intervals and speed changes of joint rotation sequences to determine the rhythmic characteristics of the movement so that the movement after style transfer has a similar rhythm.

[0199] Strength: Estimate the force of movement through the velocity and acceleration of joint rotations, making the style-transferred movements appear more powerful or softer.

[0200] Dynamic range: Calculate the range of joint rotation angles to determine the dynamic amplitude of the movement, making the movement after style transfer more exaggerated or more restrained.

[0201] Emotion: Simulate emotional expression through the speed, amplitude, and rhythm of joint movements to convey specific emotions or expressions, such as happiness, sadness, or anger.

[0202] Posture preference: Common postures in joint rotation sequences are counted to extract preferences and shift specific posture habits, such as straight back or hunched back.

[0203] The style of 3D motion is often reflected in the joint's motion trajectory, speed, acceleration and other characteristics. The joint position information more directly reflects the character's position and motion state in space and is more suitable for capturing the character's motion style. The input data is the 3D joint position, and the style encoder E s With content encoder E C Similar to using a one-dimensional temporal convolutional network structure, style features of 3D joint position sequences are extracted, such as the relative speed of movement, the relative position relationship between joints, etc.

[0204] Style feature vector Z s Contains the style information of the input motion data, such as the rhythm, speed, strength, emotion, etc. of the movement.

[0205] The processing flow of the style encoder: joint position sequence data -> 1D temporal convolution layer group -> temporal feature map (speed, acceleration, etc.) -> feature map -> feature vector Z s .

[0206] Step S430: Decoder F decodes the content features and style features.

[0207] The role of the decoder F is to transform the content feature Z c and style Z s Decode and generate animation clips with the target style. Apply the mean and standard deviation of the style features to the normalized content features. This can be achieved by the following steps:

[0208] 1) Motion content vector Z c —>A series of residual blocks (convolutional layer, AdaIn layer) —>Time signal (vector), where each residual block contains a time convolutional layer and an AdaIn layer; the convolutional layer operates on the time dimension and converts Z c Decoded into a temporal signal in a higher-dimensional feature space. The temporal signal represents a high-dimensional feature representation of joint rotations and contains detailed information about the motion. AdaIN performs style injection, i.e., affine transformations in documents.

[0209] Using adaptive instance normalization (AdaIN) as follows (6), the decoder combines the statistical information of content features and style features to generate a new motion sequence with the target style.

[0210]

[0211] Assume that AdaIN receives content input x and style input y, calculates the channel mean μ(x) and variance σ(x) of content input x; calculates the channel mean μ(y) and variance σ(y) of style input y;

[0212] Align the channel mean μ(x) and variance σ(x) of the content input x to match the value of the style input y, and adjust the mean and variance of each channel of the content input x to 0 and 1 through instance normalization.

[0213] Scale the normalized content input using the variance σ(y) of the style input y;

[0214] Shift the scaled content input by the mean μ(y) of the style input y.

[0215] The AdaIN layer constitutes a normalization layer that applies an affine transformation to the feature activations. That is, the style code Z s Converted into parameters, these parameters are used to adjust Z cThe mean and variance of the feature activations of each channel of , thereby injecting the target style.

[0216] The affine transformation introduced by the AdaIN layer is invariant in time. This transformation does not change the temporal characteristics of the motion, but only affects the non-temporal properties of the motion.

[0217] Convolutional layers with strides are used to extract and transform features. These convolutional layers can further process and manipulate the features while maintaining temporal resolution. Finally, the decoder F restores the spatial resolution of the features through upsampling to match the original input.

[0218] 2) Temporal signal (vector) -> upsampling operation -> joint rotation features at higher temporal resolution. The temporal features are upsampled and the resolution is restored through convolutional layers and convolution operations with a stride of 2.

[0219] 3) Joint rotation features -> output layer (composed of multiple linear layers) -> final motion sequence. In motion style transfer tasks, the output dimension is typically T×J×4, where T is the number of time steps, J is the number of joints, and 4 is the dimension of each joint rotation represented by a unit quaternion. The output layer maps the upsampled feature representation to the final output dimension.

[0220] The interpolation algorithm is used to ensure that the final output motion sequence has the same temporal resolution as the input motion sequence, ensuring that the generated motion sequence is temporally coherent without sudden jumps or unnatural transitions.

[0221] In an embodiment of the present application, an unpaired data training strategy is adopted to construct a content encoder and a style encoder respectively to extract content features and style features, and use a decoder to fuse the two, and finally generate a new animation clip with a target style. Among them, the content encoder is responsible for extracting information describing the content of the animation action (such as action type, body posture, motion trajectory, etc.), while the style encoder focuses on extracting style attributes (such as rhythm, speed, strength, emotion, etc.). The decoder injects style features into content features through adaptive instance normalization (AdaIN) technology, thereby achieving high-quality style transfer. In this way, the animation style transfer achieved does not require paired training data, thereby reducing data acquisition costs and improving efficiency; the content retention effect is good, and the style and content can be effectively separated, thereby retaining the motion content while migrating the style.

[0222] Based on the foregoing embodiments, an embodiment of the present application provides an animation style transfer device, which includes modules, each module includes sub-modules, each sub-module includes a unit, and can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; during implementation, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0223] Figure 5 This is a schematic diagram of the structure of the animation style transfer device provided in the embodiment of the present application, as shown in FIG. Figure 5 As shown, the apparatus 500 includes:

[0224] A content extraction module 510 is configured to extract content features of the first animation clip using a content encoder to obtain a content feature vector;

[0225] A style extraction module 520 is configured to extract style features of the second animation clip using a style encoder to obtain a style feature vector;

[0226] A decoding module 530 is configured to decode the content feature vector and the style feature vector using a decoder to obtain a third animation clip, wherein the third animation clip is an animation clip in which the style of the second animation clip is transferred to the first animation clip; and the content encoder and the style encoder are temporal convolutional neural networks.

[0227] In some embodiments, the content encoder includes a temporal convolution layer, a residual block and an average pooling layer; the content extraction module 510 includes a first analysis submodule, a first convolution processing submodule, a normalization processing submodule and a first mapping submodule, wherein the first molecular submodule is used to analyze the first animation clip to obtain joint rotation sequence data corresponding to the first animation clip; the first convolution processing submodule is used to use at least one temporal convolution layer to perform convolution processing on the joint rotation sequence data to obtain at least one temporal feature map; the normalization processing submodule is used to set the residual block after each temporal convolution layer, and use the residual block to normalize the temporal feature map to obtain a content feature map that removes style information and retains content information; the first mapping submodule is used to use the average pooling layer to map the content feature map to the content feature vector.

[0228] In some embodiments, the residual block includes a convolution layer, a normalization layer, an activation layer and a connection layer, and the normalization processing submodule includes a convolution processing unit, a normalization processing unit, a nonlinear activation function processing unit and a connection unit, wherein the convolution processing unit is used to use the convolution layer to perform convolution processing on the temporal feature map to obtain a first feature map; the normalization processing unit is used to use the normalization layer to perform normalization processing on the first feature map, remove the style information in the first feature map, retain the content information, and obtain a second feature map; the nonlinear activation function processing unit is used to use the activation layer to perform nonlinear activation function processing on the second feature map to obtain a third feature map; the connection unit is used to use the connection layer to connect the temporal feature map and the third feature map to obtain the content feature map.

[0229] In some embodiments, the content feature vector includes at least one of the following information corresponding to the first animation clip: action type, motion trajectory, and body posture.

[0230] In some embodiments, the content encoder includes a temporal convolution layer and a global pooling layer; the style extraction module 520 includes a second analysis submodule, a second convolution processing submodule and a second mapping submodule, wherein the second analysis submodule is used to analyze the second animation clip to obtain joint position sequence data corresponding to the second animation clip; the second convolution processing submodule is used to use at least one temporal convolution layer to perform convolution processing on the joint position sequence data to obtain at least one temporal feature map; the second mapping submodule is used to use the global pooling layer to map the last temporal feature map output by the last temporal convolution layer to the style feature vector.

[0231] In some embodiments, the style feature vector includes at least one of the following information corresponding to the second animation clip: motion rhythm, speed, strength, or emotion.

[0232] In some embodiments, the decoder comprises at least one residual block and an output layer comprising at least one linear layer, wherein each residual block comprises a convolutional layer and an adaptive instance normalization layer arranged after the convolutional layer; the decoding module 530 comprises a decoding submodule, a style injection submodule and a third mapping submodule, wherein the decoding submodule is configured to utilize the convolutional layer in the residual block to decode the content feature vector into a time vector in a preset dimensional feature space in the time dimension; the style injection submodule is configured to utilize the adaptive instance normalization layer in the residual block to perform style injection into the time vector based on the style feature vector to obtain a target time vector for completing the style injection; the third mapping submodule is configured to utilize the output layer to map the target time vector into a target motion sequence through a linear transformation to obtain the third animation segment based on the target motion sequence.

[0233] In some embodiments, the decoder also includes an upsampling layer, and the decoding module 530 also includes an upsampling submodule, which is used to use the upsampling layer to upsample the target time vector to obtain a joint rotation feature vector corresponding to a preset resolution; correspondingly, the third mapping submodule is also used to use the output layer to map the joint rotation feature vector into a target motion sequence through a linear transformation, so as to obtain the third animation segment based on the target motion sequence.

[0234] In some embodiments, in the process of training the style transfer network including the content encoder, the style encoder and the decoder, at least an adversarial loss function is used, wherein,

[0235] The adversarial loss function includes a generator loss function and a discriminator loss function. Using the adversarial loss function during the training process can make the third animation clip output by the decoder close to the preset real animation clip.

[0236] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0237] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0238] Correspondingly, an embodiment of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the animation style transfer method provided in the above embodiment.

[0239] Correspondingly, an embodiment of the present application provides an electronic device, Figure 6 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the hardware entity of the device 600 includes: a memory 601 and a processor 602, wherein the memory 601 stores a computer program that can be run on the processor 602, and when the processor 602 executes the program, the steps in the animation style transfer method provided in the above embodiment are implemented.

[0240] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (RAM).

[0241] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0242] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0243] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0244] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0245] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0246] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0247] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0248] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words be embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0249] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0250] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0251] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0252] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for animation style transfer, the method comprising: Extracting content features of the first animation clip using a content encoder to obtain a content feature vector; Extracting style features of the second animation clip using a style encoder to obtain a style feature vector; The content feature vector and the style feature vector are decoded using a decoder to obtain a third animation clip, wherein the third animation clip is an animation clip in which the style of the second animation clip is transferred to the first animation clip; the content encoder and the style encoder are temporal convolutional neural networks.

2. The method of claim 1, wherein the content encoder comprises a temporal convolution layer, a residual block, and an average pooling layer; and extracting content features of the first animation clip using the content encoder to obtain a content feature vector comprises: Analyzing the first animation clip to obtain joint rotation sequence data corresponding to the first animation clip; Performing convolution processing on the joint rotation sequence data using at least one temporal convolution layer to obtain at least one temporal feature graph; The residual block is set after each of the temporal convolutional layers, and the temporal feature map is normalized by using the residual block to obtain a content feature map that removes style information and retains content information; The content feature map is mapped into the content feature vector using the average pooling layer.

3. The method according to claim 2, wherein the residual block includes a convolutional layer, a normalization layer, an activation layer, and a connection layer, and the normalizing the temporal feature map using the residual block comprises: Performing convolution processing on the temporal feature map using the convolution layer to obtain a first feature map; Normalizing the first feature map using the normalization layer to remove style information from the first feature map and retain content information to obtain a second feature map; Using the activation layer to perform a nonlinear activation function on the second feature map to obtain a third feature map; The temporal feature map and the third feature map are connected using the connection layer to obtain the content feature map.

4. The method according to claim 2, wherein the content feature vector comprises at least one of the following information corresponding to the first animation clip: action type, motion trajectory, and body posture.

5. The method of claim 1, wherein the content encoder comprises a temporal convolution layer and a global pooling layer; and wherein extracting the style features of the second animation clip using the style encoder to obtain the style feature vector comprises: Analyzing the second animation clip to obtain joint position sequence data corresponding to the second animation clip; Performing convolution processing on the joint position sequence data using at least one temporal convolution layer to obtain at least one temporal feature graph; The global pooling layer is used to map the last temporal feature map output by the last temporal convolutional layer into the style feature vector. 6 . The method according to claim 5 , wherein the style feature vector comprises at least one of the following information corresponding to the second animation clip: motion rhythm, speed, strength, or emotion.

7. The method according to any one of claims 1 to 6, wherein the decoder comprises at least one residual block and an output layer comprising at least one linear layer, wherein: Each residual block includes a convolution layer and an adaptive instance normalization layer provided after the convolution layer; the decoder is used to decode the content feature vector and the style feature vector to obtain a third animation clip, including: Decoding the content feature vector into a time vector in a feature space of a preset dimension in a time dimension using a convolutional layer in the residual block; Using an adaptive instance normalization layer in the residual block, style injection is performed on the time vector based on the style feature vector to obtain a target time vector after completing the style injection; The target time vector is mapped into a target motion sequence by linear transformation using the output layer, so as to obtain the third animation clip based on the target motion sequence.

8. The method of claim 7, wherein the decoder further comprises an upsampling layer, and before using the output layer to map the target time vector into a target motion sequence through a linear transformation to obtain the third animation clip based on the target motion sequence, the method further comprises: Upsampling the target time vector using the upsampling layer to obtain a joint rotation feature vector corresponding to a preset resolution; Correspondingly, the using the output layer to map the target time vector into a target motion sequence through a linear transformation to obtain the third animation clip based on the target motion sequence includes: The output layer is used to map the joint rotation feature vector into a target motion sequence through linear transformation, so as to obtain the third animation clip based on the target motion sequence.

9. The method according to any one of claims 1 to 6, wherein in the process of training the style transfer network comprising the content encoder, the style encoder and the decoder, at least an adversarial loss function is used, wherein: The adversarial loss function includes a generator loss function and a discriminator loss function. Using the adversarial loss function during the training process can make the third animation clip output by the decoder close to the preset real animation clip.

10. An electronic device, comprising: A content encoder, configured to extract content features of the first animation clip to obtain a content feature vector; A style encoder, configured to extract style features of the second animation clip and obtain a style feature vector; A decoder is configured to decode the content feature vector and the style feature vector to obtain a third animation clip, wherein the third animation clip is an animation clip in which the style of the second animation clip is transferred to the first animation clip; and the content encoder and the style encoder are temporal convolutional neural networks.

Images