Physical reality-based action style migration method
Through the attention matrix fusion technology of the dynamic fusion module, the problem of the transfer of action styles in the existing technology that conform to physical reality is solved, effectively retaining the local semantic structure of the action content, and generating stylized actions that conform to physical reality is generated.
Patent Information
- Application Number
- CN202411994617.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
The prior art cannot achieve the transfer of action styles that conform to physical reality, and ignores the local semantic structure of action content, resulting in the destruction of action structure.
Through the dynamic fusion module, the action content and action style are regarded as two different modes. Data fusion is carried out by building an attention matrix, and the shared query vector is set to achieve feature alignment of the style mode and the content mode. Finally, the attention weighted average value of the action content and style data is calculated based on the attention matrix to generate a stylized action that conforms to physical reality.
Effectively retain the local semantic structure of the action content, generate stylized actions that conform to physical reality, avoid the damage to the action structure, and improve the visual fidelity of the generated action.
Smart Images
Figure CN119919279A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of action style transfer, and in particular to an action style transfer method based on physical reality. Background Art
[0002] Human action style can usually be regarded as a collection of action attributes such as emotions, personality, and behavior. It carries rich information to reflect the current characteristic state of the character. Action style transfer technology can achieve fast switching of different styles of the same action and efficiently generate actions with different emotional styles, which is widely used in movies, games and other fields.
[0003] The invention with publication number CN118570053A discloses an image style transfer processing method and device, an electronic device and a storage medium, and proposes a solution for transferring the style of a reference image to a specified image, so as to effectively integrate the style of the reference image while maintaining the integrity of the specified image content;
[0004] However, this solution only achieves image style transfer, but cannot achieve action style transfer of digital humans, let alone action style transfer that conforms to physical reality.
[0005] There are other papers that have achieved action style transfer by borrowing from image style transfer and other deep learning methods. However, most of them use global feature statistical methods such as AdaIN to achieve style transfer. Although such methods can transfer the global mean and variance of the action style to the feature space of the action content to transfer the style, they ignore the local semantic structure of the action content, which ultimately leads to the destruction of the action structure and the inability to achieve action style transfer that conforms to physical reality. Summary of the invention
[0006] The purpose of the present invention is to provide a method for transferring action styles based on physical reality in order to overcome the defects of the above-mentioned prior art and realize the transfer of action styles based on physical reality.
[0007] The purpose of the present invention can be achieved by the following technical solutions:
[0008] A method for transferring action styles based on physical reality, comprising the following steps:
[0009] The motion content data with the original style and the motion style data with the target style are obtained, and data fusion processing is performed through a dynamic fusion module. The processing process of the dynamic fusion module includes:
[0010] Feature extraction is performed on the action content data and the action style data respectively to obtain the feature vector of the action content and the feature vector of the action style; a shared query vector is constructed, and the feature vector of the action content and the feature vector of the action style are multiplied with the shared query vector respectively, and the attention matrix is obtained after being concatenated with each other;
[0011] The attention-weighted average values of the action content data and the action style data are calculated according to the attention matrix to obtain a final feature map; and the final feature map is decoded to obtain an action sequence after style transfer.
[0012] 2. According to the method of action style transfer based on physical reality in claim 1, it is characterized in that the acquisition process of the attention matrix is specifically as follows:
[0013] Encode the feature vector of the action content as key K c Sum value V c , encode the feature vector of the action style as key K s Sum value V c , the expression of the attention matrix is:
[0014]
[0015] Where A is the attention matrix, is matrix multiplication, Q is the shared query vector, Concat is the concatenation of two matrices, and Softmax is the activation function.
[0016] Furthermore, the expression of the final feature map is:
[0017]
[0018] In the formula, Z fin is the final feature map.
[0019] Furthermore, the method further includes inputting the decoded action sequence and the corresponding style label into a discriminator, and judging whether it is a real action or a generated fake action by the discriminator, thereby supervising the training process of the dynamic fusion module according to the judgment result;
[0020] The discriminator includes a plurality of sequentially connected discriminator blocks, each of which obtains the global features of the input action through a multi-head self-attention mechanism, and further extracts the fine-grained features of the action sequence through an action style enhancement module;
[0021] The extraction results finally output by each identification block are concatenated with the style label to obtain the output result of the discriminator. Furthermore, the expression for obtaining the global features of the input action through the multi-head self-attention mechanism is:
[0022] m i =MHSA(Linear(LN(M G )))
[0023]
[0024] In the formula, m i is the intermediate feature of the discriminative block, Q, K, V represent the query, key and value learned by the neural network, b is the parameterized attention bias, M G For input data.
[0025] Furthermore, the processing expression of the action style enhancement module is:
[0026] m i+1 =Conv(Conv G (m i ))+Conv(m i )
[0027] In the formula, m i+1 is the output feature of the identification block, m i To identify the intermediate features of the block, Conv is a convolution operation. G The convolutional layer is followed by the activation function ReLU.
[0028] Furthermore, the loss function in the training process of the dynamic fusion module includes reconstruction loss, adversarial loss, style triple loss and unit quaternion loss.
[0029] Furthermore, the calculation expression of the adversarial loss is:
[0030] L adv =||D s (M s |S)-1|| 2 +||D s (M G |S l )|| 2
[0031] Where, L adv For adversarial loss, S is the set of all styles, S1 is the style label input to the discriminator, and D s (M s |S) is the sum of all style sets S and real actions M s The output of the discriminator, D s (M G |S l ) is based on all style tags S l and false action M G The output of the discriminator.
[0032] Furthermore, the reconstruction loss is used to determine the input action style M of the given dynamic fusion module. s and the action M generated by it G To calculate, the corresponding calculation expression is:
[0033] L rec =||M G -M s ||1
[0034] Where, L rec To rebuild the losses;
[0035] The style triple loss is used to encourage latent vectors of similar styles to be mapped together, and the corresponding calculation expression is:
[0036]
[0037] Where, L trip is the style triplet loss, s1∈S,s2∈S are two different styles;
[0038] The unit quaternion loss encourages the generated quaternion to be a unit quaternion, and the corresponding calculation expression is:
[0039]
[0040] Where, L uq is the unit quaternion loss, T is the number of frames of the action, M s Enter the action style.
[0041] Furthermore, the method is used in film making, game design or virtual reality scenes.
[0042] Compared with the prior art, the present invention has the following advantages:
[0043] (1) The present invention proposes a novel framework: MSTDF, which regards action style and action content as two different modalities. By constructing an attention matrix for dynamic fusion, a shared query vector is set to enable the style modality and content modality to share feature information and extract more cross-modal relationships. Finally, the attention matrix is applied to the action style features. According to the importance of all style features and content features, dynamic weights are assigned to the target feature action. The final features can highlight the strong features and suppress the weaker features in the feature fusion module. This can better preserve the local semantic structure of the action content and generate stylized actions that conform to physical reality.
[0044] (2) The present invention also proposes an action style supervision module, namely, the MDF discriminator, which takes an action sequence and a style label as input, and then determines whether the input action is a real action with style or a fake action generated by the action style transfer module, thereby supervising the generation quality of the entire style transfer task and improving the visual fidelity of the generated action.
[0045] (3) The present invention can realize fast switching of different styles of the same action and can be widely used in multimedia scenarios such as film production and game design. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A schematic diagram of a framework of a dynamic fusion module provided in an embodiment of the present invention;
[0047] Figure 2 It is a schematic diagram of the structure of an MDF identifier provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0049] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0051] Example 1
[0052] This embodiment provides an action style transfer method based on physical reality. First, a dynamic fusion module is proposed to regard action style and action content as two different modalities, and the distribution of the two modalities in the feature space is aligning through a dynamic fusion method, thereby achieving action style transfer that conforms to physical reality and avoiding destruction of the local semantic structure of the action.
[0053] In addition, in order to improve the style consistency between the generated stylized motion and the input motion, this embodiment designs an action style supervision module, called Motion Dynamic Fusion (MDF) discriminator, which is used to supervise the generation quality of the motion.
[0054] The details are as follows:
[0055] 1. Dynamic Fusion Module
[0056] In the prior art, global feature statistics methods such as AdaIN are used to achieve style transfer. Although the global mean and variance of the action style can be transferred to the feature space of the action content to transfer the style, they ignore the local semantic structure of the action content, which ultimately leads to the destruction of the action content structure. Therefore, this embodiment proposes the MSTDF framework, which regards the action content and action style as two different signals, and aligns their feature distributions through a dynamic fusion method, thereby achieving action style transfer.
[0057] like Figure 1 As shown, the processing process includes the following steps:
[0058] The action content data with the original style and the action style data with the target style are obtained, and the data fusion processing is performed through the dynamic fusion module. The processing process of the dynamic fusion module includes:
[0059] Feature extraction is performed on the action content data and the action style data respectively to obtain the feature vector of the action content and the feature vector of the action style; a shared query vector is constructed, and the feature vector of the action content and the feature vector of the action style are multiplied with the shared query vector respectively, and the attention matrix is obtained after being concatenated with each other;
[0060] According to the attention matrix, the attention-weighted average of the action content data and the action style data is calculated to obtain the final feature map; the final feature map is decoded to obtain the action sequence after style transfer.
[0061] Specifically, the input of the dynamic fusion module is two actions: action content M with original style c∈S c ∈R 4J ×T , and an action style M with a target style s∈S s ∈R 3J×T , where S is the set of all styles, T is the number of frames of the action, and J = 21 is the number of joints in the human skeleton. Using the content encoder E mc and style encoder E ms Separately transform the two actions M c and M s Compress to low-dimensional space z and extract the feature vector of the action content and the feature vector of the action style
[0062] In order to enable the style modality and content modality to share feature information, extract more cross-modal relationships, and promote the model to learn cross-modal common representations, this embodiment designs a shared query vector Q∈R d×1 , and learn K through neural network s ,K c ,V s ,V c ∈R d×T .
[0063] The attention matrix A can be calculated by the following formula:
[0064]
[0065] in, Represents matrix multiplication. Concat means concatenating two matrices.
[0066] Subsequently, the attention matrix A is applied to the action style features. The action content and style features have different contributions to the final generated action sequence. Therefore, the model will assign dynamic weights to the target feature action according to the importance of all style features and content features.
[0067] Based on this understanding, the action content M can be calculated c ∈R 4J×T and action style M s ∈R 3J×T The weighted average of the attention of each distribution m c and m s , and then get the final feature map Z fin ∈R d×1 :
[0068]
[0069] Among them, A T ∈R 1×T , V s ,V c ∈R d×T , Represents matrix addition. This method provides the optimal fusion weight for each feature, which can highlight the strong features and suppress the weaker features in the feature fusion module.
[0070] Finally, the action decoder F is used to decode the feature map Z fin Decode to get the action sequence after style transfer Its target style label is s∈S.
[0071] In short, MSTDF achieves style transfer through dynamic fusion. Compared with the AdaIN method, MSTDF considers point-by-point statistics rather than simple local feature statistics, which can better preserve the local semantic structure of the action content and generate stylized actions that conform to physical reality.
[0072] 2. Action Style Supervision Module
[0073] In order to solve the problem that the action style generated by the action style transfer module is inconsistent with the input action style, so as to generate a motion that is more physically realistic and closer to the original style, this embodiment designs a new discriminator, Motion Dynamic Fusion (MDF) discriminator, such as Figure 2 As shown, the processing process includes:
[0074] The decoded action sequence and the corresponding style label are input into the discriminator, which determines whether it is a real action or a generated fake action, and supervises the training process of the dynamic fusion module based on the judgment result;
[0075] The discriminator consists of multiple discriminator blocks connected in sequence. Each discriminator block obtains the global features of the input action through a multi-head self-attention mechanism, and further extracts fine-grained features of the action sequence through an action style enhancement module.
[0076] The final extraction results of each identification block are concatenated with the style label to obtain the output result of the discriminator.
[0077] That is, this embodiment combines an action sequence and a style label S l ∈S as input, and then judge whether the input action is a real action with style i∈S or a fake action M generated by the action style transfer module G ∈R d′×T , thereby supervising the entire style transfer task and improving the fidelity of generated actions.
[0078] Specifically, the MDF discriminator is composed of several MDF blocks, and its output can be expressed as:
[0079]
[0080] Here, n is the summary of MDF blocks. In our implementation, n=3. Next, we will introduce the structure of MDFblocks in detail. First, we use the multi-head self-attention mechanism to obtain the global features of the input action, which can be formally expressed as:
[0081] m i =MHSA(Linear(LN(M G )))
[0082]
[0083] Among them, m i|i＞0 represents the intermediate features of MDF blocks, Q, K, V represent the query, key and value learned by the neural network, and b is the parameterized attention bias as the position encoding. On this basis, this embodiment designs an action style enhancement module to further extract fine-grained features of action sequences while avoiding the gradient disappearance of deep networks:
[0084] m i+1 =Conv(Conv G (m i ))+Conv(m i )
[0085] Among them, Conv G indicates that the convolutional layer is followed by the activation function ReLU. Then, we convert m i+1 With style tag S l ∈S together, which is the output of the MDF discriminator:
[0086] D s (M G |S1)=Concat(m i+1 ,S l )
[0087] 3. Training process of dynamic fusion module
[0088] Given action content M c ∈R 4J×T and action style M s ∈R 3J×T , the model is trained in a way similar to GAN, and the loss function used is as follows:
[0089] Reconstruction loss: Given the input action style M of the action style transfer module s ∈R 3J×T and the action M generated by it G ∈R d′×T To calculate the reconstruction loss, it is used to quantitatively measure the degree of content loss:
[0090] L rec =||M G -M s ||1
[0091] Adversarial loss: The MDF discriminator and the generator complete the entire training process by adversarial:
[0092] L adv =||Ds (M s |S)-1|| 2 +||D s (M G |S l )|| 2
[0093] Among them, S is the set of all styles, S l is the style label that is input to the discriminator.
[0094] Style triple loss: In order to encourage latent vectors of similar styles to be mapped together, thereby improving the clustering effect of styles, the corresponding expression is as follows:
[0095]
[0096] Among them, s1∈S, s2∈S are two different styles, and δ=4.
[0097] Unit quaternion loss. This loss function encourages the generated quaternion to be a unit quaternion, thereby generating more physically realistic action sequences:
[0098]
[0099] The final loss function is a combination of the above four loss functions:
[0100] L total =L rec +L adv +L trip +L uq
[0101] The method provided in this embodiment can realize fast switching of different styles of the same action, and can be widely used in multimedia scenes such as film production and game design to produce more realistic and expressive character animation.
[0102] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A method for transferring action styles based on physical reality, characterized in that: The following steps are involved: The motion content data with the original style and the motion style data with the target style are obtained, and data fusion processing is performed through a dynamic fusion module. The processing process of the dynamic fusion module includes: Feature extraction is performed on the action content data and the action style data to obtain the feature vector of the action content and the feature vector of the action style; a shared query vector is constructed, and the feature vector of the action content and the feature vector of the action style are multiplied with the shared query vector respectively, and the attention matrix is obtained after being concatenated with each other; Calculating the attention weighted average of the action content data and the action style data according to the attention matrix to obtain a final feature map; The final feature map is decoded to obtain the action sequence after style transfer.
2. The method for transferring action style based on physical reality according to claim 1, characterized in that: The acquisition process of the attention matrix is specifically as follows: Encode the feature vector of the action content as key K c Sum value V c , encode the feature vector of the action style as key K s Sum value V c , the expression of the attention matrix is: Where A is the attention matrix, is matrix multiplication, Q is the shared query vector, Concat is the concatenation of two matrices, and Softmax is the activation function.
3. The method for transferring action style based on physical reality according to claim 2, characterized in that: The expression of the final feature map is: In the formula, Z fin is the final feature map.
4. The method for transferring action style based on physical reality according to claim 1, characterized in that: The method further includes inputting the decoded action sequence and the corresponding style label into a discriminator, and judging whether it is a real action or a generated fake action by the discriminator, thereby supervising the training process of the dynamic fusion module according to the judgment result; The discriminator includes a plurality of sequentially connected discriminator blocks, each of which obtains the global features of the input action through a multi-head self-attention mechanism, and further extracts the fine-grained features of the action sequence through an action style enhancement module; The final extraction results of each identification block are concatenated with the style label to obtain the output result of the discriminator.
5. The method for transferring action style based on physical reality according to claim 4, characterized in that: The expression for obtaining the global features of the input action through the multi-head self-attention mechanism is: m i =MHSA(Linear(LN(M G ))) In the formula, m i is the intermediate feature of the discriminative block, Q, K, V represent the query, key and value learned by the neural network, b is the parameterized attention bias, M G For input data.
6. The method for transferring action style based on physical reality according to claim 4, characterized in that: The processing expression of the action style enhancement module is: m i+1 =Conv(Conv G (m i ))+Conv(m i ) In the formula, m i+1 is the output feature of the identification block, m i To identify the intermediate features of the block, Conv is a convolution operation. G The convolutional layer is followed by the activation function ReLU.
7. The method for transferring action style based on physical reality according to claim 4, characterized in that: The loss functions in the training process of the dynamic fusion module include reconstruction loss, adversarial loss, style triple loss and unit quaternion loss.
8. The method for transferring action style based on physical reality according to claim 7, characterized in that: The calculation expression of the adversarial loss is: L adv =||D s (M s |S)-1|| 2 +||D s (M G |S l )|| 2 Where, L adv To combat the loss, S is the set of all styles, S l is the style label input to the discriminator, D s (M s |S) is the sum of all style sets S and real actions M S The output of the discriminator, D s (M G |S l ) is the sum of all style labels S1 and fake actions M G The output of the discriminator.
9. The method for transferring action style based on physical reality according to claim 7, characterized in that: The reconstruction loss is used for the input action style M of the given dynamic fusion module s and the action M it generates G To calculate, the corresponding calculation expression is: L rec =||M G -M s ||1 Where, L rec To rebuild the losses; The style triple loss is used to encourage latent vectors of similar styles to be mapped together, and the corresponding calculation expression is: Where, L trip is the style triplet loss, s1∈S,s2∈S are two different styles; The unit quaternion loss encourages the generated quaternion to be a unit quaternion, and the corresponding calculation expression is: Where, L Uq is the unit quaternion loss, T is the number of frames of the action, M s Enter the action style.
10. The method for transferring action style based on physical reality according to claim 1, characterized in that: The method is used in film making, game design or virtual reality scenarios.