Automatic generation method of Laban dance score based on part specific attention Transformer model

Abstract

The invention provides a Laban dance score automatic generation method based on a part specific attention Transform model, and relates to the technical field of computer vision and image processing. Preprocessing the motion data, wherein the preprocessing comprises coordinate transformation and normalization; the part specific attention module outputs a feature sequence {g1, g2,..., gL}, and the feature sequence {g1, g2,..., gL} is input into the coder-decoder after position coding; an encoder-decoder architecture is adopted to learn dynamic characteristics of body movement of the dance along the time dimension; the decoder sequentially outputs a sequence {z1, z2,..., zL}, and maps the sequence {z1, z2,..., zL} into a Laban symbol; and arranging the generated Laban symbols according to a time sequence to obtain a Laban dance score corresponding to the target dance. According to the method, the feature correlation between different limb parts is learned through the part specific attention module, the overall human body motion features are extracted, the complex dynamic features of dance actions in time are learned by using the Transform encoder-decoder model, and finally the high-quality laban dance score is generated, so that the score recording efficiency of the laban dance score can be greatly improved.

Description

Technical Field

[0001] The present invention provides a method for automatically generating Laban dance notation based on a part-specific attention Transformer model, and relates to the technical fields of computer vision and image processing. Background Art

[0002] Traditional folk dance is an important part of the intangible cultural heritage of mankind. Laban dance notation is one of the widely used symbol systems in the world to record human dance movements. Using Laban dance notation to record and preserve traditional folk dance has the advantages of easy preservation and reproduction of dance movements, easy teaching, communication and dissemination, and is an effective means to protect the precious intangible cultural heritage of traditional dance.

[0003] At present, a large number of Laban dance notations are still drawn by hand, and the notation process is complicated and takes a lot of time and effort. In order to reduce the difficulty of drawing, some computer-aided tools have been developed, such as Laban Writer, LED&LINTEL and LabanEditor. However, using these tools to record Laban dance notations requires professional training. Moreover, even if a professional notator has rich experience in Laban dance notation, he still needs a lot of time to carefully observe a dance movement before he can write down and finally proofread the correct Laban dance notation. This process may take several days or even weeks. Therefore, using computers to automatically generate Laban dance notations has important application significance and research value.

[0004] The current mainstream methods include the method of generating Laban notation from video and the method of generating Laban notation from 3D human motion capture data. The video-based Laban notation generation method is to identify the human posture in the video, convert the human motion information into 3D spatial coordinate information, and then further identify it as Laban notation. The Laban notation generation method based on 3D human motion capture data first uses motion capture technology to obtain accurate 3D human motion data, and then generates Laban notation by analyzing and identifying the human motion capture data.

[0005] 3D human motion capture data can faithfully record and restore the dance movements, but there are still many challenges in converting it into Laban dance notation. For example:

[0006] 1. Different dancers may have different body shapes and dance styles. If the three-dimensional spatial coordinates based on human body posture are rigidly mapped to Laban dance notation symbols, large errors may occur.

[0007] 2. The writing standards of Laban dance notation require the distinction and recording of specific movements and postures of different parts of the human body, including the head, upper limbs, lower limbs, etc. The existing methods either do not make this distinction or directly separate the limbs, without considering the impact of coordination between limbs on the judgment of dance movements, which may lead to low accuracy of the dance notation.

[0008] 3. Dance movements have different durations, so it is necessary to scientifically distinguish different movements and map them into Laban notation symbols. The existing technology either does not take this into consideration or there is a large error in the way of distinguishing movements, resulting in the final generated Laban notation being inaccurate.

[0009] At present, the video-based Laban dance score generation method is involved in the existing Chinese invention patents (CN110660017A, CN109255295B):

[0010] CN110660017A provides a dance score recording and demonstration method and process based on three-dimensional posture recognition, which has the following disadvantages:

[0011] 1. Dance movements are complex and delicate, and may include complex movements such as extension, flexion, rotation, and a variety of postures. Video is two-dimensional data projected onto the camera, which has problems such as front and back occlusions and lens distortion. It is difficult to accurately estimate the human body posture, resulting in inaccurate human body posture and Laban dance notation.

[0012] 2. Laban dance notation has detailed descriptions of the postures and movements of various parts of the human body in space. However, in the process of interpreting human postures into Laban dance notation, this technology simply maps the three-dimensional spatial position coordinates into specific Laban symbols, without considering the individual body shape differences of different dancers, the specific movement combinations of each limb part, and the temporal coherence and duration of dance movements. This can easily cause errors and result in low quality of the obtained Laban dance notation.

[0013] CN109255295B discloses a method, device, equipment and storage medium for generating dance scores based on vision. This technology has the following disadvantages:

[0014] The second prior art method for generating Laban dance notation based on body morphology is as follows: for each minimum rhythmic unit of the target dance, after obtaining the body morphology of the dancer in the minimum rhythmic unit, the body morphology of the dancer in the minimum rhythmic unit is matched with the body morphology information corresponding to each dance notation symbol, so as to obtain the corresponding dance notation symbol, and then the symbols are combined to form a dance notation.

[0015] This direct matching approach also has large errors: 1) It is difficult to deal with the differences in body shape between dancers; 2) The overall matching method does not take into account the different combinations of different limbs that may lead to different Laban symbols; 3) Matching is performed at each minimum rhythmic unit, and does not take into account the different durations of different movements, and the division of movements may not be consistent with the division of rhythmic units.

[0016] Representative Chinese journal papers on the Laban dance notation generation method based on 3D human motion capture data (Wang Jiaji, Miao Zhenjiang, Zhang Xueyan, et al. Automatic Generation of Laban Dance Notation for Digitalization of National Dynamic Arts [J]. Chinese Science: Information Sciences, 2019) use speed thresholds to segment action elements, taking into account the different durations of different actions, and distinguish between upper and lower limbs for action recognition, taking into account the Laban symbols of different limb parts. However, 1) the use of speed thresholds for segmentation has a limited scope of application, for example, it may not be effective for soothing dances; 2) it only mechanically distinguishes between the upper and lower limbs, without considering parts such as the head and torso, and it is also impossible to model the influence of the coordination relationship between different limbs on the overall dance movement posture; 3) the recognition of limb movements still uses quantitative mapping of spatial directions, which is also difficult to deal with the problem of body shape differences among different dancers. Summary of the invention

[0017] The present invention analyzes and recognizes 3D human motion capture data to generate Laban dance notation. The key technology of the present invention is the recognition of 3D human motion capture data based on a deep learning model, and proposes an automatic generation method of Laban dance notation based on a part-specific attention Transformer model.

[0018] The specific technical solution is: a method for automatically generating Laban dance notation based on a part-specific attention Transformer model, comprising the following steps:

[0019] S1, preprocessing the motion data, including coordinate transformation and normalization.

[0020] For each frame of data, the part-specific attention module is used to extract spatial features by body part through the attention mechanism, and the body is divided into five parts. The part-specific attention module learns the feature correlation between different limb parts and learns the flexible limb movement features in dance. Then, the spatial features of different parts are aggregated together to form the overall features of the current frame. The model learns how to correctly extract spatial features based on the body shape differences of different dancers.

[0021] S2, the part-specific attention module outputs a feature sequence {g1,g2,...,g L}, after position encoding, it is input into the encoder-decoder. The encoder-decoder architecture is used to learn the dynamic characteristics of body movement in dance along the time dimension;

[0022] S3, the decoder outputs the sequence {z1,z2,...,z L}, mapped to Laban symbols. The generated Laban symbols are arranged in chronological order to obtain the Laban dance score corresponding to the target dance.

[0023] The detailed method is described as follows:

[0024] The specific method of S1 is:

[0025] The related joints corresponding to the body parts are divided into a group, and each body part contains several joints. Define a body part as P, and decompose a frame of motion capture data into a group of body parts {P1, P2, ..., P n Considering the structure of body skeleton data, we set n = 5. Each body part consists of several joints, which can be regarded as a joint sequence {j1, j2, ..., j m}, the number of joints of all body parts is uniformly recorded as m.

[0026] For each body part P, spatial features are extracted from the raw joint features through a multi-head attention mechanism; more specifically, a scaled dot product attention mechanism is used. The self-attention mechanism has three inputs, denoted by Q, K, and V. The feature matrix of the input sequence is represented as S = {s1, s2, ..., s m}, where m represents the length of the sequence, and the three inputs of the self-attention module are obtained through three weight matrices:

[0027]

[0028] The dimension of S is m×d model , W Q The dimension is d model ×d Q , W K The dimension is d model ×d K , W V The dimension is d model ×d V . Set Q =d K =d V =d model / n head , where n head is the number of heads of multi-head attention. Attention weight a self By inputting the dot product of Q, K and the scalar And the normalization of the softmax function is obtained as shown below:

[0029]

[0030] The superscript self represents a self The self-attention weight, T represents the transposition operation, a self The dimension is m×d V .

[0031] Each time state of the input sequence is represented by a weighted value.

[0032] Apply the mask to the dot product calculation and repeat the attention calculation n head times, and concatenate the results to form the multi-head attention output:

[0033]

[0034] Where W O is the added weight dimension, implemented by the fully connected feed-forward layer, and its dimension is a mhead is the multi-head self-attention output, whose dimension is m×d model .

[0035] More specifically, for body part P i (i∈{1,2,....,n}) feature extraction, the length of the input sequence is the number of joints m, P i The feature dimension is m×d model Finally, the features of all body parts are concatenated to form a n×m×d model All frames are arranged in time order to obtain a series of spatial features {g1,g2,...,g t ,...,g L}, where g t represents the frame feature, and L is the length of the frame sequence.

[0036] The specific method of S2 is:

[0037] The encoder-decoder architecture of Transformer is used to learn the feature sequence and convert it into the corresponding Laban symbol sequence.

[0038] The spatial feature sequence output by the part-specific attention module is {g1,g2,...,g t ,...,g L}, as the input of the subsequent modules. First, for each input g t , a linear embedding layer is used to reshape the feature channels to obtain a deeper feature representation:

[0039] xt =W g g t , (4)

[0040] where x t represents the updated feature, whose dimension is d model , W g Represents a weight matrix whose dimension is d model ×d frame Then, in order for the model to understand the relative time position of each time series state, the position information of the sequence data is encoded into the input data. The position encoding function is defined as:

[0041]

[0042] Where i represents d model The i-th dimension of ist ist, t is the time step, that is, the position in the sequence. Define the position encoding vector as p t =PE(t). In this way, the position encoding process is expressed as:

[0043] y t =x t +p t , (6)

[0044] where y t is the position encoding data for each time step.

[0045] The specific methods of S3 are:

[0046] After encoding the input sequence with position information, the processed sequence is fed into the encoder-decoder architecture for temporal modeling and target sequence generation. By repeating and stacking encoders and decoders, a deep network is constructed to learn the temporal characteristics of motion feature sequences.

[0047] First, the encoder takes the entire input sequence and learns a more compact description. The decoding process is described as:

[0048]

[0049] where Y = {y1,y2,...,y t} is the encoded position sequence and E is the encoded output.

[0050] The decoding process is then described as:

[0051] Z=f decode (E,{<BOS>}), (8)

[0052] Where Z represents the decoded output and <BOS> represents the beginning of the sequence. Using the greedy strategy to output the symbol with the highest probability, the decoder outputs the Laban symbol sequence, which is represented as Z = {z1,z2,...,zt,...,z L'}. Where zt represents the Laban symbol at time t, and L' represents the length of the output sequence. Finally, the generated Laban symbols are combined into a Laban dance score according to the time sequence.

[0053] The technical solution of the present invention learns the feature correlation between different limb parts by designing a part-specific attention module, thereby extracting the overall human body movement characteristics, and using the Transformer encoder-decoder model to learn the complex dynamic characteristics of dance movements in time, and finally generates high-quality Laban dance notation. The proposed model can continuously improve the generalization ability of diversified data of different dancers after a large amount of data iterative learning. This method can obtain a recognition accuracy of more than 95% on a widely used academic evaluation data set, reaching the most advanced level of automatic generation of Laban dance notation. It can greatly improve the notation efficiency of Laban dance notation, save a lot of manpower and time, and effectively promote the rescue excavation and recording of traditional folk dances based on Laban dance notation, thereby contributing to the protection of intangible cultural heritage. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of the method of the present invention;

[0055] Figure 2 is a schematic diagram of a location-specific attention module of the present invention;

[0056] Figure 3 Schematic diagram of the encoder-decoder architecture of the present invention. DETAILED DESCRIPTION

[0057] The specific technical solution of the present invention is described in conjunction with the accompanying drawings.

[0058] The goal of the present invention is to automatically generate Laban dance notation from a three-dimensional human motion capture data sequence. Figure 1 The process of the Laban dance score automatic generation method based on the Transformer model with part-specific attention mechanism is shown, including the following steps:

[0059] S1, the present invention preprocesses the motion data, including coordinate transformation and normalization. For each frame of data, the present invention uses a part-specific attention module to extract spatial features according to body parts through the attention mechanism, such as Figure 1As shown in the figure, the body is divided into five parts. The part-specific attention module can learn the feature correlation between different limb parts to better learn the flexible limb movement characteristics in dance. Then, the spatial features of different parts are aggregated together to form the overall features of the current frame. Moreover, after training with a large amount of diverse data, the model can learn how to correctly extract spatial features under the body shape differences of different dancers.

[0060] S2, the part-specific attention module outputs a feature sequence {g1,g2,...,g L}, after position encoding, it is input into the encoder-decoder. Here, the present invention adopts the encoder-decoder architecture to learn the dynamic characteristics of body movement in dance along the time dimension, and solves the problem of different durations of complex dance movements.

[0061] S3, the decoder outputs the sequence {z1,z2,...,z L}, mapped into Laban symbols. The generated Laban symbols are arranged in chronological order to obtain the Laban dance score corresponding to the target dance. In this way, the present invention can achieve the purpose of automatically generating Laban dance score from motion capture data.

[0062] The detailed method is described below.

[0063] (1) Part-specific attention module

[0064] In the proposed method, the present invention first proposes a part-specific attention module, such as Figure 2 As shown in Figure 2, it can be used to capture the features of flexible limb movements. The part-specific attention module can learn the feature correlation between the joints of each body part and enhance the joint features through multi-head attention. Then, the joint features are further aggregated to form the overall features of the body skeleton in each frame.

[0065] The detailed instructions are as follows.

[0066] First, the present invention introduces the criteria for dividing the body into multiple parts, and then introduces the subsequent process of extracting specific features of each part. According to the writing standard of Laban dance notation, each limb movement, including the torso, must be recorded using Laban symbols. Therefore, by considering the hierarchical structure of motion capture data, the present invention divides the relevant joints corresponding to the body parts into a group, such as Figure 2 As shown in the left figure, different colors are used to represent different body parts, and each body part contains several joints. A body part is defined as P. The present invention can decompose a frame of motion capture data into a group of body parts {P1, P2, ..., P nConsidering the structure of body skeleton data, we set n = 5. Each body part consists of several joints, which can be regarded as a joint sequence {j1, j2, ..., j m},like Figure 2 For convenience, the number of joints of all body parts is uniformly recorded as m.

[0067] For each body part P, the present invention extracts spatial features from the original joint features through a multi-head attention mechanism, which can fully simulate the relationship between every two states in the sequence to learn the global dependency between sequence states. More specifically, the present invention utilizes the scaled dot product attention mechanism. The self-attention mechanism has three inputs, represented by Q, K and V respectively. The feature matrix of the input sequence is represented as S = {s1, s2, ..., s m}, where m represents the length of the sequence. The three inputs of the self-attention module can be obtained through three weight matrices:

[0068]

[0069] The dimension of S is m×d model , W Q The dimension is d model ×d Q , W K The dimension is d model ×d K , W V The dimension is d model ×d V . Generally, the present invention sets d Q =d K =d V =d model / n head , where n head is the number of heads of multi-head attention. Therefore, the attention weight a self You can input the dot product of Q, K and the scalar And the normalization of the softmax function is obtained as shown below:

[0070]

[0071] The superscript self represents a self The self-attention weight, T represents the transposition operation, a self The dimension is m×d V .

[0072] In this way, each time state of the input sequence can be represented by a weighted value. In addition, there is no need to consider future information when performing the self-attention mechanism during decoding, so the present invention applies a mask to the dot product calculation. Then, in order to obtain more information and improve the representation ability, the attention is repeatedly calculated n times. head times, and concatenate the results to form a multi-head attention output:

[0073]

[0074] Where W O is the added weight dimension, implemented by the fully connected feed-forward layer, and its dimension is a mhead is the multi-head self-attention output, whose dimension is m×d model The multi-head attention calculation process is as follows Figure 2 shown in the lower half of .

[0075] More specifically, for body part P i (i∈{1,2,....,n}) feature extraction, the length of the input sequence is the number of joints m, P i The feature dimension is m×d model Finally, the present invention further connects the features of all body parts to form a n×m×d model By arranging all frames in time sequence, the present invention can obtain a series of spatial features {g1,g2,...,g t ,...,g L}, where g t represents the frame feature, and L is the length of the frame sequence.

[0076] (2) Encoder-Decoder Architecture

[0077] In the proposed method, the part-specific attention module receives the motion capture data sequence and outputs the corresponding motion feature sequence. In order to learn the temporal dependencies of the output feature sequence, the present invention uses the encoder-decoder architecture of Transformer to learn the feature sequence and convert it into the corresponding Laban symbol sequence.

[0078] The spatial feature sequence output by the part-specific attention module is {g1,g2,...,g t ,...,g L}, as the input of the subsequent modules. First, for each input g t , the present invention adopts a linear embedding layer to reshape the feature channel to obtain a deeper feature representation:

[0079] x t =W g g t, (4)

[0080] where x t represents the updated feature, whose dimension is d model , W g Represents a weight matrix whose dimension is d model ×d frame Then, in order for the model to understand the relative time position of each time series state, the position information of the sequence data needs to be encoded into the input data. The position encoding function can be defined as:

[0081]

[0082] Where i represents d model The i-th dimension of t is the time step, i.e., the position in the sequence. The present invention defines the position encoding vector as p t =PE(t). In this way, the position encoding process can be expressed as:

[0083] y t =x t +p t , (6)

[0084] where y t is the position encoding data for each time step.

[0085] After encoding the input sequence with position information, the processed sequence is fed into the encoder-decoder architecture for temporal modeling and target sequence generation. The detailed structure of the encoder and decoder is shown in Figure 3 By repeating and stacking encoders and decoders, the present invention constructs a deep network to learn the temporal characteristics of motion feature sequences.

[0086] Typically, learning of temporal dynamics and global dependencies of long sequences of frames is achieved through a decoder-encoder architecture. First, the encoder takes the entire input sequence and learns a more compact description, which is also obtained by the multi-head attention mechanism. The present invention describes the decoding process as:

[0087]

[0088] where Y = {y1,y2,...,y t} is the encoded position sequence, and E is the encoded output. Then, the present invention describes the decoding process as:

[0089] Z=f decode (E,{<BOS>}), (8)

[0090] Where Z represents the decoded output and <BOS> represents the beginning of the sequence. Using the greedy strategy to output the symbol with the highest probability, the decoder outputs the Laban symbol sequence, which is represented as Z = {z1,z2,...,zt,...,z L'}. Where zt represents the Laban symbol at time t, and L' represents the length of the output sequence. Finally, the generated Laban symbols are combined into a Laban dance score according to the time sequence.

[0091] The technical solution of the present invention includes body movement feature extraction based on part-specific attention modules, and dance posture temporal feature learning based on Transformer encoder-decoder architecture. The attention module in the first stage can be replaced by other neural network modules, such as convolutional neural network, graph neural network, etc., and the encoder-decoder architecture in the second stage can be replaced by other temporal modeling architectures similar to Transformer architecture, such as Mamba, RNN architecture, etc.

Claims

1. A method for automatically generating Laban dance notation based on a part-specific attention Transformer model, characterized in that: The following steps are involved: S1, preprocessing the motion data, including coordinate transformation and normalization; For each frame of data, a part-specific attention module is used to extract spatial features by body part through the attention mechanism, and the body is divided into five parts; the part-specific attention module learns the feature correlation between different limb parts and learns the flexible limb movement features in dance; then, the spatial features of different parts are aggregated together to form the overall features of the current frame; the model learns how to correctly extract spatial features based on the body shape differences of different dancers; S2, the part-specific attention module outputs a feature sequence {g1,g2,...,g L }, after position encoding, it is input into the encoder-decoder; the encoder-decoder architecture is used to learn the dynamic characteristics of body movement in the dance along the time dimension; S3, the decoder outputs the sequence {z1,z2,...,z L }, mapped into Laban symbols; the generated Laban symbols are arranged in chronological order to obtain the Laban dance score corresponding to the target dance.

2. The method for automatically generating Laban dance notation based on the position-specific attention Transformer model according to claim 1, characterized in that: The specific method of S1 is: The relevant joints corresponding to the body parts are divided into a group, and each body part contains several joints; a body part is defined as P, and a frame of motion capture data is decomposed into a group of body parts {P1, P2, ..., P n Considering the structure of body skeleton data, we set n = 5; each body part consists of several joints, which can be regarded as a joint sequence {j1, j2, ..., j m }, the number of joints of all body parts is uniformly recorded as m, where m is the sequence length; For each body part P, spatial features are extracted from the original joint features through a multi-head attention mechanism; The self-attention mechanism has three inputs, represented by Q, K and V respectively; the feature matrix of the input sequence is represented as S = {s1, s2, ..., s m }, where m represents the length of the sequence, and the three inputs of the self-attention module are obtained through three weight matrices: The dimension of S is m×d model , W Q The dimension is d model ×d Q , W K The dimension is d model ×d K , W V The dimension is d model ×d V ; Set d Q =d K =d V =d model / n head , where n head is the number of heads of multi-head attention; the attention weight a self By inputting the dot product of Q, K and the scalar And the normalization of the softmax function is obtained as shown below: The superscript self represents a self The self-attention weight, T represents the transposition operation, a self The dimension is m×d V ; Each time state of the input sequence is represented by a weighted value; Apply the mask to the dot product calculation and repeat the attention calculation n head times, and concatenate the results to form a multi-head attention output: Where W O is the added weight dimension, implemented by the fully connected feed-forward layer, and its dimension is a mhead is the multi-head self-attention output, whose dimension is m×d model ; For body parts P i (i∈{1,2,....,n}) feature extraction, the length of the input sequence is the number of joints m, P i The feature dimension is m×d model ; Finally, the features of all body parts are connected to form a n×m×d model All frames are arranged in time order to obtain a series of spatial features {g1,g2,...,g t ,...,g L }, where g t represents the frame feature, and L is the length of the frame sequence.

3. The method for automatically generating Laban dance notation based on the position-specific attention Transformer model according to claim 1, characterized in that: The specific method of S2 is: Use the Transformer encoder-decoder architecture to learn feature sequences and convert them into corresponding Laban symbol sequences; The spatial feature sequence output by the part-specific attention module is {g1,g2,...,g t ,...,g L }, as the input of the subsequent module; first, for each input g t , a linear embedding layer is used to reshape the feature channels to obtain a deeper feature representation: x t =W g g t , (4) where x t represents the updated feature, whose dimension is d model , W g Represents a weight matrix whose dimension is d model ×d frame ; Then, in order for the model to understand the relative time position of each time series state, the position information of the sequence data is encoded into the input data; the position encoding function is defined as: Where i represents d model The i-th dimension of t is the time step, that is, the position in the sequence; the position encoding vector is defined as p t =PE(t); Thus, the position encoding process is expressed as: y t =x t +p t , (6) where y t is the position encoding data for each time step.

4. The method for automatically generating Laban dance notation based on the position-specific attention Transformer model according to claim 1, characterized in that: The specific methods of S3 are: After encoding the input sequence with position information, the processed sequence is fed into the encoder-decoder architecture for temporal modeling and target sequence generation; by repeating and stacking encoders and decoders, a deep network is constructed to learn the temporal characteristics of motion feature sequences; First, the encoder takes the entire input sequence and learns a more compact description. The decoding process is described as: where Y = {y1,y2,...,y t } is the encoded position sequence, E is the encoded output; The decoding process is then described as: Z=f decode (E,{<BOS>}), (8) Where Z represents the decoded output, and <BOS> represents the beginning of the sequence. Using the greedy strategy to output the symbol with the highest probability, the decoder outputs the Laban symbol sequence, which is represented as Z = {z1, z2, ..., z t ,...,z L' }; where z t represents the Laban symbol at time t, and L' represents the length of the output sequence; Finally, the generated Laban symbols are organized into Laban dance notation according to the time sequence.

Images