Dance posture prediction method based on efficient channel attention spatio-temporal graph convolution network

Abstract

The application discloses a dance posture prediction method based on an efficient channel attention spatio-temporal graph convolution network, and comprises the following steps: S100, establishing a dance dataset; S200, preprocessing the dance dataset; S300, extracting two-dimensional dance posture key point coordinate data of a dancer in the preprocessed dance dataset to form a two-dimensional dance posture key point coordinate dataset; S400, constructing an efficient channel attention spatio-temporal graph convolution network; S500, training the efficient channel attention spatio-temporal graph convolution network; and S600, predicting a dance posture action sequence according to the trained efficient channel attention spatio-temporal graph convolution network. The method has the advantages of high dance posture prediction accuracy, strong generalization ability and low cost, and can be applied to the technical fields of dance posture prediction and automatic dance generation.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of computer vision, artificial intelligence and neural network, and particularly relates to a dance posture prediction method based on an efficient channel attention spatio-temporal graph convolution network. BACKGROUND

[0002] In recent years, automatic generation of dance has become an important research hotspot in the field of computer vision and artificial intelligence. In the film and game industry, the application demand of dance is becoming larger and larger. However, artificial dance creation not only has high cost, but also consumes a long time and has low creation efficiency, which is not conducive to industrial development. Dance posture prediction refers to predicting the posture action of a dancer in the next period of time according to the posture actions of the dancer in adjacent frames in a period of time in a dance video, thereby increasing and enriching the dance posture action, which has certain enlightenment and promoting effect on automatic creation of dance.

[0003] Common methods of dance posture prediction include a method of combining sensors such as inertial measurement units and depth cameras with traditional optimization algorithms and a deep learning method such as a long short-term memory network and a temporal convolutional neural network. Traditional optimization algorithms have the disadvantages of low prediction accuracy, the need for manual feature extraction and high cost.

[0004] A spatio-temporal graph convolution network model is a graph neural network model for processing time series data. By constructing a spatio-temporal graph of time series data, convolution operations are performed simultaneously in time and space to extract data features. In dance posture prediction, the spatio-temporal graph convolution network model models dance posture key point coordinate data as a spatio-temporal graph, then uses spatio-temporal graph convolution operations to extract spatio-temporal features of dance posture actions, and finally realizes prediction of the posture action of a dancer. SUMMARY

[0005] In view of this, the present disclosure provides a dance posture prediction method based on an efficient channel attention spatio-temporal graph convolution network, comprising the following steps:

[0006] S100: establishing a dance data set;

[0007] S200: preprocessing the dance data set;

[0008] S300: extracting two-dimensional dance posture key point coordinate data of a dance performer in the preprocessed dance data set to form a two-dimensional dance posture key point coordinate data set;

[0009] S400: constructing an efficient channel attention spatio-temporal graph convolution network;

[0010] S500: training the efficient channel attention spatio-temporal graph convolution network;

[0011] S600: predicting a dance posture action sequence according to the trained efficient channel attention spatio-temporal graph convolution network.

[0012] By the above technical solution, the efficient channel attention spatio-temporal graph convolution network is constructed, the dance posture key point coordinate data is modeled as a spatio-temporal graph, a time sequence pooling layer is used to reduce the dimension of the data, the training speed of the network is improved, the features of the dance posture key points can be effectively extracted, and the accuracy of the dance posture action prediction is improved. The method has the advantages of high dance posture prediction accuracy, strong generalization ability, low cost, etc., and can be applied to the technical fields of dance posture prediction and automatic generation of dance, etc. BRIEF DESCRIPTION OF DRAWINGS

[0013] Fig. 1 is a dance posture prediction method flowchart based on an efficient channel attention spatio-temporal graph convolution network provided in an embodiment of the present disclosure;

[0014] Fig. 2 is a dance posture key point division schematic diagram in an embodiment of the present disclosure;

[0015] Fig. 3 is an efficient channel attention spatio-temporal graph convolution network structure schematic diagram in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] To further describe the present disclosure, the following will combine the drawings of the present disclosure with the specific embodiments of the present disclosure. Figs. 1 to 3 Further description will be made.

[0017] Referring to Fig. 1 In an embodiment, a dance posture prediction method based on an efficient channel attention spatio-temporal graph convolution network is disclosed, including the following steps:

[0018] S100: establishing a dance data set;

[0019] S200: preprocessing the dance data set;

[0020] S300: extracting two-dimensional dance posture key point coordinate data of a dance performer in the preprocessed dance data set to form a two-dimensional dance posture key point coordinate data set;

[0021] S400: constructing an efficient channel attention spatio-temporal graph convolution network;

[0022] S500: training the efficient channel attention spatio-temporal graph convolution network;

[0023] S600: predicting a dance posture action sequence according to the trained efficient channel attention spatio-temporal graph convolution network.

[0024] In this embodiment, the dance posture prediction method is composed of the steps of establishing a dance dataset, preprocessing the dance dataset, extracting a dance posture key point coordinate dataset, constructing an efficient channel attention spatio-temporal graph convolution network, training the efficient channel attention spatio-temporal graph convolution network, and predicting a dance posture action sequence. The dance posture prediction method can effectively extract the features of dance posture key points, and the predicted dance posture sequence is close to the real dance posture sequence, has the advantages of high prediction accuracy and strong generalization ability, and can predict different types of dance postures.

[0025] In another embodiment, step S100 further comprises using a camera to record different types of dance videos performed by dance performers on site or collecting and downloading different types of dance videos from a public dance video website to establish the dance dataset.

[0026] For this embodiment, different types of dance videos performed by dance performers are recorded on site using a camera or different types of dance videos are collected and downloaded from a public dance video website to establish the dance dataset D as follows:

[0027]

[0028] where i is the number of dance types in the dance dataset, j is the number of the same type of dance video, N is a positive integer, and represents the number of dance types.

[0029] In another embodiment, step S200 further comprises segmenting all dance videos in the dance dataset for 10-30 seconds and normalizing the frame rate of the dance videos to 24 frames per second.

[0030] In another embodiment, step S300 further comprises:

[0031] S301: using OpenPose to extract two-dimensional dance posture key point coordinate data of dance performers in the preprocessed dance dataset frame by frame to form a two-dimensional dance posture key point coordinate dataset;

[0032] S302: storing the two-dimensional dance posture key point coordinate dataset in JSON format;

[0033] S303: dividing the two-dimensional dance posture key point coordinate dataset into a training set, a validation set, and a test set in a ratio of 6:2:2.

[0034] For this embodiment, two-dimensional posture key point coordinate data of dance performers in the dance dataset D is extracted frame by frame using OpenPose to form a two-dimensional dance posture key point coordinate dataset M as follows:

[0035] M = [P0, P1,..., PN] T]∈R N×25×2

[0036] P T is the body pose keypoint vector of the T-th frame of the dancer, and P T = [J0, J1,..., J 24 ], J i is the two-dimensional coordinate representation of the i-th node, and J i = (x i , y i ), N is a positive integer, representing the total number of dancers in each frame of the video; the two-dimensional dance pose keypoint coordinate dataset M is stored in JSON format; the two-dimensional dance pose keypoint coordinate dataset M is divided into training set M1, validation set M2 and test set M3 in the ratio of 6:2:2.

[0037] In another embodiment, the efficient channel attention spatio-temporal graph convolutional network in step S400 includes an input layer, a spatio-temporal graph convolutional layer, an efficient channel attention module, a temporal pooling layer, a fully connected layer and an output layer.

[0038] For this embodiment, the input layer of this embodiment is a spatio-temporal graph composed of dance pose keypoints, and the dance pose keypoint division method is described in Fig. 2 ; in Fig. 3 , the efficient channel attention spatio-temporal graph convolutional network of this embodiment is composed of an input layer 1, a spatio-temporal graph convolutional layer 2, an efficient channel attention module 3, a temporal pooling layer 4, a fully connected layer 5, and an output layer 6. The feature map dimension of the input layer 1 is (2, 25, 12, 8), the convolution kernel size of the spatio-temporal graph convolutional layer 2 is 1x1, the step is 1, the efficient channel attention module 3 is composed of an average pooling layer, a one-dimensional convolution and a Sigmoid activation function, the temporal pooling layer 4 uses average pooling, and the feature map dimension of the output layer 6 is (2, 25, 8).

[0039] The output of the spatio-temporal graph convolutional layer can be represented as H l+1 , where l represents the layer number, the input of the spatio-temporal graph convolutional layer at the l-th layer is H l , and the relationship between the l+1-th layer and the l-th layer can be represented as:

[0040]

[0041] where σ represents the activation function ReLU, is the adjacency matrix of the spatio-temporal graph, is the diagonal node degree matrix of , and and are the spatial convolution weight matrix and the temporal convolution weight matrix of the l-th layer, respectively. This formula is an intermediate formula for constructing the efficient channel attention spatio-temporal graph convolutional network.

[0042] The calculation of the weight ω in the efficient channel attention module is as follows:

[0043] ω = σ(C1D k (y))

[0044] where σ is an activation function, C1D k represents one-dimensional convolution with a convolution kernel size of k, and y is the aggregated features. The formula is an intermediate formula for constructing the efficient channel attention spatio-temporal graph convolutional network.

[0045] In another embodiment, the connection relationship between the network layers of the efficient channel attention spatio-temporal graph convolutional network is as follows: an input layer is sequentially connected with 3 spatio-temporal graph convolutional layers, each spatio-temporal graph convolutional layer is connected with 1 efficient channel attention module, each efficient channel attention module is connected with 1 temporal pooling layer, and at the end of the efficient channel attention spatio-temporal graph convolutional network, 1 fully connected layer and 1 output layer are connected.

[0046] In this embodiment, such connection is first to extract features from the time series data using the spatio-temporal graph convolutional layer, then to highlight important features and suppress unimportant features using the efficient channel attention module, and then to reduce the dimension using the temporal pooling layer, and finally to integrate the final features using the fully connected layer.

[0047] In another embodiment, the input layer is a spatio-temporal graph composed of two-dimensional dance posture key points.

[0048] In another embodiment, the temporal pooling layer uses average pooling.

[0049] In this embodiment, overfitting is reduced, and the model has better generalization ability.

[0050] In another embodiment, the step S500 further comprises: inputting the training set into the efficient channel attention spatio-temporal graph convolutional network for training, and in the training process, the learning rate of the efficient channel attention spatio-temporal graph convolutional network is dynamically adjusted using the Adam optimization algorithm, and the loss function is

[0051]

[0052] where Y i is the real two-dimensional dance posture key point coordinate vector value, is the two-dimensional dance posture key point coordinate vector value predicted by the model, and N represents the number of frames, which is a finite positive integer.

[0053] For this embodiment, the training set M1 is input to the high-efficiency channel attention spatio-temporal graph convolution network for training. In the training process, the learning rate of the high-efficiency channel attention spatio-temporal graph convolution network is dynamically adjusted using the Adam optimization algorithm, and the loss function is

[0054]

[0055] where Y i is the real dance pose key point coordinate vector value, is the dance pose key point coordinate vector value predicted by the model, N represents the number of frames, and N is a finite positive integer.

[0056] In this method, the training is terminated when the loss function L converges or the number of training iterations reaches 500.

[0057] In another embodiment, the method further comprises the following steps: evaluating the trained high-efficiency channel attention spatio-temporal graph convolution network. The validation set M2 is input to the trained high-efficiency channel attention spatio-temporal graph convolution network for testing, and the average distance error ADE and the final distance error FDE are introduced to evaluate the performance of the high-efficiency channel attention spatio-temporal graph convolution network.

[0058] In another embodiment, step S600 further comprises: inputting the test set to the trained high-efficiency channel attention spatio-temporal graph convolution network model for testing to predict the dance pose action sequence.

[0059] For this embodiment, the test set M3 is input to the trained high-efficiency channel attention spatio-temporal graph convolution network model for testing to predict the dance pose action sequence.

[0060] In another embodiment, 100 dance videos of 3 dances are selected from the dance data set as an example. N in the dance data set D takes the value 3. N in the two-dimensional dance pose key point coordinate data set M takes the value 1, indicating that one dancer is selected.

[0061] In another embodiment, 150 dance videos of 4 dances are selected from the dance data set as an example. N in the dance data set D takes the value 4. N in the two-dimensional dance pose key point coordinate data set M takes the value 1, indicating that one dancer is selected.

[0062] In another embodiment, 200 dance videos of 5 dances are selected from the dance data set as an example. N in the dance data set D takes the value 5. N in the two-dimensional dance pose key point coordinate data set M takes the value 1, indicating that one dancer is selected.

[0063] Although the embodiments of the present application have been described above with reference to the accompanying drawings, the present application is not limited to the above-described specific embodiments and areas of application, and the above-described specific embodiments are merely illustrative and instructive, but are not restrictive. Many modifications can be made by those skilled in the art under the teachings of the present specification and without departing from the scope of the present application as defined by the claims.

Claims

1. A dance pose prediction method based on an efficient channel attention spatiotemporal graph convolutional network, comprising the following steps: S100: Create a dance dataset; S200: Preprocess the dance dataset; S300: Extract the two-dimensional dance posture key point coordinate data of the dancers from the preprocessed dance dataset to form a two-dimensional dance posture key point coordinate dataset. S400: Constructing an efficient channel-attention spatiotemporal graph convolutional network; S500: Train the efficient channel attention spatiotemporal graph convolutional network; S600: Predict dance posture sequence based on the trained high-efficiency channel attention spatiotemporal graph convolutional network; The efficient channel attention spatiotemporal graph convolutional network described in step S400 includes an input layer, a spatiotemporal graph convolutional layer, an efficient channel attention module, a temporal pooling layer, a fully connected layer, and an output layer; The connection relationships between the network layers of the efficient channel attention spatiotemporal graph convolutional network are as follows: after the input layer, three spatiotemporal graph convolutional layers are connected in sequence; after each spatiotemporal graph convolutional layer, one efficient channel attention module is connected; after each efficient channel attention module, one temporal pooling layer is connected; and at the end of the efficient channel attention spatiotemporal graph convolutional network, one fully connected layer and one output layer are connected.

2. The method according to claim 1, step S100 further includes: Create a dance dataset by recording different types of dance performances live with cameras or by collecting and downloading different types of dance videos from publicly available dance video websites.

3. The method according to claim 1, step S200 further includes: All dance videos in the dance dataset are segmented into 10-30 second segments, and the frame rate of the dance videos is normalized to 24 frames per second.

4. The method according to claim 1, step S300 further includes: S301: Use OpenPose to extract the two-dimensional dance posture key point coordinate data of the dancers in the preprocessed dance dataset frame by frame, and form a two-dimensional dance posture key point coordinate dataset. S302: A dataset of two-dimensional dance pose keypoint coordinates stored in JSON format; S303: Divide the two-dimensional dance posture key point coordinate dataset into training set, validation set and test set in a ratio of 6:2:

2.

5. The method according to claim 1, wherein the input layer is a spatiotemporal graph composed of two-dimensional dance posture key points.

6. The method of claim 1, wherein the temporal pooling layer uses average pooling.

7. The method according to claim 1, step S500 further includes: The training set is input into the efficient channel attention spatiotemporal graph convolutional network for training. During training, the Adam optimization algorithm is used to dynamically adjust the learning rate of the efficient channel attention spatiotemporal graph convolutional network, and the loss function is... : ； in, These are the coordinate vector values ​​of key points in the actual dance posture. These are the coordinate vector values ​​of the key points of the dance posture predicted by the model. Indicates the number of frames. It is a finite positive integer.

8. The method according to claim 1, step S600 further includes: The test set is input into a well-trained, efficient channel attention spatiotemporal graph convolutional network model for testing, predicting dance posture movement sequences.

Images