A two-person motion prediction method based on multi-scale strategy and interaction feature extraction

Abstract

The application provides a two-person motion prediction method based on a multi-scale strategy and interactive feature extraction, comprising: acquiring a two-person action video segment; dividing the two-person motion sequence into a training set and a test set; aggregating human body skeleton joints to obtain a joint coordinate sequence; segmenting the two-person motion sequence of the training set and the test set to obtain a final two-person motion sequence and a corresponding action true value; constructing a two-person motion prediction model, and importing the final two-person motion sequence into the two-person motion prediction model to iteratively generate an iterated two-person motion prediction model. Through the above technical solution, the performance and accuracy of motion prediction are improved, and the difficulty of prediction is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and machine learning, and particularly relates to a two-person motion prediction method based on a multi-scale strategy and interactive feature extraction. BACKGROUND

[0002] With the development of 5G applications, the growth of multimedia data, especially image and video data, is showing an explosive trend. For example, there are more than 30 million surveillance cameras in China, which can generate 60 EB of video data every month. For another example, in recent years, network live streaming has become increasingly popular, and software such as WeChat and TikTok can generate a large amount of video data in real time during live streaming. The rapid growth of video quantity brings a new problem: how to make computers understand the content of videos. Understanding the content of videos includes methods such as action recognition, motion prediction, and temporal action localization. Human motion prediction has broad development potential in human-computer interaction, autonomous driving, and sports guidance, and is an important part of analyzing the behavior of characters in videos.

[0003] Human motion prediction is a typical spatiotemporal prediction task, which mainly studies the characteristics of high dimensionality of motion itself, high correlation between joints, and strong time sequence, with the purpose of constructing a model with low dimensionality, rich temporal and spatial relationships, and predicting the future motion of the human body with longer time and higher accuracy. With the development of artificial intelligence and deep learning technology, most of the current models use a data-driven method to predict the future motion of the human body based on skeletal joint data. In a natural scene, there are always complex interactions between characters. When predicting the motion of two people, if the cooperative relationship between the two people can be captured, and the motion prediction of the current character is assisted by the motion pattern of the two people and the historical motion information of the other person, the error of two-person motion prediction can be effectively reduced.

[0004] Existing two-person motion prediction methods mainly fall into two categories. The method based on recurrent neural network (RNN) recursively predicts the future motion of two people, which has the problem of error accumulation, and lacks modeling of spatial relationships. The method based on graph convolutional network (GCN) can effectively preserve the natural connection relationship of human skeletons in space, but the modeling of temporal relationship is relatively simple and cannot meet the needs of two-person motion scenes. In addition, the two methods have two common problems, one is that they do not consider more fine-grained motion features, and the other is that they lack a more explicit representation of the interaction between two people.

[0005] The existing two-person motion prediction method has a coarse granularity in modeling the spatial relationship between the internal joint points of the human body and cannot explicitly represent the interaction relationship between the two persons, so that the further reduction of the prediction error is inhibited. SUMMARY

[0006] Therefore, the present application provides a two-person motion prediction method based on a multi-scale strategy and interaction feature extraction, a multi-scale graph convolutional neural network, and the combination of the interaction relationship between the two persons at different granularities and the spatiotemporal relationship of the two-person motion, so as to realize the two-person motion prediction with fine granularity, strong interpretability, and low error.

[0007] According to an aspect of the present application, a two-person motion prediction method based on a multi-scale strategy and interaction feature extraction is provided, which includes the following steps: S101: acquiring a two-person motion video segment; wherein the two-person motion video segment includes an original two-person motion sequence carrying human skeleton joint point annotation information; S102: dividing the two-person motion sequence into a training set and a test set; S103: aggregating the human skeleton joint points to obtain a joint point coordinate sequence; S104: segmenting the two-person motion sequence of the training set and the test set to obtain a final two-person motion sequence and corresponding action true value; S105: constructing a two-person motion prediction model and importing the final two-person motion sequence into the two-person motion prediction model to iteratively generate an iterated two-person motion prediction model.

[0008] Further, S103 further includes: S1031: defining the human skeleton joint points as n scales, and the i-th human skeleton joint point coordinate of the n-th scale is wherein J n n represents the number of human skeleton joint points at the n-th scale, k is an integer not less than 0, and n is an integer not less than 0; S1032: processing the two-person motion sequence in the training set and the test set to obtain a joint point coordinate sequence at n different scales.

[0009] Further, S104 further includes: copying the two-person pose T times to obtain a final two-person motion sequence, and the expression is as follows: wherein s1 represents a different scale, p1 represents an action performer, N represents the Nth historical action, and T represents the number of action frames to be predicted, represents the motion sequence of the p1 performer at the s1 scale, represents the motion sequence of the p1 performer from the 1st frame to the Nth frame, represents the T times repeated Nth frame of the character action.

[0010] Further, the S105 further includes: S1051: obtaining a double-person action feature based on the final double-person motion sequence, and the expression is as follows:

[0011] wherein, PreGCN(.) is a spatial graph convolution network, pre-encoding the double-person motion sequence to convert it into a D-dimensional embedding vector, s1 represents one different scale, p1 represents one performer of the action, T represents the number of action frames to be predicted, D represents the dimension, J s1 represents the number of human skeleton joints at the s1 th scale; S1052: constructing a multi-scale feature aggregation module to convert the double-person action feature from fine granularity to coarse granularity to obtain a coarse-granularity double-person action feature; S1053: constructing a multi-scale feature separation module to convert the double-person action feature from coarse granularity to fine granularity to obtain a fine-granularity double-person action feature; S1054: constructing a motion prediction process at each scale to splice, fuse, and decode the coarse-granularity double-person action feature and the fine-granularity double-person action feature to obtain a prediction result; S1055: comparing the prediction result with the original double-person motion sequence to train with an average joint position error as a loss function, and the expression of the loss function is as follows:

[0012] wherein, represents the coordinate position predicted by the j th joint of the p th person in the t th frame, x p，t，j represents the real coordinate position of the corresponding joint of the corresponding person, and J represents the number of joints, (N+T) represents the number of predicted frames, and p1 and p2 represent two performers of the action.

[0013] Further, S1052 further includes: the multi-scale feature aggregation module includes six interaction graph convolution modules and one down-sampling module; the interaction graph convolution module includes a spatial graph convolution module, an interaction feature extraction module, and a time graph convolution module; wherein, the spatial graph convolution module is used to extract the spatial relationship between the joints inside the human body, the interaction feature extraction module is used to extract the interaction relationship between the two persons at the current scale, and the time graph convolution module is used to extract the time relationship between the front and back frames of the double-person motion sequence; the down-sampling module includes a joint aggregation module and a feature dimension increasing module.

[0014] Further, S1053 further comprises: the multi-scale feature separation module comprises six interactive graph convolution modules and one up-sampling module; the interactive graph convolution module comprises a spatial graph convolution module, an interactive feature extraction module, and a temporal graph convolution module; wherein the spatial graph convolution module is used to extract the spatial relationship between the internal joint points of the human body, the interactive feature extraction module is used to extract the interaction relationship between the two people at the current scale, and the temporal graph convolution module is used to extract the time relationship between the front and rear frames of the two people motion sequence; and the down-sampling module comprises a joint point separation module and a feature dimension reduction module.

[0015] Further, S1055 further comprises: determining the error value of the prediction result and the original two-person motion sequence, if there is no performance gain, the model training is ended, the model weight file is saved, otherwise, returning to execute S104.

[0016] Further, the method further comprises: S106: obtaining a video segment to be detected, obtaining a skeleton joint position sequence of two-person action from the video segment, inputting the iterative two-person motion prediction model, and performing prediction of two-person action.

[0017] The present application fully considers the characteristics that two-person motion prediction needs to capture more fine-grained spatial relationships, captures the cooperative relationship between the internal joint points and body parts of two people from coarse to fine at different scales, can effectively utilize the spatial relationship contained in the two-person motion sequence, improve the performance of motion prediction, and the correlation degree between the two-person skeleton key points or the two-person body parts under the current action, can effectively utilize the cooperative relationship contained in the two-person interactive action, provide auxiliary information for subsequent motion prediction, reduce the difficulty of prediction, and improve the accuracy of prediction.

[0018] The above description is only a summary of the technical scheme of the present application, in order to more clearly understand the technical means of the present application, and can be implemented according to the content of the specification, the following detailed description of the preferred embodiments of the present application and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings which form a part of the present application are used to provide further understanding of the present application, and the illustrative embodiments of the present application and their description are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 The method flowchart of the present application is shown;

[0021] Figure 2 Another method flowchart of the present application is shown;

[0022] Figure 3A framework diagram of the multi-scale feature aggregation module is shown;

[0023] Figure 4 A framework diagram of the multi-scale feature separation module is shown. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] First, combine Figure 1 The process of this invention is explained. For example... Figure 1 As shown, a method for predicting two-person motion based on multi-scale strategy and interactive feature extraction includes: S101: acquiring video clips of two-person actions; wherein, the video clips of two-person actions include an original two-person motion sequence carrying human skeletal joint point annotation information; S102: dividing the two-person motion sequence into a training set and a test set; S103: aggregating the human skeletal joint points to obtain a joint point coordinate sequence; S104: segmenting the two-person motion sequences in the training set and the test set to obtain a final two-person motion sequence and corresponding ground truth values; S105: constructing a two-person motion prediction model, and importing the final two-person motion sequence into the two-person motion prediction model, iteratively generating an iteratively updated two-person motion prediction model.

[0026] The above methods will be discussed in detail below.

[0027] Example 1

[0028] S101: Obtain a video clip of two-person actions;

[0029] In one embodiment, the two-person motion video clip includes an original two-person motion sequence carrying human skeletal joint annotation information.

[0030] S102: Divide the two-person motion sequence into a training set and a test set;

[0031] S103: Aggregate the joints of the human skeleton to obtain a joint coordinate sequence;

[0032] In one embodiment, S1031: The human skeletal joints are defined as n scales, and the coordinates of the i-th human skeletal joint at the n-th scale are... Among them, J nrepresents the number of human skeleton joints at the nth scale, k is an integer not less than 0, and n is an integer not less than 0; S1032: processing the double-person motion sequence in the training set and the test set to obtain double-person motion sequences at n different scales.

[0033] S104: segmenting the double-person motion sequence of the training set and the test set to obtain a final double-person motion sequence and a corresponding action true value;

[0034] In one embodiment, the double-person pose is copied T times to obtain a final double-person motion sequence, and the expression is as follows: wherein s1 represents one different scale, p1 represents one performer of an action, N represents the Nth historical action, and T represents the number of action frames to be predicted, represents the motion sequence of the p1 performer at the s1 scale, represents the motion sequence of the p1 performer from the 1st frame to the Nth frame, represents the Nth action of the character repeated T times.

[0035] S105: constructing a double-person motion prediction model, and importing the final double-person motion sequence into the double-person motion prediction model to iteratively generate an iterated double-person motion prediction model;

[0036] In one embodiment, S1051: obtaining a double-person action feature based on the final double-person motion sequence, and the expression is as follows: wherein PreGCN(.) is a spatial graph convolution network, which pre-encodes the double-person motion sequence to convert it into a D-dimensional embedding vector, s1 represents one different scale, p1 represents one performer of an action, T represents the number of action frames to be predicted, D represents the dimension, and J s1 represents the number of human skeleton joints at the s1 scale; S1052: constructing a multi-scale feature aggregation module to convert the double-person action feature from a fine granularity to a coarse granularity to obtain a coarse-granularity double-person action feature; S1053: constructing a multi-scale feature separation module to convert the double-person action feature from a coarse granularity to a fine granularity to obtain a fine-granularity double-person action feature; S1054: constructing a motion prediction process at each scale to splice, fuse, and decode the coarse-granularity double-person action feature and the fine-granularity double-person action feature to obtain a prediction result; S1055: comparing the prediction result with the original double-person motion sequence to train the prediction result by taking the average joint position error as a loss function, and the expression of the loss function is as follows:

[0037] wherein, represents the coordinate position predicted by the jth joint of the pth person at the tth frame, x p，t，jrepresents the real coordinate position of the corresponding joint of the corresponding person, J represents the number of joints, (N+T) represents the number of predicted frames, and p1 and p2 represent two performers of the action.

[0038] In one embodiment, as shown in FIG. 5, the multi-scale feature aggregation module includes six interactive graph convolution modules and one down-sampling module; the interactive graph convolution module includes a spatial graph convolution module, an interactive feature extraction module, and a temporal graph convolution module; wherein the spatial graph convolution module is used to extract the spatial relationship between the joints inside the human body, the interactive feature extraction module is used to extract the interaction relationship between the two persons at the current scale, and the temporal graph convolution module is used to extract the temporal relationship between the frames before and after the motion sequence of the two persons; the down-sampling module includes a joint aggregation module and a feature dimension increasing module. Figure 3 In one embodiment, as shown in FIG. 5, the multi-scale feature aggregation module includes six interactive graph convolution modules and one down-sampling module; the interactive graph convolution module includes a spatial graph convolution module, an interactive feature extraction module, and a temporal graph convolution module; wherein the spatial graph convolution module is used to extract the spatial relationship between the joints inside the human body, the interactive feature extraction module is used to extract the interaction relationship between the two persons at the current scale, and the temporal graph convolution module is used to extract the temporal relationship between the frames before and after the motion sequence of the two persons; the down-sampling module includes a joint aggregation module and a feature dimension increasing module.

[0039] Figure 4 In one embodiment, as shown in FIG. 5, the multi-scale feature aggregation module includes six interactive graph convolution modules and one down-sampling module; the interactive graph convolution module includes a spatial graph convolution module, an interactive feature extraction module, and a temporal graph convolution module; wherein the spatial graph convolution module is used to extract the spatial relationship between the joints inside the human body, the interactive feature extraction module is used to extract the interaction relationship between the two persons at the current scale, and the temporal graph convolution module is used to extract the temporal relationship between the frames before and after the motion sequence of the two persons; the down-sampling module includes a joint aggregation module and a feature dimension increasing module.

[0040] In one embodiment, S1055 further includes: determining the error value of the prediction result and the original two-person motion sequence, if there is no performance gain, the model training is ended, the model weight file is saved, otherwise, returning to execute S104.

[0041] S106: obtaining a video segment to be detected, obtaining a skeleton joint position sequence of a two-person action from the video segment, inputting the iterative two-person motion prediction model, and predicting the two-person action.

[0042] Embodiment two

[0043] S201: obtaining a two-person action video segment, the two-person action video segment containing an original two-person motion sequence of 3D coordinate information of a human body skeleton joint in each frame image;

[0044] S202: dividing the original two-person motion sequence into a training set and a test set;

[0045] In some embodiments, the number and meaning of the human body skeleton joints can be defined according to the requirements of the training set and the test set.

[0046] ​S203, defining the graph structure of the human skeleton joints at different scales, combining the original double-person motion sequence in step S201, and obtaining the joint coordinate sequence at multiple scales by aggregating the joints through calculating the average position of the joints;

[0047] In some embodiments, S203 further comprises:

[0048] S2031, defining the human skeleton joints as n scales, and defining the i-th joint coordinate at the n-th scale as wherein J n represents the number of joints at the n-th scale, and the i-th joint at the n-th scale is calculated by the j1 to j k k joints at the n-1-th scale.

[0049] S2032, processing the double-person motion sequence in the training set and the test set according to the method in step S2031 to obtain the double-person motion sequence at n different scales wherein p1, p2 represent the two performers of the action, and s1, s2…sn represent n different scales.

[0050] S204, segmenting the double-person motion sequence in the training set and the test set according to the time length (N+T) to obtain the final double-person motion sequence input and the corresponding action true value.

[0051] In some embodiments, in order to obtain the final double-person motion sequence, the double-person pose at the Nth frame is copied T times. Taking p1 at the s1 scale as an example, the final double-person motion sequence input is N represents the N frames of historical actions, and T represents the number of action frames to be predicted.

[0052] S205, constructing a double-person motion prediction algorithm model, setting and adjusting the algorithm parameter set, importing the final double-person motion sequence into the double-person motion prediction algorithm model, iteratively generating a double-person motion prediction model, and saving the model.

[0053] In some embodiments, S205 further comprises:

[0054] S2051, converting the double-person motion sequence from the coordinate domain to the embedding space domain to obtain the action feature:

[0055]

[0056] wherein PreGCN(.) is a spatial graph convolution network, and the double-person motion sequence is pre-encoded to convert it into a D-dimensional embedding vector.

[0057] S2052, as Figure 2As shown, the multi-scale feature aggregation module is constructed to convert the two-person action features from fine granularity to coarse granularity. The feature aggregation module is composed of 6 interaction graph convolution modules and a down-sampling module. Each interaction graph convolution module is composed of a spatial graph convolution layer, an interaction feature extraction layer and a temporal graph convolution layer. The down-sampling module is composed of a joint node aggregation and a feature dimension increase.

[0058] The spatial graph convolution layer is used to extract the spatial relationship between the internal joints of the human body:

[0059] F l+1 = σ(A s F l W s l )

[0060] wherein represents the input feature of the l-th layer, T represents the number of sequence frames, J represents the number of bone joints, D l represents the dimension of the input feature of the l-th layer, represents a trainable parameter matrix, represents a spatial adjacency matrix.

[0061] The interaction feature extraction layer is used to extract the interaction relationship between the two persons at the current scale. First, the action patterns of the two persons are extracted by a multi-layer perceptron (MLP):

[0062]

[0063]

[0064] wherein f key (·) and f value (·) are the same MLP composed of two fully connected layers, and hidden represents the dimension of the hidden layer feature. Then, the interaction feature matrix is generated according to the attention mechanism:

[0065]

[0066] wherein softmax(·) is a normalization function for normalizing the attention weight to [0, 1], A p1p2 is the interaction attention matrix between the person p1 and the person p2 when the person p1 is the prediction subject, which can represent the correlation degree of the joints between the two persons at the current scale. Finally, the action features of the two persons are updated according to the matrix:

[0067]

[0068]

[0069] in, and These represent the action features of p1 and p2 at the updated sn scale, respectively.

[0070] Temporal plot convolutional layers are used to extract the temporal relationship between consecutive frames in a two-person motion sequence:

[0071] F l+1 =σ(A t F l W t l )

[0072] Similar to spatial graph convolutional layers, Let T represent the input features of the l-th layer, T represent the number of frames, J represent the number of skeletal joints, and D represent the number of skeletal joints. l This represents the dimension of the features input to the l-th layer. W of the spatial graph convolutional layer mentioned above s l Shared parameters This represents the temporal adjacency matrix.

[0073] Step S2053, as follows Figure 4 As shown, a multi-scale feature separation module is constructed to transform the coarse-grained action features of the two individuals back into fine-grained features. This module is symmetrical to the multi-scale feature aggregation module in step S301 and consists of six interactive graph convolutional modules and one downsampling / upsampling module. Specifically, the upsampling module comprises keypoint separation and feature dimensionality reduction.

[0074] Step S2054: Construct the motion prediction process for each scale. First, the motion features generated by the multi-scale feature aggregation module and the separation module at each scale are concatenated, then fused using an MLP, and finally decoded to obtain the final prediction result.

[0075]

[0076] in, and These represent the action features output by the multi-scale feature aggregation module and the separation module at the sn scale, respectively. decodeGCN(.) is a spatial graph convolutional network that decodes the action features and transforms them back from the embedding space to the coordinate domain.

[0077] Step S2055: Compare the predicted results with the original two-person motion sequence, and train the model using the Mean Per Joint Position Error (MPJPE) as the final loss function.

[0078]

[0079] wherein, represents the predicted coordinate position of the jth joint of the pth person in the tth frame, x p，t，j represents the real coordinate position of the corresponding joint of the corresponding person, J represents the number of joints, and (N+T) represents the number of predicted frames. In addition, since there are n scales of prediction results in total, the loss values in the n scales are taken as the final loss value for back propagation.

[0080] In some embodiments, the test data set arranged in the reading step S201 is used to determine the error value of the result of the two-person motion prediction algorithm. If there is no performance gain, the model training is ended, the model weight file is saved, otherwise the step S204 is returned to execute.

[0081] S206, a motion prediction process of a two-person motion video is constructed. First, the skeleton joint position sequence of the two-person motion is obtained from the video through a pose estimation method, the two-person motion prediction model file saved in the loading step S205 is loaded, and the prediction of the two-person motion is performed.

[0082] Embodiment three

[0083] In order to more clearly understand the technical principles of the present application, the embodiments of the present application are specifically and in detail described below. In the embodiments, the two-person dance data set Extreme Pose Interaction (ExPI) is taken as an example, and the specific implementation method and steps are as follows:

[0084] S301, a two-person motion video segment is obtained, the two-person motion video segment contains 3D coordinate information of human body skeleton joints in each frame image, and the two-person motion sequence containing the above-mentioned labeled information is divided into a training data set T and a test data set U;

[0085] Specifically, in the step S301, the ExPI data set contains 16 kinds of two-person motions performed by 2 pairs of dancers in total, each observation point contains 115 two-person motion sequences and 30k frames of images, and 60k labeled 3D body poses and character models. The training data set T contains 120512 two-person motion sequences, and the test data set U contains 163840 two-person motion sequences.

[0086] S302, a graph structure of human body skeleton joints in different scales is defined, the original two-person motion sequence in the step S301 is combined, the joints are aggregated by calculating the average position of the joints, and the joint coordinate sequence in multiple scales is obtained;

[0087] In some embodiments, in step S301, the human skeleton joint is divided into n=4 scales. Each person in the ExPI dataset corresponds to 18 joints, and each joint aggregation mode from fine to coarse is defined. Finally, the number of joints at each scale is 18, 14, 9, and 4 respectively.

[0088] S303, the two-person motion sequence in the training set and the test set is segmented according to the time length (N+T) to obtain the two-person motion sequence input and the corresponding action true value.

[0089] In some embodiments, in step S303, N represents N frames of historical actions, which is 10 here, and T represents the number of action frames to be predicted, which is 25 here. In order to obtain the two-person motion sequence input mentioned in step S2, the two-person pose of the Nth frame is copied T times. Taking p1 at s1 scale as an example, the final two-person motion sequence is That is, each two-person motion sequence in the training set and the test set has 35 frames;

[0090] Step S304, constructing a two-person motion prediction algorithm model, setting and adjusting the algorithm parameter set, importing the final two-person motion sequence obtained in step S303 into the two-person motion prediction algorithm model, iterating the two-person motion prediction model and saving.

[0091] In some embodiments, step S304 further comprises:

[0092] Step S3041, converting the two-person motion sequence from the coordinate domain to the embedding space domain to obtain the action feature (the feature in step S304 is taken as an example at s1 scale):

[0093]

[0094] Where PreGCN(.) is a spatial graph convolution network, which pre-encodes the two-person motion sequence and converts it into a D-dimensional embedding vector, which is 64 here.

[0095] Step S3042, as shown in Figure 3 , a multi-scale feature aggregation module is constructed to convert the two-person action feature from fine to coarse. The feature aggregation module is composed of 6 interactive graph convolution modules and a down-sampling module. Each interactive graph convolution module is composed of a spatial graph convolution layer, an interactive feature extraction layer and a time graph convolution layer. The down-sampling module is composed of joint aggregation and feature dimension lifting. In the process of joint aggregation, in order not to lose the expression ability of the action feature, the feature dimension is lifted, and the feature dimension at 4 scales is 64, 128, 256 and 512 respectively.

[0096] The spatial graph convolution layer is used to extract the spatial relationship between the joints inside the human body:

[0097] F l+1 = σ(A s F l W s l )

[0098] wherein denotes the input feature of the l-th layer, T denotes the number of sequence frames, J denotes the number of skeleton joints, D l denotes the dimension of the input feature of the l-th layer, denotes the trainable parameter matrix, denotes the spatial adjacency matrix.

[0099] The interaction feature extraction layer is used to extract the interaction relationship between the two people at the current scale. First, the action patterns of the two people are extracted through the MLP:

[0100]

[0101]

[0102] wherein f key (·) and f value (·) are the same MLP, which consists of two fully connected layers, hidden denotes the dimension of the hidden layer feature, which is kept the same as the dimension of the action feature at the current scale. Then the interaction feature matrix is generated according to the attention mechanism:

[0103]

[0104] wherein softmax(·) is a normalization function, which normalizes the attention weight to [0, 1], A p1p2 is the interaction attention matrix between the person p1 and the person p2 when the person p1 is the prediction subject, which can represent the correlation degree between the two people at the current scale. Finally, the action features of the two people are updated according to this matrix:

[0105]

[0106]

[0107] wherein and denote the updated action features of p1 and p2 at the s1 scale, respectively.

[0108] The temporal graph convolution layer is used to extract the temporal relationship between the frames before and after the motion sequence of the two people:

[0109] F l+1 = σ(A t F lW t l )

[0110] and spatial graph convolutional layer, denotes the input feature of the l-th layer, T denotes the number of frames, J denotes the number of skeleton joints, D l denotes the dimension of the input feature of the l-th layer, W s l share parameters, denotes the time adjacency matrix.

[0111] Step S3043, as shown in the figure, a multi-scale feature separation module is constructed to convert the two-person action features from coarse granularity to fine granularity. The multi-scale feature separation module is symmetrical to the multi-scale feature aggregation module in step S3042 and is composed of 6 interaction graph convolutional modules and a down-up sampling module. In particular, the up-sampling module is composed of joint separation and feature dimension reduction. Figure 4

[0112] Step S3044, a motion prediction process under each scale is constructed. First, the action features generated by the multi-scale feature aggregation module and the separation module under each scale are spliced, then fused by MLP, and finally the final prediction result is decoded:

[0113]

[0114] wherein, and denote the action features output by the multi-scale feature aggregation module and the separation module under the s1 scale respectively, decodeGCN(.) is a spatial graph convolutional network that decodes the action features to convert back from the embedding space to the coordinate domain.

[0115] Step S3045, the prediction result is compared with the original two-person motion sequence, and the MPJPE is used as the final loss function for model training:

[0116]

[0117] wherein, denotes the predicted coordinate position of the j-th joint of the p-th person in the t-th frame, x p，t，j denotes the real coordinate position of the corresponding joint of the corresponding person, J denotes the number of joints, and (N+T) denotes the number of predicted frames. In addition, since there are prediction results under 4 scales in total, the loss under 4 scales is summed as the final loss value for back propagation.

[0118] ​In some embodiments, in step S304, the test data set sorted in step S301 is read to determine the error value of the result of the two-person motion prediction algorithm, if there is no performance gain, the model training is ended, the model weight file is saved, otherwise, step S303 is returned to execute.

[0119] In step S305, a two-person motion video motion prediction process is constructed. First, the skeleton joint position sequence of the two-person motion is obtained from the video by a pose estimation method, the two-person motion prediction model file saved in step S304 is loaded, and the prediction of the two-person motion is performed.

[0120] According to the method provided by the present application, at least the following technical effects are achieved: 1) The present application fully considers the characteristics of two-person motion prediction that needs to capture more fine-grained spatial relationships, and proposes a multi-scale motion feature encoding structure based on a graph convolution network, which is divided into a multi-scale feature encoding module and a multi-scale feature decoding module, and captures the cooperative relationship between the internal joints and body parts of two persons from coarse granularity to fine granularity at different scales. The present application can effectively utilize the spatial relationship contained in the two-person motion sequence and improve the performance of motion prediction. 2) The present application proposes a more explicit extraction method of two-person interaction relationship. The interaction feature matrix can be obtained by judging the current action mode at different scales, which can represent the correlation degree between the skeleton key points of two persons or the body parts of two persons under the current action. Then, the motion features of two persons are updated according to the interaction feature matrix. The present application can effectively utilize the cooperative relationship contained in the two-person interactive action, provide auxiliary information for subsequent motion prediction, reduce the difficulty of model prediction, and improve the prediction accuracy.

[0121] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0122] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification of the above embodiment according to the technical essence of the present application still belongs to the scope of the technical solution of the present application.

Claims

1. A two-person motion prediction method based on multi-scale strategy and interaction feature extraction, characterized in that, The method comprises: S101: acquiring a two-person action video segment; wherein the two-person action video segment comprises an original two-person motion sequence carrying human body skeleton joint annotation information; S102: dividing the two-person motion sequence into a training set and a test set; S103: aggregating the human body skeleton joints to obtain a joint coordinate sequence; S104: segmenting the two-person motion sequences of the training set and the test set to obtain a final two-person motion sequence and a corresponding action true value; S105: constructing a two-person motion prediction model, and importing the final two-person motion sequence into the two-person motion prediction model to iteratively generate an iterated two-person motion prediction model; The S105 further comprises: S1051: obtaining two-person action features based on the final two-person motion sequence, and the expression is as follows: where PreGCN(.) is a spatial graph convolution network that pre-encodes the two-person motion sequence into a D-dimensional embedding vector, s1 denotes a different scale, p1 denotes a performer of the action, T denotes the number of action frames to be predicted, D denotes the dimension, J s1 denotes the number of human skeleton joints at the s1th scale; S1052: constructing a multi-scale feature aggregation module, converting the two-person action features from fine granularity to coarse granularity to obtain coarse-grained two-person action features; S1053: constructing a multi-scale feature separation module, converting the two-person action features from coarse granularity to fine granularity to obtain fine-grained two-person action features; S1054: constructing a motion prediction process at each scale, splicing, fusing, and decoding the coarse-grained two-person action features and the fine-grained two-person action features to obtain a prediction result; S1055: comparing the prediction result with the original two-person motion sequence, and training with an average joint position error as a loss function, and the expression of the loss function is as follows: wherein, represents the predicted coordinate position of the jth joint of the pth person in the tth frame, x p，t，j represents the real coordinate position of the corresponding joint of the corresponding person, J represents the number of joints, (N+T) represents the number of predicted frames, and p1 and p2 represent two performers of the action.

2. The method of claim 1, wherein, The S103 further comprises: S1031: define the human body skeletal joint as n scales, and the i-th human body skeletal joint coordinate of the n-th scale is wherein Jn represents the number of human body skeletal joints under the n-th scale, k is an integer not less than 0, and n is an integer not less than 0. S1032: processing the two-person motion sequences in the training set and the test set to obtain the joint coordinate sequences at n different scales.

3. The method of claim 1, wherein, The S104 further comprises: copying the two-person pose T times to obtain a final two-person motion sequence, and the expression is as follows: wherein s1 represents one different scale, p1 represents one performer of the motion, N represents the Nth frame of historical motion, and T represents the number of frames of motion to be predicted, represents the motion sequence of p1 performer at s1 scale, represents the motion sequence of p1 performer from the 1st frame to the Nth frame, represents the Nth frame of human motion repeated T times.

4. The method of claim 1, wherein, The S1052 further comprises: The multi-scale feature aggregation module comprises six interactive graph convolution modules and a down-sampling module; The interactive graph convolution module comprises a spatial graph convolution module, an interactive feature extraction module, and a temporal graph convolution module; wherein the spatial graph convolution module is used to extract the spatial relationship between the internal joints of the human body, the interactive feature extraction module is used to extract the interaction relationship between the two persons at the current scale, and the temporal graph convolution module is used to extract the time relationship between the front and rear frames of the two-person motion sequence; The down-sampling module comprises a joint aggregation module and a feature dimension increasing module.

5. The method of claim 1, wherein, The S1053 further comprises: The multi-scale feature separation module comprises six interactive graph convolution modules and an up-sampling module; The interactive graph convolution module comprises a spatial graph convolution module, an interactive feature extraction module, and a temporal graph convolution module; wherein the spatial graph convolution module is used to extract the spatial relationship between the internal joints of the human body, the interactive feature extraction module is used to extract the interaction relationship between the two persons at the current scale, and the temporal graph convolution module is used to extract the time relationship between the front and rear frames of the two-person motion sequence; The up-sampling module comprises a joint separation module and a feature dimension reduction module.

6. The method of claim 1, wherein, The S1055 further comprises: The prediction result is compared with the original double-person motion sequence to determine an error value. If there is no performance gain, the model training is ended, a model weight file is saved, and otherwise, the execution of S104 is returned.

7. The method of claim 1, wherein, The method further includes: S106: obtaining a video segment to be detected, obtaining a sequence of bone joint positions of double-person motion from the video segment, inputting the iterated double-person motion prediction model, and performing prediction of double-person motion.

Images