Human body action recognition method, model training method, model product and equipment based on double-expansion time domain adaptive graph convolutional neural network

Abstract

The invention discloses a human motion recognition method based on a double-expansion time domain adaptive graph convolutional neural network, and the method is specifically implemented according to the following steps: firstly, designing a set of motion command set for a man-machine interaction task, carrying out the data collection, carrying out the viewpoint-independent processing through coordinate system transformation, and carrying out the data collection; the sensitivity of the action to the shooting position is reduced. In order to ensure feature extraction of local time and global time in the whole action process, convolution of different expansion factors is combined, and a double-expansion time domain convolution layer is designed. And finally, establishing a double-expansion time domain adaptive graph convolutional neural network model by combining with an adaptive space graph convolutional layer. Experimental verification is conducted on the established model through a data set, and the result shows that the overall recognition accuracy of the human body actions is 98.5%, the recognition accuracy of each type of actions reaches 97.5% or above, the recognition effect is excellent, and the requirement for man-machine interaction tasks can be met.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and action recognition, and more particularly relates to a human action recognition method based on a double extended time domain adaptive graph convolutional neural network, a model training method, a model product and equipment. BACKGROUND

[0002] Action recognition is a research hotspot and difficulty in the field of computer vision. Humans can handle things and express feelings through actions, which can carry rich semantic information, so action recognition is widely used in human-computer interaction, video monitoring, video retrieval and other fields. Researchers have proposed many methods to accurately and quickly recognize human actions. According to different input data, there are currently mainly RGB image sequences, depth image sequences and human skeleton sequences.

[0003] Using RGB image sequence and depth image sequence data for action recognition will be disturbed by the environment in which the human body is located, and a large amount of data needs to be processed. For action recognition using human skeleton sequences, since each frame of action sequence of the human body only contains skeleton features, it will not be disturbed by the environment, and the data to be processed is also less, so action recognition on human skeleton sequence data is more lightweight and has better robustness. The traditional method for action recognition based on human skeleton sequence is dynamic time warping method, which extracts features from each collected action to establish an action template library. When performing action recognition, the distance between the action to be recognized and each action in the action template library is compared, and the action with the shortest distance is the recognized action. Since the dynamic time warping method needs to be compared with each action in the template library, when the number of action categories is large, the calculation time will greatly increase. In recent years, with the development of deep learning, many researchers have used deep learning to recognize skeleton sequence actions. For example, graph convolutional neural network is used for skeleton sequence action recognition, but it destroys the spatial topology of the skeleton and cannot well handle the time sequence relationship of the action. Subsequent research has used adaptive graph convolutional neural network to recognize skeleton actions, but most research has not considered the time difference between individual human actions, resulting in reduced recognition accuracy. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the present application provides a skeleton-based graph convolutional neural network action recognition method, which aims to optimize the deep learning network model, thereby solving the technical problem of reduced recognition accuracy caused by the time difference between individual human actions.

[0005] To achieve the above object, according to one aspect of the present application, a human action recognition model training method based on a double extended time domain adaptive graph convolutional neural network is provided, comprising the following steps:

[0006] The collected data is subjected to viewpoint-independent processing, including translation and rotation of the coordinate system, so that the spatial positions of the actions are basically close to each other, obtaining skeleton joint information and vector information of adjacent skeleton joints;

[0007] The skeleton joint information and the vector information of adjacent skeleton joints are taken as inputs, and the corresponding skeleton action is taken as output, and a double extended time domain adaptive graph convolutional neural network is trained, wherein the spatial and temporal features of the input information are extracted and labeled training through the adaptive spatial graph convolutional layer and the double extended time domain convolutional layer in the double extended time domain adaptive graph convolutional neural network, obtaining a human action recognition model based on the double extended time domain adaptive graph convolutional neural network;

[0008] The double extended time domain adaptive graph convolutional neural network comprises a BN layer, an adaptive graph convolutional block, a convolutional block combination, a global pooling layer and a Softmax layer connected in sequence, wherein:

[0009] The BN layer performs standardization processing on the input action sequence data;

[0010] One adaptive graph convolutional block is taken as the second layer and does not perform residual operation;

[0011] The number of convolutional block combinations is at least one, and each convolutional block combination is composed of one double extended time domain adaptive graph convolutional block and two adaptive graph convolutional blocks, and all of them perform residual operation.

[0012] Further, the translation of the coordinate system is to make the spine base of the first frame of the sequence as the origin, and to complete the coordinate system translation by subtracting the origin coordinates from all skeleton nodes in the action sequence.

[0013] Further, the rotation of the coordinate system is to use the method of solving the rotation matrix by using the quaternion, to rotate the coordinate system, wherein the direction from the left hip joint to the right hip joint is the x-axis direction, and the direction from the spine base to the middle spine is the z-axis direction, so as to convert the data from the camera coordinate system to the human body coordinate system.

[0014] Further, the double extended time domain adaptive graph convolutional neural network model inputs two types of data information, including:

[0015] The first type of data is skeleton joint information v i,t :

[0016] v i,t =(x i,t ,y i,t ,z i,t )

[0017] Where, x i,t ,y i,t ,z i,t Let x, y, z be the x, y, z coordinates of the i-th bone point in frame t;

[0018] The second type of input data is vector information b of adjacent skeleton joints. i,j,t :

[0019] b i,j,t =(x j,t -x i,t ,y j,t -y i,t ,z j,t -z i,t )

[0020] Where, x j,t ,y j,t ,z j,t Let x, y, z be the x, y, z coordinates of the j-th bone point in frame t, and let i be the bone point adjacent to j-th bone point.

[0021] Furthermore, both the adaptive graph convolution block and the double-extended temporal adaptive graph convolution block include adaptive spatial convolution operations and temporal convolution operations executed sequentially. The convolution kernel of the temporal convolution operation in the adaptive graph convolution block has no dilation factor, while the dilation factor of the convolution kernel of the temporal convolution operation in the double-extended temporal adaptive graph convolution block varies with the number of layers.

[0022] Furthermore, the graph convolutional layer in the graph convolution of a single-frame spatial graph is represented by the following formula:

[0023]

[0024] In the formula, H l Let K be the feature map of the l-th layer of the deep learning model, represented by the tensor (C, T, N), where C is the number of channels, T is the number of frames per sample, and N is the number of nodes; K v This represents the size of the convolution kernel in the spatial dimension; Let C represent the weight matrix of the k-th 1×1 convolution. in C represents the number of channels in the previous layer. out This represents the number of channels in the current layer. This indicates the degree of association between other skeleton nodes and the nodes undergoing convolution, based on their distance. Denotes a normalized diagonal matrix, Λ k The value of the diagonal element is The sum of the corresponding column elements; convolving the spatiotemporal skeleton graph is a process of multiplying a standard two-dimensional convolution by a normalized adjacency matrix;

[0025] The operation of the adaptive graph convolutional layer is shown in the formula:

[0026]

[0027] Among them, A k Depend on Calculated, H l-1 It is the feature map of layer l-1; B k Initially set as A k However, it will be updated as the dataset is backpropagated during training; C k The following formula determines whether two nodes are related:

[0028]

[0029] In the formula, With W Φk Obtained through Gaussian function calculation.

[0030] Furthermore, the dual extended temporal adaptive graph convolutional block combines two convolutions with different dilation factors. The first convolution sets a smaller dilation factor in the lower layers and increases with the number of layers, using non-causal convolution to capture global temporal information; the second convolution sets a larger dilation factor in the lower layers and decreases with the number of layers, capturing local temporal information.

[0031] According to another aspect of the present invention, a human action recognition method based on a dual extended temporal adaptive graph convolutional neural network is provided, wherein a human action recognition model trained by the human action recognition model training method described in the preceding claim is used to perform human action recognition.

[0032] According to another aspect of the present invention, a human motion recognition model product is provided, including a computer program, said computer program being a human motion recognition model trained using the human motion recognition model training method as described in any of the preceding claims.

[0033] According to another aspect of the present invention, a human motion recognition device is provided, including a memory, a processor, and a computer program stored in the memory, said computer program being a human motion recognition model trained using the human motion recognition model training method as described in any of the preceding claims.

[0034] In summary, the technical solutions conceived in this invention, compared with the prior art, can achieve the following beneficial effects:

[0035] (1) By making the viewpoint independent of the skeleton information, the human body coordinate axis can be determined by the human skeleton points, and the translation and rotation matrix can be calculated by using quaternions, human body origin coordinates and camera origin coordinates, which reduces the sensitivity of the action to the shooting position and improves the robustness of recognition.

[0036] (2) The double expansion time domain adaptive graph convolutional neural network comprises 7 adaptive graph convolutional blocks, 3 double expansion time domain adaptive convolutional neural networks and a softmax classification function, the double expansion time domain adaptive graph convolutional layer combines two convolutional operations with different size expansion factors, so that the network model can capture local time features and global time features of the action time sequence, and effectively reduce the influence of the fast and slow of different people performing different actions on the recognition. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a skeleton sequence obtained by a Kinect camera in the preferred embodiment of the present application.

[0038] Figure 2 is a comparison chart before and after data viewpoint independent processing in the preferred embodiment of the present application.

[0039] Figure 3 is input skeleton joint information and a vector of an adjacent skeleton joint in the preferred embodiment of the present application.

[0040] Figure 4 is a double expansion time domain adaptive graph convolutional neural network diagram in the preferred embodiment of the present application.

[0041] Figure 5 is an adaptive graph convolutional layer in the preferred embodiment of the present application.

[0042] Figure 6 is a double expansion time domain adaptive graph convolutional layer in the preferred embodiment of the present application.

[0043] Figure 7 is a model recognition accuracy confusion matrix in the preferred embodiment of the present application.

[0044] Figure 8 is a joint definition of a Kinect camera in the preferred embodiment of the present application.

[0045] Figure 9 is a schematic diagram of establishing a time-space skeleton graph without direction for an action sequence in the preferred embodiment of the present application.

[0046] Figure 10 is a diagram of extracting an action sequence feature by a graph convolutional kernel in the preferred embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0048] The preferred human action recognition method of the dual dilated temporal adaptive graph convolutional neural network of the present application comprises:

[0049] Data predefinition and preprocessing:

[0050] Designing human-computer interaction action command set, obtaining human skeleton data, and obtaining human skeleton data in the movement process through a general camera or a depth camera;

[0051] Network input data, transforming the human skeleton data, constructing a time-space human skeleton graph, obtaining the coordinate and vector information of the joint nodes as a spatial graph, and connecting the same node of different frames to generate a time graph input into the network;

[0052] Model construction and training:

[0053] Constructing a dual dilated temporal adaptive graph convolutional neural network, including 7 adaptive graph convolutional blocks (AGCB) and 3 dual dilated temporal adaptive convolutional neural network (DT-AGCB) and a softmax classification function, for human action recognition;

[0054] Obtaining the perception weight of the edge, in each multi-relation graph convolutional layer, first adopting a dynamic edge perception module DEP to perform perception operation on the features of the edges in the joint node graph and the limb graph, to obtain the perception weight of the edge;

[0055] Extracting motion features and outputting the recognized action.

[0056] Preferably, in the dual dilated temporal adaptive graph convolutional neural network, the first layer of the model is standardized by a batch normalization layer (Batch Normalization, BN).

[0057] Then, 7 adaptive graph convolutional blocks (Adaptive Graph Convolutional Blocks, AGCB) and 3 dual dilated temporal adaptive graph convolutional blocks (Dual Dilated Temporal Adaptive Graph Convolutional Blocks, DT-AGCB) are used to extract data features.

[0058] Except for the first graph convolution block, the other graph convolution blocks all increase the residual operation to alleviate the gradient vanishing caused by the increase of the number of layers. The first three layers of each graph convolution block are adaptive spatial convolution operations, and the last three layers are time domain convolution operations.

[0059] The convolution kernel of the time domain convolution operation of the AGCB has no expansion factor, and the convolution kernel of the time domain convolution operation of the DT-AGCB changes with the number of layers.

[0060] The model is reduced in parameter quantity through a global pooling layer (GAP) at the rear end, and finally classifies the action through a Softmax layer to complete the action recognition.

[0061] In a preferred embodiment, the present application comprises the following steps:

[0062] Step 1: design a human-computer interaction action command set, collect data through a sensor, and the action command set includes semantic actions such as need / no need to hold, stop the robot from moving, and provide a tool. A skeleton sequence is collected using a Kinect camera of Microsoft, as shown in Figure 1 The joint definition of the Kinect camera is shown in Figure 8 .

[0063] Step 2: perform viewpoint-independent processing on the collected data, including coordinate system translation and rotation, so that the spatial positions of the actions are basically close to each other, obtain skeleton joint information and vector information of adjacent skeleton joints, and the data before and after processing is as shown in Figure 2 .

[0064] Step 3: establish a double-expanded time domain adaptive graph convolution neural network, take the skeleton joint information and the vector information of adjacent skeleton joints as input, and train the corresponding skeleton action as output, as shown in Figure 3 , extract spatial and temporal features through an adaptive spatial graph convolution layer and a double-expanded time domain convolution layer and perform labeling training;

[0065] Step 4: use the trained model to recognize human skeleton actions.

[0066] The process steps of step 1 are as follows:

[0067] A general action command set is designed, including twelve types of actions such as "stop moving", "move to the target object", "put the object to the target position", "open / close the gripper", "move the end forward / backward / left / right / up / down", and "return to the origin", and the action command set data is obtained by collecting a skeleton sequence using a Kinect camera of Microsoft, as shown in Figure 1As shown. Each type of action includes a certain number of slower / faster action sequences. Due to the different degrees of speed differences between people or different actions of one person, a completely identical gesture can express different meanings or perform different functions through different speeds to constitute different actions. Therefore, the present application mainly faces the accurate recognition of actions based on the speed differences of these actions, so the original action command set will contain action sequences of different speeds.

[0068] The viewpoint-independent processing procedure of step 2 is as follows:

[0069] Step 2.1: For each action sequence, make the point 1 of the first frame of the sequence as the origin, and subtract the origin coordinates v from all the skeleton nodes in the action sequence O (x O ,y O ,z O ) to obtain new coordinates.

[0070] Step 2.2: Use the method of solving rotation matrix using quaternion to complete the coordinate system rotation. For the sensor coordinate system axis vector v and the human body coordinate system axis vector v', the normal vector n((n x ,n y ,n z ) of the plane formed by the two vectors and the unit vector of the normal vector can be obtained by calculation as follows:

[0071]

[0072] The angle θ between the two vectors is:

[0073]

[0074] Then the quaternion is:

[0075]

[0076] Where w, x, y, z represent the four components of the quaternion, are the x, y, z components of the unit normal vector of the plane formed by the sensor axis vector and the human body coordinate system axis vector.

[0077] According to the quaternion, the rotation matrix R can be calculated, and the translated coordinates are rotated. The effect of the final viewpoint-independent processing is as shown in Figure 3 .

[0078]

[0079] The constructed double expansion time domain adaptive graph convolutional neural network model is as shown in Figure 4 , wherein:

[0080] (1) The first layer BN layer is a batch normalization layer, which normalizes the input action sequence data.

[0081] (2) For the input data, the graph convolution operation can extract higher-dimensional features, and the collected action sequence has multiple frames of skeleton information of the human body, wherein each frame is composed of 25 different position joints. In order to enable the graph convolution to extract the motion features of the skeleton action sequence, an undirected space-time skeleton graph is established for the action sequence, as shown in Figure 9 .

[0082] The space-time skeleton graph includes a time graph and a space graph, wherein the time graph is connected by the positions of the same node in different frames, and the space graph is connected by the 25 node positions in the same frame according to the skeleton structure of the human body.

[0083] For the space-time skeleton graph, all nodes can form a point set V = {v ij |i = 1, …, T, j = 1, …, N}, wherein i represents the sequence number of each frame; j represents the sequence number of the joint in a frame; T is the total number of time frames; N is the total number of joints; and each joint v ij can be represented by a three-dimensional coordinate. For the space-time skeleton graph, the GCN can be compared to the CNN, the three-dimensional coordinates of the skeleton nodes can be compared to the channel number of the image, and the graph convolution kernel can be compared to the convolution kernel. The feature extraction process is as shown in the figure.

[0084] In the space-time skeleton graph, the adjacency matrix is used to represent the space graph in a single frame, representing the natural connection relationship between the joints of the human skeleton. In the graph convolution of the single-frame space graph, the following formula can be used to represent:

[0085]

[0086] In the formula, H l is the feature map of the lth layer of the deep learning model, represented by a tensor (C, T, N), C is the channel number; K v is the size of the convolution kernel in the spatial dimension; represents the weight matrix of the kth 1x1 convolution, C in is the channel number of the previous layer, C out is the channel number of the current layer; represents the association degree of other skeleton nodes to the convolution node according to the distance; represents a normalized diagonal matrix, Λ k the sum of the corresponding column elements in . The convolution of the space-time skeleton graph is a process of multiplying the standard two-dimensional convolution by the normalized adjacency matrix.

[0087] AGCB is an adaptive graph convolution block, as shown in Figure 5 .

[0088] The graph convolution layer uses the following formula to represent the graph convolution in the single-frame spatial graph:

[0089]

[0090] In the formula, H l is the feature map of the lth layer of the deep learning model, represented by the tensor (C, T, N), C is the number of channels; K v is the size of the convolution kernel in the spatial dimension; represents the weight matrix of the kth 1x1 convolution, C in is the number of channels of the previous layer, C out is the number of channels of the current layer; represents the association degree of the convolution node according to the distance of other skeleton nodes; represents a normalized diagonal matrix, Λ k the element value on the diagonal is the sum of the corresponding column elements in the matrix; the convolution of the space-time skeleton graph is a process of multiplying the standard two-dimensional convolution by the normalized adjacency matrix;

[0091] The operation of the adaptive graph convolution layer is shown in the formula:

[0092]

[0093] In the formula, A k is calculated by , H l-1 is the feature map of the l-1th layer; B k is initially set to A k , but will be updated during the backpropagation of the training data set; C k determines whether two nodes are associated, and the following formula is obtained:

[0094]

[0095] In the formula, and W Φk are both calculated by a Gaussian function.

[0096] (3) DT-AGCB is a double extended time domain adaptive graph convolution layer, as shown in Figure 6As shown, the convolutional layer combines two convolutional operations with different size expansion factors, so that the network model can capture local and global time features of the action timing. The first time domain convolution sets a smaller convolution with an expansion factor at a low layer, and the expansion factor is further increased as the number of layers increases. The convolution kernel is a non-causal convolution with a size of 3. This convolution has a larger receptive field at a higher layer of the model, and does not increase the number of parameters of the model by expanding the size of the convolution kernel, so that the model is more lightweight and can be used to capture global time features of the action. The second time domain convolution is opposite to the first one, which sets a larger convolution with an expansion factor at a low layer, and the expansion factor decreases as the number of layers increases. The size of the convolution kernel is also 3. This convolution can be used to capture local time features of the action. The convolution operation of each layer in the model can be described by the following formula:

[0097]

[0098] H l = H l-1 + Relu(BN(H l ))

[0099] wherein, are feature values obtained by convolution under different expansion factors, respectively; is the weight vector of the extended time domain convolution with an expansion factor of 2 l-1 and 2 L-l-1 and a kernel size of 3; (l is the current layer; L is the total number of layers; is the weight of the 1x1 convolution; is a bias vector; [t1, t2] represents splicing of two feature maps; BN() is a BN layer operation; Relu() is a Relu activation function.

[0100] Preferably, the double extended time domain adaptive graph convolutional neural network model uses a batch normalization layer at the first layer to standardize the input action sequence data, and then extracts features of the data through 7 adaptive graph convolutional blocks (AGCB) and 3 double extended time domain convolutions (DT-AGCB). Except that the first network block of the model does not use a residual operation, the residual operation is used in other network blocks to prevent the gradient vanishing problem caused by a high number of layers. The time domain convolution kernel of the AGCB in the model has no expansion factor, and the expansion factor of the time domain convolution kernel of the DT-AGCB changes with the increase of the number of layers. The model uses a global pooling layer (Global Average Pooling) at the back to reduce the parameter quantity, and finally uses a Softmax layer to complete the classification of the action, realizing the action recognition.

[0101] In a specific test example, the development language used by the present application is Python 3.8, the deep learning platform is Tensorflow 2.4, the GPU is NVIDIA GeForce RTX 2080 Ti, and the CPU is Intel Xeon E5-2678v3. Since the data of each action class is relatively small, in order to make the trained model better, 76% of the collected data set is divided into a training set, 12% is divided into a validation set, and 12% is divided into a test set. The loss function of the model is the cross-entropy loss function, the batchsize size is set to 8, and the parameter initialization adopts the method of He et al. Through 20 times of training, the training results of the spatio-temporal graph convolutional neural network, the adaptive graph convolutional neural network and the double-expanded time-domain adaptive graph convolutional neural network are obtained, which are the validation set accuracy rates in the training process. The recognition accuracy rate confusion matrix of the test set of each action of the trained model is shown in Table 1. Figure 7 As can be seen from Table 1, the designed model has excellent recognition accuracy, and the total recognition rate of the test set is more than 98%, and the recognition rate of each action is more than 97%.

[0102] It is easy for those skilled in the art to understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a human action recognition model based on a dual extended time domain adaptive graph convolutional neural network, characterized in that, The method comprises the following steps: The collected data is subjected to viewpoint-independent processing, including translation and rotation of the coordinate system, so that the spatial positions of the actions are basically close to each other, and skeleton joint information and vector information of adjacent skeleton joints are obtained; The skeleton joint information and the vector information of adjacent skeleton joints are taken as inputs, and a corresponding skeleton action is taken as output, and a double-expansion time-domain adaptive graph convolutional neural network is trained, wherein spatial and temporal features of the input information are extracted and labeled training is performed through an adaptive spatial graph convolutional layer and a double-expansion time-domain convolutional layer in the double-expansion time-domain adaptive graph convolutional neural network, and a human action recognition model based on the double-expansion time-domain adaptive graph convolutional neural network is obtained; The double-expansion time-domain adaptive graph convolutional neural network comprises a BN layer, an adaptive graph convolutional block, a convolutional block combination, a global pooling layer and a Softmax layer connected in sequence, wherein: The BN layer performs standardization processing on the input action sequence data; One adaptive graph convolutional block is taken as the second layer and does not perform residual operation; The number of the convolutional block combinations is at least one, and each convolutional block combination is composed of one double-expansion time-domain adaptive graph convolutional block and two adaptive graph convolutional blocks, and residual operation is performed on each convolutional block combination.

2. The human motion recognition model training method based on the double expansion time domain adaptive graph convolutional neural network according to claim 1, characterized in that, The translation of the coordinate system is to take the spine base of the first frame of the sequence as the origin, and subtract the origin coordinates from all the skeleton joints in the action sequence to complete the translation of the coordinate system.

3. The human motion recognition model training method based on the double expansion time domain adaptive graph convolutional neural network according to claim 1, characterized in that, The rotation of the coordinate system is to use the method of solving the rotation matrix through the quaternion, rotate the coordinate system, take the direction from the left hip joint to the right hip joint as the x-axis direction, and take the direction from the spine base to the middle spine as the z-axis direction, so that the data is converted from the camera coordinate system to the human body coordinate system.

4. The human motion recognition model training method based on the double expansion time domain adaptive graph convolutional neural network according to claim 1, characterized in that, The double-expansion time-domain adaptive graph convolutional neural network model inputs two types of data information, including: The first data type is skeleton joint information v i,t : v i,t = (x i,t ,y i,t ,z i,t ) where x i,t ,y i,t ,z i,t are the x, y, z coordinates of the i-th bone point of the t-th frame, respectively. The second data type of the input is the vector information b of the adjacent skeleton joints i,j,t : b i,j,t = (x j,t - x i,t , y j,t - y i,t , z j,t - z i,t ) where x j,t , y j,t , and z j,t are the x, y, and z coordinates of the jth skeleton point in the tth frame, respectively, and the ith skeleton point is adjacent to the jth skeleton point.

5. The human motion recognition method based on the dual-expansion time-domain adaptive graph convolutional neural network according to claim 1, characterized in that, The adaptive graph convolutional block and the double-expansion time-domain adaptive graph convolutional block each comprise adaptive spatial convolution operation and time-domain convolution operation performed in sequence, wherein the convolution kernel of the time-domain convolution operation of the adaptive graph convolutional block has no dilation factor, and the convolution kernel of the time-domain convolution operation of the double-expansion time-domain adaptive graph convolutional block changes with the change of the layer number.

6. The method of claim 5, wherein the method further comprises: The graph convolutional layer uses the following formula to represent the graph convolution in the single-frame spatial graph: In the formula, H l is the feature map of the lth layer of the deep learning model, represented by a tensor (C, T, N), C is the number of channels, T is the number of frames of each sample, and N is the number of nodes; K v is the size of the convolution kernel in the spatial dimension; represents the weight matrix of the kth 1x1 convolution, C in is the number of channels of the previous layer, C out is the number of channels of the current layer; represents the association degree of the convolution nodes according to the distance of other skeleton nodes; represents a normalized diagonal matrix, Λ k the element value on the diagonal is the sum of the corresponding column elements in the matrix; the convolution of the space-time skeleton graph is a process of multiplying a standard two-dimensional convolution by a normalized adjacency matrix; The operation of the adaptive graph convolutional layer is shown in the formula: where A k By Calculated, H l-1 Is the feature map of l-1 layer; B k Initially set to A k But will be updated with the data set in the training back propagation; C k Decided whether the two nodes are associated, the following formula is obtained: In the formula, With W Φk Obtained by Gaussian function calculation.

7. The human motion recognition model training method based on the double expansion time domain adaptive graph convolutional neural network according to claim 5, characterized in that, The double-expansion time-domain adaptive graph convolutional block combines the convolution of two different dilation factors, the first convolution sets a smaller dilation factor at a lower layer and increases with the increase of the layer number, and uses acausal convolution to capture global temporal information; the second convolution sets a larger dilation factor at a lower layer and decreases with the increase of the layer number, and captures local temporal information.

8. A human motion recognition method based on a double expansion time domain adaptive graph convolutional neural network, characterized in that, The human action recognition model trained by the human action recognition model training method of any one of claims 1-7 is used for human action recognition.

9. A human action recognition model product comprising a computer program, characterized in that, The computer program is the human action recognition model trained by the human action recognition model training method of any one of claims 1-7.

10. A human motion recognition apparatus comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 9. The computer program is the human action recognition model trained by the human action recognition model training method of any one of claims 1-7.

Images