A human behavior recognition method based on dual-stream dynamic features

Abstract

The present invention discloses a method for human behavior recognition based on dual-stream dynamic features, comprising: first extracting the spatial three-dimensional information of human skeletal joints to obtain a human joint sequence, and converting the human joint sequence into a human skeletal sequence data set. Constructing a dynamic feature extraction module, and constructing a dynamic spatiotemporal graph convolution network through the constructed spatial graph convolution module, temporal convolution module and dynamic feature extraction module. Finally, the dynamic spatiotemporal graph convolution network is used to construct a dual-stream dynamic spatiotemporal graph convolution network, and the human joint sequence is converted into a human skeletal sequence as a dual-stream input, and finally the dual-stream is weightedly fused to realize human behavior recognition and classification. The present invention solves the problem of low accuracy of the prior art in recognizing some behaviors that are biased towards speed expression and skeleton expression, and compensates for the shortcomings of each single-stream network through weighted fusion, thereby improving the accuracy of video behavior recognition.

Description

Technical Field

[0001] The present invention belongs to the field of feature recognition, and in particular relates to a human behavior recognition method based on dual-stream dynamic features. Background Art

[0002] With the continuous development of artificial intelligence, AI has been involved in many fields, including industry, medicine, finance, transportation, education, and security. Human behavior recognition has become a major task in the field of artificial intelligence. At the same time, human behavior recognition and monitoring have been gradually applied to various fields of life, greatly reducing the burden on workers.

[0003] In the existing technology, most mainstream behavior recognition methods are based on skeleton data. Because the human skeleton does not have complex background and environmental noise, behavior recognition methods based on skeleton data are highly adaptable. In recent years, more and more human behavior recognition methods based on skeleton and graph convolution have been proposed, such as the ST-GCN network based on graph convolution, the AS-GCN network, the AGC-LSTM network, and so on.

[0004] Skeletal data refers to the joint information connecting each skeleton. Its dynamic characteristics can be obtained by analyzing the joint information in two time frames. The second-order information of skeletal data is the length and direction of the bones, which is naturally more informative and discriminative for action recognition. Similar to joint information, the changes in its direction and length over time can also be expressed as dynamic characteristics. Many methods often overlook the role of the dynamic characteristics of joint information and the dynamic characteristics of the second-order skeleton information in human action recognition. This can have a certain degree of impact on the recognition results of many actions that are more prone to velocity and skeletal expression, resulting in low recognition accuracy. Summary of the Invention

[0005] The purpose of the present invention is to provide a human behavior recognition method based on dual-stream dynamic features, which can effectively improve the effect of human behavior recognition by fusing the dynamic features of joints and skeleton.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a human behavior recognition method based on dual-stream dynamic features, comprising the following steps:

[0007] S1. Use the human behavior video captured by the camera as the human behavior recognition sample and label each human behavior category;

[0008] S2. Use the Openpose algorithm to extract the spatial three-dimensional information of human skeletal joints and create a behavior classification joint data set;

[0009] S3, converting the classified joint point dataset into a human skeleton sequence dataset;

[0010] S4, building a dynamics extraction module to convert input data into dynamic features;

[0011] S5. Build spatial graph convolution module and temporal convolution module;

[0012] S6. The joint and bone dynamic features are used as two streams in parallel through the spatial graph convolution module and the temporal convolution module built in step S5, and then weighted fused to achieve behavior recognition and classification.

[0013] Furthermore, the specific steps of S2 include:

[0014] Use Openpose to process a single frame of image and extract the 3D coordinate information of the human skeleton joints in the image;

[0015] Each human body in the human behavior video has 25 joints, each joint point includes a three-dimensional coordinate point, and each category contains T frames of pictures. The skeletons of the corresponding picture frames of each video are stacked together in chronological order to form a tensor of size (C, N, T), where C=3 represents the three-dimensional coordinates X, Y, Z, N represents the 25 joint points, and T represents the number of video frames, which is the three-dimensional coordinate feature sequence data of the skeleton points;

[0016] With the feature matrix V∈R C×N×T Represents the characteristic information of human joints. For a given video skeleton sequence, the formula of the joint point S is as follows:

[0017] S={V k,t |k=1,…,N,t=1,…,T}

[0018] Where V k,t Described as the kth joint point of the person in the video at time t.

[0019] Furthermore, the S3 is specifically:

[0020] Define the joint close to the center of gravity of the skeleton as the source joint, and the joint far from the center of gravity as the target joint;

[0021] Each bone is represented as a vector pointing from the current source joint to the current target joint. Suppose the source joint V of the bone is 1,t Position p 1,t =(x 1,t ,y 1,t ,z 1,t ), target joint V 2,t Position p 2,t =(x 2,t ,y 2,t ,z 2,t ), then the vector calculation formula of the bone is:

[0022] M v1,v2 =(x 2,t -x 1,t ,y 2,t -y 1,t ,z 2,t -z 1,t )

[0023] Design an empty bone with a value of 0 at the central joint;

[0024] With the feature matrix M∈R C×N×T Represents the characteristic information of human skeleton. For a given video skeleton sequence, the skeleton formula is described as follows:

[0025] P={M k,t |k=1,…,N,t=1,…,T}

[0026] Among them, M kt Represents the kth skeleton of the person in the video at time t.

[0027] Furthermore, the specific steps of the dynamics extraction module are:

[0028] For a given joint point V k,t , using spatial three-dimensional information p k,t =(x k,t ,y k,t ,z k,t )∈R 3 and speed information v k,t To represent the dynamics, where x k,t ,y k,t ,z k,t Respectively represent the coordinate information of the three dimensions of the k-th joint point in the t-th frame;

[0029] The speed information is the difference between the t-th frame and the t-1-th frame, and the speed information v k，t The calculation formula is:

[0030] v k,t =*p k,t -p k,t-1 )

[0031] Fill the expanded matrix with 0 elements in the 0th frame to make the current structure the same as the position information;

[0032] Encode the position information and speed information and embed them into a high-dimensional matrix. The calculation formula is:

[0033]

[0034] in is the weight matrix, C1 is the size of the joint data, ReLu represents the linear rectification activation function, and b1 and b2 are bias vectors.

[0035] High-dimensional position information and high-dimensional velocity information Perform additive fusion to obtain the joint dynamic feature z k，t , and its calculation formula is:

[0036]

[0037] where z k,t It represents the dynamic characteristics of the joints after fusing high-dimensional position information and high-dimensional velocity information.

[0038] For skeleton M k,t , the three-dimensional vector information of the bone p ′ k,t =(x ′ k,t ,y ′ k,t ,z ′ k,t )∈R 3 and vector change speed information v ′ k,t =(p ′ k,t -p ′ k,t-1 ), perform high-dimensional embedding and addition fusion to obtain the skeletal dynamics feature z′ k，t ,

[0039]

[0040] where p ′ k,t Represents the three-dimensional vector information of the bone, Indicates the bone vector velocity change information, z ′ k,t It represents the dynamic characteristics of the fusion of the three-dimensional vector information of the skeleton and the velocity change information of the skeleton vector.

[0041] Furthermore, the specific steps of S5 are:

[0042] The correlation between the features themselves is first calculated through the dynamic information features to obtain the correlation adjacency matrix. Taking the dynamic information features of joints as an example, the formula is:

[0043] R t (i,j)=θ(z t,i ) T β(z t,j )

[0044] where Rt (i, j) represents the correlation from the i-th joint to the j-th joint in the t-th layer, where θ and β are composed of fully connected layers;

[0045] The construction of the spatial graph convolution module is specifically as follows:

[0046] Correlation adjacency matrix R t The dimension is T×N×N, which is combined with the dimension of T×N×C in The dynamic characteristics of the matrix are multiplied to obtain T×N×C in The new tensor, C in is the dimension of the input feature map;

[0047] The multiplication result and the dynamic features before multiplication are added after passing through the convolution layer respectively.

[0048] The result of the addition is first passed through the normalization layer and then through the ReLu layer to obtain a dimension of T×N×C out The output feature map of , that is, a spatial graph convolution layer is constructed.

[0049] The construction of the temporal convolution module is specifically as follows:

[0050] The dimension is T×N×C in The features of are obtained through the spatial pooling layer T×1×C in The output feature map of

[0051] Then, a convolution layer with a convolution kernel of 1×3 and 512 convolution kernels, a normalization layer, a ReLu linear activation layer, and a random dropout layer are used to obtain a network with a dimension of T×1×C. in The new tensor of

[0052] The features obtained above are then passed through a convolution layer with a convolution kernel of 1 and 1024 convolution kernels, and then through a normalization layer and a ReLu linear activation layer to obtain a dimension of T×1×C out The new tensor.

[0053] Furthermore, the specific steps of S6 are:

[0054] The spatial graph convolution module and the temporal graph convolution module in step S5 are connected together with a global average pooling layer, a flattening layer, and a fully connected layer to form a dynamic spatiotemporal graph convolution network, where the number of output neurons in the fully connected layer is the total number of human behavior categories;

[0055] The joint dynamic features and the skeleton dynamic features obtained in step S4 are input into the dynamic spatiotemporal graph convolutional network as two streams, and the two output results are Y1 and Y2 respectively;

[0056] Set a learnable parameter α, and finally achieve behavior recognition and classification by fusing Y1 and Y2 through parameter weighting. The final recognition result Y is:

[0057] Y=αY1+(1-α)Y2.

[0058] Where Y1 represents the output of the joint dynamics feature input into the dynamic spatiotemporal graph convolutional network. Y2 represents the output of the skeleton dynamics feature input into the dynamic spatiotemporal graph convolutional network. α is a learnable parameter, and Y is the final recognition result.

[0059] The beneficial technical effects of the present invention are at least as follows:

[0060] (1) This invention uses a two-stream spatiotemporal graph convolutional network based on dynamic features. It uses the first-order joint point information and second-order skeleton vector information of the human skeleton as the input of human behavior, extracts its dynamic features, enriches the information expression of its features, and makes up for the lack of speed information and skeleton vector information in many networks. It solves the problem of low recognition accuracy of some behaviors that are biased towards speed expression and skeleton expression in existing technologies.

[0061] (2) The dual-stream weighted fusion with learnable parameters can enable the networks to better compensate for the shortcomings of their respective single-stream networks and improve the accuracy of video behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative effort.

[0063] Figure 1 This is a flow chart of a human behavior recognition method based on dual-stream dynamic features of the present invention.

[0064] Figure 2 This is the flow chart of the dynamic spatiotemporal graph convolutional network of the present invention. DETAILED DESCRIPTION

[0065] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention. Specific implementation method one:

[0067] See also Figure 1 The present invention provides a method for human behavior recognition based on dual-stream dynamic features, comprising the following steps:

[0068] Step 1: Use a camera to shoot human behavior videos as human behavior recognition samples, and label each behavior category.

[0069] Step 2: Use the Openpose algorithm to extract the spatial three-dimensional information of human skeletal joints and create a behavior classification joint data set.

[0070] Step 3: Convert the human joint point sequence into a human skeleton sequence dataset.

[0071] Step 4: Construct a dynamics extraction module to convert input data into dynamic features

[0072] Step 5: Construct spatial graph convolution module and temporal convolution module

[0073] Step 6: Build a dynamic spatiotemporal graph convolution network through the spatial graph convolution module and the temporal convolution module, and use the joint and bone dynamic features as two streams in parallel through post-weighted fusion to achieve behavior recognition and classification.

[0074] In step 2, the Openpose tool is used to process the single-frame image and extract the three-dimensional coordinate information of the human skeleton joint points in the image.

[0075] Each human body in the video has 25 joints, each of which has 3-dimensional coordinates. Each category contains T frames of images. The skeletons of the corresponding frames of each video are stacked together in chronological order to form a tensor of size (C, N, T), where C = 3 represents the three-dimensional coordinates X, Y, Z, N is the 25 joints, and T represents the number of video frames. This is the three-dimensional coordinate feature sequence data of the skeleton points. Use the feature matrix V∈R C×N×T Represents the characteristic information of human joints. For a given video skeleton sequence, the formula for its joints is described as follows:

[0076] S={V k,t |k=1,…,N,t=1,…,T} (1)

[0077] Where V kt Described as the kth joint point of the person in the video at time t.

[0078] In step 3, define the joints close to the center of gravity of the bone as source joints and the joints far from the center of gravity as target joints. Each bone is represented as a vector pointing from its source joint to its target joint. Suppose the source joint V of the bone is 1,t Position p 1,t =(x 1,t ,y 1,t ,z 1,t ), target joint V 2,t Position p2,t =(x 2,t ,y 2,t ,z 2,t ), then the vector calculation formula of the bone is:

[0079] M v1,v2 =(x 2,t -x 1,t ,y 2,t -y 1,t ,z 2,t -z 1,t ) (2)

[0080] Since the number of joints in an image is always one more than the number of bones, in order to make the joint data dimension the same as the bone data dimension, an empty bone with a value of 0 is designed at the central joint.

[0081] With the feature matrix M∈R C×N×T Represents the characteristic information of human skeleton. For a given video skeleton sequence, the skeleton formula is described as follows:

[0082] P={M k,t |k=1,…,N,t=1,…,T} (3)

[0083] Where T is the total length of the video sequence, N is the total number of bones in the human skeleton, and M kt Described as the kth skeleton of a person in the video at time t.

[0084] In step 4, the kinetic extraction process is as follows:

[0085] For a given joint point V k,t , using its spatial three-dimensional information p k,t =(x k,t ,y k,t ,z k,t )∈R 3 and speed information v k,t To define the dynamics. k,t ,y k,t ,z k,t Respectively represent the coordinate information of the three dimensions of the kth joint point in the tth frame

[0086] The velocity information is the difference between the t-th frame and the t-1-th frame, and its calculation formula is:

[0087] v k,t =(p k,t -p k,t-1 ) (4)

[0088] Since the velocity information is the difference between the tth frame and the t-1th frame, the calculated motion velocity flow will be missing the motion velocity information of frame 0. Therefore, the matrix is ​​expanded by padding the 0th frame with zero elements so that its structure is the same as the position information.

[0089] Then the position information and speed information are encoded and embedded into a high-dimensional matrix. Taking the position high-dimensional encoding as an example, the high-dimensional embedding calculation formula is:

[0090]

[0091] in is the weight matrix, C1 is the size of the joint data, which is 128 here, ReLu represents the linear rectification activation function, and b1 and b2 are bias vectors.

[0092] Its matrix dimensions are converted from (3, 25, T) to (128, 25, T). and high-dimensional velocity information The joint dynamic information features are obtained by adding and fusion, and the calculation formula is:

[0093]

[0094] Similarly, for the skeleton M k,t , use the same method to calculate the dynamic information, and convert the three-dimensional vector information p′ of the bone k,t =(x′ k,t ,y′ k,t ,z′ k,t )∈R 3 and vector change speed information v′ k,t =(p′ k,t -p′ k,t-1 ), high-dimensional embedding and addition fusion are performed to obtain the skeletal dynamics information features

[0095]

[0096] For step 5, the correlation between the dynamic information features is first calculated to obtain the correlation adjacency matrix. Taking the dynamic information features of joints as an example, the formula is:

[0097] R t (i,j)=θ(z t,i ) T β(z t,j )(8)

[0098] Its meaning is expressed as the correlation from the i-th joint to the j-th joint in the t-th layer, where θ and β are composed of fully connected layers.

[0099] Constructing a spatial graph convolution (GCN) module:

[0100] Correlation adjacency matrix R t The dimension is T×N×N, which is combined with the dimension of T×N×C in The dynamic characteristics of the matrix are multiplied to obtain T×N×C in A new tensor of C. in is the dimension of the input feature map.

[0101] The multiplication result and the dynamic features before multiplication are added after passing through the convolution layer respectively.

[0102] The result of the addition is first passed through the normalization layer and then through the ReLu layer to obtain a dimension of T×N×C out The output feature map of . That is, a spatial graph convolution layer is constructed.

[0103] Four layers of spatial graph convolution layers are designed as the spatial graph convolution module. The convolution kernel size of each layer of graph convolution is 1×1, and the number of convolution kernels is 128, 256, 512, and 512 respectively.

[0104] Construct a temporal convolution (CNN) module:

[0105] First, the dimension is T×N×C in The features of are obtained through the spatial pooling layer T×1×C in The output feature map of . Then pass through a convolution layer with a convolution kernel of 1×3 and 512 convolution kernels, a normalization layer, a ReLu linear activation layer and a random inactivation layer to obtain a dimension of T×1×C in Here C in The size is 512.

[0106] The features obtained above are then passed through a convolution layer with a convolution kernel of 1 and 1024 convolution kernels, and then through a normalization layer and a ReLu linear activation layer to obtain a dimension of T×1×C out The new tensor.

[0107] See also Figure 2 In step 6, the spatial graph convolution module and temporal graph convolution module from step 5 are connected together with a global average pooling layer, a flattening layer, and a fully connected layer to form a dynamic spatiotemporal graph convolution network. The number of output neurons in the fully connected layer is the total number of human behavior categories.

[0108] The joint dynamic features and bone dynamic features obtained in step 4 are input as dual streams into the modified dynamic spatiotemporal graph convolutional network, and finally two output results Y1 and Y2 are obtained.

[0109] Set a learnable parameter α, and finally achieve behavior recognition and classification by fusing Y1 and Y2 through parameter weighting. The final recognition result Y is:

[0110] Y=αY1+(1-α)Y2 (9)

[0111] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A human behavior recognition method based on dual-stream dynamic features, characterized in that: The steps include: S1. Use the human behavior video captured by the camera as the human behavior recognition sample and label each human behavior category; S2. Use the Openpose algorithm to extract the spatial three-dimensional information of human skeletal joints and create a behavior classification joint data set; S3, converting the classified joint point dataset into a human skeleton sequence dataset; S4, building a dynamics extraction module to convert input data into dynamic features; S5. Build spatial graph convolution module and temporal convolution module; S6. The joint and bone dynamic features are used as two streams in parallel through the spatial graph convolution module and the temporal convolution module built in step S5, and then weighted fusion is performed to achieve behavior recognition and classification. The specific steps of the kinetic extraction module are: For a given joint point V k,t , using spatial three-dimensional information p k,t =(x k,t ,y k,t ,z k,t )∈R 3 and speed information v k,t To represent the dynamics, where x k,t ,y k,t ,z k,t Respectively represent the coordinate information of the three dimensions of the k-th joint point in the t-th frame; The speed information is the difference between the t-th frame and the t-1-th frame, and the speed information v k，t The calculation formula is: v k,t =(p k,t -p k,t-1 ) Among them, p k,t is the spatial three-dimensional information of the t-th frame, p k,t-1 is the spatial three-dimensional information of the t-1th frame; Fill the expanded matrix with 0 elements in the 0th frame to make the current structure the same as the position information; Encode the position information and speed information and embed them into a high-dimensional matrix. The calculation formula is: in is the weight matrix, C1 is the size of the joint data, ReLu represents the linear rectification activation function, b1 and b2 are the bias vectors, High-dimensional position information and high-dimensional velocity information Perform additive fusion to obtain the joint dynamic feature z k，t , and its calculation formula is: where z k,t Represents the dynamic characteristics of the joint after fusing high-dimensional position information and high-dimensional velocity information; For bone M k,t , the three-dimensional vector information p′ of the bone k,t =(x ′ k,t ,y ′ k,t ,z ′ k,t )∈R 3 and vector change speed information v ′ k,t =(p ′ k,t -p ′ k,t-1 ), perform high-dimensional embedding and addition fusion to obtain the skeletal dynamics feature z′ k，t , where p ′ k,t Represents the three-dimensional vector information of the bone, Indicates the bone vector velocity change information, z ′ k,t It represents the dynamic characteristics of the fusion of the three-dimensional vector information of the skeleton and the velocity change information of the skeleton vector.

2. The method for human behavior recognition based on dual-stream dynamic features according to claim 1, characterized in that: The specific steps of S2 include: Use Openpose to process a single frame of image and extract the 3D coordinate information of the human skeleton joints in the image; Each human body in the human behavior video has 25 joints, each joint point includes a three-dimensional coordinate point, and each category contains T frames of pictures. The skeletons of the corresponding picture frames of each video are stacked together in chronological order to form a tensor of size (C, N, T), where C=3 represents the three-dimensional coordinates X, Y, Z, N represents the 25 joint points, and T represents the number of video frames, which is the three-dimensional coordinate feature sequence data of the skeleton points; With the feature matrix V∈R C×N×T Represents the characteristic information of human joints. For a given video skeleton sequence, the formula of the joint point S is as follows: S={V k,t |k=1,…,N,t=1,…,t} Where V k,t Described as the kth joint point of the person in the video at time t.

3. The method for human behavior recognition based on dual-stream dynamics features according to claim 2, characterized in that: The S3 is specifically: Define the joint close to the center of gravity of the skeleton as the source joint, and the joint far from the center of gravity as the target joint; Each bone is represented as a vector pointing from the current source joint to the current target joint. Suppose the source joint V of the bone is 1,t Position p 1,t =(x 1,t ,y 1,t ,z 1,t ), target joint V 2,t Position p 2,t =(x 2,t ,y 2,t ,z 2,t ), then the vector calculation formula of the bone is: M v1,v2 =(x 2,t -x 1,t ,y 2,t -y 1,t ,z 2,t -z 1,t ) Among them, x 1,t ,y 1,t ,z 1,t They represent the coordinate information of the three dimensions of the first joint point in the t-th frame, x 2,t ,y 2,t ,z 2,t Respectively represent the coordinate information of the three dimensions of the second joint point in the t-th frame, M v1,v2 is the source joint V 1,t Position and target joint V 2,t The vectors between the positions of bones; Design an empty bone with a value of 0 at the central joint; With the feature matrix M∈R C×N×T Represents the characteristic information of human skeleton. For a given video skeleton sequence, the skeleton formula is described as follows: P={M k,t |k=1,…,N,t=1,…,T} Among them, M kt Represents the kth skeleton of the person in the video at time t.

4. The method for human behavior recognition based on dual-stream dynamic features according to claim 1, characterized in that: The specific steps of S5 are: The correlation between the features themselves is first calculated through the dynamic information features to obtain the correlation adjacency matrix. Taking the dynamic information features of joints as an example, the formula is: R t (i,j)=θ(z t,i ) T β(z t,j ) where R t (i, j) represents the correlation from the i-th joint to the j-th joint in the t-th layer, where θ and β are composed of fully connected layers; The construction of the spatial graph convolution module is specifically as follows: Correlation adjacency matrix R t The dimension is T×N×N, which is combined with the dimension of T×N×C in The dynamic characteristics of the matrix are multiplied to obtain T×N×C in The new tensor, C in is the dimension of the input feature map; The multiplication result and the dynamic features before multiplication are added after passing through the convolution layer respectively; The result of the addition is first passed through the normalization layer and then through the ReLu layer to obtain a dimension of T×N×C out The output feature map of , that is, a spatial graph convolution layer is constructed, The construction of the temporal convolution module is specifically as follows: The dimension is T×N×C in The features of are obtained through the spatial pooling layer T×1×C in The output feature map of Then, a convolution layer with a convolution kernel of 1×3 and 512 convolution kernels, a normalization layer, a ReLu linear activation layer, and a random dropout layer are used to obtain a network with a dimension of T×1×C. in The new tensor of The features obtained above are then passed through a convolution layer with a convolution kernel of 1 and 1024 convolution kernels, and then through a normalization layer and a ReLu linear activation layer to obtain a dimension of T×1×C out The new tensor.

5. The method for human behavior recognition based on dual-stream dynamic features according to claim 4, characterized in that: The specific steps of S6 are: The spatial graph convolution module and the temporal graph convolution module in step S5 are connected together with a global average pooling layer, a flattening layer, and a fully connected layer to form a dynamic spatiotemporal graph convolution network, where the number of output neurons in the fully connected layer is the total number of human behavior categories; The joint dynamic features and the skeleton dynamic features obtained in step S4 are input into the dynamic spatiotemporal graph convolutional network as two streams, and the two output results are Y1 and Y2 respectively; Set a learnable parameter α, and finally achieve behavior recognition and classification by fusing Y1 and Y2 through parameter weighting. The final recognition result Y is: Y=αY1+(1-α)Y 2， Where Y1 represents the output result obtained by inputting the joint dynamic features into the dynamic spatiotemporal graph convolutional network, Y2 represents the output result obtained by inputting the skeleton dynamic features into the dynamic spatiotemporal graph convolutional network, α is a learnable parameter, and Y is the final recognition result.

Images