Fitness action recognition method based on multi-branch fusion graph convolutional network

Abstract

The invention discloses an innovative multi-branch feature fusion adaptive graph convolutional network, and aims to break through the limitation of a traditional action recognition technology in high-dimensional complex motion analysis. According to the method, an adaptive graph convolutional network AGCN and a time channel aggregation graph convolutional network TCA-GCN are fused, and an efficient multi-branch feature fusion framework is constructed. And the Transform is used as a core component, so that the integration of heterogeneous features is further optimized, and the complementarity and expression ability of the features are remarkably improved. By converting visual skeleton data into a graph structure, the method makes full use of the inherent characteristics of a human skeleton image, and realizes accurate extraction and integration of skeleton motion high-order features. A multi-branch fusion strategy is developed, the dynamic updating capability of the AGCN and the advantages of TCA-GCN in time channel feature extraction are combined, and experimental verification on a common reference data set shows that the recognition precision of key skeleton nodes in motion is remarkably improved.

Description

Technical Field

[0001] The present invention relates to action recognition technology in the field of computer vision, and in particular to a method for identifying fitness actions. Background Art

[0002] Human motion recognition is of great significance in the field of fitness. It can monitor and evaluate the user's exercise status during fitness, provide a basis for personalized fitness plans, and assist fitness coaches in analyzing students' movements and improving training results.

[0003] Action recognition faces many challenges in fitness scenarios, such as background clutter, illumination changes, scale changes, viewpoint changes, occlusions, and motion speed changes. In fitness motion analysis, the role of key skeletal nodes is crucial, and the overall coordination of the body, such as limb swing trajectory and trunk balance control, also has a significant impact on motion performance. These factors combined have led to a significant increase in the diversity and complexity of fitness movements. Early deep learning-based methods viewed human joints as a set of independent features and organized them into a feature sequence or a pseudo-image, which was input into a recurrent neural network or a convolutional neural network to predict action labels. The graph convolutional network (GCN)-based method is different from traditional manual methods, recurrent neural network (RNN) and convolutional neural network (CNN) network models. It can process data with generalized topological structures and deeply mine their features. Yan et al. first proposed the spatiotemporal graph convolutional network (ST-GCN). Since then, many GCN-based methods for human skeleton data behavior recognition have been developed on the basis of ST-GCN, such as 4S-AAGCN, MST-GCN, CTR-GCN, TCA-GCN, etc. Therefore, the method based on graph convolutional network has become one of the mainstream methods for human skeleton data behavior recognition.

[0004] Although GCN-based motion analysis technology has made significant progress in basic human posture recognition, the applicability and accuracy of existing models in fitness applications still need to be improved. The GCN model can be further optimized to focus more on the feature extraction of key bone nodes and overall body coordination. Combine multimodal data to improve the applicability and accuracy of the model in complex motion scenarios. Strengthen the research on fitness movement patterns and deeply explore the skeletal dynamics characteristics of specific movements. Summary of the invention

[0005] In view of the shortcomings of existing motion recognition methods in terms of joint motion relations and data utilization, this paper proposes an innovative multi-branch feature fusion strategy. This strategy first uses adaptive graph convolutional networks (AGCNs) and temporal channel aggregation graph convolutional networks (TCA-GCNs) to construct a multi-branch feature fusion framework. On this basis, Transformer is introduced as the core component of the framework to integrate the heterogeneous features extracted by the two branch networks and adaptively enhance their complementarity and expression capabilities. This innovation aims to provide a comprehensive and accurate analysis tool to understand highly complex motion patterns.

[0006] This framework not only achieves technological innovation in parsing complex skeleton dynamics, but also significantly improves the accuracy and sensitivity of identifying the role of key bone nodes in motion. It also opens up a new way to fine-tune the identification of specific joint effects. Through rigorous testing on widely recognized datasets, empirical research results show that this model can accurately capture the key joint dynamics in fitness movements from multiple dimensions and effectively reveal the deep connection between sports performance and joint function.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A fitness action recognition method based on a multi-branch fusion graph convolutional network, the method comprising the following steps:

[0009] Step 1) Generate a time series of the coordinates of the skeletal points to be detected based on the motion recognition video to be detected, and establish a graph adjacency matrix based on the set human skeleton image, and split it into several matrices A k ;

[0010] Step 2) preprocessing the skeleton point coordinate time series to obtain input Z;

[0011] Step 3) Input Z obtained in the previous step into the adaptive graph convolutional network (AGCN) and the temporal channel aggregation graph convolutional network (TCA-GCN) modules respectively to obtain spatiotemporal features at different levels and concatenate them as output;

[0012] Step 4) Transformer is introduced as the core component of the framework to integrate the heterogeneous features extracted by the two branch networks, and adaptively enhance their complementarity and expression capabilities to build a multi-branch feature fusion framework to achieve a deep understanding of complex sequence data;

[0013] Step 5) Input the joint coordinates, bone direction and length data into the spatiotemporal adaptive feature fusion graph convolutional network for training, obtain the corresponding initial recognition results and softmax scores, and finally output the final result by weighted addition.

[0014] Furthermore, in the step 1), the initial adjacency matrix established according to the human skeleton image is A, and the adjacency matrix is ​​split according to: A+I=∑ k A k , where A k denote the adjacency matrix of the k-th hop, and A 0 =I, A 1 =A.

[0015] Furthermore, in step 2), the skeleton point coordinate time series is processed using kernel Γ u Perform temporal convolution to obtain temporal features.

[0016] Furthermore, it is concatenated with the original sequence to extract spatiotemporal features. All preprocessing can be written as:

[0017]

[0018] in, is the processed video representation, * represents temporal convolution, T represents the length of the time series, N represents the number of nodes, and C represents the number of feature channels. u represents the number of filters used for temporal convolution, V represents the time series of bone point coordinates, and * and Represent temporal convolution and feature concatenation respectively.

[0019] Furthermore, in step 3), in order to extract spatial features from the topological structure of the human skeleton, the results are respectively subjected to graph convolution. The specific method is as follows: using an adaptive graph convolutional network (AGCN), applying a normalized embedded Gaussian function to calculate the similarity between two joints, which can not only reflect the connectivity between the two joints, but also determine the strength of the connection between them. The specific calculation process is expressed as follows:

[0020]

[0021] Among them, v i ,v j represents two different nodes, and N is the number of nodes. If P represents the similarity matrix calculated after embedding the Gaussian function, the new adjacency matrix is ​​expressed as:

[0022]

[0023] Among them, the spatiotemporal graph convolution network model is composed of a BN layer and a multi-scale graph convolution module. The multi-scale graph convolution module contains nine layers of feature extraction modules, and its output channels are 64, 64, 64, 128, 128, 128, 256, 256, 256. Each layer of feature extraction modules includes a second residual module, and an adaptive spatial domain graph convolution unit, an activation function, an attention module, and a temporal domain graph convolution unit connected in sequence. The input end of the second residual module is connected to the input end of the adaptive spatial domain graph convolution unit, and the output end is connected to the first feature extraction unit, which is used to perform an addition operation.

[0024] Furthermore, a temporal channel aggregation graph convolutional network (TCA-GCN) is proposed to dynamically learn spatial and temporal topology and effectively aggregate topological features of different temporal and channel dimensions for skeleton-based action recognition. This part uses channel topology modeling to dynamically obtain the spatial channel topology, which consists of a correlation modeling function and a refinement function. The correlation modeling function uses the input feature X∈R T×N×C Model channel dependencies between joints to obtain channel-specific dependencies C 1 represents a specific number of channels. The refinement function adds the channel-related Q to the adjacency matrix obtained by spatial configuration partitioning to obtain the channel topology The specific method formula is as follows:

[0025] S = α·Q + μ(A k ) (4)

[0026] Among them, A k is the k-th channel adjacency matrix, μ(*) represents the normalization and dimension transformation operation of the third-order adjacency matrix, and α is a trainable parameter representing the connection strength.

[0027] In the temporal aggregation part, a temporal adaptive weight aggregation method is proposed based on temporal adaptive convolution to process skeleton features in the temporal dimension. This method can generate calibrated weights in the temporal dimension based on the input features, and aggregate them with the prior topology in the temporal dimension to complete the temporal dynamic topological representation.

[0028] Specifically, we first generate the time weights from the skeleton sequence as input X∈R T×N×C , use the generating function to multiply the initial weight, perform dimensional transformation to generate time weight, perform time aggregation with the prior topology, and obtain a new high-dimensional feature representation As shown in the formula:

[0029] A out =TA(W,X)=(W 1 X 1 )||...||(WT X T ) (5)

[0030] Among them, W 1 X 1 It is expressed as: TA(*) represents the aggregation function, w(j) represents the generating function, x(j) represents the initial weight, and W(*) represents the dimensionality reduction function. is the time weight feature, and its expression is:

[0031] W=α t ·W 0 (6)

[0032] Among them, α t =G(x 1 ,...,x t ), α t is the calibration weight with output channels, using different dimensional methods based on bone features, is the initial weight. In this part, the focus is on aggregating the spatiotemporal features of the first two parts to complete the balancing process. When aggregating in the channel dimension, the graph convolution in the channel dimension is expressed as follows:

[0033]

[0034] Among them, A out is the output feature of adaptive aggregation, B i Indicates v j The node range, S is the output feature of the channel direction topology, v i i is the i-th node, v j For the jth node, the final output result It is expressed as follows:

[0035]

[0036] in, It is expressed as: CA is the channel aggregation function.

[0037] Furthermore, in step 4), a Transformer fusion network model is established, features are input from the two network models, and the fusion result is output. The output features of the two feature extractors are fused through a Transformer network based on multi-head cross attention. Using m-head cross attention, the operation process of Transformer is expressed as:

[0038]

[0039] Among them, CA j is the j head cross attention, BQj for Linear projection of the appearance. Kj and E Vj yes The linear projection of dim represents B Qj and E Kj The dimension of , W′ is the weight matrix of multi-head attention. Then use a linear layer, followed by a Tanh activation, and stack the residual connection and feedforward layer, which is expressed as:

[0040]

[0041]

[0042] Where FFN stands for Feed-Forward Network. F The loss between and the label is expressed as:

[0043] O F =tanh(Linear(F,θ F )) (13)

[0044] L inter =MSE(O F ,Y) (14)

[0045] Among them, L inter is the total loss function and Y represents the label.

[0046] Furthermore, in step 5), the present invention experimentally verifies the framework on a public benchmark dataset and constructs a four-stream framework. Each of the four spatiotemporal graph convolutional network models is a group, corresponding to four types of flow data: J-Stream, B-Stream, JM-Stream, and BM-Stream. These four types of flow data are: joint flow V', bone flow (where e is the bone vector for each bone, ), joint motion flow M joint (where m joint is the vector difference of each joint in adjacent frame images, ) and skeletal motion flow M bone (where m bone is the vector difference of each bone in adjacent frame images, ). Specifically, the first-order information (joint coordinates) and second-order information (bone direction and length) are input into four independent streams for processing. Subsequently, the human action video to be recognized is converted into four stream data and input into the trained spatiotemporal graph convolutional network model to obtain the corresponding initial prediction results and softmax scores, obtain the action scores, and then predict the action labels based on them. Experimental results show that the performance of this framework is significantly better than the baseline model in tasks such as action recognition and advanced skeleton motion analysis.

[0047] Compared with the prior art, the beneficial effects of the present invention are mainly manifested in:

[0048] 1. This paper develops an innovative multi-branch fusion strategy. Through the dual-branch network structure of AGCN and TCA-GCN, it accurately extracts and integrates the high-order features of skeleton motion, deeply mines its semantic information, and provides an efficient analysis tool for understanding complex motion patterns.

[0049] 2. The present invention introduces a Transformer fusion module, which serves as a bridge to connect different features, thereby achieving deep and detailed feature fusion; this strategy not only optimizes the extraction of motion state information and improves the accuracy of action recognition, but also provides an effective tool for the precise analysis of highly complex motion patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0051] Figure 2 It is a flow chart of input data preprocessing of the present invention;

[0052] Figure 3 It is the AGCN network structure diagram of the present invention;

[0053] Figure 4 It is a TCA-GCN network structure diagram of the present invention;

[0054] Figure 5 It is a schematic diagram of the Transformer fusion module of the present invention. DETAILED DESCRIPTION

[0055] The present invention is further described below in conjunction with the accompanying drawings.

[0056] Reference Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 As shown in Table 1 and Table 2, the present invention describes a fitness action recognition method based on a multi-branch fusion graph convolutional network, comprising the following steps:

[0057] like Figure 1 As shown in FIG. 1 , it is a schematic diagram of the overall framework of the present invention. According to the present invention, a fitness action recognition method based on a multi-branch fusion graph convolutional network is provided, which includes the following steps:

[0058] Step 1) Convert the input video stream of Τ×N×C into the corresponding bone point coordinate time series, where Τ is the number of frames played in a video; N is the number of human skeleton joints; and C is the coordinate vector dimension of each joint.

[0059] The initial adjacency matrix established according to the human skeleton image is A, and the basis for splitting the adjacency matrix is: A+I=∑ k A k , where A k denote the adjacency matrix of the k-th hop, and A 0 =I, A 1 =A.

[0060] Step 2) The position of the data can only provide limited information, making it difficult to accurately identify human motion. In order to enrich the kinematic information of the input data, the present invention preprocesses the skeleton position sequence data, obtains the skeleton velocity sequence data and skeleton acceleration sequence data features, and Figure 2 The preprocessing process shown in the figure uses the kernel Γ on the skeleton point coordinate time series u Perform temporal convolution to obtain temporal features.

[0061] Specifically, it is concatenated with the original sequence to extract spatiotemporal features. All initial processing can be written as:

[0062]

[0063] in, is the processed video representation, and * represents temporal convolution.

[0064] Step 3) According to Figure 3 and Figure 4 As shown in the figure, a network model with AGCN and TCA-GCN is established. In order to extract spatial features from the topological structure of the human skeleton, the result is subjected to graph convolution. The specific method is as follows: a normalized embedded Gaussian function is applied to calculate the similarity between two joints, which can not only reflect the connectivity between the two joints, but also determine the strength of the connection between them. The specific calculation process is expressed as follows:

[0065]

[0066] Among them, v i ,v jrepresents two different nodes, and N is the number of nodes. If P represents the similarity matrix calculated after embedding the Gaussian function, the new adjacency matrix is ​​expressed as:

[0067]

[0068] Among them, the spatiotemporal graph convolution network model is composed of a BN layer and a multi-scale graph convolution module. The multi-scale graph convolution module contains nine layers of feature extraction modules, and its output channels are 64, 64, 64, 128, 128, 128, 256, 256, 256. Each layer of feature extraction modules includes a second residual module, and an adaptive spatial domain graph convolution unit, an activation function, an attention module, and a temporal domain graph convolution unit connected in sequence. The input end of the second residual module is connected to the input end of the adaptive spatial domain graph convolution unit, and the output end is connected to the first feature extraction unit, which is used to perform an addition operation.

[0069] Specifically, a temporal channel aggregation graph convolutional network (TCA-GCN) is used to dynamically learn spatial and temporal topology and effectively aggregate topological features of different temporal and channel dimensions for skeleton-based action recognition. This part uses channel topology modeling to dynamically obtain the spatial channel topology, which consists of a correlation modeling function and a refinement function. The correlation modeling function uses the input feature X∈R T×N×C Model channel dependencies between joints to obtain channel-specific dependencies The refinement function adds the channel-related Q to the adjacency matrix obtained by spatial configuration partitioning to obtain the channel topology The specific method formula is as follows:

[0070] S = α·Q + μ(A k ) (4)

[0071] Among them, A k is the k-th channel adjacency matrix, μ(*) represents the normalization and dimension transformation operation of the third-order adjacency matrix, and α is a trainable parameter representing the connection strength.

[0072] In the temporal aggregation part, based on temporal adaptive convolution, a temporal adaptive weight aggregation method is proposed to process the skeleton features in the temporal dimension. This method can generate calibrated weights in the temporal dimension based on the input features, and aggregate them with the prior topology in the temporal dimension to complete the temporal dynamic topological representation. Specifically, first, the skeleton sequence is used to generate the temporal weights as input X∈R T×N×C , use the generating function to multiply the initial weight, perform dimensional transformation to generate time weight, perform time aggregation with the prior topology, and obtain a new high-dimensional feature representation As shown in the formula:

[0073] Aout =TA(W,X)=(W 1 X 1 )||...||(W T X T ) (5)

[0074] Among them, W 1 X 1 It is expressed as: TA(*) represents the aggregation function, W(*) represents the dimensionality reduction function, is the time weight feature, and its expression is:

[0075] W=α t ·W 0 (6)

[0076] Among them, α t =G(x 1 ,...,x t ), α t is the calibration weight with output channels, using different dimensional methods based on bone features, is the initial weight. In this part, the focus is on aggregating the spatiotemporal features of the first two parts to complete the balancing process. When aggregating in the channel dimension, the graph convolution in the channel dimension is expressed as follows:

[0077]

[0078] Among them, A out is the output feature of adaptive aggregation, S is the output feature of channel direction topology, v i For the i-th node, the final output result It is expressed as follows:

[0079]

[0080] in, It is expressed as: CA is the channel aggregation function.

[0081] In step 4), according to Figure 5 As shown in the figure, a Transformer fusion network model is established to integrate the heterogeneous features output by the two models in step 3) and output the fusion result. The output features of the two models are fused through a Transformer network based on multi-head cross attention. Using m-head cross attention, the operation process of Transformer is expressed as:

[0082]

[0083] Among them, CA j is the j head cross attention, BQj for Linear projection of the appearance. Kj and E Vj yes The linear projection of dim represents B Qj and E Kj The dimension of , W′ is the weight matrix of multi-head attention. Then use a linear layer, followed by a Tanh activation, and stack the residual connection and feedforward layer, which is expressed as:

[0084]

[0085] Where FFN stands for Feed-Forward Network. F The loss between and the label is expressed as:

[0086] O F =tanh(Linear(F,θ F )) (13)

[0087] L inter =MSE(O F ,Y) (14)

[0088] Among them, L inter is the total loss function.

[0089] Step 5) As shown in Tables 1 and 2, the present invention experimentally verifies the present framework on public benchmark datasets and constructs a four-stream framework. In this framework, the multi-heart spatiotemporal attention graph convolutional network model contains twelve parallel spatiotemporal graph convolutional network models, each of which is a group of four, corresponding to the four streaming data of J-Stream, B-Stream, JM-Stream, and BM-Stream. Specifically, the first-order information (joint coordinates) and the second-order information (bone direction and length) are respectively input into four independent streams for processing. Subsequently, the softmax scores of the four streams are weighted and fused to obtain the action score, and then the action label is predicted based on it. This embodiment is compared with the advanced model in the top-1 accuracy (%) of the NTU-RGB+D 60 and NTU-RGB+D120 datasets. This model has achieved the most advanced results in almost all benchmark tests.

[0090] In the model training stage, all experiments in this embodiment are carried out under the PyTorch deep learning framework and trained using a Tesla A100 GPU. The training parameters are as follows: the initial learning rate is 0.1, the weight decay is 0.0004, the parameters are adjusted using stochastic gradient descent (SGD) with a Nesterov momentum of 0.9, the maximum number of training rounds is set to 70 times, and the learning rate is divided by 10 in the 35th and 55th training stages to minimize the loss function, thereby generating a human motion sequence that is very close to the real motion data, ensuring that the model can capture and simulate human motion more accurately. Model training is a well-known technology for those skilled in the art, so it will not be repeated here.

[0091] Table 1: Comparison of top-1 accuracy (%) with state-of-the-art methods on the NTU-RGBD 60 dataset

[0092]

[0093] Table 2: Comparison of top-1 accuracy (%) with state-of-the-art methods on the NTU-RGBD 120 dataset

[0094]

[0095]

[0096] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A fitness action recognition method based on a multi-branch fusion graph convolutional network, characterized in that: The method comprises the following steps: Step 1) Generate the coordinate time series of the skeletal points to be detected based on the motion recognition video to be detected, and establish the graph adjacency matrix based on the set human skeleton image, and split it into several matrices A k ; Step 2) preprocessing the skeleton point coordinate time series to obtain input Z; Step 3) constructing a spatiotemporal adaptive feature fusion graph convolutional network model, which includes an adaptive graph convolutional network AGCN model and a temporal channel aggregation graph convolutional network TCA-GCN model. The input Z obtained in step 2) is input into the two models respectively to obtain spatiotemporal features at different levels and splice them as output; Step 4) construct a multi-branch feature fusion framework, in which Transformer is introduced to integrate the heterogeneous features extracted by the two branch networks, and adaptively enhance their complementarity and expression ability to achieve a deep understanding of complex sequence data; Step 5) Extract joint coordinates, bone direction and length data from the time series of the bone point coordinates to be tested, input the extracted data into the spatiotemporal adaptive feature fusion graph convolutional network for training, obtain the corresponding initial recognition results and softmax scores, and finally output the final results by weighted addition.

2. A method for identifying fitness movements based on a multi-branch fusion graph convolutional network as claimed in claim 1, characterized in that: In step 1), the partitioning of the adjacency matrix is ​​based on the following formula: A+I=∑ k A k , where A k They respectively represent the adjacency matrices with a distance k from them, and A0=I, A1=A.

3. A method for identifying fitness movements based on a multi-branch fusion graph convolutional network as claimed in claim 2, characterized in that: In step 2), the kernel Γ is used on the skeleton point coordinate time series u Perform temporal convolution to obtain temporal features, and concatenate them with the original sequence to extract spatiotemporal features; All preprocessing is expressed as: in, is the processed video representation, * represents temporal convolution, T represents the length of the time series, N represents the number of nodes, C represents the number of feature channels, and C u represents the number of filters used for temporal convolution, V represents the time series of bone point coordinates, and * and Represent temporal convolution and feature concatenation respectively.

4. The method for identifying fitness movements based on a multi-branch fusion graph convolutional network as claimed in claim 2, characterized in that: In step 3), in order to extract spatial features from the topological structure of the human skeleton, the result is subjected to graph convolution, specifically as follows: a normalized embedded Gaussian function is applied to calculate the similarity between two joint points, and the calculation process is expressed as follows: Among them, v i ,v j represents two different nodes, N is the number of nodes; let P represent the similarity matrix calculated after embedding the Gaussian function, then the new adjacency matrix is ​​expressed as:

5. A method for identifying fitness movements based on a multi-branch fusion graph convolutional network as claimed in claim 4, characterized in that: In the step 3), the spatiotemporal graph convolution network model includes a BN layer and a multi-scale graph convolution module; the multi-scale graph convolution module includes nine layers of feature extraction modules, each layer of feature extraction modules includes a second residual module, and an adaptive spatial domain graph convolution unit, an activation function, an attention module, and a temporal domain graph convolution unit connected in sequence; the input end of the second residual module is connected to the input end of the adaptive spatial domain graph convolution unit, and the output end is connected to the first feature extraction unit, and the first feature extraction unit is used to perform an addition operation.

6. The method for identifying fitness movements based on a multi-branch fusion graph convolutional network as claimed in claim 4, characterized in that: In step 3), the temporal channel aggregation graph convolutional network TCA-GCN model uses channel topology modeling to dynamically obtain the spatial channel topology, which includes a related modeling function and a refinement function; the related modeling function uses the input feature X∈R T ×N×C Model channel dependencies between joints to obtain channel-specific dependencies C 1 Indicates a specific number of channels; The refinement function adds the channel-related Q to the adjacency matrix obtained by spatial configuration partitioning to obtain the channel topology The specific method formula is as follows: S=α·Q+μ(A k ) (4) Among them, A k is the k-th channel adjacency matrix, μ(*) represents the normalization and dimension transformation operation of the third-order adjacency matrix, α is a trainable parameter, and Q represents the connection strength.

7. The method for identifying fitness movements based on a multi-branch fusion graph convolutional network as claimed in claim 4, characterized in that: In the step 3), when the time channel aggregation graph convolution network TCA-GCN model aggregates, the skeleton features of the time dimension are processed based on the time adaptive weight aggregation method; the time adaptive weight aggregation method generates calibrated weights in the time dimension based on the input features, and aggregates them with the prior topology in the time dimension, thereby completing the time dynamic topology representation; specifically as follows: First, the skeleton sequence generates the time weight as input X∈R T×N×C , use the generating function to multiply the initial weight, perform dimensional transformation to generate time weight, perform time aggregation with the prior topology, and obtain a new high-dimensional feature representation As shown in the formula: A out =TA(W,X)=(W1X1)||...||(W T X T ) (5) Among them, W1X1 is expressed as: TA(*) represents the aggregation function, w(j) represents the generating function, x(j) represents the initial weight, and W(*) represents the dimensionality reduction function. is the time weight feature, and its expression is: In=α t W0 (6) Among them, α t =G(x1,...,x t ), α t is the calibration weight with output channels, using different dimensional methods based on bone features, is the initial weight; when aggregating on the channel dimension, the graph convolution of the channel dimension is expressed as follows: Among them, A out is the output feature of adaptive aggregation, B i Indicates v j The node range, S is the output feature of the channel direction topology, v i is the i-th node, v j For the jth node, the final output result It is expressed as follows: in, It is expressed as: CA is the channel aggregation function.

8. The method for identifying fitness movements based on a multi-branch fusion graph convolutional network as claimed in claim 2, characterized in that: In step 4), a Transformer fusion network model is established to integrate the heterogeneous features output by the two models in step 3) and output a fusion result; specifically, as follows: The output features of the two models are fused through a Transformer network based on multi-head cross attention; using m-head cross attention, the operation process of Transformer is expressed as: Among them, CA j is j head cross attention, B Qj for Linear projection of the appearance; E Kj and E Vj yes The linear projection of dim represents B Qj and E Kj The dimension of , W′ is the weight matrix of multi-head attention; then a linear layer is used, followed by a Tanh activation, and the residual connection and feedforward layer are superimposed, which is expressed as: Among them, FFN stands for feed-forward network; calculation O F The loss between the label is expressed as: O F =tanh(Linear(F,θ F )) (13) L inter =MSE(O F ,Y) (14) Among them, L inter is the total loss function and Y represents the label.

9. The method for identifying fitness movements based on a multi-branch fusion graph convolutional network as claimed in claim 2, characterized in that: In the step 5), the framework is experimentally verified to establish a four-stream framework, in which each four spatiotemporal graph convolutional network models are a group, corresponding to four flow state data of joint stream J-Stream, skeleton stream B-Stream, ligament stream JM-Stream, and joint motion stream BM-Stream respectively; the specific process of step 5) is as follows: 1) Input joint coordinates, bone orientation and length data into four independent streams for processing; 2) Convert the human behavior video to be identified into four types of flow data and input them into the trained spatiotemporal graph convolutional network model to obtain the corresponding initial prediction results and softmax scores to obtain the action scores; 3) Predict action labels.

Images