A fitness action recognition method based on multi-branch fusion graph convolutional network

Abstract

The present invention discloses an innovative multi-branch feature fusion adaptive graph convolutional network, which aims to break through the limitations of traditional action recognition technology in high-dimensional complex motion analysis. This method integrates the adaptive graph convolutional network AGCN and the temporal channel aggregation graph convolutional network TCA-GCN to construct an efficient multi-branch feature fusion framework. And with Transformer as the core component, it further optimizes the integration of heterogeneous features and significantly improves the complementarity and expression ability of features. By converting visual skeleton data into a graph structure, the present invention fully utilizes the inherent characteristics of human skeleton images and realizes the accurate extraction and integration of high-order features of skeleton motion. A multi-branch fusion strategy is developed to combine the dynamic update capability of AGCN with the advantages of TCA-GCN in temporal channel feature extraction. Experimental verification on public benchmark datasets shows that the recognition accuracy of key skeletal nodes in motion is significantly improved.

Description

Technical Field

[0001] The present invention relates to action recognition technology in the field of computer vision, and in particular to a fitness-oriented action recognition method. Background Art

[0002] Human motion recognition is of great significance in the fitness field. It can monitor and evaluate the user's exercise status during exercise, provide a basis for personalized fitness plans, and assist fitness coaches in analyzing students' movements to improve training results.

[0003] Action recognition faces many challenges in fitness scenarios, such as background clutter, illumination changes, scale changes, viewpoint changes, occlusions, and motion speed changes. In fitness motion analysis, the role of key skeletal nodes is crucial. At the same time, the overall coordination of the body, such as limb swing trajectory and trunk balance control, also has a significant impact on motion performance. These factors combined have led to a significant increase in the diversity and complexity of fitness movements. Early deep learning-based methods viewed human joints as a set of independent features and organized them into a feature sequence or a pseudo-image, which was input into a recurrent neural network or a convolutional neural network to predict action labels. The graph convolutional network (GCN)-based method is different from traditional manual methods, recurrent neural network (RNN) and convolutional neural network (CNN) network models. It can process data with generalized topological structures and deeply explore its features. Yan et al. first proposed the spatiotemporal graph convolutional network (ST-GCN). Since then, many GCN-based methods for human skeleton data behavior recognition have been developed on the basis of ST-GCN, such as 4S-AAGCN, MST-GCN, CTR-GCN, TCA-GCN, etc. Therefore, the graph convolutional network-based method has become one of the mainstream methods for human skeleton data behavior recognition.

[0004] While GCN-based motion analysis technology has made significant progress in basic human posture recognition, the applicability and accuracy of existing models in fitness applications still need to be improved. GCN models can be further optimized to focus on feature extraction of key skeletal nodes and overall body coordination. Incorporating multimodal data can improve the model's applicability and accuracy in complex motion scenarios. Research on fitness movement patterns should be strengthened to further explore the skeletal dynamics characteristics of specific movements. Summary of the Invention

[0005] In response to the shortcomings of existing motion recognition methods in terms of joint motion relationships and data utilization, this paper proposes an innovative multi-branch feature fusion strategy. This strategy first uses the adaptive graph convolutional network (AGCN) and the temporal channel aggregation graph convolutional network (TCA-GCN) to construct a multi-branch feature fusion framework. On this basis, the Transformer is introduced as the core component of the framework to integrate the heterogeneous features extracted by the two branch networks and adaptively enhance their complementarity and expression capabilities. This innovation aims to provide a comprehensive and accurate analysis tool to understand highly complex motion patterns.

[0006] This framework not only achieves technological innovation in parsing complex skeletal dynamics but also significantly improves the accuracy and sensitivity of identifying the effects of key skeletal nodes during movement. It also opens up a new approach for fine-grained identification of specific joint effects. Through rigorous testing on widely recognized datasets, empirical results demonstrate that this model can accurately capture the dynamics of key joints in fitness movements from multiple dimensions and effectively reveal the deep connection between athletic performance and joint function.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A fitness action recognition method based on a multi-branch fusion graph convolutional network, the method comprising the following steps:

[0009] Step 1) Generate the time series of the coordinates of the skeleton points to be tested based on the motion recognition video to be detected, and at the same time establish the graph adjacency matrix based on the set human skeleton image and split it into several matrices A k ;

[0010] Step 2) Preprocess the skeleton point coordinate time series to obtain input Z;

[0011] Step 3) Input the Z obtained in the previous step into the adaptive graph convolutional network (AGCN) and the temporal channel aggregation graph convolutional network (TCA-GCN) modules respectively to obtain spatiotemporal features at different levels and splice them as output;

[0012] Step 4) Introducing the Transformer as the core component of the framework, it integrates the heterogeneous features extracted by the two branch networks and adaptively enhances their complementarity and expressiveness, building a multi-branch feature fusion framework to achieve a deep understanding of complex sequence data;

[0013] Step 5) Input the joint coordinates, bone direction and length data into the spatiotemporal adaptive feature fusion graph convolutional network for training respectively, obtain the corresponding initial recognition results and softmax scores, and finally output the final result by weighted addition.

[0014] Furthermore, in step 1), the initial adjacency matrix A is established based on the human skeleton image, and the adjacency matrix is ​​split according to: A+I=∑ k A k , where A k They represent the adjacency matrices of the k-th hop, and A0=I, A1=A.

[0015] Furthermore, in step 2), the skeleton point coordinate time series is processed using kernel Γ u Perform temporal convolution to obtain temporal features.

[0016] Furthermore, it is concatenated with the original sequence to extract spatiotemporal features. All preprocessing can be written as:

[0017]

[0018] in, is the processed video representation, * represents temporal convolution, T represents the length of the time series, N represents the number of nodes, and C represents the number of feature channels. u represents the number of filters used for temporal convolution, V represents the time series of bone point coordinates, and Represent temporal convolution and feature concatenation respectively.

[0019] Furthermore, in step 3), in order to extract spatial features from the topological structure of the human skeleton, the results are subjected to graph convolution. The specific method is as follows: using an adaptive graph convolutional network (AGCN), applying a normalized embedded Gaussian function to calculate the similarity between two joints, which can not only reflect the connectivity between the two joints, but also determine the strength of the connection between them. The specific calculation process is expressed as follows:

[0020]

[0021] Among them, v i ,v j Represents two different nodes, and N is the number of nodes. If P represents the similarity matrix calculated after embedding the Gaussian function, the new adjacency matrix is ​​expressed as:

[0022]

[0023] The spatiotemporal graph convolutional network model consists of a batch normalization (BN) layer and a multi-scale graph convolution module. The multi-scale graph convolution module contains nine layers of feature extraction modules, whose output channels are 64, 64, 64, 128, 128, 128, 256, 256, and 256, respectively. Each layer of the feature extraction module includes a second residual module, as well as an adaptive spatial domain graph convolution unit, an activation function, an attention module, and a temporal domain graph convolution unit connected in sequence. The input of the second residual module is connected to the input of the adaptive spatial domain graph convolution unit, and the output is connected to the first feature extraction unit, which is used to perform an addition operation.

[0024] Furthermore, a temporal channel aggregation graph convolutional network (TCA-GCN) is proposed to dynamically learn spatial and temporal topology and effectively aggregate topological features of different time and channel dimensions for skeleton-based action recognition. This part uses channel topology modeling to dynamically obtain the spatial channel topology, which consists of a correlation modeling function and a refinement function. The correlation modeling function uses the input feature X∈R T×N×C Model channel correlations between joints to obtain channel-specific correlations C 1 Represents a specific number of channels. The refinement function adds the channel-related Q to the adjacency matrix obtained by spatial configuration partitioning to obtain the channel topology The specific method formula is as follows:

[0025] S=α·Q+μ(A k ) (4)

[0026] Among them, A k is the k-th channel adjacency matrix, μ(*) represents the normalization and dimension transformation operation of the third-order adjacency matrix, and α is a trainable parameter representing the connection strength.

[0027] In the temporal aggregation part, a temporal adaptive weight aggregation method is proposed based on temporal adaptive convolution to process skeleton features in the temporal dimension. This method generates calibrated weights in the temporal dimension based on the input features and aggregates them with the prior topology in the temporal dimension to complete the temporal dynamic topological representation.

[0028] Specifically, first, the skeleton sequence is used to generate the time weight as input X∈R T×N×C , use the generating function to multiply the initial weight, perform dimensional transformation to generate time weight, perform time aggregation with the prior topology, and obtain a new high-dimensional feature representation As shown in the formula:

[0029] A out =TA(W,X)=(W1X1)||...||(W T X T) (5)

[0030] Among them, W1X1 is expressed as: TA(*) represents the aggregation function, w(j) represents the generating function, x(j) represents the initial weight, W(*) represents the dimensionality reduction function, is the time weight feature, and its expression is:

[0031] W=α t ·W0 (6)

[0032] Among them, α t =G(x1,...,x t ), α t is the calibration weight with output channels, using different dimensional methods based on bone features, is the initial weight. In this part, the focus is on aggregating the spatiotemporal features of the first two parts to complete the balancing process. When aggregating in the channel dimension, the graph convolution in the channel dimension is expressed as follows:

[0033]

[0034] Among them, A out is the output feature of adaptive aggregation, B i Indicates v j The node range, S is the output feature of the channel direction topology, v i i is the i-th node, v j For the jth node, the final output result It is expressed as follows:

[0035]

[0036] in, Expressed as: CA is the channel aggregation function.

[0037] Furthermore, in step 4), a Transformer fusion network model is established, which inputs features from the two network models and outputs the fusion result. The output features of the two feature extractors are fused through a Transformer network based on multi-head cross attention. Using m-head cross attention, the Transformer operation process is expressed as:

[0038]

[0039] Among them, CA j is j head cross attention, B Qj for Linear projection of the appearance. Kj and E Vj yes Linear projection of dim represents B Qj and E Kj The dimension of W′ is the weight matrix of the multi-head attention. Then a linear layer is used, followed by a Tanh activation, and the residual connection and feedforward layer are superimposed, which is expressed as:

[0040]

[0041]

[0042] Where FFN stands for Feed Forward Network. F The loss between and the label is expressed as:

[0043] O F =tanh(Linear(F,θ F )) (13)

[0044] L inter =MSE(O F ,Y) (14)

[0045] Among them, L inter is the total loss function, and Y represents the label.

[0046] Furthermore, in step 5), the present invention experimentally verifies the framework on a public benchmark dataset and constructs a four-stream framework. In which, each of the four spatiotemporal graph convolutional network models is a group, corresponding to the four flow data types of J-Stream, B-Stream, JM-Stream, and BM-Stream. These four flow data types are: joint flow V', skeleton flow (where e is the bone vector for each bone, ), joint motion flow M joint (where m joint is the vector difference of each joint in adjacent frame images, ) and skeletal motion flow M bone (where m bone is the vector difference of each bone in adjacent frame images, Specifically, the first-order information (joint coordinates) and second-order information (bone orientation and length) are fed into four independent streams for processing. Subsequently, the human action video to be recognized is converted into four streams of data and fed into a trained spatiotemporal graph convolutional network model to obtain the corresponding initial prediction results and softmax scores, resulting in action scores, which are then used to predict action labels. Experimental results show that this framework significantly outperforms baseline models in tasks such as action recognition and advanced skeletal motion analysis.

[0047] Compared with the prior art, the beneficial effects of the present invention are mainly manifested in:

[0048] 1. This paper develops an innovative multi-branch fusion strategy. Through the dual-branch network structure of AGCN and TCA-GCN, it accurately extracts and integrates the high-order features of skeleton motion, deeply mines its semantic information, and provides an efficient analysis tool for understanding complex motion patterns.

[0049] 2. The present invention introduces a Transformer fusion module, which serves as a bridge connecting different features to achieve deep and detailed feature fusion. This strategy not only optimizes the extraction of motion state information and improves the accuracy of action recognition, but also provides an effective tool for the precise analysis of highly complex motion patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0051] Figure 2 It is a flow chart of input data preprocessing of the present invention;

[0052] Figure 3 This is a diagram of the AGCN network structure of the present invention;

[0053] Figure 4 This is a diagram of the TCA-GCN network structure of the present invention;

[0054] Figure 5 It is a schematic diagram of the Transformer fusion module of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below with reference to the accompanying drawings.

[0056] Reference Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 As shown in Table 1 and Table 2, the present invention describes a fitness action recognition method based on a multi-branch fusion graph convolutional network, comprising the following steps:

[0057] like Figure 1 FIG. 1 is a schematic diagram of the overall framework of the present invention. According to the present invention, a method for identifying fitness movements based on a multi-branch fusion graph convolutional network is provided, which includes the following steps:

[0058] Step 1) Convert the input video stream of T×N×C into the corresponding bone point coordinate time series, where T is the number of frames played in a video; N is the number of human skeleton joints; and C is the coordinate vector dimension of each joint.

[0059] The initial adjacency matrix established based on the human skeleton image is A, and the basis for splitting the adjacency matrix is: A+I=∑ k A k , where A k They represent the adjacency matrices of the k-th hop, and A0=I, A1=A.

[0060] Step 2) The position of the data can provide limited information, making it difficult to accurately identify human motion. In order to enrich the kinematic information of the input data, the present invention preprocesses the skeleton position sequence data, obtains the skeleton velocity sequence data and skeleton acceleration sequence data features, and Figure 2 The preprocessing process shown uses kernel Γ on the time series of skeleton point coordinates u Perform temporal convolution to obtain temporal features.

[0061] Specifically, it is concatenated with the original sequence to extract spatiotemporal features. All the initial processing can be written as:

[0062]

[0063] in, is the processed video representation, and * represents temporal convolution.

[0064] Step 3) According to Figure 3 and Figure 4 As shown in the figure, a network model combining AGCN and TCA-GCN is established. To extract spatial features from the topological structure of the human skeleton, the results are subjected to graph convolution. The specific method is as follows: a normalized embedded Gaussian function is applied to calculate the similarity between two joints. This not only reflects the connectivity between the two joints, but also determines the strength of the connection between them. The specific calculation process is expressed as follows:

[0065]

[0066] Among them, v i ,v j Represents two different nodes, and N is the number of nodes. If P represents the similarity matrix calculated after embedding the Gaussian function, the new adjacency matrix is ​​expressed as:

[0067]

[0068] The spatiotemporal graph convolutional network model consists of a batch normalization (BN) layer and a multi-scale graph convolution module. The multi-scale graph convolution module contains nine layers of feature extraction modules, whose output channels are 64, 64, 64, 128, 128, 128, 256, 256, and 256, respectively. Each layer of the feature extraction module includes a second residual module, as well as an adaptive spatial domain graph convolution unit, an activation function, an attention module, and a temporal domain graph convolution unit connected in sequence. The input of the second residual module is connected to the input of the adaptive spatial domain graph convolution unit, and the output is connected to the first feature extraction unit, which is used to perform an addition operation.

[0069] Specifically, a temporal channel aggregation graph convolutional network (TCA-GCN) is used to dynamically learn spatial and temporal topologies and effectively aggregate topological features of different time and channel dimensions for skeleton-based action recognition. This part uses channel topology modeling to dynamically obtain the spatial channel topology, which consists of a correlation modeling function and a refinement function. The correlation modeling function uses the input feature X∈R T×N×C Model channel correlations between joints to obtain channel-specific correlations The refinement function adds the channel correlation Q to the adjacency matrix obtained by spatial configuration partitioning to obtain the channel topology The specific method formula is as follows:

[0070] S=α·Q+μ(A k ) (4)

[0071] Among them, A k is the k-th channel adjacency matrix, μ(*) represents the normalization and dimension transformation operation of the third-order adjacency matrix, and α is a trainable parameter representing the connection strength.

[0072] In the temporal aggregation part, based on time-adaptive convolution, a time-adaptive weight aggregation method is proposed to process the skeleton features in the time dimension. This method can generate calibrated weights in the time dimension based on the input features and aggregate them with the prior topology in the time dimension to complete the temporal dynamic topological representation. Specifically, first, the skeleton sequence is used to generate the temporal weights as input X∈R T×N×C , use the generating function to multiply the initial weight, perform dimensional transformation to generate time weight, perform time aggregation with the prior topology, and obtain a new high-dimensional feature representation As shown in the formula:

[0073] A out =TA(W,X)=(W1X1)||...||(W T X T ) (5)

[0074] Among them, W1X1 is expressed as: TA(*) represents the aggregation function, W(*) represents the dimensionality reduction function, is the time weight feature, and its expression is:

[0075] W=α t ·W0 (6)

[0076] Among them, α t =G(x1,...,x t ), α t is the calibration weight with output channels, using different dimensional methods based on bone features, is the initial weight. In this part, the focus is on aggregating the spatiotemporal features of the first two parts to complete the balancing process. When aggregating in the channel dimension, the graph convolution in the channel dimension is expressed as follows:

[0077]

[0078] Among them, A out is the output feature of adaptive aggregation, S is the output feature of channel direction topology, v i For the i-th node, the final output result It is expressed as follows:

[0079]

[0080] in, Expressed as: CA is the channel aggregation function.

[0081] In step 4), according to Figure 5 As shown in Figure 1, a Transformer fusion network model is established to integrate the heterogeneous features output by the two models in step 3) and output the fusion result. The output features of the two models are fused through a Transformer network based on multi-head cross attention. Using m-head cross attention, the Transformer operation process is expressed as:

[0082]

[0083] Among them, CA j is j head cross attention, B Qj for Linear projection of the appearance. Kj and E Vj yes Linear projection of dim represents B Qj and E Kj The dimension of W′ is the weight matrix of the multi-head attention. Then a linear layer is used, followed by a Tanh activation, and the residual connection and feedforward layer are superimposed, which is expressed as:

[0084]

[0085] Where FFN stands for Feed Forward Network. F The loss between and the label is expressed as:

[0086] O F =tanh(Linear(F,θ F )) (13)

[0087] L inter =MSE(O F ,Y) (14)

[0088] Among them, L inter is the total loss function.

[0089] Step 5) As shown in Tables 1 and 2, the present invention experimentally verifies the present framework on public benchmark datasets and constructs a four-stream framework. In this framework, the multi-heart spatiotemporal attention graph convolutional network model contains twelve parallel spatiotemporal graph convolutional network models, each group of four, corresponding to the four flow data of J-Stream, B-Stream, JM-Stream, and BM-Stream respectively. Specifically, the first-order information (joint coordinates) and the second-order information (skeleton direction and length) are input into four independent streams for processing. Subsequently, the softmax scores of the four streams are weighted and fused to obtain the action score, and the action label is predicted based on it. This embodiment is compared with the advanced model in the top-1 accuracy (%) of the NTU-RGB+D 60 and NTU-RGB+D120 datasets. The present model has achieved the most advanced results in almost all benchmark tests.

[0090] In the model training phase, all experiments in this embodiment were carried out under the PyTorch deep learning framework and trained using a Tesla A100 GPU. The training parameters are as follows: the initial learning rate is 0.1, the weight decay is 0.0004, the parameters are adjusted using stochastic gradient descent (SGD) with a Nesterov momentum of 0.9, the maximum number of training rounds is set to 70 times, and the learning rate is divided by 10 in the 35th and 55th training stages to minimize the loss function, thereby generating a human motion sequence that is very close to the real motion data, ensuring that the model can capture and simulate human motion more accurately. Model training is a well-known technology for those skilled in the art, so it will not be described in detail.

[0091] Table 1: Comparison of top-1 accuracy (%) with state-of-the-art methods on the NTU-RGBD 60 dataset

[0092]

[0093] Table 2: Comparison of top-1 accuracy (%) with state-of-the-art methods on the NTU-RGBD 120 dataset

[0094]

[0095]

[0096] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A fitness action recognition method based on a multi-branch fusion graph convolutional network, characterized in that: The method comprises the following steps: Step 1) Generate the time series of the coordinates of the skeleton points to be detected based on the motion recognition video to be detected, and at the same time establish the graph adjacency matrix based on the set human skeleton image and split it into several matrices A k ; Step 2) Preprocess the skeleton point coordinate time series to obtain input Z; Step 3) Construct a spatiotemporal adaptive feature fusion graph convolutional network model. The spatiotemporal adaptive feature fusion graph convolutional network model includes an adaptive graph convolutional network (AGCN) model and a temporal channel aggregation graph convolutional network (TCA-GCN) model. The input Z obtained in step 2) is input into the two models respectively to obtain spatiotemporal features at different levels and splice them as output. Step 4) Construct a multi-branch feature fusion framework. The Transformer is introduced into the multi-branch feature fusion framework to integrate the heterogeneous features extracted by the two branch networks and adaptively enhance their complementarity and expression ability to achieve a deep understanding of complex sequence data. Step 5) Extract joint coordinates, bone direction and length data from the time series of the bone point coordinates to be tested, input the extracted data into the spatiotemporal adaptive feature fusion graph convolutional network for training, obtain the corresponding initial recognition results and softmax scores, and finally output the final result by weighted addition.

2. The method for fitness movement recognition based on a multi-branch fusion graph convolutional network according to claim 1, characterized in that: In step 1), the splitting of the adjacency matrix is ​​based on the following formula: A+I=∑ k A k , where A k They represent the adjacency matrices with a distance k from them, and A0=I, A1=A.

3. The method for fitness movement recognition based on a multi-branch fusion graph convolutional network according to claim 2, characterized in that: In step 2), the kernel Γ is used for the time series of the skeleton point coordinates u Perform temporal convolution to obtain temporal features, and concatenate them with the original sequence to extract spatiotemporal features; All preprocessing is expressed as: in, is the processed video representation, * represents temporal convolution, T represents the length of the time series, N represents the number of nodes, C represents the number of feature channels, and C u represents the number of filters used for temporal convolution, V represents the time series of bone point coordinates, and Represent temporal convolution and feature concatenation respectively.

4. The method for fitness movement recognition based on a multi-branch fusion graph convolutional network according to claim 2, wherein: In step 3), in order to extract spatial features from the topological structure of the human skeleton, the result is subjected to graph convolution, specifically as follows: a normalized embedded Gaussian function is applied to calculate the similarity between two joint points. The calculation process is expressed as follows: Among them, v i ,v j Represents two different nodes, N is the number of nodes; let P represent the similarity matrix calculated after embedding the Gaussian function, then the new adjacency matrix is ​​expressed as:

5. The method for fitness movement recognition based on a multi-branch fusion graph convolutional network according to claim 4, characterized in that: In the step 3), the spatiotemporal graph convolution network model includes a BN layer and a multi-scale graph convolution module; the multi-scale graph convolution module includes nine layers of feature extraction modules, each layer of feature extraction module includes a second residual module, and an adaptive spatial domain graph convolution unit, an activation function, an attention module, and a temporal domain graph convolution unit connected in sequence; the input end of the second residual module is connected to the input end of the adaptive spatial domain graph convolution unit, and the output end is connected to the first feature extraction unit, and the first feature extraction unit is used to perform an addition operation.

6. The method for fitness movement recognition based on a multi-branch fusion graph convolutional network according to claim 4, characterized in that: In step 3), the temporal channel aggregation graph convolutional network TCA-GCN model uses channel topology modeling to dynamically obtain the spatial channel topology, which includes a related modeling function and a refinement function; the related modeling function uses the input feature X∈R T ×N×C Model channel correlations between joints to obtain channel-specific correlations C 1 Indicates a specific number of channels; The refinement function adds the channel correlation Q to the adjacency matrix obtained by spatial configuration partitioning to obtain the channel topology The specific method formula is as follows: S=α·Q+μ(A k ) (4) Among them, A k is the k-th channel adjacency matrix, μ(*) represents the normalization and dimension transformation operation of the third-order adjacency matrix, α is a trainable parameter, and Q represents the connection strength.

7. The method for fitness movement recognition based on a multi-branch fusion graph convolutional network according to claim 4, characterized in that: In step 3), when the temporal channel aggregation graph convolutional network TCA-GCN model aggregates, the skeleton features of the time dimension are processed based on the time adaptive weight aggregation method; the time adaptive weight aggregation method generates calibrated weights in the time dimension based on the input features, and aggregates them with the prior topology in the time dimension, thereby completing the temporal dynamic topological representation; specifically as follows: First, the skeleton sequence generates the time weight as input X∈R T×N×C , use the generating function to multiply the initial weight, perform dimensional transformation to generate time weight, perform time aggregation with the prior topology, and obtain a new high-dimensional feature representation As shown in the formula: A out =TA(W,X)=(W1X1)||...||(W T X T ) (5) Among them, W1X1 is expressed as: TA(*) represents the aggregation function, w(j) represents the generating function, x(j) represents the initial weight, and W(*) represents the dimensionality reduction function. is the time weight feature, and its expression is: W=α t W0 (6) Among them, α t =G(x1,...,x t ), α t is the calibration weight with output channels, using different dimensional methods based on bone features, is the initial weight; when aggregating on the channel dimension, the graph convolution of the channel dimension is expressed as follows: Among them, A out is the output feature of adaptive aggregation, B i Indicates v j The node range, S is the output feature of the channel direction topology, v i is the i-th node, v j For the jth node, the final output result It is expressed as follows: in, Expressed as: CA is the channel aggregation function.

8. The method for fitness movement recognition based on a multi-branch fusion graph convolutional network according to claim 2, wherein: In step 4), a Transformer fusion network model is established to integrate the heterogeneous features output by the two models in step 3) and output a fusion result; specifically, the following: The output features of the two models are fused through a Transformer network based on multi-head cross attention; using m-head cross attention, the operation process of the Transformer is expressed as: Among them, CA j is j head cross attention, B Qj for Linear projection of appearance; E Kj and E Vj yes Linear projection of dim represents B Qj and E Kj The dimension of W′ is the weight matrix of multi-head attention; then a linear layer is used, followed by a Tanh activation, and the residual connection and feedforward layer are superimposed, which is expressed as: Among them, FFN stands for feed-forward network; calculation O F The loss between and the label is expressed as: O F =tanh(Linear(F,θ F )) (13) L inter =MSE(O F ,Y) (14) Among them, L inter is the total loss function, and Y represents the label.

9. The method for fitness movement recognition based on a multi-branch fusion graph convolutional network according to claim 2, wherein: In step 5), this framework is experimentally verified to establish a four-stream framework. In the four-stream framework, each four spatiotemporal graph convolutional network models form a group, corresponding to the four flow state data of joint stream J-Stream, skeleton stream B-Stream, ligament stream JM-Stream, and joint motion stream BM-Stream respectively; the specific process of step 5) is as follows: 1) Joint coordinates, bone orientation, and length data are input into four independent streams for processing; 2) Convert the human action video to be identified into four types of flow data and input them into the trained spatiotemporal graph convolutional network model to obtain the corresponding initial prediction results and softmax scores to obtain the action scores; 3) Predict action labels.

Images