Sports action recognition method based on 3D space-time attention and slowfast network

Abstract

The invention discloses a sports motion recognition method based on 3D space-time attention and a slowfast network, and the method comprises the steps: obtaining an original human body motion video, sampling the original human body motion video through different frame rates, generating Slow branch and Fast branch input images, and taking the Slow branch and Fast branch input images as training samples; constructing a dual-branch network based on spatio-temporal feature extraction; designing a loss function, and training the dual-branch network based on spatial-temporal feature extraction by using the training sample; and sampling a to-be-identified human body behavior video through different frame rates to obtain corresponding video frame images, and inputting the video frame images into the trained dual-branch network based on spatial-temporal feature extraction to obtain an identified human body behavior category. According to the sports action recognition method based on the 3D space-time attention and slowfast network, the problems that in the prior art, the locality of the dependency relationship between the features in the space-time dimension is captured, and the model complexity is high are solved.

Description

Technical Field

[0001] The invention belongs to the technical field of human behavior recognition, and in particular relates to a sports action recognition method based on 3D spatiotemporal attention and slowfast network. Background Art

[0002] Behavior recognition technology has been widely used in many fields, such as smart sports, athlete motion analysis, game tactics research, and intelligent training assistance. Sports videos contain a lot of colorful content, which is easier to attract users than text, images, and audio information. However, due to the strong spatiotemporal dynamics, diverse motion changes, and complex backgrounds of sports movements, traditional methods are difficult to fully capture the fine-grained information of the motion, thus affecting the recognition effect. Therefore, the key issue of current research is to efficiently integrate 3D spatiotemporal feature modeling and feature extraction for sports movements to improve the accuracy and robustness of motion recognition.

[0003] Human action recognition refers to the process of identifying, analyzing and understanding human actions, postures, motion trajectories, etc. in surveillance videos or images through computer vision and artificial intelligence technologies. In the process of learning video action features, it is necessary not only to consider the image information in the spatial dimension, but also to not ignore the timing information in the temporal dimension. However, existing methods capture the dependencies between long-term action features in the temporal dimension in a local manner and ignore the connection between the spatial dimension and the temporal dimension, which greatly increases the complexity and computational cost of the model. Summary of the invention

[0004] The purpose of the present invention is to provide a sports action recognition method based on 3D spatiotemporal attention and slowfast network, which solves the problems of locality and high model complexity in capturing the dependency relationship between features in the spatiotemporal dimension in the prior art.

[0005] The technical solution adopted by the present invention is a sports action recognition method based on 3D spatiotemporal attention and slowfast network, which specifically includes the following steps:

[0006] S1: Get the original human motion video, sample it at different frame rates, generate Slow branch and Fast branch input images, and use them as training samples;

[0007] S2: Construct a dual-branch network based on spatiotemporal feature extraction;

[0008] S3: Design a loss function and use training samples to train a dual-branch network based on spatiotemporal feature extraction;

[0009] S4: The human behavior video to be identified is sampled at different frame rates to obtain the corresponding video frame image, which is input into the trained dual-branch network based on spatiotemporal feature extraction to obtain the identified human behavior category.

[0010] The present invention is also characterized in that:

[0011] The specific process of S1 is as follows:

[0012] For each original video sequence X∈R T×H×W×3 Sampling is performed according to the time dimension to generate the Slow branch input image And Fast branch input image Where T represents the number of video frames, H represents the height of the video frame, W represents the width of the video frame, 3 represents the number of channels of the video frame, and T s =T / τ s , T f =T / τ f ; H×W is the spatial resolution of each frame;

[0013] At low frame rate τ s Sampling is performed to extract long-time-scale static feature images from the original video and generate the Slow branch input image. The formula is:

[0014] X slow =X[::τ s ] (1)

[0015] In the formula,: : τ s Indicates that every τ from the time dimension s Frame sampling 1 frame;

[0016] At high frame rate τ f Sampling is performed to capture short-time-scale dynamic feature images from the original video and generate the Fast branch input image. The formula is:

[0017] X fast =X[::τ f ] (2)

[0018] In the formula,: : τ f Indicates that every τ from the time dimension f Frame sampling 1 frame, τ f Usually less than τ s ;

[0019] The input data of the two branches of an original video is used as a set of training samples, each of which corresponds to an action category label y∈R c , where C is the total number of action categories.

[0020] In S2, a dual-branch network based on spatiotemporal feature extraction is constructed. The dual-branch network based on spatiotemporal feature extraction includes a slow branch and a fast branch. Each branch includes a feature extraction module and an attention mechanism module connected in sequence. A cross-branch interaction module is connected between the feature extraction modules of the slow branch and the fast branch. The attention mechanism modules of the two branches are connected to the classification module.

[0021] The feature extraction module is composed of Resnet50 as the backbone network. Resnet50 contains five sequentially connected blocks: res1~res5; res1 includes a 7×7 convolution layer and a maximum pooling layer. The 7×7 convolution layer preliminarily extracts the spatiotemporal features of the video frame, and then inputs the maximum pooling layer for spatial downsampling; res2 consists of a Bottleneck1 residual block and two Bottleneck2 residual blocks, res3 consists of a Bottleneck1 residual block and three Bottleneck2 residual blocks, res4 consists of a Bottleneck1 residual block and five Bottleneck2 residual blocks, and res5 consists of a Bottleneck1 residual block and two Bottleneck2 residual blocks; the Bottleneck1 residual block is composed of a 1×1 convolution layer connected to a 3×3 convolution layer and a 1×1 convolution layer, and then connected to a 1×1 convolution layer; the Bottleneck2 residual block is composed of a 1×1 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer.

[0022] The cross-branch interaction module includes a convolutional layer and a connection layer. The features extracted by the Fsat branch are aligned through the convolutional layer and then concatenated with the features extracted by each Block of the Slow branch through the connection layer to serve as the input of the next Block on the Slow branch.

[0023] The attention mechanism module includes a channel attention module and a spatial attention module connected in sequence; the channel attention module includes a maximum pooling layer, an average pooling layer, an MLP layer, and a sigmoid function; the spatial attention module includes a maximum pooling layer, an average pooling layer, a 7×7 convolution layer, and a sigmoid function;

[0024] The classification module includes a global average pooling layer, a fully connected layer, and a Softmax layer. First, the feature map is globally average pooled in the time and space dimensions, and then the global feature vectors of the Slow branch and the Fast branch are concatenated in the channel dimension. The concatenated features are linearly transformed through the fully connected layer to map the high-dimensional features to the dimension of the number of action categories. Finally, the output of the fully connected layer is converted into a category probability distribution through the Softmax layer.

[0025] The loss function in S3 uses the cross entropy loss function LCE , the expression is:

[0026] L CE = -log(p y ) (3)

[0027] In the formula, p y It is the predicted probability of the true category y by the dual-branch network based on spatiotemporal feature extraction;

[0028] The average loss for a training set of M samples is for:

[0029]

[0030] In S3, training samples are used to train the dual-branch network based on spatiotemporal feature extraction. The specific process is as follows:

[0031] S3.1: Input the two branch data of the training sample into the Slow branch and the Fast branch respectively; the Slow branch receives the low frame rate τ s Sampled video frames, the Fast branch receives high frame rate τ f Sampled video frames;

[0032] S3.2: The Slow branch gradually downsamples the input data through the feature extraction module to extract multi-scale global features in time and space dimensions; the Fast branch gradually downsamples the input data through the feature extraction module to extract multi-scale global features in time and space dimensions;

[0033] In the process of extracting multi-scale global features in the Slow branch and the Fast branch, the high-temporal-resolution information captured in the Fast branch is transferred to the Slow branch through the cross-branch interaction module to supplement the temporal dynamic details;

[0034] S3.3: Through the attention mechanism module, the global features extracted from the Fast branch and the Slow branch are enhanced with channel attention and spatial attention, and the corresponding spatiotemporal features are output respectively;

[0035] S3.4: Perform global average pooling on the spatiotemporal features output by the Slow branch and the Fast branch to obtain the feature vector of each path, concatenate the feature vectors of the two paths into a final global feature vector, and output the action category prediction result through the fully connected layer and the Softmax layer operation.

[0036] The Slow branch and the Fast branch in S3.2 extract multi-scale global features, which specifically include the following steps:

[0037] The processing of the Slow branch:

[0038] S3.2.1: Input the image into the Slow branch for feature extraction, and use the convolution operation to expand the image from 3 channels to 64 channels to obtain the expanded feature Y1 Slow :

[0039] Y1 Slo w=Conv3D(X Slow , W1 Slow ) (5)

[0040] Where Conv3D(·) represents the convolution operation, W1 Slow Represents the convolution kernel weight of the Slow branch convolution layer;

[0041] S3.2.2: The expanded feature Y1 Slow Using convolution for downsampling, the temporal dimension remains unchanged and the spatial resolution is reduced to 1 / 4 of the original input, expressed as:

[0042]

[0043] In the formula, represents the output feature of the maximum pooling layer, MaxPool3D(·) represents the maximum pooling operation, k represents the pooling window, the size is 3×3, and s represents the pooling operation compensation, the size is 2;

[0044] S3.2.3: Use Resnet50 to extract the downsampled features layer by layer. The core formula of each residual block is:

[0045]

[0046] In the formula, It is a combination of convolution operations, including 1×1, 3×3, and 1×1 convolution layers; ReLu() represents the activation function operation, and l represents the number of residual block layers; in high-level res4 and res5, non-degenerate convolution is enabled in the time dimension, and the kernel size is 3×3×3; the feature dimensions of the final output after all residual blocks are:

[0047]

[0048] Where Y Slow represents the output feature tensor of the Slow branch, T Slow Indicates the time frame number of the Slow branch output feature;

[0049] Fast branch processing process:

[0050] S3.2.1: Input the preliminary features into the Fast branch for feature extraction, and use the convolution operation to expand the preliminary features from 3 channels to 8 channels to obtain the expanded features Y1 Fast ;

[0051]

[0052] Where W1 Fast Represents the convolution kernel weight of the Fast branch convolution layer;

[0053] S3.2.2: The expanded feature Y1 Fast Using convolution for downsampling, the temporal dimension remains unchanged and the spatial resolution is reduced to 1 / 4 of the original input, expressed as:

[0054]

[0055] In the formula, Represents the output features of the maximum pooling layer;

[0056] S3.2.3: Use Resnet50 to extract the downsampled features layer by layer. The core formula of each residual block is:

[0057]

[0058] The feature dimension of the final output after all residual blocks is:

[0059]

[0060] In S3.2, the high-temporal-resolution information captured in the Fast branch is transferred to the Slow branch through the cross-branch interaction module to supplement the temporal dynamic details. Specifically:

[0061] In the cross-branch interaction module, the features of the Fast branch are aligned to match the time dimension of the Slow branch. The formula is:

[0062]

[0063] In the formula, represents the features after time alignment operation; W TimeConv represents the temporal convolution kernel, and stride is the ratio of temporal downsampling;

[0064] The aligned Fast branch features and Slow branch features are merged at the same time resolution. Feature interaction is performed after the network's maximum pooling layer, res2, res3, and res4. The features of the Fast branch and the Slow branch are horizontally connected and merged, which is expressed as:

[0065]

[0066] In the formula, The alignment operation of the Fast branch feature represented by Y Fuse，l Indicates the features after lateral connection in stage l; Y Slow，l Indicates the characteristics of the slow branch in stage l; Y Fast，l Represents the features of the Fast branch of stage l, where stage l refers to one of the maximum pooling layer, res2, res3, and res4.

[0067] S3.3 specifically includes the following steps:

[0068] S3.3.1: Input the global features of the Fast branch and the global features of the Slow branch into the attention mechanism module respectively. First, input them into the channel attention module to perform channel attention calculation:

[0069] Y c =σ(F avg +F max ) (15)

[0070] Y ca =Y c ⊙X (16)

[0071] Among them, F avg is the result of the feature after global average pooling and processing by the fully connected layer, F max is the result of the feature after global maximum pooling processed by the fully connected layer, X is the channel dimension of the input feature; Y c represents the channel attention weight; Y ca represents the features of the output of the channel attention module; ⊙ represents the dot multiplication operation; σ(·) represents the Sigmoid activation function;

[0072] Step 3.3.2: Input the features of the Fast branch and the Slow branch after the channel attention module into the spatial attention module to perform spatial attention calculation, which is expressed as:

[0073] Y s =σ(Conv3D(Y spatial ) (17)

[0074] Y spatial =Concat(Avg(Y ca , aim=1), Max(Y ca ,dim=1) (18)

[0075] Y sa =Y s ⊙Y ca (19)

[0076] Where Y s represents the spatial attention weight; Y spatial Indicates the concatenation of spatial features; Concat indicates the concatenation operation; Avg indicates the average pooling operation; Max indicates the maximum pooling operation; dim indicates the channel dimension of the operation; Y sa Represents the features output by the spatial attention module.

[0077] S3.4 specifically includes the following steps:

[0078] S3.4.1: The fused spatiotemporal features of the Slow branch and the spatiotemporal features of the Fast branch are globally averaged and pooled in time and space dimensions, expressed as:

[0079] f = GlobalAvgPool(Y sa ) (20)

[0080] Where f represents the global feature vector; GlobalAvgPool represents the global average pooling operation;

[0081] S3.4.2: Concatenate the corresponding global feature vectors of the Slow branch and the Fast branch along the channel dimension to form a joint global feature representation, expressed as:

[0082] f Concat =Concat(f Slow , f Fast ,dim=channel) (21)

[0083] In the formula, f Concat represents the joint feature vector after concatenation; f Fast represents the global feature vector of the Fast branch after global average pooling, f Slow It represents the global feature vector of the Slow branch after global average pooling, and channel represents the channel dimension of the feature;

[0084] S3.4.3: The concatenated joint feature vector f Concat Input to the fully connected layer for mapping to the dimension of the number of categories. The calculation formula of the fully connected layer is:

[0085] z=W fc ·f Concat +b fc (twenty two)

[0086] Where: W fc is the weight matrix of the fully connected layer, b fc is the bias vector, z is the unnormalized class score;

[0087] S3.4.4: The unnormalized scores are converted to class probability distributions through the Softmax function, expressed as:

[0088]

[0089] Where P represents the predicted probability of category i; N class Indicates the total number of categories corresponding to the training samples; z i Represents the unnormalized score of the i-th class of output;

[0090] S3.4.5: According to the predicted probability distribution, select the category with the highest probability as the final classification result, expressed as:

[0091]

[0092] In the formula, y pred Indicates the final classification result.

[0093] The beneficial effects of the present invention are:

[0094] The sports action recognition method based on 3D spatiotemporal attention and slowfast network introduces spatial attention and channel attention mechanisms through the attention enhancement module, effectively combines the global information of spatial dimension and time dimension, and makes up for the limitation of existing methods in capturing long-term action feature dependency in time dimension. The introduction of attention mechanism module can achieve significant performance improvement with less computational overhead, thereby effectively reducing the overall complexity of the model and effectively enhancing the action recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 It is the overall network diagram of the sports action recognition method based on 3D spatiotemporal attention and slowfast network of the present invention;

[0096] Figure 2 It is an architecture diagram of a feature extraction module in a sports action recognition method based on 3D spatiotemporal attention and slowfast network of the present invention;

[0097] Figure 3 It is an architecture diagram of the cross-branch interaction module in the sports action recognition method based on 3D spatiotemporal attention and slowfast network of the present invention;

[0098] Figure 4 It is an architecture diagram of the attention mechanism module in the sports action recognition method based on 3D spatiotemporal attention and slowfast network of the present invention;

[0099] Figure 5It is an architecture diagram of a classification module in a sports action recognition method based on 3D spatiotemporal attention and slowfast network of the present invention;

[0100] Figure 6 It is a video frame image of a human behavior recognition video;

[0101] Figure 7 It is a detection result diagram of a video frame diagram using the method of the present invention. DETAILED DESCRIPTION

[0102] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0103] The present invention provides a sports action recognition method based on 3D spatiotemporal attention and slowfast network, which specifically comprises the following steps:

[0104] S1: Obtain the original human motion video, sample the original human motion video at different frame rates, generate Slow branch and Fast branch input images, and use them as training samples.

[0105] For each original video sequence X∈R T×H×W×3 Sampling is performed according to the time dimension to generate the Slow branch input image And Fast branch input image Where T represents the number of video frames, H represents the height of the video frame, W represents the width of the video frame, 3 represents the number of channels of the video frame, and T s =T / τ s , T f =T / τ f ; H×W is the spatial resolution of each frame;

[0106] The Slow branch has a low frame rate τ s Sampling is performed to capture long-time-scale low-speed motion and extract long-time-scale static feature images from the original video. The formula is:

[0107] X slow =X[::τ s ] (1)

[0108] In the formula,: : τ s Indicates that every τ from the time dimension s Frame sampling 1 frame;

[0109] The Fast branch has a low frame rate τ f Sampling is performed to extract global features of static scenes and slow motions. The Fast branch is responsible for capturing fast motions in short time scales and capturing dynamic feature images in short time scales from the original video. The formula is:

[0110] X fast=X[::τ f ] (2)

[0111] In the formula,: : τ f Indicates that every τ from the time dimension f Frame sampling 1 frame, τ f Usually less than τ s The Fast branch input image can better capture the rapidly changing local dynamic features.

[0112] The input data of the two branches of an original video is used as a set of training samples, each of which corresponds to an action category label y∈R C , where C is the total number of action categories.

[0113] S2: Construct a dual-branch network based on spatiotemporal feature extraction.

[0114] The constructed dual-branch network based on spatiotemporal feature extraction includes a slow branch and a fast branch. Each branch includes a feature extraction module and an attention mechanism module connected in sequence. The feature extraction modules of the slow branch and the fast branch are connected by a cross-branch interaction module. The attention mechanism modules of the two branches are connected to the classification module. The overall architecture is shown in the figure below. Figure 1 shown.

[0115] The feature extraction module is composed of Resnet50 as the backbone network, such as Figure 2 As shown in the figure, Resnet50 contains five sequentially connected blocks: res1~res5; res1 includes a 7×7 convolution layer and a maximum pooling layer. The 7×7 convolution layer preliminarily extracts the spatiotemporal features of the video frame, and then inputs the maximum pooling layer for spatial downsampling; res2 consists of a Bottleneck1 residual block and two Bottleneck2 residual blocks, res3 consists of a Bottleneck1 residual block and three Bottleneck2 residual blocks, res4 consists of a Bottleneck1 residual block and five Bottleneck2 residual blocks, and res5 consists of a Bottleneck1 residual block and two Bottleneck2 residual blocks; the Bottleneck1 residual block is composed of a 1×1 convolution layer connected to a 3×3 convolution layer and a 1×1 convolution layer respectively, and then connected to a 1×1 convolution layer; the Bottleneck2 residual block is composed of a 1×1 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer.

[0116] The cross-branch interaction module includes convolutional layers and connection layers, such as Figure 3As shown in the figure, the features extracted by the Fsat branch are aligned through the convolution layer, and then concatenated with the features extracted by the Slow branch through each Block through the connection layer, and then used as the input of the next Block on the Slow branch. This allows the Slow path to receive high-temporal-resolution information from the Fast path.

[0117] Attention mechanism modules, such as Figure 4 As shown, it includes a channel attention module and a spatial attention module connected in sequence; the channel attention module includes a maximum pooling layer, an average pooling layer, an MLP layer, and a sigmoid function; the spatial attention module includes a maximum pooling layer, an average pooling layer, a 7×7 convolution layer, and a sigmoid function. The original feature map is input into the channel attention module for processing, and the obtained feature map is multiplied with the original feature map and input into the spatial attention module. The feature map processed by the spatial attention module is multiplied with its input feature map to obtain the final attention-enhanced feature.

[0118] Classification modules, such as Figure 5 As shown in the figure, it includes a global average pooling layer, a fully connected layer, and a Sottmax layer. First, the feature map is globally average pooled in the time and space dimensions, and then the global feature vectors of the Slow branch and the Fast branch are concatenated in the channel dimension. The concatenated features are linearly transformed through the fully connected layer to map the high-dimensional features to the dimension of the number of action categories. Finally, the output of the fully connected layer is converted into a category probability distribution through the Soffmax layer.

[0119] S3: Design a loss function and use training samples to train a dual-branch network based on spatiotemporal feature extraction.

[0120] Using the cross entropy loss function L CE , the expression is:

[0121] L CE = -log(p y ) (3)

[0122] In the formula, p y It is the predicted probability of the true category y by the dual-branch network based on spatiotemporal feature extraction;

[0123] The average loss for a training set of M samples is for:

[0124]

[0125] The training samples obtained by S1 are used to train the dual-branch network based on spatiotemporal feature extraction based on the above loss function. The specific process is as follows:

[0126] S3.1: Input the two branch data of the training sample into the Slow branch and the Fast branch respectively; the Slow branch receives the low frame rate τ s Sampled video frames, the Fast branch receives high frame rate τ f Sampled video frames.

[0127] S3.2: The Slow branch gradually downsamples the input data through the feature extraction module to extract multi-scale global features in time and space dimensions; the Fast branch gradually downsamples the input data through the feature extraction module to extract multi-scale global features in time and space dimensions.

[0128] The Slow branch and the Fast branch receive video frames at the same time, and their processing is parallel, as follows:

[0129] The processing of the Slow branch:

[0130] S3.2.1: Input the image into the Slow branch for feature extraction, and use the convolution operation to expand the image from 3 channels to 64 channels to obtain the expanded feature Y1 Slow :

[0131] Y1 Slo w=Conv3D(X Slow , W1 Slow ) (5)

[0132] Where Conv3D(·) represents the convolution operation, W1 Slow Represents the convolution kernel weight of the Slow branch convolution layer;

[0133] S3.2.2: The expanded feature Y1 Slow Using convolution for downsampling, the temporal dimension remains unchanged and the spatial resolution is reduced to 1 / 4 of the original input, expressed as:

[0134]

[0135] In the formula, represents the output feature of the maximum pooling layer, MaxPool3D(·) represents the maximum pooling operation, k represents the pooling window, the size is 3×3, and s represents the pooling operation compensation, the size is 2;

[0136] S3.2.3: Use Resnet50 to extract the downsampled features layer by layer. The core formula of each residual block is:

[0137]

[0138] In the formula, It is a combination of convolution operations, including 1×1, 3×3, and 1×1 convolution layers; ReLu() represents the activation function operation, and l represents the number of residual block layers; in high-level res4 and res5, non-degenerate convolution is enabled in the time dimension, and the kernel size is 3×3×3; the feature dimensions of the final output after all residual blocks are:

[0139]

[0140] Where Y Slow represents the output feature tensor of the Slow branch, T Slow Indicates the time frame number of the Slow branch output feature;

[0141] Fast branch processing process:

[0142] S3.2.1: Input the preliminary features into the Fast branch for feature extraction, and use the convolution operation to expand the preliminary features from 3 channels to 8 channels to obtain the expanded features Y1 Fast ;

[0143]

[0144] Where W1 Fast Represents the convolution kernel weight of the Fast branch convolution layer;

[0145] S3.2.2: The expanded feature Y1 Fast Using convolution for downsampling, the temporal dimension remains unchanged and the spatial resolution is reduced to 1 / 4 of the original input, expressed as:

[0146]

[0147] In the formula, Represents the output features of the maximum pooling layer;

[0148] S3.2.3: Use Resnet50 to extract the downsampled features layer by layer. The core formula of each residual block is:

[0149]

[0150] The feature dimension of the final output after all residual blocks is:

[0151]

[0152] In the process of extracting multi-scale global features from the Slow branch and the Fast branch, the high-temporal-resolution information captured in the Fast branch is transferred to the Slow branch through the cross-branch interaction module to supplement the temporal dynamic details. The specific process is as follows:

[0153] In the cross-branch interaction module, the features of the Fast branch are aligned to match the time dimension of the Slow branch. The formula is:

[0154]

[0155] In the formula, represents the features after time alignment operation; W TimeConv represents the temporal convolution kernel, and stride is the ratio of temporal downsampling;

[0156] The aligned Fast branch features and Slow branch features are merged at the same time resolution, and feature interaction is performed after the network's maximum pooling layer, res2, res3, and res4. The features of the Fast branch and the Slow branch are horizontally connected and merged, which is expressed as:

[0157]

[0158] In the formula, The alignment operation of the Fast branch feature represented by Y Fuse，l Indicates the features after lateral connection in stage l; Y Slow，l Indicates the characteristics of the slow branch in stage l; Y Fast，l Represents the features of the Fast branch of stage l, where stage l refers to one of the maximum pooling layer, res2, res3, and res4.

[0159] S3.3: Through the attention mechanism module, the global features extracted from the Fast branch and the Slow branch are enhanced with channel attention and spatial attention, and the corresponding spatiotemporal features are output respectively.

[0160] The specific steps include:

[0161] S3.3.1: Input the global features of the Fast branch and the global features of the Slow branch into the attention mechanism module respectively. First, input them into the channel attention module to perform channel attention calculation:

[0162] Y c =σ(F avg +F max ) (15)

[0163] Y ca =Y c ⊙X (16)

[0164] Among them, F avg is the result of the feature after global average pooling and processing by the fully connected layer, F max is the result of the feature after global maximum pooling processed by the fully connected layer, X is the channel dimension of the input feature; Yc represents the channel attention weight; Y ca represents the features of the output of the channel attention module; ⊙ represents the dot multiplication operation; σ(·) represents the Sigmoid activation function;

[0165] Step 3.3.2: Input the features of the Fast branch and the Slow branch after the channel attention module into the spatial attention module to perform spatial attention calculation, which is expressed as:

[0166] Y s =σ(Conv3D(Y spatial ) (17)

[0167] Y spatial =Concat(Avg(Y ca , dim=1),Max(Y ca ,dim=1) (18)

[0168] Y sa =Y s ⊙Y ca (19)

[0169] Where Y s represents the spatial attention weight; Y spatial Indicates the concatenation of spatial features; Concat indicates the concatenation operation; Avg indicates the average pooling operation; Max indicates the maximum pooling operation; dim indicates the channel dimension of the operation; Y sa Represents the features output by the spatial attention module.

[0170] S3.4: Perform global average pooling on the spatiotemporal features output by the Slow branch and the Fast branch to obtain the feature vector of each path, concatenate the feature vectors of the two paths into a final global feature vector, and output the action category prediction result through the fully connected layer and the Softmax layer operation.

[0171] The specific steps include:

[0172] S3.4.1: The fused spatiotemporal features of the Slow branch and the spatiotemporal features of the Fast branch are globally averaged and pooled in time and space dimensions, expressed as:

[0173] f = GlobalAvgPool(Y sa ) (20)

[0174] Where f represents the global feature vector; GlobalAvgPool represents the global average pooling operation;

[0175] S3.4.2: Concatenate the corresponding global feature vectors of the S1ow branch and the Fast branch along the channel dimension to form a joint global feature representation, expressed as:

[0176] f Concat =Concat(f Slow , f Fast ,dim=channel) (21)

[0177] In the formula, f Concat represents the joint feature vector after concatenation; f Fast represents the global feature vector of the Fast branch after global average pooling, f Slow It represents the global feature vector of the Slow branch after global average pooling, and channel represents the channel dimension of the feature;

[0178] S3.4.3: The concatenated joint feature vector f Concat Input to the fully connected layer for mapping to the dimension of the number of categories. The calculation formula of the fully connected layer is:

[0179] z=W fc ·f Concat +b fc (twenty two)

[0180] Where: W fc is the weight matrix of the fully connected layer, b fc is the bias vector, z is the unnormalized class score;

[0181] S3.4.4: The unnormalized scores are converted to class probability distributions through the Softmax function, expressed as:

[0182]

[0183] Where P represents the predicted probability of category i; N class Indicates the total number of categories corresponding to the training samples; z i Represents the unnormalized score of the i-th class of output;

[0184] S3.4.5: According to the predicted probability distribution, select the category with the highest probability as the final classification result, expressed as:

[0185]

[0186] In the formula, y pred Indicates the final classification result.

[0187] S4: The human behavior video to be identified is sampled at different frame rates to obtain the corresponding video frame image, which is input into the trained dual-branch network based on spatiotemporal feature extraction to obtain the identified human behavior category.

[0188] The sports action recognition method based on 3D spatiotemporal attention and slowfast network of the present invention is used to recognize 101 human action videos of the dataset UCF101, and the top1 accuracy (top1 _ acc), top5 accuracy (top5_acc), the results are shown in Table 1:

[0189] Table 1

[0190]

[0191] According to Table 1, the sports action recognition method based on 3D spatiotemporal attention and slowfast network has very good performance in terms of top 1 accuracy and top 5 accuracy measured on the dataset.

[0192] Example 1

[0193] This embodiment provides a sports action recognition method based on 3D spatiotemporal attention and slowfast network, which specifically includes the following steps:

[0194] S1: Get the original human motion video, sample it at different frame rates, generate Slow branch and Fast branch input images, and use them as training samples;

[0195] S2: Construct a dual-branch network based on spatiotemporal feature extraction;

[0196] S3: Design a loss function and use training samples to train a dual-branch network based on spatiotemporal feature extraction;

[0197] S4: The human behavior video to be identified is sampled at different frame rates to obtain the corresponding video frame image, which is input into the trained dual-branch network based on spatiotemporal feature extraction to obtain the identified human behavior category.

[0198] Example 2

[0199] On the basis of Example 1, the specific process of S1 is as follows:

[0200] For each original video sequence X∈R T×H×W×3 Sampling is performed according to the time dimension to generate the Slow branch input image And Fast branch input image Where T represents the number of video frames, H represents the height of the video frame, W represents the width of the video frame, 3 represents the number of channels of the video frame, and Ts =T / τ s , T f =T / τ f ; H×W is the spatial resolution of each frame;

[0201] At low frame rate τ s Sampling is performed to extract long-time-scale static feature images from the original video and generate the Slow branch input image. The formula is:

[0202] X slow =X[::τ s ] (1)

[0203] In the formula,: : τ s Indicates that every τ from the time dimension s Frame sampling 1 frame;

[0204] At high frame rate τ f Sampling is performed to capture short-time-scale dynamic feature images from the original video and generate the Fast branch input image. The formula is:

[0205] X fast =X[::τ f ] (2)

[0206] In the formula,: : τ f Indicates that every τ from the time dimension f Frame sampling 1 frame, τ f Usually less than τ s ;

[0207] The input images of the two branches of an original video are used as a set of training samples, each of which corresponds to an action category label y∈R C , where C is the total number of action categories.

[0208] Example 3

[0209] On the basis of Example 2, a dual-branch network based on spatiotemporal feature extraction is constructed, including a Slow branch and a Fast branch, each branch including a feature extraction module and an attention mechanism module connected in sequence; a cross-branch interaction module is connected between the feature extraction modules of the Slow branch and the Fast branch; the attention mechanism modules of the two branches are connected to the classification module;

[0210] The feature extraction module is composed of Resnet50 as the backbone network. Resnet50 includes five sequentially connected blocks: res1 to res5; res1 includes a 7×7 convolution layer and a maximum pooling layer. The 7×7 convolution layer preliminarily extracts the spatiotemporal features of the video frame and then inputs the maximum pooling layer for spatial downsampling; res2 is composed of a Bottleneck1 residual block and two Bottleneck2 residual blocks, res3 is composed of a Bottleneck1 residual block and three Bottleneck2 residual blocks, res4 is composed of a Bottleneck1 residual block and five Bottleneck2 residual blocks, and res5 is composed of a Bottleneck1 residual block and two Bottleneck2 residual blocks; the Bottleneck1 residual block is composed of a 1×1 convolution layer connected to a 3Δ3 convolution layer and a 1×1 convolution layer respectively, and then connected to a 1×1 convolution layer; the Bottleneck2 residual block is composed of a 1×1 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer connected;

[0211] The cross-branch interaction module includes a convolutional layer and a connection layer. The features extracted by the Fsat branch are aligned through the convolutional layer and then concatenated with the features extracted by each Block of the Slow branch through the connection layer to serve as the input of the next Block on the Slow branch.

[0212] The attention mechanism module includes a channel attention module and a spatial attention module connected in sequence; the channel attention module includes a maximum pooling layer, an average pooling layer, an MLP layer, and a sigmoid function; the spatial attention module includes a maximum pooling layer, an average pooling layer, a 7×7 convolution layer, and a sigmoid function;

[0213] The classification module includes a global average pooling layer, a fully connected layer, and a Softmax layer. First, the feature map is globally average pooled in the time and space dimensions, and then the global feature vectors of the Slow branch and the Fast branch are concatenated in the channel dimension. The concatenated features are linearly transformed through the fully connected layer to map the high-dimensional features to the dimension of the number of action categories. Finally, the output of the fully connected layer is converted into a category probability distribution through the Softmax layer.

[0214] Example 4

[0215] Based on Example 3, the loss function adopts the cross entropy loss function L CE , the expression is:

[0216] L CE = -log(p y ) (3)

[0217] In the formula, py It is the predicted probability of the true category y by the dual-branch network based on spatiotemporal feature extraction;

[0218] The average loss for a training set of M samples is for:

[0219]

[0220] Example 5

[0221] On the basis of Example 4, the dual-branch network based on spatiotemporal feature extraction is trained using training samples, and the specific process is as follows:

[0222] S3.1: Input the two branch data of the training sample into the Slow branch and the Fast branch respectively; the Slow branch receives the low frame rate τ s Sampled video frames, the Fast branch receives high frame rate τ f Sampled video frames;

[0223] S3.2: The Slow branch gradually downsamples the input data through the feature extraction module to extract multi-scale global features in time and space dimensions; the Fast branch gradually downsamples the input data through the feature extraction module to extract multi-scale global features in time and space dimensions;

[0224] The processing of the Slow branch:

[0225] S3.2.1: Input the image into the Slow branch for feature extraction, and use the convolution operation to expand the image from 3 channels to 64 channels to obtain the expanded feature Y1 Slow :

[0226]

[0227] Where Conv3D(·) represents the convolution operation, W1 Slow Represents the convolution kernel weight of the Slow branch convolution layer;

[0228] S3.2.2: The expanded feature Y1 Slow Using convolution for downsampling, the temporal dimension remains unchanged and the spatial resolution is reduced to 1 / 4 of the original input, expressed as:

[0229]

[0230] In the formula, represents the output feature of the maximum pooling layer, MaxPool3D(·) represents the maximum pooling operation, k represents the pooling window, the size is 3×3, and s represents the pooling operation compensation, the size is 2;

[0231] S3.2.3: Use Resnet50 to extract the downsampled features layer by layer. The core formula of each residual block is:

[0232]

[0233] In the formula, It is a combination of convolution operations, including 1×1, 3×3, and 1×1 convolution layers; ReLu() represents the activation function operation, and l represents the number of residual block layers; in high-level res4 and res5, non-degenerate convolution is enabled in the time dimension, and the kernel size is 3×3×3; the feature dimensions of the final output after all residual blocks are:

[0234]

[0235] Where Y Slow represents the output feature tensor of the Slow branch, T Slow Indicates the time frame number of the Slow branch output feature;

[0236] Fast branch processing process:

[0237] S3.2.1: Input the preliminary features into the Fast branch for feature extraction, and use the convolution operation to expand the preliminary features from 3 channels to 8 channels to obtain the expanded features Y1 Fast ;

[0238]

[0239] Where W1 Fast Represents the convolution kernel weight of the Fast branch convolution layer;

[0240] S3.2.2: The expanded feature Y1 Fast Using convolution for downsampling, the temporal dimension remains unchanged and the spatial resolution is reduced to 1 / 4 of the original input, expressed as:

[0241]

[0242] In the formula, Represents the output features of the maximum pooling layer;

[0243] S3.2.3: Use Resnet50 to extract the downsampled features layer by layer. The core formula of each residual block is:

[0244]

[0245] The feature dimension of the final output after all residual blocks is:

[0246]

[0247] In the process of extracting multi-scale global features in the Slow branch and the Fast branch, the high-temporal-resolution information captured in the Fast branch is transferred to the Slow branch through the cross-branch interaction module to supplement the temporal dynamic details;

[0248] In the cross-branch interaction module, the features of the Fast branch are aligned to match the time dimension of the Slow branch. The formula is:

[0249]

[0250] In the formula, represents the features after time alignment operation; W TimeConv represents the temporal convolution kernel, and stride is the ratio of temporal downsampling;

[0251] The aligned Fast branch features and Slow branch features are merged at the same time resolution, and feature interaction is performed after the network's maximum pooling layer, res2, res3, and res4. The features of the Fast branch and the Slow branch are horizontally connected and merged, which is expressed as:

[0252]

[0253] In the formula, The alignment operation Y of the Fast branch feature is represented by Fuse，l Represents the feature Y after the l-stage lateral connection Slow，l The feature Y representing the slow branch of stage l Fast，l Represents the features of the Fast branch of stage l, where stage l refers to one of the maximum pooling layer, res2, res3, and res4.

[0254] S3.3: Through the attention mechanism module, the global features extracted from the Fast branch and the Slow branch are enhanced with channel attention and spatial attention, and the corresponding spatiotemporal features are output respectively;

[0255] S3.3.1: Input the global features of the Fast branch and the global features of the Slow branch into the attention mechanism module respectively. First, input them into the channel attention module to perform channel attention calculation:

[0256] Y c =σ(F avg +F max ) (15)

[0257] Y ca =Y c ⊙X (16)

[0258] Among them, F avg is the result of the feature after global average pooling and processing by the fully connected layer, F max is the result of the feature after global maximum pooling processed by the fully connected layer, X is the channel dimension of the input feature; Y c represents the channel attention weight; Y ca represents the features of the output of the channel attention module; ⊙ represents the dot multiplication operation; σ(·) represents the Sigmoid activation function;

[0259] Step 3.3.2: Input the features of the Fast branch and the Slow branch after the channel attention module into the spatial attention module to perform spatial attention calculation, which is expressed as:

[0260] Y s =σ(Conv3D(Y spatial ) (17)

[0261] Y spatial =Concat(Avg(Y ca , dim=1),Max(Y ca ,dim=1) (18)

[0262] Y sa =Y s ⊙Y ca (19)

[0263] Where Y s represents the spatial attention weight; Y spatial Indicates the concatenation of spatial features; Concat indicates the concatenation operation; Avg indicates the average pooling operation; Max indicates the maximum pooling operation; dim indicates the channel dimension of the operation; Y sa Represents the features output by the spatial attention module.

[0264] S3.4: Perform global average pooling on the spatiotemporal features output by the Slow branch and the Fast branch to obtain the feature vector of each path, concatenate the feature vectors of the two paths into a final global feature vector, and output the action category prediction result through the fully connected layer and the Softmax layer operation.

[0265] S3.4.1: The fused spatiotemporal features of the Slow branch and the spatiotemporal features of the Fast branch are globally averaged and pooled in time and space dimensions, expressed as:

[0266] f = GlobalAvgPool(Y sa ) (20)

[0267] Where f represents the global feature vector; GlobalAvgPool represents the global average pooling operation;

[0268] S3.4.2: Concatenate the corresponding global feature vectors of the Slow branch and the Fast branch along the channel dimension to form a joint global feature representation, expressed as:

[0269] f Concat =Concat(f slow , f Fast ,dim=channel) (21)

[0270] In the formula, f Concat represents the joint feature vector after concatenation; f Fast represents the global feature vector of the Fast branch after global average pooling, f Slow It represents the global feature vector of the Slow branch after global average pooling, and channel represents the channel dimension of the feature;

[0271] S3.4.3: The concatenated joint feature vector f Concat Input to the fully connected layer for mapping to the dimension of the number of categories. The calculation formula of the fully connected layer is:

[0272] z=W fc ·f Concat +b fc (twenty two)

[0273] Where: W fc is the weight matrix of the fully connected layer, b fc is the bias vector, z is the unnormalized class score;

[0274] S3.4.4: The unnormalized scores are converted to class probability distributions through the Softmax function, expressed as:

[0275]

[0276] Where P represents the predicted probability of category i; N class Indicates the total number of categories corresponding to the training samples; z i Represents the unnormalized score of the i-th class of output;

[0277] S3.4.5: According to the predicted probability distribution, select the category with the highest probability as the final classification result, expressed as:

[0278]

[0279] In the formula, y pred Indicates the final classification result.

[0280] Example 6

[0281] The sports action recognition method based on 3D spatiotemporal attention and slowfast network of the present invention is used for Figure 6 The video screen shown is identified, and the identification result screen is as follows Figure 7 shown.

[0282] according to Figure 6 , Figure 7 By comparison, it can be seen that the sports action recognition method of the present invention has a higher recognition accuracy. The human behavior recognition technology of the present invention can more effectively recognize human motion information in sports videos, thereby improving the understanding of video scenes.

Claims

1. A sports action recognition method based on 3D spatiotemporal attention and slowfast network, characterized in that: The specific steps include: SI: Get the original human motion video, sample it at different frame rates, generate Slow branch and Fast branch input images, and use them as training samples; S2: Construct a dual-branch network based on spatiotemporal feature extraction; S3: Design a loss function and use training samples to train a dual-branch network based on spatiotemporal feature extraction; S4: The human behavior video to be identified is sampled at different frame rates to obtain the corresponding video frame image, which is input into the trained dual-branch network based on spatiotemporal feature extraction to obtain the identified human behavior category.

2. The sports action recognition method based on 3D spatiotemporal attention and slowfast network according to claim 1, characterized in that: The specific process of S1 is as follows: For each original video sequence X∈R T×H×W×3 Sampling is performed according to the time dimension to generate the Slow branch input image And Fast branch input image Where T represents the number of video frames, H represents the height of the video frame, W represents the width of the video frame, 3 represents the number of channels of the video frame, and T s =T / τ s , T f =T / τ f ; H×W is the spatial resolution of each frame; At low frame rate τ s Sampling is performed to extract long-time-scale static feature images from the original video and generate the Slow branch input image. The formula is: X slow =X[::τ s ] (1) In the formula,: : τ s Indicates that every τ from the time dimension s Frame sampling 1 frame; At high frame rate τ f Sampling is performed to capture short-time-scale dynamic feature images from the original video and generate the Fast branch input image. The formula is: X fast =X[::τ f ] (2) In the formula,: : τ f Indicates that every τ from the time dimension f Frame sampling 1 frame, τ f Usually less than τ s ; The input images of the two branches of an original video are used as a set of training samples, each of which corresponds to an action category label y∈R C , where C is the total number of action categories.

3. The sports action recognition method based on 3D spatiotemporal attention and slowfast network according to claim 1, characterized in that: In S2, a dual-branch network based on spatiotemporal feature extraction is constructed, wherein the dual-branch network based on spatiotemporal feature extraction includes a Slow branch and a Fast branch, each branch includes a feature extraction module and an attention mechanism module connected in sequence; a cross-branch interaction module is connected between the feature extraction modules of the Slow branch and the Fast branch; and the attention mechanism modules of the two branches are connected to the classification module; The feature extraction module is composed of Resnet50 as the backbone network. Resnet50 includes five sequentially connected blocks: res1~res5; res1 includes a 7×7 convolution layer and a maximum pooling layer. The 7×7 convolution layer preliminarily extracts the spatiotemporal features of the video frame and then inputs the maximum pooling layer for spatial downsampling; res2 is composed of a Bottleneck1 residual block and two Bottleneck2 residual blocks, res3 is composed of a Bottleneck1 residual block and three Bottleneck2 residual blocks, res4 is composed of a Bottleneck1 residual block and five Bottleneck2 residual blocks, and res5 is composed of a Bottleneck1 residual block and two Bottleneck2 residual blocks; the Bottleneck1 residual block is composed of a 1×1 convolution layer connected to a 3×3 convolution layer and a 1×1 convolution layer respectively, and then connected to a 1×1 convolution layer; the Bottleneck2 residual block is composed of a 1×1 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer connected; The cross-branch interaction module includes a convolution layer and a connection layer. The features extracted by the Fsat branch are aligned through the convolution layer and then concatenated with the features extracted by the Slow branch through each Block through the connection layer as the input of the next Block on the Slow branch. The attention mechanism module includes a channel attention module and a spatial attention module connected in sequence; the channel attention module includes a maximum pooling layer, an average pooling layer, an MLP layer, and a sigmoid function; the spatial attention module includes a maximum pooling layer, an average pooling layer, a 7×7 convolution layer, and a sigmoid function; The classification module includes a global average pooling layer, a fully connected layer, and a Softmax layer; first, the feature map is globally average pooled in the time and space dimensions, and then the global feature vectors of the Slow branch and the Fast branch are spliced ​​in the channel dimension; the spliced ​​features are linearly transformed through the fully connected layer to map the high-dimensional features to the dimension of the number of action categories, and finally the output of the fully connected layer is converted into a category probability distribution through the Softmax layer.

4. The sports action recognition method based on 3D spatiotemporal attention and slowfast network according to claim 1, characterized in that: The loss function described in S3 adopts the cross entropy loss function L CE , the expression is: t CE =-log(p y ) (3) In the formula, p y It is the predicted probability of the true category y by the dual-branch network based on spatiotemporal feature extraction; The average loss for a training set of M samples is for:

5. The sports action recognition method based on 3D spatiotemporal attention and slowfast network according to claim 1, characterized in that: In S3, training samples are used to train the dual-branch network based on spatiotemporal feature extraction. The specific process is as follows: S3.1: Input the two branch data of the training sample into the Slow branch and the Fast branch respectively; the Slow branch receives the low frame rate τ s Sampled video frames, the Fast branch receives high frame rate τ f Sampled video frames; S3.2: The Slow branch gradually downsamples the input data through the feature extraction module to extract multi-scale global features in time and space dimensions; the Fast branch gradually downsamples the input data through the feature extraction module to extract multi-scale global features in time and space dimensions; In the process of extracting multi-scale global features in the Slow branch and the Fast branch, the high-temporal-resolution information captured in the Fast branch is transferred to the Slow branch through the cross-branch interaction module to supplement the temporal dynamic details; S3.3: Through the attention mechanism module, the global features extracted from the Fast branch and the Slow branch are enhanced with channel attention and spatial attention, and the corresponding spatiotemporal features are output respectively; S3.4: Perform global average pooling on the spatiotemporal features output by the Slow branch and the Fast branch to obtain the feature vector of each path, concatenate the feature vectors of the two paths into a final global feature vector, and output the action category prediction result through the fully connected layer and the Softmax layer operation.

6. The sports action recognition method based on 3D spatiotemporal attention and slowfast network according to claim 5, characterized in that: The Slow branch and the Fast branch in S3.2 extract multi-scale global features, which specifically include the following steps: The processing of the Slow branch: S3.2.1: Input the image into the Slow branch for feature extraction, and use the convolution operation to expand the image from 3 channels to 64 channels to obtain the expanded feature Y1 Slow : Where Conv3D(·) represents the convolution operation, W1 Slow Represents the convolution kernel weight of the Slow branch convolution layer; S3.2.2: The expanded feature Y1 Slow Using convolution for downsampling, the temporal dimension remains unchanged and the spatial resolution is reduced to 1 / 4 of the original input, expressed as: In the formula, represents the output feature of the maximum pooling layer, MaxPool3D(·) represents the maximum pooling operation, k represents the pooling window, the size is 3×3, and s represents the pooling operation compensation, the size is 2; S3.2.3: Use Resnet50 to extract the downsampled features layer by layer. The core formula of each residual block is: In the formula, It is a combination of convolution operations, including 1×1, 3×3, and 1×1 convolution layers; ReLu() represents the activation function operation, and l represents the number of residual block layers; in high-level res4 and res5, non-degenerate convolution is enabled in the time dimension, and the kernel size is 3×3×3; the feature dimensions of the final output after all residual blocks are: Where Y Slow represents the output feature tensor of the Slow branch, T Slow Indicates the time frame number of the Slow branch output feature; Fast branch processing process: S3.2.1: Input the preliminary features into the Fast branch for feature extraction, and use the convolution operation to expand the preliminary features from 3 channels to 8 channels to obtain the expanded features Y1 Fast ; Where W1 Fast Represents the convolution kernel weight of the Fast branch convolution layer; S3.2.2: The expanded feature Y1 Fast Using convolution for downsampling, the temporal dimension remains unchanged and the spatial resolution is reduced to 1 / 4 of the original input, expressed as: In the formula, Represents the output features of the maximum pooling layer; S3.2.3: Use Resnet50 to extract the downsampled features layer by layer. The core formula of each residual block is: The feature dimension of the final output after all residual blocks is:

7. The sports action recognition method based on 3D spatiotemporal attention and slowfast network according to claim 6, characterized in that: In S3.2, the high-temporal-resolution information captured in the Fast branch is transferred to the Slow branch through the cross-branch interaction module to supplement the temporal dynamic details. Specifically: In the cross-branch interaction module, the features of the Fast branch are aligned to match the time dimension of the Slow branch. The formula is: In the formula, represents the features after time alignment operation; W TimeConv represents the temporal convolution kernel, and stride is the ratio of temporal downsampling; The aligned Fast branch features and Slow branch features are merged at the same time resolution. Feature interaction is performed after the network's maximum pooling layer, res2, res3, and res4. The features of the Fast branch and the Slow branch are horizontally connected and merged, which is expressed as: In the formula, The alignment operation of the Fast branch feature represented by Y Fuse，l Indicates the features after lateral connection in stage l; Y Slow，l Represents the characteristics of the slow branch in stage l; Y Fast，l Represents the features of the Fast branch of stage l, where stage l refers to one of the maximum pooling layer, res2, res3, and res4.

8. The sports action recognition method based on 3D spatiotemporal attention and slowfast network according to claim 7, characterized in that: S3.3 specifically includes the following steps: S3.3.1: Input the global features of the Fast branch and the global features of the Slow branch into the attention mechanism module respectively. First, input them into the channel attention module to perform channel attention calculation: Y c =σ(F avg +F max ) (15) AND ca =And c ⊙X (16) Among them, F avg is the result of the feature after global average pooling and processing by the fully connected layer, F max is the result of the feature after global maximum pooling processed by the fully connected layer, X is the channel dimension of the input feature; Y c represents the channel attention weight; Y ca represents the features of the output of the channel attention module; ⊙ represents the dot multiplication operation; σ(·) represents the Sigmoid activation function; Step 3.3.2: Input the features of the Fast branch and the Slow branch after the channel attention module into the spatial attention module to perform spatial attention calculation, which is expressed as: AND s =σ(Conv3D(Y spatial ) (17) AND spatial =Concat(Avg(Y ca (dim) 1), Max(Y ca ,dim=1) (18) AND sa =And s ⊙And ca (19) Where Y s represents the spatial attention weight; Y spatial Indicates the concatenation of spatial features; Concat indicates the concatenation operation; Avg indicates the average pooling operation; Max indicates the maximum pooling operation; dim indicates the channel dimension of the operation; Y sa Represents the features output by the spatial attention module.

9. The sports action recognition method based on 3D spatiotemporal attention and slowfast network according to claim 8, characterized in that: S3.4 specifically includes the following steps: S3.4.1: The fused spatiotemporal features of the Slow branch and the spatiotemporal features of the Fast branch are globally averaged and pooled in time and space dimensions, expressed as: f=GlobalAvgPool(Y sa ) (20) Where f represents the global feature vector; GlobalAvgPool represents the global average pooling operation; S3.4.2: Concatenate the corresponding global feature vectors of the Slow branch and the Fast branch along the channel dimension to form a joint global feature representation, expressed as: f Concat =Concat(f Slow ,f Fast ,dim=channel) (21) In the formula, f Concat represents the joint feature vector after concatenation; f Fast represents the global feature vector of the Fast branch after global average pooling, f Slow It represents the global feature vector of the Slow branch after global average pooling, and channel represents the channel dimension of the feature; S3.4.3: The concatenated joint feature vector f Concat Input to the fully connected layer for mapping to the dimension of the number of categories. The calculation formula of the fully connected layer is: z=W fc ·F Concat +b fc (22) Where: W fc is the weight matrix of the fully connected layer, b fc is the bias vector, z is the unnormalized class score; S3.4.4: The unnormalized scores are converted to class probability distributions through the Softmax function, expressed as: Where P represents the predicted probability of category i; N class Indicates the total number of categories corresponding to the training samples; z i Represents the unnormalized score of the i-th class of output; S3.4.5: According to the predicted probability distribution, select the category with the highest probability as the final classification result, expressed as: In the formula, y pred Indicates the final classification result.

Images