A fine-grained basketball action recognition method based on global context awareness

Abstract

The present invention relates to a fine-grained basketball action recognition method based on global context perception, which belongs to the field of computer vision and comprises the following steps: S1: for a basketball action video, a MTSA module is used to aggregate frames of different distances in a time pyramid manner; S2: a SCI module is used to perform cross-dimensional interaction of features along two directions and channels in space respectively; S3: a GCA-Module is obtained by fusing the MTSA module and the SCI using dynamic convolution; S4: a GCA-Module block is introduced into a ResNet-50 to form a GCA-Block; S5: a plurality of GCA-Blocks are stacked to build a fine-grained basketball action recognition model GCA-Net; S6: a training data set is constructed; S7: the model is trained, predicted and optimized to obtain the trained model, and fine-grained basketball action recognition is performed on the basketball action video.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and relates to a fine-grained basketball action recognition method based on global context perception. Background Art

[0002] With the continuous development of science and technology, the competition in competitive sports reflects the competition of comprehensive scientific and technological strength of various countries to a large extent. Analyzing and diagnosing competitive sports through various scientific and technological means is an important means to overcome the traditional training mode of repeated guidance based on the subjective experience of coaches, and it is also an important way to improve the training level and competition results of athletes. At present, deep learning technology has brought many changes to the intelligent sports industry. Computer vision technology characterized by deep learning has been applied in many single-player sports (such as diving, skiing, gymnastics, etc.), and has played an important role in helping athletes achieve good results in competitions. However, the application of computer vision in multi-player ball sports (such as basketball) has just started. Among them, the use of computer vision methods for fine-grained action recognition in basketball videos is one of the research hotspots. It is also the basis for completing advanced applications such as automatic detection of key events in the game, intelligent analysis of skills and tactics, and automatic generation of highlights.

[0003] In the field of action recognition, fine-grained action recognition aims to distinguish subclasses of a broad action class, which have high similarity. In some current studies, the development of fine-grained action recognition is mainly constrained by two challenges: on the one hand, the difficulty is that fine-grained action recognition usually needs to find local small differences between a series of similar actions; on the other hand, the difficulty is that the current model lacks effective time domain modeling information. Basketball is a team competitive sport, and there are two main difficulties in fine-grained action recognition:

[0004] (1) Similarity of actions. Different from coarse-grained action recognition, fine-grained action recognition aims to distinguish a series of subclasses of broad action classes, which have high similarity and low inter-class differences. Since the inter-class differences of fine-grained actions are small, different actions in a certain frame may be similar. Therefore, it is difficult to effectively recognize fine-grained basketball actions only through a single frame or short-term temporal information.

[0005] (2) Background similarity. Different from coarse-grained action recognition, the scene information in the video can provide an effective basis for action recognition. In basketball videos, there are a lot of similar backgrounds, and there is also a lot of interference from off-court personnel, such as referees, spectators, coaches, etc., so it is difficult to focus on the players themselves where these fine-grained actions occur.

[0006] Therefore, compared with coarse-grained action recognition, fine-grained basketball action recognition is more challenging. Summary of the invention

[0007] In view of this, an object of the present invention is to provide a fine-grained basketball action recognition method based on global context perception.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A fine-grained basketball action recognition method based on global context perception includes the following steps:

[0010] S1: For basketball action videos, the multi-time scale aggregation strategy of the multi-time scale aggregation (MTSA) module is used to aggregate frames of different distances using a temporal pyramid.

[0011] S2: Using the hybrid attention mechanism of cross-dimensional interaction in the space-channel interaction (SCI) module, cross-dimensional interaction of features is performed along the two directions of space and the channel respectively;

[0012] S3: Using dynamic convolution, the output of the MTSA module is used as an adaptive convolution kernel to dynamically assign different weight values ​​to the feature map output by the SCI module, thereby fusing the two modules into a global context-aware module GCA-Module;

[0013] S4: The global context awareness module GCA-Module is introduced after the first 1×1 convolution of each ResNet-Block of the original ResNet-50 network structure to form a GCA-Block;

[0014] S5: Stack multiple GCA-Blocks to build a global context-aware network GCA-Net as a fine-grained basketball action recognition model;

[0015] S6: Using RGB basketball action video images as the input of the network, a sparse temporal sampling strategy is used to extract frames at equal intervals from the original basketball action video images, and the extracted frames are subjected to data augmentation by random horizontal flipping and corner cropping;

[0016] S7: In the training phase, ImageNet pre-trained weights are used for initialization, and the stochastic gradient descent algorithm is used to learn the parameters of the network; in the prediction phase, the softmax classifier is used to obtain the final classification probability of the category; in the optimization phase, the cross entropy loss function and the ternary loss function are jointly optimized to reduce the error between the true value and the predicted value; this step is repeated until the number of iterations reaches the set number of iterations, the model training is completed, and fine-grained basketball action recognition is performed on the basketball action video.

[0017] Further, step S1 specifically includes the following steps:

[0018] For the original input feature map X∈R C×T×H×W , the global average pooling operation is used to compress it in the spatial dimension, and the new feature map obtained is F∈R C×T , the feature vector contained in the feature map is F:{f 1 ,f 2 ,…,f T}, where C represents the number of channels, T represents the number of segments into which the video is segmented when sparse temporal sampling is used, and H and W represent the spatial size of the input feature map;

[0019] Then, a time pyramid is used to obtain contextual information at different time scales; the total number of layers in the pyramid is log 2 T, as the number of layers increases, the expansion rate gradually increases to cover different time scales; in the i-th layer of the time pyramid, the feature group F i The number of features contained in is 2 i-1 , F i It is expressed as:

[0020]

[0021]

[0022] in, Indicates the second layer of the i-th layer in the time pyramid i-1 feature vectors; Indicates that the expansion rate in the i-th layer of the time pyramid is r = 2 i-1 The convolution kernel of , in each layer, the size s of the convolution kernel and the corresponding expansion rate r satisfy s·r=T; * represents a one-dimensional convolution operation;

[0023] Then, the temporal context features obtained from different layers in the temporal pyramid are aggregated in a splicing manner as the updated temporal features F':

[0024]

[0025] Finally, two fully connected layers are used to aggregate the temporal features with a global receptive field; the weights are perceived by softmax to learn the interdependence of video frames at different time scales. The process is shown in the following formula:

[0026] w=softmax(W 2 (σ(W 1 F′))

[0027] Where w∈R C×(T-1)represents the weight value generated by the MTSA module, σ represents the ReLU activation function, and W 1 and W 2 are the learnable parameters of the two fully connected layers.

[0028] Further, step S2 specifically includes the following steps:

[0029] For the input feature map X∈R C×T×H×W , after dimension conversion, X H ∈R T×H×C×W Dimensional pooling is performed along the H direction to obtain a finer-grained tensor representation. The operation is expressed as:

[0030] Dimensional pooling(X)=concat[Avgpool 1d (X),Maxpool 1d (X)]

[0031] Among them, 1d represents the first dimension of the input feature map X, on which global average pooling GAP and global maximum pooling GMP are performed; concat means concatenating the pooled feature maps in the first dimension;

[0032] Through the above operations, information interaction between channels C and W is established, and the feature map obtained is X' H ∈R T×2×C×W ;

[0033] Then, information interaction between channels C and H is established to input feature map X∈R C×T×H×W , after dimension exchange, we get X W ∈R T×W×C×H ; Repeat the dimension pooling operation, and the final feature map obtained is X' W ∈R T×2×C×H ;

[0034] Then, the attention generation operation is performed; the feature X' generated after pooling H and X' W Each of them passes through a parameter-sharing two-dimensional convolution, and then passes through a batch normalization layer and a Sigmoid activation layer to obtain the corresponding weights; they are multiplied with the original feature X respectively;

[0035] Finally, the outputs generated in the above process are averaged to achieve feature aggregation. The whole process is expressed as:

[0036]

[0037] Among them, w H ∈R T×1×C×W and w W ∈RT×1×C×H represents the attention weight generated by the SCI module, σ represents the Sigmoid activation function, and · represents the dot product of the elements; and Indicates that in X H ·w H and X w ·w w Perform a dimension swap operation on the input to keep the original feature map X∈R C×T×H×W The dimensions are consistent; y∈R C×T×H×W Represents the final output produced by the space-channel interaction module.

[0038] Further, step S3 specifically includes the following steps:

[0039] The global context timing information w∈R generated by the MTSA module C×(T-1) As an adaptive convolution kernel, it is used to dynamically assign the feature map y∈R output by the SCI module C×T×H×W With different weights, the final output is:

[0040] Y=w'*y'

[0041] Among them, w'∈R C×1×(T-1)×1 is the adaptive weight value generated by the output w of the MTSA module after the Reshape operation. For each different input, its value is different; y'∈R 1×C×T×(HW) is the feature map generated by the Reshape operation after the feature map y activated by the SCI module; * represents the two-dimensional convolution operation; the final output Y∈R C×T×H×W It is a global context feature map obtained by adaptive convolution with dynamic kernel.

[0042] Furthermore, in the original ResNet-50 structure, there are a total of 4 stages, and different stages contain 3, 4, 6 and 3 original Resnet-Blocks respectively; according to the original structure of ResNet-50, GCA-Blocks are stacked in each stage according to the number of 3:4:6:3, and finally a global context-aware network GCA-Net is formed.

[0043] Furthermore, in the training phase, the enhanced RGB frames are fed into the global context-aware network for fine-grained action feature extraction, ImageNet pre-trained weights are used to speed up the network convergence, and the stochastic gradient descent algorithm is used to learn the network parameters; then the MultiStepLR method is used to dynamically adjust the learning rate changes;

[0044] In the prediction stage, for a target dataset with a total number of action categories C, a fully connected layer is used to obtain the final prediction vector I∈RC . Then, the softmax classifier is used to obtain the final classification probability of the category, that is:

[0045]

[0046] in, represents the predicted probability of action category i;

[0047] In the optimization stage, the cross entropy loss function and the ternary loss function are jointly optimized to reduce the error between the true value and the predicted value. The expression is:

[0048] Loss = (1-λ)L 1 +λL 2

[0049] Among them, L 1 represents the cross entropy loss function, L 2 represents the ternary loss function, and λ is a hyperparameter. The greater the difference between the predicted value and the true value, the greater the Loss value; through continuous iteration and optimization, the optimal Loss value is finally obtained to enhance the classification performance of the model.

[0050] Furthermore, the trained fine-grained basketball action recognition model can realize the recognition of 26 fine-grained basketball actions. The fine-grained basketball actions are divided into three categories, namely passing, dribbling and shooting. Each category contains more fine-grained basketball actions, among which:

[0051] Shooting is divided into 11 categories: free throws, tip-ins, layups, blocks, one-handed slam dunks, one-handed throws, standing jump shots, step-back jump shots, pull-up jump shots, turnaround jump shots, and two-handed slam dunks.

[0052] Passing is divided into 8 categories: hand-to-hand pass, one-handed rebound pass, one-handed chest pass, one-handed shoulder pass, one-handed side pass, two-handed rebound pass, two-handed chest pass, and two-handed overhead pass;

[0053] Dribbling is divided into seven categories: dribbling behind the back, dribbling with changing hands, dribbling with a change of direction in front of the body without changing hands, dribbling with a change of hands without changing direction in front of the body, dribbling between the legs, dribbling with changing hands and good dribbling.

[0054] The beneficial effects of the present invention are as follows: the fine-grained basketball action recognition algorithm based on global context perception proposed in the present invention introduces the MTSA module and the SCI module on the basis of the original ResNet-50 to focus on different aspects of the features. In the MTSA module, the time pyramid setting is adopted, and one-dimensional convolutions with different expansion rates are used to obtain the contextual connection of the video in the time domain dimension. In the SCI module, through the information interaction between two different dimensions of space and channel, richer category attributes and spatial information are obtained. Then, the long-term time information with a global receptive field is embedded in the short-term space-channel information in the form of convolution, and the final model can realize effective action discrimination under the guidance of global context information.

[0055] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 This is the overall structure diagram of the global context-aware network (GCA-Net);

[0058] Figure 2 It is the structural diagram of the multi-time scale aggregation (MTSA) module;

[0059] Figure 3 It is the structural diagram of the space-channel interaction (SCI) module;

[0060] Figure 4 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0062] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0063] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0064] See also Figure 1 to Figure 4 ,This invention proposes a fine-grained basketball action recognition method based on global context ,awareness, such as Figure 1 The designed global context-aware network is shown, which includes the following steps:

[0065] S1: To solve the problem of difficulty in judging similar basketball actions from short-term temporal information, a multi-time scale aggregation (MTSA) module is designed. In this module, a multi-time scale aggregation strategy is proposed. This strategy uses the setting of time pyramid to aggregate frames of different distances and enhance the model's ability to obtain long-term temporal information.

[0066] Figure 2 This is a schematic diagram of the multi-time scale aggregation module, showing the time pyramid structure when T=8. The total number of pyramid layers is log 2 T, the entire operation process in this module is:

[0067] For the original input feature map X∈R C×T×H×W , the global average pooling operation is used to compress it in the spatial dimension, and the new feature map obtained is F∈R C×T , the feature vector contained in the feature map is F:{f 1 ,f 2 ,…,f T}. Where C represents the number of channels, T represents the number of segments the video is segmented into when sparse temporal sampling is used, and H and W represent the spatial size of the input feature map.

[0068] Subsequently, the temporal pyramid is used to obtain temporal context information of different scales. In this module, dilated convolutions with different dilation rates are set. As the number of pyramid layers increases, the dilation rate gradually increases. In the i-th layer of the temporal pyramid, the feature group F i The number of features contained in is 2 i-1 , F i It can be expressed as:

[0069]

[0070]

[0071] in, Indicates the second layer of the i-th layer in the time pyramid i-1 feature vectors. Indicates that the expansion rate in the i-th layer of the time pyramid is r = 2 i-1 The convolution kernel of , in each layer, the size s of the convolution kernel and the corresponding expansion rate r satisfy s·r=T. Here, * represents a one-dimensional convolution operation.

[0072] Then, the temporal context features obtained from different layers in the temporal pyramid are aggregated in a splicing manner as the updated temporal features F':

[0073]

[0074] Finally, two fully connected layers are used to aggregate the temporal features with a global receptive field, and softmax is used to learn the interdependence of video frames at different time scales. The final features obtained are as follows:

[0075] w=softmax(W 2 (σ(W 1 F′))

[0076] Where w∈R C×(T-1) represents the weight value generated by the MTSA module, σ represents the ReLU activation function, and W 1 and W 2 are the learnable parameters of the two fully connected layers.

[0077] S2: To solve the problem of difficulty in extracting more discriminative feature representations from basketball videos with a large number of similar backgrounds, a spatial-channel interaction (SCI) module is designed. In this module, a hybrid attention mechanism with cross-dimensional interaction is introduced, which performs cross-dimensional interaction of features along the two directions of space and channels respectively, and complementarily improves the expression ability of important features in the network.

[0078] Figure 3 This is the structure diagram of the space-channel interaction module. The entire operation process in this module is:

[0079] For the original input feature map X∈R C×T×H×W The dimensions are swapped in different branches, and the new features obtained are X H ∈R T×H×C×W and X W ∈R T×W×C×H .

[0080] Subsequently, the above two types of features are subjected to dimensional pooling operations to obtain a rich tensor representation. Since GAP can cover the entire input image to obtain global information, it is susceptible to interference from cluttered backgrounds; while GMP can retain the most distinctive partial aggregation features in the input image and can effectively exclude background clutter, but it cannot contain global information from the entire input image. Therefore, for a given input X, a combination of global maximum pooling and global average pooling is used to reduce the adverse effects of using a single type of pooling operation. The operation can be expressed as:

[0081] Dimensional pooling(X)=concat[Avgpool 1d (X),Maxpool 1d (X)]

[0082] Among them, 1d represents the first dimension of the input feature map X, global average pooling (GAP) and global maximum pooling (GMP) are both performed on this dimension, and concat means concatenating the pooled feature maps in the first dimension.

[0083] Through the dimension pooling operation, the dependencies between the two different dimensions (C, W) and (C, H) can be captured respectively, and the final feature obtained is X' H ∈R T×2×C×W and X W ∈R T×W×C×H .

[0084] Then, the attention generation operation is performed. The feature X' generated by the left and right branches after pooling H and X' W Each of them is passed through a parameter-sharing 2D convolution, and then through a batch normalization layer and a sigmoid activation layer to obtain the corresponding weights. They are multiplied with the original features of the corresponding branch to accurately highlight the area of ​​interest. For example, for the large category of shooting, the network should pay more attention to the player holding the ball, the basketball, and the position around the basket, thereby providing a more effective judgment basis for the recognition of basketball actions.

[0085] Finally, the outputs of the two branches are averaged to achieve feature aggregation. The whole process can be expressed as:

[0086]

[0087] Among them, w H ∈R T×1×C×W and w W ∈R T×1×C×H represents the attention weight generated by the SCI module, σ represents the Sigmoid activation function, and · represents the dot product of the elements. and Indicates that in X H ·w H and X w ·w w Perform a dimension swap operation on the input to keep the original feature map X∈R C×T×H×W The dimensions of y∈R C×T×H×W Represents the final output produced by the space-channel interaction module.

[0088] S3: Using dynamic convolution, the long-term temporal information with global receptive field generated by the multi-time scale aggregation module is fused with the short-term space-channel information generated by the space-channel interaction module. The resulting global context-aware module (GCA-Module) has the ability to obtain richer spatiotemporal information.

[0089] A 2D convolution is used to aggregate the long-term temporal information w∈R with a global receptive field generated by the multi-time scale aggregation MTSA module. C×(T-1) As an adaptive convolution kernel, it is used to dynamically assign the enhanced feature map y∈R output by the spatial-channel interaction module C×T×H×W With different weights, the final output is:

[0090] Y=w'*y'

[0091] Among them, w'∈R C×1×(T-1)×1 is the adaptive weight value generated by the output w of the MTSA module after the Reshape operation. For each different input, its value is different. y'∈R 1×C×T×(H·W) is the feature map generated by the Reshape operation after the feature map y activated by the SCI module. The * here represents a two-dimensional convolution operation. The final output Y∈R C×T×H×W It is a global context feature map obtained by adaptive convolution with dynamic kernel.

[0092] S4: Based on the traditional 2D convolutional neural network ResNet-50, a global context-aware module (GCA-Module) is introduced after the first 1×1 convolution of each ResNet-Block to form a GCA-Block.

[0093] Specifically, in the original ResNet-50 structure, there are a total of 4 stages, and different stages contain 3, 4, 6, and 3 original Resnet-Blocks respectively. In the global context-aware network, according to the original structure of ResNet-50, GCA-Blocks are stacked in each stage in the ratio of 3:4:6:3 to finally form a global context-aware network. Among them, GCA-Block is constructed by introducing the global context-aware module after the first 1×1 of the original ResNet-Block in S4.

[0094] S5: By stacking multiple GCA-Blocks, a global context-aware network (GCA-Net) is built as a fine-grained basketball action recognition model.

[0095] like Figure 4 As shown in the figure, after building a fine-grained basketball action recognition model, its training, prediction and optimization process is as follows:

[0096] S6: For the input video, a sparse temporal sampling strategy is used to divide the video into multiple segments with equal spacing, and then a frame is randomly extracted from each segment, and the extracted frames are combined to represent the action category.

[0097] S7: The sampled video frames are subjected to data augmentation by random horizontal flipping and corner cropping. The size of each cropped frame is 224×224, which is used as input into the fine-grained basketball action recognition model for training.

[0098] S8: During the training phase, the enhanced RGB frames are fed into the global context-aware network for fine-grained action feature extraction. ImageNet pre-trained weights are used to speed up the network convergence, and the stochastic gradient descent algorithm is used to learn the network parameters. Then, the MultiStepLR method is used to dynamically adjust the learning rate.

[0099] S9: In the prediction stage, for a target dataset with a total number of action categories C, a fully connected layer is used to obtain the final prediction vector I∈R C . Then, the softmax classifier is used to obtain the final classification probability of the category, that is:

[0100]

[0101] in, represents the predicted probability of action category i.

[0102] S10: In the optimization stage, the cross entropy loss function and the ternary loss function are jointly optimized to reduce the error between the true value and the predicted value. The expression is:

[0103] Loss = (1-λ)L 1 +λL 2

[0104] Among them, L 1 represents the cross entropy loss function, L 2 represents the ternary loss function, and λ is a hyperparameter. The greater the difference between the predicted value and the true value, the greater the Loss value. Through continuous iteration and optimization, the optimal Loss value is finally obtained to enhance the classification performance of the model.

[0105] S11: Repeat S8 to S11. Each time an epoch is iterated, the model parameters of the current epoch are saved, and the classification accuracy of the current epoch is compared with the classification accuracy of the previous optimal model. When the set maximum epoch is reached, the fine-grained action classification model with the best recognition accuracy is output. The trained model can recognize 26 fine-grained basketball actions. Among them, these fine-grained basketball actions can be divided into three categories, namely passing, dribbling and shooting, and each category contains more fine-grained basketball actions. For example, shooting can be divided into 11 categories: free throws, tip-ins, layups, blocks, one-handed dunks, one-handed throws, stationary jump shots, step-back jump shots, pull-up jump shots, turnaround jump shots, and two-handed dunks; passing can be divided into 8 categories: hand-to-hand pass, one-handed rebound pass, one-handed chest pass, one-handed shoulder pass, one-handed side pass, two-handed rebound pass, two-handed chest pass, and two-handed overhead pass; dribbling can be divided into 7 categories: behind-the-back dribbling, change-of-hand dribbling, dribbling with a change of direction in front of the body without changing hands, dribbling with a change of hands without changing direction in front of the body, dribbling between the legs, change-of-hand dribbling, and good dribbling.

[0106] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which may include: ROM, RAM, disk or CD, etc.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A fine-grained basketball action recognition method based on global context awareness, characterized by: The following steps are involved: S1: For basketball action videos, the multi-time scale aggregation strategy of the multi-time scale aggregation module MTSA is used to aggregate frames of different distances using a time pyramid. S2: Using the hybrid attention mechanism of cross-dimensional interaction in the space-channel interaction module SCI, cross-dimensional interaction of features is performed along the two directions of space and the channel respectively; S3: Using dynamic convolution, the output of the MTSA module is used as an adaptive convolution kernel to dynamically assign different weight values ​​to the feature map output by the SCI module, thereby fusing the two modules into a global context-aware module GCA-Module; S4: The global context awareness module GCA-Module is introduced after the first 1×1 convolution of each ResNet-Block of the original ResNet-50 network structure to form a GCA-Block; S5: Stack multiple GCA-Blocks to build a global context-aware network GCA-Net as a fine-grained basketball action recognition model; S6: Using RGB basketball action video images as the input of the network, a sparse temporal sampling strategy is used to extract frames at equal intervals from the original basketball action video images, and the extracted frames are subjected to data augmentation by random horizontal flipping and corner cropping; S7: In the training phase, ImageNet pre-trained weights are used for initialization, and the stochastic gradient descent algorithm is used to learn the parameters of the network; in the prediction phase, the softmax classifier is used to obtain the final classification probability of the category; in the optimization phase, the cross entropy loss function and the ternary loss function are jointly optimized to reduce the error between the true value and the predicted value; this step is repeated until the number of iterations reaches the set number of iterations, the model training is completed, and fine-grained basketball action recognition is performed on the basketball action video.

2. The fine-grained basketball action recognition method based on global context perception according to claim 1 is characterized in that: Step S1 specifically includes the following steps: For the original input feature map X∈R C×T×H×W , the global average pooling operation is used to compress it in the spatial dimension, and the new feature map obtained is F∈R C×T , the feature vector contained in the feature map is F:{f1,f2,…,f T }, where C represents the number of channels, T represents the number of segments into which the video is segmented when sparse temporal sampling is used, and H and W represent the spatial size of the input feature map; Subsequently, a time pyramid is used to obtain contextual information at different time scales. The total number of layers in the pyramid is log2T. As the number of layers increases, the expansion rate gradually increases to cover different time scales. At the i-th layer of the time pyramid, the feature group F i The number of features contained in is 2 i-1 , F i It is expressed as: in, Indicates the second layer of the i-th layer in the time pyramid i-1 feature vectors; Indicates that the expansion rate in the i-th layer of the time pyramid is r = 2 i-1 The convolution kernel of , in each layer, the size s of the convolution kernel and the corresponding expansion rate r satisfy s·r=T; * represents a one-dimensional convolution operation; Then, the temporal context features obtained from different layers in the temporal pyramid are aggregated in a splicing manner as the updated temporal features F': Finally, two fully connected layers are used to aggregate the temporal features with a global receptive field; the weights are perceived by softmax to learn the interdependence of video frames at different time scales. The process is shown in the following formula: w = softmax(W2(σ(W1F′)) Where w∈R C×(T-1) represents the weight value generated by the MTSA module, σ represents the ReLU activation function, and W1 and W2 are the learnable parameters of the two fully connected layers.

3. The fine-grained basketball action recognition method based on global context perception according to claim 1 is characterized in that: Step S2 specifically includes the following steps: For the input feature map X∈R C×T×H×W , after dimension conversion, X H ∈R T×H×C×W Dimensional pooling is performed along the H direction to obtain a finer-grained tensor representation. The operation is expressed as: Dimensional pooling(X)=concat[Avgpool 1d (X),Maxpool 1d (X)] Among them, 1d represents the first dimension of the input feature map X, on which global average pooling GAP and global maximum pooling GMP are performed; concat means concatenating the pooled feature maps in the first dimension; Through the above operations, information interaction between channels C and W is established, and the feature map obtained is X' H ∈R T×2×C×W ; Then, information interaction between channels C and H is established to input feature map X∈R C×T×H×W , after dimension exchange, we get X W ∈R T×W×C×H ; Repeat the dimension pooling operation, and the final feature map obtained is X' W ∈R T×2×C×H ; Then, the attention generation operation is performed; the feature X' generated after pooling H and X' W Each of them passes through a parameter-sharing two-dimensional convolution, and then passes through a batch normalization layer and a Sigmoid activation layer to obtain the corresponding weights; they are multiplied with the original feature X respectively; Finally, the outputs generated in the above process are averaged to achieve feature aggregation. The whole process is expressed as: Among them, w H ∈R T×1×C×W and w W ∈R T×1×C×H represents the attention weight generated by the SCI module, σ represents the Sigmoid activation function, and · represents the dot product of the elements; and Indicated in and Perform a dimension swap operation on the input to keep the original feature map X∈R C×T×H×W The dimensions are consistent; y∈R C×T×H×W Represents the final output produced by the space-channel interaction module.

4. The fine-grained basketball action recognition method based on global context perception according to claim 1 is characterized in that: Step S3 specifically includes the following steps: The global context timing information w∈R generated by the MTSA module C×(T-1) As an adaptive convolution kernel, it is used to dynamically assign the feature map y∈R output by the SCI module C×T×H×W With different weights, the final output is: Y=w'*y' Among them, w'∈R C×1×(T-1)×1 is the adaptive weight value generated by the output w of the MTSA module after the Reshape operation. For each different input, its value is different; y'∈R 1×C×T×(H·W) is the feature map generated by the Reshape operation after the feature map y activated by the SCI module; * represents the two-dimensional convolution operation; the final output Y∈R C×T×H×W It is a global context feature map obtained by adaptive convolution with dynamic kernel.

5. The fine-grained basketball action recognition method based on global context perception according to claim 1 is characterized in that: In the original ResNet-50 structure, there are a total of 4 stages, with 3, 4, 6 and 3 original Resnet-Blocks in different stages respectively; according to the original structure of ResNet-50, GCA-Blocks are stacked in each stage in the ratio of 3:4:6:3, finally forming the global context-aware network GCA-Net.

6. The method for fine-grained basketball action recognition based on global context perception according to claim 1, characterized in that: In the training phase, the enhanced RGB frames are fed into the global context-aware network for fine-grained action feature extraction. ImageNet pre-trained weights are used to speed up the network convergence, and the stochastic gradient descent algorithm is used to learn the network parameters. Then the MultiStepLR method is used to dynamically adjust the learning rate. In the prediction stage, for a target dataset with a total number of action categories C, a fully connected layer is used to obtain the final prediction vector I∈R C ; Subsequently, the softmax classifier is used to obtain the final classification probability of the category, namely: in, represents the predicted probability of action category i; In the optimization stage, the cross entropy loss function and the ternary loss function are jointly optimized to reduce the error between the true value and the predicted value. The expression is: Loss = (1-λ)L1+λL2 Among them, L1 represents the cross entropy loss function, L2 represents the ternary loss function, and λ is a hyperparameter; the greater the difference between the predicted value and the true value, the greater the Loss value; through continuous iteration and optimization, the most ideal Loss value is finally obtained to enhance the classification performance of the model.

7. The method for fine-grained basketball action recognition based on global context perception according to claim 1, characterized in that: The trained fine-grained basketball action recognition model can recognize 26 fine-grained basketball actions. The fine-grained basketball actions are divided into three categories, namely passing, dribbling and shooting. Each category contains more fine-grained basketball actions, including: Shooting is divided into 11 categories: free throws, tip-ins, layups, blocks, one-handed slam dunks, one-handed throws, standing jump shots, step-back jump shots, pull-up jump shots, turnaround jump shots, and two-handed slam dunks. Passing is divided into 8 categories: hand-to-hand pass, one-handed rebound pass, one-handed chest pass, one-handed shoulder pass, one-handed side pass, two-handed rebound pass, two-handed chest pass, and two-handed overhead pass; Dribbling is divided into seven categories: dribbling behind the back, dribbling with changing hands, dribbling with a change of direction in front of the body without changing hands, dribbling with a change of hands without changing direction in front of the body, dribbling between the legs, dribbling with changing hands and good dribbling.

Images