Action recognition method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to an action recognition method, which comprises the steps of extracting an original action fragment from a to-be-recognized video; carrying out downsampling processing on the original action segments to obtain action segments with a unified frame number; segmenting each frame of picture of the action fragment into a preset number of picture blocks; inputting each picture block into an action recognition model, wherein the action recognition model comprises an embedding layer, a spatial-temporal feature attention layer and a classification layer; linearly mapping each picture block into an embedded vector through an embedded layer to obtain an initial embedded vector of the picture block; performing attention weight calculation on the initial embedding vector of the picture block through a spatio-temporal feature attention layer to obtain a fusion feature embedding vector; and classifying the fusion feature embedding vectors through a classification layer. The invention further provides an action recognition device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the action fragments can be stored in a block chain. The accuracy and the reliability of action recognition are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a motion recognition method, device, computer equipment and storage medium. Background Art

[0002] In the training and competition of badminton, it is of great significance to accurately monitor and evaluate the athletes' movements. As an emerging method, video badminton action recognition technology can capture and analyze the athletes' badminton movements in detail through video analysis, thereby providing objective and quantitative data support for coaches. The application of this technology enables coaches to provide targeted guidance to athletes' training based on detailed data, helping athletes to improve and optimize their action techniques, thereby improving their competitive level.

[0003] Moreover, by continuously monitoring the badminton movements of athletes, it is also possible to promptly detect possible problems in the athletes' movement habits, such as poor movement patterns or a tendency to over-rely on certain muscle groups, which are often important factors leading to sports injuries. Therefore, video badminton movement recognition technology can warn of potential sports injury risks at an early stage and provide effective protection measures for athletes.

[0004] In the rehabilitation stage of athletes, video badminton motion recognition technology also plays an irreplaceable role. By tracking the rehabilitation progress of athletes, coaches can ensure the scientificity and effectiveness of rehabilitation training, adjust rehabilitation plans in time, and promote the rapid recovery of athletes.

[0005] However, the existing badminton batting motion recognition technology still has some limitations. Traditional batting motion prediction methods mainly focus on the motion itself, but in actual games, many batting types have little difference in form, and it is difficult to accurately distinguish batting motions based on the motion itself. Summary of the invention

[0006] The purpose of the embodiments of the present application is to propose an action recognition method, device, computer equipment and storage medium to solve the technical problem that the existing badminton action recognition technology only focuses on the action itself, making it difficult to accurately and effectively distinguish the hitting action.

[0007] In order to solve the above technical problems, the present application provides an action recognition method, which adopts the following technical solution:

[0008] Obtain a video to be identified, and extract an original action segment from the video to be identified;

[0009] Downsampling the original action clips to obtain uniform action clips with a preset target frame number;

[0010] Using an image segmentation algorithm to segment each frame of the action clip to obtain a preset number of image blocks;

[0011] Inputting each of the image blocks into a trained action recognition model, wherein the action recognition model includes an embedding layer, a spatiotemporal feature attention layer, and a classification layer;

[0012] Linearly mapping each of the picture blocks into an embedding vector through the embedding layer to obtain an initial embedding vector of the picture block;

[0013] By using the spatiotemporal feature attention layer, the time attention mechanism and the space attention mechanism are used to calculate the attention weight of the initial embedding vector of the picture block to obtain a fused feature embedding vector that fuses the time feature and the space feature;

[0014] The fused feature embedding vector is input into the classification layer to obtain an action recognition result.

[0015] Furthermore, the step of extracting the original action clip from the video to be identified includes:

[0016] Preprocessing the video to be identified to obtain enhanced video data;

[0017] According to a preset motion action model, a template matching algorithm is used to analyze the video frames of the enhanced video data to identify the start frame and the end frame of the motion action;

[0018] Analyze the video frames between the start frame and the end frame by using a target detection algorithm to determine the start position coordinates and the end position coordinates;

[0019] If the number of video frames between the start frame and the end frame is greater than a preset frame number threshold, the motion action is determined to be complete, otherwise the motion action is determined to be incomplete;

[0020] When determining that the motion is complete, tracking the motion process using a motion target tracking algorithm according to the starting position coordinates and the ending position coordinates to obtain a complete motion trajectory;

[0021] The original action segments are determined according to the complete action trajectory, and the original action segments are sorted in chronological order to generate an action segment set.

[0022] Furthermore, the step of downsampling the original action clips to obtain action clips with a uniform preset target frame number includes:

[0023] Obtaining original frame rate and resolution information of the original action clip;

[0024] Calculating a downsampling ratio of the original action clip according to a preset target number of frames;

[0025] A linear interpolation algorithm is used to downsample the video frames of the original action clip according to the downsampling ratio to obtain an action clip with a preset target frame number.

[0026] Furthermore, the spatiotemporal feature attention layer includes a temporal attention network and a spatial attention network, and the step of using the temporal attention mechanism and the spatial attention mechanism to calculate the attention weight of the initial embedding vector of the picture block to obtain a fused feature embedding vector that fuses the temporal feature and the spatial feature includes:

[0027] Inputting the initial embedding vector of the picture block into the temporal attention network and the spatial attention network respectively;

[0028] By using the temporal attention network, the temporal feature attention weight of each picture block in each layer of the temporal attention network in the corresponding frame is calculated using the temporal attention mechanism to obtain a temporal feature vector of the picture block;

[0029] By using the spatial attention network, the spatial feature attention weight of each picture block in each layer of the spatial attention network in the corresponding frame is calculated using the spatial attention mechanism to obtain a spatial feature vector of the picture block;

[0030] The temporal feature vector of the picture block and the spatial feature vector of the picture block are fused to obtain a fused feature embedding vector.

[0031] Furthermore, the step of using the temporal attention mechanism to calculate the temporal feature attention weight of each picture block in each layer of the temporal attention network in the corresponding frame to obtain the temporal feature vector of the picture block includes:

[0032] The Softmax function is used to calculate the time attention weight matrix between each of the image blocks and all frames;

[0033] According to the temporal attention weight matrix, weighted summation is performed on the initial embedding vector of each of the picture blocks to obtain the temporal weighted features of each of the picture blocks on all frames;

[0034] Using the time-weighted features, calculating a new time embedding vector for each of the picture blocks;

[0035] The new temporal embedding vector is used as the input of the next layer of the temporal attention network, and the above steps are repeated until all layers of the temporal attention network are processed to obtain the temporal attention embedding vector of each image block in all frames of the last layer of the temporal attention network;

[0036] The temporal attention embedding vectors of all the picture blocks are concatenated to obtain a temporal feature vector of the picture block.

[0037] Furthermore, the step of using the spatial attention mechanism to calculate the spatial feature attention weight of each image block in each layer of the spatial attention network in the corresponding frame includes:

[0038] Calculating the spatial relationship between each of the picture blocks according to the initial embedding vector of the picture block;

[0039] The Softmax function is used to normalize the spatial relationship to obtain the spatial attention weight;

[0040] According to the spatial attention weight, weighted summation is performed on the initial embedding vector of each of the picture blocks to obtain a spatial weighted feature of each of the picture blocks on all frames;

[0041] Using the spatial weighted features, calculating a new spatial embedding vector for each of the picture blocks;

[0042] The new spatial embedding vector is used as the input of the next layer of the spatial attention network, and the above steps are repeated until all layers of the spatial attention network are processed to obtain the spatial attention embedding vector of each image block in the last layer of the spatial attention network on all frames;

[0043] The spatial attention embedding vectors of all the picture blocks are concatenated to obtain a spatial feature vector of the picture block.

[0044] Furthermore, the step of inputting the fused feature embedding vector into the classification layer to obtain the action recognition result includes:

[0045] The fused feature embedding vector is input into the classification layer, and the predicted probability distribution of the hitting action corresponding to the action clip is obtained through forward propagation calculation;

[0046] According to the predicted probability distribution, the action type that the action clip most likely corresponds to is determined as an action recognition result.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a motion recognition device, which adopts the following technical solution:

[0048] An acquisition module is used to acquire a video to be identified and extract an original action segment from the video to be identified;

[0049] A downsampling module, used for downsampling the original action clips to obtain action clips with a uniform preset target frame number;

[0050] A segmentation module, used to segment each frame of the action clip using an image segmentation algorithm to obtain a preset number of picture blocks;

[0051] An input module, used to input each of the image blocks into a trained action recognition model, wherein the action recognition model includes an embedding layer, a spatiotemporal feature attention layer, and a classification layer;

[0052] An embedding module, used for linearly mapping each of the picture blocks into an embedding vector through the embedding layer to obtain an initial embedding vector of the picture block;

[0053] An attention calculation module is used to calculate the attention weight of the initial embedding vector of the picture block by using the time attention mechanism and the space attention mechanism through the time-space feature attention layer to obtain a fused feature embedding vector that fuses the time feature and the space feature;

[0054] The classification module is used to input the fusion feature embedding vector into the classification layer to obtain the action recognition result.

[0055] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0056] The computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned action recognition method when executing the computer-readable instructions.

[0057] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0058] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the action recognition method described above are implemented.

[0059] Compared with the prior art, this application has the following beneficial effects:

[0060] The present application provides an action recognition method, by downsampling the original action fragment extracted from the video to be recognized, obtaining the action fragment with the uniform frame number, optimizing the data processing and analysis process, and improving the stability and robustness of the algorithm; by adopting the image segmentation algorithm to segment each frame of the action fragment, it is possible to focus on the key action area more accurately, which is helpful to analyze the action details more carefully, improve the accuracy of action recognition and enhance the robustness of the algorithm; by inputting each picture block into the trained action recognition model, combining the time and space attention mechanism for feature extraction and fusion, it is possible to improve the feature extraction ability, help capture the spatial relationship and time relationship of the action, dynamically adjust the weight, and enhance the model's processing ability for long sequences, improve the model's interpretability, enhance the model's generalization ability, and finally classify the fused feature embedding vector through the classification layer, realize accurate recognition of the action, and significantly improve the accuracy and reliability of action recognition. The present application can not only be used for badminton analysis and teaching, but also can be extended to other similar sports action recognition fields, providing new technical support for intelligent sports analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0063] Figure 2 is a flow chart of an embodiment of the action recognition method according to the present application;

[0064] Figure 3 is a structural schematic diagram of an embodiment of a motion recognition device according to the present application;

[0065] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0069] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0070] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0071] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0072] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0073] It should be noted that the action recognition method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the action recognition device is generally arranged in the server / terminal device.

[0074] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0075] Continue to refer Figure 2 , shows a flow chart of an embodiment of the action recognition method according to the present application, comprising the following steps:

[0076] Step S201, obtaining a video to be identified, and extracting an original action segment from the video to be identified.

[0077] The sports video data to be analyzed is obtained as the video to be identified. The video to be identified may be a game video or a training video of various sports. Taking badminton as an example, the video to be identified is a badminton game video or a training video.

[0078] In this embodiment, the electronic device (eg, Figure 1 The server / terminal device shown in the figure can receive the acquired video to be identified through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0079] In some optional implementations of this embodiment, the step of extracting the original action segment from the video to be identified includes:

[0080] Preprocess the video to be identified to obtain enhanced video data;

[0081] According to the preset motion action model, the template matching algorithm is used to analyze the video frames of the enhanced video data to identify the start frame and the end frame of the motion action;

[0082] The video frames between the start frame and the end frame are analyzed by the target detection algorithm to determine the start position coordinates and the end position coordinates;

[0083] If the number of video frames between the start frame and the end frame is greater than the preset frame number threshold, the motion action is determined to be complete, otherwise the motion action is determined to be incomplete;

[0084] When judging that the motion is complete, the motion process is tracked using a motion target tracking algorithm based on the starting position coordinates and the ending position coordinates to obtain a complete motion trajectory;

[0085] The original action clips are determined according to the complete action trajectory, and the original action clips are sorted in chronological order to generate an action clip set.

[0086] It is determined whether the acquired video to be identified is in a preset standard format. If not, the video to be identified is converted into the preset standard format. Then, the video to be identified is preprocessed, and the preprocessing includes denoising and enhancement. Specifically, the video to be identified is denoised using a wavelet transform algorithm to remove noise interference in the video and obtain denoised video data; the denoised video data is enhanced using algorithms such as histogram equalization and contrast enhancement to improve the quality and clarity of the video and obtain enhanced video data.

[0087] In this embodiment, the enhanced video data is input into a preset motion action model (such as a batting action model), and video frame images are extracted frame by frame; feature extraction is performed on each video frame image to obtain a feature vector of each video frame; a similarity matching calculation is performed between the feature vector of the video frame and the feature vector of the preset motion action model; if the similarity is greater than a preset threshold, the video frame is judged to be a target action frame (batting action frame), and the frame number of the video frame is recorded; based on continuous motion action frames (such as batting action frames), the starting frame number and the ending frame number of the motion action (such as the batting action) are determined; and the identified starting frame number and the ending frame number are output as the starting frame and the ending frame of the motion action.

[0088] The target detection algorithms include but are not limited to KLT tracking algorithm, active contour model, CamShift algorithm, YOLO (You Only Look Once) algorithm, posture estimation algorithm, etc. You can select the appropriate algorithm according to the specific application scenario and requirements.

[0089] In badminton, the starting position coordinates are the hitting position coordinates of the hitting party, the ending position coordinates are the receiving position coordinates of the receiving party, the starting frame is the starting frame of the hitting action of the hitting party, and the ending frame is the starting frame of the receiving action of the receiving party. When the number of video frames between the starting frame and the ending frame is greater than the preset frame number threshold, the hitting action is considered to be complete. For a complete hitting action, a moving target tracking algorithm is used to track the hitting process according to the hitting position coordinates and the receiving position coordinates to obtain a complete hitting action trajectory.

[0090] In some optional implementations, the action segment set is post-processed, including removing duplicate action segments and filtering out action segments that meet the duration requirement.

[0091] By extracting action clips from the video to be identified, the video quality can be improved, motion movements can be accurately located, data management and analysis can be facilitated, and the subsequent data processing efficiency can be improved.

[0092] Step S202 , downsampling the original action clips to obtain action clips with a uniform preset target frame number.

[0093] Specifically, the original frame rate and resolution information of the original action clip are obtained; according to a preset target frame number, the downsampling ratio of the original action clip is calculated; and a linear interpolation algorithm is used to downsample the video frames of the original action clip according to the downsampling ratio to obtain an action clip with a preset target frame number.

[0094] Assume that the preset target frame number is T, that is, the frame number of each action segment is T, and the input action segment is represented by X∈R H ×W×3×T , where H and W represent the height and width of each frame; R represents all action clips.

[0095] In the downsampling process, if the current frame is a key frame, the frame is retained without downsampling; the downsampled frame is re-encoded to obtain an action clip with a uniform number of frames. In this embodiment, the steps of downsampling the video frames of the original action clip according to the downsampling ratio using a linear interpolation algorithm include:

[0096] According to the preset downsampling ratio, the number of key frames and the inter-frame interval that need to be retained are determined; the original video frame sequence of the original action clip is traversed, and the retained key frames are extracted according to the determined inter-frame interval, and the timestamp information of each key frame is recorded; a linear interpolation algorithm is used to insert a number of transition frames between two adjacent key frames according to the timestamp information to achieve a smooth transition between adjacent key frames; the interpolated transition frames and the extracted key frames are re-encoded to obtain an action clip with a preset target number of frames after downsampling.

[0097] In some optional implementations, the bit rate and resolution of action clips with a uniform number of frames are adjusted according to the encoding parameters of the original video to ensure image quality.

[0098] By downsampling the original action clips and unifying the frame number, the amount of data can be reduced, the computational complexity can be lowered, the processing speed can be increased, while key action information can be maintained and storage and transmission efficiency can be optimized.

[0099] It should be emphasized that in order to further ensure the privacy and security of the action clips, the above action clips can also be stored in a blockchain node.

[0100] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0101] Step S203: Use an image segmentation algorithm to segment each frame of the action clip to obtain a preset number of picture blocks.

[0102] In this embodiment, the number of frames to be processed is determined according to the video frame rate of the action clip; for each frame, a preset image segmentation algorithm is used to segment each frame. The image segmentation algorithm needs to adaptively adjust the segmentation granularity according to the image content; after processing by the image segmentation algorithm, it is determined whether the number of image blocks obtained by segmenting each frame is equal to the fixed value preset number. If not, the parameters of the image segmentation algorithm are adjusted and segmentation is performed again until the number of image blocks is met.

[0103] Assume that the dimension of each picture is H×W, and divide each frame into p blocks. There are T pictures in total (one frame is one picture). The divided picture blocks are represented as follows:

[0104]

[0105] Among them, x (p,t) Represents the pth picture block of the tth frame.

[0106] Specifically, the frequency domain information of each frame of the image is extracted by Fourier transform to obtain the texture features of the image; the Canny operator is used to detect the edge of the image to obtain the edge features; the HSV color space is used to quantize the image to obtain the color features; the image complexity is determined according to the texture features, edge features and color features; the segmentation granularity is determined according to the complexity of the image, where the higher the complexity, the smaller the initial segmentation granularity; the image is segmented by the GrabCut algorithm based on graph cuts, and the preliminary segmentation results are obtained by the maximum flow minimum cut algorithm; the color consistency of the pixels in the image area and the integrity of the image area boundary are used as evaluation indicators to evaluate the segmentation quality. When the segmentation quality score is greater than the preset score, it is considered that the segmentation result meets the expectations and is directly output; otherwise, according to the difference between the current score and the target score, the segmentation granularity is adjusted by the gradient descent method, the image is segmented again, and the segmentation quality is evaluated again, and it is iterated until the segmentation quality requirements are met or the maximum number of iterations is reached. Through this adaptive adjustment strategy, the segmentation accuracy can be improved as much as possible while ensuring the segmentation efficiency.

[0107] Step S204, input each image block into the trained action recognition model, wherein the action recognition model includes an embedding layer, a spatiotemporal feature attention layer and a classification layer.

[0108] In this embodiment, the action recognition model is pre-trained. The action recognition model includes an embedding layer, a spatiotemporal feature attention layer and a classification layer. The image block is converted into an embedding vector through the embedding layer, and then the temporal features and spatial features are extracted through the temporal self-attention mechanism and the spatial self-attention mechanism of the spatiotemporal feature attention layer. Finally, the classification layer performs action prediction based on the embedding vector that integrates the temporal features and the spatial features.

[0109] When training the action recognition model, a large number of action clip videos are obtained, and each action clip video is labeled with the real batting action type, and then divided into training sets and test sets. The action recognition model is trained and evaluated, and it is continuously iterated and optimized to obtain a trained action recognition model with recognition accuracy that meets the requirements.

[0110] Step S205: linearly map each picture block to an embedding vector through an embedding layer to obtain an initial embedding vector of the picture block.

[0111] In this embodiment, each image block x (p,t) Linearly map to an initial embedding vector It is expressed as follows:

[0112]

[0113] Among them, Ex p,t Represents image block x (p,t) The linear transformation matrix is A vector representing position information.

[0114] Step S206, through the spatiotemporal feature attention layer, the time attention mechanism and the space attention mechanism are used to calculate the attention weight of the initial embedding vector of the picture block to obtain a fused feature embedding vector that fuses the time feature and the space feature.

[0115] In this embodiment, the spatiotemporal feature attention layer includes a temporal attention network and a spatial attention network. Both the temporal attention network and the spatial attention network adopt a Transfomer attention structure. The Transfomer attention structure includes multiple layers of stacked attention modules. Each layer of attention modules includes a multi-head attention layer and a feedforward neural network. The attention weight of each image block is calculated through the self-attention mechanism of the multi-head attention layer, and the attention weight is input into the feedforward neural network layer for further processing to extract deeper features.

[0116] In some optional implementations of this embodiment, the step of using the temporal attention mechanism and the spatial attention mechanism to calculate the attention weight of the initial embedding vector of the picture block to obtain a fused feature embedding vector that fuses the temporal feature and the spatial feature includes:

[0117] The initial embedding vector of the image block is input into the temporal attention network and the spatial attention network respectively;

[0118] Through the temporal attention network, the temporal attention mechanism is used to calculate the temporal feature attention weight of each picture block in each layer of the temporal attention network in the corresponding frame, and the temporal feature vector of the picture block is obtained;

[0119] Through the spatial attention network, the spatial attention mechanism is used to calculate the spatial feature attention weight of each picture block in each layer of the spatial attention network in the corresponding frame, and the spatial feature vector of the picture block is obtained;

[0120] The temporal feature vector of the picture block and the spatial feature vector of the picture block are fused to obtain a fused feature embedding vector.

[0121] Among them, the temporal attention network extracts temporal features related to the action, and the spatial attention network extracts spatial features related to the action. Taking badminton as an example, it is necessary to capture the spatial relationship between the type of hitting and the serving position and the receiving position, as well as the temporal relationship between the speed and trajectory of the ball. That is, the type of hitting action is highly correlated with the hitting speed. For example, when the ball is drawn, the duration is very short, and the duration of the high ball is relatively long. The type of hitting action is highly correlated with the position information of the server and the receiver. For example, when a small ball is dropped, the receiver will quickly rush from the backcourt to the frontcourt.

[0122] In this embodiment, the process of calculating the temporal feature attention weight of the temporal attention network is as follows:

[0123] The Softmax function is used to calculate the temporal attention weight matrix between each image block and all frames; according to the temporal attention weight matrix, the initial embedding vector of each image block is weighted and summed to obtain the temporal weighted features of each image block on all frames; the temporal weighted features are used to calculate the new temporal embedding vector of each image block; the new temporal embedding vector is used as the input of the next layer of the temporal attention network, and the above steps are repeated until all layers of the temporal attention network are processed to obtain the temporal attention embedding vector of each image block in the last layer of the temporal attention network on all frames; the temporal attention embedding vectors of all image blocks are spliced ​​to obtain the temporal feature vector of the image block.

[0124] For the temporal self-attention mechanism, the attention mechanism of the p-th block and the t-th frame in the l-th layer of the temporal attention network is expressed as:

[0125]

[0126] Among them, softmax() represents a normalization function, the image block p does not change in space, and traverses all time dimensions, that is, t'=1,…,T. represents the query vector, and represents the key vector, D h Represents the embedding vector dimension. The embedding vector of the first layer is the initial embedding vector of the picture block.

[0127] The temporal attention embedding vector at layer l of the temporal attention network is represented as follows:

[0128]

[0129] Among them, f(·) represents the calculation process of the Transformer attention structure.

[0130] The calculated temporal attention embedding vector z for each image block over all frames (( l p ) ,t) The splicing is performed to obtain the temporal feature vector of the picture block.

[0131] By extracting the temporal feature vector of the image block through the temporal attention mechanism, it is possible to capture the temporal relationship between video frames, enhance the accuracy of action recognition, and more flexibly adapt to the changes of different actions, thereby improving the robustness of the model. In addition, it is possible to have a deeper understanding of the dynamic information in the video, thereby more accurately identifying complex actions and scenes.

[0132] In this embodiment, the process of calculating the spatial feature attention weights of the spatial attention network is as follows:

[0133] The spatial relationship between each of the picture blocks is calculated according to the initial embedding vector of the picture block; the spatial relationship is normalized using the Softmax function to obtain the spatial attention weight; according to the spatial attention weight, the initial embedding vector of each picture block is weighted and summed to obtain the spatial weighted features of each picture block on all frames; the spatial weighted features are used to calculate the new spatial embedding vector of each picture block; the new spatial embedding vector is used as the input of the next layer of the spatial attention network, and the above steps are repeated until all layers of the spatial attention network are processed to obtain the spatial attention embedding vector of each picture block in the last layer of the spatial attention network on all frames; the spatial attention embedding vectors of all picture blocks are spliced ​​to obtain the spatial feature vector of the picture block.

[0134] For the spatial self-attention mechanism, the attention mechanism of the p-th block and the t-th frame in the l-th layer of the spatial attention network is expressed as:

[0135]

[0136] Here, the softmax() represents a normalization function, the image block p does not change in time, and traverses all spatial dimensions, that is, p'=1,…,P. represents the query vector, and represents the key vector, D h Represents the embedding vector dimension. The embedding vector of the first layer is the initial embedding vector of the picture block.

[0137] The spatial attention embedding vector at layer l of the spatial attention network is represented as follows:

[0138]

[0139] Among them, f(·) represents the calculation process of the Transformer attention structure.

[0140] The calculated spatial attention embedding vector z for each image block on all frames ( ( l p ) ,t) Perform splicing to obtain the spatial feature vector of the image block.

[0141] By extracting the spatial feature vector of the image block through the spatial attention mechanism, the motion action can be described more comprehensively, thereby improving the accuracy of motion recognition. It can also enhance the robustness of motion recognition, thereby improving the stability and reliability of motion recognition.

[0142] The temporal feature vector of the picture block and the spatial feature vector of the picture block are fused by concatenation or weighted summation to obtain a fused feature embedding vector.

[0143] By obtaining the fusion feature embedding vector that integrates temporal features and spatial features, we can obtain more comprehensive information describing sports actions and more accurately capture the full picture of the action, thereby improving the precision and accuracy of action recognition. In competitive sports, we can more accurately judge the opponent's actions and intentions, thereby formulating more effective game strategies and improving the competitive level.

[0144] Step S207, input the fused feature embedding vector into the classification layer to obtain the action recognition result.

[0145] Specifically, the fused feature embedding vector is input into the classification layer, and the predicted probability distribution of the batting action corresponding to the action clip is obtained through forward propagation calculation; based on the predicted probability distribution, the action type that the action clip most likely corresponds to is determined as the action recognition result.

[0146] In some optional implementations of this embodiment, the classification layer uses a linear neural network and uses the embedding vector of the last layer of the spatiotemporal feature attention layer Input into the linear neural network, we can get the prediction result y, which is expressed as follows:

[0147]

[0148] Where LN(·) represents a linear neural network.

[0149] In some optional implementations, if the maximum probability value in the predicted probability distribution exceeds a preset probability threshold, the corresponding action type is used as the final recognition result, otherwise it is determined as a recognition failure; the action fragments that failed to be recognized are re-recognized.

[0150] Predicting action recognition results through the classification layer can improve the accuracy of action classification, thereby improving the accuracy of action prediction, enhancing the interpretability of the model, and improving the robustness of the algorithm.

[0151] The present application downsamples the original action clips extracted from the video to be identified to obtain action clips with a uniform number of frames, which can optimize the data processing and analysis process and improve the stability and robustness of the algorithm; by using an image segmentation algorithm to segment each frame of the action clip, it can focus more accurately on the key action area, help to analyze the action details more carefully, improve the accuracy of action recognition and enhance the robustness of the algorithm; by inputting each picture block into a trained action recognition model, combining the time and space attention mechanism for feature extraction and fusion, it can improve the feature extraction ability, help capture the spatial relationship and time relationship of the action, dynamically adjust the weight, and enhance the model's processing ability for long sequences, improve the model's interpretability, enhance the model's generalization ability, and finally classify the fused feature embedding vector through the classification layer to achieve accurate recognition of the action, significantly improving the accuracy and reliability of action recognition. The present application can not only be used for badminton analysis and teaching, but also can be extended to other similar sports action recognition fields, providing new technical support for intelligent sports analysis.

[0152] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0153] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0154] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0155] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0156] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a motion recognition device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0157] like Figure 3 As shown, the motion recognition device 300 described in this embodiment includes: an acquisition module 301, a downsampling module 302, a segmentation module 303, an input module 304, an embedding module 305, an attention calculation module 306 and a classification module 307. Among them:

[0158] The acquisition module 301 is used to acquire a video to be identified and extract an original action segment from the video to be identified;

[0159] The downsampling module 302 is used to perform downsampling processing on the original action clips to obtain action clips with a uniform preset target frame number;

[0160] The segmentation module 303 is used to segment each frame of the action clip using an image segmentation algorithm to obtain a preset number of picture blocks;

[0161] The input module 304 is used to input each of the picture blocks into the trained action recognition model, wherein the action recognition model includes an embedding layer, a spatiotemporal feature attention layer and a classification layer;

[0162] The embedding module 305 is used to linearly map each of the picture blocks into an embedding vector through the embedding layer to obtain an initial embedding vector of the picture block;

[0163] The attention calculation module 306 is used to calculate the attention weight of the initial embedding vector of the picture block by using the time attention mechanism and the space attention mechanism through the time-space feature attention layer to obtain a fusion feature embedding vector that fuses the time feature and the space feature;

[0164] The classification module 307 is used to input the fusion feature embedding vector into the classification layer to obtain the action recognition result.

[0165] It should be emphasized that in order to further ensure the privacy and security of the action clips, the above action clips can also be stored in a blockchain node.

[0166] Based on the above-mentioned action recognition device 300, by downsampling the original action clips extracted from the video to be recognized to obtain action clips with a uniform frame number, the data processing and analysis process can be optimized, and the stability and robustness of the algorithm can be improved; by using an image segmentation algorithm to divide each frame of the action clip, it is possible to focus on the key action area more accurately, which is helpful to analyze the action details more carefully, improve the accuracy of action recognition and enhance the robustness of the algorithm; by inputting each image block into a trained action recognition model, combining the time and space attention mechanism for feature extraction and fusion, the feature extraction capability can be improved, which is helpful to capture the spatial and temporal relationships of the action, dynamically adjust the weights, and at the same time enhance the model's ability to process long sequences, improve the model's interpretability, and enhance the model's generalization ability. Finally, the fused feature embedding vector is classified through the classification layer to achieve accurate recognition of the action, significantly improving the accuracy and reliability of action recognition.

[0167] In some optional implementations of this embodiment, the acquisition module 301 includes:

[0168] A preprocessing submodule, used for preprocessing the video to be identified to obtain enhanced video data;

[0169] A frame analysis submodule, used to analyze the video frames of the enhanced video data according to a preset motion action model using a template matching algorithm to identify the start frame and the end frame of the motion action;

[0170] The target detection submodule is used to analyze the video frames between the start frame and the end frame by using a target detection algorithm to determine the start position coordinates and the end position coordinates;

[0171] A judgment submodule, configured to judge that the motion is complete if the number of video frames between the start frame and the end frame is greater than a preset frame number threshold, and judge that the motion is incomplete otherwise;

[0172] A tracking submodule, for tracking the motion process using a motion target tracking algorithm according to the starting position coordinates and the ending position coordinates to obtain a complete motion trajectory when determining that the motion is complete;

[0173] The acquisition submodule is used to determine the original action segments according to the complete action trajectory, sort the original action segments in chronological order, and generate an action segment set.

[0174] By extracting action clips from the video to be identified, the video quality can be improved, motion movements can be accurately located, data management and analysis can be facilitated, and the efficiency of subsequent data processing can be improved.

[0175] In some optional implementations, the downsampling module 302 is further configured to:

[0176] Obtaining original frame rate and resolution information of the original action clip;

[0177] Calculating a downsampling ratio of the original action clip according to a preset target number of frames;

[0178] A linear interpolation algorithm is used to downsample the video frames of the original action clip according to the downsampling ratio to obtain an action clip with a preset target frame number.

[0179] By downsampling the original action clips and unifying the frame number, the amount of data can be reduced, the computational complexity can be lowered, the processing speed can be increased, while key action information can be maintained and storage and transmission efficiency can be optimized.

[0180] In some optional implementations of this embodiment, the spatiotemporal feature attention layer includes a temporal attention network and a spatial attention network, and the attention calculation module 306 includes:

[0181] An input submodule, used to input the initial embedding vector of the picture block into the temporal attention network and the spatial attention network respectively;

[0182] A temporal attention calculation submodule, configured to calculate the temporal feature attention weight of each of the picture blocks in each layer of the temporal attention network in the corresponding frame by using the temporal attention mechanism through the temporal attention network, and obtain a temporal feature vector of the picture block;

[0183] A spatial attention calculation submodule, used to calculate the spatial feature attention weight of each picture block in each layer of the spatial attention network in the corresponding frame by using the spatial attention mechanism through the spatial attention network, and obtain the spatial feature vector of the picture block;

[0184] The fusion submodule is used to fuse the temporal feature vector of the picture block with the spatial feature vector of the picture block to obtain a fused feature embedding vector.

[0185] By obtaining a fused feature embedding vector that fuses temporal features and spatial features, we can obtain more comprehensive information describing motion actions and more accurately capture the overall picture of the action, thereby improving the precision and accuracy of action recognition.

[0186] In this embodiment, the time attention calculation submodule is further used to:

[0187] The Softmax function is used to calculate the time attention weight matrix between each of the image blocks and all frames;

[0188] According to the temporal attention weight matrix, weighted summation is performed on the initial embedding vector of each of the picture blocks to obtain the temporal weighted features of each of the picture blocks on all frames;

[0189] Using the time-weighted features, calculating a new time embedding vector for each of the picture blocks;

[0190] The new temporal embedding vector is used as the input of the next layer of the temporal attention network, and the above steps are repeated until all layers of the temporal attention network are processed to obtain the temporal attention embedding vector of each image block in all frames of the last layer of the temporal attention network;

[0191] The temporal attention embedding vectors of all the picture blocks are concatenated to obtain a temporal feature vector of the picture block.

[0192] By extracting the temporal feature vector of the image block through the temporal attention mechanism, it is possible to capture the temporal relationship between video frames, enhance the accuracy of action recognition, and more flexibly adapt to the changes of different actions, thereby improving the robustness of the model.

[0193] In some optional implementations of this embodiment, the spatial attention calculation submodule is further used to:

[0194] Calculating the spatial relationship between each of the picture blocks according to the initial embedding vector of the picture block;

[0195] The Softmax function is used to normalize the spatial relationship to obtain the spatial attention weight;

[0196] According to the spatial attention weight, weighted summation is performed on the initial embedding vector of each of the picture blocks to obtain a spatial weighted feature of each of the picture blocks on all frames;

[0197] Using the spatial weighted features, calculating a new spatial embedding vector for each of the picture blocks;

[0198] The new spatial embedding vector is used as the input of the next layer of the spatial attention network, and the above steps are repeated until all layers of the spatial attention network are processed to obtain the spatial attention embedding vector of each image block in the last layer of the spatial attention network on all frames;

[0199] The spatial attention embedding vectors of all the picture blocks are concatenated to obtain a spatial feature vector of the picture block.

[0200] By extracting the spatial feature vector of the image block through the spatial attention mechanism, the motion action can be described more comprehensively, thereby improving the accuracy of motion recognition. It can also enhance the robustness of motion recognition, thereby improving the stability and reliability of motion recognition.

[0201] In some optional implementations, the classification module 307 is further configured to:

[0202] The fused feature embedding vector is input into the classification layer, and the predicted probability distribution of the hitting action corresponding to the action clip is obtained through forward propagation calculation;

[0203] According to the predicted probability distribution, the action type that the action clip most likely corresponds to is determined as an action recognition result.

[0204] Predicting action recognition results through the classification layer can improve the accuracy of action classification, thereby improving the accuracy of action prediction, enhancing the interpretability of the model, and improving the robustness of the algorithm.

[0205] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0206] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0207] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0208] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the action recognition method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0209] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the action recognition method.

[0210] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0211] By downsampling the original action clips extracted from the video to be identified to obtain action clips with a uniform frame number, the data processing and analysis process can be optimized and the stability and robustness of the algorithm can be improved. By using the image segmentation algorithm to divide each frame of the action clip, it is possible to focus on the key action area more accurately, which helps to analyze the action details more carefully, improve the accuracy of action recognition and enhance the robustness of the algorithm. By inputting each image block into the trained action recognition model and combining the time and space attention mechanism for feature extraction and fusion, the feature extraction capability can be improved, which helps to capture the spatial and temporal relationships of the action, dynamically adjust the weights, and enhance the model's ability to process long sequences, improve the model's interpretability, and enhance the model's generalization ability. Finally, the fused feature embedding vector is classified through the classification layer to achieve accurate recognition of the action, significantly improving the accuracy and reliability of action recognition.

[0212] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the action recognition method as described above.

[0213] By downsampling the original action clips extracted from the video to be identified to obtain action clips with a uniform frame number, the data processing and analysis process can be optimized and the stability and robustness of the algorithm can be improved. By using the image segmentation algorithm to divide each frame of the action clip, it is possible to focus on the key action area more accurately, which helps to analyze the action details more carefully, improve the accuracy of action recognition and enhance the robustness of the algorithm. By inputting each image block into the trained action recognition model and combining the time and space attention mechanism for feature extraction and fusion, the feature extraction capability can be improved, which helps to capture the spatial and temporal relationships of the action, dynamically adjust the weights, and enhance the model's ability to process long sequences, improve the model's interpretability, and enhance the model's generalization ability. Finally, the fused feature embedding vector is classified through the classification layer to achieve accurate recognition of the action, significantly improving the accuracy and reliability of action recognition.

[0214] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0215] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for motion recognition, characterized in that: The steps include: Obtain a video to be identified, and extract an original action segment from the video to be identified; Downsampling the original action clips to obtain uniform action clips with a preset target frame number; Using an image segmentation algorithm to segment each frame of the action clip to obtain a preset number of image blocks; Inputting each of the image blocks into a trained action recognition model, wherein the action recognition model includes an embedding layer, a spatiotemporal feature attention layer, and a classification layer; Linearly mapping each of the picture blocks into an embedding vector through the embedding layer to obtain an initial embedding vector of the picture block; By using the spatiotemporal feature attention layer, the time attention mechanism and the space attention mechanism are used to calculate the attention weight of the initial embedding vector of the picture block to obtain a fused feature embedding vector that fuses the time feature and the space feature; The fused feature embedding vector is input into the classification layer to obtain an action recognition result.

2. The action recognition method according to claim 1, characterized in that: The step of extracting the original action clip from the video to be identified comprises: Preprocessing the video to be identified to obtain enhanced video data; According to a preset motion action model, a template matching algorithm is used to analyze the video frames of the enhanced video data to identify the start frame and the end frame of the motion action; Analyze the video frames between the start frame and the end frame by using a target detection algorithm to determine the start position coordinates and the end position coordinates; If the number of video frames between the start frame and the end frame is greater than a preset frame number threshold, the motion action is determined to be complete, otherwise the motion action is determined to be incomplete; When determining that the motion is complete, tracking the motion process using a motion target tracking algorithm according to the starting position coordinates and the ending position coordinates to obtain a complete motion trajectory; The original action segments are determined according to the complete action trajectory, and the original action segments are sorted in chronological order to generate an action segment set.

3. The action recognition method according to claim 1, characterized in that: The step of downsampling the original action clips to obtain action clips with a uniform preset target frame number comprises: Obtaining original frame rate and resolution information of the original action clip; Calculating a downsampling ratio of the original action clip according to a preset target number of frames; A linear interpolation algorithm is used to downsample the video frames of the original action clip according to the downsampling ratio to obtain an action clip with a preset target frame number.

4. The action recognition method according to claim 1, characterized in that: The spatiotemporal feature attention layer includes a temporal attention network and a spatial attention network. The step of using the temporal attention mechanism and the spatial attention mechanism to calculate the attention weight of the initial embedding vector of the picture block to obtain a fused feature embedding vector that fuses the temporal feature and the spatial feature includes: Inputting the initial embedding vector of the picture block into the temporal attention network and the spatial attention network respectively; By using the temporal attention network, the temporal feature attention weight of each picture block in each layer of the temporal attention network in the corresponding frame is calculated using the temporal attention mechanism to obtain a temporal feature vector of the picture block; By using the spatial attention network, the spatial feature attention weight of each picture block in each layer of the spatial attention network in the corresponding frame is calculated using the spatial attention mechanism to obtain a spatial feature vector of the picture block; The temporal feature vector of the picture block and the spatial feature vector of the picture block are fused to obtain a fused feature embedding vector.

5. The action recognition method according to claim 4, characterized in that: The step of using the temporal attention mechanism to calculate the temporal feature attention weight of each picture block in each layer of the temporal attention network in the corresponding frame to obtain the temporal feature vector of the picture block includes: The Softmax function is used to calculate the time attention weight matrix between each of the image blocks and all frames; According to the temporal attention weight matrix, weighted summation is performed on the initial embedding vector of each of the picture blocks to obtain the temporal weighted features of each of the picture blocks on all frames; Using the time-weighted features, calculating a new time embedding vector for each of the picture blocks; The new temporal embedding vector is used as the input of the next layer of the temporal attention network, and the above steps are repeated until all layers of the temporal attention network are processed to obtain the temporal attention embedding vector of each image block in all frames of the last layer of the temporal attention network; The temporal attention embedding vectors of all the picture blocks are concatenated to obtain a temporal feature vector of the picture block.

6. The action recognition method according to claim 4, characterized in that: The step of using the spatial attention mechanism to calculate the spatial feature attention weights of each of the picture blocks in each layer of the spatial attention network in the corresponding frame includes: Calculating the spatial relationship between each of the picture blocks according to the initial embedding vector of the picture block; The Softmax function is used to normalize the spatial relationship to obtain the spatial attention weight; According to the spatial attention weight, weighted summation is performed on the initial embedding vector of each of the picture blocks to obtain a spatial weighted feature of each of the picture blocks on all frames; Using the spatial weighted features, calculating a new spatial embedding vector for each of the picture blocks; The new spatial embedding vector is used as the input of the next layer of the spatial attention network, and the above steps are repeated until all layers of the spatial attention network are processed to obtain the spatial attention embedding vector of each image block in the last layer of the spatial attention network on all frames; The spatial attention embedding vectors of all the picture blocks are concatenated to obtain a spatial feature vector of the picture block.

7. The action recognition method according to claim 1, characterized in that: The step of inputting the fusion feature embedding vector into the classification layer to obtain the action recognition result comprises: The fused feature embedding vector is input into the classification layer, and the predicted probability distribution of the hitting action corresponding to the action clip is obtained through forward propagation calculation; According to the predicted probability distribution, the action type that the action clip most likely corresponds to is determined as an action recognition result.

8. A motion recognition device, characterized in that: include: An acquisition module is used to acquire a video to be identified and extract an original action segment from the video to be identified; A downsampling module, used for downsampling the original action clips to obtain action clips with a uniform preset target frame number; A segmentation module, used to segment each frame of the action clip using an image segmentation algorithm to obtain a preset number of picture blocks; An input module, used to input each of the image blocks into a trained action recognition model, wherein the action recognition model includes an embedding layer, a spatiotemporal feature attention layer, and a classification layer; An embedding module, used for linearly mapping each of the picture blocks into an embedding vector through the embedding layer to obtain an initial embedding vector of the picture block; An attention calculation module is used to calculate the attention weight of the initial embedding vector of the picture block by using the time attention mechanism and the space attention mechanism through the time-space feature attention layer to obtain a fused feature embedding vector that fuses the time feature and the space feature; The classification module is used to input the fusion feature embedding vector into the classification layer to obtain the action recognition result.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the action recognition method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the action recognition method according to any one of claims 1 to 7 are implemented.

Images