Motion recognition and model training method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an action recognition and model training method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining key point heat maps corresponding to a dynamic target in a to-be-recognized video, and stacking the key point heat maps according to a time sequence to obtain a three-dimensional heat map stacking body corresponding to the dynamic target; inputting the three-dimensional heat map stack into a pre-trained action recognition model, wherein the network structure of the action recognition model is an improved attitude three-dimensional convolutional network obtained by adding a convolutional block attention module in a residual layer of the attitude three-dimensional convolutional network; and based on the three-dimensional heat map stack, determining an action recognition result of the dynamic target by improving the attitude three-dimensional convolutional network. According to the embodiment of the invention, the recognition accuracy when the action of the dynamic target in the video is recognized can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, electronic equipment and storage medium for motion recognition and model training. Background Art

[0002] Action recognition of dynamic targets in video data has important application value in security monitoring, sports events and other fields. Action recognition belongs to spatiotemporal detection, which requires comprehensive consideration of temporal and spatial information to identify and understand actions in video data. This involves not only the analysis of the static features of the target, but also the use of time series information to identify and classify different action behaviors. The challenge of action recognition is how to effectively combine spatial and temporal features to achieve accurate understanding of video data, which requires the algorithm to be able to process video data and extract meaningful spatiotemporal features from it.

[0003] The action recognition methods in the prior art include key point-based methods and RGB video-based methods. Since human key points can provide a lot of important information for human action recognition. Even without detailed features such as background, RGB color, and target appearance in the video, the target's behavior can be effectively understood by extracting the position transformation information of key points, and it is robust to changes in lighting and scenes. Therefore, the action recognition method based on key points has less computational complexity and stronger adaptability to different scenes than the RGB video-based method, and is more suitable for real-time behavior recognition, and has received more and more attention.

[0004] With the expansion of motion recognition application scenarios, people have higher and higher requirements for the accuracy of motion recognition. The key point-based motion recognition method in the prior art cannot meet people's accuracy requirements. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, electronic device and storage medium for action recognition and model training, which can improve the recognition accuracy when recognizing the action of a dynamic target in a video.

[0006] In a first aspect, an embodiment of the present invention provides a method for motion recognition, comprising:

[0007] Obtain a key point heat map corresponding to a dynamic target in a video to be identified, and stack the key point heat maps in chronological order to obtain a three-dimensional heat map stack corresponding to the dynamic target;

[0008] Inputting the three-dimensional heat map stack into a pre-trained action recognition model, wherein the network structure of the action recognition model is an improved posture three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the posture three-dimensional convolutional network; and

[0009] Based on the three-dimensional heat map stack, the action recognition result of the dynamic target is determined by the improved posture three-dimensional convolutional network.

[0010] In a second aspect, an embodiment of the present invention provides a method for training an action recognition model, comprising:

[0011] Add action category labels to sample videos to obtain labeled sample videos;

[0012] Obtaining a key point heat map corresponding to the dynamic target in the labeled sample video, and stacking the key point heat maps in chronological order to obtain a three-dimensional heat map stack corresponding to the labeled sample video;

[0013] Inputting the three-dimensional heat map stack into the action recognition model to be trained, wherein the network structure of the action recognition model is an improved posture three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the posture three-dimensional convolutional network;

[0014] Based on the three-dimensional heat map stack, determining the action recognition result of the dynamic target through the improved posture three-dimensional convolutional network; and

[0015] The function value of the training loss function is determined based on the action recognition result, and the network parameters of the action recognition model are adjusted according to the function value of the training loss function.

[0016] In a third aspect, an embodiment of the present invention provides a motion recognition device, comprising:

[0017] The module for acquiring the stack of heat maps to be tested is used to acquire the key point heat maps corresponding to the dynamic target in the video to be identified, and to stack the key point heat maps in chronological order to obtain a three-dimensional heat map stack corresponding to the dynamic target;

[0018] A heat map stack input module to be tested, used for inputting the three-dimensional heat map stack into a pre-trained action recognition model, wherein the network structure of the action recognition model is an improved attitude three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the attitude three-dimensional convolutional network; and

[0019] The action recognition module to be tested is used to determine the action recognition result of the dynamic target through the improved posture three-dimensional convolutional network based on the three-dimensional heat map stack.

[0020] In a fourth aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the action recognition method or action recognition model training method as described in any one of the embodiments of the present invention is implemented.

[0021] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the action recognition method or action recognition model training method as described in any one of the embodiments of the present invention.

[0022] The embodiments of the present invention provide an action recognition and model training method, device, electronic device and storage medium. The training network structure is an improved posture three-dimensional convolutional network action recognition model in which a convolution block attention module is added to the residual layer of the posture three-dimensional convolutional network, and the three-dimensional heat map stack corresponding to the dynamic target in the video to be identified is obtained. Then, based on the three-dimensional heat map stack, the action recognition result of the dynamic target is obtained by identifying the pre-trained action recognition model, thereby improving the recognition accuracy of the action of the dynamic target in the video to be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 is a flow chart of a motion recognition method provided by an embodiment of the present invention;

[0025] Figure 2 It is a schematic diagram of a skeleton point information heat map group and a three-dimensional heat map stack in the action recognition method provided by an embodiment of the present invention;

[0026] Figure 3 It is a schematic diagram of the structure of a PoseConv3D network and an improved PoseConv3D network in the action recognition method provided by an embodiment of the present invention;

[0027] Figure 4 is another flowchart of the action recognition method provided by an embodiment of the present invention;

[0028] Figure 5 It is a flow chart of the action recognition model training method provided by an embodiment of the present invention;

[0029] Figure 6 is a structural schematic diagram of a motion recognition device provided by an embodiment of the present invention;

[0030] Figure 7 is a structural schematic diagram of a motion recognition model training device provided by an embodiment of the present invention;

[0031] Figure 8It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Figure 1 The present invention provides a flow chart of a method for motion recognition. The present invention can be applied to the scene of identifying dynamic targets in a video. The method can be executed by the motion recognition device provided by the present invention. The device can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated into an electronic device, such as a computer, a server, etc. The following embodiments will be described by taking the device integrated into an electronic device as an example. Figure 1 , the method may specifically include the following steps:

[0035] Step 101, obtain the key point heat map corresponding to the dynamic target in the video to be identified, and stack the key point heat map in chronological order to obtain a three-dimensional heat map stack corresponding to the dynamic target. This step can be beneficial to obtain the recognition result of the dynamic target through the pre-trained action recognition model based on the three-dimensional heat map stack corresponding to the dynamic target.

[0036] Specifically, the video to be identified may be a real-time video acquired by a video shooting device, or may be a playback video.

[0037] Specifically, the dynamic target may be a dynamic person, animal and / or equipment.

[0038] Optionally, when the dynamic target is a person, the key point heat map may be a heat map of human skeleton point information of the person in each frame of the video to be identified.

[0039] Specifically, the above-mentioned skeleton point information may be coordinate information of the skeleton point.

[0040] In a specific example, the human skeleton point information may include information of 17 skeleton points, and the key point heat map may be a skeleton point information heat map group including 17 heat maps corresponding to each frame image, wherein each heat map includes information of one skeleton point, specifically as follows: Figure 2 shown.

[0041] Optionally, the process of stacking key point heat maps in chronological order to obtain a three-dimensional heat map stack corresponding to the dynamic target includes: stacking multiple groups of heat maps corresponding to a video clip of an action of the dynamic target in chronological order to obtain the three-dimensional heat map stack.

[0042] Specifically, the three-dimensional heat map stack is heat map data having three dimensions, namely, time dimension, height dimension and width dimension.

[0043] In a specific example, when the dynamic target is a person, the three-dimensional heat map stack can be as follows: Figure 2 shown.

[0044] Optionally, the process of obtaining a key point heat map corresponding to a dynamic target in a video to be identified includes:

[0045] The pre-trained target detection model and key point detection model are used to obtain the key point heat map corresponding to the above dynamic targets.

[0046] Specifically, the above-mentioned target detection model can be a neural network model, for example, it can be a target detection model such as a Faster Regional Convolutional Neural Network (Faster RCNN) model, a Real-Time Detection, Embedding, and Tracking (RTDETR) model or a YOLOv8 model.

[0047] Specifically, the above-mentioned key point detection model may also be a neural network model, for example, a high-resolution network (High-Resolution Net, HRNet) model.

[0048] Step 102, input the three-dimensional heat map stack into the pre-trained action recognition model, the network structure of the action recognition model is an improved attitude three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the attitude three-dimensional convolutional network. This step can help determine the action recognition result of the dynamic target based on the input three-dimensional heat map stack by adding the improved attitude three-dimensional convolutional network with the convolutional block attention module in the residual layer.

[0049] Specifically, the above-mentioned posture three-dimensional convolutional network can be understood as a PoseConv3D network. Specifically, the specific structure of the above-mentioned PoseConv3D network is as follows Figure 3 shown.

[0050] Specifically, the above-mentioned Convolutional Block Attention Module (CBAM) mainly includes two submodules, a channel attention submodule and a spatial attention submodule. The channel attention submodule learns the importance of different channels by considering the global information of each channel of the feature map. Global average pooling and global maximum pooling are used to extract the global statistical features of the feature map, and then the channel attention map is generated through the fully connected layer and activation function. The channel attention map is used to adjust the weights of each channel in the original feature map so that the model can focus on more informative channels. The spatial attention submodule further focuses on the specific positions inside the feature map and learns the importance of spatial positions. This module usually uses the maximum and average values ​​of channel information to generate a spatial attention map. By applying a small convolution kernel, the module can capture spatial context information and generate a spatial attention map for adjusting the weight of each spatial position.

[0051] The embodiment of the present invention performs pooling operations on three-dimensional data through a convolutional block attention module, so that it processes data with three spatial dimensions including a time dimension, performs three-dimensional convolution instead of two-dimensional convolution, and can fully extract the motion features of dynamic targets in the three-dimensional heat map stack, so as to ultimately improve the recognition accuracy of the motion recognition results of dynamic targets.

[0052] Optionally, the improved posture 3D convolutional network includes: a first-stage convolutional layer, a second-stage residual layer, a third-stage residual layer, a fourth-stage residual layer, a fifth-stage pooling layer, and a sixth-stage fully connected layer.

[0053] Optionally, a convolution block attention module is added to each residual block of the second-stage residual layer, the third-stage residual layer and the fourth-stage residual layer; for each residual block, the convolution block attention module is added after the convolution block of the current residual block and before the residual connection.

[0054] Specifically, the structure of the above improved posture 3D convolutional network is as follows: Figure 3 shown.

[0055] Specifically, the second-stage residual layer, the third-stage residual layer and the fourth-stage residual layer respectively include: 4 residual blocks, 6 residual blocks and 3 residual blocks.

[0056] Specifically, a convolutional block attention module is added before the residual connection of each residual block in each residual layer.

[0057] Specifically, the above-mentioned convolutional attention module can also be added to each residual block of any residual layer among the second-stage residual layer, the third-stage residual layer and the fourth-stage residual layer.

[0058] Specifically, the above-mentioned convolutional attention module can be added to some residual blocks in the second-stage residual layer, the third-stage residual layer, and the fourth-stage residual layer, respectively.

[0059] Specifically, when adding the above-mentioned convolutional attention module in each residual block, the above-mentioned convolutional attention module can also be added after each convolutional block of the residual block.

[0060] In an optional specific example, the number of convolutional attention modules in each residual layer is determined by changing the position and number of convolutional attention modules in the residual layer and analyzing the recognition accuracy change trend of the improved posture 3D convolutional network including different numbers of CBMA modules.

[0061] Specifically, when using the improved PoseConv3D network model including different numbers of CBMA modules to recognize the actions of dynamic targets in the video, the accuracy is improved. However, due to the increase in the number of model parameters, the amount of calculation in the recognition process will increase, and the time spent on action recognition will increase and the memory consumption will increase.

[0062] Optionally, based on the changing trend of the corresponding accuracy, the changing trend of the time consumed for the corresponding action recognition, and the upper limit of the available memory, the number of the above-mentioned attention modules is determined to be 13, and the position is before the residual connection of each residual block of each residual layer, directly connecting the corresponding residual connection setting.

[0063] Step 103, based on the three-dimensional heat map stack, determine the action recognition result of the dynamic target by improving the posture three-dimensional convolutional network. This step is based on steps 101 and 102, and the pre-trained network structure is an improved posture three-dimensional convolutional network action recognition model in which a convolution block attention module is added to the residual layer of the posture three-dimensional convolutional network. The action recognition result of the dynamic target is obtained based on the three-dimensional heat map stack corresponding to the dynamic target in the video to be recognized, which can improve the recognition accuracy of the action of the dynamic target in the video to be recognized.

[0064] Optionally, the process of determining the action recognition result of the dynamic target by improving the posture 3D convolutional network based on the 3D heat map stack includes:

[0065] The action features of the three-dimensional heat map stack are extracted in sequence through the first-stage convolutional layer, the second-stage residual layer, the third-stage residual layer and the fourth-stage residual layer to obtain the target action features; the target action features are averaged pooled through the fifth-stage pooling layer to obtain the pooled output features; and the action recognition results are obtained based on the pooled output features through the sixth-stage fully connected layer.

[0066] Optionally, the process of extracting action features from a three-dimensional heat map stack through the second-stage residual layer, the third-stage residual layer, and the fourth-stage residual layer includes: for each residual block, using the convolution block of the current residual block to perform convolution processing on the input features of the current residual block to obtain the convolution features of the current residual block; using the convolution block attention module of the current residual block to adjust the attention of the convolution features of the current residual block to obtain the attention adjustment features of the current residual block; and performing residual connection on the input features and the attention adjustment features of the current residual block to obtain the output features of the current residual block.

[0067] The action recognition method provided by the embodiment of the present invention is further described below.

[0068] like Figure 4 As shown, Figure 1 Step 101 in the method may include the following steps:

[0069] Step 1011, selecting a currently applicable target detection model from multiple candidate target detection models based on the image region features corresponding to the dynamic target.

[0070] Optionally, the image region feature corresponding to the dynamic target may be the clarity of the corresponding image and / or the position of the corresponding image region in the entire image.

[0071] Optionally, the process of selecting the currently applicable target detection model from multiple candidate target detection models based on image area features corresponding to dynamic targets includes: selecting the currently applicable target detection model from multiple candidate target detection models with different network structures that are trained based on the same sample video.

[0072] Optionally, the process of selecting the currently applicable target detection model from multiple candidate target detection models based on image area features corresponding to dynamic targets includes: selecting the currently applicable target detection model from multiple candidate target detection models of the same network structure trained based on different types of sample videos.

[0073] Optionally, the candidate target detection model is a RTDETR model.

[0074] Step 1012: Detect and obtain dynamic targets from the video to be identified using the currently applicable target detection model.

[0075] Step 1013: Detect key points of the dynamic target using a key point detection model, and generate a key point heat map based on the key points of the dynamic target.

[0076] Optionally, the key point detection model is a HRNet model.

[0077] Specifically, the HRNet network structure is designed to solve tasks such as image segmentation and human posture estimation, where high-resolution feature maps are essential for accurate spatial positioning. Compared with other deep learning models, such as CNN convolutional neural networks, which usually gradually reduce the spatial resolution of feature maps in deep layers to obtain more abstract feature representations, HRNet can enhance or maintain this resolution at all levels while maintaining high spatial resolution. This design allows HRNet to effectively fuse features of different scales throughout the network, optimizing the capture of details and global information, rather than simply pursuing deep abstract feature representations.

[0078] Specifically, the HRNet network starts with a high-resolution subnet, gradually adds low-resolution subnets in parallel to capture features of different scales, and exchanges and fuses features at all levels. When processing images, HRNet can simultaneously utilize high-resolution detail features and low-resolution global features, thereby achieving accurate spatial positioning and efficient information integration in the task of human pose estimation.

[0079] Specifically, the final output of HRNet is a set of heat maps, each key point corresponds to a heat map, the heat map is as follows Figure 3 As shown. For common human posture estimation tasks, there are usually 17 key points. Therefore, HRNet will output 17 heat maps, and the size of each heat map is consistent with the output resolution of the final layer of the model, which is usually 1 / 4 of the input image resolution. Each pixel value on each heat map represents the confidence that the corresponding key point appears at the corresponding position. The position of the highest value on each heat map is usually used to predict the position of the corresponding key point. The key point data obtained is arranged in the order of the skeleton points to obtain the skeleton point data.

[0080] Figure 5A flowchart of a method for training an action recognition model provided in an embodiment of the present invention is provided. This embodiment is applicable to scenarios where dynamic targets in a video are to be recognized. The method can be performed by a method for training an action recognition model provided in an embodiment of the present invention, and the method can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated into an electronic device, such as a computer, a server, etc. The following embodiments will be described using the example of the device being integrated into an electronic device. Figure 5 , the method may specifically include the following steps:

[0081] Step 501: add action category labels to sample videos to obtain labeled sample videos.

[0082] Specifically, the sample video may be a video obtained by adjusting the size, contrast, sharpness, color temperature, saturation, etc. of the original video including the dynamic target.

[0083] Specifically, the labeled sample videos can be horizontally flipped to increase the diversity of training data and improve the generalization ability of the action recognition model.

[0084] Step 502: Obtain key point heat maps corresponding to dynamic targets in the labeled sample video, and stack the key point heat maps in chronological order to obtain a three-dimensional heat map stack corresponding to the labeled sample video.

[0085] Step 503: input the three-dimensional heat map stack into the action recognition model to be trained. The network structure of the action recognition model is an improved posture three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the posture three-dimensional convolutional network.

[0086] Step 504, based on the three-dimensional heat map stack, determine the action recognition result of the dynamic target by improving the posture three-dimensional convolutional network.

[0087] Step 505 , determining a function value of a training loss function based on the action recognition result, and adjusting network parameters of the action recognition model according to the function value of the training loss function.

[0088] Optionally, the above training loss function is a category-balanced focal loss function.

[0089] Specifically, the PoseConv3D network model in the prior art uses the cross entropy loss function as the loss function for the classification task, and the cross entropy loss function does not take into account the problem of class imbalance. The embodiment of the present invention uses a class-balanced focal loss function (i.e., CB Focal loss) as a loss function, which can alleviate the class imbalance problem by introducing different weight factors for samples of different categories under the premise of considering the difficulty of classifying samples and the class imbalance problem.

[0090] In the embodiment of the present invention, the action recognition model is trained by using a category-balanced focal loss function, thereby avoiding the problem of category imbalance in the training process, and improving the accuracy of the trained action recognition model in classifying action recognition of dynamic targets.

[0091] Optionally, the process of adjusting the network parameters of the action recognition model according to the function value of the training loss function includes: adjusting the network parameters of the action recognition model through an Adam optimizer according to the function value of the training loss.

[0092] Specifically, the above-mentioned Adam optimizer is the Adam optimizer.

[0093] Specifically, the SGD optimizer is used in the prior art to train the PoseConv3D network model, but a major disadvantage of SGD is that its convergence speed is usually slower, and the SGD optimizer is more sensitive to the selection of learning rate and other hyperparameters. The Adam optimizer algorithm combines the exponentially weighted moving average of momentum and gradient squares, can adaptively adjust the learning rate of each parameter, and can handle the optimization problems of sparse gradients and non-stationary objective functions, and has a fast convergence speed, and is widely used in the training of deep learning models.

[0094] Preferably, the network parameters of the action recognition model are adjusted by an AdamW optimizer in an Adam optimizer.

[0095] Specifically, the AdamW optimizer is an improved adaptive learning rate optimization algorithm that implements weight decay more effectively by modifying the standard Adam optimizer. Unlike traditional L2 regularization, AdamW treats weight decay as an independent part of the optimization process and adjusts the parameters directly instead of incorporating it into the gradient. This separation decouples weight decay from the learning rate of each parameter, providing a more stable training process and better generalization when training neural networks. This feature makes AdamW more effective when dealing with deep learning models, especially when there are a large number of parameters and complex structures, and often achieves better results than standard Adam.

[0096] The embodiment of the present invention adjusts the network parameters of the action recognition model through the Adam optimizer, so that the action recognition model with higher recognition accuracy can be trained more efficiently and stably.

[0097] Specifically, A, B and C are used to represent adding a CBAM module to the network structure of PoseConv3D, using the CB Focal loss function instead of the cross entropy loss function, using the Adam optimizer instead of the SGD optimizer, using the PoseConv3D network model in the prior art, and A, B, C, the combination of A, B, C, and the combination of A, B, C. The corresponding improved PoseConv3D network model is used to perform action classification recognition on a fine-grained gymnastics action classification task including 99 action classifications. The recognition accuracy is shown in Table 1:

[0098] method Accuracy PoseConv3D 0.907 A 0.944 B 0.912 C 0.911 A and B 0.944 A and C 0.942 B and C 0.923 AB and C 0.948

[0099] Table 1

[0100] As shown in Table 1, after the introduction of the CBAM module, the action recognition accuracy of the PoseConv3D network model increased from 0.907 to 0.944. After the introduction of the CBAM module and the use of the CB Focal loss function, the accuracy was 0.944 and did not improve further; after the introduction of the CBAM module and the use of the Adam optimizer, the accuracy was 0.942, which was lower than the introduction of the CBAM module alone; after the introduction of the CBAM module, the CB Focal loss loss function and the Adam optimizer, the accuracy was improved to 0.948 compared with the introduction of the CBAM module alone. Therefore, the three improvement measures work together to promote the improvement of model performance, and the three measures work together to significantly improve the accuracy of the model.

[0101] Figure 6 is a structural diagram of a motion recognition device provided by an embodiment of the present invention, and the device is suitable for executing the motion recognition method provided by an embodiment of the present invention. Figure 6 As shown, the device may specifically include:

[0102] The heat map stack acquisition module 601 to be tested is used to obtain the key point heat map corresponding to the dynamic target in the video to be identified, and stack the key point heat map in time order to obtain the three-dimensional heat map stack corresponding to the dynamic target. It is beneficial to obtain the recognition result of the dynamic target through the pre-trained action recognition model based on the three-dimensional heat map stack corresponding to the dynamic target.

[0103] Optionally, the above-mentioned test heat map stack acquisition module 601 can be specifically used to select a currently applicable target detection model from multiple candidate target detection models based on image area features corresponding to the dynamic target; detect and acquire the dynamic target from the video to be identified through the currently applicable target detection model; and detect the key points of the dynamic target through the key point detection model, and generate a key point heat map based on the key points of the dynamic target.

[0104] The heat map stack input module 602 to be tested is used to input the three-dimensional heat map stack into the pre-trained action recognition model, and the network structure of the action recognition model is an improved posture three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the posture three-dimensional convolutional network. It is beneficial to determine the action recognition result of the dynamic target based on the input three-dimensional heat map stack by adding the convolutional block attention module to the residual layer.

[0105] Optionally, the improved posture 3D convolutional network includes: a first-stage convolutional layer, a second-stage residual layer, a third-stage residual layer, a fourth-stage residual layer, a fifth-stage pooling layer, and a sixth-stage fully connected layer.

[0106] Optionally, a convolution block attention module is added to each residual block of the second-stage residual layer, the third-stage residual layer and the fourth-stage residual layer; for each residual block, the convolution block attention module is added after the convolution block of the current residual block and before the residual connection.

[0107] The action recognition module 603 to be tested is used to determine the action recognition result of the dynamic target through the improved posture three-dimensional convolutional network based on the three-dimensional heat map stack. On the basis of modules 601 and 602, the pre-trained network structure is an action recognition model of the improved posture three-dimensional convolutional network with a convolution block attention module added to the residual layer of the posture three-dimensional convolutional network. The action recognition result of the dynamic target is obtained based on the three-dimensional heat map stack corresponding to the dynamic target in the video to be recognized, which can improve the recognition accuracy of the action of the dynamic target in the video to be recognized.

[0108] Optionally, the above-mentioned action recognition module 603 to be tested can be specifically used to extract action features of the three-dimensional heat map stack through the first-stage convolutional layer, the second-stage residual layer, the third-stage residual layer and the fourth-stage residual layer in sequence to obtain target action features; perform average pooling processing on the target action features through the fifth-stage pooling layer to obtain pooled output features; and obtain action recognition results based on the pooled output features through the sixth-stage fully connected layer.

[0109] Optionally, the above-mentioned action recognition module 603 to be tested can be specifically used to, for each residual block, use the convolution block of the current residual block to perform convolution processing on the input features of the current residual block to obtain the convolution features of the current residual block; use the convolution block attention module of the current residual block to adjust the attention of the convolution features of the current residual block to obtain the attention adjustment features of the current residual block; and perform residual connection on the input features and attention adjustment features of the current residual block to obtain the output features of the current residual block.

[0110] Figure 7is a structural diagram of a motion recognition model training device provided by an embodiment of the present invention, and the device is suitable for executing the motion recognition model training method provided by an embodiment of the present invention. Figure 7 As shown, the device may specifically include:

[0111] The sample heat map stack acquisition module 701 is used to add action category labels to the sample video to obtain the labeled sample video, obtain the key point heat map corresponding to the dynamic target in the labeled sample video, and stack the key point heat maps in chronological order to obtain a three-dimensional heat map stack corresponding to the labeled sample video.

[0112] The sample heat map stack input module 702 is used to input the three-dimensional heat map stack into the action recognition model to be trained. The network structure of the action recognition model is an improved posture three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the posture three-dimensional convolutional network.

[0113] The sample action recognition module 703 is used to determine the action recognition result of the dynamic target by improving the posture three-dimensional convolutional network based on the three-dimensional heat map stack.

[0114] The parameter adjustment module 704 is used to determine the function value of the training loss function based on the action recognition result, and adjust the network parameters of the action recognition model according to the function value of the training loss function.

[0115] Optionally, the training loss function is a class-balanced focal loss function.

[0116] Optionally, the parameter adjustment module 704 can be specifically used to adjust the network parameters of the action recognition model through the Adam optimizer according to the function value of the training loss.

[0117] The action recognition model training device provided by the embodiment of the present invention can train an action recognition model that can greatly improve the recognition accuracy when classifying and identifying the actions of dynamic targets in a video based on a three-dimensional heat map stack.

[0118] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0119] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the action recognition method or action recognition model training method provided in any of the above embodiments is implemented.

[0120] An embodiment of the present invention further provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the action recognition method or action recognition model training method provided in any of the above embodiments is implemented.

[0121] The embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the action recognition method or action recognition model training method as described in any one of the embodiments of the present invention.

[0122] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of an electronic device suitable for implementing an embodiment of the present invention. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0123] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0124] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0125] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0126] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0127] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be provided in a processor, for example, may be described as: a processor including a module for acquiring a heat map stack to be tested, an input module for acquiring a heat map stack to be tested, and an action recognition module to be tested; or, may be described as: a processor including a module for acquiring a labeled sample video, a sample heat map stack input module, a sample action recognition module, and a parameter adjustment module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0129] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device implements: obtaining a key point heat map corresponding to a dynamic target in a video to be identified, and stacking the key point heat map in chronological order to obtain a three-dimensional heat map stack corresponding to the dynamic target; inputting the three-dimensional heat map stack into a pre-trained action recognition model, the network structure of the action recognition model is an improved posture three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the posture three-dimensional convolutional network; and determining the action recognition result of the dynamic target through the improved posture three-dimensional convolutional network based on the three-dimensional heat map stack.

[0130] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for motion recognition, characterized in that: include: Obtain a key point heat map corresponding to a dynamic target in a video to be identified, and stack the key point heat maps in chronological order to obtain a three-dimensional heat map stack corresponding to the dynamic target; Inputting the three-dimensional heat map stack into a pre-trained action recognition model, wherein the network structure of the action recognition model is an improved posture three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the posture three-dimensional convolutional network; as well as Based on the three-dimensional heat map stack, the action recognition result of the dynamic target is determined by the improved posture three-dimensional convolutional network.

2. The action recognition method according to claim 1, characterized in that: The improved posture three-dimensional convolutional network includes: a first-stage convolutional layer, a second-stage residual layer, a third-stage residual layer, a fourth-stage residual layer, a fifth-stage pooling layer, and a sixth-stage fully connected layer; The step of determining the action recognition result of the dynamic target based on the three-dimensional heat map stack by using the improved posture three-dimensional convolutional network includes: Extracting action features from the three-dimensional heat map stack through the first-stage convolutional layer, the second-stage residual layer, the third-stage residual layer, and the fourth-stage residual layer in sequence to obtain target action features; Performing average pooling processing on the target action feature through the fifth stage pooling layer to obtain a pooling output feature; and The action recognition result is obtained through the sixth-stage fully connected layer based on the pooled output features.

3. The action recognition method according to claim 2, characterized in that: The convolution block attention module is added to each residual block of the second-stage residual layer, the third-stage residual layer and the fourth-stage residual layer; for each residual block, the convolution block attention module is added after the convolution block of the current residual block and before the residual connection; The extracting action features from the three-dimensional heat map stack through the second-stage residual layer, the third-stage residual layer and the fourth-stage residual layer includes: For each of the residual blocks, using the convolution block of the current residual block to perform convolution processing on the input features of the current residual block to obtain the convolution features of the current residual block; Using the convolution block attention module of the current residual block to perform attention adjustment on the convolution feature of the current residual block to obtain the attention adjustment feature of the current residual block; and Perform a residual connection on the input features of the current residual block and the attention adjustment features to obtain the output features of the current residual block.

4. The action recognition method according to claim 1, characterized in that: The step of obtaining a key point heat map corresponding to a dynamic target in a video to be identified includes: Selecting a currently applicable target detection model from a plurality of candidate target detection models based on the image region features corresponding to the dynamic target; Detecting and acquiring the dynamic target from the video to be identified by using the currently applicable target detection model; and The key points of the dynamic target are detected by a key point detection model, and the key point heat map is generated based on the key points of the dynamic target.

5. A method for training an action recognition model, characterized in that: include: Add action category labels to sample videos to obtain labeled sample videos; Obtaining a key point heat map corresponding to the dynamic target in the labeled sample video, and stacking the key point heat maps in chronological order to obtain a three-dimensional heat map stack corresponding to the labeled sample video; Inputting the three-dimensional heat map stack into the action recognition model to be trained, wherein the network structure of the action recognition model is an improved posture three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the posture three-dimensional convolutional network; Based on the three-dimensional heat map stack, determining the action recognition result of the dynamic target through the improved posture three-dimensional convolutional network; as well as The function value of the training loss function is determined based on the action recognition result, and the network parameters of the action recognition model are adjusted according to the function value of the training loss function.

6. The method for training an action recognition model according to claim 5, characterized in that: The training loss function is a class-balanced focal loss function.

7. The method for training an action recognition model according to claim 5, characterized in that: The adjusting the network parameters of the action recognition model according to the function value of the training loss function includes: According to the function value of the training loss, the network parameters of the action recognition model are adjusted through the Adam optimizer.

8. A motion recognition device, characterized in that: include: The module for acquiring the stack of heat maps to be tested is used to acquire the key point heat maps corresponding to the dynamic target in the video to be identified, and to stack the key point heat maps in chronological order to obtain a three-dimensional heat map stack corresponding to the dynamic target; A heat map stack input module to be tested, used for inputting the three-dimensional heat map stack into a pre-trained action recognition model, wherein the network structure of the action recognition model is an improved attitude three-dimensional convolutional network obtained by adding a convolutional block attention module to the residual layer of the attitude three-dimensional convolutional network; as well as The action recognition module to be tested is used to determine the action recognition result of the dynamic target through the improved posture three-dimensional convolutional network based on the three-dimensional heat map stack.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the action recognition method as described in any one of claims 1 to 4, or the action recognition model training method as described in any one of claims 5 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the action recognition method as described in any one of claims 1 to 4, or the action recognition model training method as described in any one of claims 5 to 7.

Images