Human body action recognition method based on long short-term memory network and Transform algorithm

Abstract

The invention relates to a human body action recognition method and device based on a long short-term memory network and a Transform algorithm and computer equipment. The method comprises the following steps: acquiring a key point thermodynamic diagram corresponding to image data; the key point thermodynamic diagram comprises key point data of the target object; extracting a target position feature of a target dimension in the key point thermodynamic diagram according to the target recurrent neural network, and constructing a position feature diagram of the target object based on the target position feature and the key point thermodynamic diagram; and performing image recognition on the position feature map based on the sequence model and the activation function to obtain an action recognition result of the target object. By adopting the method, the efficiency of the human body action recognition method can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, apparatus, computer device, computer-readable storage medium and computer program product for human motion recognition based on a long short-term memory network and a Transformer algorithm. Background Art

[0002] Understanding human behavior is one of the most important tasks in computer vision. Human action recognition methods can be used to identify images and obtain human action categories, thus providing a data foundation for a wide range of applications such as human-computer interaction, robotics, and game control.

[0003] The current human action recognition method extracts the position features of each dimension of each key point data of the target object based on the GCN network (Graph Convolution Network), and constructs a key point feature map based on the position features of each dimension. Then, the key point feature map is used for image recognition based on the MLP (Multilayer Perceptron) network to obtain the action recognition result of the target object.

[0004] However, in the current human action recognition method, the GCN network extracts the position features of each dimension of the key point data, which makes the data volume of the key point feature map large, and then makes the MLP network spend a lot of time processing the key point feature map. Therefore, the efficiency of the current image recognition method is low. Summary of the invention

[0005] Based on this, it is necessary to provide a human motion recognition method, device, computer equipment, computer-readable storage medium and computer program product based on long short-term memory network and Transformer algorithm to address the above technical problems.

[0006] In a first aspect, the present application provides a method for human action recognition based on a long short-term memory network and a Transformer algorithm, comprising:

[0007] Obtaining a key point heat map corresponding to the image data; the key point heat map includes key point data of the target object;

[0008] Extracting target position features of the target dimension in the key point heat map according to the target recurrent neural network, and constructing a position feature map of the target object based on the target position features and the key point heat map;

[0009] Image recognition is performed on the position feature map based on a sequence model and an activation function to obtain an action recognition result of the target object.

[0010] In one embodiment, extracting target position features of the target dimension in the key point heat map according to the target recurrent neural network, and constructing a position feature map of the target object based on the target position features and the key point heat map, includes:

[0011] Inputting the key point heat map into the target recurrent neural network, and extracting the full-dimensional position features of each key point data through the target recurrent neural network;

[0012] Based on the target recurrent neural network, the position features of the full dimension are screened to obtain the target position features of the target dimension;

[0013] A feature matrix diagram is constructed according to each of the target position features, and the feature matrix diagram and the key point heat map are spliced ​​to obtain a position feature map of the target object.

[0014] In one embodiment, performing image recognition on the position feature map based on the sequence model and the activation function to obtain the action recognition result of the target object includes:

[0015] Dividing the position feature graph to obtain initial position feature subgraphs, and position encoding each of the initial position feature subgraphs to obtain position feature subgraphs;

[0016] Determining position relationship data based on the position relationship between each of the position feature subgraphs;

[0017] Image recognition is performed on each of the position feature subgraphs according to the sequence model, the position relationship data and the activation function to obtain an output result, and an action recognition result of the target object is determined based on the output result.

[0018] In one embodiment, performing image recognition on each of the position feature subgraphs according to the sequence model, the position relationship data and the activation function to obtain an output result includes:

[0019] Determine each of the position feature subgraphs as a patch embedding, and determine the target action category data and the position relationship data as a class token; the target action category data is the action feature data of the action category to be classified;

[0020] Performing image recognition on the patch embedding and the class token according to the sequence model to obtain an initial output result;

[0021] The initial output result is mapped according to the activation function to obtain an output result.

[0022] In one embodiment, the output result includes a probability of a target action category, and determining the action recognition result of the target object based on the output result includes:

[0023] Determining whether the probability of the target action category is greater than or equal to a preset probability threshold;

[0024] If the probability is greater than or equal to the probability threshold, determining that the action recognition result of the target object is a target action category;

[0025] If the probability is less than the probability threshold, it is determined that the action recognition result of the target object is a non-target action category.

[0026] In one of the embodiments, before extracting the target position features of the target dimension in the key point heat map according to the target recurrent neural network and constructing the position feature map of the target object based on the target position features and the key point heat map, the method further includes:

[0027] Acquire an image sample set; the image sample set includes an image training sample set and an image verification sample set;

[0028] Training the recurrent neural network based on the image training sample set to obtain a trained recurrent neural network, and verifying the trained recurrent neural network based on the image verification sample set to obtain a verification result;

[0029] Determining whether the verification result meets a preset training completion condition;

[0030] When the verification result satisfies the training completion condition, the trained recurrent neural network is determined to be the target recurrent neural network.

[0031] In one embodiment, the verification result is a loss value, and after determining whether the verification result satisfies a preset training completion condition, the method further includes:

[0032] When the verification result does not satisfy the training completion condition, determining a gradient value based on the loss value and a chain algorithm;

[0033] Updating parameters of the trained recurrent neural network according to the gradient descent method and the gradient value; the parameters are used to determine the target dimension;

[0034] The step of training the recurrent neural network based on the image training sample set is performed until the verification result satisfies the training completion condition.

[0035] In one embodiment, obtaining a key point heat map corresponding to the image data includes:

[0036] Acquire initial image data, and preprocess the initial image data according to a preset image format to obtain image data;

[0037] Based on the human body posture estimation model, key point data of the target object in the image data is extracted to obtain a key point heat map.

[0038] In the second aspect, the present application also provides a human action recognition device based on a long short-term memory network and a Transformer algorithm, comprising:

[0039] An acquisition module is used to acquire a key point heat map corresponding to the image data; the key point heat map contains key point data of the target object;

[0040] An extraction module, configured to extract target position features of a target dimension in the key point heat map according to a target recurrent neural network, and construct a position feature map of the target object based on the target position features and the key point heat map;

[0041] The recognition module is used to perform image recognition on the position feature map based on a sequence model and an activation function to obtain an action recognition result of the target object.

[0042] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0043] Obtaining a key point heat map corresponding to the image data; the key point heat map includes key point data of the target object;

[0044] Extracting target position features of the target dimension in the key point heat map according to the target recurrent neural network, and constructing a position feature map of the target object based on the target position features and the key point heat map;

[0045] Image recognition is performed on the position feature map based on a sequence model and an activation function to obtain an action recognition result of the target object.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0047] Obtaining a key point heat map corresponding to the image data; the key point heat map includes key point data of the target object;

[0048] Extracting target position features of the target dimension in the key point heat map according to the target recurrent neural network, and constructing a position feature map of the target object based on the target position features and the key point heat map;

[0049] Image recognition is performed on the position feature map based on a sequence model and an activation function to obtain an action recognition result of the target object.

[0050] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0051] Obtaining a key point heat map corresponding to the image data; the key point heat map includes key point data of the target object;

[0052] Extracting target position features of the target dimension in the key point heat map according to the target recurrent neural network, and constructing a position feature map of the target object based on the target position features and the key point heat map;

[0053] Image recognition is performed on the position feature map based on a sequence model and an activation function to obtain an action recognition result of the target object.

[0054] The above-mentioned human action recognition method, device, computer equipment, computer-readable storage medium and computer program product based on long short-term memory network and Transformer algorithm obtain the key point heat map corresponding to the image data; the key point heat map contains the key point data of the target object; the target position features of the target dimension in the key point heat map are extracted according to the target recurrent neural network, and the position feature map of the target object is constructed based on the target position features and the key point heat map; the position feature map is image recognized based on the sequence model and activation function to obtain the action recognition result of the target object. According to this method, the target position features of the target dimension in the key point heat map are extracted through the recurrent neural network, and the position feature map is constructed based on the target position features and the key point heat map, which reduces the data volume of the position feature map. Then, the position feature map with less data volume is image recognized based on the sequence model and activation function to obtain the action recognition result, which reduces the data processing time and improves the efficiency of the human action recognition method. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 1 is a flow chart of a method for human action recognition based on a long short-term memory network and a Transformer algorithm in one embodiment;

[0057] Figure 2 A schematic diagram of a process for determining a position feature map in one embodiment;

[0058] Figure 3 A schematic diagram of a process for determining an action recognition result in one embodiment;

[0059] Figure 4 A schematic diagram of a process for determining an output result in one embodiment;

[0060] Figure 5 A schematic diagram of a model for determining an initial output result in one embodiment;

[0061] Figure 6 A schematic diagram of a process for determining a target recurrent neural network in one embodiment;

[0062] Figure 7 A schematic diagram of a process for updating parameters of a trained recurrent neural network in one embodiment;

[0063] Figure 8 A schematic diagram of a process for determining an action recognition result based on an output result in one embodiment;

[0064] Fig. 9 A schematic diagram of a process for obtaining a key point heat map in one embodiment;

[0065] Fig.10 is a flow chart of a method for human action recognition based on a long short-term memory network and a Transformer algorithm in another embodiment;

[0066] Fig.11 is a structural block diagram of a human motion recognition device based on a long short-term memory network and a Transformer algorithm in one embodiment;

[0067] Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] Understanding human behavior is one of the most important tasks in computer vision. Human action recognition methods can be used to identify images and obtain human action categories, thus providing a data foundation for a wide range of applications such as human-computer interaction, robotics, and game control.

[0070] The current human action recognition method extracts the positional features of each key point data of the target object in the image data in each dimension through the GCN network. For each key point data, the GCN network determines the positional features of the key point data in each dimension with other key point data based on the positional features of the key point data in each dimension, and determines the dependency relationship between the key point represented by the key point data and other key points, thereby constructing a skeleton feature map of the target object. The GCN network constructs a key point feature map based on the skeleton feature map of the target object and the positional features of each key point data in each dimension. Then, the MLP network performs image recognition based on the key point feature map of the target object to obtain the action recognition result of the target object.

[0071] However, the current human action recognition method has a large amount of data in the key point feature map, because the key point feature map contains the position characteristics of each key point data in each dimension and the skeleton feature map of the target object. Therefore, the MLP network will take a lot of time to process a large amount of data, and the efficiency is low. In addition, in the process of constructing the key point feature map of the GCN network, a lot of time will be spent on determining the dependency relationship between the key points, further reducing the processing efficiency of the human action recognition method.

[0072] Based on the above-mentioned traditional technology, the embodiment of the present application provides a method for human motion recognition based on a long short-term memory network and a Transformer algorithm, which extracts the target position features of the target dimension in the key point heat map through a recurrent neural network, and constructs a position feature map based on the target position features and the key point heat map, thereby reducing the amount of data in the position feature map. Then, image recognition is performed on the position feature map with a small amount of data based on the sequence model and the activation function to obtain the action recognition result, thereby reducing the data processing time and improving the efficiency of the human motion recognition method. In addition, the present application extracts the key point data of the target object in the image data according to the fixed key points through the human posture estimation model, thereby avoiding the determination of the dependency between the key points based on the position features of the key point data, and can quickly obtain the key point human body map, further improving the efficiency of the human motion recognition method.

[0073] In one embodiment, Figure 1As shown, a human motion recognition method based on a long short-term memory network and a Transformer algorithm is provided. The embodiment of the present application takes the method applied to a computer device as an example for explanation. The embodiment of the present application does not limit the execution device of the human motion recognition method based on a long short-term memory network and a Transformer algorithm, and includes the following steps 102 to 106:

[0074] Step 102, obtaining a key point heat map corresponding to the image data.

[0075] Among them, the key point heat map contains the key point data of the target object.

[0076] In implementation, a computer device acquires image data containing a target object, and extracts key point data of the target object in the image data based on a human posture estimation model to obtain a key point heat map.

[0077] Specifically, the computer device acquires initial image data and preprocesses the initial image data to obtain image data. Then, the computer device extracts key point data of the target object in the image data according to the human posture estimation model to obtain a key point heat map.

[0078] In an optional embodiment, the computer device obtains video data of the target object and transcodes the video data to obtain each image data. Then, the computer device extracts key point data of the target object in the image data according to the human posture estimation model for each image data to obtain a key point heat map.

[0079] Step 104, extracting target position features of the target dimension in the key point heat map according to the target recurrent neural network, and constructing a position feature map of the target object based on the target position features and the key point heat map.

[0080] In implementation, the computer device obtains an image sample set, and determines a target recurrent neural network based on the image sample set and the recurrent neural network. Then, the computer device inputs the key point heat map into the target recurrent neural network, and extracts the target position features of the target dimension in the key point heat map through the target recurrent neural network. Then, the computer device constructs a position feature map of the target object based on the target position features and the key point heat map.

[0081] Specifically, the computer obtains an image sample set, and trains a recurrent neural network based on the image sample set to obtain a target recurrent neural network that meets the training completion condition. Then, the computer device inputs the key point heat map into the target recurrent neural network, extracts each position feature through the target recurrent neural network, and selects the target position feature of the target dimension from each position feature. Then, the computer device constructs a position feature map of the target object based on the target position feature and the key point heat map.

[0082] Step 106, performing image recognition on the position feature map based on the sequence model and the activation function to obtain an action recognition result of the target object.

[0083] In implementation, the computer device preprocesses the position feature graph to obtain each position feature subgraph, and then performs image recognition on each position feature subgraph according to the sequence model and the activation function to obtain an image recognition result of the target object.

[0084] Specifically, the computer device performs division processing and position encoding processing on the position feature graph to obtain each position feature subgraph. Then, the computer device determines the position relationship between each position feature subgraph, thereby generating position relationship data. The computer device performs image recognition on each position feature subgraph according to the position relationship data, the sequence model and the activation function to obtain the output result of the target object, and determines the action recognition result of the target object based on the output result.

[0085] In the above-mentioned human action recognition method based on long short-term memory network and Transformer algorithm, the target position features of the target dimension in the key point heat map are extracted through the recurrent neural network, and the position feature map is constructed based on the target position features and the key point heat map, thereby reducing the data volume of the position feature map. Then, image recognition is performed on the position feature map with less data volume based on the sequence model and activation function to obtain the action recognition result, thereby reducing the data processing time and improving the efficiency of the human action recognition method.

[0086] In an exemplary embodiment, Figure 2 As shown, the specific processing process of step 104 includes steps 202 to 206. Among them:

[0087] Step 202: input the key point heat map into the target recurrent neural network, and extract the full-dimensional position features of each key point data through the target recurrent neural network.

[0088] Among them, the target recurrent neural network is the target LSTM network, and the key point heat map contains 17 key point data of the target object.

[0089] During implementation, the computer device inputs the key point heat map into the target LSTM network, and extracts the full-dimensional position features of each key point data through the target LSTM network.

[0090] Specifically, the computer device inputs the key point heat map into the target LSTM network, and extracts the position features of each of the 17 key point data in 226 dimensions through the target LSTM network.

[0091] In an exemplary embodiment, the target LSTM network is pre-set with a processing order of 17 key point data, and the processing order represents the order from top to bottom between the 17 key points when the target object is standing normally. The computer device inputs the key point heat map into the target LSTM network, and extracts the position features of each of the 17 key point data in 226 dimensions through the target LSTM network and the processing order. The position features of each key point data are extracted by the processing order, and the serialized modeling of the key point data is realized.

[0092] Step 204, based on the target recurrent neural network, the position features of all dimensions are screened to obtain the target position features of the target dimension.

[0093] During implementation, the computer device filters the position features of all dimensions corresponding to each key point data through a target recurrent neural network to obtain the target position features of the target dimension.

[0094] Specifically, the parameters in the target recurrent neural network can determine the target position features of the target dimension required for each key point data. Therefore, the target recurrent neural network in the computer device screens the position features of the full dimension corresponding to each key point data to obtain the target position features of the target dimension.

[0095] In an optional embodiment, the computer device includes a key point heat map corresponding to each image data. The computer device inputs the key point heat map into the target LSTM network for each key point heat map, and extracts the position features of each key point data in 226 dimensions of the 17 key point data through the target LSTM network and the processing sequence. Then, the computer device screens the position features of the 226 dimensions through the target LSTM network to obtain the target position features of the target dimension. Since the target recurrent neural network is a target LSTM network, and the target LSTM network processes multiple key point heat maps corresponding to a video data, the position features will also have some time series features. The target position features of the target dimension are extracted through the target LSTM network, and the position of each human key point in each key point heat map is converted into a richer time series feature representation.

[0096] Step 206, construct a feature matrix diagram according to each target position feature, and splice the feature matrix diagram and the key point heat map to obtain a position feature map of the target object.

[0097] In implementation, the computer device constructs a feature matrix diagram based on the target position features corresponding to each key point data. Then, the computer device splices the feature matrix diagram and the key point heat map together to obtain a position feature map of the target object.

[0098] Specifically, the computer device combines the target position features corresponding to each key point data to obtain a 17*64*48 feature matrix. According to the correspondence between the target dimension of the feature matrix and the dimension of the key point heat map, the computer device splices the feature matrix and the key point heat map together to obtain a position feature map of the target object.

[0099] In an exemplary embodiment, the computer device contains the target position features of each key point heat map. For each key point heat map, the computer device combines the target position features corresponding to the key point heat map to obtain a 17*64*48 feature matrix map. The computer device splices the feature matrix map and the key point heat map together according to the correspondence between the target dimension of the feature matrix map and the dimension of the key point heat map to obtain a position feature map of the target object.

[0100] In this embodiment, the key point data in the key point heat map is modeled by the target recurrent neural network, so as to serialize each key point data and extract dynamic information and long-term dependencies in the sequence. At the same time, the target recurrent neural network can also help the sequence model learn a more robust feature representation and reduce errors caused by factors such as occlusion and perspective changes. In addition, the target recurrent neural network is used to serialize and model the key point data of the human body so that the sequence model can extract additional time series features in the two-dimensional posture data, which helps to enhance the feature representation of the key point heat map, thereby improving the accuracy of action recognition.

[0101] In an exemplary embodiment, Figure 3 As shown, the specific processing process of step 106 includes steps 302 to 306. Among them:

[0102] Step 302: divide the position feature graph to obtain initial position feature subgraphs, and perform position encoding on each initial position feature subgraph to obtain a position feature subgraph.

[0103] In the implementation, the computer device is pre-set with an initial position feature subgraph format and a position encoding algorithm. The computer device divides the position feature graph according to the initial position feature subgraph format to obtain each initial position feature subgraph. Then, the computer device performs position encoding on each initial position feature subgraph according to the preset position encoding algorithm to obtain a position feature subgraph corresponding to the target object.

[0104] In an exemplary embodiment, there are multiple position feature graphs in a computer device. The position encoding algorithm is an absolute position encoding algorithm. For each position feature graph, the computer device divides the position feature graph according to the initial position feature subgraph format to obtain each initial position feature subgraph corresponding to the position feature graph. Then, the computer device performs position encoding processing on each initial position feature subgraph according to the absolute position encoding algorithm to obtain a position feature subgraph corresponding to the target object. Due to the use of the self-attention mechanism in the subsequent sequence model, the sequence model requires additional position encoding to understand the relative position of words in the sequence. Therefore, each initial position feature subgraph is position encoded to facilitate the subsequent sequence model to use the sequence information of the 17 key point data in the position feature graph. In addition, by position encoding the initial position feature subgraph, the spatial position relationship between each position feature subgraph is retained.

[0105] Optionally, the format of the initial position feature sub-image is determined according to the size of the sequence model and the target image data. The format of the initial position feature sub-image can be but is not limited to being set to 13px*13px. The position encoding algorithm can be an absolute position encoding algorithm, a relative position encoding algorithm, and a common position encoding algorithm, etc. The embodiments of the present application do not limit the format of the initial position feature sub-image and the position encoding algorithm.

[0106] Step 304: Determine position relationship data based on the position relationship between each position feature subgraph.

[0107] In implementation, the computer device determines, for each position feature subgraph in each position feature subgraph, the position relationship between the position feature subgraph and other position feature subgraphs in each position feature subgraph, and generates position relationship data according to the position relationship.

[0108] In an exemplary embodiment, the positional relationship characterizes the relative position and relative direction between two positional feature subgraphs. The computer device includes positional feature subgraphs corresponding to a plurality of positional feature graphs. The computer device determines the positional relationship between each positional feature subgraph and other positional feature subgraphs for each positional feature subgraph corresponding to each positional feature graph. The positional relationship is the relative direction and relative position of the positional feature subgraph and other positional feature subgraphs in the positional feature graph. The computer device generates positional relationship data according to each positional relationship.

[0109] Step 306 , performing image recognition on each position feature subgraph according to the sequence model, the position relationship data and the activation function to obtain an output result, and determining the action recognition result of the target object based on the output result.

[0110] In the implementation, the computer device constructs class tokens and patch embeddings based on the position relationship data and the feature subgraphs of each position. Then, the computer device processes the class tokens and patch embeddings according to the sequence model to obtain an initial output result, and performs data processing on the initial output result based on the activation function to obtain an output result. The output result of the computer device and a preset probability threshold determine the action recognition result of the target object.

[0111] Specifically, the output result includes the probability of the target action category, the class token is the classtoken in the neural network, and the patch embedding is the Position Embedding of the neural network. The computer device determines each position feature subgraph as the Position Embedding, and constructs the class token based on the position relationship data. Then, the computer device performs image recognition processing on the Position Embedding and the class token based on the sequence model to obtain the initial output result, and maps the initial output result according to the activation function to obtain the output result. Then, the computer device determines the size relationship between the probability and the probability threshold to obtain the judgment result. The computer device determines the action recognition result of the target object based on the judgment result.

[0112] In this embodiment, by dividing and position encoding the position feature graph, each position feature subgraph is obtained, and the position relationship data is determined based on each position feature subgraph, and the spatial position relationship between each position feature subgraph is retained. Then, based on the sequence model, the position relationship data and the position feature subgraph, the action recognition result is determined, and the action type of the target object in the image data is clarified.

[0113] In an exemplary embodiment, Figure 4 As shown, the specific processing process of performing image recognition on each position feature sub-graph according to the sequence model and the activation function to obtain the output result in step 306 includes steps 402 to 406. Among them:

[0114] Step 402 , each position feature subgraph is determined as a patch embedding, and the target action category data and the position relationship data are determined as class tokens.

[0115] The target action category data is the action feature data of the action category to be classified. The patch embedding is the Position Embedding of the deep learning model. The class token is the class token in the deep learning model.

[0116] In implementation, the computer device determines each position feature sub-graph as a patch embedding and obtains target action category data. Then, the computer device splices the target action category data and the position relationship data to obtain a class token.

[0117] Specifically, the computer device determines each position feature subgraph as a Position Embedding. When the computer device needs to identify the target action category of the target object in the image data, the user determines the identified target action category and inputs the target action category data of the target action category into the computer device. The computer device obtains the target action category data. Then, the computer device splices the position relationship data and the target action category data to obtain a class token.

[0118] For example, the computer device determines each position feature subgraph as a Position Embedding. The user determines that the target action category that the target object needs to identify is falling, and inputs the falling data into the computer device. The computer device obtains the falling data, and splices the position relationship data and the falling data to obtain the class token.

[0119] In an exemplary embodiment, when a user needs to identify multiple target action categories of a target user, the user inputs target action category data of multiple target action categories into a computer device. The computer device obtains each target action category. Then, for each target action category, the computer device splices the target action category and the time series feature data together to obtain a class token.

[0120] Step 404 , image recognition is performed on the patch embedding and the class token according to the sequence model to obtain an initial output result.

[0121] In implementation, a computer device inputs patch embeddings and class tokens into a sequence model, and the sequence model performs image recognition processing on the patch embeddings and class tokens to obtain an initial output result.

[0122] Specifically, the sequence model is a Transformer model (a deep neural network model based on the self-attention mechanism). The computer device inputs the patch embeddings and class tokens into the Transformer model, and uses the Transformer model to extract the positions and mutual connections between each key point of the target object from the patch embeddings and class tokens, and performs image recognition based on the positions and connection relationships to obtain the initial output results.

[0123] In an exemplary embodiment, a sequence model for identifying multiple target action categories is provided in a computer device, and the computer device includes class tokens of multiple target action categories. The computer device determines the sequence model corresponding to the class token for each target action category according to the target action category of the class token. Then, the computer device embeds the class token and the patch into the sequence model corresponding to the input class token, and performs image recognition of the target object by embedding the class token and the patch through the sequence model to obtain an initial output classification result. The initial output classification result characterizes whether the action performed by the target object is of the target action category.

[0124] For example, a computer device is provided with a sequence model for identifying falling, normal walking and riding, and the computer device contains class tokens for falling, normal walking and riding. The computer device determines that the sequence model corresponding to the class token of falling is the sequence model for identifying falling, and inputs the class token of falling and the patch embedding into the sequence model for identifying falling, and performs image recognition on the class token of falling and the patch embedding through the sequence model for identifying falling, and obtains an initial output result. The initial output result characterizes whether the action performed by the target object in the image data is falling.

[0125] The computer device determines that the sequence model corresponding to the normal walking class token is the sequence model for identifying normal walking, and inputs the normal walking class token and patch embedding into the sequence model for identifying normal walking, and performs image recognition on the normal walking class token and patch embedding through the sequence model for identifying normal walking to obtain an initial output result. The initial output result represents whether the action performed by the target object in the image data is normal walking.

[0126] The computer device determines that the sequence model corresponding to the cycling class token is the cycling recognition sequence model, and inputs the cycling class token and patch embedding into the cycling recognition sequence model, and performs image recognition on the cycling class token and patch embedding through the cycling recognition sequence model to obtain an initial output result. The initial output result indicates whether the action performed by the target object in the image data is cycling.

[0127] Step 406, mapping the initial output result according to the activation function to obtain an output result.

[0128] In implementation, an activation function is set in the computer device, and the computer device performs mapping processing on the initial output result according to the activation function to obtain the output result.

[0129] Specifically, the mapping function is a Softmax (soft maximum) layer of a neural network. The computer device inputs the initial output result into the Softmax layer, and the Softmax layer performs mapping processing on the initial output result to obtain the output result.

[0130] For example, the initial output result is that the ratio of the action category to the non-action category is 2:8. The computer device inputs the initial output result to the Softmax layer, and the Softmax layer performs mapping processing on the initial output result so that the sum of the ratios of the action category to the non-action category is 1, and the output result is that the ratio of the action category to the non-action category is 0.2:0.8.

[0131] In an optional embodiment, a normalization algorithm and a mapping function are provided in the computer device. The normalization algorithm is the Normalization layer of the neural network, and the mapping function is the Softmax layer of the neural network. The computer device inputs the initial output result into the Normalization layer, and normalizes the initial output result through the Normalization layer to obtain the standardized initial output result. Then, the computer device inputs the standardized initial output result into the Softmax layer, and maps the standardized initial output result through the Softmax layer to obtain the output result.

[0132] In an exemplary embodiment, Figure 5 A schematic diagram of the model for determining the initial output results. Figure 5 As shown, the computer device splices the feature matrix map (LSTM Features) and the key point heat map (Keypoints feature) to obtain the position feature map of the target object. Then, the computer device divides and position encodes the position feature map to obtain each position feature sub-map. The computer device constructs a class token (Class Toeken) based on the target action category data and the position relationship data between each position feature sub-map, and determines each position feature sub-map as a patch embedding. The computer device inputs the class token and the patch embedding into the Transformer model to obtain the initial output result, and determines the action recognition result (Class) of the target object based on the initial output.

[0133] In this embodiment, by inputting patch embedding and class tokens into the sequence model, the sequence model can understand the dependencies between key points under different human actions, and determine the initial output results based on the dependencies, and then determine the output results based on the initial output results, so as to facilitate the subsequent determination of the action recognition results of the target object.

[0134] In an exemplary embodiment, Figure 6 As shown, the output result includes the probability of the target action category, and the specific processing process of determining the action recognition result of the target object based on the output result in step 306 includes steps 602 to 606. Among them:

[0135] Step 602: determine whether the probability of the target action category is greater than or equal to a preset probability threshold.

[0136] In implementation, a probability threshold is preset in the computer device, and the computer device determines whether the probability of the action category is greater than or equal to the preset probability threshold.

[0137] In an exemplary embodiment, the action category is falling. The computer device determines whether the probability of falling is greater than or equal to a preset probability threshold. The probability is the confidence of whether the target object has performed an action of the target action category. The greater the probability, the higher the confidence of the target object performing an action of the target action category; the smaller the probability, the lower the confidence of the target object performing an action of the target action category.

[0138] In an optional embodiment, for each action category, a probability threshold corresponding to the action category may be set, or a probability threshold may be uniformly set for each action category. If a probability threshold is set for each action category, the computer device determines the probability threshold corresponding to the action category and determines whether the probability of the action category is greater than or equal to the probability threshold.

[0139] Optionally, the probability threshold can be set to, but is not limited to, 90%, which is determined based on image recognition requirements. The embodiment of the present application does not limit the probability threshold.

[0140] Step 604: If the probability is greater than or equal to the probability threshold, determine that the action recognition result of the target object is a target action category.

[0141] In implementation, if the probability is greater than or equal to the probability threshold, it indicates that in the image data, the probability of the target object performing an action of the target action category is relatively high. Therefore, the computer device determines the action recognition result of the target object as the target action category.

[0142] In an exemplary embodiment, the action category is falling. If the probability is greater than or equal to the probability threshold, it indicates that the probability of the target object falling in the image data is relatively high. Therefore, the computer device determines the action recognition result of the target object as falling.

[0143] In an optional embodiment, the computer device includes probabilities of multiple target action categories. The computer device determines the target action category with the highest probability as the action recognition result of the target object. For example, the computer device includes the probabilities of falling, normal walking, and riding. The probability of falling is 95%, the probability of normal walking is 90%, and the probability of riding is 95%. The computer device determines that the target action category with the highest probability is riding, and determines the action recognition result of the target object as riding.

[0144] Step 606: If the probability is less than the probability threshold, determine that the action recognition result of the target object is a non-target action category.

[0145] In implementation, if the probability is less than the probability threshold, it indicates that in the image data, the probability of the target object performing the action of the target action category is small. Therefore, the computer device determines the action recognition result of the target object as a non-target action category.

[0146] In an exemplary embodiment, the action category is falling. If the probability is less than the probability threshold, it indicates that the probability of the target object falling in the image data is small. Therefore, the computer device determines the action recognition result of the target object as not falling.

[0147] In this embodiment, the action recognition result of the target object is determined based on the size relationship between the output result and the probability threshold, and the target action category of the action performed by the target object in the image data is clarified.

[0148] In an exemplary embodiment, before extracting the target position features according to the target recurrent neural network, it is necessary to first train the recurrent neural network to obtain the target neural network. Figure 7 As shown, before step 104 is executed, the specific processing process of the human action recognition method based on the long short-term memory network and the Transformer algorithm also includes steps 702 to 708. Among them:

[0149] Step 702: Obtain an image sample set.

[0150] The image sample set includes an image training sample set and an image verification sample set.

[0151] In the implementation, the computer device obtains an initial image sample set, and annotates each initial sample image in the initial image sample set to obtain a sample image. The computer device constructs an image sample set based on each sample image. Then, the computer device segments the image sample set according to a preset segmentation ratio to obtain an image training sample set and an image verification sample set.

[0152] Specifically, the computer device obtains an initial image sample set, and the sample images in the initial image sample set include a first initial sample image representing a target action category and a second initial sample image representing a non-target action category. The target category is the action category that needs to be identified. The computer device sets a first label for the first initial sample image to obtain a first sample image, and sets a second label for the second initial sample image to obtain a second sample image. Then, the computer device constructs an image sample set based on each first sample image and each second sample image. Then, the computer device segments the image sample set according to a preset segmentation ratio to obtain an image training sample set and an image verification sample set.

[0153] In an exemplary embodiment, the target action category is falling. The computer device obtains an initial image sample set, and the sample images in the initial image sample set include a first initial sample image representing falling and a second initial sample image representing non-falling (other behavior categories). The computer device sets a first label 1 for the first initial sample image to obtain a first sample image, and sets a second label 0 for the second initial sample image to obtain a second sample image. Then, the computer device constructs an image sample set based on each first sample image and each second sample image. Then, the computer device segments the image sample set according to the segmentation ratio of image training sample set: image verification sample set = 8:2 to obtain an image training sample set and an image verification sample set.

[0154] In an optional embodiment, the recurrent network model needs to identify multiple target action categories, including falling, normal walking and riding. The computer device obtains each initial image sample set. The sample images in the initial image sample set include a first initial image sample representing a fall, a second initial image sample representing a normal walk, a third initial image sample representing a ride, and a fourth initial image sample representing other action categories (non-fall, normal walking and riding). Then, the computer device sets a first label 1 for the first initial sample image to obtain a first sample image, and sets a second label 0 for the second initial sample image, the third initial sample image and the fourth initial sample image to obtain a second sample image, a third sample image and a fourth sample image. The computer device constructs a fall image sample set according to each first sample image, each second sample image, each third sample image and the fourth sample image. Then, the computer device segments the fall image sample set according to the segmentation ratio of image training sample set: image verification sample set = 8:2 to obtain a fall image training sample set and a fall image verification sample set.

[0155] The computer device sets a first label 1 for the second initial sample image to obtain a second sample image, and sets a second label 0 for the first initial sample image, the third initial sample image, and the fourth initial sample image to obtain a first sample image, a third sample image, and a fourth sample image. The computer device constructs a normal walking image sample set according to each first sample image, each second sample image, each third sample image, and the fourth sample image. Then, the computer device segments the normal walking image sample set according to a segmentation ratio of image training sample set: image verification sample set = 8:2 to obtain a normal walking image training sample set and a normal walking image verification sample set.

[0156] The computer device sets a first label 1 for the third initial sample image to obtain a third sample image, and sets a second label 0 for the first initial sample image, the second initial sample image, and the fourth initial sample image to obtain a first sample image, a second sample image, and a fourth sample image. The computer device constructs a riding image sample set according to each first sample image, each second sample image, each third sample image, and the fourth sample image. Then, the computer device segments the riding image sample set according to a segmentation ratio of image training sample set: image verification sample set = 8:2 to obtain a riding image training sample set and a riding image verification sample set.

[0157] Optionally, the first tag may be set to 1, and the second tag may be set to 0. The embodiment of the present application does not limit the first tag and the second tag.

[0158] Optionally, the segmentation ratio can be but is not limited to image training sample set: image verification sample set = 8:2, which is determined according to the training requirements of the recurrent neural network. The embodiment of the present application does not limit the segmentation ratio.

[0159] Step 704: train the recurrent neural network based on the image training sample set to obtain a trained recurrent neural network, and verify the trained recurrent neural network based on the image verification sample set to obtain a verification result.

[0160] In the implementation, the computer device is pre-set with a recurrent neural network. The computer device inputs the sample images in the image training sample set into the recurrent neural network, performs image recognition on the sample images through the recurrent neural network, and obtains the trained recurrent neural network. Then, the computer device inputs the sample set in the image verification sample set into the trained recurrent neural network to obtain an output result. Then, the computer device performs data calculation on the output result and the label of each sample image in the image verification sample set according to the verification algorithm to obtain a verification result.

[0161] Specifically, the recurrent neural network is an LSTM (Long Short-Term Memory) network. The verification algorithm is a loss function algorithm. The computer device inputs the sample images in the image training sample set into the LSTM network, performs image recognition on the sample images through the LSTM network, and obtains the trained LSTM network. Then, the computer device inputs the sample set in the image verification sample set into the trained LSTM network to obtain an output result. The output result is the position feature of the target object in the sample image. The computer device calculates each output result and the label of each sample image in the image verification sample set according to the loss function algorithm to obtain the loss value of the trained LSTM network.

[0162] In an exemplary embodiment, the image sample set is a wrestling image sample set. The wrestling image sample set includes a wrestling image training sample set and a wrestling image verification sample set. The computer device inputs the sample images in the wrestling image training sample set into the LSTM network, performs image recognition on the sample images through the LSTM network, and obtains a trained LSTM network. Then, the computer device inputs the sample set in the wrestling image verification sample set into the trained LSTM network to obtain an output result. The computer device calculates each output result and the label of each sample image in the wrestling image verification sample set according to the loss function algorithm to obtain a loss value of the trained LSTM network.

[0163] Step 706, determining whether the verification result satisfies the preset training completion condition.

[0164] In implementation, the training completion condition of the recurrent neural network is pre-set in the computer device. The computer device determines whether the verification result meets the training completion condition of the recurrent neural network. If the verification result meets the training completion condition, the computer device executes the following step 708. If the verification result does not meet the training completion condition, the computer device executes the following step 802.

[0165] Specifically, the verification result is a loss value, and the training completion condition is a loss threshold. A loss threshold of the recurrent neural network is preset in the computer device. The computer device determines whether the loss value is less than or equal to the loss threshold. If the loss value is less than or equal to the loss threshold, the computer device determines that the verification result meets the preset training completion condition. If the loss value is greater than the loss threshold, the computer device determines that the verification result does not meet the training completion condition.

[0166] In an exemplary embodiment, the loss value is 0.2 and the loss threshold is 0.1. The computer device determines that the loss value is greater than the loss threshold, and determines that the verification result does not meet the training completion condition.

[0167] Optionally, the loss threshold can be set to, but is not limited to, 0.1, which is determined according to the training requirements of the recurrent neural network. The embodiment of the present application does not limit the loss threshold.

[0168] Step 708: When the verification result satisfies the training completion condition, the trained recurrent neural network is determined to be the target recurrent neural network.

[0169] In the implementation, when the verification result meets the training completion condition, the computer device uses the trained recurrent neural network as the target recurrent neural network, and uses the target recurrent neural network in the subsequent execution of the human action recognition method. The target recurrent neural network can extract the position features of the target action category.

[0170] Specifically, the recurrent neural network is an LSTM network. When the verification result satisfies the training completion condition, the computer device uses the trained LSTM network as the target LSTM network.

[0171] In an exemplary embodiment, the computer device trains the LSTM network based on the wrestling image sample set. When the verification result satisfies the training completion condition, the computer device determines that the trained LSTM network is a target LSTM network, and determines that the target LSTM network can extract the position features of the target object when wrestling.

[0172] In an exemplary embodiment, a computer device contains a wrestling image sample set, a normal walking image sample set, and a cycling image sample set. The computer device trains a recurrent neural network based on the wrestling image sample set to obtain a first target recurrent neural network that meets the training completion condition. At this time, the first target recurrent neural network can accurately extract the position features of the target object in the image data when it wrestles. Then, the computer device trains the first target recurrent neural network based on the normal walking image sample set to obtain a second target recurrent neural network that meets the training completion condition. At this time, the second target recurrent neural network can accurately extract the position features of the target object in the image data when it walks normally and when it wrestles. Then, the computer device trains the second target recurrent neural network based on the cycling image sample set to obtain a third target recurrent neural network that meets the training completion condition. At this time, the third target recurrent neural network can accurately extract the position features of the target object in the image data when it rides, when the target object walks normally, and when it wrestles.

[0173] In this embodiment, the recurrent neural network is trained by the image sample set to obtain a target recurrent neural network that meets the training completion condition, thereby improving the accuracy of the target recurrent neural network. Furthermore, the target position feature is extracted based on the target recurrent neural network, thereby improving the accuracy of the target position feature and the accuracy of the human action recognition method.

[0174] In an exemplary embodiment, when the verification result does not meet the training completion condition, the parameters of the recurrent neural network need to be updated. The verification result is a loss value, such as Figure 8 As shown, before step 706 is executed, the specific processing process of the human action recognition method based on the long short-term memory network and the Transformer algorithm also includes steps 802 to 806. Among them:

[0175] Step 802: When the verification result does not meet the training completion condition, the gradient value is determined based on the loss value and the chain algorithm.

[0176] In implementation, a chain algorithm is pre-set in the computer device. When the verification result does not meet the training completion condition, the computer device performs data calculations on the loss value according to the preset chain algorithm to obtain a gradient value.

[0177] In an exemplary embodiment, in a composite function, the total derivative is equal to the product of the partial derivatives of each function. In a recurrent neural network, the computer device starts from the output layer of the recurrent neural network and calculates the gradient of the loss value with respect to the parameters of each layer layer by layer. Specifically, the recurrent neural network contains an automatic differentiation tool. The computer device automatically processes the loss value through the automatic differentiation tool to obtain the gradient of the recurrent neural network.

[0178] Optionally, the loss function may be, but is not limited to, mean square error or cross entropy loss, etc. This application does not limit the loss function.

[0179] Step 804, updating the parameters of the trained recurrent neural network according to the gradient descent method and the gradient value.

[0180] Among them, the parameter is used to determine the target dimension.

[0181] In implementation, the computer device processes the gradient value according to the gradient descent method to update the parameters of the trained recurrent neural network.

[0182] Specifically, the basic formula for parameter update is shown in the following formula (1):

[0183] (1)

[0184] In the above formula (1), are model parameters, is the loss function, is the learning rate, is the gradient of the loss function with respect to the parameters.

[0185] During the updating process of the parameters of the recurrent neural network, the recurrent neural network will automatically learn the target position features of the required target dimension, thereby facilitating the subsequent screening of the target position features of the target dimension.

[0186] Optionally, the parameters of the recurrent neural network may be updated using, but not limited to, a gradient descent method and other optimization algorithms (such as Adam (Adaptive Moment Estimation)). This application does not limit the optimization algorithm.

[0187] Step 806, executing the step of training the recurrent neural network based on the image training sample set until the verification result meets the training completion condition.

[0188] In implementation, the computer device executes the above step 704 until the verification result of the recurrent neural network meets the training completion condition, and the computer device determines the trained recurrent neural network as the target recurrent neural network. The specific processing process of step 704 has been described in detail in the above embodiment, and the embodiment of this application will not be repeated here.

[0189] In this embodiment, when the verification result does not meet the training completion conditions, the parameters of the recurrent neural network are updated based on the loss value, thereby improving the accuracy of the recurrent neural network, and then the target position features of the target dimension are determined based on the parameters of the recurrent neural network, thereby improving the accuracy of the target position features.

[0190] In an exemplary embodiment, Fig. 9 As shown, the specific processing process of step 102 includes steps 902 to 904. Among them:

[0191] Step 902: acquiring initial image data, and preprocessing the initial image data according to a preset image format to obtain image data.

[0192] In implementation, an image format is pre-set in the computer device, the computer device acquires initial image data, and then adjusts the initial image data to image data in the image format.

[0193] Specifically, the computer device and the image acquisition device are connected to a network. The image acquisition device acquires initial image data of the target object and transmits the initial image data to the computer device through the network connection. The computer device acquires the initial image data. Then, the computer device adjusts the format of the initial image data according to a preset image format to obtain image data.

[0194] In an exemplary embodiment, the image acquisition device is a monitoring device. The computer device needs to identify the action category of the target object to determine whether the behavior of the target object is abnormal. The monitoring device collects video data of the target object and transmits the video data to the computer device in the form of a video stream. The computer device receives the video data and transcodes the video data to obtain initial image data of each frame. Then, the computer device preprocesses the format of the initial image data according to a preset image format for each frame of initial image data to obtain image data so that the length and width of the image data meet the processing requirements of the human posture estimation model. The image data is an RGB (Red, Green, Blue) image.

[0195] Optionally, the image format needs to meet the input requirements of the human posture estimation model, which may be but is not limited to 256px (pixel) * 192px. The embodiment of the present application does not limit the image format.

[0196] Optionally, the target object is generally a person, including men, women and children, and the image data may include, but is not limited to, one target object. The embodiment of the present application does not limit the target object.

[0197] Step 904: extract key point data of the target object in the image data based on the human posture estimation model to obtain a key point heat map.

[0198] Among them, the key point data is the position data and confidence data of the key point. In human posture estimation, the key point refers to the specific parts of the human body used to describe the posture, such as the head, shoulder, elbow, wrist, hip, knee, ankle, etc.

[0199] In implementation, a human posture estimation model is pre-set in the computer device. The computer device inputs image data into the human posture estimation model, extracts key point data of the target image in the image data through the human posture estimation model, and obtains a key point heat map.

[0200] Specifically, the human pose estimation model is the ViT Pose (ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation, a simple visual Transformer baseline for human pose estimation, where Transformer is a neural network structure) model. The computer device inputs the image data into the ViT Pose model, extracts the key point data of the target image in the image data through the human pose estimation model, and obtains a key point heat map.

[0201] In an exemplary embodiment, there are multiple frames of image data in a computer device. For each image data, the computer device inputs the image data into a ViT Pose model, performs image recognition on a target object in the image data through the ViT Pose model, and extracts 17 key point data of the target object, thereby constructing a key point heat map corresponding to the target object based on the 17 key point data. Among them, the ViT Pose model is a pre-trained ViT Pose model, and the ViT Pose model contains a multi-layer Transformer structure. The human body key point heat map of the target image is accurately and efficiently predicted by the pre-trained ViT Pose model using a multi-layer Transformer structure.

[0202] In this embodiment, the initial image data is preprocessed, and the key point data of the target object in the preprocessed image data are extracted according to the human body posture estimation model to obtain a human body key point heat map, which clarifies the relationship between the joints of the target object and facilitates subsequent data processing of the human body key point heat map.

[0203] In an exemplary embodiment, a human action recognition method based on long short-term memory network and Transformer algorithm is provided, such as Fig.10 shown. Fig.10 1 is a schematic diagram of a processing flow of a human motion recognition method based on a long short-term memory network and a Transformer algorithm in another embodiment. The human motion recognition method based on a long short-term memory network and a Transformer algorithm includes:

[0204] Step 1001, obtain initial image data, and pre-process the initial image data to obtain image data. The image data is RGB image data.

[0205] Step 1002: extract key point data of the target object in the image data based on the ViT Pose model to obtain a key point heat map.

[0206] Step 1003, extract the target position features of the target dimension in the key point heat map according to the target LSTM model, and concatenate the key point heat map and each target position feature to obtain a position feature map.

[0207] Step 1004 , dividing and position encoding the position feature map to obtain each position feature sub-map, and determining each position feature sub-map as a patch embedding.

[0208] Step 1005, determining position relationship data based on each position feature subgraph, and constructing a class token based on the position relationship data and the target action category data.

[0209] Step 1006 , performing image recognition on the class token and the patch embedding based on the sequence model and the activation function to obtain an output result.

[0210] Step 1007: determine the action recognition result of the target object based on the output result.

[0211] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0212] Based on the same inventive concept, the embodiment of the present application also provides a human motion recognition device based on a long short-term memory network and a Transformer algorithm for implementing the human motion recognition method based on a long short-term memory network and a Transformer algorithm involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the human motion recognition device based on a long short-term memory network and a Transformer algorithm provided below can refer to the limitations of the human motion recognition method based on a long short-term memory network and a Transformer algorithm above, and will not be repeated here.

[0213] In an exemplary embodiment, Fig.11 As shown, a human action recognition device 1100 based on a long short-term memory network and a Transformer algorithm is provided, comprising: an acquisition module 1101, an extraction module 1102 and a recognition module 1103, wherein:

[0214] The acquisition module 1101 is used to acquire a key point heat map corresponding to the image data; the key point heat map contains key point data of the target object.

[0215] The determination module 1102 is used to extract the target position features of the target dimension in the key point heat map according to the target recurrent neural network, and construct a position feature map of the target object based on the target position features and the key point heat map.

[0216] The recognition module 1103 is used to perform image recognition on the position feature map based on the sequence model and the activation function to obtain the action recognition result of the target object.

[0217] In an exemplary embodiment, the extraction module 1102 includes:

[0218] The first extraction submodule is used to input the key point heat map into the target recurrent neural network, and extract the full-dimensional position features of each key point data through the target recurrent neural network.

[0219] The first screening submodule is used to screen the position features of all dimensions based on the target recurrent neural network to obtain the target position features of the target dimension.

[0220] The first construction submodule is used to construct a feature matrix diagram according to each target position feature, and to splice the feature matrix diagram and the key point heat map to obtain a position feature map of the target object.

[0221] In an exemplary embodiment, the identification module 1103 includes:

[0222] The first division submodule is used to divide the position feature map to obtain initial position feature submaps, and position encode each initial position feature submap to obtain a position feature submap;

[0223] A first determination submodule, used to determine position relationship data based on the position relationship between each position feature subgraph;

[0224] The first recognition submodule is used to perform image recognition on each position feature subgraph according to the sequence model, the position relationship data and the activation function, obtain an output result, and determine the action recognition result of the target object based on the output result.

[0225] In an exemplary embodiment, the first identification submodule includes a second identification submodule and a second determination submodule. The second identification submodule includes:

[0226] The third determination submodule is used to determine each position feature subgraph as a patch embedding, and determine the target action category data and the position relationship data as a class token; the target action category data is the action feature data of the action category to be classified.

[0227] The third recognition submodule is used to perform image recognition on patch embeddings and class tokens according to the sequence model to obtain an initial output result.

[0228] The first processing submodule is used to perform mapping processing on the initial output result according to the activation function to obtain the output result.

[0229] In an exemplary embodiment, the output result includes the probability of the target action category, and the first recognition submodule includes a second recognition submodule and a second determination submodule. The second determination submodule includes:

[0230] The first judgment submodule is used to judge whether the probability of the target action category is greater than or equal to a preset probability threshold.

[0231] The fourth determination submodule is used to determine that the action recognition result of the target object is a target action category if the probability is greater than or equal to the probability threshold.

[0232] The fifth determination submodule is used to determine that the action recognition result of the target object is a non-target action category if the probability is less than the probability threshold.

[0233] In an exemplary embodiment, the human motion recognition device 1100 based on the long short-term memory network and the Transformer algorithm further includes:

[0234] The first acquisition module is used to acquire an image sample set; the image sample set includes an image training sample set and an image verification sample set.

[0235] The training module is used to train the recurrent neural network based on the image training sample set to obtain the trained recurrent neural network, and to verify the trained recurrent neural network based on the image verification sample set to obtain a verification result.

[0236] The first judgment module is used to judge whether the verification result meets the preset training completion condition.

[0237] The first determination module is used to determine that the trained recurrent neural network is a target recurrent neural network when the verification result meets the training completion condition.

[0238] In an exemplary embodiment, the verification result is a loss value, and the human action recognition device 1100 based on the long short-term memory network and the Transformer algorithm further includes:

[0239] The second determination module is used to determine the gradient value based on the loss value and the chain algorithm when the verification result does not meet the training completion condition.

[0240] An update module is used to update the parameters of the trained recurrent neural network according to the gradient descent method and the gradient value; the parameters are used to determine the target dimension.

[0241] The execution module is used to execute the steps of training the recurrent neural network based on the image training sample set until the verification result meets the training completion condition.

[0242] In an exemplary embodiment, the acquisition module 1101 includes:

[0243] The second acquisition submodule is used to acquire initial image data and preprocess the initial image data according to a preset image format to obtain image data.

[0244] The first extraction submodule is used to extract key point data of the target object in the image data based on the human posture estimation model to obtain a key point heat map.

[0245] Each module in the above-mentioned human action recognition device based on long short-term memory network and Transformer algorithm can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0246] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.12As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a human motion recognition method based on a long short-term memory network and a Transformer algorithm is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0247] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0248] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0249] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0250] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0251] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0252] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0253] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A human action recognition method based on long short-term memory network and Transformer algorithm, characterized in that: The method comprises: Obtaining a key point heat map corresponding to the image data; the key point heat map includes key point data of the target object; Extracting target position features of the target dimension in the key point heat map according to the target recurrent neural network, and constructing a position feature map of the target object based on the target position features and the key point heat map; Image recognition is performed on the position feature map based on a sequence model and an activation function to obtain an action recognition result of the target object.

2. The method according to claim 1, characterized in that The step of extracting target position features of the target dimension in the key point heat map according to the target recurrent neural network, and constructing a position feature map of the target object based on the target position features and the key point heat map, includes: Inputting the key point heat map into the target recurrent neural network, and extracting the full-dimensional position features of each key point data through the target recurrent neural network; Based on the target recurrent neural network, the position features of the full dimension are screened to obtain the target position features of the target dimension; A feature matrix diagram is constructed according to each of the target position features, and the feature matrix diagram and the key point heat map are spliced ​​to obtain a position feature map of the target object.

3. The method according to claim 1, characterized in that The performing image recognition on the position feature map based on the sequence model and the activation function to obtain the action recognition result of the target object includes: Dividing the position feature graph to obtain initial position feature subgraphs, and position encoding each of the initial position feature subgraphs to obtain position feature subgraphs; Determining position relationship data based on the position relationship between each of the position feature subgraphs; Image recognition is performed on each of the position feature subgraphs according to the sequence model, the position relationship data and the activation function to obtain an output result, and an action recognition result of the target object is determined based on the output result.

4. The method according to claim 3, characterized in that The performing image recognition on each of the position feature subgraphs according to the sequence model, the position relationship data and the activation function to obtain an output result includes: Determine each of the position feature subgraphs as a patch embedding, and determine the target action category data and the position relationship data as a class token; the target action category data is the action feature data of the action category to be classified; Performing image recognition on the patch embedding and the class token according to the sequence model to obtain an initial output result; The initial output result is mapped according to the activation function to obtain an output result.

5. The method according to claim 3, characterized in that: The output result includes the probability of the target action category, and the step of determining the action recognition result of the target object based on the output result includes: Determining whether the probability of the target action category is greater than or equal to a preset probability threshold; If the probability is greater than or equal to the probability threshold, determining that the action recognition result of the target object is a target action category; If the probability is less than the probability threshold, it is determined that the action recognition result of the target object is a non-target action category.

6. The method according to claim 1, characterized in that Before extracting the target position features of the target dimension in the key point heat map according to the target recurrent neural network and constructing the position feature map of the target object based on the target position features and the key point heat map, the method further includes: Acquire an image sample set; the image sample set includes an image training sample set and an image verification sample set; Training the recurrent neural network based on the image training sample set to obtain a trained recurrent neural network, and verifying the trained recurrent neural network based on the image verification sample set to obtain a verification result; Determining whether the verification result meets a preset training completion condition; When the verification result satisfies the training completion condition, the trained recurrent neural network is determined to be the target recurrent neural network.

7. The method according to claim 6, characterized in that The verification result is a loss value. After determining whether the verification result meets a preset training completion condition, the method further includes: When the verification result does not satisfy the training completion condition, determining a gradient value based on the loss value and a chain algorithm; Updating parameters of the trained recurrent neural network according to the gradient descent method and the gradient value; the parameters are used to determine the target dimension; The step of training the recurrent neural network based on the image training sample set is performed until the verification result satisfies the training completion condition.

8. The method according to claim 1, characterized in that The step of obtaining a key point heat map corresponding to the image data includes: Acquire initial image data, and preprocess the initial image data according to a preset image format to obtain image data; Based on the human body posture estimation model, key point data of the target object in the image data is extracted to obtain a key point heat map.

9. A human action recognition device based on long short-term memory network and Transformer algorithm, characterized in that: The device comprises: An acquisition module is used to acquire a key point heat map corresponding to the image data; the key point heat map contains key point data of the target object; An extraction module, configured to extract target position features of a target dimension in the key point heat map according to a target recurrent neural network, and construct a position feature map of the target object based on the target position features and the key point heat map; The recognition module is used to perform image recognition on the position feature map based on a sequence model and an activation function to obtain an action recognition result of the target object.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images