Human action recognition method, training method of recognition model thereof, and electronic device

Abstract

The application provides a human action recognition method, a training method of a recognition model thereof, and an electronic device. The human action recognition model comprises a feature construction module and a feature extraction module. The method comprises: acquiring video data containing human action; constructing features of the video data based on the feature construction module to obtain a fixed-step feature map corresponding to the video data; extracting features of the fixed-step feature map based on the feature extraction module to obtain a predicted human action category; iteratively training the human action recognition model according to the predicted human action category and a real human action category until a preset condition is reached to obtain a trained human action recognition model. In the training method, the feature construction module can be used to construct features of a variable-length video frame sequence to obtain a fixed-step feature map for training, thereby solving the problem of variable length of the input video data sequence in the model training process.

Description

Technical Field

[0001] This application relates to the field of intelligent recognition technology, and in particular to a human motion recognition method, a training method for the recognition model, and an electronic device. Background Technology

[0002] With the increasing prevalence of the Internet of Things (IoT) system, which connects everything, a massive amount of video data is generated daily. Recognizing human movement within videos is a crucial research topic in video understanding. Videos consist of sequences of images, and frame rate measures the number of frames a camera or motion capture system can capture per second. Each frame is a static image, but continuous frames displayed on a computer or digital device create a constantly changing video. Therefore, a higher frame rate results in smoother and more realistic video animations; generally, a higher frame rate better reproduces the realism of the scene.

[0003] Human motion recognition encompasses behaviors such as walking, running, standing, jumping, and sitting. Identifying human actions in videos is a highly challenging task, but its ability to recognize human actions in videos can greatly benefit intelligent surveillance, human-computer interaction, and healthcare. In using video to recognize and judge human actions, a single image is insufficient; it requires combining information from multiple frames of the video for comprehensive judgment. Different actions require different numbers of frames to recognize, and the time required for transitions between each action also varies. This results in the input of human motion detection models being of variable length, while the input size for deep learning is fixed. Therefore, how to input variable-length video data into human motion detection models for training or recognition is a pressing problem in the field of motion recognition. Summary of the Invention

[0004] This application provides a human motion recognition method and a training method and electronic device for the recognition model. In this training method, a feature construction module can be used to construct features from a video frame sequence of variable length to obtain a feature map with a fixed step length for training, thereby solving the problem of variable length of input video data sequence during model training and subsequent recognition.

[0005] In a first aspect, embodiments of this application provide a training method for a human motion recognition model. The human motion recognition model includes a feature construction module and a feature extraction module. The method includes: acquiring video data containing human motion, wherein the video data includes multiple frames of image data, and each frame of image data is labeled with a real human motion category; constructing features on the video data based on the feature construction module to obtain a fixed step-length feature map corresponding to the video data; extracting features from the fixed step-length feature map based on the feature extraction module to obtain a predicted human motion category; and iteratively training the human motion recognition model according to the predicted human motion category and the real human motion category until a preset condition is met to obtain a trained human motion recognition model.

[0006] In some embodiments, the feature construction module includes a feature encoding submodule, a feature fusion submodule, and a feature decoding submodule. The step of constructing features from the video data based on the feature construction module to obtain a fixed-step feature map corresponding to the video data includes: inputting the video data into the feature encoding submodule for feature encoding to obtain feature maps corresponding to each frame of the image data; inputting each feature map into the feature fusion submodule for feature fusion to obtain a feature fusion map corresponding to the video data; and inputting the feature fusion map into the feature decoding submodule for feature decoding to obtain the fixed-step feature map.

[0007] In some embodiments, the feature encoding submodule includes a convolutional neural network, the feature fusion submodule includes an addition module, and the feature decoding submodule includes a convolutional layer and an upsampling layer connected in sequence.

[0008] In some embodiments, the feature extraction module includes a first feature extraction submodule, a second feature extraction submodule, and a classification submodule. The step of extracting features from the fixed-step-length feature map based on the feature extraction module to obtain a predicted human action category includes: inputting the fixed-step-length feature map into the first feature extraction submodule for first feature extraction to obtain a first feature vector; inputting the first feature vector into the second feature extraction submodule for second feature extraction to obtain a second feature vector; and inputting the second feature vector into the classification submodule for classification to obtain the predicted human action category.

[0009] In some embodiments, the first feature extraction submodule includes a plurality of convolutional submodules and a first fully connected layer connected in sequence, wherein the convolutional submodule includes a 3D convolutional kernel and a 3D max pooling layer connected in sequence; the second feature extraction submodule includes a gated recurrent unit; and the classification submodule includes a second fully connected layer and a softmax function layer connected in sequence.

[0010] In some embodiments, the step of iteratively training the human action recognition model based on the predicted human action category and the real human action category until a preset condition is met to obtain the trained human action recognition model includes: obtaining the total loss between the predicted human action category and the real human action category based on the center-cross loss function, and iteratively adjusting the parameters of the human action recognition model using the Adam algorithm until the preset condition is met to obtain the trained human action recognition model.

[0011] In some embodiments, the center-cross loss function includes a classification loss function and a center loss function. Obtaining the total loss between the predicted human action category and the actual human action category based on the center-cross loss function includes: obtaining a classification loss based on the classification loss function; obtaining a center loss based on the center loss function; and obtaining the total loss based on the classification loss and the center loss.

[0012] Secondly, embodiments of this application provide a human motion recognition method, which includes: acquiring video data to be recognized; inputting the video data to be recognized into a human motion recognition model to obtain the human motion category corresponding to the video data to be recognized, wherein the human motion recognition model is trained using the training method described in any of the embodiments of the first aspect above.

[0013] Thirdly, embodiments of this application provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in any of the embodiments of the first or second aspect above.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method described in any one of the embodiments of the first or second aspect above.

[0015] Fifthly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in any one of the embodiments of the first or second aspect above.

[0016] Compared with existing technologies, the beneficial effects of this application are as follows: Unlike existing technologies, this application provides a human motion recognition method and a training method and electronic device for its recognition model. The human motion recognition model includes a feature construction module and a feature extraction module. The method includes: acquiring video data containing human motion, wherein the video data contains multiple frame image data, each frame image data is labeled with a real human motion category; based on the feature construction module, performing feature construction on the video data to obtain a fixed-step-length feature map corresponding to the video data; based on the feature extraction module, performing feature extraction on the fixed-step-length feature map to obtain a predicted human motion category; and iteratively training the human motion recognition model according to the predicted human motion category and the real human motion category until a preset condition is met, thus obtaining the trained human motion recognition model. In this training method, the feature construction module can be used to construct features on variable-length video frame sequences to obtain a fixed-step-length feature map for training, thereby solving the problem of variable-length input video data sequences during model training and recognition. Attached Figure Description

[0017] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise stated, the figures in the drawings do not constitute a limitation on scale.

[0018] Figure 1 This is a schematic diagram illustrating an application scenario of a human motion recognition system provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0020] Figure 3 This is a flowchart illustrating a training method for a human motion recognition model provided in an embodiment of this application.

[0021] Figure 4 This is one of the embodiments provided in this application. Figure 3 A flowchart illustrating step S20;

[0022] Figure 5 This is a partial structural diagram of a human motion recognition model provided in an embodiment of this application;

[0023] Figure 6 This is a flowchart illustrating step S30 of three steps provided in an embodiment of this application;

[0024] Figure 7 This is a partial structural diagram of another human motion recognition model provided in an embodiment of this application;

[0025] Figure 8 This is a schematic diagram of the structure of a gated loop unit provided in an embodiment of this application;

[0026] Figure 9 This is one of the embodiments provided in this application. Figure 3 A flowchart illustrating step S40;

[0027] Figure 10 This is one of the embodiments provided in this application. Figure 9 A flowchart illustrating step S41;

[0028] Figure 11 This is a flowchart illustrating a human motion recognition method provided in an embodiment of this application. Detailed Implementation

[0029] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0030] To facilitate understanding of this application, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0031] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram, in some cases, they can be divided differently from those in the device. In addition, the terms "first" and "second" used herein do not limit the data or execution order, but only distinguish between identical or similar items with essentially the same function and effect.

[0032] Currently, human action recognition algorithms can be mainly divided into traditional methods and deep learning-based methods. Traditional methods require manual feature extraction, typically involving three steps: extracting features from the input video data, encoding the features, and finally using a classifier to classify the encoded features. Deep learning-based methods use convolutional neural networks, recurrent neural networks, and graph convolutional neural networks based on skeleton information to learn spatiotemporal information in videos. Feature extraction is a crucial step in the field of human action recognition. Extracting spatial, spatiotemporal, and motion features is highly challenging, and each feature has diverse representations. Therefore, feature extraction remains a pressing problem in action recognition. Furthermore, for video data, judging human actions requires comprehensive analysis of information from multiple frames; a single image is insufficient. Different actions require different numbers of frames to recognize, and the time required for transitions between actions also varies. This results in variable-length inputs for the model, whereas deep learning requires a fixed input size.

[0033] To address the aforementioned technical issues, embodiments of this application provide a human motion recognition method and a training method and electronic device for the recognition model. In the human motion recognition model obtained by the training method, after obtaining video data containing human motion, a feature construction module is used to convert the variable-length video data into a feature map with a fixed step length for human motion recognition.

[0034] As an example, see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the human motion recognition system provided in this application embodiment. Terminal 10 connects to server 20 via a network, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0035] Terminal 10 can be used to acquire training data and build a human motion recognition model. For example, those skilled in the art can download prepared training data to the terminal and build the model structure of the human motion recognition model. The training data includes several video datasets. It is understood that terminal 10 can also be used to acquire video data for training or testing. For example, a video data acquisition device can send the acquired video data for training or testing to terminal 10, thereby allowing terminal 10 to acquire the video data for training or testing. In some embodiments, terminal 10 can be integrated with a video acquisition device.

[0036] In some embodiments, terminal 10 locally executes the training method of the human motion recognition model provided in this application embodiment to train the designed model using training data, determine the final model parameters, and thus preset the model configuration to obtain the trained human motion recognition model. In some embodiments, terminal 10 may also send training data and a constructed human motion recognition model stored on the terminal by those skilled in the art to server 20 via a network. Server 20 receives the training data and the human motion recognition model, trains the human motion recognition model using the training data, determines the final model parameters, and then sends the final model parameters to terminal 10. Terminal 10 saves the final model parameters, enabling the human motion recognition model to be configured with the final model parameters, thereby obtaining the trained human motion recognition model.

[0037] The following describes an exemplary application of the electronic device 100 provided in the embodiments of this application for training a human motion recognition model or for human motion recognition. It is understood that the electronic device 100 can both train a human motion recognition model and use the human motion recognition model to perform human motion recognition.

[0038] The electronic device 100 provided in some embodiments of this application can be a server, such as a server deployed in the cloud. When the server is used to train a human motion recognition model, it iteratively trains the human motion recognition model using video data provided by other devices or those skilled in the art, determines the final model parameters, and then configures the classifier with these final model parameters to obtain the trained human motion recognition model. When the server is used for human motion recognition, it calls the built-in trained human motion recognition model, provides the human motion recognition device with the video data to be tested, performs corresponding calculations, and obtains the corresponding human motion.

[0039] In other embodiments of this application, the electronic device 100 can also be various types of terminals such as laptops, desktop computers, or mobile devices. When the terminal is used to train a human motion recognition model, those skilled in the art input prepared video data into the terminal and design a human motion recognition model on the terminal. The terminal uses the video data to iteratively train the human motion recognition model, determines the final model parameters, and then configures the human motion recognition model with the final model parameters to obtain the trained human motion recognition model. When the terminal is used for human motion recognition, the test video data is processed accordingly to obtain a feature map with a fixed step length. Then, the built-in trained human motion recognition model is called to perform corresponding calculations on the feature map with the fixed step length to obtain the human motion.

[0040] For details, please see Figure 2The document illustrates the hardware structure of an electronic device 100 capable of executing the training method for the human motion recognition model and the human motion recognition method described in this application. The electronic device 100 may be... Figure 1 Terminal 10 is shown.

[0041] Please see Figure 2 The electronic device 100 includes a processor 101 and a memory 102 connected via a communication link. Here, the communication link can be established via a bus. Figure 2 The bus connection between China and Israel is illustrated by example. It is understood that... Figure 2 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0042] The processor 101 is configured to support the electronic device 100 in executing methods for training human action recognition models or corresponding functions within human action recognition methods. The processor 101 can be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0043] The memory 102, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the method for training a human motion recognition model or the human motion recognition method in the embodiments of this application. The processor 101, by running the non-transitory software programs, instructions, and modules stored in the memory 102, can implement the method for training a human motion recognition model or the human motion recognition method in any of the following method embodiments.

[0044] Memory 102 may include volatile memory (VM), such as random access memory (RAM); memory 1002 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 102 may also include combinations of the above types of memory.

[0045] It is understood that electronic devices also include other supporting hardware and software. Hardware may include antennas, various sensors, microphones, etc. Software may include operating systems, which are programs that manage and control the hardware and software resources of electronic devices. Software may also include various applications (apps). Other parts of the improved electronic devices not involved in the embodiments of this application will not be described here.

[0046] In some embodiments, the electronic device may also be a terminal or a server, etc. In this application embodiment, no limitations are placed on the form of the electronic device; it only needs to be able to acquire video data and have computing processing capabilities.

[0047] As can be understood from the above, the human motion recognition method provided in this application embodiment can be implemented by various types of electronic devices with processing capabilities, such as being executed by the processor of an electronic device or by other devices with computing capabilities. Other devices with computing capabilities can be smart terminals or servers that are communicatively connected to the electronic device.

[0048] The following describes the method for training a human motion recognition model provided in this application, using exemplary applications and implementations of the electronic device provided in the embodiments of this application. Please refer to... Figure 3 , Figure 3 This is a flowchart illustrating a method for training a human motion recognition model according to an embodiment of this application. It is understood that the execution entity of this training method can be one or more processors of an electronic device. The human motion recognition model includes a feature construction module and a feature extraction module, and the training method includes:

[0049] Step S10: Obtain video data containing human movements, wherein the video data contains multiple frames of image data, and each frame of image data is labeled with the category of real human movement.

[0050] Electronic devices can acquire video data through video capture devices, such as cameras, or by downloading video data containing human motion from the network. The resolution of the video data can be selected according to actual needs; for example, the resolution of the video data can be 1024*768.

[0051] It should be noted that each frame of the video data is labeled with a real human action category; these categories include actions such as walking, running, standing, jumping, and sitting. Users can label each video frame with real human actions, facilitating subsequent labeling of these categories. The specific encoding method for these categories can be found in existing technologies and will not be detailed here.

[0052] Step S20: Based on the feature construction module, perform feature construction on the video data to obtain a fixed step size feature map corresponding to the video data.

[0053] A fixed-step feature map is a feature map with a fixed step size N, where N can be 10 or other values, set according to actual needs. If N is 10, the fixed-step feature map includes 10 feature maps. It's understood that the fixed step size N should be less than the number of frames in the video data. Specifically, the feature construction module can perform feature encoding and decoding on the video data, converting variable-length video data into fixed-step feature maps.

[0054] Step S30: Based on the feature extraction module, perform feature extraction on the fixed step size feature map to obtain the predicted human action category.

[0055] Specifically, the feature extraction module can perform image mapping on the fixed step size feature map to obtain a fixed number of feature vectors corresponding to the video data, such as 10 feature vectors; then, the feature extraction module can classify the fixed number of feature vectors and output the predicted human action category.

[0056] Step S40: Based on the predicted human action category and the actual human action category, iteratively train the human action recognition model until the preset conditions are met, and obtain the trained human action recognition model.

[0057] The human action recognition model is trained by using predicted human action categories and real human action categories. The parameters of the human action recognition model are continuously adjusted until a preset condition is met, such as the human action recognition model converges or the predicted human action category output by the human action recognition model becomes closer and closer to the real human action category. At this point, the parameter is used as the human action recognition model parameter, thus obtaining the trained human action recognition model.

[0058] In the training method provided in this application embodiment, variable-length video data is converted into fixed-step feature maps for training through a feature construction module, thereby solving the problem of variable-length video data sequences input during model training. This allows the model to better cope with different completion time sequences of different actions during training, and the human action recognition model obtained after subsequent training can also better cope with various variable-length test video data, thereby realizing the recognition of human actions in various variable-length test video data.

[0059] In some embodiments, the feature construction module includes a feature encoding submodule, a feature fusion submodule, and a feature decoding submodule; please refer to the relevant documentation. Figure 4 and Figure 5 Step S20 includes:

[0060] Step S21: Input the video data into the feature encoding submodule for feature encoding to obtain the feature map corresponding to each frame of image data.

[0061] Specifically, the feature encoding submodule includes Convolutional Neural Networks (CNNs). A CNN is a feedforward neural network characterized by local connectivity and weight sharing. The basic structure of a CNN generally includes an input layer, hidden layers, and an output layer. While layers are fully connected, nodes within each layer are unconnected. Therefore, feedforward neural networks do not consider the correlation between data; the network's output is only related to the network's input at the current moment. Thus, CNNs can extract features from each frame of image data in video data, learning the shape features of human movements within a single frame.

[0062] For example, a CNN can use a 3*3 kernel with a stride of 2. The number of kernels can be 32, 64, 128, or 256. Finally, after passing through the CNN, each frame of the video data can be used to obtain a feature map of size 64*48*256.

[0063] Step S22: Input each feature map into the feature fusion submodule to perform feature fusion and obtain the feature fusion map corresponding to the video data.

[0064] Specifically, the feature fusion submodule can use an addition module for feature fusion. After obtaining the feature map of size 64*48*256 corresponding to each frame of image data, the addition module is used for feature fusion. For example, the feature fusion map F corresponding to the video data can be obtained using the following formula:

[0065] F = f1 + f2 + ... + fi + ... + fe;

[0066] Where fi is the image data of the i-th frame in the video data, and e is the total number of frames in the video data.

[0067] Step S23: Input the feature fusion map into the feature decoding submodule for feature decoding to obtain a fixed step size feature map.

[0068] After obtaining the feature fusion map, the feature decoding submodule performs feature decoding. Its network structure includes sequentially connected convolutional layers and upsampling layers. The convolutional layers can use 3x3 kernels with a stride of 1, and the number of kernels can be 128, 64, 64, or 3*N to perform convolution operations on the feature fusion map. Simultaneously, the upsampling layer has a stride of 2 to enlarge the feature map after convolution. Finally, the feature decoding submodule outputs a feature map with a fixed stride of N and a size of 1024*768*3.

[0069] In this embodiment, feature construction can be performed on variable-length video data through feature encoding submodule, feature fusion submodule, and feature decoding submodule to obtain feature maps with fixed step length. This can solve the problem of variable-length video data sequences input during model training, enabling the model to better handle different completion time sequences of different actions during training. Furthermore, the human action recognition model obtained after subsequent training can better handle various variable-length test video data, thereby enabling human action recognition on various variable-length test video data.

[0070] In some embodiments, the feature extraction module includes a first feature extraction submodule, a second feature extraction submodule, and a classification submodule. Please refer to the relevant documentation. Figure 6 and Figure 7 Step S30 includes:

[0071] Step S31: Input the fixed step size feature map into the first feature extraction submodule to perform the first feature extraction and obtain the first feature vector.

[0072] The first feature extraction submodule includes multiple convolutional submodules connected in sequence and a first fully connected layer. The convolutional submodules include 3D convolutional kernels and 3D max pooling layers connected in sequence. The 3D convolutional kernels can perform convolution operations to extract feature maps, and the 3D max pooling layers can perform dimensionality reduction operations.

[0073] For example, the first feature extraction submodule includes a first convolutional submodule, a second convolutional submodule, a third convolutional submodule, a fourth convolutional submodule, and a first fully connected layer connected in sequence. The first convolutional submodule has a 3D convolutional kernel size of 3*3*N*32, with a padding value and stride of 1, and a 3D max-pooling layer size of 2*2. The second convolutional submodule has a 3D convolutional kernel size of 3*3*32*64, with a stride and padding value of 1, and a 3D max-pooling layer size of 2*2. The third convolutional submodule has a 3D convolutional kernel size of 3*3*64*128, with a stride and padding value of 1, and a 3D max-pooling layer size of 2*2. The fourth convolutional submodule has a 3D convolutional kernel size of 3*3*128*256, with a stride and padding value of 1, and a 3D max-pooling layer size of 2*2.

[0074] Among them, the size W of the feature map output by the 3D convolution kernel out The calculation formula is as follows:

[0075]

[0076] padding is the padding value, stride is the padding step size, and w in F is the size of the feature map input to the 3D convolution kernel, and F is the feature fusion map.

[0077] In the first feature extraction submodule described above, the feature map with a fixed stride of N is processed by the 3D convolution kernel of the first convolution submodule to obtain a feature map of size 1024*768*3*N. After passing through the 3D max pooling layer of the first convolution submodule, a feature map of size 512*384*32 is obtained. Then, after passing through the second convolution submodule, a feature map of size 256*192*64 is obtained. Next, after passing through the third convolution submodule, a feature map of size 128*96*128 is obtained. Then, after passing through the fourth convolution submodule, a feature map of size 64*48*256 is obtained. Finally, after passing through a fully connected layer, an N*1*1024 feature vector is output. Wherein, in Figure 7 In the illustrated embodiment, N is 10. In practical applications, the number of convolutional submodules, the size of the 3D convolutional kernel, and the size of the 3D max-pooling layer can all be set according to actual needs, and are not limited to the constraints of this embodiment. It can be seen that a fixed number of feature vectors can be obtained through the first feature extraction submodule, and the number of these feature vectors is equal to the fixed stride N.

[0078] Step S32: Input the first feature vector into the second feature extraction submodule to perform second feature extraction and obtain the second feature vector.

[0079] Specifically, the second feature extraction submodule includes a gated recurrent unit (GRU). The structure of the GRU is as follows: Figure 8 As shown, GRU supports gating of the hidden state, indicating that the model has a dedicated mechanism to determine when to update and reset the hidden state. GRU combines the forget gate and input gate into an update gate, and also combines the cell state and the hidden state. The update gate determines the amount of prior information to be passed to the future; a larger update gate value means more state information is introduced from the previous time step. The reset gate determines the amount of prior information to be forgotten; a smaller reset gate value indicates that the information from the previous time step is less important than the current one, and therefore should be ignored. The current memory gate is used, which calculates the current state value by combining the previous hidden state with the current input. Because each hidden unit has separate reset and update gates, each hidden unit will learn to capture dependencies across different time ranges. Units that learn to capture short-term dependencies will tend to use the reset gate, while those that capture long-term dependencies will tend to use the update gate.

[0080] In GRU, the inputs to the reset and update gates are both the input x of the current time step. t The hidden state h of the previous time step t-1 Given that the outputs of the two gates are provided by two fully connected layers containing sigmoid activation functions, update gate z. t Reset gate r t Current memory status and output value h t The calculation expressions are as follows:

[0081] z t =σ(W z ·[h t-1 x t ])

[0082] r t =σ(W r ·[h t-1 x t ])

[0083]

[0084]

[0085] Among them, W r W z W and h are weighting coefficients. t-1 It is the output value of the previous GRU unit.

[0086] In this embodiment, GRU is used to extract features from the first feature vector, which helps to extract long-term spatiotemporal features from video data. Furthermore, the hidden layers can retain the temporal information between different sequences as time changes, enabling the extracted features to learn the temporal features of human actions.

[0087] Step S33: Input the second feature vector into the classification submodule for classification to obtain the predicted human action category.

[0088] Specifically, the classification submodule consists of a second fully connected layer and a softmax function layer connected sequentially. Finally, after the second feature vector is input into the second fully connected layer and the softmax function layer for classification calculation, the corresponding predicted human action class can be output.

[0089] In this embodiment, 3D convolution and GRU networks are used simultaneously for long-term temporal modeling to extract long-term spatiotemporal features, which can better learn the spatiotemporal features of human movements. The human movement recognition model obtained after subsequent training can improve the accuracy of human movement recognition.

[0090] In some of these embodiments, please refer to Figure 9 Step S40 includes:

[0091] Step S41: Obtain the total loss between the predicted human action category and the real human action category based on the center crossover loss function, and use the Adam algorithm to iteratively adjust the parameters of the human action recognition model until the preset conditions are met, thus obtaining the trained human action recognition model.

[0092] Specifically, the Adam algorithm is used to iteratively optimize the parameters of the human action recognition model. The number of iterations can be set to 100,000, the initial learning rate to 0.001, and the weight decay to 0.0005. Every 1000 iterations, the learning rate decays to 1 / 10 of its original value. In this embodiment, training the model using the Adam algorithm yields the various parameters of the human action recognition model, resulting in the trained human action recognition model.

[0093] In addition, in this embodiment, the center crossover loss function is used as the loss function, which can speed up the model convergence speed and improve the accuracy of the human action recognition model after training.

[0094] Specifically, in some embodiments, the center-cross loss function includes a classification loss function and a center loss function; please refer to [link to relevant documentation]. Figure 10 Step S41 includes:

[0095] Step S411: Obtain the classification loss based on the classification loss function.

[0096] In training a human action recognition model, introducing a classification loss function can amplify the differences between features of different categories of human actions, which helps to accelerate model convergence. For example, the classification loss Ls can be calculated using the following formula:

[0097]

[0098] Where n is the total number of categories of human movements, m is the total number of training samples, and x ij Let W be the predicted feature of the i-th type of human action in the j-th training sample, W be the parameter of the softmax function layer, and b be the bias value of the softmax function layer. T This represents the transpose of W.

[0099] Step S412: Obtain the central loss based on the central loss function.

[0100] In training human action recognition models, a center loss function can be introduced to reduce the feature difference between features of similar human actions and the central features of human action categories, i.e., to constrain the distance between features of similar human actions. For example, the center loss Lc can be calculated using the following formula:

[0101] ;

[0102] Among them, C yi X is the central feature of the i-th type of human movement. i Let be the predicted features of the i-th type of human movement.

[0103] Step S413: Obtain the total loss based on the classification loss and center loss.

[0104] For example, the total loss can be calculated using the following formula:

[0105] L = L s +λ*L c ;

[0106] Wherein, λ is the coefficient of the center loss function. By adjusting λ, the ratio of the center loss function to the classification loss function can be adjusted. It can be set to 10. In practical applications, it can be set according to actual needs.

[0107] In this embodiment of the application, the central cross-entropy loss function is constructed in the above manner and used as a constraint, which can effectively accelerate the convergence speed of the human action recognition model. Moreover, the human action recognition model after training has good feature discrimination ability. After feature extraction, the same type of human action has similar features, while the features of different types of human action have large differences after feature extraction, which can improve the accuracy of the human action recognition model after subsequent training.

[0108] This application also provides a method for human motion recognition. Please refer to [link / reference]. Figure 11 The human motion recognition method includes:

[0109] Step S100: Obtain the video data to be identified.

[0110] The specific method by which the electronic device acquires the video data to be identified can be referred to step S10 in the training embodiment, and will not be repeated here.

[0111] Step S200: Input the video data to be identified into the human motion recognition model to obtain the human motion category corresponding to the video data to be identified. The human motion recognition model is trained using the training method described in any embodiment of the first aspect.

[0112] The electronic device has a built-in human motion recognition application. The human motion recognition model is encapsulated within this application. The application calls the human motion recognition model to perform human motion recognition on the aforementioned test human actions. After a series of calculations, it outputs a predicted human motion recognition category. It is understood that this human motion recognition model is trained using the method described in the above embodiments for training the human motion recognition detection model. It has the same structure and function as the human motion recognition model described in the above embodiments, and will not be elaborated further here.

[0113] This application also provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example, to execute the training method and human action recognition method for the human action recognition model described in the above embodiments. The computer-readable storage medium can be a storage medium such as FRAM, ROM, PROM, EPROM, EEE PROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; it can also be various devices including one or any combination of the above-mentioned memories. The executable instructions can be in the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0114] This application also provides a computer program product, including a computing program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform the training method for the human motion recognition model and the human motion recognition method described in the above embodiments.

[0115] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for at least one computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for a human motion recognition model, characterized in that, The human motion recognition model includes a feature construction module and a feature extraction module. The feature construction module includes a feature encoding submodule, a feature fusion submodule, and a feature decoding submodule. The method includes: Acquire video data containing human movements, wherein the video data contains multiple frames of image data, and each frame of image data is labeled with a real human movement category; The video data is input into the feature encoding submodule for feature encoding to obtain feature maps corresponding to each frame of the image data; Each of the aforementioned feature maps is input into the feature fusion submodule for feature fusion to obtain the feature fusion map corresponding to the video data; The feature fusion map is input into the feature decoding submodule for feature decoding to obtain a fixed step size feature map corresponding to the video data; based on the feature extraction module, features are extracted from the fixed step size feature map to obtain the predicted human action category; Based on the predicted human action category and the actual human action category, the human action recognition model is iteratively trained until a preset condition is met, resulting in a trained human action recognition model.

2. The method according to claim 1, characterized in that, The feature encoding submodule includes a convolutional neural network, the feature fusion submodule includes an addition module, and the feature decoding submodule includes a convolutional layer and an upsampling layer connected in sequence.

3. The method according to claim 1, characterized in that, The feature extraction module includes a first feature extraction submodule, a second feature extraction submodule, and a classification submodule. Based on the feature extraction module, feature extraction is performed on the fixed-step-length feature map to obtain the predicted human action category, including: The fixed step size feature map is input into the first feature extraction submodule to perform first feature extraction, and a first feature vector is obtained; The first feature vector is input into the second feature extraction submodule to perform second feature extraction, thereby obtaining the second feature vector; The second feature vector is input into the classification submodule for classification to obtain the predicted human action category.

4. The method according to claim 3, characterized in that, The first feature extraction submodule includes multiple convolutional submodules and a first fully connected layer connected in sequence. The convolutional submodule includes a 3D convolutional kernel and a 3D max pooling layer connected in sequence. The second feature extraction submodule includes a gated loop unit; The classification submodule includes a second fully connected layer and a softmax function layer connected in sequence.

5. The method according to claim 1, characterized in that, The step of iteratively training the human action recognition model based on the predicted human action category and the actual human action category until a preset condition is met, to obtain the trained human action recognition model, includes: The total loss between the predicted human action category and the actual human action category is obtained based on the center crossover loss function. The parameters of the human action recognition model are iteratively adjusted using the Adam algorithm until the preset conditions are met, thus obtaining the trained human action recognition model.

6. The method according to claim 5, characterized in that, The center-cross loss function includes a classification loss function and a center loss function. The step of obtaining the total loss between the predicted human action category and the actual human action category based on the center-cross loss function includes: Based on the classification loss function, the classification loss is obtained; Based on the aforementioned central loss function, the central loss is obtained; The total loss is obtained based on the classification loss and the center loss.

7. A method for human motion recognition, characterized in that, include: Obtain the video data to be identified; The video data to be identified is input into the human motion recognition model to obtain the human motion category corresponding to the video data to be identified, wherein the human motion recognition model is trained using the training method described in any one of claims 1-6.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 7.

Images