Action recognition model training method and apparatus, action recognition method and apparatus

Abstract

This disclosure provides a method and apparatus for training an action recognition model, relating to the fields of computer vision and deep learning. The specific implementation scheme includes: acquiring a pre-set sample set; acquiring a pre-established action recognition network, which includes a convolutional neural network, a recurrent neural network, and a fully connected layer connected to the convolutional neural network for classifying image attributes; performing the following training steps: calculating attribute loss values ​​based on attribute-labeled samples selected from the sample set, using a convolutional neural network and a fully connected layer; calculating action loss values ​​based on action-labeled samples selected from the sample set, using a convolutional neural network and a recurrent neural network; obtaining the action recognition network loss value from the attribute loss value and the action loss value; and using the action recognition network as the action recognition model when the training completion condition is met. This implementation improves the accuracy of action recognition.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular, to the technical field of computer vision, deep learning and the like, and more particularly to a motion recognition model training method and device, a motion recognition method and device, an electronic device, a computer readable medium and a computer program product. BACKGROUND

[0002] In recent years, in the deep learning (DL) method of motion recognition, a relatively common practice is to use a convolutional neural network (CNN) to extract features of image frames in a video, and then input the extracted features into a recurrent neural network (RNN) to extract the time sequence relationship in the multiple continuous image frame features. However, this structure is relatively cumbersome, and it is difficult for the convolutional neural network and the recurrent neural network to form a unified whole, so it is difficult to form a good feature extraction ability for the target. SUMMARY

[0003] A motion recognition model training method and device, a motion recognition method and device, an electronic device, a computer readable medium and a computer program product are provided.

[0004] According to a first aspect, a motion recognition model training method is provided, which comprises: obtaining a preset sample set, wherein the sample set comprises attribute-labeled samples and action-labeled samples; obtaining a pre-established motion recognition network, the motion recognition network comprising a convolutional neural network, a recurrent neural network, and a fully connected layer connected to the convolutional neural network and used for classifying image attributes; performing the following training steps: based on the attribute-labeled samples selected from the sample set, using the convolutional neural network and the fully connected layer, calculating an attribute loss value; based on the action-labeled samples selected from the sample set, using the convolutional neural network and the recurrent neural network, calculating an action loss value; obtaining a motion recognition network loss value from the attribute loss value and the action loss value; and in response to the motion recognition network meeting a training completion condition, taking the motion recognition network as a motion recognition model.

[0005] According to a second aspect, a motion recognition method is provided, which comprises: obtaining a video sequence to be subjected to motion recognition; inputting the video sequence into a motion recognition model generated by the method described in any of the implementations of the first aspect, and outputting an action classification result of the video sequence.

[0006] According to a third aspect, an action recognition model training apparatus is provided, the apparatus comprising: a sample obtaining unit configured to obtain a preset sample set, wherein the sample set comprises attribute-labeled samples and action-labeled samples; a network obtaining unit configured to obtain a pre-established action recognition network, the action recognition network comprising a convolutional neural network, a recurrent neural network, and a fully connected layer connected to the convolutional neural network and configured to classify image attributes; an attribute calculating unit configured to calculate an attribute loss value based on an attribute-labeled sample selected from the sample set, using the convolutional neural network and the fully connected layer; an action calculating unit configured to calculate an action loss value based on an action-labeled sample selected from the sample set, using the convolutional neural network and the recurrent neural network; an obtaining unit configured to obtain an action recognition network loss value from the attribute loss value and the action loss value; and an output unit configured to output the action recognition network as an action recognition model in response to the action recognition network satisfying a training completion condition.

[0007] According to a fourth aspect, an action recognition apparatus is provided, the apparatus comprising: an obtaining unit configured to obtain a video sequence to be subjected to action recognition; and a classification unit configured to input the video sequence into an action recognition model generated by the method described in any implementation form of the first aspect, and output an action classification result of the video sequence.

[0008] According to a fifth aspect, an electronic device is provided, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any implementation form of the first aspect or the second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being configured to cause a computer to perform the method described in any implementation form of the first aspect or the second aspect.

[0010] According to a seventh aspect, a computer program product is provided, comprising a computer program configured to implement the method described in any implementation form of the first aspect or the second aspect when executed by a processor.

[0011] The action recognition model training method and device provided by the embodiments of the present disclosure first acquire a preset sample set, wherein the sample set includes attribute-labeled samples and action-labeled samples; secondly, a pre-established action recognition network is acquired, the action recognition network includes a convolutional neural network, a recurrent neural network, and a fully connected layer connected with the convolutional neural network and used for classifying image attributes; thirdly, based on the attribute-labeled samples selected from the sample set, the convolutional neural network and the fully connected layer are used to calculate an attribute loss value; fourthly, based on the action-labeled samples selected from the sample set, the convolutional neural network and the recurrent neural network are used to calculate an action loss value; fifthly, the action recognition network loss value is obtained from the attribute loss value and the action loss value; finally, in response to the action recognition network satisfying a training completion condition, the action recognition network is taken as an action recognition model. Thus, when the action recognition task is trained, the sample attribute classification training of the convolutional neural network is simultaneously performed, the feature extraction capability of the action recognition network is improved, and the accuracy of action recognition is improved.

[0012] The action recognition method and device provided by the embodiments of the present disclosure acquire a video sequence to be subjected to action recognition; the video sequence to be subjected to action recognition is input into an action recognition model generated by the action recognition model training method of the embodiments, and an action classification result of the video sequence to be subjected to action recognition output by the action recognition model is obtained. Thus, the processing of the video sequence to be subjected to action recognition by the action recognition model with increased attribute classification can improve the accuracy of action recognition and ensure the continuity of the action recognition result.

[0013] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them:

[0015] Figure 1 is a flowchart of one embodiment of the action recognition model training method according to the present disclosure;

[0016] Figure 2 is a structural schematic diagram of one embodiment of the action recognition network training in the embodiments of the present disclosure;

[0017] Figure 3 is a flowchart of one embodiment of the action recognition method according to the present disclosure;

[0018] Figure 4 is a structural schematic diagram of one embodiment of the action recognition model training device according to the present disclosure;

[0019] Figure 5 is a structural schematic diagram of an action recognition device according to one embodiment of the present disclosure;

[0020] Figure 6 is a block diagram of an electronic device for implementing an action recognition model training method or an action recognition method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which various details of embodiments of the present disclosure are set forth to assist in the understanding of the present disclosure. It will be apparent to those skilled in the art that various changes and modifications can be made thereto without departing from the scope and spirit of the present disclosure. Also, the description is made in the order of the following description for the sake of clarity and conciseness.

[0022] Most of the existing action recognition models composed of convolutional neural networks and recurrent neural networks are in an end-to-end training mode, i.e., the action frame image sequence is input to the convolutional neural network, and the action class of the object in the action frame image sequence output by the recurrent neural network is obtained. However, the feature extraction capability of the convolutional neural network part is not improved, and the convolutional neural network and the recurrent neural network cannot be formed into a unified whole.

[0023] Figure 1 Flow 100 according to one embodiment of an action recognition model training method of the present disclosure is shown, and the action recognition model training method includes the following steps:

[0024] In step 101, a preset sample set is obtained.

[0025] In this embodiment, the execution subject on which the action recognition model training method runs can obtain the sample set in various ways. For example, the execution subject can obtain the sample set stored in the database server through wired connection or wireless connection. For another example, the user can collect samples through a terminal, so that the execution subject can receive the samples collected by the terminal and store the samples locally to generate a sample set.

[0026] Here, the sample set can include at least one sample. The sample can include attribute-labeled samples and / or action-labeled samples. Attribute-labeled samples and action-labeled samples are two different types of samples. Attribute-labeled samples are obtained by labeling the attributes of the objects to be identified in the video dataset. Each object to be identified in the attribute-labeled sample has an attribute label. Attributes refer to the external features of the object to be identified, and specific attributes can include features such as gender, identity information, clothing, and actions. These attributes are semantic information other than the target action. By using attribute-labeled samples to train the convolutional neural network, its feature extraction capability can be improved.

[0027] In this embodiment, the process of annotating attribute-labeled samples to obtain attribute tags is as follows:

[0028] 1) Select attribute set A = {a1, a2, ..., a...} n}. Where a i (i = 1, 2, ..., n) represents a certain attribute, such as the target's identity, clothing, gender, etc.

[0029] 2) Based on the selected attribute set, extract image frames from the video data, thereby completing the sample set partitioning D = {D1, D2, ..., D}. n}, where D i (i = 1, 2, ..., n) corresponds to a i Attribute-annotated samples.

[0030] In this embodiment, the action annotation sample is a sample in the video dataset that labels the target actions of the object to be identified. Each object to be identified in the action annotation sample has an action label. Specifically, all image frames in the video dataset in which the object to be identified performs the same action (i.e., all image frames from the first frame to the last frame) are selected, and the obtained image frames are sorted according to the different actions. The resulting image frame sequence is the action annotation sample. In this embodiment, an image frame refers to a single image frame.

[0031] Step 102: Obtain the pre-established action recognition network.

[0032] The action recognition network includes convolutional neural networks, recurrent neural networks, and fully connected layers connected to the convolutional neural networks for classifying image attributes.

[0033] In this embodiment, the action recognition network includes a convolutional neural network (CNN) and a recurrent neural network (RNN). The output of the CNN is connected to the input of the RNN, and both the CNN and the RNN are neural networks. The CNN consists of neurons with learnable weights and constant biases. Each neuron receives some inputs, performs a dot product calculation, and outputs a score for each category.

[0034] Recurrent neural networks (RNNs) are a type of neural network used to process sequential data. Unlike other neural networks, RNNs are better at processing sequential information, that is, identifying relationships between successive inputs. For example, in Natural Language Processing (NLP), understanding a sentence in isolation is insufficient; it requires processing the entire sequence of words connected together.

[0035] In this embodiment, a fully connected layer is introduced into the action recognition network. This fully connected layer is connected to the output of the convolutional neural network and is used to perform attribute classification on the feature image output by the convolutional neural network. The fully connected layer F may include multiple fully connected sub-layers F. i (i = 1, 2, ..., n), each fully connected sublayer corresponds to the classification of an attribute-labeled sample, F = {F1, F2, ..., Fn} n The input to the fully connected layer F is the feature vector of the image in the input dataset D extracted by the convolutional neural network. According to D... i Data and attribute labels for F i It is trained with a convolutional neural network.

[0036] Step 103: Based on the attribute-labeled samples selected from the sample set, use a convolutional neural network and a fully connected layer to calculate the attribute loss value.

[0037] In this embodiment, the convolutional neural network and fully connected layers are used as an attribute classification network. The attribute classification network is trained using attribute-labeled samples selected from the sample set. The attribute loss value is calculated by the loss function of the attribute classification network. The attribute loss value can be reduced by adjusting the parameters of the attribute classification network.

[0038] like Figure 2 As shown, after selecting attribute-labeled samples from the sample set, the attribute-labeled samples are processed by a CNN to generate feature vectors. These feature vectors are then input into a fully connected layer for image attribute classification to obtain the attribute classification results. Based on these attribute classification results and the attribute labels on the attribute-labeled samples, the attribute loss value can be calculated.

[0039] Step 104: Based on the action-annotated samples selected from the sample set, use convolutional neural networks and recurrent neural networks to calculate the action loss value.

[0040] In this embodiment, the convolutional neural network and the recurrent neural network are taken as an action classification network, the action classification network is trained through the action labeled samples selected from the sample set, the action loss value is calculated by the loss function of the action classification network, and the action loss value can be made smaller by adjusting the parameters of the action classification network.

[0041] As shown in Figure 2 After the action labeled samples in the sample set are selected, the action labeled samples pass through the CNN to generate a feature vector, the feature vector is input into the RNN for image action classification to obtain an action classification result, and the action loss value can be calculated based on the action classification result and the action label on the action labeled sample.

[0042] In this embodiment, each iteration training of the action recognition network can be divided into two different stages which are performed simultaneously, that is, in the current iteration process, the first stage is the training stage of the attribute classification network to obtain the attribute loss value, and the second stage is the training stage of the action classification network to obtain the action loss value, and the action recognition network loss value is obtained based on the attribute loss value and the action loss value after the first stage and the second stage are completed.

[0043] In step 105, the action recognition network loss value is obtained from the attribute loss value and the action loss value.

[0044] In this embodiment, the action recognition network loss value is the loss value of the entire action recognition network, and the action recognition network loss value is equal to the sum of the attribute loss value and the action loss value, as shown in Figure 2 After the action recognition network loss value is obtained, the parameters of the CNN or the RNN or the fully connected layer can be adjusted by gradient back propagation to make the action recognition network loss value converge, and finally reach the minimum convergence value of the action recognition network loss value.

[0045] Alternatively, the action recognition network loss value can also be equal to the sum of a first value and a second value, wherein the first value is equal to the product of a first weight value and the attribute loss value, and the second value is equal to the product of a second weight value and the action loss value. The first weight value and the second weight value can be set according to requirements, for example, the first weight value is greater than the second weight value.

[0046] In step 106, in response to the action recognition network meeting the training completion condition, the action recognition network is taken as an action recognition model.

[0047] In this embodiment, the training completion condition includes at least one of the following: the number of training iterations of the action recognition network reaches a predetermined iteration threshold, and the action recognition network loss value is less than a predetermined loss value threshold. For example, the number of training iterations of the action recognition network reaches 5,000 times. The action recognition network loss value is less than 0.05.

[0048] In some optional implementations of the embodiment, in response to the action recognition network not satisfying the training completion condition, the related parameters in the action recognition network are adjusted to make the action recognition network loss value converge, and based on the adjusted action recognition network, the training steps 103-105 are continuously executed.

[0049] In the embodiment, if the training is not completed, the parameters of the attribute classification network or the action classification network are adjusted to make the action recognition network loss value converge. Specifically, the parameters of the convolutional neural network can be kept unchanged first, and the steps 103-105 are repeatedly executed to adjust the parameters of the recurrent neural network, so that the action recognition network loss value gradually decreases until convergence. Specifically, the parameters of the recurrent neural network can also be kept unchanged first, and the steps 103-105 are repeatedly executed to adjust the parameters of the convolutional neural network, so that the action recognition network loss value gradually decreases until convergence, thereby ensuring the convergence effect of the action recognition network.

[0050] The action recognition model training method provided by the embodiment of the disclosure first acquires a preset sample set, wherein the sample set includes attribute-labeled samples and action-labeled samples; secondly, a pre-established action recognition network is acquired, the action recognition network includes a convolutional neural network, a recurrent neural network, and a fully connected layer connected with the convolutional neural network and used for classifying image attributes; thirdly, based on the attribute-labeled samples selected from the sample set, the convolutional neural network and the fully connected layer are used to calculate an attribute loss value; fourthly, based on the action-labeled samples selected from the sample set, the convolutional neural network and the recurrent neural network are used to calculate an action loss value; fifthly, the action recognition network loss value is obtained from the attribute loss value and the action loss value; and finally, in response to the action recognition network satisfying a training completion condition, the action recognition network is taken as an action recognition model. Thus, when the action recognition task is trained, the sample attribute classification training of the convolutional neural network is simultaneously performed, the feature extraction capability of the action recognition network is improved, and the accuracy of the action recognition is improved.

[0051] In some optional implementations of the embodiment, based on the attribute-labeled samples selected from the sample set, the convolutional neural network and the fully connected layer are used to calculate the attribute loss value, including: inputting the attribute-labeled samples into the convolutional neural network to obtain a feature vector output by the convolutional neural network; inputting the feature vector into the fully connected layer to obtain an attribute classification result of the attribute-labeled samples output by the fully connected layer; and based on the attribute classification result, calculating the attribute loss value.

[0052] In this embodiment, different structures of convolutional neural networks can be used to process the attribute annotation samples to obtain the feature vectors. For example, the convolutional neural network includes an input layer, a convolutional layer, and a pooling layer. For another example, the convolutional neural network includes an input layer, a first convolutional layer, a second convolutional layer, and a pooling layer.

[0053] In this optional implementation, based on the convolutional neural network and the fully connected layer, the attribute classification result of the attribute annotation sample output by the fully connected layer is obtained, and based on the attribute classification result, the attribute loss value is calculated, thereby ensuring the accuracy of the attribute loss value.

[0054] Optionally, before the attribute annotation sample is input into the convolutional neural network, the attribute annotation sample with the attribute label in the sample set can also be determined through the pre-set attribute label, thereby ensuring that the attribute annotation sample input into the convolutional neural network is a real attribute annotation sample.

[0055] In some optional implementations of this embodiment, the above-mentioned convolutional neural network includes an input layer, a convolutional layer, and a global pooling layer; the attribute annotation sample is input into the convolutional neural network to obtain the feature vector output by the convolutional neural network, including: the attribute annotation sample is input into the input layer for image preprocessing to obtain a preprocessed image; the preprocessed image is input into the convolutional layer for convolution operation to obtain a feature map; and the feature map is input into the global pooling layer for global average pooling to obtain the feature vector.

[0056] In this embodiment, the image preprocessing can include mean removal, normalization, and dimension reduction processing, wherein the mean removal refers to centering all dimensions of the input data to 0, summing and averaging all samples, and then subtracting the mean sample from all samples to remove the mean. The normalization refers to normalizing the data amplitude to the same range, for example, normalizing each feature to [-1, 1]. The dimension reduction processing refers to performing decorrelation processing on the attribute annotation sample to cancel the correlation of each dimension, so that the features are independent of each other.

[0057] The pooling layer is used for feature dimension reduction, compression of data and parameter quantity, and reduction of overfitting. In this embodiment, the global pooling layer is used to define a spatial neighborhood and calculate the average value from the modified feature map in the window.

[0058] In this optional implementation, when the convolutional neural network includes an input layer, a convolutional layer, and a global pooling layer, the global pooling layer is used to perform global average pooling on the feature map output by the convolutional layer, thereby achieving the effect of average pooling on the feature map, providing an optional implementation for obtaining the feature vector, and ensuring the reliability of the feature vector.

[0059] In some optional implementations of the embodiment, the convolutional neural network comprises an input layer, a convolutional layer, and a region pooling layer; the attribute-labeled sample is input into the convolutional neural network to obtain a feature vector output by the convolutional neural network, comprising: inputting the attribute-labeled sample into the input layer for image preprocessing to obtain a preprocessed image; inputting the preprocessed image into the convolutional layer for convolution operation to obtain a feature map; dividing the feature map into a plurality of different regions; inputting the feature map in each region into the region pooling layer for region average pooling to obtain the feature vector.

[0060] To solve the problem that the traditional global average pooling discards the position information of the elements in the original feature map, the region average pooling provided in the embodiment divides the original feature map into regions, and performs average pooling processing on the elements in each region, and the calculation method is as follows:

[0061]

[0062] In formula (1), is the value of the region-pooled map corresponding to the (i,j) position of the kth channel of the CNN output feature map; s m , s n is the step length of average pooling in the x,y dimension; M k is the kth channel of the feature map. As can be seen from formula (1), the region average pooling performed by the region pooling layer can partially restore the position information in the feature map, and finally the generated region average pooling map is unfolded into a feature vector, which is used for attribute classification of the CNN post-connection layer and input of the RNN.

[0063] In this optional implementation, after obtaining the feature map output by the convolutional layer, the feature map is regionally divided, and each region is regionally averaged and pooled, and finally a feature vector is obtained. The original global average pooling operation is improved to region average pooling, so that the discarded position information can be partially retained, and the information contained in the feature vector is more abundant.

[0064] In some optional implementations of the embodiment, the above action-labeled sample is a multi-frame labeled image, and based on the action-labeled sample selected from the sample set, a convolutional neural network and a recurrent neural network are used to calculate an action loss value, comprising: sequentially inputting each image in the multi-frame labeled image into the convolutional neural network to obtain a feature vector corresponding to each image in the multi-frame labeled image; packing the feature vector corresponding to each image in the multi-frame labeled image, and inputting it into the recurrent neural network to obtain an action classification result of the multi-frame labeled image output by the recurrent neural network; and calculating the action loss value based on the action classification result.

[0065] In this optional implementation, the multiple-frame labeled images are all images between a first image and a last image labeled for an action of the object to be recognized. The multiple-frame labeled images can be images labeled for one action (for example, shaking shoulders or lifting legs). The multiple-frame labeled images can also be images labeled for multiple continuous actions (for example, holding a cup and drinking water).

[0066] In this optional implementation, after all image frames in the multiple-frame labeled images are input into the convolutional neural network to obtain feature vectors of all image frames, the feature vectors of all image frames are packed and input into the recurrent neural network, so as to ensure the reliability of the recurrent neural network in recognizing the action labeled sample.

[0067] Optionally, the action labeled sample is a multiple-frame labeled image. Based on the action labeled sample selected from the sample set, the convolutional neural network and the recurrent neural network are used to calculate an action loss value, including: sequentially inputting each image in the multiple-frame labeled images into the convolutional neural network to obtain a feature vector corresponding to each image in the multiple-frame labeled images; removing feature vectors with the same value in the feature vectors, and packing the feature vectors corresponding to the remaining images in the multiple-frame labeled images to input into the recurrent neural network to obtain an action classification result of the multiple-frame labeled images output by the recurrent neural network; and calculating the action loss value based on the action classification result.

[0068] Further, based on the action recognition model training method provided in the above embodiments, an embodiment of an action recognition method is further provided in the disclosure. The action recognition method provided in the embodiment combines computer vision, deep learning, and other artificial intelligence fields.

[0069] Referring to Figure 3 , a flow 300 of one embodiment of the action recognition method according to the disclosure is shown. The action recognition method provided in the embodiment includes the following steps:

[0070] In step 301, a video sequence to be subjected to action recognition is obtained.

[0071] In the embodiment, the video sequence is a plurality of video images with spatio-temporal association arranged in a certain order. The plurality of video images can or can not include an object to be recognized. In the embodiment, the object to be recognized is an object to be subjected to action recognition. When the plurality of video images includes the object to be recognized, the type of action of the object to be recognized can be recognized by the action recognition model, and multiple continuous actions in the plurality of images can also be recognized.

[0072] In the embodiment, the obtained plurality of video images can be color images and / or grayscale images, and the format of the obtained plurality of video images is not limited in the disclosure.

[0073] The execution subject of the action recognition method can acquire the video sequence to be subjected to action recognition in multiple manners. For example, the execution subject can acquire the video sequence to be subjected to action recognition stored in the database server through wired connection or wireless connection. For another example, the execution subject can also receive the video sequence to be subjected to action recognition collected by a terminal or other device.

[0074] In step 302, the video sequence is input into the action recognition model, and an action classification result of the video sequence is output.

[0075] In this embodiment, the execution subject can input the video sequence acquired in step 301 into the action recognition model, so as to obtain the action classification result of the object to be recognized in the video sequence.

[0076] In this embodiment, the action recognition model can be obtained by training the method described in the above Figure 1 Figure 1 In this embodiment, the specific training process can be referred to the related description of the above

[0077] In this embodiment, the action recognition model can be used to recognize the action of the object to be recognized in the video sequence and classify the action. The action recognition model can be used to recognize different actions of different persons in the video sequence and classify the recognized different actions, so as to obtain the classification result of different actions. For example, a series of video sequences of persons playing ball are input into the action recognition model, and the action recognition model outputs all playing ball actions of each person and the types of each playing ball action.

[0078] It should be noted that the action recognition method of this embodiment can be used to test the action recognition model generated by the above embodiments. Furthermore, the action recognition model can be continuously optimized according to the action classification result. The method can also be the actual application method of the action recognition model generated by the above embodiments. Using the action recognition model generated by the above embodiments to recognize the action of the object in the video sequence can help to improve the accuracy of action recognition.

[0079] The action recognition method provided in this embodiment acquires the video sequence to be subjected to action recognition; inputs the video sequence to be subjected to action recognition into the action recognition model generated by the action recognition model training method of this embodiment, and obtains the action classification result of the video sequence to be subjected to action recognition output by the action recognition model. Therefore, the processing of the video sequence to be subjected to action recognition by the action recognition model with increased attribute classification can improve the accuracy of action recognition and ensure the continuity of the action recognition result.

[0080] Further reference is made to Figure 4 ​As an implementation of the method shown in the above figures, the disclosure provides an embodiment of an action recognition model training device, which corresponds to the method embodiment shown in Figure 1 and can be applied to various electronic devices.

[0081] As shown in Figure 4 The action recognition model training device 400 provided in this embodiment includes a sample acquisition unit 401, a network acquisition unit 402, an attribute calculation unit 403, an action calculation unit 404, a obtaining unit 405, and an output unit 406. The sample acquisition unit 401 can be configured to acquire a preset sample set, wherein the sample set includes attribute-labeled samples and action-labeled samples. The network acquisition unit 402 can be configured to acquire a pre-established action recognition network, which includes a convolutional neural network, a recurrent neural network, and a fully connected layer connected to the convolutional neural network and used for classifying image attributes. The attribute calculation unit 403 can be configured to calculate an attribute loss value based on the attribute-labeled samples selected from the sample set, using the convolutional neural network and the fully connected layer. The action calculation unit 404 can be configured to calculate an action loss value based on the action-labeled samples selected from the sample set, using the convolutional neural network and the recurrent neural network. The obtaining unit 405 can be configured to obtain an action recognition network loss value from the attribute loss value and the action loss value. The output unit 406 can be configured to output the action recognition network as an action recognition model in response to the action recognition network meeting a training completion condition.

[0082] In this embodiment, the specific processing of the sample acquisition unit 401, the network acquisition unit 402, the attribute calculation unit 403, the action calculation unit 404, the obtaining unit 405, and the output unit 406 in the action recognition model training device 400 and the technical effects brought by the specific processing can be respectively referred to the related descriptions of the steps 101, 102, 103, 104, 105, and 106 in the corresponding embodiment. Figure 1 The related descriptions of the steps 101, 102, 103, 104, 105, and 106 in the corresponding embodiment, which will not be repeated here.

[0083] In some optional implementations of this embodiment, the device 400 further includes an adjustment unit (not shown in the figure). The adjustment unit can be configured to adjust the related parameters in the action recognition network to make the action recognition network loss value converge in response to the action recognition network not meeting the training completion condition. The attribute calculation unit 403, the action calculation unit 404, the obtaining unit 405, and the output unit 406 continue to perform the training steps based on the adjusted action recognition network.

[0084] In some optional implementations of the present embodiment, the attribute calculation unit 403 includes a vector obtaining module (not shown in the figure), a result obtaining module (not shown in the figure), and an attribute calculation module (not shown in the figure). The vector obtaining module can be configured to input the attribute annotation sample into the convolutional neural network to obtain a feature vector output by the convolutional neural network. The result obtaining module can be configured to input the feature vector into the fully connected layer to obtain an attribute classification result of the attribute annotation sample output by the fully connected layer. The attribute calculation module can be configured to calculate the attribute loss value based on the attribute classification result.

[0085] In some optional implementations of the present embodiment, the convolutional neural network includes an input layer, a convolutional layer, and a global pooling layer; and the vector obtaining module includes a processing submodule (not shown in the figure), an operation submodule (not shown in the figure), and a global pooling submodule (not shown in the figure). The processing submodule can be configured to input the attribute annotation sample into the input layer to perform image preprocessing to obtain a preprocessed image. The operation submodule can be configured to input the preprocessed image into the convolutional layer to perform convolution operation to obtain a feature map. The global pooling submodule can be configured to input the feature map into the global pooling layer to perform global average pooling to obtain the feature vector.

[0086] In some optional implementations of the present embodiment, the convolutional neural network includes an input layer, a convolutional layer, and a region pooling layer; and the vector obtaining module includes a processing submodule (not shown in the figure), an operation submodule (not shown in the figure), a division submodule (not shown in the figure), and a region pooling submodule (not shown in the figure). The processing submodule can be configured to input the attribute annotation sample into the input layer to perform image preprocessing to obtain a preprocessed image. The operation submodule can be configured to input the preprocessed image into the convolutional layer to perform convolution operation to obtain a feature map. The division submodule can be configured to divide the feature map into a plurality of different regions. The region pooling submodule can be configured to input the feature map in each region into the region pooling layer to perform regional average pooling to obtain the feature vector.

[0087] In some optional implementations of this embodiment, the action annotation samples are multi-frame annotated images, and the action calculation unit 404 includes: a feature acquisition module (not shown in the figure), a classification acquisition module (not shown in the figure), and an action calculation module (not shown in the figure). The feature acquisition module can be configured to sequentially input each frame of the multi-frame annotated images into a convolutional neural network to obtain a feature vector corresponding to each frame. The classification acquisition module can be configured to package the feature vectors corresponding to each frame of the multi-frame annotated images and input them into a recurrent neural network to obtain the action classification result of the multi-frame annotated images output by the recurrent neural network. The action calculation module can be configured to calculate an action loss value based on the action classification result.

[0088] The action recognition model training apparatus provided in the embodiments of this disclosure firstly involves a sample acquisition unit 401 acquiring a preset sample set, which includes attribute-labeled samples and action-labeled samples. Secondly, a network acquisition unit 402 acquires a pre-established action recognition network, which includes a convolutional neural network, a recurrent neural network, and a fully connected layer connected to the convolutional neural network for classifying image attributes. Thirdly, an attribute calculation unit 403 calculates attribute loss values ​​based on attribute-labeled samples selected from the sample set, using a convolutional neural network and a fully connected layer. Fourthly, an action calculation unit 404 calculates action loss values ​​based on action-labeled samples selected from the sample set, using a convolutional neural network and a recurrent neural network. Fifthly, an output unit 405 obtains the action recognition network loss value from the attribute loss value and the action loss value. Finally, an output unit 406, in response to the action recognition network meeting the training completion conditions, uses the action recognition network as an action recognition model. Thus, during the training of the action recognition task, sample attribute classification training of the convolutional neural network is performed simultaneously, improving the feature extraction capability of the action recognition network and increasing the accuracy of action recognition.

[0089] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an action recognition device, which is similar to... Figure 3 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0090] like Figure 5 As shown, the action recognition device 500 provided in this embodiment includes: an acquisition unit 501 and a classification unit 502. The acquisition unit 501 can be configured to acquire a video sequence to be identified. The classification unit 502 can be configured to input the video sequence as described above. Figure 1 The action recognition model generated by the method described in the embodiment outputs the action classification results of the video sequence.

[0091] In this embodiment, the specific processing of the acquisition unit 501 and the classification unit 502 in the action recognition device 500, and the resulting technical effects, can be found in the following references: Figure 3 The relevant descriptions of steps 301 and 302 in the corresponding embodiments will not be repeated here.

[0092] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0093] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0094] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0095] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0096] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0097] The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the action recognition model training method or the action recognition method. For example, in some embodiments, the action recognition model training method or the action recognition method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the action recognition model training method or the action recognition method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the action recognition model training method or the action recognition method by any other suitable means, such as by means of firmware.

[0098] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0099] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a general purpose computer, special purpose computer, or other programmable computing apparatuses, processors or controllers to produce a machine, such that the program code, when executed by the processor or controller, implements the functions / operations specified in the flowcharts and / or block diagrams. The program code can execute entirely on a machine, partly on a machine, as a stand-alone software package, partly on a machine and partly on a remote machine or entirely on a remote machine or server.

[0100] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0101] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0102] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0103] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0104] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the steps. For example, the steps described in the present disclosure can be executed in parallel, in series, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

[0105] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training an action recognition model, the method comprising: obtaining a preset sample set, wherein the sample set comprises attribute-labeled samples and action-labeled samples; obtaining a pre-established action recognition network, the action recognition network comprising a convolutional neural network, a recurrent neural network, and a fully connected layer connected to the convolutional neural network and configured to classify image attributes; performing the following training steps: calculating an attribute loss value based on an attribute-labeled sample selected from the sample set, using the convolutional neural network and the fully connected layer; calculating an action loss value based on an action-labeled sample selected from the sample set, using the convolutional neural network and the recurrent neural network; obtaining an action recognition network loss value from the attribute loss value and the action loss value; and in response to the action recognition network satisfying a training completion condition, regarding the action recognition network as an action recognition model; wherein the attribute loss value is calculated based on the attribute-labeled sample selected from the sample set, using the convolutional neural network and the fully connected layer, comprises: determining attribute-labeled samples in the sample set that have a preset attribute label; inputting the attribute-labeled samples into the convolutional neural network to obtain a feature vector output by the convolutional neural network; inputting the feature vector into the fully connected layer to obtain an attribute classification result of the attribute-labeled samples output by the fully connected layer; and calculating an attribute loss value based on the attribute classification result; wherein the convolutional neural network comprises an input layer, a convolutional layer, and a region pooling layer; and inputting the attribute-labeled samples into the convolutional neural network to obtain the feature vector output by the convolutional neural network comprises: inputting the attribute-labeled samples into the input layer to perform image preprocessing to obtain a preprocessed image; inputting the preprocessed image into the convolutional layer to perform convolutional operation to obtain a feature map; dividing the feature map into a plurality of different regions, and inputting the feature map in each region into the region pooling layer to perform region average pooling to obtain a feature vector. 2.The method of claim 1, further comprising: in response to the action recognition network not satisfying the training completion condition, adjusting related parameters in the action recognition network to make the action recognition network loss value converge, and continuing to perform the above training steps based on the adjusted action recognition network. The convolutional neural network comprises an input layer, a convolutional layer, and a global pooling layer; and inputting the attribute-labeled samples into the convolutional neural network to obtain the feature vector output by the convolutional neural network comprises: inputting the attribute-labeled samples into the input layer to perform image preprocessing to obtain a preprocessed image; inputting the preprocessed image into the convolutional layer to perform convolutional operation to obtain a feature map; and inputting the feature map into the global pooling layer to perform global average pooling to obtain a feature vector. The action-labeled sample is a multi-frame labeled image; and calculating the action loss value based on the action-labeled sample selected from the sample set, using the convolutional neural network and the recurrent neural network, comprises: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 3. The method of claim 1, wherein, ​ ​ ​ ​ 4. The method according to one of claims 1 to 3, wherein ​ inputting each of the multiple frames of labeled images into the convolutional neural network in sequence to obtain a feature vector corresponding to each of the multiple frames of labeled images; packing the feature vectors corresponding to each of the multiple frames of labeled images and inputting the packed feature vectors into the recurrent neural network to obtain an action classification result of the multiple frames of labeled images output by the recurrent neural network; calculating an action loss value based on the action classification result.

5. An action recognition method, the method comprising: obtaining a video sequence to be subjected to action recognition; inputting the video sequence into an action recognition model generated by the method of any one of claims 1-4 to output an action classification result of the video sequence.

6. An action recognition model training apparatus, the apparatus comprising: a sample obtaining unit configured to obtain a preset sample set, wherein the sample set comprises attribute-labeled samples and action-labeled samples; a network obtaining unit configured to obtain a pre-established action recognition network, the action recognition network comprising a convolutional neural network, a recurrent neural network, and a fully connected layer connected to the convolutional neural network and configured to classify image attributes; an attribute calculating unit configured to calculate an attribute loss value based on attribute-labeled samples selected from the sample set, using the convolutional neural network and the fully connected layer; an action calculating unit configured to calculate an action loss value based on action-labeled samples selected from the sample set, using the convolutional neural network and the recurrent neural network; an obtaining unit configured to obtain an action recognition network loss value from the attribute loss value and the action loss value; an output unit configured to output the action recognition network as an action recognition model in response to the action recognition network satisfying a training completion condition; wherein the attribute calculating unit comprises: a sample obtaining module configured to determine attribute-labeled samples having a preset attribute label from the sample set based on the attribute label; a vector obtaining module configured to input the attribute-labeled samples into the convolutional neural network to obtain a feature vector output by the convolutional neural network; a result obtaining module configured to input the feature vector into the fully connected layer to obtain an attribute classification result of the attribute-labeled samples output by the fully connected layer; an attribute calculating module configured to calculate an attribute loss value based on the attribute classification result; wherein the convolutional neural network comprises an input layer, a convolutional layer, and a region pooling layer; and the vector obtaining module comprises: a processing submodule configured to input the attribute-labeled samples into the input layer for image preprocessing to obtain preprocessed images; an operation submodule configured to input the preprocessed images into the convolutional layer for convolution operation to obtain feature maps; a region pooling submodule configured to divide the feature maps into multiple different regions, input the feature maps in each region into the region pooling layer for region average pooling to obtain a feature vector.

7. The apparatus of claim 6, further comprising: The adjusting unit is configured to adjust the related parameters in the action recognition network so that the action recognition network loss value converges, in response to the action recognition network not satisfying the training completion condition, and the attribute calculation unit, the action calculation unit, the obtaining unit and the output unit continue to perform the training step based on the adjusted action recognition network.

8. The apparatus of claim 6, wherein, The convolutional neural network comprises an input layer, a convolutional layer and a global pooling layer; and the vector obtaining module comprises: The processing submodule is configured to input the attribute annotation sample into the input layer for image preprocessing to obtain a preprocessed image. The operation submodule is configured to input the preprocessed image into the convolutional layer for convolution operation to obtain a feature map. The global pooling submodule is configured to input the feature map into the global pooling layer for global average pooling to obtain a feature vector.

9. The apparatus of one of claims 6-8, wherein, The action annotation sample is a multi-frame annotation image, and the action calculation unit comprises: The feature obtaining module is configured to input each frame of image in the multi-frame annotation image into the convolutional neural network in sequence to obtain a feature vector corresponding to each frame of image in the multi-frame annotation image. The classification obtaining module is configured to input the feature vector corresponding to each frame of image in the multi-frame annotation image after being packed into the recurrent neural network to obtain an action classification result of the multi-frame annotation image output by the recurrent neural network. The action calculation module is configured to calculate an action loss value based on the action classification result.

10. An action recognition apparatus, the apparatus comprising: an obtaining unit configured to obtain a video sequence to be subjected to action recognition; a classification unit configured to input the video sequence into an action recognition model generated by the method of any one of claims 1-4 to output an action classification result of the video sequence.

11. An electronic device, comprising: comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-5.

13. A computer program product comprising computer program which, when executed by a processor, implements the method of any one of claims 1-5.

13. A computer program product comprising computer program which, when executed by a processor, implements the method of any one of claims 1-5.

Images