Action recognition model training method and device, and action recognition method and device

Abstract

The present disclosure provides a motion recognition model training method and device, and relates to the technical fields of computer vision, deep learning and the like. The specific implementation scheme is as follows: obtaining a sample set, the sample set comprising at least one motion image sample; obtaining a pre-established motion recognition network, the motion recognition network comprising a convolutional neural network and a recurrent neural network connected to the convolutional neural network, elements of a feature vector output by the convolutional neural network to the recurrent neural network comprising: an average pooling value, an activation center position value, the activation center position value being calculated from a value and an element position value of an element in a feature map output by a convolutional layer; selecting a motion image sample in the sample set, and training the motion recognition network using the selected motion image sample; and in response to determining that the motion recognition network meets a training completion condition, taking the motion recognition network as a motion recognition model. The embodiment improves the accuracy of motion recognition.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the fields of computer vision, deep learning, etc., and in particular to an action recognition model training method and apparatus, action recognition method and apparatus, electronic equipment, computer-readable medium, and computer program product. Background Technology

[0002] In recent years, a common approach in deep learning (DL) methods for action recognition is to use convolutional neural networks (CNNs) to extract features from image frames in a video, and then input the extracted features into a recurrent neural network (RNN) to extract the time series relationship between features from multiple consecutive image frames. The method by which convolutional neural networks obtain feature vectors is to perform global averaging on the last layer of the feature map, which will lose the positional information of the elements in the feature map. Summary of the Invention

[0003] A method and apparatus for training an action recognition model, an action recognition method and apparatus, an electronic device, a computer-readable medium, and a computer program product are provided.

[0004] According to a first aspect, a method for training an action recognition model is provided. The method includes: acquiring a sample set, the sample set including at least one action image sample; acquiring a pre-established action recognition network, the action recognition network including a convolutional neural network and a recurrent neural network connected to the convolutional neural network, wherein the elements of the feature vector output by the convolutional neural network to the recurrent neural network include: an average pooling value and an activation center position value, the activation center position value being calculated from the values ​​and position values ​​of elements in the feature map output by the convolutional layer; selecting action image samples from the sample set, and training the action recognition network using the selected action image samples; and, in response to determining that the action recognition network meets the training completion conditions, using the action recognition network as an action recognition model.

[0005] According to the second aspect, an action recognition method is provided, the method comprising: acquiring a video sequence to be recognized for action; inputting the video sequence into an action recognition model generated by the method described in any implementation of the first aspect; and outputting the action classification result of the video sequence.

[0006] According to a third aspect, an action recognition model training apparatus is provided, the apparatus comprising: a sample acquisition unit configured to acquire a sample set, the sample set including at least one action image sample; a network acquisition unit configured to acquire a pre-established action recognition network, the action recognition network including a convolutional neural network and a recurrent neural network connected to the convolutional neural network, wherein the elements of the feature vector output by the convolutional neural network to the recurrent neural network include: an average pooling value and an activation center position value, the activation center position value being calculated from the value and position value of an element in the feature map output by the convolutional layer; a selection unit configured to select action image samples from the sample set; a training unit configured to train the action recognition network using the selected action image samples; and an output unit configured to, in response to determining that the action recognition network meets the training completion conditions, use the action recognition network as an action recognition model.

[0007] According to the fourth aspect, an action recognition device is also provided, the device comprising: an acquisition unit configured to acquire a video sequence to be action recognized; and a classification unit configured to input the video sequence into an action recognition model generated by a method described in any implementation of the first aspect, and output an action classification result of the video sequence.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first or second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first or second aspect.

[0010] According to a seventh aspect, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in either the first or second aspect.

[0011] The action recognition model training method and apparatus provided in the embodiments of this disclosure first acquire a sample set, which includes at least one action image sample; second, acquire a pre-established action recognition network, which includes a convolutional neural network and a recurrent neural network connected to the convolutional neural network, wherein the elements of the feature vector output by the convolutional neural network to the recurrent neural network include: average pooling value and activation center position value; third, select action image samples from the sample set; then, train the action recognition network using the selected action image samples; finally, in response to determining that the action recognition network meets the training completion conditions, use the action recognition network as an action recognition model. Thus, by setting activation center position values ​​in the feature vector, the positional information of elements in the feature map of the convolutional neural network is preserved. Furthermore, explicitly extracting this positional information and inputting it into the recurrent neural network can assist the action recognition network in completing action recognition, improving the accuracy of action recognition, while not affecting the original network structure, thus simplifying implementation.

[0012] The action recognition method and apparatus provided in the embodiments of this disclosure acquire a video sequence to be recognized for action recognition; input the video sequence to be recognized for action recognition into an action recognition model generated by the action recognition model training method of this embodiment, and obtain the action classification result of the video sequence to be recognized for action recognition output by the action recognition model. Therefore, by using an action recognition model with added activation center position values ​​to process the video sequence to be recognized for action recognition, the accuracy of action recognition can be improved, and the continuity of action recognition results can be ensured.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 This is a flowchart of an embodiment of the action recognition model training method according to the present disclosure;

[0016] Figure 2 This is a schematic diagram of a structure for training an action recognition network in an embodiment of this disclosure;

[0017] Figure 3 This is a flowchart of an embodiment of the action recognition method according to the present disclosure;

[0018] Figure 4 This is a schematic diagram of a structure of an embodiment of the action recognition model training device according to the present disclosure;

[0019] Figure 5 This is a schematic diagram of a structure of an embodiment of the motion recognition device according to the present disclosure;

[0020] Figure 6 This is a block diagram of an electronic device used to implement the action recognition model training method or action recognition method of the embodiments of this disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] Most existing action recognition models composed of convolutional neural networks (CNNs) and recurrent neural networks (RNNs) employ end-to-end training. This means that a sequence of action frame images is input into the CNN, and the RNN outputs the action category of the object in the sequence of action frame images. However, this method does not consider the information contained in the feature vectors output by the CNN. The feature vectors input to the RNN are obtained by global average pooling of the feature map from the last layer of the CNN. This method only retains the magnitude of the elements in each channel of the feature map, discarding the original positional information of the elements. For action recognition models, the positional changes of each element in the feature map are crucial, and this is a common feature in time series problems.

[0023] Figure 1 A flowchart 100 is shown as an embodiment of an action recognition model training method according to the present disclosure, the action recognition model training method comprising the following steps:

[0024] Step 101: Obtain the sample set.

[0025] In this embodiment, the execution entity running the action recognition model training method can obtain the sample set in various ways. For example, the execution entity can obtain the sample set stored in the database server through a wired or wireless connection. Alternatively, a user can collect samples through a terminal. In this case, the execution entity can receive the samples collected by the terminal and store them locally, thereby generating a sample set.

[0026] Here, the sample set may include at least one action image sample. The action image sample is a sample in the video dataset that labels the target actions of the object to be identified. Each object to be identified in the action-labeled sample has an action label. Specifically, all image frames in the video dataset where the object to be identified performs the same action (i.e., all image frames from the first frame to the last frame) are selected, and the resulting image frames are sorted according to the different actions. The resulting sequence of image frames is the action-labeled sample. In this embodiment, an image frame refers to a single image frame.

[0027] Step 102: Obtain the pre-established action recognition network.

[0028] The action recognition network includes a convolutional neural network and a recurrent neural network connected to the convolutional neural network. The elements of the feature vector output by the convolutional neural network to the recurrent neural network include: average pooling value and activation center position value. The activation center position value is calculated from the value and position value of the element in the feature map output by the convolutional layer.

[0029] In this embodiment, the action recognition network includes a convolutional neural network (CNN) and a recurrent neural network (RNN). The output of the CNN is connected to the input of the RNN, and both the CNN and the RNN are neural networks. The CNN consists of neurons with learnable weights and constant biases. Each neuron receives some inputs, performs a dot product calculation, and outputs a score for each category.

[0030] Recurrent neural networks (RNNs) are a type of neural network used to process sequential data. Unlike other neural networks, RNNs are better at processing sequential information, that is, identifying relationships between successive inputs. For example, in Natural Language Processing (NLP), understanding a sentence in isolation is insufficient; it requires processing the entire sequence of words connected together.

[0031] In this embodiment, the elements in the feature map are the basic units that make up the feature map. Each element is calculated from the values ​​and positions of the elements in the feature map output by the convolutional layer, based on the activation center position values ​​of the feature map. For example, this calculation method can be to take a weighted sum of the values ​​and positions of each element and then divide it by the sum of all elements.

[0032] When performing classification tasks, convolutional neural networks only need to use the size of each element in each channel of the feature map to indicate the presence of relevant features in the current sample to determine the category. However, in problems such as action recognition, which require information from consecutive frames, the size and position of each element in each channel of the feature map are crucial. Therefore, it is essential to preserve as much of the original information of the feature map as possible to improve the recognition performance of action recognition networks.

[0033] The elements of a feature vector are the basic units that make up the feature vector. The elements of the feature vector output by a traditional convolutional neural network are generally global average pooling values. However, in this embodiment, in addition to the average pooling values, the elements of the feature vector output by the convolutional neural network also include activation center position values. The activation center position values ​​are calculated based on the values ​​and position values ​​of the elements in the feature map output by the convolutional layer of the convolutional neural network. That is, the activation center position values ​​are positional information related to the position of the elements in the feature map. Compared with traditional feature vectors, the feature vector output by the convolutional neural network to the recurrent neural network provided in this embodiment contains the positional information of the elements in the feature map.

[0034] Step 103: Select motion image samples from the sample set.

[0035] In this embodiment, the executing entity can select motion image samples from the sample set obtained in step 101, and perform the training steps 104 to 105. The method and number of samples selected are not limited in this application. For example, all motion image samples including one action of a single object can be randomly selected, or motion image samples including multiple actions of multiple objects can be selected from the sample set.

[0036] Step 104: Train the action recognition network using selected action image samples.

[0037] In this embodiment, the action recognition network is used to detect the actions of objects in an image and the type of those actions. It is a pre-built network. After selecting action image samples, the aforementioned executing entity uses these samples to train the action recognition network. After multiple training, evaluation, and parameter tuning processes, a trained action recognition model is obtained.

[0038] Specifically, such as Figure 2 As shown, the training process of the action recognition model can be as follows: The selected action image samples can be trained based on the action recognition network. Figure 2 The difference between the action detection results of the input sample and the action labels of the selected action image samples determines the error of the action recognition network. The parameters of the action recognition network are iteratively adjusted using gradient backpropagation to gradually reduce the error. When the error of the action recognition network converges to a certain range or the number of iterations reaches a preset threshold, parameter adjustment can be stopped, resulting in a trained action recognition model.

[0039] In this embodiment, as Figure 2As shown, the action recognition network includes CNN and RNN. The action recognition network is iteratively trained using action image samples selected from the sample set. Each iteration of the action recognition network has an output result. The action loss value of the action recognition network is calculated during each iteration of training. Specifically, the action loss value can be calculated by the loss function of the action recognition network. The action loss value can be reduced by adjusting the parameters of the action recognition network.

[0040] Furthermore, in Figure 2 In each iteration of training, after selecting action image samples from the sample set, the action image samples are processed by a CNN to generate feature vectors. These feature vectors are then input into an RNN for image action classification to obtain action classification results. Based on these action classification results and the action labels on the action image samples, the action loss value can be calculated.

[0041] Step 105: In response to the action recognition network meeting the training completion conditions, the action recognition network is used as the action recognition model.

[0042] In this embodiment, the training completion condition includes at least one of the following: the number of training iterations of the action recognition network reaches a predetermined iteration threshold, and the loss value of the action recognition network is less than a predetermined loss value threshold. For example, the action recognition network has been trained for 5,000 iterations. The loss value of the action recognition network is less than 0.05.

[0043] In some optional implementations of this embodiment, in response to the action recognition network not meeting the training completion conditions, the relevant parameters in the action recognition network are adjusted so that the action loss value of the action recognition network converges, and based on the adjusted action recognition network, the above training steps 103-105 are continued.

[0044] In this embodiment, if training is not complete, the parameters of the attribute classification network or the action classification network are adjusted to bring the action loss value of the action recognition network to converge. Specifically, the parameters of the convolutional neural network can be kept unchanged, and steps 103 to 105 can be repeatedly executed to adjust the parameters of the recurrent neural network, so that the action loss value gradually decreases until convergence. Alternatively, the parameters of the recurrent neural network can be kept unchanged, and steps 103 to 105 can be repeatedly executed to adjust the parameters of the convolutional neural network, so that the action loss value gradually decreases until convergence. Optionally, the convergence effect of the training process of the action recognition network can be ensured by simultaneously adjusting the parameters of the convolutional neural network and the recurrent neural network to gradually decrease the action loss value until convergence.

[0045] The action recognition model training method provided in this disclosure first obtains a sample set, which includes at least one action image sample; second, it obtains a pre-established action recognition network, which includes a convolutional neural network (CNN) and a recurrent neural network (RNN) connected to the CNN, wherein the elements of the feature vector output by the CNN to the RNN include: average pooling value and activation center position value; third, it selects action image samples from the sample set; then, it trains the action recognition network using the selected action image samples; finally, in response to determining that the action recognition network meets the training completion conditions, it uses the action recognition network as an action recognition model. Thus, by setting activation center position values ​​in the feature vector, the positional information of elements in the feature map of the CNN is preserved. Furthermore, explicitly extracting this positional information and inputting it into the RNN can assist the action recognition network in completing action recognition, improving the accuracy of action recognition, while not affecting the original network structure, and simplifying implementation.

[0046] In some optional implementations of this embodiment, the activation center position value includes: the average center position value in the width direction and the average center position value in the height direction of the spliced ​​structure; the average center position value in the width direction is equal to the sum of the products of the values ​​of each element in the width direction and the position value of that element divided by the sum of the values ​​of all elements; the average center position value in the height direction is equal to the sum of the products of the values ​​of each element in the height direction and the position value of that element divided by the sum of the values ​​of all elements.

[0047] In this embodiment, the height direction refers to the direction in which the height h of the feature map output by the convolutional layer is located, and the width direction refers to the direction in which the width w of the feature map output by the convolutional layer is located. The average center position value in the width direction is shown in Equation (1), and the average center position value in the height direction is shown in Equation (2).

[0048]

[0049]

[0050] Among them, cx i cy i These represent the average center position values ​​in the width and height directions of the i-th channel activation location portion of the output feature vector, respectively. Equations (1) and (2) above are weighted averages, and the position value of the activation center is determined according to the size of the elements in the feature map.

[0051] In this optional implementation, setting the activation center position value to the average center position value in the width direction and the average center position value in the height direction of the splicing can improve the reliability of the location of the activation center position value and ensure the accuracy of the position information in the feature vector.

[0052] Optionally, the activation center position value can be the average center position value in the width direction.

[0053] Optionally, the activation center position value can also be the average center position value in the height direction.

[0054] In some optional implementations of this embodiment, the convolutional neural network further includes: a global pooling layer; the action image samples include multiple frames of action-annotated images, and the action recognition network is trained using selected action image samples, including: sequentially inputting each frame of action-annotated images in the selected action image samples into the convolutional layer for convolution operations to obtain feature maps corresponding to each frame of action-annotated images; for each feature map, inputting the feature map into the global pooling layer for global average pooling to obtain the average pooling value corresponding to the feature map; calculating the activation center position value corresponding to the feature map; concatenating the average pooling value and the activation center position value of the feature map to obtain the feature vector corresponding to the feature map; packaging all feature vectors corresponding to the multiple frames of action-annotated images and inputting them into the recurrent neural network to obtain the action classification result of the multiple frames of action-annotated images output by the recurrent neural network; and calculating the action loss value based on the action classification result.

[0055] In this embodiment, the pooling layer is used for feature dimensionality reduction, compressing the amount of data and parameters, and reducing overfitting. In this embodiment, the global pooling layer is used to define a spatial neighborhood and calculate the average value from the corrected feature map within the window.

[0056] In this embodiment, the average pooling value obtained by the global pooling layer is calculated as shown in the following formula (3):

[0057]

[0058] In equation (3), v i is the i-th element of the average pooling part of the output feature vector; h and w are the height and width of the feature map output by the convolutional neural network, respectively. It is the element at position (j,k) of the i-th channel of the feature map.

[0059] In this optional implementation, when the activation center position value includes the average center position value in the width direction and the average center position value in the height direction, the average pooling value is concatenated with the activation center position value to form a feature vector. Its dimension is 3 times that of the feature map of the last layer of the convolutional neural network. This vector is then input into the recurrent neural network to extract the time sequence information of the object's actions.

[0060] In this optional implementation, the multi-frame action annotation image is all the images from the first frame to the last frame after the action of the object to be identified is labeled. The multi-frame action annotation image can be an image after labeling an action (e.g., shaking the shoulder, raising the leg), or it can be an image after labeling multiple consecutive actions (e.g., taking a cup to drink water).

[0061] In this optional implementation, after inputting all image frames from the multi-frame action-annotated image into a convolutional neural network to obtain the feature vectors of all image frames, the feature vectors of all image frames are packaged and input into a recurrent neural network, which can ensure the reliability of the recurrent neural network in recognizing action-annotated samples.

[0062] Optionally, the above-mentioned action annotation samples are multi-frame action annotation images. Based on the action annotation samples selected from the sample set, a convolutional neural network and a recurrent neural network are used to calculate the action loss value, including: inputting each frame of the multi-frame action annotation images into the convolutional neural network in sequence to obtain the feature vector corresponding to each frame of the multi-frame action annotation images; removing feature vectors with the same value from the feature vectors, packing the feature vectors corresponding to the remaining images in the multi-frame action annotation images, and inputting them into the recurrent neural network to obtain the action classification result of the multi-frame action annotation images output by the recurrent neural network; and calculating the action loss value based on the action classification result.

[0063] In this optional implementation, when the convolutional neural network includes convolutional layers and global pooling layers, the global pooling layer is used to perform global average pooling on the feature map output by the convolutional layer, achieving the effect of average pooling of the feature map. This provides an optional implementation for obtaining feature vectors and ensures the reliability of the obtained feature vectors.

[0064] In some optional implementations of this embodiment, the convolutional neural network further includes: a region pooling layer; the action image samples include multiple frames of action-annotated images, and the action recognition network is trained using selected action image samples, including: sequentially inputting each frame of action-annotated images in the selected action image samples into the convolutional layer for convolution operations to obtain feature maps corresponding to each frame of action-annotated images; for each feature map, dividing the feature map into multiple different regions, inputting the feature map into the region pooling layer for region average pooling to obtain the average pooling value corresponding to the feature map; calculating the activation center position value corresponding to the feature map; concatenating the average pooling value and the activation center position value of the feature map to obtain the feature vector corresponding to the feature map; packaging all feature vectors corresponding to multiple frames of action-annotated images and inputting them into a recurrent neural network to obtain the action classification result of the multiple frames of action-annotated images output by the recurrent neural network; and calculating the action loss value based on the action classification result.

[0065] To address the issue that traditional global average pooling discards the positional information of elements in the original feature map, this embodiment provides a region-based average pooling method. This method divides the original feature map into regions and performs average pooling on elements within each region. The calculation method is as follows:

[0066]

[0067] In equation (4), The region pooling map corresponds to the value at position (i,j) of the k-th channel in the CNN output feature map; s m s n The step size for average pooling along the x, y dimensions; M k It is the k-th channel of the feature map. As can be seen from Equation (4), the region average pooling performed by the region pooling layer can partially restore the position information in the feature map. Finally, the generated region average pooling map is expanded into a feature vector, which is used as the feature vector input to the RNN after the CNN.

[0068] In this optional implementation, after obtaining the feature map output by the convolutional layer, the feature map is divided into regions, and regional average pooling is performed on each region to finally obtain the feature vector. The original global average pooling operation is improved to regional average pooling, which can partially retain the original discarded positional information and make the feature vector contain richer information.

[0069] In some optional implementations of this embodiment, the convolutional neural network is also used to downsample the feature maps corresponding to each frame of action annotation images output by the convolutional layer, so as to reduce the number of channels in the feature maps.

[0070] In this optional implementation, by adding another convolutional layer to the convolutional neural network, the convolutional neural network can downsample the feature map output by the current convolutional layer to the pooling layer (global pooling layer or region pooling layer). This allows the feature vector generated by the current convolutional layer to be downsampled by another convolutional layer before generating the feature vector when the feature vector dimension is too high. This minimizes the additional computational load and ensures the reliability of the action recognition network training.

[0071] Furthermore, based on the action recognition model training method provided in the above embodiments, this disclosure also provides an embodiment of an action recognition method, which combines artificial intelligence fields such as computer vision and deep learning.

[0072] See Figure 3 The diagram illustrates a flow 300 of an embodiment of the action recognition method according to the present disclosure. The action recognition method provided in this embodiment includes the following steps:

[0073] Step 301: Obtain the video sequence to be used for action recognition.

[0074] In this embodiment, the video sequence is a series of video images that are spatially and temporally related and arranged in a certain order. The series of video images may or may not include the object to be identified. In this embodiment, the object to be identified is the object to be identified by action recognition. When the series of video images includes the object to be identified, the type of action of the object to be identified can be identified by the action recognition model. It is also possible to identify multiple consecutive actions in the series of images.

[0075] In this embodiment, the acquired multiple video images may be color images and / or grayscale images, etc., and the format of the acquired multiple video images is not limited in this disclosure.

[0076] The entity executing the action recognition method can acquire the video sequence to be recognized in various ways. For example, the entity can obtain the video sequence stored in a database server via a wired or wireless connection. Alternatively, the entity can receive the video sequence to be recognized from a terminal or other device.

[0077] Step 302: Input the video sequence into the action recognition model and output the action classification result of the video sequence.

[0078] In this embodiment, the executing entity can input the video sequence obtained in step 301 into the action recognition model to obtain the action classification result of the object to be identified in the video sequence.

[0079] In this embodiment, the action recognition model can be as described above. Figure 1 The training was performed using the method described in the embodiments; for details of the training process, please refer to [link / reference]. Figure 1 The relevant descriptions of the embodiments will not be repeated here.

[0080] In this embodiment, the action recognition model can be used to identify the actions of an object to be identified in a video sequence, and classify the actions to obtain classification results. The action recognition model can be used to identify different actions of different people in a video sequence, and classify the identified different actions to obtain classification results for different actions. For example, if a series of video sequences of people playing ball are input into the action recognition model, the action recognition model will output all the ball-playing actions of each person, as well as the type of each ball-playing action.

[0081] It should be noted that the action recognition method in this embodiment can be used to test the action recognition models generated in the above embodiments. Furthermore, the action recognition model can be continuously optimized based on the action classification results. This method can also be a practical application of the action recognition models generated in the above embodiments. Using the action recognition models generated in the above embodiments to perform object action recognition in video sequences helps improve the accuracy of action recognition.

[0082] The action recognition method provided in this embodiment obtains a video sequence to be recognized; the video sequence is then input into an action recognition model generated using the action recognition model training method of this embodiment, resulting in the action classification result of the video sequence to be recognized output by the action recognition model. Therefore, processing the video sequence to be recognized using an action recognition model with added activation center position values ​​can improve the accuracy of action recognition and ensure the continuity of action recognition results.

[0083] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an action recognition model training device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0084] like Figure 4 As shown, the action recognition model training device 400 provided in this embodiment includes: a sample acquisition unit 401, a network acquisition unit 402, a selection unit 403, a training unit 404, and an output unit 405. The sample acquisition unit 401 can be configured to acquire a sample set, which includes at least one action image sample. The network acquisition unit 402 can be configured to acquire a pre-established action recognition network, which includes a convolutional neural network and a recurrent neural network connected to the convolutional neural network. The elements of the feature vector output by the convolutional neural network to the recurrent neural network include: an average pooling value and an activation center position value. The activation center position value is calculated from the value and position value of the element in the feature map output by the convolutional layer. The selection unit 403 can be configured to select action image samples from the sample set. The training unit 404 can be configured to train the action recognition network using the selected action image samples. The output unit 405 can be configured to use the action recognition network as an action recognition model in response to determining that the action recognition network meets the training completion conditions.

[0085] In this embodiment, the specific processing of the sample acquisition unit 401, network acquisition unit 402, selection unit 403, training unit 404, and output unit 405 in the action recognition model training device 400, and the resulting technical effects, can be found in reference to [reference needed]. Figure 1 The relevant descriptions of steps 101, 102, 103, 104, and 105 in the corresponding embodiments will not be repeated here.

[0086] In some optional implementations of this embodiment, the device 400 further includes an adjustment unit (not shown in the figure). The adjustment unit can be configured to adjust the relevant parameters in the action recognition network so that the action loss value of the action recognition network converges when the action recognition network does not meet the training completion conditions, and the selection unit 403, training unit 404, and output unit 405 continue training based on the adjusted action recognition network.

[0087] In some optional implementations of this embodiment, the above-mentioned activation center position value includes: the average center position value in the width direction and the average center position value in the height direction of the splicing; the average center position value in the width direction is equal to the sum of the products of the values ​​of each element in the width direction and the position value of that element divided by the sum of the values ​​of all elements; the average center position value in the height direction is equal to the sum of the products of the values ​​of each element in the height direction and the position value of that element divided by the sum of the values ​​of all elements.

[0088] In some optional implementations of this embodiment, the convolutional neural network includes a global pooling layer; the action image samples include multiple frames of action-annotated images; and the training unit 404 includes a feature acquisition module (not shown in the figure), a pooling acquisition module (not shown in the figure), a position calculation module (not shown in the figure), a vector acquisition module (not shown in the figure), an action classification module (not shown in the figure), and a loss calculation module (not shown in the figure). The feature acquisition module can be configured to sequentially input each frame of action-annotated images from the selected action image samples into the convolutional layer for convolution operations to obtain feature maps corresponding to each frame of action-annotated images. The pooling acquisition module can be configured to input each feature map into the global pooling layer for global average pooling to obtain the average pooling value corresponding to the feature map. The position calculation module can be configured to calculate the activation center position value corresponding to the feature map. The vector acquisition module can be configured to concatenate the average pooling value and the activation center position value of the feature map to obtain the feature vector corresponding to the feature map. The action classification module described above can be configured to package all feature vectors corresponding to multiple frames of action-annotated images, input them into a recurrent neural network, and obtain the action classification results of the multiple frames of action-annotated images output by the recurrent neural network. The loss calculation module described above can be configured to calculate the action loss value based on the action classification results.

[0089] In some optional implementations of this embodiment, the convolutional neural network further includes: a region pooling layer; the action image samples include multiple frames of action-annotated images, and the training unit 404 includes: a feature acquisition module (not shown in the figure), a region partitioning module (not shown in the figure), a center calculation module (not shown in the figure), a vector concatenation module (not shown in the figure), an action classification module (not shown in the figure), and a loss calculation module (not shown in the figure). The feature acquisition module can be configured to sequentially input each frame of action-annotated images from the selected action image samples into the convolutional layer for convolution operations to obtain feature maps corresponding to each frame of action-annotated images. The region partitioning module can be configured to divide each feature map into multiple different regions, input the feature map into the region pooling layer for region average pooling, and obtain the average pooling value corresponding to the feature map. The center calculation module can be configured to calculate the activation center position value corresponding to the feature map. The vector concatenation module can be configured to concatenate the average pooling value and the activation center position value of the feature map to obtain the feature vector corresponding to the feature map. The action classification module described above can be configured to package all feature vectors corresponding to multiple frames of action-annotated images, input them into a recurrent neural network, and obtain the action classification results of the multiple frames of action-annotated images output by the recurrent neural network. The loss calculation module described above can be configured to calculate the action loss value based on the action classification results.

[0090] In some optional implementations of this embodiment, the convolutional neural network is also used to downsample the feature maps corresponding to each frame of action annotation images output by the convolutional layer, so as to reduce the number of channels in the feature maps.

[0091] The action recognition model training apparatus provided in the embodiments of this disclosure firstly involves a sample acquisition unit 401 acquiring a sample set, which includes at least one action image sample. Secondly, a network acquisition unit 402 acquires a pre-established action recognition network, which includes a convolutional neural network (CNN) and a recurrent neural network (RNN) connected to the CNN. The elements of the feature vector output by the CNN to the RNN include: average pooling value and activation center position value. Thirdly, a selection unit 403 selects action image samples from the sample set. Next, a training unit 404 trains the action recognition network using the selected action image samples. Finally, an output unit 405, in response to determining that the action recognition network meets the training completion conditions, uses the action recognition network as an action recognition model. Thus, by setting activation center position values ​​in the feature vector, the positional information of elements in the feature map of the CNN is preserved. Furthermore, explicitly extracting this positional information and inputting it into the RNN can assist the action recognition network in completing action recognition, improving the accuracy of action recognition, while not affecting the original network structure, thus simplifying implementation.

[0092] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an action recognition device, which is similar to... Figure 3 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0093] like Figure 5 As shown, the action recognition device 500 provided in this embodiment includes: an acquisition unit 501 and a classification unit 502. The acquisition unit 501 can be configured to acquire a video sequence to be identified. The classification unit 502 can be configured to input the video sequence as described above. Figure 1 The action recognition model generated by the method described in the embodiment outputs the action classification results of the video sequence.

[0094] In this embodiment, the specific processing of the acquisition unit 501 and the classification unit 502 in the action recognition device 500, and the resulting technical effects, can be found in the following references: Figure 3 The relevant descriptions of steps 301 and 302 in the corresponding embodiments will not be repeated here.

[0095] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0096] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0097] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0098] like Figure 6As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0099] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0100] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as action recognition model training methods or action recognition methods. For example, in some embodiments, the action recognition model training method or action recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the action recognition model training method or action recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform an action recognition model training method or an action recognition method by any other suitable means (e.g., by means of firmware).

[0101] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a general-purpose computer, a special-purpose computer, or other programmable motion recognition model training device, the processor, or the controller of the motion recognition device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0105] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0106] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0107] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0108] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training an action recognition model, the method comprising: Obtain a sample set, the sample set including at least one motion image sample; A pre-established action recognition network is obtained. This network includes a convolutional neural network (CNN) and a recurrent neural network (RNN) connected to the CNN. The feature vector output by the CNN to the RNN contains elements of an average pooling value and activation center position values. These activation center position values ​​are calculated from the values ​​and positions of elements in the feature map output by the convolutional layers. The activation center position values ​​include: the average center position value in the width direction and the average center position value in the height direction. The average center position value in the width direction is equal to the sum of the products of the values ​​and positions of each element in the width direction divided by the sum of all element values. The average center position value in the height direction is equal to the sum of the products of the values ​​and positions of each element in the height direction divided by the sum of all element values. Select action image samples from the sample set, and use the selected action image samples to train the action recognition network; In response to determining that the action recognition network meets the training completion conditions, the action recognition network is used as an action recognition model.

2. The method according to claim 1, further comprising: In response to determining that the action recognition network does not meet the training completion conditions, the relevant parameters in the action recognition network are adjusted so that the action loss value of the action recognition network converges. Based on the adjusted action recognition network, action image samples in the sample set are selected again, and the selected action image samples are used to train the action recognition network.

3. The method according to claim 1 or 2, wherein, The convolutional neural network further includes a global pooling layer; the action image samples include multiple frames of action-annotated images, and training the action recognition network using selected action image samples includes: The selected action image samples are sequentially input into the convolutional layer for convolution operation to obtain the feature map corresponding to each action image. For each feature map, the feature map is input into the global pooling layer for global average pooling to obtain the average pooling value corresponding to the feature map; Calculate the activation center location value corresponding to this feature map; By concatenating the average pooling value and the activation center position value of the feature map, the feature vector corresponding to the feature map is obtained. After packaging all feature vectors corresponding to the multi-frame action annotation images, the recurrent neural network is input to obtain the action classification result of the multi-frame action annotation images output by the recurrent neural network. Based on the action classification results, the action loss value is calculated.

4. The method according to claim 1, wherein, The convolutional neural network further includes: a region pooling layer; the action image samples include multiple frames of action-annotated images, and training the action recognition network using selected action image samples includes: Each frame of the selected action image sample is sequentially input into the convolutional layer for convolution operation to obtain the feature map corresponding to each frame of action image. For each feature map, the feature map is divided into multiple different regions, and the feature map is input into the region pooling layer for region average pooling to obtain the average pooling value corresponding to the feature map. Calculate the activation center location value corresponding to this feature map; By concatenating the average pooling value and the activation center position value of the feature map, the feature vector corresponding to the feature map is obtained. After packaging all feature vectors corresponding to the multi-frame action annotation images, the recurrent neural network is input to obtain the action classification result of the multi-frame action annotation images output by the recurrent neural network. Based on the action classification results, the action loss value is calculated.

5. The method according to claim 1, wherein, The convolutional neural network is also used to downsample the feature maps corresponding to each frame of action annotation images output by the convolutional layer, so as to reduce the number of channels in the feature maps.

6. An action recognition method, the method comprising: Obtain the video sequence to be used for action recognition; The video sequence is input into an action recognition model generated using the method described in any one of claims 1-5, and the action classification result of the video sequence is output.

7. An action recognition model training device, the device comprising: A sample acquisition unit is configured to acquire a sample set, the sample set including at least one motion image sample; A network acquisition unit is configured to acquire a pre-established action recognition network, which includes a convolutional neural network (CNN) and a recurrent neural network (RNN) connected to the CNN. The feature vector output by the CNN to the RNN includes elements of an average pooling value and activation center position values. The activation center position values ​​are calculated from the values ​​and position values ​​of elements in the feature map output by the convolutional layer. The activation center position values ​​include: the average center position value in the width direction and the average center position value in the height direction. The average center position value in the width direction is equal to the sum of the products of the values ​​of each element in the width direction and their respective position values, divided by the sum of all element values. The average center position value in the height direction is equal to the sum of the products of the values ​​of each element in the height direction and their respective position values, divided by the sum of all element values. The selection unit is configured to select motion image samples from the sample set; The training unit is configured to train the action recognition network using selected action image samples; The output unit is configured to use the action recognition network as an action recognition model in response to determining that the action recognition network meets the training completion conditions.

8. The apparatus according to claim 7, further comprising: The adjustment unit is configured to adjust relevant parameters in the action recognition network to converge the action loss value of the action recognition network in response to the action recognition network not meeting the training completion conditions. The selection unit, the training unit, and the output unit then continue training based on the adjusted action recognition network.

9. The apparatus according to claim 7 or 8, wherein, The convolutional neural network further includes: a global pooling layer; the action image samples include multiple frames of action-annotated images; the training unit includes: The feature acquisition module is configured to sequentially input each frame of action annotation images from the selected action image samples into the convolutional layer for convolution operations to obtain the feature map corresponding to each frame of action annotation images. The pooling module is configured to input the feature map into the global pooling layer for global average pooling for each feature map, and obtain the average pooling value corresponding to the feature map. The location calculation module is configured to calculate the activation center location value corresponding to the feature map; The vector acquisition module is configured to concatenate the average pooling value and activation center position value of the feature map to obtain the feature vector corresponding to the feature map. The action classification module is configured to package all feature vectors corresponding to multi-frame action-annotated images and input them into the recurrent neural network to obtain the action classification result of the multi-frame action-annotated images output by the recurrent neural network. The loss calculation module is configured to calculate the action loss value based on the action classification result.

10. The apparatus according to claim 7, wherein, The convolutional neural network further includes: a region pooling layer; the action image samples include multiple frames of action-annotated images; the training unit includes: The feature acquisition module is configured to sequentially input each frame of action annotation images from the selected action image samples into the convolutional layer for convolution operations to obtain the feature map corresponding to each frame of action annotation images. The region segmentation module is configured to divide each feature map into multiple different regions, input the feature map into the region pooling layer for region average pooling, and obtain the average pooling value corresponding to the feature map. The center calculation module is configured to calculate the activation center location value corresponding to the feature map; The vector concatenation module is configured to concatenate the average pooling value and the activation center position value of the feature map to obtain the feature vector corresponding to the feature map. The action classification module is configured to package all feature vectors corresponding to multi-frame action-annotated images and input them into the recurrent neural network to obtain the action classification result of the multi-frame action-annotated images output by the recurrent neural network. The loss calculation module is configured to calculate the action loss value based on the action classification result.

11. The apparatus according to claim 7, wherein, The convolutional neural network is also used to downsample the feature maps corresponding to each frame of action annotation images output by the convolutional layer, so as to reduce the number of channels in the feature maps.

12. An action recognition device, the device comprising: The acquisition unit is configured to acquire the video sequence to be subjected to action recognition. The classification unit is configured to input the video sequence into an action recognition model generated by the method as described in any one of claims 1-5, and output the action classification result of the video sequence.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

Images