Method and apparatus for action recognition

Abstract

The embodiment of the present specification provides a motion recognition method and device, wherein the motion recognition method comprises: inputting a to-be-processed video into a motion recognition model, and performing convolution processing on the to-be-processed video through a two-dimensional convolution network in the motion recognition model to generate a corresponding first convolution vector, wherein the to-be-processed video contains a to-be-recognized motion of a target object, performing convolution processing on the to-be-processed video and the first convolution vector through a three-dimensional convolution network in the motion recognition model to generate a corresponding second convolution vector, processing the second convolution vector through a classification network of the motion recognition model to recognize the to-be-recognized motion, and generating and outputting a corresponding motion recognition result.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a motion recognition method. One or more embodiments of the present specification also relate to a motion recognition apparatus, a computing device, and a computer-readable storage medium. BACKGROUND

[0002] Currently, in many scenarios, the motion of a target object (e.g., a person) in a video needs to be recognized to determine the motion type of the person's motion. For example, in public places, in order to avoid the occurrence of harmful events such as climbing, fighting, and illegal operations that endanger personal and public safety, the motion of a person in a video obtained in a public place needs to be recognized. For example, in a human-computer interaction system, in order to understand the behavior of a person, the motion of a person in a video obtained by the human-computer interaction system needs to be recognized.

[0003] In order to realize the recognition of the motion of a person in a video, currently, motion recognition based on two-dimensional pose estimation and motion recognition based on three-dimensional pose estimation are mostly used. In the process of motion recognition based on two-dimensional pose estimation, the lack of depth information of the pose of a person makes the motion recognition accuracy of this method relatively low. In the process of motion recognition based on three-dimensional pose estimation, the depth information of the pose of a person is considered, but this method cannot be effectively applied in a motion recognition scene with limited computing resources and high real-time requirements while ensuring the accuracy of pose estimation and motion recognition. Therefore, an effective method is urgently needed to solve such problems. SUMMARY

[0004] In view of this, embodiments of the present specification provide a motion recognition method. One or more embodiments of the present specification also relate to a motion recognition apparatus, a computing device, and a computer-readable storage medium to solve the technical defects in the prior art.

[0005] According to a first aspect of embodiments of the present specification, a motion recognition method is provided, including:

[0006] inputting a to-be-processed video into a motion recognition model, and performing convolution processing on the to-be-processed video by a two-dimensional convolution network in the motion recognition model to generate a corresponding first convolution vector, wherein the to-be-processed video contains a to-be-recognized motion of a target object;

[0007] performing convolution processing on the to-be-processed video and the first convolution vector by a three-dimensional convolution network in the motion recognition model to generate a corresponding second convolution vector;

[0008] The second convolution vector is processed through a classification network of the action recognition model to recognize the to-be-recognized action, generate a corresponding action recognition result, and output.

[0009] According to a second aspect of an embodiment of the present specification, an action recognition device is provided, comprising:

[0010] An input module is configured to input a to-be-processed video into an action recognition model, and perform convolution processing on the to-be-processed video through a two-dimensional convolution network in the action recognition model to generate a corresponding first convolution vector, wherein the to-be-processed video contains a to-be-recognized action of a target object;

[0011] A processing module is configured to perform convolution processing on the to-be-processed video and the first convolution vector through a three-dimensional convolution network in the action recognition model to generate a corresponding second convolution vector;

[0012] An output module is configured to process the second convolution vector through a classification network of the action recognition model to recognize the to-be-recognized action, generate a corresponding action recognition result, and output.

[0013] According to a third aspect of an embodiment of the present specification, a computing device is provided, comprising:

[0014] a memory and a processor;

[0015] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions to implement steps of the action recognition method.

[0016] According to a fourth aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, and the instructions are executed by a processor to implement steps of the action recognition method.

[0017] One embodiment of the present specification inputs a to-be-processed video into an action recognition model, and performs convolution processing on the to-be-processed video through a two-dimensional convolution network in the action recognition model to generate a corresponding first convolution vector, wherein the to-be-processed video contains a to-be-recognized action of a target object, performs convolution processing on the to-be-processed video and the first convolution vector through a three-dimensional convolution network in the action recognition model to generate a corresponding second convolution vector, and processes the second convolution vector through a classification network of the action recognition model to recognize the to-be-recognized action, generate a corresponding action recognition result, and output.

[0018] The embodiment of the present specification combines a two-dimensional convolution network and a three-dimensional convolution network to obtain an action recognition model, which greatly simplifies the calculation complexity while ensuring similar feature extraction effect as the three-dimensional convolution, that is, through the network structure of the mixed two-dimensional convolution network and three-dimensional convolution network, the spatiotemporal features of the video to be processed can be better extracted under the premise of low calculation complexity, which is conducive to realizing better video action recognition effect, thereby being conducive to improving the accuracy and recognition efficiency of the action recognition result. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a schematic diagram of an action recognition process provided by an embodiment of the present specification;

[0020] Figure 2 is a processing flowchart of an action recognition method provided by an embodiment of the present specification;

[0021] Figure 3a is a processing flowchart of an action recognition method provided by an embodiment of the present specification;

[0022] Figure 3b is a processing flowchart of an action recognition method provided by an embodiment of the present specification;

[0023] Figure 3c is a processing flowchart of an action recognition method provided by an embodiment of the present specification;

[0024] Figure 4 is a processing flowchart of an action recognition method provided by an embodiment of the present specification;

[0025] Figure 5 is a schematic diagram of an action recognition device provided by an embodiment of the present specification;

[0026] Figure 6 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0027] In the following description, a lot of specific details are set forth in order to facilitate a thorough understanding of the present specification. However, the present specification can be implemented in many different ways than described herein, and those skilled in the art can make similar extensions without departing from the connotation of the present specification, so the present specification is not limited by the specific implementation disclosed below.

[0028] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0030] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0031] Video action recognition: This refers to extracting three-dimensional features in time and space from a video to determine the type of human action in the video. Usually, the action type is defined within a certain range.

[0032] 3D convolutional networks: also known as three-dimensional convolutional networks, can simultaneously perform convolution operations on image or video frames in both the spatial and temporal dimensions. Therefore, 3D convolution is often used in video-related tasks to extract both temporal and spatial features at the same time.

[0033] This specification provides an action recognition method, and also relates to an action recognition device, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0034] Figure 1 A schematic diagram of an action recognition process provided according to one embodiment of this specification is shown.

[0035] As facial recognition systems continue to develop, the recognition process has been gradually simplified. For example, the startup process has been simplified to improve the user experience, and the number of repeated recognition attempts has been reduced to increase recognition speed. However, with the optimization of the facial recognition experience, more security issues have also been exposed. Therefore, in practical applications, more security algorithms are needed to ensure the security of the recognition process and results.

[0036] In an actual face recognition system, similar situations may occur during the face brushing process, for example, the user's hand movements, the user's face from unobstructed to obstructed, the user turning away during the face brushing process, and the like. These are all actions related to the user, which can be feature extracted and action recognized through video. If the complete action of the user can be accurately recognized during the entire face brushing process, a more secure risk control chassis can be established in the face recognition system, thereby laying a solid foundation for further simplification and optimization of the system.

[0037] Based on this, the embodiments of the present specification propose an action recognition method based on a two-dimensional convolutional network and a three-dimensional convolutional network, which is used to obtain the action recognition result of the user from the video of the face recognition process through the way of spatio-temporal feature extraction, thereby assisting the safety judgment of the system.

[0038] Specifically, in the case that the to-be-processed video contains at least two video frames, a target video frame can be screened from the at least two video frames, and spatial feature extraction is performed on the target video frame, and the extracted spatial features are subjected to two-dimensional convolution processing.

[0039] In the screening of the target video frame, the spatial features of each video frame in the to-be-processed video can be extracted in turn through a first two-dimensional convolutional layer, and then based on the extraction result of the spatial features, the target video frame is screened from the at least two video frames contained in the to-be-processed video. The target video frame can be a key frame in the to-be-processed video. In actual application, if the action information of the target object contained in a certain video frame changes compared with the action information of the target object contained in the video frame before it, the video frame can be determined as the target video frame. Then the spatial features of the target video frame can be subjected to two-dimensional convolution processing to obtain the corresponding spatial vector.

[0040] Since the to-be-processed video contains at least two video frames, and the two-dimensional convolutional network can only process a single image or a single video frame, the two-dimensional convolutional network cannot directly obtain the time sequence features contained in the to-be-processed video. Based on this, when the first two-dimensional convolutional layer is used to perform time sequence convolution processing on the to-be-processed video, the number of video frames before the target video frame in the to-be-processed video can be determined first, and the video frames form a first video frame set. The number of video frames after the target video frame in the to-be-processed video can also be determined, and the video frames form a second video frame set. The video frames in the first video frame set are adjacent to each other, and one video frame is adjacent to the target video frame. Similarly, the video frames in the second video frame set are adjacent to each other, and one video frame is adjacent to the target video frame. Then, the optical flow information between the target video frame and each video frame in the first video frame set and the second video frame set can be determined, so as to extract the time sequence features between the target video frame and any video frame in the first video frame set and the second video frame set based on the optical flow information. Then, the two-dimensional convolution processing can be performed on the time sequence features to obtain a corresponding time sequence vector. The spatial vector and the time sequence vector are fused to generate a corresponding fusion result. The second two-dimensional convolutional layer is used to perform convolution processing on the fusion result to generate a corresponding first convolution vector.

[0041] Then, the first three-dimensional convolutional layer can be used to extract the time sequence features and the spatial features of the to-be-processed video. The feature extraction result is fused with the first convolution vector, and the fusion result is subjected to convolution processing to generate a first space-time vector. The second three-dimensional convolutional layer is used to perform convolution processing on the first space-time vector to generate a corresponding second space-time vector. The third three-dimensional convolutional layer is used to perform convolution processing on the second space-time vector to generate a corresponding second convolution vector. The pooling layer is used to perform pooling processing on the second convolution vector to generate a corresponding processing result. The full connection layer is used to process the processing result to perform action recognition on the to-be-recognized video, generate a corresponding action recognition result, and output the action recognition result.

[0042] The action recognition model is obtained by combining the two-dimensional convolutional network and the three-dimensional convolutional network. The calculation complexity is greatly simplified while ensuring similar feature extraction effect as the three-dimensional convolution. That is, the network structure of the two-dimensional convolutional network and the three-dimensional convolutional network is mixed. The space-time features of the to-be-processed video can be better extracted under the premise of low calculation complexity. The space-time information extraction capability of the optical flow and the three-dimensional convolutional network is fully utilized, which is conducive to achieving better video action recognition effect.

[0043] Figure 2 A processing flowchart of an action recognition method provided by an embodiment of the present specification is shown, including steps 202 to 206.

[0044] Step 202: Input the video to be processed into the action recognition model, and perform convolution processing on the video to be processed through the two-dimensional convolutional network in the action recognition model to generate the corresponding first convolution vector, wherein the video to be processed contains the action to be recognized of the target object.

[0045] Specifically, the target object is the object that performs the corresponding limb movements, including but not limited to people, animals, robots, etc.; the video to be processed is the video generated by capturing the movements of the target object using a corresponding device. This device can be a device equipped with an image capture device. The video to be processed can be generated by directly shooting video using this device, or it can be generated by first capturing images of the movement process when the target object performs the corresponding movements, and then stitching the captured images together to generate the video. The specific method can be determined according to actual needs and is not limited here.

[0046] Since the video to be processed contains the action to be identified of the target object, the action recognition of the target object can be achieved through the video to be processed. Specifically, the video to be processed can be input into the action recognition model, and the action recognition model can be used to process the features contained in each video frame of the video to be processed, so as to identify the action to be identified of the target object contained in the video to be processed based on the processing result.

[0047] In practical applications, this action recognition model can be composed of a combination of a two-dimensional convolutional network (2D convolutional network), a three-dimensional convolutional network (3D convolutional network), and a classification network. Therefore, after the video to be processed is input into the action recognition model, the two-dimensional convolutional network first performs two-dimensional convolution processing on each video frame (one frame, two frames, or multiple frames) in the video to be processed to generate the corresponding first convolutional vector. Then, the three-dimensional convolutional network processes the first convolutional vector and the video to be processed to recognize the action to be recognized of the target object contained in the video to be processed based on the processing result.

[0048] For example, if the video to be processed contains T video frames, after inputting the video into the action recognition model, a two-dimensional convolutional network can be used to sequentially extract the spatial features of each of the T video frames. The extraction result can be {x1, x2, x3, ..., x...} T}, where x1 is the spatial feature of the first video frame in the video to be processed, x2 is the spatial feature of the second video frame in the video to be processed, and so on. The spatial feature is a two-dimensional vector. Then, the spatial features of each video frame can be processed by two-dimensional convolution to obtain the spatial vector corresponding to each video frame.

[0049] In addition, in addition to extracting the spatial features of each video frame, the optical flow information between any two adjacent video frames in the T-frame video frame can be determined, so as to determine the time sequence features between the two video frames based on the optical flow information, and the determination result is {y1, y2, y3, …, y T-1} wherein y1 is the time sequence feature between the first video frame and the second video frame in the to-be-processed video, y2 is the time sequence feature between the second video frame and the third video frame, and so on.

[0050] After extracting the spatial features and the time sequence features, the spatial features and the time sequence features can be fused to generate a first convolution vector of a unified dimension.

[0051] In addition, after the two-dimensional convolution network extracts the spatial features of each video frame in the to-be-processed video, the target video frame can be screened from at least two video frames included in the to-be-processed video based on the extraction result of the spatial features. The target video frame can be a key frame in the to-be-processed video. In actual application, if the action information of the target object included in a video frame changes compared with the action information of the target object included in a video frame before the video frame, the video frame can be determined as the target video frame. Then, the spatial features of the target video frame can be subjected to two-dimensional convolution processing to obtain a corresponding spatial vector.

[0052] Correspondingly, after the target video frame is determined, a certain number of video frames before the target video frame in the to-be-processed video frame can be determined, and the video frames form a first video frame set. A certain number of video frames after the target video frame can be determined, and the video frames form a second video frame set. The video frames in the first video frame set are adjacent to each other, and one video frame is adjacent to the target video frame. Similarly, the video frames in the second video frame set are adjacent to each other, and one video frame is adjacent to the target video frame. Then, the optical flow information between the target video frame and each video frame in the first video frame set and the second video frame set can be determined based on the optical flow information, so as to extract the time sequence features between the target video frame and any video frame in the first video frame set and the second video frame set based on the optical flow information. Then, the time sequence features can be subjected to two-dimensional convolution processing to obtain a corresponding time sequence vector.

[0053] After the spatial vector and the time sequence vector are generated, the spatial vector and the time sequence vector can be fused to generate a first convolution vector of a unified dimension.

[0054] In specific implementation, the two-dimensional convolution network includes a first two-dimensional convolution layer and a second two-dimensional convolution layer.

[0055] Correspondingly, the two-dimensional convolution network in the action recognition model is used to perform convolution processing on the to-be-processed video to generate a corresponding first convolution vector, which includes:

[0056] The video to be processed is subjected to temporal convolution processing through the first two-dimensional convolutional layer to generate a first temporal vector, and the video to be processed is subjected to spatial convolution processing through the first two-dimensional convolutional layer to generate a first spatial vector.

[0057] The first spatial vector and the first temporal vector are fused to generate the corresponding fusion result;

[0058] The fusion result is processed by convolution through the second two-dimensional convolutional layer to generate the corresponding first convolutional vector.

[0059] Furthermore, the video to be processed contains at least two video frames;

[0060] Accordingly, the step of performing temporal convolution processing on the video to be processed through the first two-dimensional convolutional layer to generate a first temporal vector includes:

[0061] The optical flow images corresponding to any two adjacent video frames in the at least two video frames are determined. Temporal features are extracted from the optical flow images through the first two-dimensional convolutional layer, and the feature extraction results are convolved to generate a first temporal vector.

[0062] A schematic diagram of another action recognition process provided in the embodiments of this specification is shown below. Figure 3a As shown, Figure 3a In the action recognition model, a two-dimensional convolutional network is included, which contains two two-dimensional convolutional layers, namely the first two-dimensional convolutional layer and the second two-dimensional convolutional layer.

[0063] Since the video to be processed contains at least two video frames, and a two-dimensional convolutional network can only process a single image or a single video frame, it is not possible to directly obtain the temporal features contained in the video to be processed through a two-dimensional convolutional network. Based on this, in the embodiments of this specification, when performing temporal convolution processing on the video to be processed through the first two-dimensional convolutional layer, the motion information between any two adjacent video frames in at least two video frames can be determined first, that is, the motion information of the target object between any two video frames. Then, an optical flow image corresponding to any two adjacent video frames is generated based on the motion information. Then, the temporal features of the optical flow image are extracted through the first two-dimensional convolutional layer, and the extraction result is subjected to two-dimensional convolution processing to generate the corresponding temporal vector.

[0064] Optical flow, which refers to the velocity of motion patterns in time-varying images, i.e., the instantaneous velocity of pixels moving on the imaging plane of a moving object in space, is an important method for motion image analysis.

[0065] In cases where two-dimensional convolutional networks cannot directly obtain the temporal features of the video to be processed, in order to ensure the accuracy of the action recognition results output by the action recognition model, the embodiments of this specification can determine the optical flow image corresponding to two adjacent video frames based on the motion information of the target object contained between any two adjacent video frames in the video to be processed.

[0066] Specifically, feature points (corner points) can be extracted from any two adjacent video frames. Then, each feature point is tracked within the two video frames to determine its motion information between the two adjacent frames, including but not limited to direction and speed. Based on this motion information, the motion bias in the x and y directions of each feature point as it moves from video frame A to video frame B is determined. The resulting optical flow tracking image, which contains not only the motion information of each feature point but also the 3D structural information of the relevant scene, is then used to extract temporal features between any two adjacent video frames. These temporal features are then subjected to 2D convolution to generate corresponding temporal vectors.

[0067] For example, if the video to be processed contains T video frames, then the optical flow images corresponding to any two adjacent video frames are generated, and the corresponding generation result is {p1, p2, p3, ..., p...} T-1}, where p1 is the optical flow image corresponding to the first and second video frames in the video to be processed, p2 is the optical flow image corresponding to the second and third video frames, and so on.

[0068] Based on optical flow images, the temporal characteristics between any two adjacent video frames can be determined, and the result can be {y1, y2, y3, ..., y...}. T-1}, where y1 is the temporal feature between the first and second video frames in the video to be processed, y2 is the temporal feature between the second and third video frames, and so on.

[0069] In addition to generating the first temporal vector, the video to be processed can be spatially convolved through the first two-dimensional convolutional layer to generate the first spatial vector. The first spatial vector and the first temporal vector are then fused to generate the corresponding fusion result. The fusion result is then convolved through the second two-dimensional convolutional layer to generate the corresponding first convolutional vector.

[0070] In specific implementation, the video to be processed is spatially convolutionally processed through the first two-dimensional convolutional layer to generate a first spatial vector, including:

[0071] Spatial features are extracted from the target video frames in the at least two video frames through the first two-dimensional convolutional layer, and the feature extraction results are convolved to generate a first spatial vector.

[0072] Furthermore, spatial features are extracted from the target video frames in the at least two video frames through the first two-dimensional convolutional layer, and the feature extraction results are subjected to convolution processing, including:

[0073] Spatial features are extracted from each video frame through the first two-dimensional convolutional layer, and the target video frame is determined from the at least two video frames based on the extraction results.

[0074] The spatial features of the target video frame are convolved through the first two-dimensional convolutional layer to generate a first spatial vector.

[0075] Specifically, when the video to be processed contains at least two video frames, a target video frame can be selected from the at least two video frames, spatial features can be extracted from the target video frame, and the extracted spatial features can be processed by two-dimensional convolution.

[0076] In the process of selecting target video frames, the spatial features of each video frame in the video to be processed can be extracted sequentially through the first two-dimensional convolutional layer. Then, based on the extraction results of the spatial features, target video frames can be selected from at least two video frames contained in the video to be processed. The target video frame can be a keyframe in the video to be processed. In practical applications, if the action information of the target object in a video frame changes compared with the action information of the target object in the video frame before a certain video frame, then the video frame can be identified as the target video frame. Then, the spatial features of the target video frame can be processed by two-dimensional convolution to obtain the corresponding spatial features.

[0077] Specifically, since the action information of the target object contained in the target video frame has changed compared with the action information of the target object contained in the video frames before this video frame, extracting spatial features from the target video frame and performing action recognition based on the extracted spatial features is beneficial to improving the recognition efficiency of action recognition.

[0078] Furthermore, temporal convolution processing is performed on the video to be processed through the first two-dimensional convolutional layer, including:

[0079] A predetermined number of video frames to be processed are determined among the at least two video frames that are adjacent to the target video frame, and the corresponding optical flow image is determined based on the motion information of the target object contained in the target video frame and the video frames to be processed.

[0080] Temporal features are extracted from the optical flow image through the first two-dimensional convolutional layer, and the feature extraction results are then subjected to convolution processing.

[0081] Specifically, a schematic diagram of a two-dimensional convolutional layer provided in the embodiments of this specification is shown below. Figure 3b As shown.

[0082] When determining a target video frame from at least two video frames of a video to be processed, and extracting spatial features from the target video frame, if it is necessary to determine the temporal features of the video to be processed, the target video frame can be used as a basis. A certain number of video frames preceding the target video frame can be determined within the video to be processed; these video frames form a first set. A certain number of video frames following the target video frame can be determined; these video frames form a second set. In the first set, all video frames are pairwise adjacent, and one video frame is adjacent to the target video frame. Similarly, in the second set, all video frames are pairwise adjacent, and one video frame is adjacent to the target video frame.

[0083] Then, optical flow images between each video frame in the first video frame set and the second video frame set and the target video frame can be determined, so as to extract temporal features between the target video frame and any video frame in the first video frame set and the second video frame set based on the optical flow images.

[0084] For example, if the video to be processed contains 20 video frames, and the 10th video frame is determined as the target video frame, then the 5th to 9th video frames can be determined as the first video set, and the 11th to 15th video frames can be determined as the second video set. Then, the optical flow images corresponding to the 10th and 5th video frames, the 10th and 6th video frames, ..., and the 10th and 15th video frames are determined. Then, the temporal features of each optical flow image are extracted, and the temporal features of each optical flow image are processed by two-dimensional convolution through the first two-dimensional convolutional layer to generate the corresponding temporal vector.

[0085] In cases where two-dimensional convolutional networks cannot directly obtain the temporal features of the video to be processed, the embodiments of this specification determine a preset number of video frames to be processed adjacent to the target video frame, and determine the optical flow image corresponding to the target video frame and each video frame to be processed based on the motion information of the target object contained in the target video frame and the video frames to be processed. Then, the temporal features of the optical flow image are extracted, and action recognition is performed based on the temporal information and spatial features, which helps to improve the accuracy and efficiency of action recognition results.

[0086] In specific implementation, temporal convolution processing is performed on the video to be processed, including:

[0087] Extract the spatial features of the target video frame in the video to be processed, and perform convolution processing on the spatial features according to the preset convolution kernel and the first preset stride to generate a first spatial vector;

[0088] Accordingly, the convolutional processing of the fusion result through the second two-dimensional convolutional layer includes:

[0089] The fusion result is processed by the second two-dimensional convolutional layer according to the preset convolutional kernel and the second preset stride.

[0090] Specifically, in the embodiments of this specification, the temporal and spatial features are processed by two-dimensional convolution through a first two-dimensional convolutional layer to generate corresponding temporal and spatial vectors. Then, the fusion result of the temporal and spatial vectors is processed by a second two-dimensional convolutional layer. The purpose is to reduce the dimensionality of the fusion result so that the fusion result becomes a feature vector of a fixed dimension, thereby satisfying the input conditions of a three-dimensional convolutional network.

[0091] In practical applications, the number of convolutional kernels, kernel size, and convolution step used by the first two-dimensional convolutional layer to process temporal and spatial features can be determined according to actual needs and are not restricted here; similarly, the number of convolutional kernels, kernel size, and convolution step used by the second two-dimensional convolutional layer to process temporal and spatial vectors can be determined according to actual needs and are not restricted here.

[0092] Step 204: The video to be processed and the first convolution vector are convolved through the three-dimensional convolutional network in the action recognition model to generate the corresponding second convolution vector.

[0093] Specifically, after processing video frames in the video to be processed through the two-dimensional convolutional network in the action recognition model to obtain the fusion result of the corresponding temporal vector and spatial vector, the fusion result can be further processed through the three-dimensional convolutional network in the action recognition model.

[0094] Furthermore, since a 3D convolutional network can directly extract all temporal and spatial features contained in the video to be processed, in order to ensure the accuracy of the model output, the video to be processed in the embodiments of this specification can be used as input to a 3D convolutional network in addition to being directly input to a 2D convolutional network. This allows the 3D convolutional network to extract the spatiotemporal features (the fusion result of temporal and spatial features) in the video to be processed. Then, the spatiotemporal features are fused again with the fusion result output by the 2D convolutional network, and the fusion result is further convolved. Finally, the action to be identified of the target object contained in the video to be processed is identified based on the convolution processing result.

[0095] In practice, the three-dimensional convolutional network includes a first three-dimensional convolutional layer, a second three-dimensional convolutional layer, and a third three-dimensional convolutional layer;

[0096] Accordingly, the step of performing convolution processing on the video to be processed and the first convolution vector through the three-dimensional convolutional network in the action recognition model to generate a corresponding second convolution vector includes:

[0097] The first three-dimensional convolutional layer is used to extract temporal and spatial features from the video to be processed. The feature extraction results are then fused with the first convolutional vector, and the fused results are further processed by convolution to generate a first spatiotemporal vector.

[0098] The first spatiotemporal vector is convolved by the second three-dimensional convolutional layer to generate the corresponding second spatiotemporal vector.

[0099] The second spatiotemporal vector is convolved by the third three-dimensional convolutional layer to generate the corresponding second convolutional vector.

[0100] Specifically, such as Figure 3a As shown, the action recognition model contains a three-dimensional convolutional network, which in turn contains three convolutional layers: the first three-dimensional convolutional layer, the second three-dimensional convolutional layer, and the third three-dimensional convolutional layer.

[0101] In the embodiments of this specification, after the second two-dimensional convolutional layer outputs the first convolutional vector, the first convolutional vector and the video to be processed are used together as the input of the first three-dimensional convolutional layer to extract the spatiotemporal features (the fusion result of temporal features and spatial features) in the video to be processed. Then, the spatiotemporal features are fused again with the first convolutional vector output by the second two-dimensional convolutional layer, and the obtained fusion result is further processed by three-dimensional convolution. Finally, the action to be identified of the target object contained in the video to be processed is identified according to the convolution processing result.

[0102] The first 3D convolutional layer performs 3D convolution processing on the fusion result to generate a first spatiotemporal vector, the second 3D convolutional layer performs 3D convolution processing on the first spatiotemporal vector, and the third 3D convolutional layer performs 3D convolution processing on the second spatiotemporal vector output by the second 3D convolutional layer. The purpose of all these processes is to reduce the dimensionality of the first and second spatiotemporal vectors so that the final output second convolutional vector becomes a feature vector of fixed dimension, thereby satisfying the input conditions of the classification network.

[0103] In practical applications, the number of convolutional kernels, kernel size, and convolution step used by the first 3D convolutional layer to perform convolution processing on the fusion result can be determined according to actual needs and are not restricted here. Similarly, the number of convolutional kernels, kernel size, and convolution step used by the second 3D convolutional layer to perform 3D convolution processing on the first spatiotemporal vector, and the third 3D convolutional layer to perform 3D convolution processing on the output of the second 3D convolutional layer, can be determined according to actual needs and are not restricted here.

[0104] Furthermore, to reduce the computational complexity of the 3D convolutional network, the embodiments in this specification can decompose the 3D convolution into two sequential 2D convolutions. The specific decomposition result of the 3D convolutional layer is as follows: Figure 3c As shown, when performing three-dimensional convolution processing on the fusion result (t×d×d) of the spatiotemporal features and the first convolution vector, we can first perform two-dimensional convolution on each spatial feature (t 1×d×d spatial features) contained in the fusion result, then fuse the two-dimensional convolution results corresponding to each spatial feature, and then perform two-dimensional convolution on the fusion result (t×1×1, where t represents the dimension) in the time dimension to obtain the corresponding three-dimensional convolution result.

[0105] This processing method significantly simplifies computational complexity while maintaining feature extraction effects similar to 3D convolution. By using a hybrid network structure of 2D and 3D convolutional networks, it can better extract the spatiotemporal features of the video under low computational complexity and make full use of the spatiotemporal information extraction capabilities of optical flow and 3D convolutional networks, which is conducive to achieving better video action recognition results.

[0106] Step 206: The second convolutional vector is processed by the classification network of the action recognition model to identify the action to be recognized, generate the corresponding action recognition result and output it.

[0107] Specifically, after extracting the spatiotemporal features of the video to be processed by combining two-dimensional and three-dimensional convolutional networks, pooling operations can be performed on the spatiotemporal features. Then, the pooling results are processed through a fully connected layer for action classification, so as to recognize the actions of the target objects in the video to be processed.

[0108] In practice, the classification network includes a pooling layer and a fully connected layer;

[0109] Accordingly, the second convolutional vector is processed by the classification network of the action recognition model to identify the action to be recognized, generate the corresponding action recognition result, and output it, including:

[0110] The second convolutional vector is pooled using the pooling layer to generate the corresponding processing result.

[0111] The video to be identified is subjected to action recognition through the fully connected layer and the processing results, and the corresponding action recognition results are generated and output.

[0112] Specifically, the pooling layer reduces the dimensionality of data by mimicking the human visual system, representing the image with higher-level features; while the fully connected layer acts as a "classifier" in the entire convolutional neural network. Therefore, in the embodiments of this specification, after obtaining the spatiotemporal features (second convolutional vector) of the video to be processed, the spatiotemporal features can first be input into the pooling layer of the classification network for dimensionality reduction, so that the vector dimension of the dimensionality-reduced spatiotemporal features meets the input conditions of the fully connected layer. Then, the dimensionality-reduced spatiotemporal features are input into the fully connected layer for classification processing, and the classification result output by the fully connected layer can be used as the recognition result of the action to be identified in the target object in the video to be processed.

[0113] In addition, when recognizing the action of the target object in the video to be processed, in addition to directly using the video to be processed as the input of the action recognition network, the video to be processed can also be segmented, each segment can be extracted separately for spatiotemporal features, and then the spatiotemporal features of each video segment can be spliced ​​and fused by network structures such as LSTM to obtain better recognition results.

[0114] When the action recognition method provided in the embodiments of this specification is applied to the field of facial recognition, the security capabilities of the facial recognition system can be built by recognizing user actions.

[0115] The embodiments in this specification combine two-dimensional convolutional networks and three-dimensional convolutional networks to obtain an action recognition model. This allows for the simultaneous combination of temporal information extracted by the two-dimensional convolutional network and the original temporal information in the video to be processed, from the perspective of temporal information, to obtain feature fusion temporal information. This helps to improve the robustness of the recognition results output by the action recognition model.

[0116] One embodiment of this specification involves inputting a video to be processed into an action recognition model, and then performing convolution processing on the video using a two-dimensional convolutional network in the action recognition model to generate a corresponding first convolutional vector. The video contains actions to be recognized from a target object. A three-dimensional convolutional network in the action recognition model is then used to perform convolution processing on the video and the first convolutional vector to generate a corresponding second convolutional vector. The classification network of the action recognition model then processes the second convolutional vector to recognize the actions to be recognized, generating and outputting a corresponding action recognition result.

[0117] The embodiments in this specification combine two-dimensional and three-dimensional convolutional networks to obtain an action recognition model. While ensuring feature extraction effects similar to those of three-dimensional convolution, the computational complexity is greatly simplified. That is, through the network structure of the hybrid two-dimensional and three-dimensional convolutional networks, the spatiotemporal features of the video to be processed can be extracted better with low computational complexity. It also makes full use of the spatiotemporal information extraction capabilities of optical flow and three-dimensional convolutional networks, which is conducive to achieving better video action recognition results, thereby improving the accuracy and recognition efficiency of action recognition results.

[0118] The following is in conjunction with the appendix Figure 4 Taking the application of the action recognition method provided in this specification in a real-world scenario as an example, the action recognition method will be further explained. Figure 4 The flowchart of an action recognition method provided in one embodiment of this specification is shown, and the specific steps include steps 402 to 422.

[0119] Step 402: Input the video to be processed into the action recognition model.

[0120] The video to be processed contains at least two video frames, each containing the action to be identified of the target object; the action recognition model includes a two-dimensional convolutional network, a three-dimensional convolutional network, and a classification network. The two-dimensional convolutional network includes a first two-dimensional convolutional layer and a second two-dimensional convolutional layer, the three-dimensional convolutional network includes a first three-dimensional convolutional layer, a second three-dimensional convolutional layer, and a third three-dimensional convolutional layer, and the classification network includes a pooling layer and a fully connected layer.

[0121] Step 404: Determine the target video frame from at least two video frames, and perform convolution processing on the spatial features of the target video frame through a first two-dimensional convolutional layer to generate a first spatial vector.

[0122] Step 406: Determine a preset number of video frames to be processed that are adjacent to the target video frame in at least two video frames, and determine the corresponding optical flow image based on the motion information of the target object contained in the target video frame and the video frames to be processed.

[0123] Step 408: Temporal features are extracted from the optical flow image through the first two-dimensional convolutional layer, and the feature extraction results are convolved to generate the first temporal vector.

[0124] Step 410: The first spatial vector and the first temporal vector are fused to generate the corresponding fusion result.

[0125] Step 412: The fusion result is processed by convolution through the second two-dimensional convolutional layer to generate the corresponding first convolutional vector.

[0126] Step 414: Through the first three-dimensional convolutional layer, temporal and spatial features are extracted from the video to be processed. The feature extraction results are fused with the first convolutional vector, and the fused results are convolved to generate the first spatiotemporal vector.

[0127] Step 416: The first spatiotemporal vector is convolved by the second three-dimensional convolutional layer to generate the corresponding second spatiotemporal vector.

[0128] Step 418: The second spatiotemporal vector is convolved by the third three-dimensional convolutional layer to generate the corresponding second convolutional vector.

[0129] Step 420: The second convolutional vector is pooled using a pooling layer to generate the corresponding processing result.

[0130] Step 422: Perform action recognition on the video to be recognized through a fully connected layer, generate the corresponding action recognition result, and output it.

[0131] The embodiments in this specification combine two-dimensional and three-dimensional convolutional networks to obtain an action recognition model. While ensuring feature extraction effects similar to those of three-dimensional convolution, the computational complexity is greatly simplified. That is, through the network structure of the hybrid two-dimensional and three-dimensional convolutional networks, the spatiotemporal features of the video to be processed can be extracted better with low computational complexity. It also makes full use of the spatiotemporal information extraction capabilities of optical flow and three-dimensional convolutional networks, which is conducive to achieving better video action recognition results, thereby improving the accuracy and recognition efficiency of action recognition results.

[0132] Corresponding to the above method embodiments, this specification also provides embodiments of action recognition devices. Figure 5 A schematic diagram of an action recognition device according to one embodiment of this specification is shown. Figure 5 As shown, the device includes:

[0133] The input module 502 is configured to input the video to be processed into the action recognition model, and to perform convolution processing on the video to be processed through the two-dimensional convolutional network in the action recognition model to generate a corresponding first convolution vector, wherein the video to be processed contains the action to be recognized of the target object;

[0134] Processing module 504 is configured to perform convolution processing on the video to be processed and the first convolution vector through the three-dimensional convolutional network in the action recognition model to generate a corresponding second convolution vector.

[0135] The output module 506 is configured to process the second convolutional vector through the classification network of the action recognition model to recognize the action to be recognized, generate the corresponding action recognition result and output it.

[0136] Optionally, the two-dimensional convolutional network includes a first two-dimensional convolutional layer and a second two-dimensional convolutional layer;

[0137] Accordingly, the input module 502 is further configured as follows:

[0138] The video to be processed is subjected to temporal convolution processing through the first two-dimensional convolutional layer to generate a first temporal vector, and the video to be processed is subjected to spatial convolution processing through the first two-dimensional convolutional layer to generate a first spatial vector.

[0139] The first spatial vector and the first temporal vector are fused to generate the corresponding fusion result;

[0140] The fusion result is processed by convolution through the second two-dimensional convolutional layer to generate the corresponding first convolutional vector.

[0141] Optionally, the video to be processed contains at least two video frames;

[0142] Accordingly, the input module 502 is further configured as follows:

[0143] The optical flow images corresponding to any two adjacent video frames in the at least two video frames are determined. Temporal features are extracted from the optical flow images through the first two-dimensional convolutional layer, and the feature extraction results are convolved to generate a first temporal vector.

[0144] Optionally, the input module 502 is further configured to:

[0145] Spatial features are extracted from the target video frames in the at least two video frames through the first two-dimensional convolutional layer, and the feature extraction results are convolved to generate a first spatial vector.

[0146] Optionally, the input module 502 is further configured to:

[0147] Spatial features are extracted from each video frame through the first two-dimensional convolutional layer, and the target video frame is determined from the at least two video frames based on the extraction results.

[0148] The spatial features of the target video frame are processed by convolution through the first two-dimensional convolutional layer.

[0149] Optionally, the input module 502 is further configured to:

[0150] A predetermined number of video frames to be processed are determined among the at least two video frames that are adjacent to the target video frame, and the corresponding optical flow image is determined based on the motion information of the target object contained in the target video frame and the video frames to be processed.

[0151] Temporal features are extracted from the optical flow image through the first two-dimensional convolutional layer, and the feature extraction results are then subjected to convolution processing.

[0152] Optionally, the input module 502 is further configured to:

[0153] Extract the spatial features of the target video frame in the video to be processed, and perform convolution processing on the spatial features according to the preset convolution kernel and the first preset stride to generate a first spatial vector;

[0154] Accordingly, the input module 502 is further configured as follows:

[0155] The fusion result is processed by the second two-dimensional convolutional layer according to the preset convolutional kernel and the second preset stride.

[0156] Optionally, the three-dimensional convolutional network includes a first three-dimensional convolutional layer, a second three-dimensional convolutional layer, and a third three-dimensional convolutional layer;

[0157] Accordingly, the processing module 504 is further configured to:

[0158] The first three-dimensional convolutional layer is used to extract temporal and spatial features from the video to be processed. The feature extraction results are then fused with the first convolutional vector, and the fused results are further processed by convolution to generate a first spatiotemporal vector.

[0159] The first spatiotemporal vector is convolved by the second three-dimensional convolutional layer to generate the corresponding second spatiotemporal vector.

[0160] The second spatiotemporal vector is convolved by the third three-dimensional convolutional layer to generate the corresponding second convolutional vector.

[0161] Optionally, the classification network includes a pooling layer and a fully connected layer;

[0162] Accordingly, the output module 506 is further configured as follows:

[0163] The second convolutional vector is pooled using the pooling layer to generate the corresponding processing result.

[0164] The video to be identified is subjected to action recognition through the fully connected layer and the processing results, and the corresponding action recognition results are generated and output.

[0165] The above is a schematic scheme of an action recognition device according to this embodiment. It should be noted that the technical solution of this action recognition device and the technical solution of the action recognition method described above belong to the same concept. For details not described in detail in the technical solution of the action recognition device, please refer to the description of the technical solution of the action recognition method described above.

[0166] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0167] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0168] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0169] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 600 can also be a mobile or stationary server.

[0170] The memory 610 is used to store computer-executable instructions, and the processor 620 is used to execute the following computer-executable instructions to implement the steps of the action recognition method.

[0171] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described action recognition method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described action recognition method.

[0172] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, are used to implement the steps of the action recognition method.

[0173] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above-described action recognition method belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-described action recognition method.

[0174] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0175] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0176] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0177] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0178] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An action recognition method, comprising: The video to be processed is input into the action recognition model, and the video to be processed is convolved through the two-dimensional convolutional network in the action recognition model to generate the corresponding first convolutional vector. The video to be processed contains the action to be recognized of the target object, and the two-dimensional convolutional network is a convolutional network that extracts the spatial and temporal features of the video to be processed. The video to be processed and the first convolutional vector are convolved through the three-dimensional convolutional network in the action recognition model to generate a corresponding second convolutional vector. The three-dimensional convolutional network is a convolutional network that extracts the fusion result of temporal and spatial features of the video to be processed. The convolutional processing of the video to be processed and the first convolutional vector includes extracting temporal and spatial features of the video to be processed, fusing the feature extraction result with the first convolutional vector, and performing convolution processing on the fusion result. The second convolutional vector is processed by the classification network of the action recognition model to identify the action to be recognized, generate the corresponding action recognition result, and output it.

2. The action recognition method according to claim 1, wherein the two-dimensional convolutional network comprises a first two-dimensional convolutional layer and a second two-dimensional convolutional layer; Accordingly, the step of performing convolution processing on the video to be processed through the two-dimensional convolutional network in the action recognition model to generate a corresponding first convolution vector includes: The video to be processed is subjected to temporal convolution processing through the first two-dimensional convolutional layer to generate a first temporal vector, and the video to be processed is subjected to spatial convolution processing through the first two-dimensional convolutional layer to generate a first spatial vector. The first spatial vector and the first temporal vector are fused to generate the corresponding fusion result; The fusion result is processed by convolution through the second two-dimensional convolutional layer to generate the corresponding first convolutional vector.

3. The action recognition method according to claim 2, wherein the video to be processed comprises at least two video frames; Accordingly, the step of performing temporal convolution processing on the video to be processed through the first two-dimensional convolutional layer to generate a first temporal vector includes: The optical flow images corresponding to any two adjacent video frames in the at least two video frames are determined. Temporal features are extracted from the optical flow images through the first two-dimensional convolutional layer, and the feature extraction results are convolved to generate a first temporal vector.

4. The action recognition method according to claim 2, wherein the video to be processed comprises at least two video frames; Accordingly, the step of performing spatial convolution processing on the video to be processed through the first two-dimensional convolutional layer to generate a first spatial vector includes: Spatial features are extracted from the target video frames in the at least two video frames through the first two-dimensional convolutional layer, and the feature extraction results are convolved to generate a first spatial vector.

5. The action recognition method according to claim 4, wherein the step of extracting spatial features from the target video frame in the at least two video frames through the first two-dimensional convolutional layer, and performing convolution processing on the feature extraction results, includes: Spatial features are extracted from each video frame through the first two-dimensional convolutional layer, and the target video frame is determined from the at least two video frames based on the extraction results. The spatial features of the target video frame are convolved through the first two-dimensional convolutional layer to generate a first spatial vector.

6. The action recognition method according to claim 4, wherein performing temporal convolution processing on the video to be processed through the first two-dimensional convolutional layer comprises: A predetermined number of video frames to be processed are determined among the at least two video frames that are adjacent to the target video frame, and the corresponding optical flow image is determined based on the motion information of the target object contained in the target video frame and the video frames to be processed. Temporal features are extracted from the optical flow image through the first two-dimensional convolutional layer, and the feature extraction results are then subjected to convolution processing.

7. The action recognition method according to claim 2, wherein performing temporal convolution processing on the video to be processed includes: Extract the spatial features of the target video frame in the video to be processed, and perform convolution processing on the spatial features according to the preset convolution kernel and the first preset stride to generate a first spatial vector; Accordingly, the convolutional processing of the fusion result through the second two-dimensional convolutional layer includes: The fusion result is processed by the second two-dimensional convolutional layer according to the preset convolutional kernel and the second preset stride.

8. The action recognition method according to any one of claims 1 to 7, wherein the three-dimensional convolutional network comprises a first three-dimensional convolutional layer, a second three-dimensional convolutional layer, and a third three-dimensional convolutional layer; Accordingly, the step of performing convolution processing on the video to be processed and the first convolution vector through the three-dimensional convolutional network in the action recognition model to generate a corresponding second convolution vector includes: The first three-dimensional convolutional layer is used to extract temporal and spatial features from the video to be processed. The feature extraction results are then fused with the first convolutional vector, and the fused results are further processed by convolution to generate a first spatiotemporal vector. The first spatiotemporal vector is convolved by the second three-dimensional convolutional layer to generate the corresponding second spatiotemporal vector. The second spatiotemporal vector is convolved by the third three-dimensional convolutional layer to generate the corresponding second convolutional vector.

9. The action recognition method according to claim 1, wherein the classification network comprises a pooling layer and a fully connected layer; Accordingly, the step of processing the second convolutional vector through the classification network of the action recognition model to identify the action to be recognized, generating and outputting the corresponding action recognition result includes: The second convolutional vector is pooled using the pooling layer to generate the corresponding processing result. The action to be identified is identified by the fully connected layer and the processing result, and the corresponding action recognition result is generated and output.

10. An action recognition device, comprising: The input module is configured to input the video to be processed into the action recognition model, and to perform convolution processing on the video to be processed through the two-dimensional convolutional network in the action recognition model to generate a corresponding first convolutional vector. The video to be processed contains the action to be recognized of the target object, and the two-dimensional convolutional network is a convolutional network that extracts the spatial and temporal features of the video to be processed. The processing module is configured to perform convolution processing on the video to be processed and the first convolution vector through the three-dimensional convolutional network in the action recognition model to generate a corresponding second convolution vector. The three-dimensional convolutional network is a convolutional network that extracts the fusion result of temporal and spatial features of the video to be processed. The convolution processing on the video to be processed and the first convolution vector includes extracting temporal and spatial features from the video to be processed, fusing the feature extraction result with the first convolution vector, and performing convolution processing on the fusion result. The output module is configured to process the second convolutional vector through the classification network of the action recognition model to identify the action to be recognized, generate the corresponding action recognition result, and output it.

11. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the action recognition method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the action recognition method according to any one of claims 1 to 9.

Images