Video identification method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a video identification method and device, electronic equipment, a storage medium and a program product. The video identification method comprises the following steps: extracting spatial features of a current frame included in a video to obtain the spatial features of the current frame; based on the spatial features of the current frame, obtaining the fast time sequence features of the current frame through a fast branch of a fast and slow RNN formed by combining a fast and slow network and a recurrent neural network (RNN), and obtaining the slow time sequence features of the current frame through a slow branch of the fast and slow RNN; and obtaining an action recognition result of the current frame based on the fast time sequence feature and the slow time sequence feature of the current frame.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more specifically, to a video recognition method and apparatus, electronic device, storage medium, and program product for action recognition. Background Technology

[0002] Online Action Detection (OAD) is a challenging task in video understanding within the field of computer vision. Its goal is to accurately identify ongoing actions from a video stream. With the proliferation of interactive devices, OAD technology has significant application value in video surveillance, smart cockpits, and human-computer interaction. The main challenge of OAD is recognizing the current action based solely on historical information as video frames arrive, which requires learning long-range temporal dependencies.

[0003] In related technologies, OAD (Online Action Recognition) schemes have been proposed, such as 3D Convolutional Neural Networks (3D-CNN), video recognition methods based on Transformer networks, and video recognition methods based on Recurrent Neural Networks (RNNs). However, these technologies all have different problems when performing video recognition. For example, 3D-CNN uses a sliding window approach to take overlapping video segments as input and classifies the action types in the input video segments to obtain the video classification results. Therefore, 3D-CNN suffers from redundant computation and cannot guarantee the real-time performance required for online action recognition tasks. RNN-based video recognition methods are difficult to train, resulting in insufficient model accuracy. As for Transformer-based video recognition methods, the computational complexity of their attention mechanisms is high, making it difficult to guarantee real-time inference.

[0004] Therefore, there is a need for a video recognition technology that can improve the accuracy of action recognition while satisfying the real-time requirements of reasoning. Summary of the Invention

[0005] To at least address the aforementioned problems and / or drawbacks, embodiments of this disclosure provide a video recognition method and apparatus, electronic device, storage medium, and program product for motion recognition.

[0006] According to one aspect of the embodiments of this disclosure, a video recognition method is provided, comprising: obtaining spatial features of the current frame by performing spatial feature extraction on the current frame included in the video; obtaining fast temporal features of the current frame by using the fast branch of a fast-slow RNN formed by combining a fast-slow network and a recurrent neural network (RNN) and obtaining slow temporal features of the current frame by using the slow branch of the fast-slow RNN; and obtaining an action recognition result of the current frame based on the fast temporal features and the slow temporal features of the current frame.

[0007] Optionally, the steps of obtaining fast temporal features of the current frame and slow temporal features of the current frame based on the spatial features of the current frame by using the fast branch of the fast and slow RNN formed by combining fast and slow networks and recurrent neural networks (RNNs) and the slow branch of the fast and slow RNNs include: obtaining fast temporal features of the current frame by using the fast RNN computation unit of the fast branch and obtaining slow temporal features of the current frame by using the slow RNN computation unit of the slow branch, based on the temporal features of the previous frame that precedes the current frame in terms of timing and the spatial features of the current frame.

[0008] Optionally, a plurality of first frames including the current frame, which are processed by a fast RNN computation unit, have a first frame rate, and a plurality of second frames, which are processed by a slow RNN computation unit, have a second frame rate, wherein the first frame rate is greater than the second frame rate and the plurality of second frames are included in the plurality of first frames.

[0009] Optionally, the step of obtaining the fast temporal features of the current frame by using a fast RNN computation unit with fast branches, based on the temporal features of the previous frame and the spatial features of the current frame, includes: obtaining the fast temporal features of the current frame by performing computation processing using a fast RNN computation unit based on the fast and slow temporal features of the previous frame and the spatial features of the current frame.

[0010] Optionally, the step of obtaining the slow temporal features of the current frame by using a slow RNN computation unit with a slow branch, based on the temporal features of the previous frame preceding the current frame and the spatial features of the current frame, includes: determining whether the current frame is included in the plurality of second frames; in response to determining that the current frame is included in the plurality of second frames, performing computation processing by using a slow RNN computation unit to obtain the slow temporal features of the current frame based on the slow temporal features of the previous frame and the spatial features of the current frame; and in response to determining that the current frame is not included in the plurality of second frames, determining the slow temporal features of the previous frame as the slow temporal features of the current frame.

[0011] Optionally, the step of determining whether the current frame is included in the plurality of second frames includes: if the frame number of the current frame is divisible by the step size of the fast and slow RNN related to the number of frames processed by the skipped slow RNN computation units, then the current frame is determined to be included in the plurality of second frames, wherein the step size is determined based on the first frame rate and the second frame rate.

[0012] Optionally, the step of obtaining the action recognition result of the current frame based on the fast temporal features and slow temporal features of the current frame includes: fusing the fast temporal features and slow temporal features of the current frame to obtain the fused temporal features of the current frame; and performing action recognition of the current frame based on the fused temporal features of the current frame to obtain the action recognition result of the current frame.

[0013] Optionally, the step of performing action recognition on the current frame based on the fused temporal features of the current frame to obtain the action recognition result of the current frame includes: updating the fast temporal features and slow temporal features of the current frame based on the fused temporal features of the current frame; obtaining the fast action prediction result of the current frame by performing action prediction based on the updated fast temporal features of the current frame, and obtaining the slow action prediction result of the current frame by performing action prediction based on the updated slow temporal features of the current frame; and obtaining the action recognition result of the current frame by performing feature fusion and filtering on the fast action prediction result and the slow action prediction result of the current frame.

[0014] According to one aspect of the embodiments of this disclosure, a video recognition device is provided, comprising: a spatial feature acquisition module configured to obtain spatial features of a current frame by extracting spatial features from a current frame included in a video; a temporal feature acquisition module configured to obtain fast temporal features of the current frame based on the spatial features of the current frame by obtaining fast temporal features of the current frame through a fast branch of a fast-slow RNN formed by combining a fast-slow network and a recurrent neural network (RNN) and obtaining slow temporal features of the current frame through a slow branch of the fast-slow RNN; and an action recognition module configured to obtain an action recognition result of the current frame based on the fast temporal features and the slow temporal features of the current frame.

[0015] Optionally, the temporal feature acquisition module is configured to obtain the fast temporal features of the current frame based on the spatial features of the current frame by using the fast branch of a fast and slow RNN formed by combining fast and slow networks and a recurrent neural network (RNN) and obtaining the slow temporal features of the current frame by using the slow branch of the fast and slow RNN: based on the temporal features of the previous frame that precedes the current frame in terms of temporal sequence and the spatial features of the current frame, the fast temporal features of the current frame are obtained by using the fast RNN computation unit of the fast branch and the slow temporal features of the current frame are obtained by using the slow RNN computation unit of the slow branch.

[0016] Optionally, a plurality of first frames including the current frame, which are processed by a fast RNN computation unit, have a first frame rate, and a plurality of second frames, which are processed by a slow RNN computation unit, have a second frame rate, wherein the first frame rate is greater than the second frame rate and the plurality of second frames are included in the plurality of first frames.

[0017] Optionally, the temporal feature acquisition module is configured to obtain the fast temporal features of the current frame by using a fast RNN computation unit with fast branches, based on the temporal features of the previous frame preceding the current frame and the spatial features of the current frame in terms of temporal sequence: based on the fast and slow temporal features of the previous frame and the spatial features of the current frame, the fast temporal features of the current frame are obtained by performing computation processing using a fast RNN computation unit.

[0018] Optionally, the temporal feature acquisition module is configured to obtain the slow temporal features of the current frame by using a slow RNN computation unit with a slow branch, based on the temporal features of the previous frame preceding the current frame and the spatial features of the current frame, through the following operations: determining whether the current frame is included in the plurality of second frames; in response to determining that the current frame is included in the plurality of second frames, obtaining the slow temporal features of the current frame by performing computation processing using a slow RNN computation unit based on the slow temporal features of the previous frame and the spatial features of the current frame; and in response to determining that the current frame is not included in the plurality of second frames, determining the slow temporal features of the previous frame as the slow temporal features of the current frame.

[0019] Optionally, the timing feature acquisition module is configured to determine whether the current frame is included in the plurality of second frames by the following operation: if the frame number of the current frame is divisible by the step size of the fast and slow RNN related to the number of frames processed by the skipped slow RNN computation unit, then the current frame is determined to be included in the plurality of second frames, wherein the step size is determined based on the first frame rate and the second frame rate.

[0020] Optionally, the action recognition module is configured to obtain the action recognition result of the current frame based on the fast temporal features and slow temporal features of the current frame by performing feature fusion on the fast temporal features and slow temporal features of the current frame to obtain the fused temporal features of the current frame; and to perform action recognition of the current frame based on the fused temporal features of the current frame to obtain the action recognition result of the current frame.

[0021] Optionally, the action recognition module is configured to perform action recognition of the current frame based on the fused temporal features of the current frame through the following operations to obtain the action recognition result of the current frame: updating the fast temporal features and slow temporal features of the current frame based on the fused temporal features of the current frame; obtaining the fast action prediction result of the current frame by performing action prediction based on the updated fast temporal features of the current frame, and obtaining the slow action prediction result of the current frame by performing action prediction based on the updated slow temporal features of the current frame; and obtaining the action recognition result of the current frame by performing feature fusion and filtering on the fast action prediction result and the slow action prediction result of the current frame.

[0022] According to one aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the video recognition method as described above.

[0023] According to one aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the video recognition method as described above.

[0024] According to one aspect of the embodiments of this disclosure, a computer program product is provided, including computer-executable instructions, wherein the computer-executable instructions, when executed by at least one processor, implement the video recognition method as described above.

[0025] The embodiments of this disclosure can learn significant changes in action within low frame rate video sequences through slow branches and retain longer-term memories. Furthermore, fast branches combine continuous high frame rate actions with skipping low frame rate actions to effectively extract temporal information of actions, thereby achieving more accurate action recognition for each frame and improving action recognition precision. In addition, the embodiments of this disclosure can learn long-range temporal dependencies, reducing redundant computation and storage resource consumption by recognizing actions in the current frame of the video stream only based on historical information, and ensuring the real-time performance of online video recognition tasks. Moreover, the embodiments of this disclosure are easily supported by hardware due to the simplified structure of the SlowFast-RNN, formed by combining fast and slow networks with recurrent neural networks (RNNs), offering the advantage of easy deployment. Furthermore, the SlowFast-RNN used in the embodiments of this disclosure is easy to train, reducing the time cost of neural network training.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The above and other aspects, features and advantages of certain embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0028] Figure 1 This is an example illustrating an application scenario of the video recognition method according to embodiments of the present disclosure.

[0029] Figure 2 This is a flowchart illustrating a video recognition method according to an embodiment of the present disclosure.

[0030] Figure 3A This is a diagram illustrating the structure of a SlowFast-RNN according to an embodiment of the present disclosure.

[0031] Figure 3B This is a diagram illustrating the operation of a SlowFast-RNN according to an embodiment of the present disclosure.

[0032] Figure 4 This is a diagram illustrating the fusion and identification processing according to an embodiment of the present disclosure.

[0033] Figure 5 This is a block diagram illustrating a video recognition device according to an embodiment of the present disclosure. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0035] The terms “comprising” or “including” as used in this specification indicate the presence of the stated features, integrals, steps, operations, elements, components, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof. It should be understood that although the terms “first,” “second,” “third,” etc., are used to describe various information, the information should not be limited by these terms. These terms are used only to distinguish one type of information from another. For example, without departing from the scope of this disclosure, first information may be referred to as second information; and similarly, second information may be referred to as first information. As used herein, depending on the context, the term “in response to…” may be understood to mean “when,” “in,” or “if.”

[0036] Furthermore, in this disclosure, "at least one of..." refers to three parallel cases: "any one of...", "any combination of...", and "the whole of...". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" includes the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0037] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as understood based on the disclosure of this application and as commonly understood by one of ordinary skill in the art to which this disclosure pertains, and should not be interpreted ideally or overly formally. The use of the term “may” in relation to examples or embodiments (e.g., “...may include”) indicates the existence of at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto. Unless otherwise expressly defined, the singular form of a term also includes the plural form. Furthermore, the same reference numerals denote the same elements.

[0038] Before introducing the embodiments of this disclosure, some key terms and related technologies involved in the embodiments of this disclosure will be introduced.

[0039] The Slow-Fast Network (also known as the Fast Network) is a dual-path fast-slow model network used for video recognition. Similar to 3D-CNN, the Slow-Fast Network uses a CNN to classify video segments to obtain video classification results. The Slow-Fast Network employs a dual-path structure with a fast path and a slow path. The fast path captures temporal dynamic information, while the slow path captures spatial semantic information. The parameters of the two paths are independent.

[0040] A Recurrent Neural Network (RNN) is a type of neural network that uses sequential data as input and processes that data. It is well-suited for processing and predicting time dependencies and temporal information in sequential data. The RNN computational unit is the core basic unit in an RNN.

[0041] To at least address the shortcomings of the aforementioned related technologies, this disclosure provides a technique based on a SlowFast-RNN, which combines fast and slow networks with RNNs. Specifically, it combines the ideas of fast and slow networks, utilizing RNNs to implement the fast and slow branches of the fast and slow networks respectively, thus obtaining a SlowFast-RNN. Specifically, it provides a video recognition method and apparatus, electronic device, storage medium, and program product based on SlowFast-RNN for action recognition. The SlowFast-RNN-based technique provided in this disclosure can learn long-range temporal dependencies while ensuring real-time inference, thereby improving the accuracy of action recognition. Furthermore, the SlowFast-RNN model used in the SlowFast-RNN-based technique provided in this disclosure has a simple structure, is easy to train, and has high training efficiency.

[0042] The following reference Figures 1 to 5 This disclosure describes a SlowFast-RNN-based technique according to embodiments of the present disclosure.

[0043] Figure 1 This is an example illustrating an application scenario of the video recognition method according to embodiments of the present disclosure.

[0044] Reference Figure 1 The video recognition method according to embodiments of this disclosure can be applied to a video comprising multiple frames (or video frames) 101-1 to 101-t to obtain action recognition results 104-1 to 104-t for each frame included in the video, wherein the frame number t is a positive integer. According to embodiments of this disclosure, the video may include, but is not limited to, a streaming video stream comprising multiple video frames in a temporal sequence. That is, the video recognition method according to embodiments of this disclosure can be applied to OAD (On-Demand Awareness) of real-time transmitted video.

[0045] exist Figure 1 In the process, each of the multiple frames 101-1 to 101-t contains a person, and each of these frames 101-1 to 101-t can be used as input to obtain the action recognition result for each frame.

[0046] According to embodiments of this disclosure, spatial features of each input frame can first be extracted using a backbone network to obtain the spatial features of each frame as input to a SlowFast-RNN. Then, the SlowFast-RNN can be used to output the action recognition result for each frame based on the spatial features of each input frame. For example, in... Figure 1In this process, the Backbone network 102 can obtain the corresponding spatial features from each of the multiple frames 101-1 to 101-t, and input the spatial features of each frame into the SlowFast-RNN 103 to obtain the action recognition result for that frame. For example, the action recognition result 104-1 of frame 101-1 is "no gesture", which means that the person in frame 101-1 has no gesture. As another example, the action recognition result 104-(tn) of frame 101-(tn) is "affirmative", which means that the person in frame 101-(tn) has a "affirmative" meaning. Here, n is an integer greater than or equal to 1 and less than the frame number t (the determination of the value of n will be described below, and will not be elaborated here).

[0047] here, Figure 1 The Backbone network 102 with multiple inputs shown is only illustrative of how the Backbone network 102 can process multiple frames included in a video, and is not intended to limit the scope to relationships between the multiple inputs or between the multiple outputs. For example, the Backbone network 102 can perform separate feature extraction for each frame, and the feature extraction processing for each frame may be independent of each other. However, this disclosure is not limited thereto, and different types or more Backbone networks may be used to process multiple frames.

[0048] like Figure 1 As shown, the video recognition method according to the embodiments of the present disclosure can process real-time transmitted video streams. Each frame in the video stream has a frame number according to the time sequence. The video recognition method according to the embodiments of the present disclosure can be executed for each frame to obtain the action recognition result of each frame in the real-time transmitted video stream. Accordingly, the action recognition can be performed for each frame using a SlowFast-RNN with a simple structure.

[0049] However, Figure 1 The application scenarios shown are merely examples, and the application scenarios of the video recognition method according to the embodiments of this disclosure are not limited to these. For example, the application scenarios may include video surveillance, smart cockpits, and human-computer interaction.

[0050] The following reference Figure 2 A video recognition method according to embodiments of the present disclosure will be described in detail. Figure 2 This is a flowchart illustrating a video recognition method according to an embodiment of the present disclosure.

[0051] Reference Figure 2 In operation S210, the current frame included in the video can be accessed. Frame t Perform spatial feature extraction to obtain the spatial features of the current frame. Ft .

[0052] As an example, spatial features can be extracted separately for each of the multiple frames included in a video, obtaining spatial features F in the form of feature vectors for each frame. Since the spatial features are obtained individually for each frame, in this disclosure, spatial features can be understood as feature representations for a single frame image.

[0053] In addition, the frame number t can represent the current frame. Frame t The sequence number of multiple frames in the video, arranged according to time sequence, and reflecting the current frame. Frame t Relationship with time in the video, for example, the current frame Frame t It can be the frame corresponding to time t, and the current frame. Frame t-1 It can be the frame corresponding to time t-1.

[0054] As an example, the unit time interval between time t and time t-1 (i.e., the time interval between two adjacent frames) can be a time interval determined based on the video's frame rate, or it can be a time interval determined based on a preset frame sampling frequency for the video. For example, when the unit time interval is equal to the time interval between two adjacent frames based on the video's frame rate, the video recognition method according to embodiments of this disclosure can identify actions frame by frame. That is, the multiple first frames processed by the fast branch RNN computing unit of the SlowFast-RNN are all the frames of the original video. When the unit time interval is equal to the time interval between two adjacent frames determined based on a preset frame sampling frequency for the video, the video recognition method according to embodiments of this disclosure can identify actions on a portion of the frames sampled according to the frame sampling frequency. That is, the multiple first frames processed by the fast branch RNN computing unit are a portion of the frames obtained by sampling all the frames in the original video according to the frame sampling frequency. The relevant details of the computational processing performed by the fast branch RNN computing unit are described below, and will not be elaborated here. Furthermore, in this disclosure, for ease of description, regardless of whether the multiple first frames are all video frames of the original video or sampled video frames, their frame numbers are determined to be consecutive. That is, even if the multiple frames processed by the video recognition method are obtained by sampling, they are regarded as having consecutive frame numbers.

[0055] According to embodiments of this disclosure, it is possible to use a Backbone network (such as...) Figure 1The Backbone network 102 extracts spatial features from the frames. Specifically, a video comprising multiple frames can be understood as, for example, an image sequence in the temporal dimension. The Backbone network can convert the image sequence into a feature sequence with spatial features by extracting spatial features from each frame in the image sequence. This feature sequence may include multiple feature vectors with spatial features from multiple frames in the temporal dimension. Therefore, in this paper, spatial features... F It has the form of an eigenvector.

[0056] As an example, the backbone network may include or may be a two-dimensional convolutional neural network (2D-CNN), such as a deep residual network ResNet50. By using 2D-CNNs with relatively simple operators, the limitations of certain hardware platforms that may prevent them from supporting computational operations can be avoided, reducing subsequent deployment hassles. Furthermore, although this disclosure illustrates the use of 2D-CNNs for spatial feature extraction, it is not limited thereto; any suitable neural network or model can be used for feature extraction from video frames.

[0057] In this disclosure, the extracted spatial features will be used as SlowFast-RNN (such as, Figure 1 The computation of the SlowFast-RNN103 is processed to obtain temporal features H with temporal information. The obtained temporal features H can be in the form of feature vectors. Therefore, as a result of operation S210, the Backbone network (such as...) processes the input to obtain temporal features H with temporal information. Figure 1 The Backbone network 102 can be based on the current frame at time t of the input. Frame t To output the spatial features of the current frame F t And it can display the spatial features of the current frame. F t Input into SlowFast-RNN (such as, Figure 1 The SlowFast-RNN (103) is used to obtain temporal features H and action recognition results. That is, the Backbone network is used to obtain independent spatial features for each frame, and the SlowFast-RNN is used to obtain temporal (action) features associated with multiple frames.

[0058] In operation S220, spatial features of the current frame can be used as a basis. F t Fast temporal features of the current frame are obtained through the fast branches of SlowFast-RNN. The slow temporal features of the current frame are obtained through the slow branch of SlowFast-RNN. That is, the operation of SlowFast-RNN may include obtaining the temporal features of a frame, including fast temporal features and slow temporal features, from the spatial features of an input frame.

[0059] According to embodiments of this disclosure, a SlowFast-RNN can be a network obtained by combining a SlowFast network with an RNN. See below for further details. Figure 3A and Figure 3B The structure and operation of SlowFast-RNN according to embodiments of the present disclosure are described in detail. Figure 3A This is a diagram illustrating the structure of a SlowFast-RNN according to an embodiment of the present disclosure. Figure 3B This is a diagram illustrating the operation of a SlowFast-RNN according to an embodiment of the present disclosure.

[0060] According to embodiments of this disclosure, a SlowFast-RNN may include a Fast-RNN branch (also called a fast branch) as a fast path, a Slow-RNN branch (also called a slow branch) as a slow path, and a fusion and recognition unit. For example, as Figure 3A As shown, the fast branch in the SlowFast-RNN 300 may include at least one Fast-RNN computation unit (such as a Fast-RNN Cell, also called a Fast RNN unit or Fast-RNN cell) 310, wherein a plurality of first frames, including the current frame, have a first frame rate, and computation processing is performed by at least one Fast-RNN computation unit 310. The slow branch in the SlowFast-RNN 300 may include at least one Slow-RNN computation unit (such as a Slow-RNN Cell, also called a Slow RNN unit or Slow-RNN cell), wherein a plurality of second frames have a second frame rate and computation processing is performed by at least one Slow-RNN computation unit 320. Furthermore, the SlowFast-RNN 300 also includes a fusion and recognition unit 330.

[0061] According to embodiments of this disclosure, multiple frames including the current frame processed by the fast RNN computation unit 310 of the fast branch can be understood as multiple first frames with a first frame rate, and multiple frames processed by the slow RNN computation unit 320 of the slow branch can be understood as multiple second frames with a second frame rate. As an example, in the SlowFast-RNN 300, when the spatial features of multiple first frames are processed by the fast branch, only the spatial features of a subset of the multiple first frames (i.e., multiple second frames) are processed by the slow branch. This method of slow branching, which only processes the spatial features of a subset of frames, can be understood as frame skipping in the slow branch.

[0062] As an example, a second frame is obtained from every n first frames of multiple first frames corresponding to the fast branch (i.e., a second frame is sampled every n first frames). The slow branch only processes the spatial features of the obtained second frame without obtaining the spatial features of the remaining first frames. In this case, the first frame rate of the multiple first frames corresponding to the fast branch is high and the second frame rate of the multiple second frames corresponding to the slow branch is low.

[0063] According to embodiments of this disclosure, a SlowFast-RNN can be trained to obtain fast temporal features of a frame based on the spatial features F of the input frame. H fast and slow time series characteristics H slow The final output frame is the action recognition result, Out.

[0064] For example, such as Figure 3A As shown, the current frame that will be identified by the SlowFast-RNN 300 is... Frame t For the current frame Frame t The SlowFast-RNN 300 can be configured to obtain fast temporal features through fast branches and slow branches respectively. and slow time series characteristics And it is configured to use a fusion and recognition unit based on fast temporal features. and slow time series characteristics Obtain action recognition results Out t For example, at least one Fast-RNN computation unit 310 can be configured to be based on fast temporal features of the previous frame. and slow time series characteristics For the current input frame Frame t Spatial features F t Perform computational processing and output the fast temporal features of the current frame. At least one Slow-RNN computation unit 320 can be configured (when for the current frame of input) Frame t Spatial features F t When performing slow branch computation processing, it is based on the slow temporal features of the previous frame. For the current input frame Frame t Spatial features F t Perform computational processing and output the slow temporal characteristics of the frame. The fusion and recognition unit 330 can be configured to be based on the current frame. Frame t fast timing characteristics and slow time series characteristics Perform fusion and recognition to obtain action recognition results Out t .

[0065] In this disclosure, a single Fast-RNN computation unit 310 can be used to perform the same computational processing on the spatial features of each of a plurality of first frames, including the current frame. However, although only one Fast-RNN computation unit 310 is shown herein, this disclosure does not limit the type or number of Fast-RNN computation units, and different types or more Fast-RNN computation units may be used to perform computational processing on the spatial features of each first frame through cooperative operation.

[0066] As an example, Fast-RNN Cell 310 can be a gated recurrent unit (GRU), as described below. FastRNNCell ( ) can represent the computation method corresponding to GRU. Here, this disclosure does not limit the type of computation unit of Fast-RNN, and any other applicable computation unit can be used.

[0067] Furthermore, in this disclosure, a single Slow-RNN computation unit 320 can be used to perform the same computational processing on the spatial features of multiple second frames processed by the slow branch. However, although only one Slow-RNN computation unit 320 is shown here, this disclosure does not limit the number of Slow-RNN computation units; multiple Slow-RNN computation units can also be used to perform computational processing on the features of each frame processed by the slow branch through cooperative operation. Moreover, according to embodiments of this disclosure, depending on the actual application, the Fast-RNN computation unit may have the same or different computational methods as the Slow-RNN computation unit, and the Fast-RNN computation unit may have independent model parameters from the Slow-RNN computation unit.

[0068] As an example, the Slow-RNN Cell can be a GRU, as described below. SlowRNNCell ( ) can represent the computation method corresponding to GRU. As another example, a Slow-RNN Cell can be a convolutional GRU, as described below. SlowRNNCell ( ) can represent the computation method corresponding to ConvGRU.

[0069] In this disclosure, there may be a lot of redundant actions between the first frames (i.e., each first frame) that need to be processed in the fast branch, while the action amplitude varies greatly between the second frames that need to be processed in the slow branch. Therefore, the feature vector dimensions of the temporal features in the two branches may differ. To address this, Fast-RNN Cell 310 can be a GRU and Slow-RNN Cell 320 can be a ConvGRU, thereby compensating for the difference in feature vector dimensions through convolution processing. Therefore, different GRUs can also be used to implement the processing of the fast branch and the computation of the slow branch respectively.

[0070] This disclosure does not restrict the type of computational unit for Slow-RNN; any other suitable computational unit may be used.

[0071] In addition, see below for reference. Figure 4 The fusion and identification unit 300 according to an embodiment of the present disclosure will not be described in detail here.

[0072] The following reference Figure 3B The operation of the SlowFast-RNN 300 according to the fundamentally disclosed embodiments is described. For ease of understanding, in Figure 3B The image shows the process of SlowFast-RNN 300 processing multiple frames in the order of frame processing. In fact, Figure 3B The processing operation is performed by a SlowFast-RNN 300 that iteratively combines the processing results of the previous frame to process the current frame.

[0073] exist Figure 3B In this context, the spatial feature F of a frame can be represented as the spatial feature F of the nth frame. t-n The spatial features F of the (t-n+1)th frame t-n+1 The spatial features F of the current t-th frame t The fast and slow temporal characteristics of a frame can be represented as the fast temporal characteristics of the nth frame. and slow time series characteristics Fast temporal features of the (t-n+1)th frame and slow time series characteristics Fast temporal features of the current t-th frame and slow time series characteristics The action recognition result Out of frame t can be represented as the action recognition result Out of frame tn. t-n The action recognition result of frame t-n+1 is output. t-n+1 The action recognition result for the current frame t is Out. t wait.

[0074] As mentioned above, there are frame skips in SlowFast-RNN. This disclosure defines the parameter related to the number of frames skipped in slow branch computation as the step size of SlowFast-RNN. When using SlowFast-RNN, the value n of the step size Step can be determined first.

[0075] Specifically, multiple first frames (such as, Figure 1 The corresponding spatial features (such as, frames 101-1 to 101-t) in the data F 1 to F t The data are input into the fast branches of the SlowFast-RNN, respectively, to obtain the corresponding fast temporal features (such as...) of multiple frames through the fast RNN computation units. to The computation here, performed by the fast RNN computation unit, includes multiple first frames with a first frame rate, including the current frame. High rate Because multiple first frames are densely packed, the first frame rate... High rate This can be understood as a high frame rate. As an example, as mentioned above, when a video recognition method processes all frames in a video, the first frame rate... High rate It can be the frame rate of the original video (i.e., the video in operation S210), or the first frame rate when only sampled frames are processed instead of all frames in the video. High rate It can be the frame rate determined based on the frame sampling frequency.

[0076] Additionally, multiple first frames (such as, Figure 1 The spatial features of multiple discontinuous (or sampled from multiple first frames) second frames (frames 101-1 to 101-t) are input into the slow branch of the SlowFast-RNN to obtain the corresponding slow temporal features of these second frames through the slow RNN computation unit. The multiple second frames processed by the slow RNN computation unit here have a second frame rate. Slow rate Because these second frames are discontinuous (or sampled from multiple first frames), the second frame rate... Slow rate This can be understood as a low frame rate. For example, the second frame rate. Slow rate The value can be set based on actual conditions or experience and can be less than [a certain value]. High rate .

[0077] As an example, it can be based on the first frame rate. High rate Second frame rate Slow rate Determine the step size of the SlowFast-RNN. For example, the value of the step size n of the SlowFast-RNN can be determined based on the following equation (1):

[0078] in, High rate The first frame rate, Slow rate The second frame rate is n, and the step size is n. For example, when the result of dividing the first frame rate by the second frame rate is not an integer, the integer value of the result can be used as the value of n.

[0079] Based on a defined step size of the SlowFast-RNN, temporal features of multiple first frames can be obtained from spatial features of multiple first frames through the fast and slow branches of the SlowFast-RNN.

[0080] For example, refer to Figure 3B When n is determined to be 4 according to equation (1) and the frame number t of the current t-th frame is 15, the Fast-RNN computation unit 310 of the SlowFast-RNN 300 can be based on the spatial features of the first frame 101-1. F 1 The computational process is performed to determine the fast temporal features of frame 101-1. Simultaneously, the Slow-RNN computational unit 320 performs computational processing based on the slow temporal features of frame 101-1 to determine the slow temporal features of frame 101-1. Then, the Fast-RNN computational unit 310 can perform computational processing based on the spatial features of frame 201-2. F 2 The computation process is performed to determine the fast temporal features of frame 2 (101-2). Simultaneously, the Slow-RNN computation unit 320 skips the computation of the slow temporal features of frame 2 (101-2) and directly uses the slow temporal features of the previous frame as the slow temporal features of frame 2 (101-2). For ease of understanding, in... Figure 3B In the diagram, for frame 2 (101-2), the Slow-RNN computation unit 320 is not shown; instead, a linear representation is used to illustrate the direct processing of the slow temporal features obtained by the Slow-RNN unit 320 for frame 1 (101-1). Used for frame 2, 101-2 The process. For frame 3 (101-3), the Fast-RNN computation unit 310 can be based on the spatial features of frame 3 (101-3). F 3The computational process is performed to determine the fast temporal features of frame 3 (101-3). Simultaneously, the Slow-RNN computation unit 320 skips the computation of the slow temporal features of frame 3 (101-3) and directly uses the slow temporal features of the previous frame as the slow temporal features of frame 3 (101-3). However, for frame 4, the Fast-RNN computation unit 310 can base its calculations on the spatial features of frame 4 (101-4). F 4 The computational processing is performed to determine the fast temporal features of frame 4 (101-4), and the Slow-RNN computational unit 320 can be based on the spatial features of frame 4 (101-4). F 4 The computational processing is performed to determine the slow timing characteristics of frame 4 (101-4). Then, for frame 5 (101-5), the SlowFast-RNN 300 performs processing similar to that of frames 2 (101-2) and 3 (101-3).

[0081] As a result, the Fast-RNN computation unit 310 targets the spatial features from frame 101-1 to frame 150-15. F 1 to F 15 The computational processing is performed, while the Slow-RNN computational unit 320 only targets the spatial features of frame 1 (101-1), frame 4 (101-4), frame 8 (101-8), and frame 12 (101-12). F 1 , F 4 , F 8 and F 12 The computational processing is performed without performing computational processing on the remaining frames, and the slow temporal features of the previous frame for each remaining frame are determined as the slow temporal features of that remaining frame. However, the above processing is only shown as an example. For example, for frame 101-1, the Slow-RNN computation unit 320 may also skip the computational processing of the slow temporal features of frame 101-1 and directly use the slow temporal features of the previous frame. The slow temporal characteristics of frame 101-1 are identified. The specific processing of the fast and slow branches of SlowFast-RNN is described below.

[0082] According to embodiments of this disclosure, during inference of the SlowFast-RNN, the SlowFast-RNN can sequentially extract features for the spatial features of each frame used for action recognition. Specifically, for the current t-th frame, the SlowFast-RNN 300 can extract features by using the spatial features of the current t-th frame. F tAnd the temporal features (i.e., fast temporal features) of the preceding frame (i.e., the (t-1)th frame) that is temporally adjacent to the current t-th frame. and slow time series characteristics Obtain the temporal features (i.e., fast temporal features) of the current frame t. and slow time series characteristics ).

[0083] According to embodiments of this disclosure, operation S220 may include: based on the temporal characteristics of the previous frame preceding the current frame and the spatial characteristics of the current frame. F t Fast temporal features of the current frame are obtained by using Fast-RNN computation units with fast branches. The slow temporal features of the current frame are obtained by using the Slow-RNN computation unit with a slow branch. .

[0084] As an example, when the current frame is the first frame (i.e., t=1), the fast temporal features of the previous frame can be used. and slow time series characteristics All are set to 0, but this disclosure is not limited to this. The initial (i.e., the fast timing feature of the previous frame) can be arbitrarily set according to the actual situation. and slow time series characteristics .

[0085] The processing of Fast-RNN computation unit 310 and Slow-RNN computation unit 320 is described below.

[0086] According to embodiments of this disclosure, in a fast branch, based on the fast temporal features of the previous frame... and slow time series characteristics and the spatial features F of the current frame t The fast temporal features of the current frame can be obtained by performing computational processing using the Fast-RNN computation unit 310. .

[0087] For example, the fast temporal features of the current t-th frame can be obtained through equation (2). :

[0088] Where FastRNNCell() represents the computation method for extracting temporal features in the Fast-RNN computation unit, and Ft and It is the input of FastRNNCell(). This is the output of FastRNNCell().

[0089] During inference in SlowFast-RNN, fast branch computation is performed on the read frame based solely on the features of the current and previous frames, without using excessive historical information from other frames. This high frame rate of continuous action recognition reduces computational complexity and minimizes storage resource consumption of useless historical data (e.g., feature data from earlier historical frames). Therefore, SlowFast-RNN's fast branches can quickly obtain temporal feature data of frames with low computational and storage costs. Furthermore, the structure of SlowFast-RNN is simplified by repeatedly executing fast branch computation using only a single Fast-RNN computation unit without complex computation unit configurations. Moreover, the computational processing of Fast-RNN units effectively compensates for potential omissions in fast branches by considering historical slow temporal features obtained from slow branches, which contain more information.

[0090] Furthermore, in this disclosure, computational processing using Slow-RNN computational units is employed to learn significant changes in actions and retain longer-term memories.

[0091] According to embodiments of this disclosure, in a slow branch, based on the slow temporal features of the previous frame... and the spatial features of the current frame F t The slow temporal features of the current frame can be obtained by performing computational processing using the Slow-RNN computation unit 320. Alternatively, the computation of the Slow-RNN computation unit 320 can be skipped, and the computation can be based on the slow temporal features of the previous frame. Directly obtain the slow temporal features of the current frame The specific method for obtaining slow-series features can be determined based on whether the current frame is included in multiple second frames processed by the Slow-RNN computation unit 320.

[0092] Because slow branches involve frame skipping, in a slow branch, firstly, it is determined whether the slow branch computation should be performed on the current frame or skipped by determining whether the current frame is included in multiple second frames, and then the corresponding processing is performed. According to embodiments of this disclosure, the step of obtaining slow temporal features using the slow RNN computation unit of the slow branch may include: determining whether the current frame is included in multiple second frames; in response to determining that the current frame is included in multiple second frames, based on the slow temporal features of the previous frame... Spatial features of the current frame F t The slow temporal features of the current frame are obtained by performing computational processing using slow RNN computational units. In response to determining that the current frame is not included in multiple second frames, the slow temporal features of the previous frame are included. Determined as a slow temporal feature of the current frame .

[0093] According to embodiments of this disclosure, the step of determining whether the current frame is included in a plurality of second frames includes: if the frame number of the current frame is divisible by the step size of the fast and slow RNNs, which is related to the number of frames skipped in the computation of the slow RNN computation unit, then the current frame is determined to be included in a plurality of second frames, wherein the step size is determined based on a first frame rate and a second frame rate. That is, the slow branch computation is performed once every n frames, thereby achieving slow branch frame skipping.

[0094] For example, the slow temporal characteristics of the current frame can be obtained through equation (3). :

[0095] in, SlowRNNCell ( ) indicates the computation method used by the Slow-RNN computation unit to extract temporal features. F t and yes SlowRNNCell The input of ( ) yes SlowRNNCell The output of parentheses () indicates that the remainder is calculated.

[0096] The following describes example operations of Fast-RNN Cell 310 and Slow-RNN Cell 320 based on the above processing.

[0097] Assume the step size n of the SlowFast-RNN 300 is 4 and the frame number t is divisible by n. For frames 1 to 3, since their frame numbers are not divisible by 4, they are determined not to be included in multiple second frames; therefore, their slow temporal characteristics... to All were determined to be equal to For example, 0. Frame 4 is determined to be included in multiple second frames because its frame number 4 is divisible by 4; therefore, based on the spatial characteristics of frame 4... F 4 And the slow temporal features of the 3rd frame, which is the frame preceding the 4th frame. Processed through Slow-RNN Cell 320 SlowRNNCell ( )Sure Next, for frames 5 through 7, since their frame numbers are not divisible by 4, they are determined not to be included in multiple second frames; therefore, their slow temporal characteristics... to All were determined to be equal to the slow temporal features of the previous frame and were ultimately determined to be equal to the slow temporal features of the 4th frame. Similar to frame 4, frame 8 was determined to be included among multiple second frames, thus based on the spatial features of frame 8. F 8 And the slow temporal features of the 7th frame, which is the frame preceding the 8th frame. Processed through Slow-RNN Cell 320 SlowRNNCell ( )Sure Repeat the process similar to that described above until the computation is completed using Slow-RNN Cell 320. SlowRNNCell ( ) Based on the spatial features of the t-th frame F t And the slow temporal features of the (t-1)th frame preceding the t-th frame. Sure Furthermore, although not shown, the timing characteristics of frame t can also be used when frame t+1 is received.

[0098] Therefore, the slow-branch approach, through computational processing and frame skipping, obtains slow temporal features that supplement historical information into the temporal features of the current frame, recording more content. This allows for a longer time interval of the processed frames without increasing the amount of historical data, thus avoiding forgetting or overfitting problems. Furthermore, the slow-branch approach's computational processing only on a subset of frames reduces unnecessary redundant computations and inference time.

[0099] After obtaining the fast temporal features of the current frame and slow time series characteristics Then, fast temporal features can be analyzed. and slow time series characteristics Processing is performed to obtain the recognition results.

[0100] In operation S230, fast timing characteristics of the current frame can be used as a basis. and slow time series characteristics Obtain the action recognition result of the current frame. Out t According to embodiments of this disclosure, operation S230 may include feature fusion operation and action recognition operation.

[0101] According to embodiments of this disclosure, operation S230 may include: [using fast timing features of the current frame] and slow time series characteristics Perform feature fusion to obtain the fused temporal features of the current frame. ; Fusion temporal features based on the current frame Perform action recognition on the current frame and obtain the action recognition result for the current frame. Outt See below for reference. Figure 4 The operation S230 is described in detail.

[0102] Figure 4 This is a schematic diagram illustrating the fusion and identification processing according to an embodiment of the present disclosure.

[0103] Reference Figure 4 First, the fast temporal features of the current frame can be analyzed. and slow time series characteristics Feature fusion is performed to obtain fused temporal features. .

[0104] As an example, feature fusion may include: in operation S401, processing the fast temporal features of the current frame. Avg Pooling (AP) is used to obtain fast temporal features. 410, In operation S402, for slow timing characteristics Average pooling is used to obtain slow time-series features. 420. Here, the fast time-series features are obtained. 410 and the obtained slow time series features 420 can have the same dimension, that is, the average pooling operations S401 and S402 are used to unify the dimensions of the two time series features.

[0105] As an example, feature fusion may also include: in operation S403, obtaining the spliced ​​temporal feature 430 of the current frame by concatenating (Cat) the fast temporal feature 410 and the slow temporal feature 420.

[0106] As an example, feature fusion may also include: in operation S404, obtaining fused temporal features by using a neural network to reduce the dimensionality and fuse the concatenated temporal features 430. 440. Among them, fusion of temporal features 440 has the same dimension as fast time series feature 410 or slow time series feature 420 and can correspond to the residual between the two time series features.

[0107] exist Figure 4 In the diagram, the neural network used to operate S404 is shown as a multilayer perceptron (MLP) 404, but this disclosure is not limited thereto. Any suitable neural network model can be used, as long as it can obtain fused temporal features with the same dimension as the individual temporal features based on fast temporal features and slow temporal features.

[0108] For example, the feature fusion operation can be expressed as equation (4):

[0109] in, F ( ) can represent a nonlinear mapping method (such as a combination of linear + layer normalization + ReLU) used for feature fusion neural network model implementation. cat ( ) can represent splicing. and These are the fast temporal feature 410 and the slow temporal feature 420, respectively, after average pooling. yes F ( ) outputs fusion timing features 440.

[0110] According to embodiments of this disclosure, when fusion timing features are obtained... In action recognition operations, it is possible to rely solely on fused temporal features. Obtain the action recognition result of the current frame Out t For example, the prediction result can be obtained through equation (5):

[0111] in, predictor ( ) can represent the calculation method of the prediction head used for action recognition. It is the fused temporal feature of the input. P t yes predictor ( ) outputs the prediction results.

[0112] For example, prediction results P t It can be at least one probability for at least one predicted action, wherein the sum of at least one probability for at least one predicted action is 1.

[0113] According to embodiments of this disclosure, prediction results can be used as a basis. P t The predicted action with the highest probability is determined as the action to be identified for the current frame (i.e., the action identification result). Out t In this disclosure, predictions can be made using any applicable neural network model's prediction head, and no specific prediction method is limited herein.

[0114] In this way, action recognition based on fused temporal features that combine fast and slow temporal features can effectively improve the accuracy of action recognition. Therefore, the embodiments of this disclosure that consider fused temporal features have the technical effect of steadily improving accuracy.

[0115] According to another embodiment of this disclosure, such as Figure 4As shown, in the action recognition operation, the fast temporal features and slow temporal features of the current frame can be updated based on the fused temporal features of the current frame. The fast action prediction result of the current frame is obtained by performing action prediction based on the updated fast temporal features of the current frame, and the slow action prediction result of the current frame is obtained by performing action prediction based on the updated slow temporal features of the current frame. Finally, the action recognition result of the current frame is obtained by feature fusion and filtering of the fast action prediction result and the slow action prediction result of the current frame.

[0116] In this embodiment, for example, equation (6) can be used to base the fusion of temporal features. Update the fast time series features, and can be based on the fused time series features through equation (7). Slow-up time-series features:

[0117] in, It can represent fast temporal features that are being updated. It can represent slow temporal features that are being updated. It can represent the fast temporal features of the current frame after average pooling. This represents the slow temporal features of the current frame after average pooling.

[0118] In this embodiment, updated fast timing features can be... and updated slow time-series features The inputs are fed into the prediction head for action prediction; for example, it can be used... Figure 4 In Predictor fast 405, the fast-motion prediction result is obtained through equation (8). Equation (8), and can be used Figure 4 In Predictor slow 406, the slow motion prediction result is obtained through equation (9). :

[0119] in, ( ) can represent the calculation method of the prediction head used for fast branch in action recognition. ( ) can represent the computation method of the prediction head used for the slow branch of action recognition. Prediction results and They are ( )and ( ) outputs the prediction results. (Compared to the prediction results) P t similar, and These can be at least one probability for at least one predicted action, wherein the sum of at least one probability for at least one predicted action in each branch is 1.

[0120] Then, as an example, feature fusion may also include: in operation S404, obtaining the final predicted output by max pooling based on the outputs corresponding to the fast branch and the outputs corresponding to the slow branch.

[0121] For example, the output of the prediction for the current frame can be obtained through equation (10). Pt :

[0122] in, max_pooling ( ) can represent max pooling on the input.

[0123] Finally, prediction results can be based on the current frame. P t Determine the action currently being identified for frame t. As mentioned above, this can be based on the prediction results. P t The predicted action with the highest probability is determined as the action recognized in the current frame (i.e., the action recognition result). Out t In this disclosure, predictions can be made using the prediction head of any applicable neural network model, and no specific prediction calculation method is limited herein.

[0124] According to embodiments of this disclosure, by preserving the ability to predict independently of the two branches and fusing the prediction results of the two branches, high frame rate continuous motion and low frame rate skipping motion can be combined to further improve prediction accuracy.

[0125] In addition, SlowFast-RNN or neural network models can be provided through training or learning.

[0126] Here, "providing through learning" refers to obtaining a SlowFast-RNN or neural network model with desired characteristics by applying a learning algorithm to multiple learning data sets. This learning can be performed within the apparatus or electronic device itself, in which the SlowFast-RNN or neural network model according to the embodiment is executed, and / or can be implemented via a separate server / system.

[0127] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using multiple learning data sets to enable, allow, or control the target device to make determinations or predictions. Examples of such learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0128] To reduce the hardware requirements for training SlowFast-RNN models and / or save training time, this disclosure also proposes to divide the input video used for training into multiple slices and supervise or process only a portion of the frames in each slice to reduce model training time.

[0129] According to embodiments of this disclosure, the number of partial frames in each slice can be limited to the value n of the step size Step of the SlowFast-RNN, for example, only n frames in each slice are determined as multiple first frames.

[0130] Specifically, during the training of the SlowFast-RNN model, the video can be divided into multiple slices of a fixed length, and only the last n consecutive frames in each slice are taken as input frames for the video recognition method. Here, the frame can be the step size value determined as described above. Therefore, according to embodiments of this disclosure, the steps for training the SlowFast-RNN include: dividing the input training video into multiple slices comprising a fixed number of frames; and performing the video recognition method described above on the last n frames of each of the multiple slices to train the SlowFast-RNN.

[0131] For example, during training, in addition to adjusting the specific model parameters of SlowFast-RNN based on the final action recognition result Out, the prediction head output of the fast branch described above can also be used. and the prediction head output of the slow branch Predicted data is used to assist in supervision during training.

[0132] In this way, by training the SlowFast-RNN model by processing only a subset of valid frames of the training video during training, the limitations on the physical devices (such as video memory) used to implement the video recognition method and the training difficulty can be reduced, effectively saving training time.

[0133] Furthermore, the developers of this disclosure have used publicly available datasets to verify the effectiveness of the video recognition method according to embodiments of this disclosure.

[0134] Specifically, the researchers used traffic control gesture datasets and the NvGesture-Oonline gesture dataset built based on NvGesture to verify the effectiveness of the video recognition method according to the embodiments of this disclosure. The evaluation metric was the average accuracy of the class (mAP). The batch size used was 32, the window size of the video slice was 128, and the initial learning rate was 0.0001, which dynamically decayed to 0.00001 as the training progressed in a cosine decay manner.

[0135] Table 1 below shows the test results of the video recognition method (Ours) of this disclosure and other related techniques on the THUMOS14 dataset. Since the operations on the THUMOS14 dataset are all feature-based, the same spatial feature extraction is used in Table 1, with only the SlowFast-RNN trained. Furthermore, for fairness, both the video recognition method of this disclosure and the MiniROAD's cells use GRU.

[0136] Table 1

[0137] As can be seen from Table 1, the video recognition method disclosed herein has the best mAP and also performs well in terms of parameter size and computational power.

[0138] Table 2 below shows the test results of the video recognition method (Ours) of this disclosure and other related techniques on the traffic control gesture dataset and the NvGesture-Oonline gesture dataset.

[0139]

[0140] In Table 2, “SlowFastGRU” indicates the test results of the video recognition method of this disclosure using SlowFast-RNN with GRU and obtaining action recognition results only based on fused temporal features (i.e., in the manner of Equation (5)). “Ours” indicates the test results of the video recognition method of this disclosure using SlowFast-RNN with temporal features updated by two branches to obtain action recognition results. As can be seen from Table 2, the video recognition method of this disclosure still has the best mAP in terms of testing on these two datasets, and the video recognition method using fast and slow temporal features updated based on fused temporal features can further improve the prediction accuracy of the model.

[0141] Furthermore, simply increasing the number of RNN layers does not improve performance; in fact, it may lead to a decrease in accuracy due to increased training difficulty. The SlowFast-RNN structure of this disclosed video recognition method, by fusing the results of RNNs at different frame rates, avoids increasing training difficulty while increasing the number of RNN layers, and delivers stable performance improvements on various types of online action recognition datasets.

[0142] Table 3 below shows the test results of the video recognition method of this disclosure for the NvGesture-Oonline gesture dataset in terms of step size.

[0143]

[0144] As shown in Table 3, when n takes any value from 2 to 4, the model performance of SlowFast-RNN is significantly improved. Furthermore, as n increases, the effect of fast branches gradually degenerates, and the model performance of SlowFast-RNN degenerates to be similar to that of MiniROAD.

[0145] In summary, the video recognition method according to the embodiments of this disclosure can learn significant changes in action in low frame rate video sequences through slow branches and retain longer-term memories, and combine continuous actions at high frame rates and jumping actions at low frame rates in fast branches to effectively extract the temporal information of actions, thereby achieving more accurate action recognition for each frame and improving the accuracy of action recognition.

[0146] Furthermore, the video recognition method according to the embodiments of this disclosure learns long-range temporal dependencies, reduces redundant computation and storage resource consumption by recognizing the action of the current frame in the video stream only based on historical information, and can guarantee the real-time performance of online video recognition tasks.

[0147] Furthermore, the video recognition method according to the embodiments of this disclosure is easy to obtain hardware support due to the simplified structure of SlowFast-RNN, and has the advantage of easy deployment.

[0148] Furthermore, the SlowFast-RNN used in the video recognition method according to the embodiments of this disclosure, by fusing the RNN calculation results at different frame rates, can avoid increasing the training difficulty while deepening the number of RNN layers. It can bring stable performance improvement on different types of online behavior recognition datasets. Moreover, the SlowFast-RNN in this disclosure is easy to train and can reduce the time cost of neural network training.

[0149] Figure 5 This is a block diagram illustrating a video recognition device according to an embodiment of the present disclosure.

[0150] Reference Figure 5 The video recognition device 500 may include a spatial feature acquisition module 510, a temporal feature acquisition module 520, and an action recognition module 530.

[0151] According to embodiments of this disclosure, the spatial feature acquisition module 510 can be configured to obtain the spatial features of the current frame by extracting spatial features from the current frame included in the video; the temporal feature acquisition module 520 can be configured to obtain the fast temporal features of the current frame based on the spatial features of the current frame by obtaining the fast temporal features of the current frame through the fast branch of the fast and slow RNN formed by combining fast and slow networks and recurrent neural networks (RNNs) and obtaining the slow temporal features of the current frame through the slow branch of the fast and slow RNNs; the action recognition module 530 can be configured to obtain the action recognition result of the current frame based on the fast and slow temporal features of the current frame.

[0152] In other words, the spatial feature acquisition module 510 can be configured to perform the operation corresponding to operation S210, the temporal feature acquisition module 520 can be configured to perform the operation corresponding to operation S220, and the action recognition module 530 can be configured to perform the operation corresponding to operation S230. Therefore, the details of the video recognition method 200 described above can be applied to various operations of the video recognition device 500.

[0153] According to embodiments of this disclosure, the temporal feature acquisition module 520 can be configured to obtain fast temporal features of the current frame based on the spatial features of the current frame by using the fast branch of a fast-slow RNN formed by combining fast and slow networks and a recurrent neural network (RNN) and obtaining slow temporal features of the current frame by using the slow branch of the fast-slow RNN: based on the temporal features of the previous frame that precedes the current frame in terms of temporal sequence and the spatial features of the current frame, the fast temporal features of the current frame are obtained by using the fast RNN computation unit of the fast branch and the slow temporal features of the current frame are obtained by using the slow RNN computation unit of the slow branch.

[0154] According to embodiments of this disclosure, a plurality of first frames including the current frame are processed by a fast RNN computing unit and have a first frame rate, and a plurality of second frames are processed by a slow RNN computing unit and have a second frame rate, wherein the first frame rate is greater than the second frame rate and the plurality of second frames are included in the plurality of first frames.

[0155] According to an embodiment of this disclosure, the temporal feature acquisition module 520 can be configured to obtain the fast temporal features of the current frame by using a fast RNN computation unit with fast branches, based on the temporal features of the previous frame preceding the current frame and the spatial features of the current frame in terms of timing: based on the fast and slow temporal features of the previous frame and the spatial features of the current frame, the fast temporal features of the current frame are obtained by performing computation processing using a fast RNN computation unit.

[0156] According to embodiments of this disclosure, the timing feature acquisition module 520 can be configured to obtain the slow timing features of the current frame by using a slow RNN computation unit with a slow branch, based on the timing features of the previous frame preceding the current frame and the spatial features of the current frame, through the following operations: determining whether the current frame is included in a plurality of second frames; in response to determining that the current frame is included in a plurality of second frames, obtaining the slow timing features of the current frame by performing computation processing using a slow RNN computation unit based on the slow timing features of the previous frame and the spatial features of the current frame; and in response to determining that the current frame is not included in a plurality of second frames, determining the slow timing features of the previous frame as the slow timing features of the current frame.

[0157] According to an embodiment of this disclosure, the timing feature acquisition module 520 can be configured to determine whether the current frame is included in a plurality of second frames by the following operation: if the frame sequence number of the current frame is divisible by the step size of the fast and slow RNNs related to the number of frames processed by the skipped slow RNN computation units, then the current frame is determined to be included in a plurality of second frames, wherein the step size is determined based on the first frame rate and the second frame rate.

[0158] According to an embodiment of this disclosure, the action recognition module 530 can be configured to obtain the action recognition result of the current frame based on the fast temporal features and slow temporal features of the current frame through the following operations: by performing feature fusion on the fast temporal features and slow temporal features of the current frame to obtain the fused temporal features of the current frame; and by performing action recognition of the current frame based on the fused temporal features of the current frame to obtain the action recognition result of the current frame.

[0159] According to embodiments of this disclosure, the action recognition module 530 can be configured to perform action recognition of the current frame based on the fused temporal features of the current frame through the following operations to obtain the action recognition result of the current frame: updating the fast temporal features and slow temporal features of the current frame based on the fused temporal features of the current frame; obtaining the fast action prediction result of the current frame by performing action prediction based on the updated fast temporal features of the current frame, and obtaining the slow action prediction result of the current frame by performing action prediction based on the updated slow temporal features of the current frame; and obtaining the action recognition result of the current frame by performing feature fusion and filtering on the fast action prediction result and the slow action prediction result of the current frame.

[0160] As already referenced above Figures 1 to 4 The description provides details on the specific operation of each module of the video recognition device 500, so it will not be described in detail here.

[0161] Furthermore, it should be understood that the various modules in the video recognition device 500 according to embodiments of this disclosure can be implemented as hardware components and / or software components. Those skilled in the art can implement the various modules, for example, using a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), based on the processes performed by each defined module.

[0162] Furthermore, according to embodiments of this disclosure, this disclosure also provides an electronic device, including: at least one processor; at least one memory storing computer-executable instructions, wherein, when executed by the at least one processor, the computer-executable instructions cause the at least one processor to perform the video recognition method as described above.

[0163] According to embodiments of this disclosure, the electronic device may be a PC computer, tablet device, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, the electronic device is not necessarily a single electronic device, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0164] In electronic devices, processors may include central processing units (CPUs), graphics processing units (GPUs), programmable logic devices, dedicated processor systems, microcontrollers, or microprocessors. By way of example and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0165] The processor can execute instructions or code stored in memory, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transport protocol.

[0166] Memory can be integrated with the processor; for example, RAM or flash memory can be housed within an integrated circuit microprocessor. Alternatively, memory can comprise a separate device, such as an external disk drive, storage array, or other storage device that can be used by any database system. Memory and processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the memory.

[0167] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device may be interconnected via buses and / or networks.

[0168] Furthermore, according to embodiments of this disclosure, a computer program product is also provided, including computer-executable instructions, wherein the computer-executable instructions, when executed by at least one processor, implement the video recognition method as described above.

[0169] Furthermore, according to embodiments of this disclosure, a computer-readable storage medium is also provided, wherein when instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the video recognition method as described above.

[0170] According to embodiments of this disclosure, examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in electronic devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0171] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0172] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the claims.

Claims

1. A video recognition method, comprising: The spatial features of the current frame are obtained by extracting spatial features from the current frame included in the video. Based on the spatial features of the current frame, the fast temporal features of the current frame are obtained through the fast branch of the fast-slow RNN formed by combining fast and slow networks and recurrent neural networks (RNNs), and the slow temporal features of the current frame are obtained through the slow branch of the fast-slow RNNs. Based on the fast and slow temporal features of the current frame, the action recognition result of the current frame is obtained.

2. The video recognition method according to claim 1, wherein, Based on the spatial features of the current frame, the steps of obtaining the fast temporal features of the current frame through the fast branch of the fast and slow RNN (formed by combining fast and slow networks and recurrent neural networks) and obtaining the slow temporal features of the current frame through the slow branch of the fast and slow RNN include: Based on the temporal features of the previous frame and the spatial features of the current frame, the fast temporal features of the current frame are obtained by using a fast RNN computation unit with a fast branch, and the slow temporal features of the current frame are obtained by using a slow RNN computation unit with a slow branch.

3. The video recognition method according to claim 2, wherein, The computational processing performed by the fast RNN computational unit includes multiple first frames, including the current frame, having a first frame rate, and multiple second frames, processed by the slow RNN computational unit, having a second frame rate. The first frame rate is greater than the second frame rate, and the plurality of second frames are included in the plurality of first frames.

4. The video recognition method according to claim 3, wherein, The steps for obtaining the fast temporal features of the current frame by using fast-branching fast RNN computation units, based on the temporal features of the previous frame preceding the current frame and the spatial features of the current frame, include: Based on the fast and slow temporal features of the previous frame and the spatial features of the current frame, the fast temporal features of the current frame are obtained by performing computational processing using a fast RNN computational unit.

5. The video recognition method according to claim 3, wherein, Based on the temporal features of the previous frame (which precedes the current frame) and the spatial features of the current frame, the steps to obtain the slow temporal features of the current frame using slow RNN computation units with slow branches include: Determine whether the current frame is included in the plurality of second frames; In response to determining that the current frame is included in the plurality of second frames, the slow temporal features of the current frame are obtained by performing computational processing using a slow RNN computational unit based on the slow temporal features of the previous frame and the spatial features of the current frame. In response to determining that the current frame is not included in the plurality of second frames, the slow timing feature of the previous frame is determined as the slow timing feature of the current frame.

6. The video recognition method according to claim 5, wherein, The steps of determining whether the current frame is included in the plurality of second frames include: The step size of the fast and slow RNNs is determined based on the first frame rate and the second frame rate, which is related to the number of frames that are skipped in the computation of the slow RNN computation unit. If the frame number of the current frame is divisible by the step size of the fast and slow RNNs, which is related to the number of frames processed by the slow RNN computation units, then the current frame is determined to be included in the plurality of second frames, wherein the step size is determined based on the first frame rate and the second frame rate.

7. The video recognition method according to claim 1, wherein, The steps to obtain the action recognition result of the current frame based on the fast and slow temporal features of the current frame include: By fusing the fast and slow temporal features of the current frame, the fused temporal features of the current frame are obtained. Action recognition of the current frame is performed based on the fused temporal features of the current frame to obtain the action recognition result of the current frame.

8. The video recognition method according to claim 7, wherein, The steps for performing action recognition on the current frame based on the fused temporal features of the current frame to obtain the action recognition result of the current frame include: Based on the fused temporal features of the current frame, update the fast temporal features and slow temporal features of the current frame; The fast motion prediction result of the current frame is obtained by performing action prediction based on the updated fast temporal features of the current frame, and the slow motion prediction result of the current frame is obtained by performing action prediction based on the updated slow temporal features of the current frame. The action recognition result of the current frame is obtained by fusing and filtering the fast motion prediction results and slow motion prediction results of the current frame.

9. A video recognition device, comprising: The spatial feature acquisition module is configured to obtain the spatial features of the current frame by performing spatial feature extraction on the current frame included in the video; The temporal feature acquisition module is configured to obtain fast temporal features of the current frame based on the spatial features of the current frame, through the fast branch of a fast-slow RNN formed by combining fast and slow networks and a recurrent neural network (RNN), and through the slow branch of the fast-slow RNN to obtain slow temporal features of the current frame. The action recognition module is configured to obtain the action recognition result of the current frame based on the fast and slow temporal features of the current frame.

10. An electronic device, comprising: At least one processor; At least one memory that stores computer-executable instructions. Wherein, when the computer-executable instructions are executed by the at least one processor, the at least one processor causes the at least one processor to perform the video recognition method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, wherein, When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the video recognition method as described in any one of claims 1 to 8.

12. A computer program product comprising computer-executable instructions, wherein, When the computer-executable instructions are executed by at least one processor, they implement the video recognition method according to any one of claims 1 to 8.

Images