Action recognition method based on deep residual network and related products

Abstract

Provided are a motion recognition method based on a deep residual network and related products. A first convolutional module in at least one convolutional module receives a video clip as input. At least one convolutional module is traversed by performing the following operations: for a current convolutional module, processing the input using a one-dimensional filter to obtain motion-related features and processing the input using a two-dimensional filter to obtain appearance-related features; shifting the motion-related features by one step along the time dimension, subtracting the shifted motion-related features from the motion-related features to obtain residual features; obtaining the output of the current convolutional module as the input of a next convolutional module until the last convolutional module in the at least one convolutional module is traversed. At least one motion included in the video clip is recognized based on the output of the last convolutional module in the at least one convolutional module. Thus, the model size is reduced and the calculation cost is lowered.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of neural networks, and in particular, to a motion recognition method based on a deep residual network and related products. BACKGROUND

[0002] The resurgence of convolutional neural networks (CNNs) and large labeled datasets has led to unprecedented progress in image classification using end-to-end trainable networks. However, human motion recognition based on videos cannot be achieved based on CNN features alone. How to effectively model temporal information, i.e., identify the correlation and causality of time, is a fundamental challenge.

[0003] Currently, there is a classical research branch that focuses on modeling motion through manually designed optical flow, and a two-stream method that uses optical flow modality and red-green-blue (RGB) separately in individual streams is one of the most successful architectures. However, optical flow is computationally expensive. SUMMARY

[0004] Embodiments provide a motion recognition method based on a deep residual network and related products to reduce the size of a model of a deep residual network for human motion recognition based on videos and reduce the computational cost.

[0005] In a first aspect, a motion recognition method based on a deep residual network is provided. The motion recognition method based on a deep residual network is applied to a deep residual network system comprising at least one convolutional module, each of the at least one convolutional module comprising at least one first convolutional layer, each of the at least one first convolutional layer having at least one one-dimensional filter and at least one two-dimensional filter, and the method comprises the following. A first convolutional module of the at least one convolutional module receives a video segment as an input. The at least one convolutional module is traversed by performing the following operations: for a current convolutional module, processing the input using the one-dimensional filter to obtain motion-related features and processing the input using the two-dimensional filter to obtain appearance-related features; shifting the motion-related features along a time dimension by one step, subtracting the shifted motion-related features from the motion-related features to obtain residual features; obtaining an output of the current convolutional module based on the appearance-related features, the motion-related features, and the residual features; and taking the output of the current convolution as an input of a next convolutional module until a last convolutional module of the at least one convolutional module is traversed. At least one motion included in the video segment is recognized based on the output of the last convolutional module of the at least one convolutional module.

[0006] In a second aspect, an action recognition apparatus based on a deep residual network is provided. The action recognition apparatus based on the deep residual network is applied to a deep residual network system comprising at least one convolutional module, each of the at least one convolutional module comprising at least one first convolutional layer, each of the at least one first convolutional layer having at least one one-dimensional filter and at least one two-dimensional filter, the apparatus comprising a receiving unit, a processing unit, and a recognizing unit. The receiving unit is configured to receive a video segment as an input at a first convolutional module of the at least one convolutional module. The processing unit is configured to traverse the at least one convolutional module by performing the following operations: for a current convolutional module, processing the input using the one-dimensional filter to obtain motion-related features and processing the input using the two-dimensional filter to obtain appearance-related features; shifting the motion-related features by one step along a time dimension, subtracting the shifted motion-related features from the motion-related features to obtain residual features; obtaining an output of the current convolutional module based on the appearance-related features, the motion-related features, and the residual features; and taking the output of the current convolutional module as an input of a next convolutional module until a last convolutional module of the at least one convolutional module is traversed. The recognizing unit is configured to recognize at least one action included in the video segment based on the output of the last convolutional module of the at least one convolutional module.

[0007] In a third aspect, a terminal device is provided, which includes a processor and a memory storing one or more programs. The one or more programs are configured to be executed by the processor, and include instructions for performing some or all of the operations in the method described in the first aspect.

[0008] In a fourth aspect, a non-transitory computer-readable storage medium is provided, which is configured to store a computer program for electronic data exchange. The computer program includes instructions for performing some or all of the operations in the method described in the first aspect.

[0009] In a fifth aspect, a computer program product is provided, which includes a non-transitory computer-readable storage medium storing a computer program. The computer program can make a computer perform some or all of the operations in the method described in the first aspect.

[0010] In the embodiments of the present application, a new deep residual network system is provided. The deep residual network system includes at least one convolution module, each of the at least one convolution module includes at least one first convolution layer, each of the at least one first convolution layer has at least one one-dimensional filter and at least one two-dimensional filter, and the deep residual network-based action recognition method is applied to the deep residual network system. The first convolution module in the at least one convolution module receives a video segment as input. The at least one convolution module is traversed by performing the following operations: for a current convolution module, processing the input using a one-dimensional filter to obtain motion-related features and processing the input using a two-dimensional filter to obtain appearance-related features; shifting the motion-related features by one step along the time dimension, subtracting the shifted motion-related features from the motion-related features to obtain residual features; obtaining the output of the current convolution module based on the appearance-related features, the motion-related features, and the residual features; taking the output of the current convolution module as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed. At least one action included in the video segment is recognized based on the output of the last convolution module in the at least one convolution module. Therefore, in the present application, a new convolution module that can be regarded as a pseudo three-dimensional convolution module is proposed, in which the standard 3D filter in the related art is decoupled to form parallel two-dimensional spatial filters and one-dimensional temporal filters. By using separable two-dimensional convolution and one-dimensional convolution instead of three-dimensional convolution, the model size and the calculation cost are greatly reduced. In addition, the 2D convolution and the 1D convolution are placed in different paths, so that different modeling can be performed for the appearance-related features and the motion-related features. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0012] Figure 1 is a schematic diagram of an RGB frame example (top) and a residual frame example (bottom).

[0013] Figure 2 is a schematic diagram of an exemplary detailed design of a deep residual network.

[0014] Figure 3 is a schematic flowchart of a deep residual network-based action recognition method according to an embodiment.

[0015] Figure 4is a schematic flow chart of a deep residual network based action recognition method according to an embodiment.

[0016] Figure 5 is a schematic diagram of an exemplary detailed design of a convolution module.

[0017] Figure 6 is a schematic structural diagram of a deep residual network based action recognition apparatus according to an embodiment.

[0018] Figure 7 is a schematic structural diagram of a terminal device according to an embodiment. DETAILED DESCRIPTION

[0019] In order to enable persons skilled in the art to better understand the solutions of the embodiments, the technical solutions in the embodiments will be clearly and completely described below with reference to the drawings in the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments, all other embodiments obtained by persons skilled in the art without creative efforts should fall within the protection scope of the present application.

[0020] The terms "first", "second", and "third" and the like in the description and claims of the present application and the drawings are used to distinguish different objects, not to describe a particular order. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0021] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor does it necessarily refer to a particular embodiment that is preferred over other embodiments. It will be explicitly understood by one of ordinary skill in the art that the embodiments described herein can be combined with other embodiments.

[0022] The terminal device related to the embodiments of the present application can include various handheld devices, vehicle-mounted devices, wearable devices, computing devices or other processing devices connected to a wireless modem, and various forms of user equipment (UE), mobile stations (MS), mobile terminals, etc. For the convenience of description, the above-mentioned devices are collectively referred to as terminal devices.

[0023] In order to facilitate better understanding of the embodiments of the present application, the related art involved in the present application will be briefly introduced below.

[0024] The resurgence of convolutional neural networks (CNNs) and large labeled datasets has enabled unprecedented progress in image classification using end-to-end trainable networks. However, human action recognition based on videos cannot be achieved based on CNN features alone. How to effectively model temporal information, i.e., identify the relevance and causality of time, is a fundamental challenge. There is a classic branch of research that focuses on modeling motion through manually designed optical flow. In the context of deep learning, the two-stream method that uses the optical flow modality and the RGB modality in separate streams is one of the most successful architectures. However, considering that the optical flow has a high computational cost, and the two-stream method that uses the optical flow modality and the RGB modality in separate streams cannot be learned end-to-end with the optical flow, this architecture is not satisfactory in terms of method.

[0025] In the present application, residual frames, i.e., the differences between adjacent RGB frames, are proposed as an optional "lightweight" motion representation, which are combined with the RGB modality for human action recognition based on videos. The reason why residual frames can be used together with the RGB modality for human action recognition based on videos can be expressed as follows. On the one hand, adjacent RGB frames largely share the information of static objects and background information, so residual frames generally mainly retain motion-specific features, such as Figure 1 as shown. Figure 1 An example of an RGB frame (top) and an example of a residual frame (bottom) are shown. As can be seen from Figure 1 , the RGB frame contains rich appearance information, while the residual frame mainly retains significant motion information. On the other hand, compared with other motion representations such as optical flow, the computational cost of residual frames can be negligible.

[0026] With the recent trend of developing three-dimensional (3D) convolutional models for video classification, a new and effective convolutional module is provided in the present application, which can be regarded as a pseudo-3D convolutional module, in which the original 3D convolution is decoupled into 2D and 1D convolutions. In addition, in order to further enhance the motion features and reduce the computational cost, the residual information in the feature space, i.e., the residual features representing the differences between the CNN features of the time-adjacent frames, can be utilized. In addition, the self-attention mechanism can also be used to recalibrate the appearance-related features and the motion-related features based on the importance of the final task, so as to further reduce the model size and the computational cost, and avoid the influence of the features that are not important to the final task on the calculation accuracy of the deep residual network system.

[0027] The action recognition method based on deep residual network and related products provided in the present application have high computing efficiency and improved performance. Specifically, the proposed residual frame (or residual feature) is a lightweight alternative compared to other motion representations (such as optical flow), and the new convolution module also helps to significantly reduce the computational cost. In addition, experiments have shown that using residual frames can significantly improve the accuracy of action recognition, which will be described in detail below.

[0028] In order to better understand the concept of residual features by those skilled in the art, the concept of residual frames is first introduced.

[0029] Suppose a video segment x ∈ R T×H×W×C where T, H, W represent the length, height and width of each frame respectively, and C represents the number of channels. The residual frame can be formed by subtracting the reference frame from the expected frame , where the step between timestamps t1 and t2 is denoted as s. More formally, the residual frame can be defined as:

[0030]

[0031] Since adjacent video frames have significant similarity in static information, residual frames usually do not contain background information and object appearance information, but retain significant motion-related information. Therefore, residual frames can be considered as a good source of extracting motion features. In addition, compared with other motion representations such as optical flow, the computational cost of residual frames can be significantly reduced.

[0032] In reality, the actions and activities contained in the video can be complex and can involve different motion speeds or motion durations. In order to cope with this uncertainty, consecutive residual frames can be stacked to form residual segments, which can be defined as:

[0033]

[0034] The above residual segments can capture fast motion on the spatial axis and slow / sustained duration motion on the time axis. Therefore, residual segments are suitable for 3D convolution, in which both short-duration and long-duration motion information can be extracted simultaneously.

[0035] However, since the object appearance and the background scene can also provide important information for recognizing the action, only the residual frame can not be enough to solve the human action recognition problem. For example, in the scene of applying eye makeup and applying lipstick, the movements of applying eye makeup and applying lipstick are similar, but the positions of the movements are different, one of which occurs around the eyes and the other of which occurs around the lips. Therefore, it is necessary to utilize both the RGB frame and the residual frame for action recognition. For this purpose, a new convolution module is provided, which is a pseudo 3D convolution module with the ability to process both the RGB frame and the residual frame.

[0036] The embodiments of the present application will be described in detail below.

[0037] Figure 2 is a schematic diagram of an exemplary detailed design of a deep residual network. As shown in Figure 2 , the deep residual network can at least include an input layer, at least one convolution layer, a pooling layer, at least one fully connected layer, and an output layer. The deep residual network is used for action recognition based on a video segment as input. The deep residual network system in the present application is based on the system of the deep residual network shown in Figure 2 .

[0038] Figure 3 is a schematic flowchart of an action recognition method based on a deep residual network according to an embodiment. The action recognition method based on the deep residual network is applied to a deep residual network system comprising at least one convolution module, each of the at least one convolution module comprising at least one first convolution layer, each of the at least one first convolution layer having at least one one-dimensional filter and at least one two-dimensional filter. As shown in Figure 3 , the method comprises the following.

[0039] 302, the first convolution module in the at least one convolution module receives a video segment as input.

[0040] Specifically, the video segment x∈R T×H×W×Cmay be received as an input to a deep residual network system, where T, H, W represent the length, height, and width of each frame, respectively, and C represents the number of channels. For RGB frames, the number of channels is 3, which represents the red (R), green (G), and blue (B) color channels. When there are no other convolutional modules or layers before the first convolutional module of the at least one convolutional module, the video clip is passed to the first convolutional module of the at least one convolutional module and is received as an input at the first convolutional module of the at least one convolutional module. When there are other layers before the first convolutional module of the at least one convolutional module, the video clip can first be processed by the other layers, then passed to the first convolutional module of the at least one convolutional module, and further received as an input at the first convolutional module of the at least one convolutional module.

[0041] The size of the filter can be represented as T x H x W, where T represents the temporal dimension, H represents the height in the spatial dimension, and W represents the width in the spatial dimension. A 1D filter can be represented as T x 1 x 1, where T is greater than 1, and a 2D filter can be represented as 1 x H x W, where at least one of H and W is greater than 1. The 1D filter is used for convolution in the temporal dimension, and the 2D filter is used for convolution in the spatial dimension.

[0042] 304, traversing the at least one convolutional module by performing the following operations.

[0043] 3042, for the current convolutional module, processing the input using a one-dimensional filter to obtain motion-related features, and processing the input using a two-dimensional filter to obtain appearance-related features.

[0044] In this application, a new convolutional module, which can be regarded as a pseudo 3D convolutional module, is provided. For the convolutional module, the 3D filter in the related art is decoupled to form a parallel 2D spatial filter and a 1D temporal filter. Therefore, by using separable 2D convolution and 1D convolution instead of 3D convolution, the model size and the calculation cost are greatly reduced, which conforms to the latest trend of the development of efficient 3D networks. In addition, the 2D convolution and the 1D convolution are placed in different paths, so that different modeling can be performed for appearance features and motion features.

[0045] 3044, shifting the motion-related features along the temporal dimension by one step, and subtracting the shifted motion-related features from the motion-related features to obtain residual features.

[0046] Specifically, when modeling the motion, the idea of residual frame is extended from the pixel level to the feature level. Assuming that the output feature from the 1D temporal convolutional module is f m ∈R T’×H’×W’×C’, the output feature of the 1D temporal convolution module is shifted by one step, e.g. step 1, along the time dimension, and then the shifted motion-related feature f m (t) is subtracted from the original motion-related feature f m (t+1), the residual feature which can be defined as:

[0047]

[0048] 3046, based on the above appearance-related feature, the above motion-related feature and the above residual feature, the output of the current convolution module is obtained.

[0049] After the pseudo 3D convolution, three features are created, where f s is the output of the 2D convolution module that preserves appearance information, f m is the output of the 1D convolution module, and f m and preserve the discriminative motion structure.

[0050] 3048, the output of the current convolution module is taken as the input of the next (i.e. subsequent) convolution module until the last convolution module in the above at least one convolution module is traversed.

[0051] 306, based on the output of the last convolution module in the above at least one convolution module, at least one action included in the video segment is identified.

[0052] In the embodiments of the present application, a new deep residual network system is provided. The deep residual network system includes at least one convolution module, each of the at least one convolution module includes at least one first convolution layer, each of the at least one first convolution layer has at least one one-dimensional filter and at least one two-dimensional filter, and the deep residual network-based action recognition method is applied to the deep residual network system. The first convolution module in the at least one convolution module receives a video clip as an input. The at least one convolution module is traversed by performing the following operations: for a current convolution module, processing the input using a one-dimensional filter to obtain a motion-related feature and processing the input using a two-dimensional filter to obtain an appearance-related feature; shifting the motion-related feature by one step along a time dimension, subtracting the shifted motion-related feature from the motion-related feature to obtain a residual feature; obtaining an output of the current convolution module based on the appearance-related feature, the motion-related feature, and the residual feature; and taking the output of the current convolution module as an input of a next convolution module until the last convolution module in the at least one convolution module is traversed. At least one action included in the video clip is recognized based on the output of the last convolution module in the at least one convolution module. Therefore, a new convolution module that can be regarded as a pseudo three-dimensional convolution module is proposed in the present application, in which a standard 3D filter in the related art is decoupled to form a two-dimensional spatial filter and a one-dimensional temporal filter in parallel. By using separable two-dimensional convolution and one-dimensional convolution instead of three-dimensional convolution, the model size and the calculation cost are greatly reduced. In addition, the 2D convolution and the 1D convolution are placed in different paths, so that different modeling can be performed for the appearance-related feature and the motion-related feature.

[0053] In one embodiment, the output of the current convolution module is obtained by: obtaining a concatenated feature by concatenating the motion-related feature, the residual feature, and the appearance-related feature in a channel dimension; and determining the concatenated feature as the output of the current convolution module.

[0054] In order to facilitate effective fusion of appearance features and motion features, the output feature of the pseudo 3D convolution module can be concatenated in the channel dimension to obtain a concatenated feature, which can be defined as:

[0055]

[0056] wherein the symbol represents concatenation.

[0057] In one embodiment, the output of the current convolutional module is obtained by: obtaining a concatenated feature by concatenating the motion-related feature, the residual feature and the appearance-related feature in the channel dimension; obtaining a channel attention mask based on the concatenated feature; and obtaining an attention feature as the output of the current convolutional module based on the channel attention mask and the concatenated feature.

[0058] To facilitate the effective fusion of the appearance-related feature and the motion-related feature, a channel self-attention mechanism can be further applied to recalibrate the output feature. Specifically, the output feature can be concatenated in the channel dimension to obtain a concatenated feature, which can be defined as:

[0059]

[0060] wherein the symbol represents concatenation. Since f m ∈R T’×H’×W’×C’ , f s ∈R T’×H’×W’×C’ , , after concatenation, f T’×H’×W’×3C’ ∈R att .

[0061] Then, a channel attention mask M att can be obtained based on the concatenated feature. In one embodiment, each of the at least one convolutional module further comprises a fully connected layer, and the channel attention mask can be obtained based on the concatenated feature in the following manner. Global pooling is performed on the concatenated feature to obtain a pooled concatenated feature, which can be denoted as pool(f). The pooled concatenated feature is multiplied by a weight matrix of the fully connected layer to obtain a weighted concatenated feature, which can be denoted as Wpool(f). The weighted concatenated feature is added with a bias to obtain a biased concatenated feature, which can be denoted as Wpool(f)+b. The biased concatenated feature is processed using a Sigmoid function to obtain the channel attention mask, which can be denoted as σ(Wpool(f)+b). Thus, the channel attention mask M att can be represented as:

[0062] M att = σ(Wpool(f)+b),

[0063] wherein represents a weight matrix parameterized by a single-layer neural network (i.e. the fully connected layer of the convolutional module), wherein, represents a bias term, pool is a global pooling operation that averages the concatenated feature f across space and time, and represents a sigmoid function. The dynamic characteristics are achieved by the channel attention mask, which conditions the dynamic features on the input features and the re-weighted channels based on the importance of the input features to the final task.

[0064] After obtaining the channel attention mask, an attention feature can be further obtained based on the channel attention mask and the concatenated feature. In one embodiment, the attention feature is obtained by performing the channel attention mask M att a channel-wise multiplication between the concatenated feature f, and obtaining the attention feature. In another embodiment, to further improve the robustness, the attention feature is obtained by performing the channel attention mask M att a channel-wise multiplication between the concatenated feature f, and obtaining the attention feature. In another embodiment, to further improve the robustness, the attention feature is obtained by performing the channel attention mask M

[0065] f att = f ® M att + f

[0066] wherein, the symbol "®" represents a channel-wise multiplication. Therefore, the residual connection can be achieved in the proposed convolution module.

[0067] In one embodiment, each of the at least one convolution module further comprises a second convolution layer located before the at least one first convolution layer, the second convolution layer comprising a three-dimensional filter with a size of 1x1x1, and traversing the at least one convolution module further comprises: using the three-dimensional filter to process the input to reduce the dimension of the input before using the one-dimensional filter to process the input to obtain the motion-related feature and using the two-dimensional filter to process the input to obtain the appearance-related feature. Using the one-dimensional filter to process the input to obtain the motion-related feature and using the two-dimensional filter to process the input to obtain the appearance-related feature comprises: using the one-dimensional filter to process the reduced-dimension input to obtain the motion-related feature, and using the two-dimensional filter to process the reduced-dimension input to obtain the appearance-related feature.

[0068] Specifically, before being processed by the 1D filter and the 2D filter of the next convolutional module, the output of the previous convolutional module can be processed by a 3D filter with a size of 1x1x1 first, so as to reduce the number of channels of the output of the previous convolutional module, thereby reducing the dimension of the output of the previous convolutional module (i.e., the input of the next convolutional module). The 3D filter can include at least one 3D filter. The number of 3D filters can be determined based on the desired dimension. For example, in order to reduce the dimension, the number of 3D filters is less than the number of channels of the output of the previous convolutional module; in order to restore the dimension, the number of 3D filters is equal to the number of channels of the output of the previous convolutional module.

[0069] In an embodiment, each of the at least one convolutional module further includes a third convolutional layer located after the at least one first convolutional layer and the second convolutional layer, the third convolutional layer including a three-dimensional filter with a size of 1x1x1, and obtaining the output of the current convolutional module further includes: processing the concatenated feature using the three-dimensional filter to increase the dimension of the concatenated feature; and using the dimension-increased concatenated feature as the output of the current convolutional module.

[0070] In an embodiment, each of the at least one convolutional module further includes a third convolutional layer located after the at least one first convolutional layer and the second convolutional layer, the third convolutional layer including a three-dimensional filter with a size of 1x1x1, and obtaining the output of the current convolutional module further includes: processing the attention feature using the three-dimensional filter to increase the dimension of the attention feature; and using the dimension-increased attention feature as the output of the current convolutional module.

[0071] Specifically, the top 1x1x1 convolution and the bottom 1x1x1 convolution are used to reduce and restore the dimension.

[0072] Further experiments were carried out, in which a variant of ResNet-60 was developed by replacing all bottleneck blocks with the proposed convolutional module.

[0073] For example, the performance of the technical solutions proposed in the present application was evaluated using the UCF101 dataset, where UCF101 is a dataset consisting of 13,330 videos containing 101 action classes. For all experiments, the top-1 accuracy and the top-5 accuracy of the split-1 validation set were reported.

[0074] Specifically, the effectiveness of different data modalities is evaluated by training the action classifier using individual RGB frames, individual residual frames, and combined inputs of RGB frames and residual frames, respectively. Second, the effect of residual frame stride on action recognition is investigated. Finally, an ablation study is also conducted to investigate the effectiveness of individual components in the proposed convolution module.

[0075] Performance comparison of different data modalities.

[0076] Table 1 shows the action recognition performance of various combinations of input modalities and network architectures. For experiments using only RGB frames or residual frames, only one stream in the data layer (i.e., the first convolutional layer) is kept, but the number of channels is doubled for fair comparison. As can be seen from Table 1, using only residual frames is about 3% higher than using only RGB frames in terms of top-1 accuracy and top-5 accuracy, which indicates that residual frames indeed contain significant motion information important for action recognition. When using both RGB frames and residual frames in different streams, the top-1 accuracy is further improved by 2.6% (from 83.0% to 85.6%), which indicates that the two data modalities preserve complementary information. Notably, compared to using standard 3D convolutions in related art, using the convolution module provided in this application not only significantly reduces the floating point operations per second (flops) (from 163G to 40G), but also provides better performance (85.6% vs. 85.0% in terms of top-1 accuracy).

[0077] Table 1: Performance comparison of different input modalities and network architectures.

[0078]

[0079] Effect of stride s.

[0080] When generating residual frames, the stride s can be changed to capture motion features at different time scales. However, it is not clear what the optimal stride is for the action recognition task. Therefore, the effect of stride is investigated, and the results are shown in Table 2. Experiments are conducted for three settings, where the input data are residual frames with stride s = 1, 2, 4, respectively. As can be seen from Table 2, the classification accuracy decreases as the stride increases. We conjecture that motion causes spatial displacement of the same object between two frames, and using a large stride can cause mismatch between motion representations.

[0081] Table 2: Performance comparison of different residual frame strides s.

[0082] Step size Val top-1 Val top-5 s＝1 83.0％ 98.3％ s＝2 82.7％ 97.1％ s＝4 80.2％ 96.0％

[0083] Ablation study.

[0084] To verify the effectiveness of different components in the proposed convolution module, an ablation study is conducted. Without loss of generality, the model is trained with the combination of RGB frames and residual frames (s = 1) as input. Table 3 shows the performance comparison of various convolution module settings. As shown in Table 3, removing the self-attention mechanism results in a 1.8% drop in top-1 accuracy (from 85.6% to 83.8%). Meanwhile, when ignoring the residual information in the feature space, the performance also drops from 85.6% to 83.5%. If both the self-attention mechanism and the residual feature are removed, the top-1 accuracy will further decrease to 82.1%. These results confirm that the channel self-attention mechanism and the residual feature are effective in improving the action recognition performance.

[0085] Table 3: Performance comparison of various convolution module settings

[0086] Method Val top-1 Val top-5 Convolutional module without self-attention and residual features 82.1％ 97.9％ Convolutional module without self-attention 83.8％ 98.4％ Convolutional module without residual features 83.5％ 98.1％ Convolutional module 85.6％ 99.2％

[0087] Figure 4 is a schematic flowchart of a deep residual network-based action recognition method according to an embodiment. The deep residual network-based action recognition method is applied to a deep residual network system comprising at least one convolution module, each of the at least one convolution module comprising at least one first convolution layer, each of the at least one first convolution layer having at least one one-dimensional filter and at least one two-dimensional filter. As shown in Figure 4 , the method comprises the following.

[0088] 402, a first convolution module in the at least one convolution module receives a video clip as input.

[0089] 404, traversing the at least one convolution module by performing the following operations.

[0090] 40502, processing the input using a three-dimensional filter to reduce the dimension of the input.

[0091] 40504, processing the reduced-dimension input using the one-dimensional filter to obtain motion-related features and processing the reduced-dimension input using the two-dimensional filter to obtain appearance-related features.

[0092] 40506, shifting the motion-related features by one step along the time dimension and subtracting the shifted motion-related features from the motion-related features to obtain residual features.

[0093] 40508, obtaining concatenated features by concatenating the motion-related features, the residual features, and the appearance-related features in the channel dimension.

[0094] 40410, performing global pooling on the concatenated features to obtain pooled concatenated features.

[0095] 40412, multiplying the pooled concatenated features by a weight matrix of a fully connected layer to obtain weighted concatenated features.

[0096] 40414, adding a bias to the weighted concatenated features to obtain biased concatenated features.

[0097] 40416, processing the biased concatenated features using a Sigmoid function to obtain a channel attention mask.

[0098] 40418, obtaining intermediate features by performing a channel-wise multiplication between the channel attention mask and the concatenated features.

[0099] 40420, adding the intermediate features to the concatenated features to obtain attention features.

[0100] 40422, processing the attention features using the three-dimensional filter to increase the dimensionality of the attention features.

[0101] 40424, using the dimensionally increased attention features as the output of the current convolutional module.

[0102] 40426, using the output of the current convolutional module as the input of the next convolutional module until the last convolutional module in the at least one convolutional module is traversed.

[0103] 406, identifying at least one action included in the video segment based on the output of the last convolutional module in the at least one convolutional module.

[0104] Figure 5 An exemplary detailed design of the proposed convolutional module is shown. The proposed convolutional module can be integrated into any standard CNN architecture, such as ResNet. To process both RGB frames and residual frames, the original data layer (i.e., the first convolutional layer) is modified to have two streams of parallel building blocks that output appearance-related features and motion-related features, respectively, each for each modality. The resulting features from the two streams are concatenated and passed to the next layer (i.e., the subsequent convolutional module). In the exemplary design of the proposed convolutional module, a 2D filter with a size of 3x3 and a 1D filter with a size of 1x1 are used as examples. Specifically, the size of the filter can be denoted as TxHxW, where T represents the temporal dimension, H represents the height in the spatial dimension, and W represents the width in the spatial dimension.

[0105] Table 4 shows an exemplary detailed design of the deep residual network system. As shown in Table 4, the deep residual network system includes four convolutional modules, denoted as res 2, res 3, res 4, and res 5, respectively. In the exemplary design of the deep residual network system, the size of the filter can be denoted by {T x S 2 ,C} to represent the time, space, and channel dimensions, specifically, in Table 4, a 2D filter with size 3 x 3 and a ID filter with size 1 x 1 are exemplified.

[0106] Table 4: Exemplary detailed design of the deep residual network system.

[0107]

[0108] In embodiments of the present application, a new deep residual network system is provided. The deep residual network system includes at least one convolutional module, each of the at least one convolutional module includes at least one first convolutional layer, each of the at least one first convolutional layer has at least one one-dimensional filter and at least one two-dimensional filter, and the action recognition method based on the deep residual network is applied to the deep residual network system. The first convolutional module in the at least one convolutional module receives a video segment as input. The at least one convolutional module is traversed by performing the following operations: for a current convolutional module, processing the input using a one-dimensional filter to obtain motion-related features and processing the input using a two-dimensional filter to obtain appearance-related features; shifting the motion-related features by one step along the time dimension, subtracting the shifted motion-related features from the motion-related features to obtain residual features; obtaining the output of the current convolutional module based on the appearance-related features, the motion-related features, and the residual features; taking the output of the current convolutional module as the input of the next convolutional module until the last convolutional module in the at least one convolutional module is traversed. At least one action included in the video segment is recognized based on the output of the last convolutional module in the at least one convolutional module. Therefore, in the present application, a new convolutional module which can be regarded as a pseudo three-dimensional convolutional module is proposed, in which the standard 3D filter in the related art is decoupled to form a parallel two-dimensional spatial filter and a one-dimensional temporal filter. By using separable two-dimensional convolution and one-dimensional convolution instead of three-dimensional convolution, the model size and the calculation cost are greatly reduced. In addition, the 2D convolution and the ID convolution are placed in different paths, so that different modeling can be performed for the appearance-related features and the motion-related features.

[0109] The above operations can refer to the detailed description of the network training operation in the action recognition method based on the deep residual network, which will not be described here.

[0110] It should be noted that, in order to implement the above functions, the electronic device comprises a hardware structure and / or a software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed in the present text, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0111] The embodiments of the present application can divide the functional units of the action recognition apparatus based on the deep residual network according to the above method examples. For example, each functional unit can be divided according to each function, or two or more functions can be integrated in one processing unit. The integrated unit can be realized in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical functional division. In actual implementation, there can be another division method.

[0112] Figure 6 is a schematic structural diagram of an action recognition apparatus based on a deep residual network according to an embodiment. The apparatus is applied to a deep residual network system, which comprises at least one convolution module. Each convolution module in the at least one convolution module comprises at least one first convolution layer. Each first convolution layer in the at least one first convolution layer has at least one one-dimensional filter and at least one two-dimensional filter. As shown in Figure 6 The action recognition apparatus based on the deep residual network comprises a receiving unit 602, a processing unit 604 and an identification unit 606.

[0113] The receiving unit 602 is configured to receive a video segment as input in a first convolution module in the at least one convolution module.

[0114] The processing unit 604 is configured to traverse the at least one convolution module by performing the following operations: for a current convolution module, processing the input using a one-dimensional filter to obtain a motion-related feature and processing the input using a two-dimensional filter to obtain an appearance-related feature; shifting the motion-related feature along a time dimension by one step, subtracting the shifted motion-related feature from the motion-related feature to obtain a residual feature; obtaining an output of the current convolution module based on the appearance-related feature, the motion-related feature, and the residual feature; and taking the output of the current convolution module as an input of a next convolution module until a last convolution module in the at least one convolution module is traversed.

[0115] The identifying unit 606 is configured to identify at least one action included in the video segment based on an output of the last convolution module in the at least one convolution module.

[0116] In the embodiments of the present application, a new deep residual network system is provided. The deep residual network system includes at least one convolution module, each convolution module in the at least one convolution module includes at least one first convolution layer, each first convolution layer in the at least one first convolution layer has at least one one-dimensional filter and at least one two-dimensional filter, and the deep residual network-based action recognition method is applied to the deep residual network system. A first convolution module in the at least one convolution module receives a video segment as an input. The at least one convolution module is traversed by performing the following operations: processing the input using a one-dimensional filter to obtain a motion-related feature and processing the input using a two-dimensional filter to obtain an appearance-related feature; shifting the motion-related feature along a time dimension by one step, subtracting the shifted motion-related feature from the motion-related feature to obtain a residual feature; obtaining an output of the convolution module based on the appearance-related feature, the motion-related feature, and the residual feature; and taking the output of the convolution module as an input of a next convolution module until a last convolution module in the at least one convolution module is traversed. At least one action included in the video segment is identified based on an output of the last convolution module in the at least one convolution module. Therefore, a new convolution module that can be regarded as a pseudo three-dimensional convolution module is proposed in the present application, in which a standard 3D filter in the related art is decoupled to form a two-dimensional spatial filter and a one-dimensional temporal filter in parallel. By using separable two-dimensional convolution and one-dimensional convolution instead of three-dimensional convolution, the model size and the calculation cost are greatly reduced. In addition, the 2D convolution and the 1D convolution are placed in different paths, so that different modeling can be performed for the appearance-related feature and the motion-related feature.

[0117] In an embodiment, in acquiring the output of the current convolution module, the processing unit 604 is specifically configured to: acquire a concatenated feature by concatenating the motion-related feature, the residual feature and the appearance-related feature in a channel dimension; and determine the concatenated feature as the output of the current convolution module.

[0118] In an embodiment, in acquiring the output of the current convolution module, the processing unit 604 is specifically configured to: acquire a concatenated feature by concatenating the motion-related feature, the residual feature and the appearance-related feature in a channel dimension; acquire a channel attention mask based on the concatenated feature; and acquire an attention feature as the output of the current convolution module based on the channel attention mask and the concatenated feature.

[0119] In an embodiment, in acquiring the attention feature based on the channel attention mask and the concatenated feature, the processing unit 604 is specifically configured to: acquire the attention feature by performing a channel-by-channel multiplication between the channel attention mask and the concatenated feature.

[0120] In an embodiment, in acquiring the attention feature based on the channel attention mask and the concatenated feature, the processing unit 604 is specifically configured to: acquire an intermediate feature by performing a channel-by-channel multiplication between the channel attention mask and the concatenated feature; and acquire the attention feature by adding the intermediate feature and the concatenated feature.

[0121] In an embodiment, each of the at least one convolution module further comprises a fully connected layer, and in acquiring the channel attention mask based on the concatenated feature, the processing unit 604 is configured to: acquire a pooled concatenated feature by performing global pooling on the concatenated feature; acquire a weighted concatenated feature by multiplying the pooled concatenated feature by a weight matrix of the fully connected layer; acquire a biased concatenated feature by adding a bias to the weighted concatenated feature; and acquire the channel attention mask by processing the biased concatenated feature using a Sigmoid function.

[0122] In an embodiment, each of the at least one convolution module further comprises a second convolution layer before the at least one first convolution layer, the second convolution layer comprising a three-dimensional filter with a size of 1x1x1, and the processing unit 604 is further configured to, in traversing the at least one convolution module, process the input using the three-dimensional filter to reduce the dimensionality of the input before processing the input using the one-dimensional filter to obtain the motion-related feature and processing the input using the two-dimensional filter to obtain the appearance-related feature; and wherein, in processing the input using the one-dimensional filter to obtain the motion-related feature and processing the input using the two-dimensional filter to obtain the appearance-related feature, the processing unit 604 is specifically configured to process the reduced-dimensionality input using the one-dimensional filter to obtain the motion-related feature and process the reduced-dimensionality input using the two-dimensional filter to obtain the appearance-related feature.

[0123] In an embodiment, each of the at least one convolution module further comprises a third convolution layer after the at least one first convolution layer and the second convolution layer, the third convolution layer comprising a three-dimensional filter with a size of 1x1x1, and the processing unit 604 is further configured to, in obtaining the output of the current convolution module, process the concatenated feature using the three-dimensional filter to increase the dimensionality of the concatenated feature; and use the increased-dimensionality concatenated feature as the output of the current convolution module.

[0124] In an embodiment, each of the at least one convolution module further comprises a third convolution layer after the at least one first convolution layer and the second convolution layer, the third convolution layer comprising a three-dimensional filter with a size of 1x1x1, and the processing unit 604 is further configured to, in obtaining the output of the current convolution module, process the attention feature using the three-dimensional filter to increase the dimensionality of the attention feature; and use the increased-dimensionality attention feature as the output of the current convolution module.

[0125] Figure 7 is a schematic structural diagram of a terminal device according to an embodiment. As shown in Figure 7As shown, the terminal device 700 includes a processor 701, a memory 702, a communication interface 703, and one or more programs 704 stored in the aforementioned memory 702 and executed by the aforementioned processor 701. The aforementioned one or more programs 704 include instructions for executing a deep residual network (ResNet) system. The aforementioned deep residual network system includes at least one convolutional module, each of the aforementioned at least one convolutional module including at least one first convolutional layer, each of the aforementioned at least one first convolutional layer having at least one one-dimensional (1D) filter and at least one two-dimensional (2D) filter. The aforementioned one or more programs 704 include instructions for performing the following operations.

[0126] A first convolutional module in the aforementioned at least one convolutional module receives a video clip as input. The aforementioned at least one convolutional module is traversed by performing the following operations: for a current convolutional module, processing the aforementioned input using a one-dimensional filter to obtain a motion-related feature and processing the aforementioned input using a two-dimensional filter to obtain an appearance-related feature; shifting the aforementioned motion-related feature along a time dimension by one step, subtracting the aforementioned shifted motion-related feature from the aforementioned motion-related feature to obtain a residual feature; obtaining an output of the aforementioned current convolutional module based on the aforementioned appearance-related feature, the aforementioned motion-related feature, and the aforementioned residual feature; taking the aforementioned output of the aforementioned current convolutional module as input of a next convolutional module until a last convolutional module in the aforementioned at least one convolutional module is traversed. At least one action included in the aforementioned video clip is identified based on an output of the aforementioned last convolutional module in the aforementioned at least one convolutional module.

[0127] In the embodiments of the present application, a new deep residual network system is provided. The deep residual network system includes at least one convolution module, each of the at least one convolution module includes at least one first convolution layer, each of the at least one first convolution layer has at least one one-dimensional filter and at least one two-dimensional filter, and the deep residual network-based action recognition method is applied to the deep residual network system. The first convolution module in the at least one convolution module receives a video clip as an input. The at least one convolution module is traversed by performing the following operations: for a current convolution module, processing the input using a one-dimensional filter to obtain a motion-related feature and processing the input using a two-dimensional filter to obtain an appearance-related feature; shifting the motion-related feature by one step along a time dimension, subtracting the shifted motion-related feature from the motion-related feature to obtain a residual feature; obtaining an output of the current convolution module based on the appearance-related feature, the motion-related feature, and the residual feature; and taking the output of the current convolution module as an input of a next convolution module until a last convolution module in the at least one convolution module is traversed. At least one action included in the video clip is recognized based on an output of the last convolution module in the at least one convolution module. Therefore, in the present application, a new convolution module that can be regarded as a pseudo three-dimensional convolution module is proposed, in which a standard 3D filter in the related art is decoupled to form a two-dimensional spatial filter and a one-dimensional temporal filter in parallel. By using separable two-dimensional convolution and one-dimensional convolution instead of three-dimensional convolution, the model size and the calculation cost are greatly reduced. In addition, the 2D convolution and the 1D convolution are placed in different paths, so that different modeling can be performed for the appearance-related feature and the motion-related feature.

[0128] In one embodiment, in obtaining the output of the current convolution module, the one or more programs 704 include instructions for performing the following operation. A concatenated feature is obtained by concatenating the motion-related feature, the residual feature, and the appearance-related feature in a channel dimension; and the concatenated feature is determined as the output of the current convolution module.

[0129] In one embodiment, in obtaining the output of the current convolution module, the one or more programs 704 include instructions for performing the following operation. A concatenated feature is obtained by concatenating the motion-related feature, the residual feature, and the appearance-related feature in a channel dimension; a channel attention mask is obtained based on the concatenated feature; and an attention feature is obtained as the output of the current convolution module based on the channel attention mask and the concatenated feature.

[0130] In one embodiment, in acquiring the attention feature based on the channel attention mask and the concatenated feature, the one or more programs 704 include instructions for performing the following operations. The attention feature is acquired by performing a channel-wise multiplication between the channel attention mask and the concatenated feature.

[0131] In one embodiment, in acquiring the attention feature based on the channel attention mask and the concatenated feature, the one or more programs 704 include instructions for performing the following operations. An intermediate feature is acquired by performing a channel-wise multiplication between the channel attention mask and the concatenated feature; and the attention feature is acquired by adding the intermediate feature and the concatenated feature.

[0132] In one embodiment, each of the at least one convolution module further includes a fully connected layer, in acquiring the channel attention mask based on the concatenated feature, the one or more programs 704 include instructions for performing the following operations. Global pooling is performed on the concatenated feature to acquire a pooled concatenated feature; a weight matrix of the fully connected layer is multiplied with the pooled concatenated feature to acquire a weighted concatenated feature; a bias is added to the weighted concatenated feature to acquire a biased concatenated feature; and the biased concatenated feature is processed using a Sigmoid function to acquire the channel attention mask.

[0133] In one embodiment, each of the at least one convolution module is located before a second convolution layer of the at least one first convolution layer, the second convolution layer includes a three-dimensional filter with a size of 1x1x1, in traversing the at least one convolution module, the one or more programs 704 further include instructions for performing the following operations. The input is processed using the three-dimensional filter to reduce a dimension of the input before the input is processed using the one-dimensional filter to acquire the motion-related feature and the input is processed using the two-dimensional filter to acquire the appearance-related feature; wherein, in processing the input using the one-dimensional filter to acquire the motion-related feature and the input using the two-dimensional filter to acquire the appearance-related feature, the one or more programs 704 include instructions for performing the following operations. The reduced-dimension input is processed using the one-dimensional filter to acquire the motion-related feature, and the reduced-dimension input is processed using the two-dimensional filter to acquire the appearance-related feature.

[0134] In an embodiment, each of the at least one convolution module further comprises a third convolution layer located after the at least one first convolution layer and the second convolution layer, the third convolution layer comprising a three-dimensional filter with a size of 1x1x1, and the one or more programs 704, in obtaining the output of the current convolution module, comprise instructions for performing the following operations. processing the concatenated feature using the three-dimensional filter to increase a dimension of the concatenated feature; and using the dimension-increased concatenated feature as the output of the current convolution module.

[0135] In an embodiment, each of the at least one convolution module further comprises a third convolution layer located after the at least one first convolution layer and the second convolution layer, the third convolution layer comprising a three-dimensional filter with a size of 1x1x1, and the one or more programs 704, in obtaining the output of the current convolution module, comprise instructions for performing the following operations. processing the attention feature using the three-dimensional filter to increase a dimension of the attention feature; and using the dimension-increased attention feature as the output of the current convolution module.

[0136] A non-transitory computer storage medium is also provided. The non-transitory computer storage medium is configured to store a program, and when the program is executed, the program is used to perform some or all of the operations of the deep residual network-based action recognition method described in the method embodiments.

[0137] A computer program product is also provided. The computer program product comprises a non-transitory computer readable storage medium storing a computer program. The computer program is capable of causing a computer to perform some or all of the operations of the deep residual network-based action recognition method described in the method embodiments.

[0138] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the described actions, because according to the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily the essential of the present application

[0139] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0140] In several embodiments provided in the present application, it should be understood that the disclosed apparatus can be implemented in other manners. For example, the division of the apparatus embodiments described above is merely illustrative, and the division of the units can be changed according to actual needs. For example, the units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0141] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0142] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0143] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0144] Those of ordinary skill in the art can understand that all or part of the steps of the various methods in the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0145] The embodiments of the present application are described in detail above, and the principles and implementation manners of the present application are described by using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present application. Meanwhile, the skilled in the art can modify the specific implementation manners and application range according to the idea of the present application. In summary, the content of the specification should not be interpreted as a limitation on the present application.

Claims

1. An action recognition method based on deep residual networks, applied to a deep residual network system including at least one convolutional module, wherein each convolutional module includes at least one first convolutional layer, and each first convolutional layer has at least one one-dimensional filter and at least one two-dimensional filter, the method comprising: The first convolutional module in the at least one convolutional module receives a video segment as input; The at least one convolutional module is traversed by performing the following operation: For the current convolutional module, a one-dimensional filter is used to process the input to obtain motion-related features, and a two-dimensional filter is used to process the input to obtain appearance-related features; The motion-related features are shifted by one step along the time dimension, and the shifted motion-related features are subtracted from the motion-related features to obtain the residual features; Based on the appearance-related features, the motion-related features, and the residual features, the output of the current convolutional module is obtained; The output of the current convolutional module is used as the input of the next convolutional module until the last convolutional module in the at least one convolutional module has been traversed. Based on the output of the last convolutional module in the at least one convolutional module, at least one action included in the video segment is identified.

2. The method according to claim 1, wherein, Obtaining the output of the current convolutional module includes: Cascaded features are obtained by cascading the motion-related features, the residual features, and the appearance-related features along the channel dimension; and The cascaded features are determined as the output of the current convolutional module.

3. The method according to claim 1, wherein, Obtaining the output of the current convolutional module includes: Cascaded features are obtained by cascading the motion-related features, the residual features, and the appearance-related features along the channel dimension. Based on the cascaded features, obtain the channel attention mask; and Based on the channel attention mask and the cascaded features, attention features are obtained as the output of the current convolutional module.

4. The method according to claim 3, characterized in that, Based on the channel attention mask and the cascaded features, obtaining the attention features includes: The attention features are obtained by performing channel-by-channel multiplication between the channel attention mask and the cascaded features.

5. The method according to claim 3, characterized in that, Based on the channel attention mask and the cascaded features, obtaining the attention features includes: Intermediate features are obtained by performing channel-by-channel multiplication between the channel attention mask and the cascaded features; and The attention feature is obtained by adding the intermediate feature to the cascaded feature.

6. The method according to any one of claims 3 to 5, characterized in that, Each of the at least one convolutional module further includes a fully connected layer, and obtaining the channel attention mask based on the cascaded features includes: Perform global pooling on the cascaded features to obtain the pooled cascaded features; Multiply the pooled cascaded features by the weight matrix of the fully connected layer to obtain weighted cascaded features; The weighted concatenated features are added to the bias to obtain the bias concatenated features; and The bias cascaded features are processed using the Sigmoid function to obtain the channel attention mask.

7. The method according to any one of claims 3 to 5, characterized in that, Each of the at least one convolutional modules further includes a second convolutional layer located before the at least one first convolutional layer, the second convolutional layer including a three-dimensional filter of size 1×1×1, and traversing the at least one convolutional module further includes: Before processing the input using the one-dimensional filter to obtain motion-related features, and before processing the input using the two-dimensional filter to obtain appearance-related features: The input is processed using the 3D filter to reduce the dimensionality of the input; The process of using a one-dimensional filter to process the input to obtain motion-related features and using the two-dimensional filter to process the input to obtain appearance-related features includes: The input after dimensionality reduction is processed using the one-dimensional filter to obtain the motion-related features, and the input after dimensionality reduction is processed using the two-dimensional filter to obtain the appearance-related features.

8. The method according to claim 7, wherein, Each of the at least one convolutional module further includes a third convolutional layer located after the at least one first convolutional layer and the second convolutional layer, the third convolutional layer including a three-dimensional filter of size 1×1×1, and obtaining the output of the current convolutional module further includes: The cascaded features are processed using the three-dimensional filter to increase the dimensionality of the cascaded features; and The cascaded features after dimensionality upgrade are used as the output of the current convolutional module.

9. The method according to claim 7, wherein, Each of the at least one convolutional module further includes a third convolutional layer located after the at least one first convolutional layer and the second convolutional layer, the third convolutional layer including a three-dimensional filter of size 1×1×1, and obtaining the output of the current convolutional module further includes: The attention features are processed using the three-dimensional filter to increase the dimensionality of the attention features; and The attention features, after being upgraded in dimensionality, are used as the output of the current convolutional module.

10. An action recognition device based on a deep residual network, applied to a deep residual network system including at least one convolutional module, each of the at least one convolutional module including at least one first convolutional layer, each of the at least one first convolutional layer having at least one one-dimensional filter and at least one two-dimensional filter, the device comprising: A receiving unit is configured to receive a video segment as input in the first convolutional module of the at least one convolutional module; A processing unit is configured to traverse the at least one convolutional module by performing the following operations: For the current convolutional module, a one-dimensional filter is used to process the input to obtain motion-related features, and a two-dimensional filter is used to process the input to obtain appearance-related features; The motion-related features are shifted by one step along the time dimension, and the shifted motion-related features are subtracted from the motion-related features to obtain the residual features; Based on the appearance-related features, the motion-related features, and the residual features, the output of the current convolutional module is obtained; The output of the current convolutional module is used as the input of the next convolutional module until the last convolutional module in the at least one convolutional module has been traversed. The recognition unit identifies at least one action included in the video segment based on the output of the last convolutional module in the at least one convolutional module.

11. The apparatus according to claim 10, wherein, In acquiring the output of the current convolutional module, the processing unit is specifically configured to: Cascaded features are obtained by cascading the motion-related features, the residual features, and the appearance-related features along the channel dimension; and The cascaded features are determined as the output of the current convolutional module.

12. The apparatus according to claim 10, wherein, In acquiring the output of the current convolutional module, the processing unit is specifically configured to: Cascaded features are obtained by cascading the motion-related features, the residual features, and the appearance-related features along the channel dimension. Based on the cascaded features, obtain the channel attention mask; as well as Based on the channel attention mask and the cascaded features, attention features are obtained as the output of the current convolutional module.

13. The apparatus according to claim 12, wherein, In acquiring the attention features based on the channel attention mask and the cascaded features, the processing unit is specifically used for: The attention features are obtained by performing channel-by-channel multiplication between the channel attention mask and the cascaded features.

14. The apparatus according to claim 12, wherein, In acquiring attention features based on the channel attention mask and the cascaded features, the processing unit is specifically used for: Intermediate features are obtained by performing channel-by-channel multiplication between the channel attention mask and the cascaded features; and The attention feature is obtained by adding the intermediate feature to the cascaded feature.

15. The apparatus according to any one of claims 12 to 14, wherein, Each of the at least one convolutional module further includes a fully connected layer, and the processing unit is specifically used for: obtaining the channel attention mask based on the cascaded features. Perform global pooling on the cascaded features to obtain the pooled cascaded features; Multiply the pooled cascaded features by the weight matrix of the fully connected layer to obtain weighted cascaded features; The weighted concatenated features are added to the bias to obtain the bias concatenated features; as well as The bias cascaded features are processed using the Sigmoid function to obtain the channel attention mask.

16. The apparatus according to any one of claims 12 to 14, wherein, Each of the at least one convolutional modules further includes a second convolutional layer located before the at least one first convolutional layer, the second convolutional layer including a three-dimensional filter of size 1×1×1. In traversing the at least one convolutional module, the processing unit is further configured to: process the input with the three-dimensional filter to reduce the dimensionality of the input before processing the input with the one-dimensional filter to obtain motion-related features and processing the input with the two-dimensional filter to obtain appearance-related features; Specifically, in processing the input using a one-dimensional filter to obtain motion-related features and processing the input using a two-dimensional filter to obtain appearance-related features, the processing unit is configured to: process the dimensionality-reduced input using the one-dimensional filter to obtain the motion-related features, and process the dimensionality-reduced input using the two-dimensional filter to obtain the appearance-related features.

17. The apparatus according to claim 16, wherein, Each of the at least one convolutional modules further includes a third convolutional layer located after the at least one first convolutional layer and the second convolutional layer, the third convolutional layer including a three-dimensional filter of size 1×1×1, and in obtaining the output of the current convolutional module, the processing unit is further configured to: process the cascaded features using the three-dimensional filter to increase the dimension of the cascaded features; as well as The cascaded features after dimensionality upgrade are used as the output of the current convolutional module.

18. The apparatus according to claim 16, wherein, Each of the at least one convolutional module further includes a third convolutional layer following the at least one first convolutional layer and the second convolutional layer, the third convolutional layer including a 1×1×1 three-dimensional filter. In acquiring the output of the current convolutional module, the processing unit is further configured to: process the attention features with the 3D filter to increase the dimension of the attention features; and The attention features, after being upgraded in dimensionality, are used as the output of the current convolutional module.

19. A terminal device, comprising a processor and a memory for storing one or more programs, wherein, The one or more programs are to be executed by the processor and include instructions for performing the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium for storing a computer program for electronic data interchange, which, when executed, causes a computer to perform the method according to any one of claims 1 to 9.

Images