Lightweight motion recognition method for deployable edge device based on time sequence modeling

Abstract

The invention provides a lightweight action recognition method for deployable edge equipment based on time sequence modeling. The lightweight action recognition method comprises the following steps: extracting a key frame; a lightweight motion recognition model based on a MobileNetV3 backbone network is constructed; and inputting the extracted key frame into the action recognition model, firstly obtaining an abstract feature fused by a spatial feature and a time feature, and then obtaining a final classification probability based on the fused feature so as to obtain an action recognition result. According to the method, firstly, key frame extraction is performed on an original video, then multi-scale spatial-temporal feature extraction is performed on input key frames, a time sequence modeling method is adopted in time sequence feature extraction, then local cross-channel interaction is realized by utilizing an ECA channel attention mechanism, and finally, spatial-temporal feature information is fused for classification to obtain a final action recognition result. And the accuracy of action recognition is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision action recognition, and in particular relates to a lightweight action recognition method for deployable edge devices based on time series modeling. Background Art

[0002] With the development of the Internet of Things (IoT) and edge computing, more and more sensor devices are deployed in various environments to collect data and conduct real-time analysis and decision-making. In many application scenarios, such as smart homes, smart health monitoring, industrial production, and intelligent transportation systems, the recognition of human behavior and actions has become crucial. Action recognition can be used to monitor human activity status, identify abnormal behavior, or as part of the user interface, such as gesture control.

[0003] Action recognition on edge devices is challenging because these devices usually have limited computing resources and storage capacity. Therefore, researchers tend to develop lightweight action recognition algorithms that can run on edge devices with low latency and low power consumption.

[0004] The action recognition method based on time series modeling is a common method. It uses a camera to collect videos containing various actions as input, and then uses machine learning or deep learning techniques to train the model to recognize different actions or behaviors. The key to this method is how to effectively capture the characteristics of time series data and design a lightweight model structure to execute in real time on edge devices. Although the network architecture based on 3D CNN and transformer has the ability of time series modeling and can extract time features, its model calculation amount and parameter amount are large, which cannot meet the real-time operation on resource-constrained edge devices. The network architecture based on 2D CNN does not have the ability of time series modeling and cannot extract time features, but its lightweight model structure has a small amount of calculation and parameter amount, which can meet the deployment on edge devices. Therefore, the introduction of the time series modeling method on the 2D CNN architecture not only meets the conditions for the model to deploy edge devices, but also ensures that the model can extract time features. The existing lightweight action recognition methods that can be deployed on edge devices still have room for improvement in recognition accuracy. Therefore, it is necessary to develop more effective models to obtain more accurate action recognition results. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a lightweight action recognition method for deployable edge devices based on time series modeling. First, the key frame extraction module is used to extract key frames from the original video frame sequence, and then the standard convolution layer is used to perform preliminary feature extraction on the input video frame. These features are then input into the improved MobileNetV3 residual block to extract spatial features and temporal features and fuse them to obtain fused features. Finally, the fused features are input into the classification module to obtain the final classification probability, thereby obtaining the final action recognition result.

[0006] The present invention proposes a lightweight action recognition method for deployable edge devices based on time series modeling, comprising:

[0007] Key frame extraction;

[0008] Build a lightweight action recognition model based on the MobileNetV3 backbone network;

[0009] The extracted key frames are input into the action recognition model to first obtain an abstract feature fused by spatial features and temporal features, and then based on the fused features, the final classification probability is obtained to obtain the action recognition result.

[0010] Optionally, the key frame extraction module is a key frame extraction method based on image entropy and density clustering. The key frame extraction method is mainly divided into three steps, namely 1) calculating image entropy, 2) finding local extreme points, and 3) performing density clustering.

[0011] Optionally, the lightweight action recognition model based on the MobileNetV3 backbone network includes: a standard convolutional layer, an improved MobileNetV3 residual block and a classification module. The improved MobileNetV3 residual block includes: a temporal modeling module, Conv 1×1, Conv 3×3, an ECA channel attention mechanism and Conv 1×1.

[0012] Optionally, the standard convolution layer and the classification module are derived from the MobileNetV3 backbone network. The standard convolution layer is used to extract the basic features of the input image and pass it to the subsequent layers of the network for further feature extraction and processing. The classification module includes a pooling layer and a fully connected layer, which are used to map the extracted fusion features into action recognition output results.

[0013] Optionally, obtaining the spatial features and temporal features includes: inputting the basic features of the image extracted by the standard convolutional layer into the improved MobileNetV3 residual block, using the improved MobileNetV3 residual block to extract temporal features and spatial features and generate fusion features.

[0014] Optionally, the temporal feature extraction module is a TAM (temporal adaptive module). The TAM is a modular module that can be integrated into a 2D CNN to generate a powerful video architecture with very little additional computational cost. The TAM consists of two branches: a local branch (L) and a global branch (G). The local branch uses temporal convolution to generate position-sensitive importance maps to enhance local features, while the global branch uses a fully connected layer to generate position-invariant kernels for temporal aggregation.

[0015] Optionally, the ECA channel attention mechanism replaces the SE channel attention mechanism in the original MobileNetV3 residual block to avoid dimensionality reduction, achieve local cross-channel interaction, and effectively capture cross-channel interaction information.

[0016] Optionally, the improved MobileNetV3 residual block is divided into: a residual block using a residual connection and a residual block not using a residual connection. When the step size is 1 and the input and output channels are the same, the residual block uses a residual connection, and the residual block contains a TAM temporal modeling module. When the step size is 2 or the input and output channels are different, the residual block does not use a residual connection, and the residual block does not contain a TAM.

[0017] Compared with the prior art, the present invention has the following advantages and technical effects:

[0018] The present invention uses key frame extraction technology to extract representative K frame images from the original video frames as model input, which can extract the main content of the data sequence under the premise of processing a small number of frames and avoid the interference of feature information in non-key frames.

[0019] The present invention is based on MobileNetV3 as the feature extraction backbone network, so that the model can run in real time on resource-constrained edge devices.

[0020] The present invention introduces TAM for temporal modeling, which enables the 2D CNN network, which originally does not have temporal modeling capabilities, to have better temporal modeling capabilities while only increasing a small amount of computing cost. This not only ensures the accuracy of action recognition but also ensures the deployment of the model on edge devices with limited computing resources.

[0021] The present invention introduces the ECA channel attention mechanism to replace the original SE attention mechanism. It can not only avoid dimensionality reduction but also realize local cross-channel interaction of the model while only adding a small number of parameters. It improves the accuracy of model action recognition while ensuring the ability to deploy edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 The present invention is a flowchart of a lightweight motion recognition method that can be deployed on edge devices according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] The present invention proposes a lightweight action recognition method for deployable edge devices based on time series modeling, comprising:

[0027] Key frame extraction;

[0028] Build a lightweight action recognition model based on the MobileNetV3 backbone network;

[0029] The extracted key frames are input into the action recognition model to first obtain an abstract feature fused by spatial features and temporal features, and then based on the fused features, the final classification probability is obtained to obtain the action recognition result.

[0030] Furthermore, the key frame extraction module is a key frame extraction method based on image entropy and density clustering. The key frame extraction method is mainly divided into three steps, namely 1) calculating image entropy, 2) finding local extreme points, and 3) performing density clustering.

[0031] Furthermore, the lightweight action recognition model based on the MobileNetV3 backbone network includes: a standard convolutional layer, an improved MobileNetV3 residual block and a classification module. The improved MobileNetV3 residual block includes: a temporal modeling module, Conv 1×1, Conv 3×3, an ECA channel attention mechanism and Conv 1×1.

[0032] Furthermore, the standard convolution layer and the classification module are derived from the MobileNetV3 backbone network. The standard convolution layer is used to extract the basic features of the input image and pass it to the subsequent layers of the network for further feature extraction and processing. The classification module includes a pooling layer and a fully connected layer, which are used to map the extracted fusion features into action recognition output results.

[0033] Furthermore, obtaining the spatial features and temporal features includes: inputting the basic features of the image extracted by the standard convolutional layer into the improved MobileNetV3 residual block, using the improved MobileNetV3 residual block to extract temporal features and spatial features and generate fusion features.

[0034] Furthermore, the temporal feature extraction module is a TAM (temporal adaptive module). The TAM is a modular module that can be integrated into a 2D CNN to generate a powerful video architecture with very little additional computational cost. The TAM consists of two branches: a local branch (L) and a global branch (G). The local branch uses temporal convolution to generate a position-sensitive importance map to enhance local features, while the global branch uses a fully connected layer to generate a position-invariant kernel for temporal aggregation.

[0035] Furthermore, the ECA channel attention mechanism replaces the SE channel attention mechanism in the original MobileNetV3 residual block to avoid dimensionality reduction, achieve local cross-channel interaction, and effectively capture the information of cross-channel interaction.

[0036] Furthermore, the improved MobileNetV3 residual block is divided into: a residual block using residual connection and a residual block not using residual connection. When the step size is 1 and the input and output channels are the same, the residual block uses residual connection, and the residual block contains a TAM temporal modeling module. When the step size is 2 or the input and output channels are different, the residual block does not use residual connection, and the residual block does not contain TAM.

[0037] Example

[0038] like Figure 1 As shown, this embodiment provides a lightweight action recognition method for deployable edge devices based on time series modeling, which specifically includes the following steps:

[0039] Step 1: key frame extraction;

[0040] The most representative key frames of K frames are extracted from the original video as input data of the action recognition model. The key frame extraction is a key frame extraction method based on image entropy and density clustering. Our key frame extraction method is mainly divided into three steps, namely 1) calculating image entropy, 2) finding local extreme points, and 3) performing density clustering.

[0041] Step 2: Input the extracted video key frames into a lightweight action recognition model based on the MobileNetV3 backbone network. The model includes a standard convolutional layer, an improved MobileNetV3 residual block, and a classification module. The MobileNetV3 backbone network has two branches, Small and Large. The Small branch contains 11 residual blocks, and the Large branch contains 15 residual blocks. In this embodiment, the Large branch will be taken as an example.

[0042] Step 3: The keyframes extracted in step 1 are first input into the standard convolutional layer described in step 2 to obtain preliminary feature extraction and then passed to the subsequent layers of the network for further feature extraction and processing.

[0043] Step 4: Input the preliminary features obtained in step 3 into the improved MobileNetV3 residual block described in step 2, the improved MobileNetV3 residual block includes: a temporal modeling module, Conv1×1, Conv 3×3, an ECA channel attention mechanism, and Conv 1×1. The temporal modeling module is TAM (temporal adaptive module).

[0044] Step 5: If the step size is 1 and the input and output channels are the same, the preliminary features obtained in step 3 are input into the TAM in step 4, and the video key frames are modeled in time series so that the model obtains temporal feature information, and then the features are input into the Conv 1×1 convolution layer described in step 4 to obtain further spatiotemporal features. If the step size is 2 or the input and output channels are different, the residual block does not contain TAM, and the preliminary features of step 3 will be input into the Conv 1×1 convolution layer described in step 4 to obtain further spatial features.

[0045] Step 6: Input the features obtained in step 5 into the Conv 3×3 convolutional layer described in step 4 to obtain further spatial features.

[0046] Step 7: Input the features obtained in step 6 into the ECA channel attention mechanism described in step 4 to achieve local cross-channel interaction and obtain further features.

[0047] Step 8: Input the features obtained in step 7 into the Conv 1×1 convolutional layer described in step 4 to obtain further spatial features.

[0048] Step 9: If the step size is 1 and the input and output channels are the same, the residual block uses residual connection to merge with the upper layer features and pass them to the lower layer module. If the step size is 2 or the input and output channels are different, the residual block does not use residual connection and passes the features to the lower layer module.

[0049] Step 10: Input the features obtained in step 9 into the lower residual block. Since we take the Large branch in the MobileNetV3 backbone network as an example, we will repeat steps 4 to 9 14 times until all 15 residual blocks are passed to obtain the final fused features.

[0050] Step 11: Input the fusion features obtained in step 10 into the classification module to obtain the final action recognition result.

[0051] In order to test the action recognition performance of step 11, this embodiment uses a test set to verify it. In this embodiment, Top-1 is used as an evaluation indicator, and the Top-1 accuracy is the ratio of the number of correct predictions of the model on the entire data set to the total number.

[0052] As an improvement to the lightweight action recognition model, this embodiment proposes a lightweight action recognition model that can be deployed on edge devices based on time series modeling. Among them, the key frame extraction method is used to extract the input frame, and the MobileNetV3 backbone network is used to extract the spatiotemporal features of the input frame at multiple scales, which is also convenient for deploying the model on the edge device. Through the improvement of the residual block, not only the 2D CNN that originally did not have the time series modeling capability has a better time series modeling capability, but also the accuracy of action recognition can be improved.

[0053] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A lightweight action recognition method for deployable edge devices based on time series modeling, characterized in that: include: Key frame extraction; Build a lightweight action recognition model based on the MobileNetV3 backbone network; The extracted key frames are input into the action recognition model to first obtain an abstract feature fused by spatial features and temporal features, and then based on the fused features, the final classification probability is obtained to obtain the action recognition result.

2. According to claim 1, the lightweight action recognition method for deployable edge devices based on time series modeling is characterized in that: The key frame extraction method is a key frame extraction method based on image entropy and density clustering. The key frame extraction method is mainly divided into three steps, namely 1) calculating image entropy, 2) finding local extreme points, and 3) performing density clustering.

3. According to claim 1, the lightweight action recognition method for deployable edge devices based on time series modeling is characterized in that: The lightweight action recognition model based on time series modeling and deployable on edge devices includes: a standard convolutional layer, an improved MobileNetV3 residual block, and a classification module. The improved MobileNetV3 residual block includes: a time series modeling module, Conv 1×1, Conv 3×3, an ECA channel attention mechanism, and Conv 1×1.

4. According to claim 1, the lightweight action recognition method for deployable edge devices based on time series modeling is characterized in that: The method for obtaining the spatial features and temporal features is to input the basic features of the image extracted by the standard convolutional layer into the improved MobileNetV3 residual block, and then use the improved MobileNetV3 residual block to extract the temporal features and spatial features and fuse them.

5. The lightweight action recognition method for deployable edge devices based on time series modeling according to claim 3 is characterized in that: The standard convolution layer and the classification module are both derived from the MobileNetV3 backbone network. The standard convolution layer is used to extract the basic features of the input image and pass it to the subsequent layers of the network for further feature extraction and processing. The classification module includes a pooling layer and a fully connected layer, which are used to map the fused features into the action recognition output results.

6. The lightweight action recognition method for deployable edge devices based on time series modeling according to claim 3 is characterized in that: The timing modeling module is TAM (time adaptive module). The TAM consists of two branches: a local branch (L) and a global branch (G). The local branch uses temporal convolution to generate a position-sensitive importance map to enhance local features, while the global branch uses a fully connected layer to generate a position-invariant kernel for temporal aggregation. The TAM not only enables the lightweight action recognition model to have better timing modeling capabilities and thus improve recognition accuracy, but also meets the deployment conditions of the model on edge devices while increasing a small amount of computational cost.

7. The lightweight action recognition method for deployable edge devices based on time series modeling according to claim 3 is characterized in that: The ECA channel attention mechanism replaces the SE channel attention mechanism in the original MobileNetV3 residual block to avoid dimensionality reduction, achieve local cross-channel interaction, and effectively capture cross-channel interaction information.

8. The lightweight action recognition method for deployable edge devices based on time series modeling according to claim 3 is characterized in that: The improved MobileNetV3 residual block is divided into a residual block using a residual connection and a residual block not using a residual connection. When the step size is 1 and the input and output channels are the same, the residual block uses a residual connection, and the residual block contains a temporal modeling module TAM. When the step size is 2 or the input and output channels are different, the residual block does not use a residual connection, and the residual block does not contain a TAM.

Images