A time series analysis method based on a double-branch attention mechanism TCN

Abstract

The application belongs to the field of time series analysis, and discloses a time series analysis method based on a double-branch attention mechanism TCN. Step 1: input embedding processing is performed on time series data of an input video; step 2: based on the input embedding of step 1, a double-branch attention time series module group is used for feature extraction; step 3: based on the feature data of step 2, a downstream task branch of video behavior analysis is used for processing. The method is used to solve the problem of insufficient long-distance dependence modeling capability of the prior art when processing time series data.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of time series analysis, and particularly relates to a time series analysis method based on a double-branch attention mechanism TCN. BACKGROUND

[0002] In time series data processing, long-term dependence is used to describe how long the state at the current time can be affected by the state at the time ago. In tasks involving time series processing such as speech recognition, text translation, and video behavior analysis, maintaining long-term dependence determines whether the content of the next time period can be accurately predicted, so establishing effective long-term dependence is the key. The traditional method of extracting dependence information is a non-supervised or semi-supervised method. BERT (Bidirectional Encoder Representations from Transformers) is a neural network for modeling long-term dependence in sequence data. Due to its large number of parameters and large network size, it can store long-distance dependence, while the application of recurrent neural networks (RNN) and long short-term memory networks (LSTM) with fewer parameters is often limited by their insufficient ability to model long-term dependence.

[0003] The attention mechanism in the Transformer has a strong ability to store long-term dependence, so it is very suitable for processing sequence data. BERT, a language model built by Transformer, has achieved the best results in natural language processing tasks. However, training the BERT model requires a large amount of computational resources and a large amount of training data, and the network convergence speed is also very slow, so the application of the network is limited. However, the attention mechanism can be applied to other time series models to capture long-term dependence information and further improve the prediction accuracy of the model.

[0004] RNN with a ring structure has the ability to model sequence data, but it is difficult to effectively extract long-term dependence information. Temporal Convolutional Networks (TCN) designed by applying convolution to sequence problem modeling can achieve or even exceed the accuracy of RNN models, but due to the use of convolution kernel calculation, the size of the receptive field is limited when processing time series problems, so the ability to obtain long-distance dependence is still insufficient.

[0005] Therefore, the existing neural network model still needs to improve the modeling ability of long-distance dependence when processing time series data. SUMMARY

[0006] The application provides a time series analysis method based on a double-branch attention mechanism TCN, to solve the problem of insufficient long-distance dependence modeling capability in the prior art when processing time series data.

[0007] The application provides an electronic device.

[0008] The application provides a computer-readable storage medium.

[0009] The application is achieved by the following technical solutions:

[0010] A time series analysis method based on a double-branch attention mechanism TCN, characterized in that the time series analysis method comprises the following steps:

[0011] Step 1: input embedding processing is performed on time series data of an input video;

[0012] Step 2: based on the input embedding of step 1, a double-branch attention time series module group is used for feature extraction;

[0013] Step 3: based on the feature data of step 2, a downstream task branch of video behavior analysis is used for processing.

[0014] A time series analysis method based on a double-branch attention mechanism TCN, wherein the step 1 is specifically that the input time series data can also be audio, text or image;

[0015] Time series data with a time length of t is embedded into a c-dimensional sparse space.

[0016] A time series analysis method based on a double-branch attention TCN, wherein the double-branch attention time series module of step 2 is used for feature extraction of time series data, and each time series module maps an input sequence from an input dimension to a higher dimension space;

[0017] The double-branch attention time series module group comprises four time series modules, and each time series module comprises a causal dilated residual connection branch and a global attention branch.

[0018] A time series analysis method based on a double-branch attention mechanism TCN, wherein the dimension change process of data in the double-branch attention time series module processing process is as follows: the input time series data of the network model is represented as {x1, x2, x3,..., x n}, denoted as X 1×n First, each node of the input data is embedded into a c-dimensional space, and the embedding result is denoted as X c×n , and then the data is processed through the time series module, and the dimension change during the four-layer time series module processing process is represented as

[0019] X1×n → X c1×n → X c2×n → X c3×n → X c4×n = X out .

[0020] A time series analysis method based on a double-branch attention mechanism TCN, the time series module needs to fuse the feature extraction results of the sparse causal residual branch and the global attention branch, represented as

[0021] X temporal_block = X causal_dilated + X global_residual

[0022] Wherein, X temporal_block represents the output data of the time series module, X causal_dilated represents the output data of the sparse causal residual branch, X global_residual represents the output data of the global attention branch.

[0023] A time series analysis method based on a double-branch attention mechanism TCN, the causal inflation residual connection branch is composed of 2 layers of causal inflation convolution and 2 residual connections; wherein, the causal inflation convolution part keeps the structure setting in the original TCN; the residual connection is composed of linear layer, softmax and normalization layer, which is used to extract the similarity and dependence between the output results of the causal convolution layer; the calculation process of this branch is represented as

[0024] X causal_dilated = f linear1 (X cd1 )+f linear2 (X cd2 )+ X cd2

[0025] Wherein, X causal_dilated represents the output data of the sparse causal residual branch, f linear1 , f linear2 represent the residual connection calculation, X cd1 , X cd2 represent the output data of the sparse causal convolution layer.

[0026] A time series analysis method based on a double-branch attention mechanism TCN, the global attention branch is specifically,

[0027] First, a one-dimensional convolution is used to extract features from time series data, and then the feature extraction results of the convolution layer are position encoded. Because the position information in the time series is the key to predict the subsequent state, the global position information in the sequence is combined during encoding;

[0028] Secondly, the feature with position coding is input to the multi-head attention layer to extract global dependency in time series data, and a residual connection containing one-dimensional convolution operation is added to improve the training convergence speed of global attention.

[0029] The calculation process of the branch is represented as

[0030] X global_residual = X global-attention + X conv1d

[0031] Wherein, X global_residual represents the output result of the global attention branch, X global_attention represents the output result of the global attention layer, and X conv1d represents the output result of the one-dimensional convolution operation.

[0032] A time series analysis method based on a double-branch attention mechanism TCN, the downstream task branch of the downstream task branch of step 3 adopts different branch processing according to different tasks.

[0033] The beneficial effects of the present application are:

[0034] The present application effectively solves the problem that RNN cannot perform large-scale parallel processing due to the fact that it can only process one element in a sequence at a time, and the problem of intensive computation caused by saving all intermediate results in task processing, thereby improving the convergence speed of model training.

[0035] The present application effectively solves the problem of insufficient long-term dependency modeling ability of TCN due to limited receptive field by introducing attention mechanism into TCN, and more effectively utilizes sequence information through position coding. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is the method flowchart of the present application.

[0037] Figure 2 is the attention mechanism TCN network model structure diagram of the present application.

[0038] Figure 3 is the video character behavior recognition effect diagram of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0040] A time series analysis method based on a double-branch attention mechanism TCN, characterized in that the time series analysis method comprises the following steps:

[0041] Step 1: input embedding processing is performed on the time series data of the input video;

[0042] Step 2: based on the input embedding of step 1, a double-branch attention time series module group is used for feature extraction;

[0043] Step 3: based on the feature data of step 2, a downstream task branch of video behavior analysis is used for processing.

[0044] A time series analysis method based on a double-branch attention mechanism TCN, wherein the step 1 is specifically that the input time series data can also be audio, text or image;

[0045] Time series data with a time length of t is embedded into a c-dimensional sparse space.

[0046] A time series analysis method based on a double-branch attention TCN, wherein the double-branch attention time series module of step 2 is used for feature extraction of time series data, and each time series module maps the input sequence from the input dimension to a higher dimensional space;

[0047] The double-branch attention time series module group comprises four time series modules, and each time series module comprises a causal inflation residual connection branch and a global attention branch.

[0048] A time series analysis method based on a double-branch attention mechanism TCN, wherein the dimension change process of data in the processing process of the double-branch attention time series module is as follows: the input time series data of the network model is represented as {x1, x2, x3, …, x n}, denoted as X 1×n , first, each node of the input data is embedded into a c-dimensional space, and the embedding result is denoted as X c×n , then the data is processed through the time series module, and the dimension change during the processing process of the four-layer time series module is represented as

[0049] X 1×n →X c1×n →X c2×n →X c3×n →X c4×n =X out .

[0050] A time series analysis method based on a double-branch attention mechanism TCN, wherein the time series module needs to perform feature fusion on the feature extraction results of the sparse causal residual branch and the global attention branch, and is represented as

[0051] X temporal_block = X causal_dilated + X global_residual

[0052] wherein, X temporal_block represents the output data of the time series module, X causal_dilated represents the output data of the sparse causal residual branch, X alobal_residual represents the output data of the global attention branch.

[0053] A time series analysis method based on a double-branch attention mechanism TCN, the causal inflation residual connection branch is composed of 2 layers of causal inflation convolution and 2 residual connections; wherein, the causal inflation convolution part keeps the structure setting in the original TCN; the residual connection is composed of a linear layer, a softmax and a normalization layer, which is used to extract the similarity and dependency between the output results of the causal convolution layer; the calculation process of this branch is represented as

[0054] X causal_dilated = f linear1 (X cd1 )+f linear2 (X cd2 )+X cd2

[0055] wherein, X causal_dilated represents the output data of the sparse causal residual branch, f linear1 , f linear2 represent the residual connection calculation, X cd1 , X cd2 represent the output data of the sparse causal convolution layer.

[0056] A time series analysis method based on a double-branch attention mechanism TCN, the global attention branch is used to consider the global semantic similarity information, and the meaning of different position input entities is emphasized by adding position information, specifically,

[0057] Firstly, a one-dimensional convolution is used to extract features from the time series data, and then the feature extraction results of the convolution layer are position encoded, because the position information in the time series is the key to predict the subsequent state, so the global position information in the sequence is combined during encoding;

[0058] Secondly, the feature with position encoding is input into the multi-head attention layer to extract the global dependency in the time series data, and a residual connection containing one-dimensional convolution operation is added to improve the training convergence speed of the global attention;

[0059] The calculation process of this branch is represented as

[0060] X global_residual = X global_attention+X conv1d

[0061] wherein, X global_residual represents the output result of the global attention branch, X global_attention represents the output result of the global attention layer, X conv1d represents the output result of the one-dimensional convolution operation.

[0062] A time series analysis method based on a double-branch attention mechanism TCN, the downstream task branch of step 3 adopts different branch processing according to different tasks. In addition to the downstream task branch of video behavior analysis, other different tasks such as speech recognition, text translation, image classification, etc. can be processed by using the corresponding downstream task branch.

[0063] An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete the communication among each other through the communication bus;

[0064] The memory is used to store a computer program;

[0065] The processor is used to execute the program stored on the memory, and realizes the method steps described above.

[0066] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to realize the method steps described above.

[0067] Taking the recognition of the behavior of the person in the video as an example: the purpose is to give a video, such as a person making salad in the picture, to identify and analyze the behavior appearing in the making process and the corresponding start and end time. The input video can be regarded as time series data composed of a series of image frames. First, the input time series data is embedded, and then the embedded results are extracted by the double-branch attention time series module group to model the long-range dependence between image frame data, so as to predict the action classification label corresponding to each image frame. On this basis, finally, the downstream task branch combines the sequence information to judge the start and end time of each behavior, and the final returned recognition result is all the behavior names appearing in the video and the corresponding behavior start and end time. The implementation effect is shown in the accompanying Figure 3 ​

Claims

1. A time series analysis method based on a two-branch attention mechanism TCN, characterized in that, The time series analysis method comprises the following steps: Step 1: input embedding processing is performed on time series data of an input video; Step 2: based on the input embedding of step 1, feature extraction is performed by using a double-branch attention time series module group; Step 3: based on the feature data of step 2, a downstream task branch of video behavior analysis is used for processing; The double-branch attention time series module of step 2 is used for feature extraction on time series data, and each time series module maps an input sequence from an input dimension to a higher dimension space; The double-branch attention time series module group comprises four time series modules, and each time series module comprises a causal dilated residual connection branch and a global attention branch; The dimension change process of the data in the processing of the double-branch attention timing module is as follows: the input timing data of the network model is represented as , denoted as . First, each node of the input data is embedded into dimensional space, and the embedding result is denoted as . Then, the data is processed through the timing module. The dimension change during the processing of the four-layer timing module is represented as 。 2. The time series analysis method based on the dual-branch attention mechanism TCN according to claim 1, characterized in that: The input time series data of step 1 can also be audio, text or image; Embedding time series data of length into a sparse space of dimensionality .

3. The time series analysis method based on the dual-branch attention mechanism TCN according to claim 1, characterized in that: The time series module needs to perform feature fusion on the feature extraction results of the sparse causal residual branch and the global attention branch, which is represented as wherein, represents output data of the timing module, represents output data of the sparse causal difference branch, represents output data of the global attention branch.

4. The time series analysis method based on the dual-branch attention mechanism TCN according to claim 1, characterized in that: The causal dilated residual connection branch is composed of two layers of causal dilated convolution and two residual connections; wherein the causal dilated convolution part maintains the structure setting in the original TCN; the residual connection is composed of a linear layer, a softmax and a normalization layer, and is used for extracting the similarity and dependency between the output results of the causal convolution layer; the calculation process of this branch is represented as wherein, denotes output data of the sparse causal residual branch, , denotes residual connection computation, , denotes output data of the sparse causal convolutional layer.

5. The time series analysis method based on the dual-branch attention mechanism TCN according to claim 1, characterized in that: The global attention branch is specifically First, a one-dimensional convolution is used to extract features from the time series data, and then the feature extraction results of the convolution layer are position encoded, because the position information in the time series is the key to predicting the subsequent state, so the global position information in the sequence is combined during encoding; Second, the feature with position encoding is input into a multi-head attention layer to extract the global dependency in the time series data, and a residual connection comprising one-dimensional convolution operation is added to improve the training convergence speed of the global attention; The calculation process of this branch is represented as wherein, represents an output result of the global attention branch, represents an output result of the global attention layer, represents an output result of the one-dimensional convolution operation.

6. The time series analysis method based on the dual-branch attention mechanism TCN according to claim 1, characterized in that: The downstream task branch of step 3 uses different branches according to different tasks.

7. An electronic device, comprising: It comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; The memory is used for storing a computer program; The processor is used for executing the program stored on the memory to realize the method steps of any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps of any one of claims 1-6.

Images