A video classification method based on accelerated Transformer model

Abstract

The present invention discloses a video classification method based on an accelerated Transformer model, the method comprising: firstly, preprocessing the original video to obtain a video feature sequence, adding a classification token vector at the beginning of the sequence; inputting the feature long sequence into the accelerated Transformer model, and finally converting the output classification token into the classification result of the video to be classified. Since the video data is an ultra-long sequence, using the original Transformer model will consume huge computing costs, so the present invention designs an accelerated Transformer model, uses Taylor's first-order expansion formula to expand the Softmax function, obtains a linear function close to Softmax, uses this linear function to calculate the attention weights between input elements, and uses the associative law of matrix multiplication to reduce the Transformer time and space complexity to linear complexity, thereby largely solving the problem of the limitation of the Transformer input sequence length and the slow training speed problem, making the Transformer model more suitable for video classification tasks.

Description

Technical Field

[0001] The present invention relates to the field of video technology, and specifically designs a video classification method based on an accelerated Transformer model. Background Art

[0002] With the advent of the 5G era, network communications have the characteristics of faster transmission speed, larger bandwidth and lower latency, allowing users to obtain a variety of video resources from more channels, which has further aggravated the growth of video data. Faced with massive amounts of video data, how to classify these videos so that users can quickly obtain the videos they are interested in has become one of the most important and challenging research hotspots in the field of computer vision.

[0003] Traditional video classification methods are mainly based on deep 2D or 3D convolutional neural network models, but their classification performance is always limited by the receptive field problem under the convolutional neural network architecture, and they cannot model long-range or global relationships. In recent years, the Transformer model based on the multi-head self-attention mechanism has become the most advanced model in many fields of machine learning due to its significant sequence modeling capabilities. Video data is sequential and requires context to understand, so we believe that the Transformer model with a long-range self-attention mechanism is more suitable for video modeling than convolutional neural networks.

[0004] The self-attention mechanism of Transformer can be expressed as follows:

[0005]

[0006] Where Q, K, V are the input sequence matrices Through the matrix and The generated projections are used to represent the query features, keyword features, and word value features of the input. N is the length of the input sequence, d is the dimension of the element embedding representation, and d k is the feature dimension of the projection. By performing column dot product calculations on Q and K, an N×N attention matrix can be generated. This attention matrix can be used to measure the correlation between query elements and keyword elements. By normalizing the attention matrix using the Softmax function, the attention weight matrix can be further obtained. Finally, the weight matrix is ​​multiplied back to the word-value feature V to obtain the weighted result of the word-value feature containing query information and keyword information. However, this self-attention mechanism brings high performance to the model but also comes with very high computational and memory costs. The main reason for this problem is the nonlinearity of the Softmax function and the fixed matrix multiplication order.

[0007] Under the action of the Softmax function, the Attention module must first calculate the product of matrix Q and matrix K, and then multiply it by matrix V. The time and space complexity of this process is O(d k N 2 ), that is, the computational complexity of the Transformer is proportional to the square of the input sequence length N. This is extremely expensive for long sequences such as video data. Even if the video data is sampled to shorten it to a medium-length sequence, the Transformer cannot be used in an environment with limited computing resources. How to retain the advantages of the Transformer for video modeling while reducing the high amount of computation and memory required when the Transformer is applied to video data is a daunting challenge. Summary of the invention

[0008] The present invention provides a video classification method based on an accelerated Transformer model, which retains the advantage of the self-attention mechanism in the Transformer model in calculating the global receptive field, and uses a linear function approximating the Softmax function and the associative law of matrix multiplication to solve the bottleneck problem of the Softmax function, thereby creating an accelerated Transformer model, which greatly reduces the computational cost of the Transformer model for video classification tasks, thereby achieving efficient video classification.

[0009] The objective of the present invention is achieved through the following technical solution: a video classification method based on an accelerated Transformer model, comprising the following steps:

[0010] (1) Preprocessing the original video to obtain a thumbnail and several image blocks corresponding to each frame;

[0011] (2) performing feature embedding on the image block and thumbnail obtained in step (1) to obtain a D-dimensional feature embedding vector of the image block and thumbnail;

[0012] (3) Using a learnable position embedding vector, encode the spatiotemporal position information of all image blocks in step (2) to obtain an image block feature embedding vector containing the position information;

[0013] (4) Add a learnable D-dimensional vector as a classification token at the beginning of the video vector sequence, and finally obtain the input sequence of the Transformer;

[0014] (5) The input sequence in step (4) is passed through the encoder of the Transformer model to obtain an output feature sequence that integrates the spatiotemporal contextual dependencies. The Transformer model uses an accelerated linear self-attention module: the Softmax function is linearly expanded using the Taylor expansion formula, and the normalization and non-negativity of the attention weight matrix are used to obtain a linear function that is extremely close to the Softmax function in accuracy. The linear function and the associative law of matrix multiplication are used to implement a linear accelerated self-attention mechanism;

[0015] (6) Separate the first bit vector of the feature sequence output in step (5), which is the classification token. Convert the classification token vector to the corresponding category and output the category of the video.

[0016] Furthermore, the pretreatment method in step (1) comprises the following steps:

[0017] (1-1) Preset the sampling time interval and evenly extract k video frames from the video;

[0018] (1-2) dividing each frame image obtained in step (1-1) into n image blocks of equal size and non-overlapping;

[0019] (1-3) For each frame image obtained in step (1-1), use downsampling to obtain the corresponding thumbnail as global information.

[0020] Furthermore, the feature embedding method in step (2) comprises the following steps:

[0021] (2-1) converting the thumbnail and several image blocks corresponding to the video frame into one dimension;

[0022] (2-2) Use trainable linear projection to project the image patch vector and thumbnail vector into D-dimensional space.

[0023] Furthermore, in step (5), replacing the Softmax function with a linear function that approximates the Softmax function comprises the following steps:

[0024] (5-1) Using the Taylor expansion formula, the approximate linear function of the Softmax function at x = 0 is as follows:

[0025]

[0026] in I N is the N-order identity matrix, 1 N is a vector of all 1s of length N;

[0027] (5-2) Using the approximate linear function obtained in step (5-1) to replace the Softmax function, the calculation expression of the self-attention mechanism of the Transformer model is transformed into the following:

[0028]

[0029] in Respectively represent the query features, keyword features and word value features of the input sequence, is the approximate Softmax matrix, and the constant C is the conditional factor;

[0030] (5-3) Since the matrix obtained after the Softmax function calculation is a weight matrix, the approximate Softmax matrix in step (5-2) should satisfy the characteristics of the weight matrix: normalization and non-negativity. Based on these two properties, the specific range of the constant C in step (5-2) is C ≥ 2 and where q i represents the i-th column vector in the matrix Q; on this basis, the value of the constant C is selected as:

[0031] C=d k

[0032] It can be proved by derivation that the error of the linear function approximating the Softmax function in step (5-1) is

[0033] (5-4) Substituting into step (5-1), the weight matrix expression is as follows:

[0034]

[0035] Substituting the weight matrix expression into step (5-2) and simplifying it, we can get the expression of the self-attention mechanism that approximates the Softmax polynomial function as follows:

[0036]

[0037] in v j Represents the j-th column vector of the matrix V.

[0038] Furthermore, the method of converting the classification token vector into the corresponding category in step (6) includes the following steps:

[0039] (6-1) Apply layer normalization (LN) to the classification token vector;

[0040] (6-2) Pass the normalized vector in step (6-1) through a fully connected network and a Softmax layer to obtain the probability distribution of each category;

[0041] (6-3) Through the probability distribution in step (6-2), the currently predicted video category is obtained.

[0042] The beneficial effects of the present invention are as follows: the present invention proposes a video classification method based on an accelerated Transformer model, which uses a Transformer model to replace the traditional CNN model to model the video sequence, uses a self-attention mechanism to capture global contextual information, and establishes long-distance dependencies between elements, thereby extracting more powerful features. In addition, the present invention constructs an accelerated Transformer structure, uses a linear function approximating Softmax to calculate the attention weights between input elements, and through the associative law of matrix multiplication, the time and space complexity of the self-attention module is linearly related to the length of the input sequence, which solves the problem of high computational cost and slow training speed of the original Transformer model when applied to long sequence inputs, making the Transformer model more suitable for video classification tasks. At the same time, the Softmax function approximation linear function proposed by the present invention can not only be used in the Transformer model, but also in other structures with Softmax Attention, which can tap a large number of potential applications for deep learning technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural diagram of the present invention;

[0044] Figure 2 This is a complexity comparison chart of the original Transformer and the accelerated Transformer of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0046] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0047] The structural diagram of the present invention is as follows Figure 1 As shown, the present invention proposes a video classification method based on an accelerated Transformer model, comprising:

[0048] (1) Preprocess the original video to obtain a thumbnail and several image blocks corresponding to each frame.

[0049] (1-1) Preset a sampling time interval, for example, 1 second, and evenly extract k video frames from the video. The number of video frames extracted from each video is determined according to the length of the video.

[0050] (1-2) Each frame of the image with a size of H×W obtained in step (1-1) is divided into n image blocks with a size of P×P, where

[0051] (1-3) The frame image of size H×W in step (1-1) is downsampled through the pooling layer to reduce its image size to s is the reduction ratio, and the thumbnail of each frame image is obtained as global information.

[0052] (2) Perform feature embedding on the image block and thumbnail obtained in step (1) to obtain a D-dimensional feature embedding vector of the image block and thumbnail.

[0053] (2-1) The image block obtained in step (1) is converted into one dimension to obtain its corresponding vector representation, namely:

[0054]

[0055] Where p = {1, ..., n} represents the spatial position of the image block, and t = {1, ..., k} represents the index of the frame where the image block is located;

[0056] (2-2) The thumbnail obtained in step (1) is converted into one dimension to obtain its corresponding vector representation, namely:

[0057]

[0058] Where t = {1, ..., k} represents the index of the frame where the thumbnail is located;

[0059] (2-3) The image block vector x obtained in step (2-1) (p，t) It is linearly mapped to a D-dimensional embedding vector through a trainable linear projection Right now:

[0060] z (p，t) =Ex (p，t)

[0061] in represents a learnable weight matrix;

[0062] (2-4) The thumbnail vector ht obtained in step (2-2) is also linearly mapped to a D-dimensional embedding vector through a trainable linear projection Right now:

[0063] h′ t =E′h t

[0064] in Represents a learnable weight matrix.

[0065] (3) Using the learnable position embedding vector, encode the spatiotemporal position information of all image blocks in step (2) to obtain the image block feature embedding vector containing the position information; specifically:

[0066] Preparing a learnable position embedding vector The image block embedding vector z obtained in step (2-3) (p，t) Add together to get the image block embedding vector with spatiotemporal position information Right now:

[0067] z′ (p，t) =z (p，t) +p (p，t)

[0068] (4) The D-dimensional feature embedding vector of the thumbnail obtained in step (2) and the image block feature embedding vector containing position information obtained in step (3) form a video vector sequence, and a learnable D-dimensional vector z is added to the beginning of the video vector sequence. cls As a classification token, the final input sequence Z of Transformer is

[0069]

[0070] Where N = k(n+1)+1, and the superscript 0 represents the initial input sequence of Transformer.

[0071] (5) The input sequence in step (4) is passed through the encoder of the Transformer model to obtain an output feature sequence that integrates the spatiotemporal contextual dependencies. In addition, since the original Transformer cannot model such an extremely long input sequence, we replace the Softmax function in the self-attention mechanism of the Transformer model with a linear function that approximates the Softmax function.

[0072] (5-1) The encoder of the Transformer model is composed of L stacked encoding blocks. The encoding block contains a multi-head self-attention layer and a feedforward neural network layer. At the beginning of each encoding block, we need to represent the encoding vector sequence Z output by the previous encoding block. l-1 Multiply by W Q , W K , W V The query vector feature projection, key vector feature projection and value vector feature projection corresponding to each element in the sequence are calculated for the self-attention mechanism, that is:

[0073]

[0074]

[0075]

[0076] Where Q, K, and V are used to represent Z respectively. l-1 query features, keyword features, and word value features; d k is the characteristic dimension of the projection;

[0077] (5-2)The Softmax function expression is as follows:

[0078]

[0079] in x j represents the j-th element of the input vector x;

[0080] The derivative expression obtained by derivation of the Softmax function through vector differentiation is as follows:

[0081]

[0082] Where D i S j Indicates S j For element x i Derivation;

[0083] Introducing the shock function δ ij , which is expressed as follows:

[0084]

[0085] Then the Softmax derivative function can be expressed as:

[0086] D i S j =S i (δ ij -Sj )

[0087] (5-3) Due to So the Jacobian matrix at x=0 can be expressed as follows:

[0088]

[0089] Among them I N is the N-order unit matrix, 1 N is a unit vector of length N;

[0090] According to Taylor's theorem, when the function Softmax is differentiable at x = 0, there is always a first-order linear function similar to Softmax in the neighborhood of x = 0.

[0091]

[0092] (5-4) Substitute the Jacobian matrix J(0) at x = 0 into The extremely approximate linear representation of the Softmax function at x = 0 can be obtained as follows:

[0093]

[0094] in I N is the N-order unit matrix, 1 N is a unit vector of length N;

[0095] (5-5) Use the linear function in step (5-4) Instead of the Softmax function used to calculate the attention weight matrix in the Transformer model, the self-attention module in the Transformer model is approximately replaced by a linear self-attention module:

[0096]

[0097] in They represent query features, keyword features, and word value features respectively, and the constant C is the conditional factor;

[0098] (5-6) The approximate Softmax matrix in step (5-5) should satisfy the properties of the weight matrix: normalization and non-negativity. Based on these two properties, the range of the constant C is C ≥ 2 and where q i Represents the i-th column vector in the matrix Q, and then sets the specific value based on it:

[0099] C=d k

[0100] At this time, the linear function The error that approximates the Softmax function is

[0101] (5-7) The expression of the self-attention mechanism that accelerates the Transformer model is as follows:

[0102]

[0103] in v j Represents the j-th column vector of the matrix V.

[0104] The complexity comparison between the original Transformer and the accelerated Transformer of the present invention is as follows: Figure 2 As shown in the figure, the left half describes the matrix calculation order of the self-attention module in the Transformer. First, the Softmax attention weight is calculated to obtain an N×N Attention matrix, and then the Attention matrix is ​​combined with the N×d k According to the complexity calculation rule of matrix multiplication, the complexity of Transformer is O(d k N 2 ). The right half describes the matrix calculation order of the linear self-attention module in this invention. Since we linearize Softmax, there is no need to calculate the size of N×d k Instead of focusing on the dot product of the Q matrix and the K matrix, we use the associative law of matrix multiplication to first calculate the transpose of the K matrix (size d k ×N) and V matrix (size N×d k ) to obtain a value of size d k ×d k The complexity of this process is O(d k 2 N), and then calculate the Q matrix and the above d k ×d k The complexity of the matrix multiplication is also O(d k 2 N), so the complexity of the linear attention module of the present invention is O(d k 2 N). It is worth mentioning that although the linearized self-attention module of the present invention changes the order of matrix multiplication, such an operation does not affect the essence of self-attention between elements. In other words, the accelerated Transformer model constructed by the present invention significantly reduces the time and space complexity, making the Transformer more suitable for video classification tasks;

[0105] (5-8) Use the linear self-attention mechanism constructed in step (5-7) to perform spatiotemporal self-attention calculations on the matrix Q, matrix K, and matrix V obtained in step (5-1), and then use the feature encoding sequence containing spatiotemporal dependencies as the input sequence of the next encoding block after passing it through the feedforward neural network layer, until the last encoding block outputs the feature sequence of the video.

[0106] (6) Separate the first bit vector of the feature sequence output in step (5), which is the classification token. Convert the classification token vector to the corresponding category and output the category of the video.

[0107] (6-1) Use layer normalization (LN) on the classification token vector;

[0108] (6-2) Pass the normalized vector in step (6-1) through a fully connected layer and a Softmax layer to obtain the probability distribution of multiple category labels we set in advance;

[0109] (6-3) According to the probability distribution in step (6-2), select the category label with the highest probability as the current predicted video category. Figure 1 As shown in the figure, we obtained the probability distribution of 400 video categories including arm wrestling, doing yoga, blowing balloons, walking the dog, etc., and finally selected "blowing balloons" with the highest probability score as the classification result of the input video.

[0110] The above is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above disclosed methods and technical contents without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A video classification method based on an accelerated Transformer model, characterized in that: include: (1) Preprocessing the original video to obtain a thumbnail and several image blocks corresponding to each frame; (2) performing feature embedding on the image block and thumbnail obtained in step (1) to obtain a D-dimensional feature embedding vector of the image block and thumbnail; (3) Using a learnable position embedding vector, encode the spatiotemporal position information of all image blocks in step (2) to obtain an image block feature embedding vector containing the position information; (4) Add a learnable D-dimensional vector as a classification token at the beginning of the video vector sequence, and finally obtain the input sequence of the Transformer; (5) The input sequence in step (4) is passed through the encoder of the Transformer model to obtain an output feature sequence that integrates the spatiotemporal contextual dependencies. The Transformer model uses an accelerated linear self-attention module: the Softmax function is linearly expanded using the Taylor expansion formula, and the normalization and non-negativity of the attention weight matrix are used to obtain a linear function that is extremely close to the Softmax function in accuracy. The linear function and the associative law of matrix multiplication are used to implement a linear accelerated self-attention mechanism; (6) Separate the first bit vector of the feature sequence output in step (5), which is the classification token. Convert the classification token vector to the corresponding category and output the category of the video.

2. The video classification method based on the accelerated Transformer model according to claim 1, characterized in that: The pretreatment method in step (1) comprises the following steps: (1-1) Preset the sampling time interval and evenly extract k video frames from the video; (1-2) dividing each frame image obtained in step (1-1) into n image blocks of equal size and non-overlapping; (1-3) For each frame image obtained in step (1-1), use downsampling to obtain the corresponding thumbnail as global information.

3. The video classification method based on the accelerated Transformer model according to claim 1, characterized in that: The feature embedding method in step (2) comprises the following steps: (2-1) converting the thumbnail and several image blocks corresponding to the video frame into one dimension; (2-2) Use trainable linear projection to project the image patch vector and thumbnail vector into D-dimensional space.

4. The video classification method based on the accelerated Transformer model according to claim 1, characterized in that: Replacing the Softmax function with a linear function approximating the Softmax function in step (5) comprises the following steps: (5-1) Using the Taylor expansion formula, the approximate linear function of the Softmax function at x = 0 is as follows: The input vector N is the length of the initial input sequence of the Transformer model, I N is the N-order identity matrix, 1 N is a vector of all 1s of length N; (5-2) Using the approximate linear function obtained in step (5-1) to replace the Softmax function, the calculation expression of the self-attention mechanism of the Transformer model is transformed into the following: in Respectively represent the query features, keyword features and word value features of the input sequence, d k is the characteristic dimension of Q, K, V, is the approximate Softmax matrix, and the constant C is the conditional factor; (5-3) According to step (5-2), the approximate Softmax matrix should satisfy the characteristics of the weight matrix: normalization and non-negativity. The specific range of the constant C in step (5-2) is calculated to be C ≥ 2 and where q i represents the i-th column vector in the matrix Q; (5-4) Substituting into step (5-1), the weight matrix expression is as follows: Substitute the weight matrix expression into step (5-2) and simplify it to obtain the expression of the self-attention mechanism that approximates the Softmax polynomial function as follows: in v j Represents the j-th column vector of the matrix V.

5. The video classification method based on the accelerated Transformer model according to claim 4, characterized in that: In the step (5-3), the value of the constant C is selected as C=d k .

6. The video classification method based on the accelerated Transformer model according to claim 1, characterized in that: The method of converting the classification token vector into the corresponding category in step (6) comprises the following steps: (6-1) Apply layer normalization to the classification token vector; (6-2) Pass the normalized vector in step (6-1) through a fully connected network and a Softmax layer to obtain the probability distribution of each category; According to the probability distribution, the currently predicted video category is obtained.

Images