A video recognition method based on multilayer perceptron fine-grained tensor decomposition compression

Abstract

The application belongs to the technical field of computer vision processing, and particularly relates to a video recognition method based on multilayer perceptron fine-grained tensor decomposition compression, which is characterized by processing the trained multilayer perceptron to obtain the compressed multilayer perceptron. In the process, firstly, the fine-grained tensor decomposition form is used to compress the full connection weight in the multilayer perceptron; then, a reconstruction-free algorithm between the input vector and the weight after the fine-grained tensor decomposition is designed, so that the algorithm output is equal to the original vector weight matrix multiplication output; after the compression of all the full connection weights to be compressed is completed, the obtained multilayer perceptron is fine-tuned to obtain the final result. The method achieves the purpose of fast calculation on the basis of compression, and improves the deployability and computational efficiency of the multilayer perceptron in processing video recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision processing, and particularly relates to a video recognition method based on multi-layer perceptron (full connection neural network) fine-grained tensor decomposition compression. BACKGROUND

[0002] In recent years, video recognition is widely applied in the fields of equipment manufacturing, monitoring security, unmanned driving and the like. With the development and maturity of deep learning technology, a recurrent neural network (RNN) and a Transformer neural network model composed of a multi-layer perceptron and having a time series data processing capability become one of the main choices of video recognition technology. However, the current mainstream RNN, Transformer and other multi-layer perceptron models often rely on a large scale to obtain strong expression capability, so that the space and time complexity of the neural network model is higher and higher, and the storage and computing resources consumed are more and more. In the embedded environment such as a resource-limited mobile device, the multi-layer perceptron must be compressed through certain technical means.

[0003] In 2013, Denil et al. and Sainath et al. respectively proved in the documents Predicting parameters in deep learning and Low-rank matrix factorization for deep neural network training with high-dimensional output targets that the deep neural network structure is actually redundant, that is, the size of the deep neural network can be reduced through some weight compression means without causing obvious accuracy loss. Therefore, many methods have been proposed to compress the deep neural network.

[0004] Among them, tensor decomposition is one of the widely used compression methods, specifically, each layer weight of the multi-layer perception is regarded as a high-order tensor, and then the mature tensor decomposition mathematical method is used to decompose it into the form of product of multiple small tensors, so that the number of multi-layer perception weight parameters is reduced. These tensor decomposition methods include: tensor train (TT) decomposition, tensor ring (TR) decomposition, hierarchical Tucker (HT) decomposition, block term (BT) decomposition and the like. Different tensor decompositions have their own mathematical expressions and high compression rates. Since Novikov et al. published the document Tensorizing neural networks in 2015, these tensor decomposition methods have been applied in multi-layer perception and the deep neural network composed of the same, and good compression effects have been achieved.

[0005] However, the existing multi-layer perception tensor decomposition compression method has not solved the problem of fast calculation between the compressed weight in the form of tensor decomposition and the input vector. The main reason is that the purpose of calculation between the input vector and the weight in the form of tensor decomposition is to convert the input dimension into the output dimension through the contraction operation, that is, to eliminate the input module and add the output module. At the same time, the weight in the form of tensor decomposition is composed of multiple factor tensors or factor matrices, which decompose and scatter the input and output modules, so that the input often needs to be constantly reconstructed in order to adjust the input module to be eliminated to the position matched with the corresponding module of the factor to be calculated. However, in the actual calculation process, the data in the computer memory is frequently reconstructed, and new storage space must be constantly applied and data must be copied in the new order, which makes the existing multi-layer perception tensor decomposition compression method actually less efficient than the ordinary multi-layer perception. SUMMARY

[0006] (I) Technical problems to be solved

[0007] The technical problem to be solved by the present application is how to solve the problem that the neural network composed of the existing multi-layer perception used in the video recognition technology is large in scale, and the related tensor decomposition compression means cannot meet the fast calculation in the resource limited environment.

[0008] (II) Technical solutions

[0009] To solve the above technical problems, the present application provides a video recognition method based on multi-layer perception fine-grained tensor decomposition compression, which comprises:

[0010] Step A: taking the total frame number T o Randomly collect video data of sensors such as cameras, cameras and the like, denoted as where Wo H represents the width of each video image frame. o T represents the height of each video frame. o That is, the total number of frames in the video data, which is also the duration, C o This indicates the number of channels in each frame of video image;

[0011] Step B: Construct a multilayer perceptron for video data Perform identification;

[0012] Step C: Output the results of video data recognition.

[0013] Step B includes:

[0014] Step B1: For the multilayer perceptron used to recognize video data, define the input dimension of each frame as D. o =W o H o C o Its time dimension is defined as T. o Multilayer perceptron is denoted as The corresponding video data is segmented into T o There are D input vectors, each with a length of D. o Output to step B2;

[0015] Step B2: Process the multilayer perceptron output from step B1. The trained multilayer perceptron is denoted as […]. Output to step B3;

[0016] Step B3: Apply fine-grained tensor decomposition and compression of the multilayer perceptron trained in step B2. The process is performed to obtain a fully compressed multilayer perceptron.

[0017] Step B4: Complete the compressed multilayer perceptron using step B3. The video data is identified, and the results of the video data recognition are generated for output.

[0018] Step B3 includes:

[0019] Step 1: Train the multilayer perceptron Let one of the fully connected layers be y = f(xW), and let f(•) be the nonlinear activation function of that layer. Read the weight matrix of the fully connected layer. M-dimensional input vector and N-dimensional output vector

[0020] Step 2: Arrange M and N according to the constraints and If the factorization is into the product of I integers, then the weight matrix The compression can be done in the form of fine-grained tensor decomposition, i.e.,

[0021]

[0022] where, is the i-th factor matrix of the k-th decomposition of the group A, is the i-th factor matrix of the k-th decomposition of the group B, is the rank of the k-th decomposition of the group A, is the rank of the k-th decomposition of the group B, and is the identity diagonal tensor, vec(·) is the transformation that stretches a tensor into a vector, the operator symbol · denotes the contraction, and the operator symbol ° denotes the outer product; where k ∈ {1, 2, …, K} and i ∈ {1, 2, …, I}; K and I are integers;

[0023] Step 3: define the input vector x and the weights after fine-grained tensor decomposition between x and xW, so that its output is equal to the output of xW;

[0024] After step 3 is completed, if there are still full connection layers in the multi-layer perceptron that have not been compressed, then jump to step 1 to compress the next full connection layer, if all the full connection layers to be compressed in the multi-layer perceptron have been compressed, a new multi-layer perceptron Step 4 is entered;

[0025] Step 4: use the error back propagation algorithm to train the new multi-layer perceptron that has completed compression for a limited number of times to fine-tune it to the best performance.

[0026] In step 2, M and N are factorized by the same number of factors I, and each factor m i or n i is not less than 2.

[0027] In step 2, the factors of the group A after the contraction operation · will eliminate all the ranks to obtain an I-order tensor with the size of m1×m2×…×m I , and the factors of the group B after the contraction operation · will eliminate all the ranks to obtain an I-order tensor with the size of n1×n2×…×n Itwo first order tensors are stretched into M-dimensional and N-dimensional vectors by vec(·), and the two vectors are outer product to get the k-th M x N matrix, and all K matrices are added to restore the original weight matrix

[0028] In step 2, the weight in the form of fine-grained tensor decomposition can be completely restored to the form of the weight matrix before compression through the contraction operation·, the outer product operation°, and the summation∑, which shows that the transformation of the weight matrix into the form of fine-grained tensor decomposition is correct in dimension transformation.

[0029] In step 3, the reconstruction-free algorithm★ specifically refers to:

[0030] First, the input vector x is folded into a first order tensor according to the constraint Then, the factor matrices with odd index i in groups A and B and the factor matrices with the next even index i+1 are contracted to eliminate the rank and That is, define and define

[0031] Then, starting from i=1, let x left contract Eliminate the leftmost two modules m i ×m i+1 , right outer product Supplement the rightmost two modules n i ×n i+1 , and get K intermediate results, that is Next, add K x k , and the result is equal to i+2 and Continue to calculate the left contraction and right outer product and then add them, and so on until all m i are eliminated; finally get the first order tensor Stretch it into an N-dimensional output vector and the vector value is equal to the direct calculation of xW.

[0032] In step 3, the unit diagonal tensor and have no effect on the calculation result.

[0033] In step 3, the input vector x is folded into a first order tensor according to the constraint ​​The process does not change the storage location of the data in the actual computer memory, but only the index pointing to the specific data is changed according to M=m1xm2...xm I Sequentially arrange; the x left contraction The two modules of the x leftmost position are exactly The module m of x i Xm i+1 Without reconstruction of x or , the contraction calculation can be directly performed; the right outer product Without reconstruction of x or Directly supplement the module n of x The rightmost position in the form of outer product Xn i Xn i+1 Without reconstruction of x or ; the I-order tensor Is stretched into an N-dimensional output vector The specific data index is rearranged according to n1xn2...xn I =N again, without reconstruction; therefore, there is no reconstruction operation in the algorithm.

[0034] In step 4, the original multi-layer perception Has been compressed into a new multi-layer perception Although the factor matrix in the form of fine-grained tensor decomposition can theoretically guarantee the recovery of the original weight matrix, there will be cumulative error after the multi-layer full connection compression, so the new multi-layer perception Need to be retrained for a certain number of times to achieve the required performance.

[0035] (Three) beneficial effects

[0036] In view of the problem that the neural network composed of the existing multi-layer perception used in the video recognition technology is large in size, and the related tensor decomposition compression means cannot meet the fast calculation in the resource limited environment, the present application provides a multi-layer perception fine-grained tensor decomposition compression method, which converts the full connection layer weight matrix in the multi-layer perception into a fine-grained multi-path grouping factor matrix outer product summation mode, and enables the calculation between the weight in the form of fine-grained tensor decomposition and the input vector to be free of reconstruction, so as to achieve the purpose of fast calculation on the basis of compression and improve the deployability and calculation efficiency of the multi-layer perception in processing video recognition.

[0037] Specifically, the technical scheme of the present application uses fine-grained tensor decomposition to compress the weight matrix in the multi-layer perception, and provides a reconstruction-free algorithm between the input vector and the weight in the form of fine-grained tensor decomposition, so that the multi-layer perception can obtain a smaller neural network size after compression and can also perform fast calculation with the input. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a flow chart of the technical solution of the present application;

[0039] Figure 2 is a topological structure diagram of fine-grained tensor decomposition;

[0040] Figure 3 is a schematic diagram of the storage order of matrix elements in memory;

[0041] Figure 4 is a schematic diagram of the reconstruction-free algorithm between input data and fine-grained tensor decomposition form weights;

[0042] Figure 5 is an RNN structure diagram for tensor decomposition compression experiments on the UCF11 dataset. DETAILED DESCRIPTION

[0043] In order to make the purpose, content, and advantages of the present application clearer, the specific embodiments of the present application are described in further detail below in combination with the drawings and examples.

[0044] To solve the above technical problems, the present application provides a multi-layer perceptron fine-grained tensor decomposition compression-based video recognition method, which comprises the following steps:

[0045] Step A: divide the video data into T o frames, randomly collect video data of sensors such as cameras and cameras, denoted as where W o represents the width of each frame of video image, H o represents the height of each frame of video image, T o , i.e. the total number of frames of video data, i.e. the time length, C o represents the number of channels of each frame of video image;

[0046] Step B: construct a multi-layer perceptron to recognize the video data ;

[0047] Step C: output the result after recognizing the video data.

[0048] The step B comprises:

[0049] Step B1: define the input dimension of each frame of the multi-layer perceptron used to recognize the video data as D o = W o H o C o , define the time dimension as T o , and denote the multi-layer perceptron as The corresponding video data is divided into T oinput vectors, each of which has a length of D o , and output to step B2;

[0050] Step B2: training the multi-layer perceptron output from step B1 , and the trained multi-layer perceptron is denoted as , and output to step B3;

[0051] Step B3: processing the multi-layer perceptron trained in step B2 using multi-layer perceptron fine-grained tensor decomposition compression to obtain a compressed multi-layer perceptron

[0052] Step B4: using the compressed multi-layer perceptron in step B3 to identify video data to generate a result of the identified video data to be output.

[0053] The step B3 includes:

[0054] Step 1: reading the trained multi-layer perceptron , and reading the weight matrix of the full connection as y = f(xW), where f(·) denotes the nonlinear activation function of the layer, and M-dimensional input vector , and N-dimensional output vector

[0055] Step 2: factorizing M and N into the product of I integers according to constraints and , the weight matrix can be compressed in the form of fine-grained tensor decomposition, that is:

[0056]

[0057] wherein, is the i-th factor matrix of the k-th decomposition in group A, is the i-th factor matrix of the k-th decomposition in group B, is the rank of the k-th decomposition in group A, is the rank of the k-th decomposition in group B, and is a unit diagonal tensor, vec(·) is a transformation that stretches a tensor into a vector, the operation symbol · represents contraction, and the operation symbol represents the outer product; wherein k ∈ {1, 2, …, K}, i ∈ {1, 2, …, I}; K and I are integers;

[0058] Step 3: defining the input vector x and the weight after fine-grained tensor decomposition​​ reconstruction algorithm★, so that its output is equal to the output of xW;

[0059] After step 3 is completed, if there are still full connection layers in the multilayer perceptron that have not been compressed, then jump to step 1 to compress the next full connection layer, if all the full connection layers to be compressed in the multilayer perceptron have been compressed, a new multilayer perceptron is obtained Step 4 is entered;

[0060] Step 4: using the error back propagation algorithm to train the new multilayer perceptron with a limited number of times to fine-tune it to the best performance.

[0061] In step 2, M and N are factorized by the same number of factors I, and each factor m i or n i is not less than 2.

[0062] In step 2, the factors of group A After the contraction operation, all ranks are eliminated An I-order tensor with size m1×m2×…×m I is obtained, and the factors of group B After the contraction operation, all ranks are eliminated An I-order tensor with size n1×n2×…×n I is obtained, and the two I-order tensors are stretched into M-dimensional and N-dimensional vectors respectively by vec(·), and the two vectors are subjected to outer product operation An M×N matrix of the kth path is obtained, and all K matrices are added to recover the original weight matrix

[0063] In step 2, the weight in the form of fine-grained tensor decomposition, its factor matrix is contracted by contraction operation, outer product operation and summation ∑, can be completely restored to the form of weight matrix before compression, which shows that the transformation of weight matrix into the form of fine-grained tensor decomposition is correct in dimension transformation.

[0064] In step 3, the reconstruction-free algorithm★ specifically refers to:

[0065] First, fold the input vector x according to the constraint into an I-order tensor Then, contract the factor matrices of group A and group B with all odd i and their adjacent even i+1 with larger index values to eliminate the rank and That is, define and define

[0066] Then, starting from i=1, let x be left-bound. Eliminate the two leftmost modulo m i ×m i+1 Right outer product Add the two modulo n on the far right i ×n i+1 A total of K intermediate results were obtained, that is Next, K Add them together, and the result is the same as what i+2 represents. and Continue performing the calculation of left contraction and right outer product followed by summation, and so on until all m are calculated. i All were eliminated; finally, an I-order tensor was obtained. Stretch it into an N-dimensional output vector Furthermore, the vector value is equal to the directly calculated xW.

[0067] In step 3, the unit diagonal tensor and It has no impact on the calculation results.

[0068] In step 3, the input vector x is determined according to constraints. Folded into an I-order tensor This process does not change the storage location of the data in the actual computer memory; it only modifies the indexes pointing to the specific data according to M = m1 × m2 × … × m I Arranged sequentially; the χ left-indented The two moduli at the leftmost position of χ are exactly... The model m i ×m i+1 No need for x or Reconstruction allows for direct shrunk calculations; the right outer product of x... This refers to directly adding the product in the form of a product at the rightmost position of x. modulus n i ×n i+1 There is no need to consider x or Reconstruction is performed; the first-order tensor Stretched into an N-dimensional output vector The specific data index is in the order of n1×n2×…×n I =N can be rearranged again without reconstruction; therefore, there is no reconstruction operation in the algorithm ★.

[0069] In step 4, the original multilayer perceptron Fine-grained tensor decomposition and compression have become a new type of multilayer perceptron. Although the factor matrix in the form of fine-grained tensor decomposition can guarantee the recovery of the original weight matrix in theory, there is accumulated error after the multi-layer full connection compression, so the new multi-layer perceptron A certain number of retraining needs to be performed to achieve the required performance.

[0070] In the above technical solution, steps A, B1, B2, B4 and C are all conventional technical means, and the innovation of the present application is mainly concentrated in step B3. Therefore, specific embodiments are provided for this link as follows.

[0071] Example 1

[0072] This embodiment is directed to step B2, and the technical solution adopted includes the following 4 steps as shown in the figure: Figure 1

[0073] Step 1: For the multi-layer perceptron that has been trained Let one of the full connections be y=f(xW), f(•) be the nonlinear activation function of this layer, and read the full connection weight matrix M-dimensional input vector and N-dimensional output vector

[0074] The so-called full connection means that there is a weight connection between each input neuron and each output neuron of the layer. All input neurons can be represented as an input vector x, and all output neurons can be represented as an output vector y. After each element in x and y is connected, all weight connections can form a weight matrix W. This step generally starts from the first full connection layer that needs to be compressed in the multi-layer perceptron. If other full connection layers still need to be compressed, then return to this step again.

[0075] Step 2: According to the constraints and factorize M and N into the product of I integers, then the weight matrix can be compressed in the form of fine-grained tensor decomposition, that is:

[0076]

[0077] wherein, is the i(th) factor matrix of the k(th) decomposition of group A (k∈{1,2,…,K}, i∈{1,2,…,I}), is the i(th) factor matrix of the k(th) decomposition of group B, is the rank of the k(th) decomposition of group A, is the rank of the k(th) decomposition of group B, and ​It is a unit diagonal tensor. vec(·) is a transformation that stretches the tensor into a vector. The · operator represents contraction, and the ° operator represents outer product.

[0078] In this step, M and N are factored with the same number of factors I, and each factor m i or n i Not less than 2, i.e., m i ≥2,n i ≥2. If there is a factor less than 2, that is, equal to 1, then its corresponding factor matrix... or It will become length or The vector.

[0079] The most crucial step in this process is clarifying the fine-grained tensor decomposition topology, which can be achieved by combining [the following steps]. Figure 2 A detailed description is provided. The factors of group A are... After the shrinking operation, all ranks will be eliminated. The resulting dimensions are m1×m2×…×m I The first-order tensor, the factors of group B After the shrinking operation, all ranks will be eliminated. The resulting size is n1×n2×…×n I These two I-order tensors are stretched into M-dimensional and N-dimensional vectors by vec(·), respectively. These two vectors are then subjected to an outer product operation ° to obtain the k-th path M×N matrix. Adding all K-path matrices together restores the original weight matrix. That is, the weights of the fine-grained tensor decomposition form described in this step, whose factor matrix is ​​obtained through the shrinking operation and the outer product operation. And by summing ∑, the weight matrix can be completely restored to its original form before compression, indicating that the transformation of the weight matrix into the fine-grained tensor decomposition form is correct in terms of dimensionality transformation.

[0080] Obtained from the weight matrix W Figure 2 The process of obtaining the fine-grained tensor decomposition form can generally be regarded as an optimization estimation problem. That is, the optimization problem is defined as follows:

[0081]

[0082] The optimization objective of the above equation is to minimize the value of the norm ||·||. A smaller norm value means that the fine-grained tensor decomposition form can more accurately estimate the original weight matrix W. The optimization problem described above can be solved using common methods such as least squares and gradient descent.

[0083] Step 3: Define the input vector x and the weights after fine-grained tensor decomposition. reconstruction-free algorithm★, so that its output is equal to that of xW.

[0084] In this step, the reconstruction-free algorithm★ specifically refers to:

[0085] First, the input vector x is folded into an I-order tensor according to the constraint Then, all factor matrices with odd subscript i in groups A and B and their adjacent even i+1 factor matrices with larger subscript values are combined to eliminate the rank and that is, and Then, starting from i=1, let x left combine Eliminate the leftmost two modules m i ×m i+1 , right outer product Supplement the rightmost two modules n i ×n i+1 , a total of K intermediate results, that is Next, add the K The result obtained is equal to the and Continue to do left combination and right outer product addition, and so on until all m i are eliminated; finally, an I-order tensor is obtained, which is stretched into an N-dimensional output vector and the vector value is equal to that of directly calculating xW. The unit diagonal tensor and have no effect on the calculation result.

[0086] The concise pseudo-code description of the reconstruction-free algorithm★ is shown in the following table:

[0087]

[0088]

[0089] The input vector x is folded into an I-order tensor according to the constraint This process does not change the storage location of data in the actual computer memory, but only rearranges the indexes pointing to specific data according to M=m1×m2×…×m I in turn. Taking Figure 3 as an example, a matrix has two directions of coordinates in theory, but it is actually one-dimensional in computer memory; that is Figure 3 The 3×4 matrix is stretched into a vector, or a vector of length 12 is arranged according to​​Figure 3 Folding the data into a 3x4 matrix in sequence does not require changing the actual storage location of the data. Therefore, as long as the order of the data remains unchanged, there is no need for reconstruction.

[0090] The Left contraction refer to The two modules on the far left are exactly... The model mi×mi +1 No need for or Reconstruction allows for direct shrunk calculations; the aforementioned Right outer product In The rightmost position is filled directly in the form of a product. modulus n i ×n i+1 There is no need to or Refactor. Figure 4 The process of left-shrinking and right-outproducting the input and factors one by one is vividly depicted, showing that the order of the moduli is not changed throughout the entire process. Finally, the I-order tensor... Stretched into an N-dimensional output vector The specific data index is in the order of n1×n2×…×n I =N can be rearranged again without reconstruction. Therefore, the algorithm does not involve any reconstruction operation. Although transposing the factor matrix with even indices is strictly speaking a reconstruction operation, i.e. and However, it can actually be stored in transpose form when storing the factor matrix; the transpose symbol here only serves to provide a clear description.

[0091] After step 3 is completed, if there are still uncompressed fully connected layers in the multilayer perceptron, the process jumps back to step 1 to compress the next fully connected layer. If all uncompressed fully connected layers in the multilayer perceptron have been compressed, a new multilayer perceptron is obtained. Proceed to step 4. The relevant decision-making route is in... Figure 1 This is illustrated in the flowchart.

[0092] Step 4: Apply the backpropagation algorithm to the newly compressed multilayer perceptron. Perform a limited number of training iterations to fine-tune it to optimal performance.

[0093] In this step, the original multilayer perceptron Fine-grained tensor decomposition and compression have become a new type of multilayer perceptron. Although the factor matrix in the form of fine-grained tensor decomposition can guarantee the recovery of the original weight matrix in theory, there is accumulated error after multi-layer full connection compression, so the new multi-layer perceptron A certain number of retraining needs to be performed to achieve the required performance.

[0094] In order to better illustrate the beneficial effects of the present application, especially in horizontal comparison with existing TT, TR, HT, BT and other tensor decomposition compression methods, the Long Short-Term Memory (LSTM) neural network model compression experiment on the UCF11 public video dataset is given.

[0095] LSTM is a commonly used specific form of RNN, which can be generally represented as:

[0096] y f =σ(W f x(t)+R f h(t-1)+b f )

[0097] y i =σ(W i x(t)+R i h(t-1)+b i )

[0098] y z =σ(W z x(t)+R z h(t-1)+b z )

[0099] y o =σ(W o x(t)+R o h(t-1)+b o )

[0100] c(t)=y f ⊙c(t-1)+y i ⊙y z

[0101] h(t)=y o ⊙σ(c(t))

[0102] Where f refers to the forget gate, i refers to the input gate, z refers to the state gate, o refers to the state gate, x(t) is the input in space at t, h(t-1) refers to the state input at t-1, W θ (θ=f,i,z,o) is the input weight matrix, R θ is the recurrent weight matrix, and b θHere, is the bias vector, and ⊙ indicates element-wise multiplication. The above six equations describe the neural network structure at time t. The neural network structure is the same before and after time t, but the values ​​of the input and output quantities x(t), h(t-1), and c(t) are different.

[0103] When designing the neural network parameters, the TT decomposition compressed LSTM references the paper "Tensor-trained recurrent neural networks for video classification," the TR decomposition compressed LSTM references the paper "Compressing recurrent neural networks with tensor ring for action recognition," the HT decomposition compressed LSTM references the paper "Compressing recurrent neural networks using hierarchical tuckertensor decomposition," and the BT decomposition compressed LSTM references the paper "Learning compact recurrent neural networks with block-term tensor decomposition." Note that in these four papers on TT, TR, HT, and BT, the later publishers all conducted LSTM compression experiments on the UCF11 dataset based on the earlier publishers' work. Therefore, the LSTM neural network models and tensor decomposition parameters of these papers can be directly used as the comparison objects for the experiments in this invention.

[0104] Specifically, each frame in the UCF11 video data is sampled as a 160×120×3 RGB image, which is stretched into a vector of length 57600 as the input x(t); the LSTM has only one hidden layer, and the output dimension is set to 2304; the LSTM duration is set to 6; the overall LSTM structure diagram is as follows. Figure 5 As shown. Input weight matrix W θ The size is 57600×2304, and the cyclic weight matrix R θ The dimensions are 2304×2304, the modulus length of 57600 is decomposed into 15×16×16×15, and the modulus length of 2304 is decomposed into 8×6×6×8; the ranks of TT, TR, HT, and BT are all set to 4, and the rank of the fine-grained tensor decomposition is... and All values ​​are set to 4, and K is set to 6. The Adam optimizer is used for training, with an initial learning rate of 0.001, and the training iterations through the UCF11 dataset are performed 400 times.

[0105] The experimental results are shown in the following table. It can be seen that the LSTM compressed by the fine-grained tensor decomposition method is the fastest in inference time. Although the remaining tensor decomposition methods can also achieve theoretical computation reduction, they are difficult to be significantly faster than the original LSTM in actual operation due to more or less data reconstruction.

[0106]

[0107] The above merely describes the preferred embodiments of the present application. It should be noted that those skilled in the art can make several improvements and modifications without departing from the technical principles of the present application, and these improvements and modifications should also be considered as falling within the protection scope of the present application.

Claims

1.A method for video recognition based on multi-layer perceptron fine-grained tensor decomposition compression, characterized in that, The method comprises: Step A: total frame number Randomly collect video data of camera, camera, denoted as Wherein, Indicates the width of each frame of video image, Indicates the height of each frame of video image, That is, the total frame number of video data is also the time length, Indicates the number of channels of each frame of video image; Step B: Constructing a multilayer perceptron for video data Performing the identification; Step C: output the result after the video data is identified; Wherein, the step B comprises: Step B1: define a multi-layer perceptron for recognizing video data, define its input dimension of each frame as , define its time dimension as , the multi-layer perceptron is denoted as , and the corresponding video data is divided into input vectors, each input vector has a length of , and output to step B2; Step B2: Multi-Layer Perceptron on the output of Step B1 is trained, and the trained multi-layer perceptron is denoted as , and output to Step B3; Step B3: compressing the trained multi-layer perceptron of step B2 using multi-layer perceptron fine-grained tensor decomposition performing processing to obtain a compressed multi-layer perceptron ; Step B4: Compressed multi-layer perceptron is completed using step B3 The video data is identified to generate a result after identification of the video data to be output. Wherein, the step B3 comprises: Step 1: For a trained multi-layer perceptron , let one of the fully connected layers be , Let the non-linear activation function of this layer be , Read the fully connected weight matrix , the input vector of dimension , and the output vector of dimension ; Step 2: Compute and According to the constraints and Factorization into integers, the weight matrix can be compressed in the form of fine-grained tensor decomposition, i.e.: wherein is the th factor matrix of the QR decomposition, is the th factor matrix of the QR decomposition, is the rank of the QR decomposition, is the rank of the QR decomposition, and is the identity diagonal tensor, is the transformation that stretches a tensor into a vector, the operator symbol denotes the contraction, the operator symbol denotes the outer product; wherein , ; K and I are both integers; Step 3: Define the input vector with the weights after fine-grained tensor decomposition reconstruction-free algorithm between so that its output is equal to the output of . After step 3 is completed, if there are still full connection layers that have not been compressed in the multilayer perception machine, jump to step 1 to compress the next full connection layer, if all the full connection layers to be compressed in the multilayer perception machine have been compressed, a new multilayer perception machine is obtained , and step 4 is entered; Step 4: Using the error backpropagation algorithm to the new completed compressed multilayer perceptron A limited number of training is performed to fine tune it to optimal performance. 2.The method of claim 1, wherein, In step 2, the number of factors is and the same factor is used to factorize, and each factor or is not less than 2. 3.The method of claim 2, wherein, The step 2 in, Factor of the group After the contraction operation Will eliminate all the rank Get the size of The Order tensor, Factor of the group After the contraction operation Will eliminate all the rank , get the size of The Order tensor, the two Order tensor respectively by Stretching Dimension and Dimension vector, the two vectors again by the outer product operation Get the first Road Matrix, all Road matrix can be restored to the original weight matrix ; In step 2, the weights in the form of fine-grained tensor decomposition, whose factor matrices are obtained by the contraction operation , the outer product operation , and the summation ∑, can be fully recovered to the form of the weight matrix before compression, which indicates that the transformation of the weight matrix to the form of fine-grained tensor decomposition is correct in dimension transformation. 4.The method of claim 3, wherein, In step 3, the reconfiguration-free algorithm Specifically refers to: First, the input vector is constrained to be a rank tensor ; then the group of factor matrices whose indices are all odd and whose index values are the next even larger than the largest index value in the group are collapsed, eliminating and ranks ; and defining Then from start, let left contract the leftmost two modules , right outer product fill in the rightmost two modules , a total of intermediate results, namely ; then add , the result is the same as pointed to and continue to do left contract and right outer product after adding the calculation, and so on until all are eliminated; finally get order tensor , stretch it to dimension output vector , and the vector value is equal to the direct calculation .​ 5.The method of claim 4, wherein, In step 3, the unit diagonal tensor and has no effect on the computation result. 6.The method of claim 5, wherein, In step 3, the input vector According to constraints Fold into order tensor This process does not change the actual storage location of the data in the computer's memory; it only adjusts the indexes that point to the specific data according to... Arranged sequentially; the aforementioned Left contraction ,refer to The two modules on the far left are exactly... model No need for or After reconstruction, shrinkage calculation can be performed directly; Right outer product , refers to The rightmost position is filled directly in the form of a product. model There is no need to or Reconstruction is performed; the order tensor Stretch to 3D output vector For specific data indexes according to The algorithm can be rearranged again without reconstruction; therefore, the algorithm described above... There are no refactoring operations. 7.The method of claim 1, wherein, In step 4, the original multi-layer perceptron Compressed by fine-grained tensor decomposition has become a new multi-layer perceptron Although the factor matrix in the form of fine-grained tensor decomposition can theoretically guarantee recovery to the original weight matrix, there will be cumulative errors after multi-layer full connection compression, so the new multi-layer perceptron A certain number of retraining is needed to achieve the required performance. 8.The method of claim 7, wherein, The method uses fine-grained tensor decomposition to compress the weight matrix in the multi-layer perceptron, and provides a reconstruction-free algorithm between the input vector and the weight in the form of fine-grained tensor decomposition, so that the multi-layer perceptron can have a smaller neural network size after compression and can also calculate quickly with the input.

Images