An online video understanding method based on time sequence reusable convolution

Abstract

The application discloses an online video understanding method based on time sequence reusable convolution, which comprises the following steps: 1) video data set pretreatment; 2) training of a 3D convolution neural network; 3) design and establishment of time sequence reusable convolution operation; 4) replacement of conventional 3D convolution with time sequence reusable convolution; 5) verification of the calculation accuracy of the network in the online video understanding task; and 6) testing of the network and calculation of the network calculation complexity. The method can reduce the calculation amount of the online video understanding task and improve the efficiency of the network in real-time tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to machine vision and deep learning technology, and particularly relates to an online video understanding method based on time sequence reusable convolution. BACKGROUND

[0002] Video understanding refers to extracting required information by using a video modeling method to understand events or behaviors occurring in the video. Compared with single image target classification and target recognition, the video understanding task needs to extract spatial and temporal feature information. For example, in a traffic scene, a single image can only determine the position information of the vehicle in the image, and cannot measure the motion information such as direction and speed of the vehicle, while the video understanding task needs to measure the above-mentioned time sequence information. In addition, video understanding has many application values in the fields of security, medical treatment, human-computer interaction and automatic driving.

[0003] In early video understanding research, most video understanding tasks use a method of manually designing features to achieve the tasks. With the development of deep learning networks, deep learning gradually replaces the method of manually designing features and becomes the mainstream method in the field of video understanding, due to the advantages of efficient feature extraction, strong learning ability, high adaptability and portability. At present, the deep learning network architecture in the field of video understanding mainly includes a double-flow method, a 2D network + time sequence module method and a 3D network method.

[0004] The double-flow method uses one network to train an image to extract spatial features, and opens another network to train an optical flow image to extract time features. This method divides the time and space information, uses a convolutional neural network to extract the features of the RGB flow and the optical flow respectively, and then fuses the prediction results of the two flows to obtain the final prediction of the video segment. The main difference between most double-flow algorithms is the fusion method of the two flows. For example, the earliest double-flow network (Two-Stream Network) simply weights and fuses the optical flow and the RGB flow. However, due to the extraction of the optical flow consuming a large amount of time and storage space, the double-flow network often needs a long inference time and a large memory space, which has great limitations in application.

[0005] The 2D network + temporal module method uses a 2D network from the image domain to extract spatial information, and then uses a module capable of extracting temporal information to extract temporal features, which are then fused together. Common structures capable of extracting temporal information include Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks. This method is advantageous due to its high efficiency and low memory requirements. However, because the initial 2D neural network lacks temporal modeling capabilities during feature extraction, it cannot capture low-level motion features, thus its performance is not very good.

[0006] 3D networks extend 2D networks from image processing to 3D, enabling them to extract temporal information. Early 3D networks, due to their large parameter count and training difficulties, were not widely used in video understanding. The emergence of I3D (Inflated 3D ConvNet) allowed excellent 2D image processing networks to be inflated into 3D networks to solve video problems, and they could be pre-trained on large 2D datasets. Subsequently, 3D networks developed rapidly and gradually became a mainstream method in video understanding. In recent years, the accuracy of 3D networks has continuously improved, but they still suffer from large parameter counts and poor real-time performance. 3D convolutional neural networks have achieved the best computational / accuracy tradeoff in current offline video understanding tasks, but in online video understanding tasks, there is still a problem of repeated computation of a large number of video frames. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing an online video understanding method based on temporally reusable convolution. This method can reduce the computational load of online video understanding tasks and improve the efficiency of networks in real-time tasks.

[0008] The technical solution to achieve the objective of this invention is:

[0009] An online video understanding method based on temporally reusable convolution includes the following steps:

[0010] 1) Video dataset preprocessing: Frame extraction and cropping are performed on the original video dataset. This involves randomly selecting frames and dividing the data into training and validation sets at a 9:1 ratio. Each video is then extracted into a sequence of frame images, which are then cropped. If the width of the video is greater than its height, left, center, and right cropping is used; if the height is greater than the width, top, center, and bottom cropping is used. The cropped frame images have a resolution of 224×224.

[0011] 2) Training a 3D Convolutional Neural Network: The X3D network was selected as the basic network architecture for the online video understanding task. The X3D network is a 3D convolutional neural network constructed using a structure search strategy. During training, it started with the smallest baseline network, X2D, and varied the sampling frame interval γ. τ Number of sampling frames γ t Spatial resolution γ s Network depth γ d Network width γ w Bottleneck layer width γ b The network is expanded in six dimensions, including:

[0012] 2-1) Determine the complexity c of the target network as a standard to measure whether the computational cost of the network meets the requirements;

[0013] 2-2) Perform forward expansion on the network, expanding only one dimension at a time while keeping other dimensions unchanged. Each expansion should be incremental, with the expansion size measured by increasing the network complexity by approximately double. Specifically:

[0014] Sampling frame interval γ τ Each expansion results in a size that is 0.5 times the original size.

[0015] Number of sampling frames γ t Each expansion doubles the original size;

[0016] Spatial resolution γ s Each expansion returns it to its original size. times;

[0017] Network Depth γ d Each expansion results in a size 2.2 times the original size;

[0018] Network width γ w Each expansion doubles the original size;

[0019] Bottleneck layer width γ b Each expansion results in a size 2.25 times the original size;

[0020] 2-3) After each network expansion, train the network using the image sequences from the training set in step 1);

[0021] 2-4) After training is completed, the network is tested using the image sequences of the test set in step 1), the number of correct predictions is counted, and the accuracy of the network is obtained by dividing the number of correct predictions by the total number of predictions.

[0022] 2-5) Use the ptflops package to calculate the number of floating-point operations per second (FLOPs) of the network;

[0023] 2-6) Repeat steps 2-2) to 2-5) until the target complexity of the network reaches c. If the computational complexity exceeds the target complexity c, cancel the expansion. Record the accuracy and FLOPs of each network.

[0024] 2-7) In step 2-6), select the two networks with higher accuracy, preserving their network structure and weights. The two selected networks are X3D-M and X3D-S. X3D-S has a smaller network structure and higher computational efficiency but lower accuracy; X3D-M has higher accuracy but higher computational cost. The network structure of X3D-M is as follows: The network takes 16 frames of images with a resolution of 224×224 as input, passes through four residual blocks (res2, res3, res4, res5), and then inputs them into the pooling and fully connected layers. The network's dimensionality parameter is: γ τ =5,γ t =16, γ s =2,γ d =2.2, γ w =1,γ b =2.25;

[0025] 2-8) Retain the weight data of the trained X3D-M and X3D-S networks;

[0026] 3) Design and establish temporally reusable convolution operations: Represent the sequence of frame images input to the network as P1, P2, P3...Pn, and represent the convolution kernels of the 3D convolution as K1, K2, with γ being the number of video sampling frames. t In the case of using conventional 3D convolution to process online video understanding tasks, the current frame image is... When the convolution occurs, the result is:

[0027]

[0028] When processing the next frame of the image, the result of the convolution is:

[0029]

[0030] At this point, there is computational redundancy:

[0031]

[0032] To eliminate the redundant computation problem in conventional 3D convolution, we now construct temporally reusable convolutions, with the following steps:

[0033] 3-1) Create a convolution result queue Q, and record the convolution result between each two frames as q:

[0034] q1 = P0·K1 + P1·K2, q n =Pn-1 ·K1+P n K2;

[0035] 3-2) Set the capacity of queue Q to the number of video sampling frames γ. t The number of data stored in queue Q cannot exceed this value;

[0036] 3-3) Each time a prediction is made for the current video frame, only the current frame image and the previous frame image are subjected to 3D convolution operation. If there is no previous frame image, no processing is performed.

[0037] 3-4) Store the result q of the convolution operation into queue Q and perform an enqueue operation. If queue Q is full, first perform a dequeue operation on the head of the queue, and then store q into the queue.

[0038] 3-5) After performing the enqueue operation, if queue Q is full, all convolution results in queue Q are combined as the convolution result for the prediction of this frame; if queue Q is not full, no combination is performed and the prediction is skipped.

[0039] 4) Replace the conventional 3D convolution with temporally reusable convolution: Analogous to the network structure in step 2), rebuild the 3D convolutional neural network. During the construction process, use the temporally reusable convolution in step 3) for feature extraction. The weight values ​​of the network are the same as the weight data retained in steps 2-8).

[0040] 5) Verify the computational accuracy of the network in the online video understanding task: Select videos with a length greater than 20s from the test set segmented in step 1) as the simulation of the online task. Input the video frame images as the online video frame stream into the 3D convolutional neural network constructed in step 4) to perform prediction. Statistically analyze the prediction results and calculate the network accuracy according to the number of correct predictions / the total number of data.

[0041] 6) Test the network and calculate the network's computational complexity: Test the network through steps 2-5) and use ptflops to calculate the network's FLOPs to measure the computational cost of the model. Compare the calculated FLOPs of the improved network with the FLOPs of the original network, i.e., the calculation results in step 2).

[0042] This technical solution replaces the conventional 3D convolution in a 3D convolutional neural network with a temporally reusable convolution method. This addresses the problem of low efficiency caused by redundant data computation in online video understanding tasks. Temporally reusable convolution uses a queue to store the convolution calculation results between two adjacent frames in each frame. When processing the next frame, the calculation results in the queue are reused, thereby eliminating computational redundancy and improving operational efficiency.

[0043] This method can reduce the computational load of online video understanding tasks and improve the efficiency of networks in real-time tasks. Attached image description:

[0044] Figure 1 This is a flowchart illustrating the method in the embodiment;

[0045] Figure 2 This is a schematic diagram of the X3D network structure in the embodiment;

[0046] Figure 3 This is a schematic diagram of the structure of a conventional 3D convolution in the embodiment;

[0047] Figure 4 This is a schematic diagram of the temporally reusable convolution structure in the embodiment;

[0048] Figure 5 This is a comparison chart of experimental results in the examples. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0050] Example:

[0051] Reference Figure 1 An online video understanding method based on temporally reusable convolution includes the following steps:

[0052] 1) Video dataset preprocessing: Frame extraction and cropping are performed on the original video dataset. This involves randomly selecting frames and dividing the data into training and validation sets at a 9:1 ratio. Each video is then extracted into a sequence of frame images, which are then cropped. If the width of the video is greater than its height, left, center, and right cropping is used; if the height is greater than the width, top, center, and bottom cropping is used. The cropped frame images have a resolution of 224×224.

[0053] 2) Training a 3D Convolutional Neural Network: The X3D network was selected as the basic network architecture for the online video understanding task. The X3D network is a 3D convolutional neural network constructed using an architecture search strategy. During training, it started with the smallest baseline network, X2D, and varied the sampling frame interval γ. τ Number of sampling frames γ t Spatial resolution γ s Network depth γ d Network width γ w Bottleneck layer width γ b The network is expanded in six dimensions, including:

[0054] 2-1) Determine the complexity c of the target network as a standard to measure whether the computational cost of the network meets the requirements;

[0055] 2-2) Perform forward expansion on the network, expanding only one dimension at a time while keeping other dimensions unchanged. Each expansion should be incremental, with the expansion size measured by increasing the network complexity by approximately double. Specifically:

[0056] Sampling frame interval γ τ Each expansion results in a size that is 0.5 times the original size.

[0057] Number of sampling frames γ t Each expansion doubles the original size;

[0058] Spatial resolution γ s Each expansion returns it to its original size. times;

[0059] Network Depth γ d Each expansion results in a size 2.2 times the original size;

[0060] Network width γ w Each expansion doubles the original size;

[0061] Bottleneck layer width γ b Each expansion results in a size 2.25 times the original size;

[0062] 2-3) After each network expansion, train the network using the image sequences from the training set in step 1);

[0063] 2-4) After training is completed, the network is tested using the image sequences of the test set in step 1), the number of correct predictions is counted, and the accuracy of the network is obtained by dividing the number of correct predictions by the total number of predictions.

[0064] 2-5) The ptflops package is used to calculate the number of floating point operations per second (FLOPs) of the network, which is used to measure the network complexity;

[0065] 2-6) Repeat steps 2-2) to 2-5) until the target complexity of the network reaches c. If the computational complexity exceeds the target complexity c, cancel the expansion and record the accuracy and FLOPs of each network.

[0066] 2-7) In step 2-6), select the two networks with higher accuracy, retaining their network structure and weights. The two selected networks are X3D-M and X3D-S. X3D-S has a smaller network structure and higher computational efficiency but lower accuracy; X3D-M has higher accuracy but higher computational cost. The network structure diagram of X3D-M is shown below. Figure 2As shown: The network takes 16 frames of images with a resolution of 224×224 as input, passes them through four residual blocks (res2, res3, res4, and res5), and then feeds them into the pooling and fully connected layers. The network's dimensionality parameter is: γ τ =5,γ t =16, γ s =2,γ d =2.2, γ w =1,γ b =2.25;

[0067] 2-8) Retain the weight data of the trained X3D-M and X3D-S networks;

[0068] 3) Design and establish temporally reusable convolution operations: Represent the sequence of frame images input to the network as P1, P2, P3...Pn, and represent the convolution kernels of the 3D convolution as K1, K2, with γ being the number of video sampling frames. t In the case of using conventional 3D convolution to process online video understanding tasks, the current frame image is... When the convolution occurs, the result is:

[0069]

[0070] When processing the next frame of the image, the result of the convolution is:

[0071]

[0072] At this point, there is computational redundancy:

[0073]

[0074] When γ t When the value is 4, the convolution result is as follows Figure 3 As shown: After calculating the convolution of the current frame image with P4 and P5, there is 2 / 3 redundant calculation in the output feature map of the result;

[0075] To eliminate the redundant computation problem in conventional 3D convolution, we now construct temporally reusable convolutions, with the following steps:

[0076] 3-1) Create a convolution result queue Q, and record the convolution result between each two frames as q:

[0077] q1=P0·K1+P1·K2,qn=P n-1 ·K1+P n K2;

[0078] 3-2) Set the capacity of queue Q to the number of video sampling frames γ. t The number of data stored in queue Q cannot exceed this value;

[0079] 3-3) Each time a prediction is made for the current video frame, only the current frame image and the previous frame image are subjected to 3D convolution operation. If there is no previous frame image, no processing is performed.

[0080] 3-4) Store the result q of the convolution operation into queue Q and perform an enqueue operation. If queue Q is full, first perform a dequeue operation on the head of the queue, and then store q into the queue.

[0081] 3-5) After performing the enqueue operation, if queue Q is full, all convolution results in queue Q are combined as the convolution result for the prediction of this frame; if queue Q is not full, no combination is performed, and the prediction is skipped. In this example, the convolution results are combined... Figure 3 and Figure 4 To further explain temporally reusable convolution: Conventional 3D convolutional neural networks extract T frames of images at equal intervals from a video as input, extract features through 3D convolution, and recognize the features. These networks have achieved a good balance between accuracy and computation for offline video understanding tasks. However, these networks still have shortcomings in online tasks. This is because in online video understanding tasks, the input is a continuous stream of video frames, and each frame needs to be combined with the information from the previous frame for prediction. For a neural network that takes T frames as input, it is necessary to save the T-1 frames and perform 3D convolution operations on the previous T-1 frames when predicting the current frame. At this time, a large number of video frames will participate in repeated 3D convolution operations.

[0082] like Figure 3 As shown, assuming the network requires 4 frames of video images as input, then at the current moment, it needs four frames P1, P2, P3, and P4 as input and performs a 3D convolution operation on them; at the next moment, it needs four frames P2, P3, P4, and P5 as input and performs a convolution operation on them. The three frames P2, P3, and P4 undergo repeated 3D convolution operations. The computational cost of a typical 3D convolution is:

[0083] K H ·K W ·K T ·M·N·F H ·F W ·F T ,

[0084] Among them, K H K represents the height of the 3D convolutional kernel. W K represents the width of the kernel. T This represents the kernel's time span, M and N represent the number of input and output channels, and F... H F W F TThese represent the height, width, and time span of the output feature map, respectively.

[0085] like Figure 4 As shown, if the network also requires 4 frames of video images as input, it is only necessary to perform a 3D convolution operation on the current frame and the previous frame, and then directly combine it with the temporarily stored results of the previous two 3D convolution operations. This can achieve 3D convolution operation on 4 frames of video images, greatly saving computing resources and improving computing efficiency. The computational cost of processing one frame by temporally reusable convolution is:

[0086] K H ·K W ·K T ·M·N·F H ·F W ,

[0087] Compared to conventional 3D convolution, temporally reusable convolution reduces computational cost by F. T In actual operation, the actual improvement in network efficiency is affected by the number of input frames of the network and the number of video frames. When the number of input frames required by the network is greater than the number of video frames, the calculation of the first few frames will become invalid due to the inability to make predictions, which will reduce efficiency. In online video understanding tasks, the number of video frames is often much greater than the number of input frames required by the network. At this time, the impact of invalid calculation on network efficiency can be ignored, and the efficiency improvement can reach the theoretical value.

[0088] 4) Replace the conventional 3D convolution with temporally reusable convolution: Analogous to the network structure in step 2), rebuild the 3D convolutional neural network. During the construction process, use the temporally reusable convolution in step 3) for feature extraction. The weight values ​​of the network are the same as the weight data retained in steps 2-8).

[0089] 5) Verify the computational accuracy of the network in the online video understanding task: Select videos with a length greater than 20s from the test set segmented in step 1) as the simulation of the online task. Input the video frame images as the online video frame stream into the 3D convolutional neural network constructed in step 4) to perform prediction. Statistically analyze the prediction results and calculate the network accuracy according to the number of correct predictions / the total number of data.

[0090] 6) Test the network and calculate its computational complexity: Test the network using steps 2-5), and calculate the network's FLOPs using ptflops to measure the model's computational cost. Compare the calculated FLOPs of the improved network with the original network's FLOPs, i.e., the results calculated in step 2). Figure 5 As shown, Figure 5The statistical results in steps 5) and 6) are presented: In human action recognition within the video understanding domain, by improving the X3D network with temporally reusable convolutions, the computational cost of the X3D-M network on the human action dataset (Kinetics-400) was reduced from 4970 MFLOPs to 325 MFLOPs, and the computational accuracy was improved from 74.6% to 75.8%. The computational cost of the X3D-S network was reduced from 2061 MFLOPs to 166 MFLOPs, and the accuracy was improved from 64.7% to 70%. The reduction in computational cost is related to the time span required by the original network. The time span parameter of X3D-M is 16, and the computational cost is reduced by approximately 15.2 times. The time span parameter of X3D-S is 13, and the computational cost is reduced by approximately 12.3 times. Since the number of video input frames no longer affects the network's computational cost, increasing the number of input frames also improves the recognition accuracy.

[0091] In summary, the method presented in this example can be applied to any video understanding method that uses 3D convolutional neural networks. It can improve the efficiency of online video understanding tasks without compromising the original video understanding accuracy. This method replaces the conventional 3D convolution of the 3D convolutional neural network with temporally reusable convolution, reusing the intermediate computational data of each frame in the network. This reduces computational redundancy, decreases the computational load of the network for processing each frame, and improves the network's operating efficiency.

Claims

1. An online video understanding method based on temporally reusable convolution, characterized in that, Includes the following steps: 1) Video dataset preprocessing: Frame extraction and cropping are performed on the original video dataset. The dataset is randomly selected and divided into training and validation sets at a 9:1 ratio. Each video is extracted into a sequence of frame images, which are then cropped. If the width of the video is greater than its height, left, center, and right cropping is used; if the height is greater than the width, top, center, and bottom cropping is used. The cropped frame images have a resolution of 224×224. 2) Training a 3D Convolutional Neural Network: The X3D network was selected as the basic network architecture for the online video understanding task. The X3D network is a 3D convolutional neural network constructed using a structure search strategy. During training, it started with the smallest baseline network, X2D, and varied the sampling frame interval γ. τ Number of sampling frames γ t Spatial resolution γ s Network depth γ d Network width γ w Bottleneck layer width γ b The network is expanded in six dimensions, including: 2-1) Determine the complexity c of the target network; 2-2) Perform forward expansion on the network, expanding only one dimension at a time while keeping other dimensions unchanged. Each expansion should be incremental, with the expansion size measured by increasing the network complexity by approximately double. Specifically: Sampling frame interval γ τ Each expansion results in a size that is 0.5 times the original size. Number of sampling frames γ t Each expansion doubles the original size; Spatial resolution γ s Each expansion returns it to its original size. times; Network Depth γ d Each expansion results in a size that is 2.2 times the original size. Network width γ w Each expansion doubles the original size; Bottleneck layer width γ b Each expansion results in a size 2.25 times the original size; 2-3) After each network expansion, train the network using the image sequences from the training set in step 1); 2-4) After training is completed, the network is tested using the image sequences of the test set in step 1), the number of correct predictions is counted, and the accuracy of the network is obtained by dividing the number of correct predictions by the total number of predictions. 2-5) Use the ptflops package to calculate the number of floating-point operations per second (FLOPs) of the network; 2-6) Repeat steps 2-2) to 2-5) until the target complexity of the network reaches c. If the computational complexity exceeds the target complexity c, cancel the expansion. Record the accuracy and FLOPs of each network. 2-7) In step 2-6), select the two networks with higher accuracy, preserving their network structure and weights. The two selected networks are X3D-M and X3D-S. The network structure of X3D-M is as follows: the network takes 16 frames of images with a resolution of 224×224 as input, passes through four residual blocks (res2, res3, res4, res5), and then inputs them into the pooling and fully connected layers. The dimensionality parameter of the network is: γ τ =5,γ t =16, γ s =2,γ d =2.2, γ w =1,γ b =2.25; 2-8) Retain the weight data of the trained X3D-M and X3D-S networks; 3) Design and establish temporally reusable convolution operations: Represent the sequence of frame images input to the network as P1, P2, P3...Pn, and represent the convolution kernels of the 3D convolution as K1, K2, with γ being the number of video sampling frames. t In the case of using conventional 3D convolution to process online video understanding tasks, the current frame image is... When the convolution occurs, the result is: When processing the next frame of the image, the result of the convolution is: At this point, there is computational redundancy: Constructing temporally reusable convolutions includes: 3-1) Create a convolution result queue Q, and record the convolution result between each two frames as q: q1=P0·K1+P1·K2,q n =P n-1 ·K1+P n ·K2; 3-2) Set the capacity of queue Q to the number of video sampling frames γ. t The number of data stored in queue Q cannot exceed this value; 3-3) Each time a prediction is made for the current video frame, only the current frame image and the previous frame image are subjected to 3D convolution operation. If there is no previous frame image, no processing is performed. 3-4) Store the result q of the convolution operation into queue Q and perform an enqueue operation. If queue Q is full, first perform a dequeue operation on the head of the queue, and then store q into the queue. 3-5) After performing the enqueue operation, if queue Q is full, all convolution results in queue Q are combined as the convolution result for the prediction of this frame; if queue Q is not full, no combination is performed and the prediction is skipped. 4) Replace the conventional 3D convolution with temporally reusable convolution: Analogous to the network structure in step 2), rebuild the 3D convolutional neural network. During the construction process, use the temporally reusable convolution in step 3) for feature extraction. The weight values ​​of the network are the same as the weight data retained in steps 2-8). 5) Verify the computational accuracy of the network in the online video understanding task: Select videos with a length greater than 20s from the test set segmented in step 1) as the simulation of the online task. Input the video frame images as the online video frame stream into the 3D convolutional neural network constructed in step 4) to perform prediction. Statistically analyze the prediction results and calculate the network accuracy according to the number of correct predictions / the total number of data. 6) Test the network and calculate the network's computational complexity: Test the network through steps 2-5) and use ptflops to calculate the network's FLOPs to measure the computational cost of the model. Compare the calculated FLOPs of the improved network with the FLOPs of the original network, i.e., the calculation results in step 2).

Images