Video data processing acceleration algorithm based on convolutional neural network

Abstract

The invention provides a video data processing acceleration algorithm based on a convolutional neural network, and relates to the technical field of data processing, and the algorithm comprises the steps: S1, constructing a neural network model for processing video data; s2, extracting image data according to the frame number and the time sequence of the video data, and preprocessing the image data; s3, performing inter-frame difference processing on the image data; s4, sequentially adding the image data into a data cache queue for acceleration processing; s5, cutting the number of channels of the neural network model according to the weight of each channel in the neural network model; s6, inputting the image data into a neural network model, and outputting video data by the neural network model according to the frame number and the time sequence of the image data; s7, training the neural network model through a back propagation algorithm; according to the invention, accelerated processing of the video data can be realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a video data processing acceleration algorithm based on convolutional neural network. Background Art

[0002] In recent years, as visual tasks become more and more complex and the amount of data processed becomes larger and larger, computers are faced with increasing computational workloads, especially for large models with more complex data scale and training cycle requirements, which often show strict limitations of larger quantity, higher quality and more complex computational requirements. Therefore, accelerating the processing of video data, whether it is image data, video data or other formats of data, has become an important research issue.

[0003] Among them, since video data shows more dense image information than image data, and has stronger inter-frame correlation and continuity between previous and subsequent frames, when processing video data, it is necessary to consider not only the two-dimensional data of single-frame image information, but also the timing information between frames, which means that video processing faces greater computational requirements.

[0004] At present, with the gradual development of hardware such as GPU, researchers have gradually begun to use video data for visual learning, such as video object segmentation, video super-resolution, etc., but there is relatively little work on accelerating the processing of video data. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a video data processing acceleration algorithm based on a convolutional neural network to achieve accelerated processing of video data.

[0006] The present invention provides a video data processing acceleration algorithm based on convolutional neural network, comprising the following steps:

[0007] S1. Build a neural network model for processing video data;

[0008] S2, extracting image data according to the frame number and timing of the video data, and preprocessing the image data;

[0009] S3, performing inter-frame difference processing on the image data;

[0010] S4, adding the image data to the data cache queue in sequence for accelerated processing;

[0011] S5, calculating the weight of each channel in the neural network model, and dynamically cutting the number of channels of the neural network model according to the weight;

[0012] S6, inputting the image data into the neural network model, and the neural network model outputs video data according to the frame number and time sequence of the image data;

[0013] S7. Train the neural network model through the back propagation algorithm.

[0014] Compared with the prior art, the present application has the following advantages: inter-frame differential processing of video data is performed to add the data to the cache queue for accelerated reading, thereby speeding up the processing of video data. Dynamic cropping is performed according to the importance of the channels of the neural network model, so the neural network model can be simplified, making the processing speed of image data faster without affecting the output.

[0015] In a possible implementation, in step S1, the neural network model is a Faster R-CNN model.

[0016] Compared with the existing technology, the Faster R-CNN model performs better than CNN and has a faster processing speed.

[0017] In a possible implementation, a BN layer is provided in the Faster R-CNN model.

[0018] Compared with the existing technology, the BN layer can speed up the convergence of the neural network. Because in a deep neural network model such as Faster R-CNN, if the data distribution of each layer is different, it will make the deep neural network model very difficult to converge and train. The BN layer can convert the data of each layer to a mean of 0 and a variance of 1, so that the distribution of data in each layer is the same, and training will be easier to converge; the BN layer can also prevent gradient explosion and gradient disappearance, and prevent overfitting.

[0019] In a possible implementation, the preprocessing in step S2 includes reducing the resolution of the image data.

[0020] Compared with the existing technology, it can reduce the capacity of the picture, reduce the spatial complexity, process faster, and retain the basic characteristics of the image data.

[0021] In a possible implementation manner, step S3 performs inter-frame difference processing by the following formula:

[0022] G k (x,y)=|P k (x,y)-P k-1 (x,y)|∩|P k+1 (x,y)-P k (x,y)|,

[0023] Wherein, P represents the target image data, G represents the result image data, x and y represent the horizontal coordinate and vertical coordinate of the pixel point in the image data respectively, k represents the kth frame image data, k-1 represents the k-1th frame image data, and k+1 represents the k+1th frame image data.

[0024] Compared with the prior art, the differential processing performed by the three-frame differential method of the above formula can effectively suppress noise and improve processing accuracy.

[0025] In a possible implementation, step S5 includes the following steps:

[0026] S51, calculating the weight of each channel in the neural network model;

[0027] S52, calculating the actual influence of the weight on the loss function;

[0028] S53. Dynamically trim channels with small influence in the neural network model according to the influence size.

[0029] Compared with the existing technology, the above technical solution can effectively cut the number of channels of the neural network model, lightweight the neural network model, and make the image data processing speed faster without affecting the output.

[0030] In a possible implementation, step S51 calculates the influence of the weight on the loss function according to the following formula:

[0031]

[0032] Where L(·) represents the loss function, N represents the weight value of the current channel, (δ0, μ0) represents the translation and scaling factors of the current channel respectively, h represents the high-order infinitesimal derived on the scaling factor δ0, k represents the high-order infinitesimal derived on the translation factor μ0, and T(δ0, μ0) represents the high-order remainder.

[0033] The above formula is the first-order Taylor expansion of the loss function L(·) at the corresponding transformation parameter (δ0, μ0) on the channel. Generally speaking, without considering the higher-order remainder, this formula limits the channel that has less impact on the neural network model. Even if the scale transformation parameter δ and the translation transformation parameter μ change, the loss value of the entire neural network model will not change dramatically, which means that the impact of the channel on network performance after being pruned is relatively small. In addition, the contribution of each channel transformation parameter to the loss function can be measured according to the absolute value of ΔL(·), and then the impact of each channel (δ0, μ0) can be obtained. Finally, the corresponding channels below the expected threshold are pruned according to the preset threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is the overall flow chart of the algorithm of the present invention;

[0035] Figure 2 Schematic diagram of the algorithm of the present invention. DETAILED DESCRIPTION

[0036] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0037] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] As attached Figure 1 As shown, this embodiment discloses a video data processing acceleration algorithm based on a convolutional neural network, comprising the following steps:

[0039] S1. Build a neural network model for processing video data;

[0040] S2, extracting image data according to the frame number and timing of the video data, and preprocessing the image data;

[0041] S3, performing inter-frame difference processing on the image data;

[0042] S4, adding the image data to the data cache queue in sequence for accelerated processing;

[0043] S5, calculating the weight of each channel in the neural network model, and dynamically cutting the number of channels of the neural network model according to the weight;

[0044] S6, inputting the image data into the neural network model, and the neural network model outputs video data according to the frame number and time sequence of the image data;

[0045] S7. Train the neural network model through the back propagation algorithm.

[0046] Specifically, in step S1, the neural network model is a Faster R-CNN model, in which a BN layer is provided, namely, a Batch Normalization layer, also known as a regularization layer. The BN layer can speed up the convergence of the neural network, because in a deep neural network model such as Faster R-CNN, if the data distribution of each layer is different, the deep neural network model will be very difficult to converge and train, and the BN layer can convert the data of each layer to a mean of 0 and a variance of 1, so that the data distribution of each layer is the same, and the training will be easier to converge.

[0047] In the neural network model, for each neuron in each batch, the BN layer calculates the mean of its input value. and variance σ 2 , using the obtained mean and variance, normalize using the following formula:

[0048]

[0049] Among them, Y represents the original input value, and Y′ represents the standardized value; after standardization, the mean of the input value becomes 0 and the variance becomes 1, thus obtaining a stable distribution.

[0050] The normalized values ​​can be scaled and translated as follows:

[0051] y (k) =δ (k) Y′+μ (k)

[0052] Among them, y (k) represents the final output, k represents the neuron number, δ (k) represents the scale transformation parameter, μ (k) Represents the translation transformation parameter. The BN layer can also prevent gradient explosion and gradient disappearance, and prevent overfitting. The Faster R-CNN model performs better than CNN and has a faster processing speed.

[0053] The video data to be processed is defined as V i ={v1,v2,…,v n},v1,v2,…,v n Represents each video data, and needs to perform a decoding operation on the video data. According to the playback timing and frame number of the video data itself, it is decoded into several frames. It is necessary to select the better quality from each frame as the image data. The image data is defined as I i ={i1,i2,…,i n}, where i1,i2,…,i n Indicates video data V i image data.

[0054] The preprocessing in step S2 is to reduce the image data I i ={i1,i2,…,i n} resolution, due to the large number of image data I i ={i1,i2,…,i n}Batch operations are required to convert to low-resolution images.

[0055] Step S3 performs inter-frame difference processing by the following formula:

[0056] G k (x,y)=|P k (x,y)-P k-1 (x,y)|∩|P k+1 (x,y)-P k (x,y)|,

[0057] Among them, P represents the target image data, G represents the result image data, x and y represent the horizontal coordinates and vertical coordinates of the pixel points in the image data, respectively, k represents the kth frame image data, k-1 represents the k-1th frame image data, and k+1 represents the k+1th frame image data. The above formula is the three-frame difference method, which operates on three adjacent frames of images and divides the three frames of images into a group, including the front frame, the middle frame and the back frame, that is, the grayscale values ​​of the corresponding pixels in the k-1th frame, the kth frame and the k+1th frame image data are subtracted, and the motion state of the object in the image data is judged by comparing the grayscale difference. If the grayscale difference of the corresponding pixel points is very small, it is considered that the object in the image data is stationary; if the grayscale difference is very large, it is considered that the grayscale change is caused by the motion of the object in the image data. Such differential processing of image data can effectively suppress noise, improve processing accuracy, and effectively extract the motion characteristics of the original video data in low-pixel image data.

[0058] In step S4, there may be a difference between the generation speed of each frame of image data and the processing speed of each frame of image data. If the processing speed cannot keep up with the generation speed, it may lead to the accumulation of frames, which in turn affects the real-time performance and performance of the entire neural network. In order to solve this problem, a data cache queue mechanism is introduced as a buffer storage for each frame of image data, so that the system can continue to process the image data of the previously received frame while receiving a new frame of image data. The data cache queue is a first-in-first-out data structure, which is a special linear table in the data structure in the computer field, used to store each frame of image data waiting to be processed. The buffer queue is implemented in the existing programming language C++, and the read and write operations are managed by pointers in C++. One of the pointers is a read pointer pointing to the head of the data cache queue, which is used to take out a frame of image data from the data cache queue for processing; the other is a write pointer pointing to the tail of the data cache queue, which is used to add a new frame of image data to the data cache queue. The size of the data cache queue, that is, the number of frames of image data that can be stored, needs to be set according to the actual situation of the system to ensure that the system can process the image data in the queue in time before the queue is full. When the system receives new image data, if the frame of image data has not been fully processed, it will be placed in the data cache queue and wait. At the same time, the system will take out a frame of image data from the queue for processing in a first-in-first-out order. After processing a frame of image data, the next frame of image data will be taken out from the queue for processing, and so on. In this way, the data cache queue can act as a buffer between the reception and processing of a frame of image data, and can process multiple frames of image data in parallel, improving efficiency and real-time performance. Image data can avoid frame accumulation through the data cache queue, reduce memory consumption, and then each frame of image data is used to input the neural network model.

[0059] Step S5 includes the following steps:

[0060] S51, calculating the weight of each channel in the neural network model;

[0061] S52, calculating the actual influence of the weight on the loss function;

[0062] S53. Dynamically trim channels with small influence in the neural network model according to the influence size.

[0063] In the above step S51, the weight of each channel in the neural network model is calculated according to the following formula:

[0064]

[0065] Where L(·) represents the loss function, N represents the weight value of the current channel, (δ0, μ0) represents the translation and scaling transformation parameters of the current channel, h represents the high-order infinitesimal derived on the scaling transformation parameter δ0, and k represents the high-order infinitesimal derived on the translation transformation parameter μ0. represents a higher-order remainder;

[0066] In order to further measure the influence of the transformation parameters (δ0, μ0) on the loss function, the change value ΔL(·) of the loss function L(·) is expanded with respect to the first-order derivative of the scale transformation parameter δ and the translation transformation parameter μ as follows:

[0067]

[0068] Specifically, the above formula is the first-order Taylor expansion of the loss function L(·) at the weight coordinates (δ0, μ0) corresponding to a channel. Generally speaking, without considering the higher-order remainder, this formula limits the channels that have little impact on the neural network model. Even if the scale transformation parameter δ and the translation transformation parameter μ change, the loss value of the entire neural network model will not change dramatically, which means that the impact of the channel on network performance after being pruned is relatively small. The weights of each channel in the neural network model can be initialized to random numbers at the beginning. These random numbers can be generated by uniform distribution or normal distribution within a certain range. The purpose is to break the symmetry and ensure that each neuron has a unique input at the beginning of training. It is also possible to use specific values ​​or ranges to initialize the weights based on experience and the needs of specific problems. However, this practice is relatively rare because random initialization is effective enough and has better generalization ability.

[0069] In the back propagation algorithm in step S7, the first-order derivative of the specific channel parameter (δ0, μ0) with respect to L(·) can be obtained by the following composite derivation:

[0070]

[0071]

[0072] Among them, Y i ′ is the input value from the i-1th neuron to the current neuron, y i is the output value of the current neuron. It should be noted that the final impact size must be regressed to the cumulative sum of the gradients of all neurons to ensure the effectiveness and reliability of the final clipping.

[0073] As attached Figure 2 As shown in Figure 1, the influence of the scale transformation parameter δ and the translation transformation parameter μ of each channel in the neural network model on the model loss is calculated according to the absolute value of ΔL(·), and the importance of each channel to the neural network model can be obtained. Then, the channels below the threshold are trimmed according to the preset minimum influence threshold; as shown in Figure 1 Figure 2 As shown in FIG. 1 , the overall channel C of the neural network model is specifically divided into channels {C1, C2, …, Cn}. The corresponding channels are pruned according to the change value ΔL(·) mapped by the scale transformation parameter δ and the translation transformation parameter μ to reduce the number of channels of the neural network model. For example, after pruning channel C1, it becomes channels {C2, C3, …, Cn}. Finally, the channels {C2, C3, …, Cn} can be transformed and then merged into the overall channel C′. The merging of channels can be dynamically adjusted according to actual conditions. For example, highly similar or redundant channels need to be merged. Finally, the channels of the overall neural network are sparsely and lightly weighted, thereby improving the operation efficiency.

[0074] It should be noted that in the hidden layer of the neural network model, various activation functions need to be set to pass backward until the output. The activation functions include Sigmoid function, ReLU function or Tanh function, etc. Technical personnel in this field can flexibly adjust according to the various activation functions in the prior art. The purpose is to enable the neural network model to reintegrate the image data into new video data according to the frame number and timing of the image data for output. The number of neurons in the hidden layer can be flexibly adjusted according to the order of magnitude of the input image data.

[0075] The algorithm of the present invention can be applied to various video data processing fields, such as video detection or video recognition.

[0076] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0077] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. Video data processing acceleration algorithm based on convolutional neural network, It is characterized in that The following steps are involved: S1. Build a neural network model for processing video data; S2, extracting image data according to the frame number and timing of the video data, and preprocessing the image data; S3, performing inter-frame difference processing on the image data; S4, adding the image data to a data cache queue in sequence for accelerated processing; S5, calculating the weight of each channel in the neural network model, and dynamically cutting the number of channels of the neural network model according to the weight; S6, inputting the image data into the neural network model, and the neural network model outputs video data according to the frame number and timing of the image data; S7. Training the neural network model through a back propagation algorithm.

2. The video data processing acceleration algorithm based on convolutional neural network according to claim 1, It is characterized in that In step S1, the neural network model is a Faster R-CNN model.

3. The video data processing acceleration algorithm based on convolutional neural network according to claim 2, It is characterized in that The Faster R-CNN model is provided with a BN layer.

4. The video data processing acceleration algorithm based on convolutional neural network according to claim 1, 2 or 3, It is characterized in that The pre-processing in step S2 includes reducing the resolution of the image data.

5. The video data processing acceleration algorithm based on convolutional neural network according to claim 1, 2 or 3, It is characterized in that The step S3 performs inter-frame difference processing by the following formula: G k (x,y)=|P k (x,y)-P k-1 (x,y)|∩|P k+1 (x,y)-P k (x,y)|, Wherein, P represents the target image data, G represents the result image data, x and y represent the horizontal coordinate and vertical coordinate of the pixel point in the image data respectively, k represents the kth frame image data, k-1 represents the k-1th frame image data, and k+1 represents the k+1th frame image data.

6. The video data processing acceleration algorithm based on convolutional neural network according to claim 1, 2 or 3, It is characterized in that The step S5 comprises the following steps: S51, calculating the weight of each channel in the neural network model; S52, calculating the actual influence of the weight on the loss function; S53. Dynamically trim channels with small influence in the neural network model according to the influence size.

7. The video data processing acceleration algorithm based on convolutional neural network according to claim 6, It is characterized in that The step S51 calculates the influence of the weight on the loss function according to the following formula: Among them, L(·) represents the loss function, N represents the weight value of the current channel, (δ 0 ,μ 0 ) represent the translation and scaling factors of the current channel, and h represents the scaling factor δ 0 The higher-order infinitesimal of the derivative, k represents the translation factor μ 0 The higher-order infinitesimal of the derivative, T(δ 0 ,μ 0 ) represents a higher-order remainder.

Images