A video classification and recognition method based on AVS inter-frame prediction coding

Abstract

A video classification and recognition method based on AVS inter-frame prediction coding includes: reading a video; obtaining a key frame (I frame): using an AVS decoder to decode and obtain a key frame (I frame) in the video coding as a starting frame; appending a prediction frame (P frame), a bidirectional prediction frame (B frame) and a key frame (I frame): appending a prediction frame (P frame), a bidirectional prediction frame (B frame) and a key frame (I frame) after the key frame (I frame) as input data of a neural network; and neural network classification prediction: inputting the intercepted AVS-coded video stream data into the neural network for video classification and recognition. The method of the present invention uses a neural network algorithm based on a self-attention mechanism, receives video stream compression data generated by inputting AVS codec, establishes a time dimension relationship between video frames, and finally realizes the function of video classification and recognition, thereby solving the problem of input data redundancy caused by video decoding inputting into the neural network in the old method.

Description

Technical Field

[0001] The invention belongs to the field of computer vision technology, and in particular relates to a video classification and recognition method based on AVS inter-frame prediction coding. Background Art

[0002] With the popularization of smart terminal devices, a large amount of audio and video is spread on the Internet, especially a large number of short videos. These short videos often need to be classified and retrieved. In the era of big data, manual labeling methods are difficult to meet the needs of massive video labeling, and the use of video intelligent classification and recognition algorithms is one of the effective ways to solve the classification of massive short videos. The current mainstream video classification and recognition algorithms are neural network algorithms based on deep learning, especially the emergence of visual self-attention neural networks (Version Transformer, ViT), which further improves video classification technology. Once these mainstream neural networks are designed and trained, the model input data volume is fixed, that is, the number of neurons at the input position is fixed. The decoded video frame data is the input data, and the data volume of this data is huge, reaching millions of levels. Therefore, the neural network requires a large number of input neurons, which will lead to the problem of large computational complexity in neural network calculations. At the same time, the digital audio and video coding standard (Audio Video Coding Standard, referred to as AVS) led by China is gradually being promoted and used, and the third-generation AVS3 standard has been launched. In the AVS standard, video coding uses inter-frame prediction compression coding technology to greatly compress video coding data.

[0003] In the prior art, video stream data is generally decoded into complete image frames, and then input into the CNN (convolutional neural network) network to perform convolution calculations on the image frames, which will result in a large amount of redundant data input and cause unnecessary calculations. Therefore, it is necessary to develop a new video decoding method to solve the problem of redundant neural network input data caused by the existing video decoding method. Summary of the invention

[0004] The present invention provides a video classification and recognition method based on AVS inter-frame prediction coding. When video stream data is input into a neural network, compressed data is sampled and input; in particular, a neural network algorithm based on a self-attention mechanism is used to receive video stream compressed data generated by input AVS codec, establish a time dimension relationship between video frames, and finally realize the function of video classification and recognition, thereby solving the problem of input data redundancy caused by video decoding input into the neural network in the old method.

[0005] The technical solution of the present invention is as follows:

[0006] The video classification and recognition method based on AVS inter-frame prediction coding of the present invention comprises the following steps: S1. reading the video; S2. obtaining the key frame (I frame): decoding with the AVS decoder, obtaining the key frame (I frame) in the video coding as the starting frame; S3. appending the prediction frame (P frame), the bidirectional prediction frame (B frame) and the key frame (I frame): appending the prediction frame (P frame), the bidirectional prediction frame (B frame) and the key frame (I frame) after the key frame (I frame) as the input data of the neural network; and S4. neural network classification prediction: inputting the intercepted AVS encoded video stream data into the neural network for video classification and recognition.

[0007] Preferably, in the above-mentioned video classification and recognition method based on AVS inter-frame prediction coding, in step S1, the AVS encoding and decoding standard technology is used to compress and encode the video into the AVS video encoding format during the video reading stage, and the AVS decoder is used to decode it to obtain the AVS decoded video stream data.

[0008] Preferably, in the above-mentioned video classification and recognition method based on AVS inter-frame prediction coding, in step S1, the video is uniformly converted into an AVS-coded video stream, and then the video is read.

[0009] Preferably, in the above-mentioned video classification and recognition method based on AVS inter-frame prediction coding, in step S2, video code stream data of a specified length is obtained from the video stream to obtain a code stream segment, and the position of the key frame (I frame) is used as the starting position of the code stream. The AVS decoder reads the video and obtains the key frame (I frame) at a certain time point from the video.

[0010] Preferably, in the above-mentioned video classification and recognition method based on AVS inter-frame prediction coding, in step S3, a sampling and filling technique is used to add a certain number of prediction frames (P frames), bidirectional prediction frames (B frames) and key frames (I frames) after the first key frame (I frame), so as to fully retain the video information and meet the requirements of neural network input.

[0011] Preferably, in the above-mentioned video classification and recognition method based on AVS inter-frame prediction coding, in step S3, the AVS-coded video code stream is input into the neural network, classification prediction calculation is performed, and the classification prediction result is output.

[0012] Preferably, in the above-mentioned video classification and recognition method based on AVS inter-frame prediction coding, in step S3, after the neural network design is completed, the number of neurons at the input position is fixed, and when the data amount of the video code stream is less than the input amount of the neural network, 0 is used for filling; when the data amount of the video code stream is more than the neural network, some subsequent appended frames will be appropriately removed.

[0013] Preferably, in the above-mentioned video classification and recognition method based on AVS inter-frame prediction coding, in step S3, when the video picture changes dramatically and the number of key frames (I frames) is large, a certain number of key frames (I frames) are sampled and removed; when the video picture content changes little and the number of key frames (I frames) is small, a certain number of prediction frames (P frames) and bidirectional prediction frames (B frames) are sampled and removed.

[0014] Preferably, in the above-mentioned video classification and recognition method based on AVS inter-frame prediction coding, in step S4, a neural network model based on a self-attention mechanism is used to receive good video stream data and perform video classification and recognition calculations.

[0015] According to the technical solution of the present invention, the beneficial effects produced are:

[0016] The video classification and recognition method based on AVS inter-frame prediction coding of the present invention aims at the problem of redundant input data of the neural network caused by video decoding, and uses AVS compression coding to compress and encode the video data stream input to the neural network, thereby reducing the amount of input data and effectively solving the problem of redundant input data caused by video decoding input to the neural network. According to the characteristics of AVS video compression coding, the method of the present invention samples the video data stream after AVS compression coding as the input data of the self-attention mechanism neural network, reducing the amount of input data, thereby avoiding the problem of a large amount of redundant data input to the neural network after video decoding, and finally realizing the video classification function under the condition of reducing redundant input data.

[0017] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0019] Figure 1 It is a flow chart of the video classification and recognition method based on AVS inter-frame prediction coding of the present invention;

[0020] Figure 2 It is a schematic diagram of inter-frame prediction coding of AVS video frames of the present invention;

[0021] Figure 3 is a schematic diagram of a self-attention mechanism neural network of the present invention; DETAILED DESCRIPTION

[0022] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0023] The video classification and recognition method based on AVS inter-frame prediction coding of the present invention is a method for solving the redundancy of neural network input data in video classification and recognition tasks. Specifically, the video data stream is encoded using an AVS encoder, and the encoded data is sampled as input data of the neural network, thereby achieving the effect of reducing redundant data input to the neural network. The AVS inter-frame prediction compression encoding and sampling algorithm of the present invention can effectively alleviate the problem of redundant neural network input data caused by video frame decoding, thereby solving the problem of a large amount of redundant data input to the neural network due to video frame decoding in the video classification and recognition algorithm.

[0024] The principle of the present invention is: using AVS inter-frame prediction coding to compress the input data of the video classification neural network, thereby realizing the function of reducing input redundant data. Specifically, the compressibility of AVS inter-frame prediction coding on data is used to reduce the input data of the neural network, thereby reducing the amount of neural network calculation, thereby solving the problem of redundant input data caused by inputting the neural network after video frame decoding in the old method.

[0025] like Figure 1 FIG. 1 is a flowchart of a video classification and recognition method based on AVS inter-frame prediction coding of the present invention, and the specific steps are as follows:

[0026] S1. Read video: Use AVS codec standard technology to read video, compress and encode the video into AVS video encoding format, and decode it using AVS decoder.

[0027] At present, the more popular video compression codec standards include MPEG-4, H.264, AVS, etc. The video encoding of different standards is different, resulting in different encoded video bitstreams for videos with the same content. A single neural network model can generally only input one of the uniformly encoded bitstreams. At the same time, in view of the AVS video codec standard, which has compression technology and specifications for efficient video inter-frame prediction, the present invention uniformly converts the video into an AVS-encoded video stream, and then, in the video reading stage, uses the AVS decoder to obtain the AVS-decoded video bitstream data.

[0028] S2. Get the key frame (I frame): Use the AVS decoder to decode and get the I frame in the video encoding as the starting frame. Because AVS encodes the I frame (key frame) integrity, the I frame can be used as the starting frame.

[0029] like Figure 2Shown is a schematic diagram of the inter-frame prediction coding of the AVS video frame of the present invention. In AVS coding, the key frame (I frame) does not depend on other frames before and after, it is a complete frame of video image; the prediction frame (P frame) only needs to use the I frame to perform prediction calculations to obtain the P frame; the B frame (bidirectional prediction frame) needs to use the P frame and the I frame at the same time to perform prediction calculations to obtain the B frame. The above inter-frame prediction coding method of the video frame is a general implementation method specified in the AVS standard and is implemented by the AVS encoder. The video classification neural network, the input data is the bitstream data of the video segment, therefore, it is necessary to obtain the video bitstream data of a specified length from the video stream to obtain the bitstream segment, and the position of the I frame is used as the starting position of the bitstream. The AVS decoder reads the video and obtains the I frame at a certain time point from the video.

[0030] S3. Append predicted frames (P frames) and bidirectional predicted frames (B frames): Append predicted frames (P frames), bidirectional predicted frames (B frames) and key frames (I frames) after I frames as input data of the neural network. Specifically, a certain number of P frames, B frames and key frames (I frames) are appended after the first I frame using sampling and padding techniques to fully retain video information while meeting the requirements of neural network input.

[0031] Video is composed of video frames. For AVS-encoded videos, a video clip is composed of multiple video I frames, P frames, and B frames. In step S3, a certain I frame is used as the starting frame, and then a certain number of P frames, B frames, and I frames are appended. For example, a video with a frame rate of 30fps and a duration of 10s has a total of 300 frames. These 300 frames are generally composed of a small number of I frames and a large number of P frames and B frames. Figure 3 As shown, it is a schematic diagram of the self-attention mechanism neural network of the present invention. The AVS-encoded video stream will be input into the neural network, and the classification prediction calculation will be performed to output the classification prediction result. After the neural network design is completed, the input size will be fixed, that is, the number of neurons at the input position is fixed. When the amount of data of the video stream is less than the input amount of the neural network, 0 is used for padding; when the amount of data of the video stream is more than the neural network, some subsequent additional frames will be appropriately removed. The way to remove frames needs to be determined according to the actual video classification content, specifically: when the video picture changes drastically and the number of I frames is large, a certain number of I frames are sampled and removed; when the video picture content changes little and the number of I frames is small, a certain number of P frames and B frames are sampled and removed. In this way, the requirement of fixed number of neural network inputs is met. Compared with the existing video classification method, the present invention no longer needs to decode P frames and B frames into images of the same size as I frames, thereby reducing redundant data input and greatly reducing the amount of calculation of the neural network. This is the core of the method of the present invention.

[0032] S4. Neural network classification prediction: The intercepted AVS encoded video stream data is input into the neural network for video classification and recognition. Specifically, a neural network model based on the self-attention mechanism is used to receive good video stream data and perform video classification and recognition calculations.

[0033] The present invention selects a neural network model based on the self-attention mechanism. Most traditional video classification videos use a CNN (convolutional neural network) model. The CNN model assumes the positional relationship of the spatial dimension, and it is difficult to establish the association of the time dimension. The neural network models based on the self-attention mechanism currently used for video classification and recognition are becoming more and more mature, such as the ViT (Version Transformer, visual self-attention neural network) neural network model, which is a general, open source algorithm model. The present invention inputs the bitstream data obtained in step S3 into the ViT model, and outputs the classification result after model calculation. At this point, the video classification and recognition task is completed.

[0034] The method of the present invention utilizes the inter-frame compression coding characteristics in AVS coding, inputs the encoded video code stream data into a self-attention mechanism neural network, uses the AVS encoder for encoding compression, and realizes the function of video classification and recognition, thereby solving the problem of neural network input data redundancy caused by video frame decoding in the existing method.

[0035] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A video classification and recognition method based on AVS inter-frame prediction coding, comprising the following steps: S1. Read the video, Video is composed of video frames. A video encoded using AVS is composed of multiple video I frames, P frames, and B frames. Using the AVS codec standard technology, in the video reading stage, the video is compressed and encoded into the AVS video encoding format, and the AVS decoder is used to decode it to obtain the AVS decoded video stream data, the video is uniformly converted into the AVS encoded video stream, and then the video is read; S2. Get the key frame, that is, the I frame: Use the AVS decoder to decode and get the key frame in the video encoding, that is, the I frame as the starting frame. Get the video bitstream data of the specified length from the video stream to get the bitstream segment, and the key frame, that is, the position of the I frame, is used as the starting position of the bitstream. The AVS decoder reads the video and gets the key frame at a certain time point, that is, the I frame, from the video. S3. appending predicted frames, i.e., P frames, bidirectional predicted frames, i.e., B frames, and key frames, i.e., I frames: appending predicted frames, i.e., P frames, bidirectional predicted frames, i.e., B frames, and key frames, i.e., I frames, after the key frames, i.e., I frames, as input data of the neural network; Using sampling and padding techniques, a certain number of predicted frames, i.e., P frames, bidirectional predicted frames, i.e., B frames, and key frames, i.e., I frames, are appended to the first key frame, i.e., I frame, to fully preserve the video information while meeting the neural network input requirements; Input the AVS-encoded video code stream into the neural network, perform classification prediction calculation, and output the classification prediction result; After the neural network is designed, the number of neurons at the input position is fixed. When the amount of data in the video stream is less than the input amount of the neural network, it is filled with 0; when the amount of data in the video stream is more than the neural network, some subsequent appended frames will be appropriately removed; When the video picture changes dramatically and the number of key frames, i.e., I frames, is large, a certain number of key frames, i.e., I frames, are sampled and removed; when the video picture content changes slightly and the number of key frames, i.e., I frames, is small, a certain number of prediction frames, i.e., P frames and bidirectional prediction frames, i.e., B frames, are sampled and removed, so as to meet the requirement of fixed number of neural network inputs; and S4. Neural network classification prediction: The intercepted AVS encoded video stream data is input into the neural network for video classification and recognition.

2. The video classification and recognition method based on AVS inter-frame prediction coding according to claim 1 is characterized in that: In step S4, a neural network model based on a self-attention mechanism is used to receive good video stream data and perform video classification and recognition calculations.

Images