A video feature extraction method

Abstract

The present invention relates to a video feature extraction method, comprising: acquiring video frame images and extracting key frame images from the video frame images; obtaining an optical flow image based on the key frame images; fusing the key frame images with the optical flow image to obtain a fused image; and inputting the fused image into a pre-trained CNN feature extraction model to perform feature extraction and obtain video features. The present invention can simultaneously achieve both effective and rapid video feature extraction.

Description

Technical Field

[0001] The present invention relates to the technical field of video feature extraction, and in particular to a video feature extraction method. Background Art

[0002] The purpose of video feature extraction is to help people find videos of interest more easily, such as completing video classification, video understanding, video target tracking, action recognition, etc. Nowadays, the clarity and scale of videos are getting larger and larger, and methods that can stably extract features from video sequences are becoming more and more important.

[0003] Currently, image-based video feature extraction methods have very limited utilization of temporal features and are significantly affected by external interference (such as occlusion, lighting, and target movement speed). Their accuracy is often not robust enough in practical applications. Combining machine learning with statistical image features, such as HoG features and HoF features, for video feature extraction can achieve relatively good results, but the extraction efficiency is very limited and the speed is slow.

[0004] Therefore, it is urgent to propose a method / system that can not only take into account the effectiveness of video feature extraction but also complete feature extraction very quickly. Summary of the Invention

[0005] The purpose of the present invention is to provide a video feature extraction method that can simultaneously take into account the effectiveness of video feature extraction and can quickly complete video feature extraction.

[0006] To achieve the above objectives, an embodiment of the present invention provides a video feature extraction method, comprising:

[0007] Acquire a video frame image, and extract a key frame image from the video frame image;

[0008] Obtaining an optical flow image according to the key frame image;

[0009] Fusing the key frame image and the optical flow image to obtain a fused image;

[0010] The fused image is input into a pre-trained CNN feature extraction model for feature extraction to obtain video features.

[0011] Preferably, extracting the key frame image from the video frame image includes:

[0012] According to a preset frequency, a frame image is periodically extracted from the video frame image as a key frame image.

[0013] Preferably, obtaining an optical flow image according to the key frame image includes:

[0014] In 10 consecutive key frame images, 1 frame image is randomly extracted from the first 5 frames and 1 frame image is randomly extracted from the last 5 frames. The horizontal vector and vertical vector of the optical flow are obtained according to the extracted 2 frame key frame images. The horizontal vector and vertical vector are used as the first and second dimensional data of the optical flow image, and the third dimensional data is added. The value of the third dimensional data of the optical flow image is set to 0 to obtain the optical flow images corresponding to the 10 key frame images.

[0015] Preferably, fusing the key frame image and the optical flow image includes:

[0016] The 3D values ​​of each key frame image and its corresponding optical flow image are averaged respectively, and the average is used as the 3D value of the fused image to obtain the fused image.

[0017] Preferably, the CNN feature extraction model includes a Resnet-50 structure, and the Resnet-50 structure includes an input layer, layer0, layer1, layer2, layer3, and layer4 connected in sequence;

[0018] The input layer is used to perform L2 regularization on the fused image input into the CNN feature extraction model;

[0019] The layer0 consists of a convolutional layer with a kernel of 7*7 and a stride of 2, a BN layer, a ReLU layer, and a pooling layer with a 3*3 maximum pooling and a stride of 2;

[0020] The layer 1 includes three residual blocks connected in sequence, the layer 2 includes four residual blocks connected in sequence, the layer 3 includes six residual blocks connected in sequence, and the layer 4 includes three residual blocks connected in sequence.

[0021] Preferably, the residual blocks of layer1 to layer4 each include a channel attention module, a motion attention module and three convolution modules connected in sequence, the channel attention module is used to extract the channel attention features of the image of the input residual block to obtain a channel attention feature map, the motion attention module is used to extract the motion attention features of the image of the input residual block to obtain a motion attention feature map, the extracted channel attention feature map and the motion attention feature map are superimposed pixel by pixel and input into the three convolution modules for convolution processing in sequence to obtain a convolution feature map, and the convolution feature map is superimposed with the image of the input residual block and output.

[0022] Preferably, the three convolution modules of layer 1 are respectively 64-dimensional 1*1 convolution, 64-dimensional 3*3 convolution, and 256-dimensional 1*1 convolution.

[0023] Preferably, the three convolution modules of layer 2 are respectively 128-dimensional 1*1 convolution, 128-dimensional 3*3 convolution, and 512-dimensional 1*1 convolution.

[0024] Preferably, the three convolution modules of layer 3 are respectively 256-dimensional 1*1 convolution, 256-dimensional 3*3 convolution, and 1024-dimensional 1*1 convolution.

[0025] Preferably, the three convolution modules of layer 4 are respectively 512-dimensional 1*1 convolution, 512-dimensional 3*3 convolution, and 1024-dimensional 1*1 convolution.

[0026] The embodiments of the present invention have at least the following beneficial effects:

[0027] The embodiment of the present invention combines the two-dimensional spatial and temporal features of the image and adopts a convolutional neural network (CNN) to extract the features of the video. The input is a method of fusing frame images and optical flow images, and key frames are extracted from the input video to reduce the amount of calculation. The optical flow images are used to obtain the motion characteristics of the video. This method can not only take into account the effectiveness of video feature extraction, but also complete video feature extraction very quickly, which is beneficial to subsequent video processing such as video classification, video understanding, and action recognition.

[0028] Other features and advantages of the embodiments of the present invention will be described in the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 The figure is a flow chart of a video feature extraction method according to an embodiment of the present invention.

[0031] Figure 2 This is a structural diagram of the CNN feature extraction model in an embodiment of the present invention.

[0032] Figure 3 2 is a structural diagram of a residual block in an embodiment of the present invention.

[0033] Figure 4 2 is a structural diagram of the channel attention module in an embodiment of the present invention.

[0034] Figure 5 2 is a structural diagram of the motion attention module in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In addition, numerous specific details are provided in the following specific examples to better illustrate the present invention. Those skilled in the art will appreciate that the present invention can be practiced without certain specific details. In some instances, means well known to those skilled in the art are not described in detail in order to highlight the main points of the present invention.

[0036] See Figure 1 , an embodiment of the present invention proposes a video feature extraction method, comprising the following steps:

[0037] Step S10: acquiring a video frame image, and extracting a key frame image from the video frame image;

[0038] Step S20: obtaining an optical flow image according to the key frame image;

[0039] Step S30: fusing the key frame image and the optical flow image to obtain a fused image;

[0040] Step S40: Input the fused image into a pre-trained CNN feature extraction model to perform feature extraction to obtain video features.

[0041] The embodiment of the present invention combines the two-dimensional spatial and temporal features of the image and adopts a convolutional neural network (CNN) to extract the features of the video. The input is a method of fusing frame images and optical flow images, and key frames are extracted from the input video to reduce the amount of calculation. The optical flow images are used to obtain the motion characteristics of the video. This method can not only take into account the effectiveness of video feature extraction, but also complete video feature extraction very quickly, which is beneficial to subsequent video processing such as video classification, video understanding, and action recognition.

[0042] In some embodiments, step S10 includes:

[0043] According to a preset frequency, a frame image is periodically extracted from the video frame image as a key frame image.

[0044] Specifically, video data contains a large amount of repeated and invalid data, and the convolutional neural network (CNN) has a relatively large amount of computation, but it is independent in the time dimension because the processing of different frames is time-invariant; therefore, the convolution operations of different frames can be processed in parallel, while some repeated data is removed and the video frames are extracted, which can accelerate the calculation and facilitate deployment to complete end-to-end learning.

[0045] Among them, the video key frame is a frame used for video compression and video encoding and decoding, which contains a frame with complete information. Other non-key frames will be compressed using the difference with the key frame. Video frames can be specifically divided into three types of IPB frames: I frame represents the key frame, which is the most complete frame picture; P frame represents the single prediction frame, which uses the previous I frame or P frame and adopts motion prediction to perform inter-frame prediction encoding; B frame represents the bidirectional prediction frame, which uses the bidirectional frame for prediction encoding; In general, the key frame I frame is the frame with the most information and the most used frame. The embodiment of the present invention selects to extract the I frame to complete, and the preset frequency is to extract one frame at an interval of one second.

[0046] In some embodiments, step S20 includes:

[0047] In 10 consecutive key frame images, 1 frame image is randomly extracted from the first 5 frames and 1 frame image is randomly extracted from the last 5 frames. The horizontal vector and vertical vector of the optical flow are obtained according to the extracted 2 frame key frame images. The horizontal vector and vertical vector are used as the first and second dimensional data of the optical flow image, and the third dimensional data is added. The value of the third dimensional data of the optical flow image is set to 0 to obtain the optical flow images corresponding to the 10 key frame images.

[0048] Specifically, the optical flow represents the speed and direction of movement of each pixel in two adjacent frames of images, and is used to describe the instantaneous motion state (motion direction and motion offset) of a moving object. Its essence is a two-dimensional vector field. Each vector represents the displacement of the point in the scene from the previous frame to the next frame, which can be decomposed into a horizontal vector and a vertical vector. The horizontal vector (i.e., the first dimension) and the vertical vector (the second dimension) of the optical flow are rescaled to the range of [0, 255] and regarded as channels of the optical flow image. The key frame image includes three RGB channels, that is, the values ​​of the pixels of the image are represented by a three-dimensional data, such as [128, 200, 13]. In this embodiment, the optical flow image is set to correspond to the key frame image for easy fusion. Since the optical flow image only has two-dimensional eigenvalues, the third dimension, i.e., the third channel, is added, and the value is set to 0 to obtain the optical flow image.

[0049] In some embodiments, step S30 includes:

[0050] The 3D values ​​of each key frame image and its corresponding optical flow image are averaged respectively, and the average is used as the 3D value of the fused image to obtain the fused image.

[0051] In some embodiments, as Figure 2As shown, the CNN feature extraction model includes a Resnet-50 structure, and the Resnet-50 structure includes an input layer, layer0, layer1, layer2, layer3, and layer4 connected in sequence;

[0052] The input layer is used to perform L2 regularization on the fused image input into the CNN feature extraction model;

[0053] Specifically, since the distribution of optical flow is different from that of RGB images, the activation values ​​of the first convolutional layer will have different distributions. After initialization with the pre-trained model, to prevent overfitting and make the optimization solution stable and fast, the embodiment of the present invention uses L2 regularization on the image data of the input model. The main idea of ​​L2 regularization is to calculate the 2-norm of each image sample and then divide each element in the sample by the norm. The result of this processing is that the 2-norm (L2-norm) of each processed sample is equal to 1.

[0054] The layer0 consists of a convolutional layer with a kernel of 7*7 and a stride of 2, a batch normalization layer, a ReLU layer, and a pooling layer with a 3*3 maximum pooling and a stride of 2.

[0055] The layer 1 includes three residual blocks connected in sequence, the layer 2 includes four residual blocks connected in sequence, the layer 3 includes six residual blocks connected in sequence, and the layer 4 includes three residual blocks connected in sequence.

[0056] Among them, the residual blocks of layer1 to layer4 each include a channel attention module, a motion attention module and three convolution modules connected in sequence. The channel attention module is used to extract the channel attention features of the image of the input residual block to obtain a channel attention feature map. The motion attention module is used to extract the motion attention features of the image of the input residual block to obtain a motion attention feature map. The extracted channel attention feature map and the motion attention feature map are superimposed pixel by pixel and then input into the three convolution modules for convolution processing in sequence to obtain a convolution feature map. The convolution feature map is superimposed with the image of the input residual block and then output.

[0057] Specifically, the key to video analysis and processing lies in the learning and understanding of temporal features. This method combines the CNN network with the attention mechanism to learn the spatial and temporal features of the video more completely, thereby achieving feature extraction of the video and better completing subsequent video classification, action recognition and other related tasks.

[0058] Among them, the three convolution modules of layer1 are 64-dimensional 1*1 convolution, 64-dimensional 3*3 convolution, and 256-dimensional 1*1 convolution.

[0059] Among them, the three convolution modules of the layer2 are 128-dimensional 1*1 convolution, 128-dimensional 3*3 convolution, and 512-dimensional 1*1 convolution.

[0060] Among them, the three convolution modules of the layer3 are 256-dimensional 1*1 convolution, 256-dimensional 3*3 convolution, and 1024-dimensional 1*1 convolution.

[0061] Among them, the three convolution modules of layer4 are 512-dimensional 1*1 convolution, 512-dimensional 3*3 convolution, and 1024-dimensional 1*1 convolution.

[0062] See Figure 2 In the embodiment of the present invention, the main part of the CNN feature extraction model is the Resnet-50 structure, which consists of 5 layers, the core of which is the residual block (Bottleneck Block). Resnet-50 uses residual blocks to weaken the strong connection between each layer.

[0063] Taking the residual block of layer1 as an example, the residual block is specifically described. Figure 3 As shown in the figure, after the image enters the residual block of layer 1, it is divided into two paths. One path first reduces the 256-dimensional channel to 64-dimensional channels through a 1*1 convolution, then undergoes a 3*3 convolution, and finally recovers it through a 1*1 convolution with 256-dimensional channels, followed by a Batch Normalization (BN) layer and a Rectified Linear Unit (ReLU) layer. The other path is directly connected, using a 1*1 convolution with 256-dimensional channels to reduce the output of the maximum pooling layer to 255*56*56, and then adding the two and outputting them through ReLU. The principles of the residual blocks of other layers 2 / layer 3 / layer 4 are similar, differing only in the channel dimension.

[0064] Table 1 Resnet50 structure diagram

[0065]

[0066] Video images have strong temporal relationships, with previous and subsequent frames interconnected. The addition of the attention mechanism module enhances the model's ability to interpret temporal features. Furthermore, the features extracted by the convolutional neural network for each frame also have temporal connections between channels. Therefore, the present invention models these features one by one and combines them to extract more video features as input to subsequent networks.

[0067] Among them, the channel attention (CE, Channel Excitation) module is based on the design of SE-Net, but because the video has an additional one-dimensional temporal information, a 1D convolution is performed on the channel layer to enhance the mutual dependence of the channels in the time domain and obtain a channel-based attention feature map. The obtained attention feature map is then multiplied by the channel of the input feature to strengthen the input feature and convoluted with the input feature (the output feature of the previous layer). The structural diagram is shown below. Figure 4 shown.

[0068] Among them, the Motion Excitation (ME) module mainly describes the movement of the action between each two adjacent frames. It uses the frame difference between the previous and next frames of the video image to first expand the 2D convolution in the time dimension, then splice the results together, and then perform subsequent pooling and convolution operations. Specifically, the input is the feature map of the image after being processed by the Layer0 layer in the Resnet50 structure. The motion feature F m Through a 2D convolution (calculation method, F m =K*F t+1 -F t ) for the two frames before and after F t+1 and F t To perform the operation, the difference between the two frames of the input X is spliced ​​in the time dimension (the last one is padded with 0), and then the obtained features are subjected to spatial pooling, 2D convolution, and Sigmoid to obtain the motion attention feature (ME feature), where Indicates element-by-element addition, the module structure diagram is as follows Figure 5 shown.

[0069] The channel attention module and motion attention module are added at the beginning of each residual block in the Resnet-50 structure. After the first layer of 7x7 convolution, the input of each layer is connected to the CE and ME attention modules, and then 1x1 and 3x3 convolutions are performed to complete feature extraction. This preserves the original Resnet structure to the greatest extent possible, allowing the use of pre-trained models from currently available public datasets, accelerating training and improving the efficiency of video feature extraction.

[0070] In summary, the embodiments of the present invention have the following advantages:

[0071] (1) The frame image and the optical flow image are fused as input. The frame image is obtained by the key frame extraction method. After obtaining the key frame, the optical flow image is randomly extracted from two frames of the 10 frames before and after the key frame for feature extraction. The optical flow features are stored in the form of a graph, and the third dimension is set to 0 to obtain the optical flow image extracted from the video. At the same time, the input is L2 regularized to prevent overfitting and accelerate network fitting.

[0072] (2) Video feature extraction using a CNN network combined with an attention mechanism. The main architecture is the Resnet-50 structure. The input of the first Bottleneck Block module in each layer is connected to the CE and ME attention modules. The CE module is used to enhance the interdependence of channels in the time domain and obtain a channel-based attention feature map. The ME module describes the movement of actions between each two adjacent frames and expands the 2D convolution in the time dimension to enhance the motion characteristics of the video sequence.

[0073] (3) In the process of video feature extraction, the mutual dependence of channels in the time domain is enhanced, and a channel-based attention feature map is obtained, so that the temporal features of the video are fully utilized. The ME module strengthens the motion features of the image between two adjacent frames. Compared with the original Resnet50 network, it does not increase the computational complexity too much, but increases the accuracy of video feature extraction.

[0074] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A video feature extraction method, characterized in that: include: Acquire a video frame image and extract a key frame image from the video frame image; periodically extract a frame image from the video frame image as a key frame image according to a preset frequency; Obtaining an optical flow image according to the key frame image, randomly extracting one frame of image from the first five frames of image and one frame of image from the last five frames of image among 10 consecutive key frame images, obtaining a horizontal vector and a vertical vector of the optical flow according to the extracted two key frame images, using the horizontal vector and the vertical vector as the first and second dimensional data of the optical flow image, adding third dimensional data, and setting the value of the third dimensional data of the optical flow image to 0, to obtain the optical flow images corresponding to the 10 key frame images; Fusing the key frame image and the optical flow image to obtain a fused image; averaging the values ​​of the three-dimensional data of each key frame image and its corresponding optical flow image, and using the mean as the value of the three-dimensional data of the fused image to obtain a fused image; The fused image is input into a pre-trained CNN feature extraction model for feature extraction to obtain video features.

2. The video feature extraction method according to claim 1, wherein The CNN feature extraction model includes a Resnet-50 structure, and the Resnet-50 structure includes an input layer, layer0, layer1, layer2, layer3, and layer4 connected in sequence; The input layer is used to perform L2 regularization on the fused image input into the CNN feature extraction model; The layer0 consists of a convolutional layer with a kernel of 7*7 and a stride of 2, a BN layer, a ReLU layer, and a pooling layer with a 3*3 maximum pooling and a stride of 2; The layer 1 includes three residual blocks connected in sequence, the layer 2 includes four residual blocks connected in sequence, the layer 3 includes six residual blocks connected in sequence, and the layer 4 includes three residual blocks connected in sequence.

3. The video feature extraction method according to claim 2, wherein: The residual blocks of layer1~layer4 each include a channel attention module, a motion attention module and three convolution modules connected in sequence. The channel attention module is used to extract the channel attention features of the image of the input residual block to obtain a channel attention feature map. The motion attention module is used to extract the motion attention features of the image of the input residual block to obtain a motion attention feature map. The extracted channel attention feature map and the motion attention feature map are superimposed pixel by pixel and then input into the three convolution modules for convolution processing in sequence to obtain a convolution feature map. The convolution feature map is superimposed with the image of the input residual block and then output.

4. The video feature extraction method according to claim 3, wherein: The three convolution modules of layer 1 are 64-dimensional 1*1 convolution, 64-dimensional 3*3 convolution, and 256-dimensional 1*1 convolution.

5. The video feature extraction method according to claim 3, wherein: The three convolution modules of layer 2 are 128-dimensional 1*1 convolution, 128-dimensional 3*3 convolution, and 512-dimensional 1*1 convolution.

6. The video feature extraction method according to claim 3, wherein: The three convolution modules of layer3 are 256-dimensional 1*1 convolution, 256-dimensional 3*3 convolution, and 1024-dimensional 1*1 convolution.

7. The video feature extraction method according to claim 3, wherein: The three convolution modules of layer4 are 512-dimensional 1*1 convolution, 512-dimensional 3*3 convolution, and 1024-dimensional 1*1 convolution.

Images