Video content description method, medium and electronic equipment

Abstract

The invention relates to a video content description method. The method comprises the following steps: S1, acquiring a video stream; s2, based on a video stream adjacent frame motion feature extraction algorithm and a clustering algorithm, extracting a preset number of key frames from the video stream; s3, extracting regional features from the preset number of key frames by adopting a target detection algorithm of regional feature extraction; s4, according to the regional features, calculating corresponding regional attention features based on a self-attention mechanism algorithm; s5, extracting video frame features of each frame by adopting a classification algorithm and a video stream adjacent frame motion feature extraction algorithm according to the video stream; and S6, according to the regional attention features and the video frame features, decoding by using a cyclic network algorithm to obtain a description text of the video content.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and deep learning, and in particular to a video content description method, medium and electronic device. Background Art

[0002] With the explosive growth of surveillance video stream data in railway public security scenarios, it is crucial to quickly analyze videos and accurately and quickly understand their content for safe railway operation. Its main application scenarios include but are not limited to the following two aspects:

[0003] On the one hand, when an emergency occurs, such as a criminal activity or an unexpected incident, every second may affect the direction and outcome of the incident. Being able to obtain intelligence as quickly as possible and conduct incident analysis means that you can take prompt action to prevent further crimes and reduce casualties and property losses. For example, in the process of hunting down a fugitive, by quickly understanding the video content, you can accurately grasp the fugitive's whereabouts and characteristics, providing key clues for the capture operation.

[0004] On the other hand, daily public security management also requires efficient video content understanding. Surveillance videos cover every corner of the city. Rapid analysis of these videos can timely discover potential safety hazards and prevent crimes. At the same time, for the security of large-scale events, rapid understanding of video content can monitor crowd dynamics, traffic conditions, etc. in real time to ensure the safety of the event.

[0005] With the development of machine learning and artificial intelligence technology, more and more models have been applied to the field of video content description. However, the research found that the defects of the existing technology are: limited feature extraction capabilities, different existing models often extract features of some aspects (for example, the common one is capturing motion features), and easily ignore features in other aspects, because the description text finally generated is usually not accurate; the model often generates some content that does not appear in the video based on some prior knowledge, especially when the data distribution of the test set and the training set is quite different.

[0006] Therefore, it is necessary to provide a more appropriate and higher quality video description method. Summary of the invention

[0007] In view of the above problems, the purpose of the present invention is to provide a video content description method, which can integrate multiple feature extraction methods and self-attention mechanisms, accurately grasp the core content of the video, carefully extract regional features and accurately classify them, comprehensively extract video frame features, and generate natural and accurate key frame content descriptions. It has broad application prospects and strong adaptability and robustness.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present application provides a video content description method, the method comprising:

[0010] S1, obtain video stream;

[0011] S2, extracting a preset number of key frames from the video stream based on a video stream adjacent frame motion feature extraction algorithm and a clustering algorithm;

[0012] S3, using a target detection algorithm for regional feature extraction to extract regional features from a preset number of key frames;

[0013] S4, calculating the corresponding regional attention features based on the self-attention mechanism algorithm according to the regional features;

[0014] S5, extracting video frame features of each frame using a classification algorithm and a video stream adjacent frame motion feature extraction algorithm according to the video stream;

[0015] S6, decoding is performed using a recurrent network algorithm according to the regional attention features and the video frame features to obtain a description text of the video content.

[0016] In one implementation, in S2, based on the motion information of objects in adjacent frames calculated by the motion feature extraction algorithm for adjacent frames of the video stream, the clustering algorithm is improved to achieve clustering of the video stream.

[0017] In one implementation, the video stream adjacent frame motion feature extraction algorithm adopts the Optical Flow algorithm to find the corresponding relationship between the previous frame and the current frame based on the change of pixels in the image sequence in the time domain and the correlation between adjacent frames, thereby calculating the motion information of objects between adjacent frames.

[0018] In one implementation, the calculation process of the clustering algorithm includes:

[0019] Randomly select a preset number of frames as cluster centers;

[0020] Start iterative calculation, calculate the distance between each frame in the video stream and the current key frame, and classify each frame into the cluster where the key frame with the smallest distance is located;

[0021] After one iteration, the average value of all frames in the cluster is calculated as the new cluster center;

[0022] Execute the next iteration calculation until the preset termination condition is met.

[0023] In one implementation, in S3, Faster RCNN is used to extract regional features from a preset number of key frames.

[0024] In one implementation, in S5, a ResNet network is used for classification; and an algorithm for extracting motion features of adjacent frames of a video stream is an Optical Flow algorithm.

[0025] In one implementation, in S6, the recurrent network algorithm includes Attention LSTM and Language LSTM.

[0026] In one implementation, the Attention LSTM takes the video frame features as input, decodes the global features of the video, the position encoding information of the video clip, and the word vector of the input text, and decodes the hidden state;

[0027] Language LSTM decodes the hidden state and regional attention features to obtain the description text.

[0028] In a second aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the video content description method described in the first aspect.

[0029] According to a third aspect, an electronic device is provided, comprising a processor and a memory; the memory stores a computer program, and the processor runs the computer program to control the electronic device to execute the video content description method according to the first aspect.

[0030] By integrating multiple algorithms, the present invention can efficiently extract key information from videos, accurately obtain regional features of key frames, accurately classify regional features, fully extract features of video frames, and finally generate sentences that can accurately describe the content of key frames, thereby providing powerful technical support for applications such as video analysis, retrieval, and summary generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of a video content description method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0033] In view of the problems of the prior art, the technical solution of the present invention provides a video content description method, the method comprising:

[0034] Get the video stream;

[0035] Extracting a preset number of key frames from the video stream based on a video stream adjacent frame motion feature extraction algorithm and a clustering algorithm;

[0036] The target detection algorithm using regional feature extraction is used to extract regional features from a preset number of key frames;

[0037] According to the regional features, based on the self-attention mechanism algorithm, the corresponding regional attention features are calculated;

[0038] According to the video stream, a classification algorithm and a video stream adjacent frame motion feature extraction algorithm are used to extract the video frame features of each frame;

[0039] According to the regional attention features and the video frame features, a recurrent network algorithm is used for decoding to obtain a description text of the video content.

[0040] Please refer to more drawings of the embodiments of the present invention, in which the method and medium provided by the present invention are further explained in more detailed embodiments of the present invention.

[0041] like Figure 1 In one aspect of the embodiments of the present application, a video content description method is provided, the method comprising:

[0042] S1, obtain video stream;

[0043] S2, extracting a preset number of key frames from the video stream based on a video stream adjacent frame motion feature extraction algorithm and a clustering algorithm;

[0044] Specifically, in this embodiment, video content description is implemented based on five modules, including a key frame extraction module, a region feature extraction module, a region attention module, a video frame feature extraction module and a sentence generation module.

[0045] In this step, the key frame extraction module includes two algorithm parts: Kmeans clustering and Optical Flow algorithm.

[0046] The algorithm first inputs the video stream data into the Kmeans clustering algorithm, randomly selects k frames as cluster centers, and sets up a benchmark in the ocean of data. Then the distance between each frame and the key frame in the video is calculated, and the frame is classified into the cluster where the key frame with the smallest distance is located. After one iteration, the average value of all frames in the cluster is calculated as the new cluster center, and the iteration cycle is continued for n times until the termination condition is met.

[0047] At the same time, the Optical Flow algorithm is added to improve the Kmeans clustering results so that the obtained k key frames can more accurately represent the original video information. Optical Flow is the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane. The optical flow method uses the changes in pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. Optical Flow optimizes the Kmeans algorithm. Since there is information about the change of object movement in the video time sequence, adding optical flow can make the Kmeans algorithm more focused on the object's motion information.

[0048] In this step, the number of key frames extracted may be, for example, 10 frames.

[0049] S3, using a target detection algorithm for regional feature extraction to extract regional features from a preset number of key frames;

[0050] Specifically, this step is implemented by a regional feature extraction module.

[0051] The regional feature extraction module uses the target detection algorithm Faster RCNN (Faster Region-based Convolutional Neural Network) to extract regional features from the 10 extracted key frames.

[0052] The Faster-RCNN network is divided into four parts, namely the backbone feature extraction network (Backbone), the candidate region proposal network, the region of interest pooling layer and the classification and regression layer. First, the 10 key frames to be detected are input into the first convolutional neural network: the VGG16 visual cluster network model, and the feature map of the image is extracted after convolution, activation and pooling operations; then the extracted feature map is input into the second network: the RPN region candidate network, the region proposal box is generated, and the normalized exponential function softmax regression is used to obtain more accurate proposal box position information; finally, the feature map obtained in the first step and the proposal box position information obtained in the second step are input into the region of interest pooling layer for processing, and the relevant target fully connected layers (FC) feature vectors are extracted, and classification regression is performed to determine the category to which the proposal box belongs, and the coordinates of these proposal boxes are fine-tuned to obtain more accurate positions.

[0053] S4, calculating the corresponding regional attention features based on the self-attention mechanism algorithm according to the regional features;

[0054] Specifically, this step is implemented by a regional attention module.

[0055] The Self-Attention mechanism algorithm is used to classify regional features, obtain category information and classification relationship matrix, and then input this information into the regional feature attention module to obtain the regional attention features.

[0056] S5, extracting video frame features of each frame using a classification algorithm and a video stream adjacent frame motion feature extraction algorithm according to the video stream;

[0057] Specifically, the video frame feature extraction module is divided into two algorithm parts: ResNet-200 (Deep Residual Network with 200 layers) and Optical Flow algorithm. For each frame of the input video data, ResNet-200 is used to classify and extract the fully connected layer features of ResNet-200. The Optical Flow method is then used to obtain the change relationship between frames and obtain the action motion information of the video frame. This processing method can make the prediction results more accurate and ultimately obtain the Frame-wise global feature information of the video frame.

[0058] ResNet is a deep convolutional residual network structure. Its core concept is to solve the gradient vanishing, gradient exploding and degradation problems in the training process of deep convolutional neural networks by introducing residual connections, so as to build a deeper network structure. The core of the ResNet-200 network is the residual block structure, which uses 200 layers of residual network layers, mainly including input layer, residual block, global average pooling layer, fully connected layer and output layer. ① Input layer: used to accept input video data. ② Residual block: As the most important part of ResNet, these blocks are connected through residual connections, so that information can be directly transmitted between layers. ResNet-200 contains multiple residual blocks, each of which contains several convolutional layers and identity mapping. ③ Global average pooling layer: The output of the last residual block is globally average pooled to convert the feature map into a one-dimensional vector. ④ Fully connected layer: Use the fully connected layer for classification or regression tasks. In ResNet-200, a fully connected layer with an appropriate number of neurons is usually used, and finally a Softmax layer is connected for classification. ⑤Output layer: Depending on the task, the output layer can be one or more neurons, which are used to output the network's prediction results.

[0059] S6, decoding is performed using a recurrent network algorithm according to the regional attention features and the video frame features to obtain a description text of the video content.

[0060] Specifically, the sentence generation module consists of two parts: Attention LSTM and Language LSTM. Attention LSTM takes the video frame-wise features as input, decodes the global features of the video, the position encoding information of the video clips and the word vector of the input text, and decodes them into hidden states. Then, Language LSTM decodes the hidden states and regional feature attention features, and decodes them into words one by one to generate sentence descriptions.

[0061] To facilitate understanding of this step, the Attention LSTM (Attention Long Short-Term Memory Network) and Language LSTM (Language Long Short-Term Memory Network) are explained as follows:

[0062] LSTM is a variant of RNN (Recurrent neural networks). Its unique feature is that it can maintain short-term memory for a long time and is widely used to process time series information. LSTM is mainly composed of forget gate, input gate, output gate and other structures. Its core idea is to introduce mechanisms such as forget gate, input gate and output gate. These structures enable LSTM neural network to capture and model long-term dependencies in time series data and extract time feature information from multi-dimensional time series data, thereby solving the problem of gradient vanishing and gradient exploding in RNN when processing long sequences.

[0063] The Attention mechanism is a mechanism that can improve the prediction effect of LSTM. In the LSTM time series prediction task, this mechanism can make the model focus more on the input information related to the current prediction moment, thereby improving the accuracy of LSTM prediction. The core concept of the Attention mechanism is to give different importance to each element in the sequence by calculating the attention weight, thereby improving performance and generalization ability.

[0064] Language LSTM is a mechanism for generating language descriptions. In the task of generating language descriptions based on video content, this mechanism enables the model to construct accurate sentence descriptions word by word based on video features and previously generated language information, thereby improving the accuracy and coherence of language generation. The core idea of ​​Language LSTM is to combine the hidden state and regional feature attention features of Attention LSTM, use structures such as forget gates, input gates, and output gates to filter and update information, and assign different weights to different input information, so as to better capture and model long-term dependencies in time series data, extract temporal feature information in multi-dimensional time series data, and generate high-quality language descriptions that are closely related to video content.

[0065] In another aspect of the embodiments of the present application, a computer storage medium is also provided accordingly.

[0066] The embodiment of the present application also provides a computer-readable storage medium, the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above method. The specific implementation process will not be repeated here.

[0067] The present application also provides an electronic device in an embodiment. The electronic device in this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned method in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program is executed by the processor, the functions of each model / unit in the device in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0068] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, a server, and a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the electronic device may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0069] The processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0070] The memory may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the electronic device. Furthermore, the memory may include both an internal storage unit of the electronic device and an external storage device. The memory is used to store computer programs and other programs and data required by the electronic device. The memory may also be used to temporarily store data that has been output or is to be output.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0072] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0073] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the above-mentioned method of each embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., which can store program codes.

[0074] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video content description method, characterized in that: The method comprises: S1, obtain video stream; S2, extracting a preset number of key frames from the video stream based on a motion feature extraction algorithm and a clustering algorithm for adjacent frames of the video stream; S3, using a target detection algorithm for regional feature extraction to extract regional features from a preset number of key frames; S4, calculating the corresponding regional attention features based on the self-attention mechanism algorithm according to the regional features; S5, extracting video frame features of each frame using a classification algorithm and a video stream adjacent frame motion feature extraction algorithm according to the video stream; S6, decoding is performed using a recurrent network algorithm according to the regional attention features and the video frame features to obtain a description text of the video content.

2. The video content description method according to claim 1, characterized in that: In S2, based on the motion information of objects in adjacent frames calculated by the motion feature extraction algorithm of adjacent frames of the video stream, the clustering algorithm is improved to achieve clustering of the video stream.

3. The video content description method according to claim 2, characterized in that: The video stream adjacent frame motion feature extraction algorithm adopts the Optical Flow algorithm to find the corresponding relationship between the previous frame and the current frame based on the change of pixels in the image sequence in the time domain and the correlation between adjacent frames, thereby calculating the motion information of objects between adjacent frames.

4. The video content description method according to claim 2, characterized in that: The calculation process of the clustering algorithm includes: Randomly select a preset number of frames as cluster centers; Start iterative calculation, calculate the distance between each frame in the video stream and the current key frame, and classify each frame into the cluster where the key frame with the smallest distance is located; After one iteration, the average value of all frames in the cluster is calculated as the new cluster center; Execute the next iteration calculation until the preset termination condition is met.

5. The video content description method according to claim 1, characterized in that: In S3, Faster RCNN is used to extract regional features from a preset number of key frames.

6. The video content description method according to claim 1, characterized in that: In S5, the ResNet network is used for classification; the motion feature extraction algorithm for adjacent frames of the video stream is the Optical Flow algorithm.

7. The video content description method according to claim 6, characterized in that: In S6, the recurrent network algorithm includes Attention LSTM and Language LSTM.

8. The video content description method according to claim 7, characterized in that: Attention LSTM, Video Frame The features are input, including the global features of the video, the position encoding information of the video clips, and the word orientation of the input text. The quantity is decoded and the hidden state is obtained by decoding; Language LSTM decodes the hidden state and regional attention features to obtain the description text.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the video content description method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores a computer program, and the processor runs the computer program to control the electronic device to execute the video content description method according to any one of claims 1 to 8.

Images