Short video labeling method based on video question answering

Abstract

The application discloses a short video labeling method based on video question answering, and belongs to the technical field of visual question answering. First, short video materials are collected from a short video platform according to different categories, and high-dimensional feature representations of visual objects and scene texts are extracted for each video frame; the interaction between two modes is simulated by using a multi-modal transformer; taking the objects or texts as answers, the corresponding questions are predicted through iterative decoding with a self-recurrence mechanism; then, for a short video to be labeled, the questions and answers are combined with the video frames and the audio respectively, wherein a group of video frames is first processed by RCNN, and the two groups are then recombined into a BERT network and a full connection layer for prediction; the predictions of the two groups are summed, and the summed vector is converted into an answer score through normalization, and the answer with the highest score is output. Finally, all the answers are taken as texts, a text classification is performed by using an RNN network, and the generated label is taken as the result of short video labeling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual question answering, and particularly relates to a short video labeling method based on video question answering. BACKGROUND

[0002] The visual question answering task is a challenging and practical task in the field of artificial intelligence, and is a multi-modal task, which is a cross-research direction mainly involving computer vision and natural language processing. Video question answering is a task type in visual question answering, and is a multi-modal reasoning task for processing video and text problems. Video question answering refers to: given a video and a question related to the video, the goal of video question answering is to combine the visual information of the video and the content of the question text, and obtain the answer to the question by performing feature fusion and reasoning on the video and the text.

[0003] Text visual question answering is a task type in visual question answering, and is a multi-modal reasoning task for processing image and text problems. Text visual question answering refers to: given an image and a question related to the image, the goal is to jointly reason to obtain an answer by recognizing the scene text and object in the image. Unlike traditional visual question answering tasks, text visual question answering pays more attention to the text information of the image, and the answer can come from a candidate answer set or from the recognized text information in the image.

[0004] For the text multi-classification task, in recent years, there are two methods of traditional machine learning and deep learning model to solve. The traditional machine learning model is dominated by statistical models, such as Naive Bayes classification (NB), K-Nearest Neighbor (KNN), Support Vector Machine (SVM), and tree model structure XGBoost and LightGBM, etc. The disadvantage is that these methods still need to be functionally designed, which is both time-consuming and expensive; in addition, they usually ignore the natural order structure or context information in the text data, which makes it difficult to learn the semantic information of words. The deep learning model is dominated by Fasttext model, TextCNN model and Bert model, among which TextCNN is more suitable for short text multi-classification tasks. However, CNN has a problem of fixed filter view, on the one hand, it cannot model longer sequence information, on the other hand, the hyperparameter adjustment of the filter is also very cumbersome. SUMMARY

[0005] Therefore, the present application aims to provide a short video annotation method based on video question answering. In order to obtain a general question set for short video annotation, the method uses a multi-modal transformer and an iterative decoder with an autoregressive mechanism to predict questions. In order to obtain an answer set for the short video to be annotated, the method uses a video frame and audio split processing method, and then uses a BERT network to generate an answer set. Finally, the method uses an RNN network to perform multi-classification on the answer set to obtain an annotation result.

[0006] To achieve the above object, the present application adopts the following technical solutions:

[0007] A short video annotation method based on video question answering, comprising the following steps:

[0008] (1) Collect short video materials from short video platforms by different categories, extract high-dimensional feature representations of visual objects and scene text for each video frame, and use a multi-modal transformer to extract feature vector sequences of different modalities, including the following sub-steps:

[0009] (1.1) Collect short video materials from short video platforms by category. The categories can be divided into drama, comedy, talent and life skills according to the content. For each video frame, use OCR to recognize scene text, and use a pre-trained 2D object detector and R-CNN to locate visual objects;

[0010] (1.2) Take one word or object from the scene text or visual object as an answer, and embed the answer word, detected visual object and all detected OCR labels into a common d-dimensional embedding space;

[0011] (1.3) Input the extracted high-dimensional feature representation into a multi-modal transformer, and the input sequence is F={F ans ,F obj ,F ocr}, where F ans , F obj , F ocr represent the feature embedding representation of the answer word, visual object and OCR label respectively;

[0012] (1.4) Model the interaction between the feature embeddings of different modalities through a multi-head attention mechanism, and then extract the d-dimensional feature vector sequence of each modality from the output of the multi-modal transformer.

[0013] (2) Take the object or text as the answer, and predict the corresponding question through iterative decoding with an autoregressive mechanism, including the following sub-steps:

[0014] (2.1) input the sequence of feature vectors into a multi-step decoding module, which outputs the predicted question word by word iteratively according to the input answer;

[0015] (2.2) in each iterative decoding step, the previously predicted word is embedded, and then the next output word is selected from a fixed word frequency vocabulary table or from the extracted OCR labels;

[0016] (2.3) when the maximum number of steps is reached, the decoding process ends, and the complete question is output; each question and answer pair is combined to form a set of question and answer pairs.

[0017] (3) for a short video to be labeled, the question and the answer are combined with the video frames and the audio, including the following sub-steps:

[0018] (3.1) for a short video to be labeled, its video frame content and audio content are processed separately as two streams. Visual objects and scene text are extracted for each video frame as a video stream; language semantic information is extracted from the audio as a semantic stream; the two streams are combined with the question and answer pair set generated in step 2 and processed separately in subsequent steps.

[0019] (4) the video frames are first processed by R-CNN, and then recombined to form a string sequence, which is input into the BERT network and the fully connected layer for

[0020] prediction, including the following sub-steps:

[0021] (4.1) the video frames are first processed by R-CNN to extract visual concept features (including objects and attributes), and the features are represented by corresponding words or noun phrases, and de-duplicated to obtain individual visual concept features from the entire scene;

[0022] (4.2) the question, the unique visual concept features, and each candidate answer are connected and rearranged into a string, and each rearranged string is tokenized to obtain a sequence;

[0023] (4.3) the sequence is input into the BERT network and a fully connected layer to obtain the answer prediction.

[0024] (5) the audio is directly recombined to form a string sequence, which is input into the BERT network and the fully connected layer for prediction, including the following sub-steps:

[0025] (5.1) the question, the semantic features of the audio, and each candidate answer are connected and rearranged into a string, and each rearranged string is tokenized to obtain a sequence;

[0026] (5.2) the sequence is input into the BERT network and a fully connected layer to obtain the answer prediction.

[0027] (6) sum the predictions of the two groups, and convert the summed vector into an answer score using normalization, output the answer with the highest score, including the following sub-steps:

[0028] (6.1) sum the predictions of the video stream and the semantic stream of each candidate answer, and convert the summed vector into an answer score using softmax, select the answer with the highest score as the final predicted answer.

[0029] (7) use the RNN network to perform text classification with all answers as text, and the generated label is the result of short video labeling

[0030] , including the following sub-steps:

[0031] (7.1) mark each word in the answer set using one-hot encoding method, input the word one-hot vector into embedding, and the output vector of embedding is the new embedding representation of the word;

[0032] (7.2) input the vector into a bidirectional LSTM layer, splice the two hidden states of the layer at each time step to obtain an input of the upper unidirectional LSTM at each time step, and finally take the hidden state of the last time step of the upper unidirectional LSTM;

[0033] (7.3) input the hidden state taken into a fully connected layer, and then through a softmax layer to obtain a classification result, which is the result of short video labeling.

[0034] Compared with the prior art, the technical scheme of the present application has the following beneficial effects:

[0035] The present application proposes a short video labeling method based on video question answering, which improves the accuracy of short video labeling by processing the visual and auditory information of the video.

[0036] The problem set acquisition of the present application only needs to be performed once, and the whole process can be automatically performed, which can be used for pre-auditing of the short video platform background, and reduces the workload of manual auditing. DETAILED DESCRIPTION

[0037] Figure 1 The flowchart of the present application;

[0038] Figure 2 The flowchart of the present application; CONCRETE EMBODIMENT

[0039] The present application will be further described below in combination with specific embodiments and drawings.

[0040] As shown in the flowchart Figure 1As shown, the short video labeling method based on video question answering of the present application includes the following steps:

[0041] (1) Collect short video materials from short video platforms by different categories, extract high-dimensional feature representation of visual objects and scene text for each video frame, and use a multi-modal transformer to extract feature vector sequences of different modalities, including the following sub-steps:

[0042] (1.1) Collect short video materials from short video platforms such as Douyin and Kuaishou according to categories. The categories can be divided into drama, comedy, talent and life skills according to the content. For each video frame, scene text is recognized using OCR, and visual objects are located using a pre-trained 2D object detector and R-CNN;

[0043] (1.2) Take one word or object from the scene text or visual object as the answer, and project the answer word, detected visual object and all detected OCR labels into a common d-dimensional embedding space as high-dimensional feature embeddings;

[0044] (1.3) Input the extracted high-dimensional feature representation into a multi-modal transformer, and the input sequence is F = {F ans ,F obj ,F ocr}, where F ans , F obj , F ocr represent the feature embedding representation of the answer word, visual object and OCR label respectively;

[0045] (1.4) Model the interaction between the feature embeddings of different modalities through a multi-head attention mechanism, and then extract the d-dimensional feature vector sequence of each modality from the output of the multi-modal transformer.

[0046] (2) Take the object or text as the answer, and predict the question corresponding to the answer through iterative decoding with an autoregressive mechanism, including the following sub-steps:

[0047] (2.1) Input the feature vector sequence into a multi-step decoding module, which iteratively outputs the predicted question word by word according to the input answer;

[0048] (2.2) In each iterative decoding step, embed the previously predicted word, and then select the next output word from a fixed word frequency vocabulary table or from the extracted OCR labels;

[0049] (2.3) When the maximum number of steps is reached, the decoding process ends and the complete question is output; each question and answer pair is combined to form a question and answer pair set.

[0050] (3) For the short video to be labeled, the question and the answer are combined with the video frame and the audio, respectively, including the following sub-steps:

[0051] (3.1) For the short video to be labeled, its video frame content and audio content are processed separately as two streams. Visual objects and scene text are extracted for each video frame as a video stream; language semantic information is extracted from the audio as a semantic stream; the two streams are combined with the question and answer pair set generated in step 2, and are processed separately in subsequent steps.

[0052] (4) The video frames are first processed by R-CNN, and then recombined to form a string sequence, which is input into the BERT network and the full connection layer for prediction, including the following sub-steps:

[0053] (4.1) The video frames are first processed by R-CNN to extract visual concept features (including objects and attributes), and the features are represented by corresponding words or noun phrases, and are de-duplicated to obtain individual visual concept features from the entire scene;

[0054] (4.2) The question, unique visual concept features, and each candidate answer are connected and rearranged into a string, and each rearranged string is tokenized to obtain a sequence;

[0055] (4.3) The sequence is input into the BERT network and a full connection layer to obtain the answer prediction.

[0056] (5) The audio is directly recombined to form a string sequence, which is input into the BERT network and the full connection layer for prediction, including the following sub-steps:

[0057] (5.1) The question, semantic features of the audio, and each candidate answer are connected and rearranged into a string, and each rearranged string is tokenized to obtain a sequence;

[0058] (5.2) The sequence is input into the BERT network and a full connection layer to obtain the answer prediction.

[0059] (6) The predictions of the two groups are summed, and the summed vector is converted into an answer score using normalization, and the answer with the highest score is output, including the following sub-steps:

[0060] (6.1) The predictions of the video stream and the semantic stream for each candidate answer are summed, and the summed vector is converted into an answer score using softmax, and the answer with the highest score is selected as the final prediction answer.

[0061] (7) All answers are used as text, and an RNN network is used for text classification to generate labels, which are used as the result of short video labeling, including the following sub-steps:

[0062] (7.1) Mark each word in the answer set with one-hot encoding method, input the word one-hot vector into embedding, and the output vector of embedding is the new embedding representation of the word;

[0063] (7.2) Input the vector into a bidirectional LSTM layer, splice the two hidden states of the layer at each time step to obtain an input of an upper one-way LSTM at each time step, and finally take the hidden state of the last time step of the upper one-way LSTM;

[0064] (7.3) Input the taken hidden state into a fully connected layer, and then pass it through a softmax layer to obtain a classification result, which is the result of short video labeling.

[0065] Embodiment

[0066] Reference Figure 2 The embodiment extracts high-dimensional feature representation of visual objects and scene text for each video frame of the video material; uses a multimodal transformer to simulate the interaction between the two modalities; takes the object or text as the answer, and predicts the corresponding question through iterative decoding with an autoregressive mechanism. For the short video to be labeled, the question and the answer are respectively combined with the video frame and the audio to arrange into a string sequence, and then re-arranged into a BERT network and a fully connected layer for prediction; the predictions of the two groups are summed, and the summed vector is converted into an answer score using normalization, and the highest score is output. Finally, all answers are used as text, and an RNN network is used for text classification to generate a label, which is the result of short video labeling.

[0067] The above only describes the preferred embodiments of the present application, and certain modifications can be made within the scope defined by the claims of the present application, but all will fall within the protection scope of the present application.

Claims

1. A short video annotation method based on video question and answer, characterized in that the method comprises the following specific steps: Step 1: Collect short video materials from short video platforms according to different categories, extract high-dimensional feature representations of visual objects and scene text for each video frame, and use a multi-modal transformer to extract feature vector sequences of different modalities; Step 2: Take the visual object or scene text as the answer, and predict the corresponding question through iterative decoding with an autoregressive mechanism; Step 3: For the short video to be annotated, combine the question and the answer with the video frame and the audio, respectively; Step 4: The video frame is first processed by R-CNN, then reorganized into a string sequence, and input into the BERT network and the full connection layer for prediction; Step 5: The audio is directly reorganized into a string sequence, and input into the BERT network and the full connection layer for prediction; Step 6: Sum the predictions of the two groups, and use normalization to convert the summed vector into an answer score, and output the answer with the highest score; Step 7: Take all the answers as text, use an RNN network for text classification, and the generated label is the result of short video annotation; Wherein, the step 1 is specifically: 1.1: Collect short video materials from short video platforms by category, which are divided into drama, comedy, talent and life skills according to content; for each video frame, scene text is recognized by OCR, and visual objects are located by pre-trained 2D object detector and R-CNN; 1.2: Take a word or object from scene text or visual object as an answer, and project the answer word, detected visual object and all detected OCR labels into a common d-dimensional embedding space as high-dimensional feature embedding; 1.3: input the extracted high-dimensional feature representation into a multi-modal transformer, the input sequence is F = {F ans , F obj , F ocr}, where F ans , F obj , F ocr represent the feature embedding representation of the answer word, visual object, and OCR label respectively; 1.4: Model the interaction between feature embeddings of different modalities through multi-head attention mechanism, and then extract the d-dimensional feature vector sequence of each modality from the output of the multi-modal transformer; The step 2 is specifically: 2.1: Input the feature vector sequence into the multi-step decoding module, which iteratively outputs the predicted question word by word according to the input answer; 2.2: In each iterative decoding step, embed the previously predicted word, and then select the next output word from the fixed word frequency vocabulary or from the extracted OCR labels; 2.3: When the maximum number of steps is reached, the decoding process ends and the complete question is output; each question and answer pair is combined to form a set of question and answer pairs; The step 3 is specifically: 3.1: For the short video to be annotated, separate its video frame content and audio content as two streams; extract visual objects and scene text for each video frame as a video stream; extract language semantic information from the audio as a semantic stream; combine the video stream and the semantic stream with the question set and the corresponding candidate answers generated in step 2, and process them separately in the subsequent steps; The step 4 is specifically: 4.1: The video frame is first processed by R-CNN to extract visual concept features including objects and attributes, and the features are represented by corresponding words or noun phrases, and the unique visual concept features are obtained from the entire scene; 4.2: Connect the question, unique visual concept features and each candidate answer and rearrange them into a string, and each rearranged string is tokenized to obtain a sequence; 4.3: Input the sequence into the BERT network and a fully connected layer to obtain the answer prediction; The step 5 is specifically: 5.1: Connect the question, audio semantic features and each candidate answer and rearrange them into a string, and each rearranged string is tokenized to obtain a sequence; 5.2: Input the sequence into the BERT network and a fully connected layer to obtain the answer prediction; The step 6 is specifically: 6.1: Sum the predictions of the video stream and the semantic stream for each candidate answer, and use softmax to convert the summed vector into an answer score, and select the answer with the highest score as the final predicted answer; The step 7 is specifically: 7.1: Use one-hot encoding method to mark each word in the answer set, and input the word one-hot vector into the embedding, and the output vector of the embedding is the new embedding representation of the word; 7.2: input the vector to a bidirectional LSTM layer, concatenate the two hidden states of the layer at each time step as an input of the upper unidirectional LSTM at each time step, and finally take the hidden state of the upper unidirectional LSTM at the last time step; 7.3: input the taken hidden state to a fully connected layer, and then pass the hidden state through a softmax layer to obtain a classification result, which is taken as the result of short video labeling.

Images