Short video annotation method based on video questions and answers
Through the short video annotation method based on video Q&A, combined with multimodal transformer and deep learning model, the semantic features of video content are extracted, and the problem of insufficient segmentation accuracy in the existing technology is solved, and the precise segmentation and high-quality management of film and television content is realized. .
Patent Information
- Application Number
- CN202411038142.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing film and television segment segmentation methods mainly rely on physical features and ignore the semantic information of the video content, resulting in insufficient segmentation accuracy and difficult to meet users' needs for high-quality film and television content management and recommendation.
Using a short video labeling method based on video Q&A, collecting materials from multiple short video platforms, extracting high-dimensional feature representations of visual objects and scene text, using a multimodal transformer model to extract feature vector sequences of different modes, combined with pre-training The deep learning model recognizes the changes in objects and scenes in video, automatically generates Q&A pairs and generates video tags, and stores them in the database.
It realizes accurate segmentation of film and television content, improves the accuracy and intelligence level of segmentation, and can effectively utilize multimodal information to improve the understanding and analysis ability of video content.
Smart Images

Figure CN118968383A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video question answering, and in particular to a short video annotation method based on video question answering. Background Art
[0002] Visual question answering is a challenging and practical task in the field of artificial intelligence. It is a multimodal task, mainly involving an interdisciplinary research direction of computer vision and natural language processing. Video question answering is a type of task in visual question answering, which is a multimodal reasoning task for processing video and text questions. Video question answering means: given a video and questions related to the video, the goal of video question answering is to combine the visual information of the video and the text content of the question, and obtain the answer to the question by fusing the features of the video and the text and reasoning.
[0003] Text Visual Question Answering is a type of task in visual question answering, and is a multimodal reasoning task for processing image and text questions. Text Visual Question Answering means that given an image and a question related to the image, the goal is to perform joint reasoning to obtain the answer by identifying the scene text and objects in the image. Unlike traditional visual question answering tasks, Text Visual Question Answering focuses more on the text information of the image, and the answer can come from a candidate answer set or from the text information recognized in the image;
[0004] Existing methods for segmenting video clips mainly rely on physical features (such as frame switching, lens changes, etc.) for segmentation. However, this method ignores the semantic information of the video content, resulting in insufficient segmentation accuracy and difficulty in meeting users' needs for high-quality video content management and recommendation. Therefore, there is an urgent need for a method that can understand the semantics of video content and automatically segment clips to improve the accuracy and intelligence of segmentation. Summary of the invention
[0005] The purpose of the present invention is to provide a short video annotation method based on video question and answer to solve the problems raised in the above background technology.
[0006] To achieve the above object, the present invention provides the following technical solution: a short video annotation method based on video question and answer, comprising the following steps:
[0007] S1. Collect short video materials from multiple short video platforms according to different video types, and set the frame frequency of each type of video according to the duration of each video;
[0008] S2, extract high-dimensional feature representations of visual objects and scene text for each video frame, and use multimodal transformers to extract feature vector sequences of different modalities;
[0009] S3, using the pre-trained deep learning model to identify objects in the video frame and combine with the scene detection model to further identify scene changes in the video;
[0010] S4, after extracting the object and scene information, produce the corresponding question-answer pair template, and automatically generate the question-answer pair through natural language generation technology;
[0011] S5. Generate video tags based on the question and answer pairs, associate the tags with the videos, and store them in a database.
[0012] Specifically, the short video material types in S1 include action, static, interview, landscape and security monitoring videos. The duration of the above different types of videos is divided into multiple time periods, and the basic frame extraction frequency is set according to the video type. At the same time, the frame extraction frequency is dynamically adjusted according to the video duration, and then the frame extraction interval of each time period is calculated.
[0013] Specifically, the basic frame extraction frequency setting sets the frame extraction frequency according to the video type:
[0014]
[0015] where f t is the basic frame extraction frequency corresponding to the video type, the number of frames extracted per second, the time field adjustment coefficient setting, and the total video duration T to adjust the frame extraction frequency:
[0016]
[0017] Among them, T max is a maximum duration reference value, the reference value is 3600 seconds, where f d Duration adjustment coefficient: As the video duration increases, the frequency is adjusted.
[0018] Specifically, the dynamic frame extraction frequency calculation combines the basic frame extraction frequency and the duration adjustment coefficient to calculate the dynamic frame extraction frequency:
[0019] f dynamic =f t ×f d
[0020] The frame extraction interval calculation is based on the dynamic frame extraction frequency to calculate the frame extraction interval:
[0021]
[0022] Among them, N is the total length of the video.
[0023] Specifically, in S2: for the extracted frames, the FasterR-CNN object detection model is used to decode each video, and a frame sequence is extracted according to the set frame extraction frequency, the video frame is input into the model, and the detection results including the detected object bounding box and category label are obtained, the frame object detection cooperates with the convolutional network neural network to extract the high-level feature representation of each detected object, and the backbone network of the FasterR-CNN model is used to directly extract the feature representation;
[0024] The OCR technology is used to detect and recognize scene text in each video frame, and the pre-trained model is used to encode the recognized text into a high-level feature representation. Specifically, the video file is decoded and the frame sequence is extracted according to the set frame extraction frequency. The pre-trained OCR model is used to input the video frame into the OCR model to obtain the detected text area and recognized text content. The pre-trained text embedding model is used to encode the recognized text into a high-dimensional feature representation, the recognized text is segmented, and it is encoded into a high-dimensional feature representation through the BERT model.
[0025] Specifically, in the multimodal feature fusion, the visual object features and the scene text features are combined into a multimodal feature vector, which is then serialized and used as input data for the multimodal transformer model. Secondly, the input multimodal feature vector sequence is positionally encoded through the multimodal transformer processing, and the input multimodal feature vector sequence is processed using the multimodal transformer to extract the correlation features between different modalities.
[0026] Specifically, the visual object features include: extracting visual object features from each video frame using the FasterR-CNN model, the specific steps are as follows:
[0027] Input preprocessing: Video frames are input into the FasterR-CNN model for preprocessing, including normalization and resizing;
[0028] Object detection: Use the FasterR-CNN model to detect objects in video frames and obtain the bounding box and category information of the objects;
[0029] Feature extraction: Extract feature vectors from detected object regions, usually using a ResNet deep convolutional neural network to extract high-dimensional features;
[0030] The scene text feature extraction includes extracting scene text features from each video frame using OCR (optical character recognition) technology, and the specific steps are as follows:
[0031] Text detection: Use text detection algorithms (such as EAST text detector) to detect text regions in video frames;
[0032] Text recognition: Use OCR technology to identify the detected text area and obtain the text content;
[0033] Feature representation: Encode the recognized text to generate a high-dimensional feature representation of the text, and use a pre-trained word embedding model or language model to convert the text into a feature vector;
[0034] Then perform feature fusion:
[0035] The visual object features and scene text features are fused to generate a multimodal feature vector. The specific steps are as follows:
[0036] Feature alignment: Align visual object features and scene text features, using position alignment and time alignment strategies to ensure that features correspond at the same time point or spatial area;
[0037] Feature concatenation: Concatenate the aligned visual object features and scene text features to form a unified feature vector, using vector concatenation, weighted summation, and other feature fusion methods:
[0038] F multi =[F visual ,F text ]
[0039] Among them, F multi is the multimodal feature vector, F visual is the visual object feature vector, F text is the scene text feature vector;
[0040] Specifically, the feature serialization is as follows: the multimodal feature vector is serialized to form a multimodal feature vector sequence, which is used as input to the multimodal transformer model for processing. The specific steps are as follows:
[0041] Feature vector serialization: Arrange the multimodal feature vectors of each video frame in chronological order to form a feature vector sequence:
[0042] S={F multi,1 ,F multi,2 ,…,F multi,T}
[0043] Among them, S is the multimodal feature vector sequence, F multi,1 is the multimodal feature vector of the tt-th frame, T is the total number of video frames, and then is input into the multimodal transformer model: the feature vector sequence is taken as input and sent into the multimodal transformer model for processing to generate feature vector sequences of different modalities.
[0044] Specifically, it is input into the multimodal transformer model for processing, following the steps below, which is mainly divided into two parts: multimodal feature vector serialization and processing the input multimodal feature vector sequence through the multimodal transformer;
[0045] The Multimodal Transformer handles: Positional encoding:
[0046] Before inputting the feature vector sequence into the transformer model, it needs to be positionally encoded so that the model can utilize the sequential information;
[0047] The position encoding formula is as follows:
[0048]
[0049] Where pos is the position, i is the dimension index, and d is the dimension of the feature vector
[0050] Add the positional encoding to the sequence of multimodal feature vectors:
[0051] F input,t =F multi,t +PE t
[0052] Multi-mode transformer processing:
[0053] Use the multimodal transformer model to process the input multimodal feature vector sequence and extract the correlation features between different modes;
[0054] The specific steps are as follows:
[0055] Multi-head self-attention mechanism: calculates the attention weights of the feature vector at each position in the input sequence and the feature vectors at other positions;
[0056]
[0057] Where Q is the query vector, K is the key vector, V is the value vector, and d k is the dimension of the feature vector;
[0058] Feedforward neural network: The feature vector of each position is transformed nonlinearly through two layers of feedforward neural network;
[0059] FFN(x)=max(0,xW1+b1)W2+b2
[0060] Layer normalization and residual connection: After multi-head self-attention and feed-forward neural network, layer normalization and residual connection are performed;
[0061] F output,t =LayerNorm(x+FFN(Attention(Q,K,V))).
[0062] Specifically, the video frames are extracted according to the video duration and frame rate. The total number of video frames is set to N, and the frame rate is f. Then the video frame of the i-th frame can be expressed as:
[0063] Frame i =V(i),i=1,2,…,N;
[0064] Using the pre-trained object detection model, for each video frame, the model output includes the object category, bounding box and confidence. Let the set of object categories detected in the i-th frame be C i , the bounding box set is B i :
[0065]
[0066] The pre-trained scene detection model (such as Places365) is used to perform scene detection on each video frame. Each video frame will be classified into a scene category. Let the scene category of the i-th frame be S i , the scene probability vector is P i , then:
[0067] S i = argmaxP i
[0068] P i ={p i1 ,p i2 ,…,p iM}
[0069] Combine the results of object detection and scene detection to identify scene changes and object changes in the video, and analyze the change trend in the video by counting the objects and scenes detected in each frame;
[0070] Count the number of occurrences of each object category in all frames:
[0071]
[0072] Among them, δ is the indicator function, when c = c ij When δ(c,c ij )=1, otherwise δ(c,c ij )=0;
[0073] By combining the results of object detection and scene detection, we can get a comprehensive analysis result for each video frame; we can calculate the feature vector of each video frame, integrating object and scene information:
[0074] F i =[C i ,Si ]
[0075] Finally, the feature vector sequence of the video frame can be expressed as:
[0076] F={F1,F2,…,F N}.
[0077] Compared with the prior art, the present invention has the following beneficial effects:
[0078] 1. The present invention provides a method for automatic segmentation of film and television segments based on semantic analysis of film and television content. Through the steps of multimodal data collection, preprocessing, semantic analysis, multimodal fusion and segment segmentation, the accurate segmentation of film and television content is achieved. The system can accurately identify semantic boundaries in complex film and television content and has broad application prospects and market value;
[0079] 2. The automatic segmentation method of film and television segments based on semantic analysis of film and television content of the present invention has made improvements in feature standardization, feature fusion and time synchronization, which significantly improves the segmentation accuracy and intelligence level of the system and has a wider application prospect and market value;
[0080] 3. The present invention combines visual object features and scene text features into a multimodal feature vector, and uses a multimodal transformer model to extract correlation features between different modalities. This can effectively utilize multimodal information and improve the ability to understand and analyze video content. BRIEF DESCRIPTION OF THE DRAWINGS
[0081] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0082] Figure 1 It is the overall structural view of the present invention;
[0083] Figure 2 The process of extracting visual object features using the FasterR-CNN model of the present invention;
[0084] Figure 3 A flow chart of the steps of extracting scene text features using the OCR technology of the present invention;
[0085] Figure 4 It is a flow chart of the feature fusion steps of the present invention. DETAILED DESCRIPTION
[0086] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0087] See also Figures 1 to 4 , the present invention provides a technical solution:
[0088] A short video annotation method based on video question answering includes the following steps:
[0089] S1. Collect short video materials from multiple short video platforms according to different video types, and set the frame frequency of each type of video according to the duration of each video;
[0090] Short video material types include action, static, interview, landscape and security monitoring videos. The duration of the above different types of videos is divided into multiple time periods, and the basic frame extraction frequency is set according to the video type. The frame extraction frequency is dynamically adjusted according to the video length, and then the frame extraction interval of each time period is calculated.
[0091] The basic frame extraction frequency setting sets the frame extraction frequency according to the video type:
[0092]
[0093] where f t is the basic frame extraction frequency corresponding to the video type, the number of frames extracted per second, the time field adjustment coefficient setting, and the total video duration T to adjust the frame extraction frequency:
[0094]
[0095] Among them, T max is a maximum duration reference value, the reference value is 3600 seconds, where f d Duration adjustment coefficient: As the video duration increases, the frequency is adjusted.
[0096] The dynamic frame extraction frequency calculation combines the basic frame extraction frequency and the duration adjustment coefficient to calculate the dynamic frame extraction frequency:
[0097] f dynamic =f t ×f d
[0098] The frame extraction interval calculation is based on the dynamic frame extraction frequency to calculate the frame extraction interval:
[0099]
[0100] Among them, N is the total length of the video.
[0101] S2, extract high-dimensional feature representations of visual objects and scene text for each video frame, and use multimodal transformers to extract feature vector sequences of different modalities;
[0102] For the extracted frames, the FasterR-CNN object detection model is used to decode each video, and the frame sequence is extracted according to the set frame extraction frequency. The video frames are input into the model to obtain the detection results including the detected object bounding box and category label. The frame object detection is combined with the convolutional network neural network to extract the high-level feature representation of each detected object, and the backbone network of the FasterR-CNN model is used to directly extract the feature representation.
[0103] The automatic segmentation method of film and television segments based on semantic analysis of film and television content of the present invention has made improvements in feature standardization, feature fusion and time synchronization, which significantly improves the segmentation accuracy and intelligence level of the system and has broader application prospects and market value.
[0104] The OCR technology is used to detect and recognize scene text in each video frame, and the pre-trained model is used to encode the recognized text into a high-level feature representation. Specifically, the video file is decoded and the frame sequence is extracted according to the set frame extraction frequency. The pre-trained OCR model is used to input the video frame into the OCR model to obtain the detected text area and recognized text content. The pre-trained text embedding model is used to encode the recognized text into a high-dimensional feature representation, the recognized text is segmented, and it is encoded into a high-dimensional feature representation through the BERT model.
[0105] Specifically, in the multimodal feature fusion, the visual object features and the scene text features are combined into a multimodal feature vector, which is then serialized and used as input data for the multimodal transformer model. Secondly, the input multimodal feature vector sequence is positionally encoded through the multimodal transformer processing, and the input multimodal feature vector sequence is processed using the multimodal transformer to extract the correlation features between different modalities.
[0106] The visual object features and scene text features are combined into a multimodal feature vector, and the multimodal transformer model is used to extract the correlation features between different modalities. This can effectively utilize multimodal information and improve the ability to understand and analyze video content.
[0107] Specifically, the visual object features include: extracting visual object features from each video frame using the FasterR-CNN model, the specific steps are as follows:
[0108] Input preprocessing: Video frames are input into the FasterR-CNN model for preprocessing, including normalization and resizing;
[0109] Object detection: Use the FasterR-CNN model to detect objects in video frames and obtain the bounding box and category information of the objects;
[0110] Feature extraction: Extract feature vectors from detected object regions, usually using a ResNet deep convolutional neural network to extract high-dimensional features;
[0111] The scene text feature extraction includes extracting scene text features from each video frame using OCR (optical character recognition) technology, and the specific steps are as follows:
[0112] Text detection: Use text detection algorithms (such as EAST text detector) to detect text regions in video frames;
[0113] Text recognition: Use OCR technology (such as Tesseract or Google Vision API) to identify the detected text area and obtain the text content;
[0114] Feature representation: Encode the recognized text to generate a high-dimensional feature representation of the text, and use a pre-trained word embedding model or language model to convert the text into a feature vector;
[0115] Then perform feature fusion:
[0116] The visual object features and scene text features are fused to generate a multimodal feature vector. The specific steps are as follows:
[0117] Feature alignment: Align visual object features and scene text features, using position alignment and time alignment strategies to ensure that features correspond at the same time point or spatial area;
[0118] Feature concatenation: Concatenate the aligned visual object features and scene text features to form a unified feature vector, using vector concatenation, weighted summation, and other feature fusion methods:
[0119] Through the above steps, the visual object features and scene text features are successfully fused into a multimodal feature vector, serialized, and input into the multimodal transformer model for further feature extraction and analysis. This method can effectively combine visual and text information to improve the understanding and analysis capabilities of video content.
[0120] F multi =[F visual ,F text ]
[0121] Among them, Fmulti is the multimodal feature vector, F visual is the visual object feature vector, F text is the scene text feature vector;
[0122] Specifically, the feature serialization is as follows: the multimodal feature vector is serialized to form a multimodal feature vector sequence, which is used as input to the multimodal transformer model for processing. The specific steps are as follows:
[0123] Feature vector serialization: Arrange the multimodal feature vectors of each video frame in chronological order to form a feature vector sequence:
[0124] S={F multi,1 ,F multi,2 ,…,F multi,T}
[0125] Among them, S is the multimodal feature vector sequence, F multi,1 is the multimodal feature vector of the tt-th frame, T is the total number of video frames, and then is input into the multimodal transformer model: the feature vector sequence is taken as input and sent into the multimodal transformer model for processing to generate feature vector sequences of different modalities.
[0126] Specifically, it is input into the multimodal transformer model for processing, following the steps below, which is mainly divided into two parts: multimodal feature vector serialization and processing the input multimodal feature vector sequence through the multimodal transformer;
[0127] The Multimodal Transformer handles: Positional encoding:
[0128] Before inputting the feature vector sequence into the transformer model, it needs to be positionally encoded so that the model can utilize the sequential information;
[0129] The position encoding formula is as follows:
[0130]
[0131] Where pos is the position, i is the dimension index, and d is the dimension of the feature vector
[0132] Add the positional encoding to the sequence of multimodal feature vectors:
[0133] F input,t =F multi,t +PE t
[0134] Multi-mode transformer processing:
[0135] Use the multimodal transformer model to process the input multimodal feature vector sequence and extract the correlation features between different modes;
[0136] The specific steps are as follows:
[0137] Multi-head self-attention mechanism: calculates the attention weights of the feature vector at each position in the input sequence and the feature vectors at other positions;
[0138]
[0139] Where Q is the query vector, K is the key vector, V is the value vector, and d k is the dimension of the feature vector;
[0140] Feedforward neural network: The feature vector of each position is transformed nonlinearly through two layers of feedforward neural network;
[0141] FFN(x)=max(0,xW1+b1)W2+b2
[0142] Layer normalization and residual connection: After multi-head self-attention and feed-forward neural network, layer normalization and residual connection are performed;
[0143] F output,t =LayerNorm(x+FFN(Attention(Q,K,V))).
[0144] Specifically, the video frames are extracted according to the video duration and frame rate. The total number of video frames is set to N, and the frame rate is f. Then the video frame of the i-th frame can be expressed as:
[0145] Frame i =V(i),i=1,2,…,N;
[0146] Using the pre-trained object detection model, for each video frame, the model output includes the object category, bounding box and confidence. Let the set of object categories detected in the i-th frame be C i , the bounding box set is B i :
[0147]
[0148] The pre-trained scene detection model (such as Places365) is used to perform scene detection on each video frame. Each video frame will be classified into a scene category. Let the scene category of the i-th frame be S i , the scene probability vector is P i , then:
[0149] S i = argmaxP i
[0150] P i ={p i1 ,p i2 ,…,p iM}
[0151] Combine the results of object detection and scene detection to identify scene changes and object changes in the video, and analyze the change trend in the video by counting the objects and scenes detected in each frame;
[0152] Count the number of occurrences of each object category in all frames:
[0153]
[0154] Among them, δ is the indicator function, when c = c ij When δ(c,c ij )=1, otherwise δ(c,c ij )=0;
[0155] By combining the results of object detection and scene detection, we can get a comprehensive analysis result for each video frame; we can calculate the feature vector of each video frame, integrating object and scene information:
[0156] F i =[C i ,S i ]
[0157] Finally, the feature vector sequence of the video frame can be expressed as:
[0158] F={F1,F2,…,F N}.
[0159] S3, using the pre-trained deep learning model to identify objects in the video frame and combine with the scene detection model to further identify scene changes in the video;
[0160] S4, after extracting the object and scene information, produce the corresponding question-answer pair template, and automatically generate the question-answer pair through natural language generation technology;
[0161] S5. Generate video tags based on the question and answer pairs, associate the tags with the videos, and store them in a database.
[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A short video annotation method based on video question answering, characterized in that: The steps include: S1. Collect short video materials from multiple short video platforms according to different video types, and set the frame frequency of each type of video according to the duration of each video; S2, extract high-dimensional feature representations of visual objects and scene text for each video frame, and use multimodal transformers to extract feature vector sequences of different modalities; S3, using the pre-trained deep learning model to identify objects in the video frame and combine with the scene detection model to further identify scene changes in the video; S4, after extracting the object and scene information, produce the corresponding question-answer pair template, and automatically generate the question-answer pair through natural language generation technology; S5. Generate video tags based on the question and answer pairs, associate the tags with the videos, and store them in a database.
2. A short video annotation method based on video question and answer according to claim 1, characterized in that: The types of short video materials in S1 include action, static, interview, landscape and security monitoring videos. The duration of the above different types of videos is divided into multiple time periods, and the basic frame extraction frequency is set according to the video type. The frame extraction frequency is dynamically adjusted according to the video duration, and then the frame extraction interval of each time period is calculated.
3. A short video annotation method based on video question and answer according to claim 2, characterized in that: The basic frame extraction frequency setting sets the frame extraction frequency according to the video type: where f t is the basic frame extraction frequency corresponding to the video type, the number of frames extracted per second, the time field adjustment coefficient setting, and the total video duration T to adjust the frame extraction frequency: Among them, T max is a maximum duration reference value, the reference value is 3600 seconds, where f d Duration adjustment coefficient: As the video duration increases, the frequency is adjusted.
4. A short video annotation method based on video question and answer according to claim 2, characterized in that: The dynamic frame extraction frequency calculation combines the basic frame extraction frequency and the duration adjustment coefficient to calculate the dynamic frame extraction frequency: f dynamic =f t ×f d The frame extraction interval calculation is based on the dynamic frame extraction frequency to calculate the frame extraction interval: Among them, N is the total length of the video.
5. A short video annotation method based on video question and answer according to claim 2, characterized in that: In S2: for the extracted frames, the FasterR-CNN object detection model is used to decode each video, and a frame sequence is extracted according to the set frame extraction frequency, the video frame is input into the model, and the detection results including the detected object bounding box and category label are obtained, the frame object detection cooperates with the convolutional network neural network to extract the high-level feature representation of each detected object, and the backbone network of the FasterR-CNN model is used to directly extract the feature representation; Use OCR technology to detect and recognize scene text in each video frame, and use a pre-trained model to encode the recognized text into a high-level feature representation; Specifically, it decodes the video file and extracts the frame sequence according to the set frame extraction frequency. Using the pre-trained OCR model, the video frame is input into the OCR model to obtain the detected text area and recognized text content. The pre-trained text embedding model is used to encode the recognized text into a high-dimensional feature representation. The recognized text is segmented and encoded into a high-dimensional feature representation through the BERT model.
6. A short video annotation method based on video question and answer according to claim 5, characterized in that: In the multimodal feature fusion, the visual object features and the scene text features are combined into a multimodal feature vector, and then the multimodal feature vector is serialized and used as input data for the multimodal transformer model. Secondly, the input multimodal feature vector sequence is positionally encoded through multimodal transformer processing, and the input multimodal feature vector sequence is processed using the multimodal transformer to extract the correlation features between different modalities.
7. A short video annotation method based on video question and answer according to claim 6, characterized in that: The visual object features include: extracting visual object features from each video frame using the FasterR-CNN model, the specific steps are as follows: Input preprocessing: Video frames are input into the FasterR-CNN model for preprocessing, including normalization and resizing; Object detection: Use the FasterR-CNN model to detect objects in video frames and obtain the bounding box and category information of the objects; Feature extraction: Extract feature vectors from detected object regions, usually using a ResNet deep convolutional neural network to extract high-dimensional features; The scene text feature extraction includes extracting scene text features from each video frame using OCR (optical character recognition) technology, and the specific steps are as follows: Text detection: Use text detection algorithms (such as EAST text detector) to detect text regions in video frames; Text recognition: Use OCR technology (such as Tesseract or Google Vision API) to identify the detected text area and obtain the text content; Feature representation: Encode the recognized text to generate a high-dimensional feature representation of the text, and use a pre-trained word embedding model or language model to convert the text into a feature vector; Then perform feature fusion: The visual object features and scene text features are fused to generate a multimodal feature vector. The specific steps are as follows: Feature alignment: Align visual object features and scene text features, using position alignment and time alignment strategies to ensure that features correspond at the same time point or spatial area; Feature concatenation: Concatenate the aligned visual object features and scene text features to form a unified feature vector, using vector concatenation, weighted summation, and other feature fusion methods: F multi =[F visual ,F text ] Among them, F multi is the multimodal feature vector, F visual is the visual object feature vector, F text is the scene text feature vector.
8. A short video annotation method based on video question and answer according to claim 7, characterized in that: The feature serialization: serialize the multimodal feature vector to form a multimodal feature vector sequence, which is used as input to the multimodal transformer model for processing. The specific steps are as follows: Feature vector serialization: Arrange the multimodal feature vectors of each video frame in chronological order to form a feature vector sequence: S={F multi,1 ,F multi,2 ,…,F multi,T } Among them, S is the multimodal feature vector sequence, F multi,1 is the multimodal feature vector of the tt-th frame, T is the total number of video frames, and then is input into the multimodal transformer model: the feature vector sequence is taken as input and sent into the multimodal transformer model for processing to generate feature vector sequences of different modalities.
9. A short video annotation method based on video question and answer according to claim 8, characterized in that: It is input into the multimodal transformer model for processing, and the following steps are followed, which are mainly divided into two parts: multimodal feature vector serialization and processing the input multimodal feature vector sequence through the multimodal transformer; The Multimodal Transformer handles: Positional encoding: Before inputting the feature vector sequence into the transformer model, it needs to be positionally encoded so that the model can utilize the sequential information; The position encoding formula is as follows: Where pos is the position, i is the dimension index, and d is the dimension of the feature vector Add the positional encoding to the sequence of multimodal feature vectors: F input,t =F multi,t +PE t Multi-mode transformer processing: Use the multimodal transformer model to process the input multimodal feature vector sequence and extract the correlation features between different modes; The specific steps are as follows: Multi-head self-attention mechanism: calculates the attention weights of the feature vector at each position in the input sequence and the feature vectors at other positions; Where Q is the query vector, K is the key vector, V is the value vector, and d k is the dimension of the feature vector; Feedforward neural network: The feature vector of each position is transformed nonlinearly through two layers of feedforward neural network; FFN(x)=max(0,xW1+b1)W2+b2 Layer normalization and residual connection: After multi-head self-attention and feed-forward neural network, layer normalization and residual connection are performed; F output,t =LayerNorm(x+FFN(Attention(Q,K,V)))。 10. A short video annotation method based on video question and answer according to claim 9, characterized in that: The video frames are extracted according to the video duration and frame rate. The total number of video frames is set to N and the frame rate is f. The video frame of the i-th frame can be expressed as: Frame i =V(i),i=1,2,…,N; Using the pre-trained object detection model, for each video frame, the model output includes the object category, bounding box and confidence. Let the set of object categories detected in the i-th frame be C i , the bounding box set is B i : The pre-trained scene detection model (such as Places365) is used to perform scene detection on each video frame. Each video frame will be classified into a scene category. Let the scene category of the i-th frame be S i , the scene probability vector is P i , then: S i =argmaxP i P i ={p i1 ,p i2 ,…,p iM } Combine the results of object detection and scene detection to identify scene changes and object changes in the video, and analyze the change trend in the video by counting the objects and scenes detected in each frame; Count the number of occurrences of each object category in all frames: Among them, δ is the indicator function, when c = c ij When δ(c,c ij )=1, otherwise δ(c,c ij )=0; By combining the results of object detection and scene detection, we can get a comprehensive analysis result for each video frame; we can calculate the feature vector of each video frame, integrating object and scene information: F i =[C i ,S i ] Finally, the feature vector sequence of the video frame can be expressed as: F={F1,F2,…,F N }。
Citation Information
Cited By
Intelligent scene toning method and device based on feature recognition
CN120375107A
A method and device for intelligent scene color adjustment based on feature recognition
CN120375107B
Scene text video question answering method and system based on selection and focusing mechanism
CN122157282A
Scene text video question answering method and system based on selection and focus mechanism
CN122157282B
