Video subtitle generation method, device and storage medium

Abstract

The present invention relates to a method, device, and storage medium for generating video subtitles. The method acquires raw video data and samples to obtain video frames; generates frame subtitles using a trained image subtitle model; obtains object detection regions and corresponding labels for each frame using an object detection model; and extracts video embedding features using a video encoder; derives subgraphs from a pre-acquired knowledge graph as additional input features, and uses a subtitle temporal encoder to perform temporal feature encoding on each frame of subtitles to obtain corresponding frame subtitle embedding features; performs clustering and counting, and encodes the clustered and counted regional features using a regional spatiotemporal encoder to obtain clustered and counted regional embedding features; and inputs the obtained multimodal features into a multimodal decoder model to obtain the final video subtitles. Compared with existing technologies, the present invention has the following advantages.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, device and storage medium for generating video subtitles. Background Art

[0002] Video captioning is a task that combines computer vision and natural language processing techniques to automatically generate captions for video content. Deep learning models (such as ResNet and VitTransformer) extract visual features from video frames and may also incorporate multimodal information such as audio, speech transcripts, frame object detection results, and knowledge graphs. This information is then processed by sequence-to-sequence (Seq2Seq) models, such as long short-term memory (LSTM) networks or Transformers, to generate coherent captions. Video captioning has a wide range of applications in areas such as accessibility services, content retrieval, and automatic caption generation. It can improve the understanding of video content for people with visual impairments and enhance the efficiency of video search. With the continuous development of technology, the capabilities of video captioning are gradually improving, and it is of great value in promoting the application of artificial intelligence in multimedia content understanding and generation. In community settings, video captioning technology has enormous potential and can significantly impact community interactivity and accessibility. For example, real-time captioning can be provided for public safety videos to help people with hearing impairments obtain emergency information, or subtitles can be automatically generated for live community events, enabling residents of different language backgrounds to participate in the activities. As technology advances, video captioning technology is expected to provide more accurate and nuanced analysis of video content, such as identifying specific community activities or cultural elements, and generating more culturally sensitive and personalized descriptions.

[0003] However, existing methods for generating video captions have certain limitations. First, most existing methods directly sample video frames from raw video data, extracting multimodal features such as audio and text transcriptions for feature fusion. These features aim to extract as much video-related information as possible, thereby improving the accuracy of model output. However, they ignore the temporal characteristics of video frames, and the accuracy of model output still needs to be improved. Second, many existing methods use object detection output to extract regions containing video objects as key regions, thereby encouraging the model to focus on these regions during inference. To reduce noise, they select a fixed number of k regions for each frame based on the confidence level of the object detection output. This selection method ignores the distinction between object detection and captioning tasks, reducing the effectiveness of the model output. Therefore, how to further improve the accuracy and effectiveness of video caption generation has become a challenge that needs to be addressed in this field. Summary of the Invention

[0004] The purpose of the present invention is to provide a video subtitle generation method, device and storage medium in order to overcome the defects of the above-mentioned prior art. The method is based on an image subtitle model and region cluster selection, which can improve the accuracy and effectiveness of the output of the video subtitle generation model on the basis of the existing method.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to a first aspect of the present invention, a method for generating video subtitles is provided, comprising the following steps: obtaining original video data and sampling to obtain video frames; based on the video frames, generating frame subtitles using a trained image subtitle model, obtaining target detection areas and corresponding labels for each video frame using a trained target detection model, and extracting video embedding features using a video encoder model; based on the frame subtitles, deriving subgraphs from a pre-acquired knowledge graph as additional input features, and encoding the temporal features of each frame subtitle using a subtitle temporal encoder to obtain corresponding frame subtitle embedding features; clustering and counting the frame subtitles and the target detection areas and corresponding labels, and encoding the clustered and counted regional features using a regional spatiotemporal encoder to obtain clustered and counted regional embedding features; and inputting the video embedding features, the additional input features, the frame subtitle embedding features, and the clustered and counted regional embedding features into a preset multimodal decoder model to obtain final video subtitles.

[0007] As an optimal technical solution, the specific process of clustering and counting based on the frame subtitles, the target detection area and the corresponding labels includes: extracting all nouns in the frame subtitles, screening words used to describe objects and obtaining corresponding feature vectors, and calculating the cosine similarity between words; clustering the words based on a pre-set similarity threshold, and selecting the top k clusters with the largest number of samples; extracting the feature vectors of the output labels of the target detection model, and judging whether each label belongs to the top k clusters. If so, retaining the target detection area corresponding to the label, otherwise filtering the target detection area corresponding to the label.

[0008] As a preferred technical solution, the specific process of determining whether each label belongs to the first k clusters includes: obtaining the average sample features in each of the first k clusters; calculating the cosine similarity between the current label and the average sample features in each cluster; when the cosine similarity exceeds the pre-set similarity threshold, the current label belongs to the corresponding cluster.

[0009] As a preferred technical solution, the cosine similarity between the words is expressed as:

[0010]

[0011] In the formula, similarity represents similarity calculation, and X and Y represent the feature vectors of different words.

[0012] As a preferred technical solution, the multimodal decoder model includes a multimodal BERT model.

[0013] As a preferred technical solution, a regional spatiotemporal encoder is used to encode the clustered and counted regional features. The specific process includes:

[0014] imf i =Flatten(ResNet(region i ))

[0015] imr i =Concat(imf i ,Norm(bbox i ))

[0016] rf=MHA((imr 1, imr2,...,imr n )+TE)

[0017] Where imf i Indicates the i-th region feature (region i ), imr i It represents the regional features with added position information, rf represents the overall features obtained after fusing all regional features, the Flatten operation flattens the features into one dimension, the ResNet operation refers to the use of the ResNet model to extract regional visualization features, the Concat operation is used for the concatenation and splicing of features, Norm represents the normalization operation, bboxi represents the i-th bounding box, MHA represents the calculation of the multi-head attention mechanism, and TE is the temporal encoding.

[0018] As a preferred technical solution, the image caption model includes an LLaVA model.

[0019] As a preferred technical solution, the target detection model includes a Faster R-CNN model.

[0020] According to a second aspect of the present invention, a video subtitle generation device is provided, comprising a memory, a processor, and a program stored in the memory, wherein the processor implements the method when executing the program.

[0021] According to a third aspect of the present invention, there is provided a storage medium having a program stored thereon, wherein the program implements the method described above when executed.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention provides a video subtitle generation method based on an image subtitle model and regional clustering selection. This method goes one step further than existing video generation models by utilizing an image subtitle model trained on a large number of datasets to generate frame-level subtitles for videos. The method also performs temporal feature encoding on the frame-level subtitles, enabling the model to perceive the subtitle information and temporal information of each video frame. This further enhances the model's capabilities and improves the accuracy of video subtitle generation.

[0024] 2. The video subtitle generation method provided by the present invention clusters and counts target detection categories based on the content of frame subtitles. It then calculates and compares similarities to filter out redundant regions and selects regions corresponding to all categories in the top k most frequently appearing clusters in the frame subtitles as key regions. This region selection and screening method utilizes information from the frame subtitles and is more effective than existing confidence-based selection methods.

[0025] 3. The video subtitle generation device provided by the present invention adopts a video subtitle generation method based on an image subtitle model and region clustering selection, which significantly improves the efficiency of video subtitle generation while ensuring the accuracy of video subtitle generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the flow of the video subtitle generation method provided by the present invention;

[0027] Figure 2 Schematic diagram of the framework flow of the video subtitle generation method in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0029] Example

[0030] like Figure 1 As shown in FIG, the present invention provides a method for generating video subtitles based on an image subtitle model and region cluster selection. Figure 2 As shown, in this embodiment, the important components of implementing the method include:

[0031] (1) Sampling

[0032] Get the original video data and sample it to get the video frame.

[0033] (2) Extract three types of features from video frames

[0034] 1. Use a video encoder model (such as Video Swin Transformer) to extract video embedding features;

[0035] 2. Use a trained object detector / object detection model (such as Faster R-CNN) to obtain the object detection area and corresponding label for each frame;

[0036] 3. Use a trained image captioning model (such as LLaVA) to generate frame captions.

[0037] (3) Knowledge Graph and Subtitle Temporal Encoder

[0038] Knowledge graph: Based on the keywords in the frame captions, subgraphs are derived from public large-scale knowledge graphs (such as ConceptNet) as additional knowledge input to obtain additional input features.

[0039] Subtitle temporal encoder: Encodes the temporal features of each frame of subtitles. The encoder can choose LSTM, Transformer, etc. to obtain the corresponding frame subtitle embedding features.

[0040] (4) Calculate similarity

[0041] Use the stanza toolkit to extract all nouns in the frame captions, use wordnet to filter out words used to describe objects in the nouns, use the word2vec model to obtain the feature vectors of the words, and calculate the cosine similarity between the words:

[0042]

[0043] Where similarity represents similarity calculation, and X and Y represent the feature vectors of different words used to describe objects.

[0044] (5) Clustering and counting

[0045] The words are clustered based on a pre-set feature similarity threshold, and the top k clusters with the largest number of samples are selected to remove noise. Then, WordNet is used to extract the feature vector of the label output by the object detector. The feature vector is then determined to determine whether the label belongs to the top k clusters. If so, the region corresponding to the label is retained and used as the model input; otherwise, the region is filtered out.

[0046] The specific process of determining whether a label belongs to the top k clusters includes:

[0047] Get the average features of samples in each cluster of the first k clusters;

[0048] The cosine similarity between the current label and the average feature of samples in each cluster is calculated. When the cosine similarity exceeds the pre-set similarity threshold, the current label belongs to the corresponding cluster.

[0049] This step mainly filters out most of the areas output by target detection, most of which are redundant, with the aim of leaving areas corresponding to labels that appear more frequently in the frame caption text, similar to denoising.

[0050] (6) Regional spatiotemporal encoder

[0051] The clustered and counted regional features are encoded using a regional spatiotemporal encoder to obtain clustered and counted regional embedding features. The specific encoding process includes:

[0052] imf i =Flatten(ResNet(region i )) (2)

[0053] imr i =Concat(imf i ,Norm(bbox i )) (3)

[0054] rf=MHA((imr 1, imr2,…,imr n )+TE) (4)

[0055] Where imf i Indicates the i-th region feature (region i );imr i Represents the regional features with added position information; rf is the overall feature obtained by fusing all regional features. The Flatten operation flattens the features into one dimension; the ResNet operation refers to extracting regional visualization features using the ResNet model; the Concat operation is used for concatenation and splicing of features; Norm represents the normalization operation; bboxi represents the i-th bounding box (rectangular shape), and the content of the bounding box is (x1, y1, x2, y2), representing the pixel coordinates of the two vertices on the main diagonal of the bounding box. The image framed by the bounding box is the "region" referred to in this embodiment; MHA represents the calculation of the multi-head attention mechanism, and TE is temporal encoding.

[0056] (7) Multimodal BERT model

[0057] Using the BERT model as the decoder, the multimodal features obtained by Guo Chenchao, namely the video embedding features, additional input features, frame subtitle embedding features, and clustered and counted region embedding features, are input into the multimodal BERT model to obtain the final video subtitles output by the model.

[0058] Applying the above method within a community, the specific process is as follows:

[0059] 1. Video Capture

[0060] Video data of community scenes is collected through public cameras in the community or uploaded by user mobile terminals, and the video is evenly sampled to obtain video frames.

[0061] 2. Generate frame subtitles and frame target detection areas

[0062] The image captioning model LLaVA is used to generate captions for each frame, and the target detection model Faster R-CNN is used to generate candidate regions for each frame. Based on the frame captions, the stanza toolkit is used to extract nouns in the captions, and then wordnet is used to further filter out nouns used to describe objects. The word2vec model is used to obtain word vectors for the words, and then the word vectors are used to calculate the cosine similarity between words. Based on a given similarity threshold, DBSCAN is used to cluster the words, and the k clusters with the largest number of cluster samples are selected. The word2vec model is then used to extract the labels output by the target detector. The cosine similarity is used to determine whether the label belongs to the top k clusters. If so, the area corresponding to the label is retained, and the retained area is the key area. Otherwise, the result is filtered out.

[0063] 3. Multimodal Data Construction

[0064] (1) Video Embedding Construction

[0065] Use the pre-trained Video Swin Transformer model to extract video embedding features.

[0066] (2) Embedded construction in key areas

[0067] First, the ResNet model is used to extract the image features of the key areas and flatten the features into one dimension. The features are then concatenated with the normalized bbox. The triangular position embedding is used as the temporal feature to add to the concatenated features. The features are then input into a multi-layer Transformer model and temporally encoded using its multi-head attention mechanism to obtain the embedded features of the key areas.

[0068] (3) Embedded construction of frame subtitles

[0069] Use the pre-trained BERT-Base model to extract the subtitle embedding of each frame, take the vector corresponding to the output sentence start [CLS] label to represent the feature embedding of the corresponding subtitle, use triangular position embedding to add frame timing information to the subtitle feature, and then use the multi-layer Transformer model for temporal encoding to obtain the embedding feature of the frame subtitle.

[0070] (4) Construction of knowledge graph

[0071] The stanza tool is used to analyze the components of the frame subtitles to form a structured syntax tree. "Adjective-noun", "noun-noun", and "adverb-verb" phrases are collected from the syntax tree. These phrases are used to retrieve and export relevant subgraphs from the large knowledge graph ConceptNet, and the knowledge contained in the subgraphs is used as prompt words in natural language as input to the decoder BERT model.

[0072] 4. Multimodal Data Fusion

[0073] The constructed feature encodings of each modality are input into the BERT model as a decoder, and the global self-attention mechanism of the BERT model is used to fuse the modal features and generate the final output.

[0074] To demonstrate the effectiveness of this method, we conducted experiments on the existing MSRVTT video captioning dataset and compared it with existing methods. Following the same dataset partitioning method as the existing SwinBERT method, the dataset was divided into a training-validation set and a test set. Training was performed on the training-validation set and testing was performed on the test set, with the test results recorded. The experimental results are shown in Table 1.

[0075] Table 1 Experimental comparison results of this method and existing methods

[0076] method Bleu_4 METEOR ROUGE_L CIDEr SwinBERT 42.0 29.9 62.1 53.8 PickNet 41.3 27.7 59.8 44.1 SibNet 40.9 27.5 60.2 47.5 OA-BTG 41.4 28.2 - 46.9 GRU-EVE 38.3 28.4 60.7 48.1 MGSA 42.4 27.6 - 47.5 POS+CG 42.0 28.2 61.6 48.7 POS+VCT 42.3 29.7 62.8 49.1 SAAT 39.9 27.7 61.2 51.0 STG-KD 40.5 28.3 60.9 47.1 PMI-CAP 42.1 28.7 - 49.4 ORG-TRL 43.6 28.8 62.1 50.9 This method 43.8 30.1 63.3 55.6

[0077] Experimental results show that the method provided in this embodiment is superior to existing methods in most indicators.

[0078] Compared to existing methods for generating video subtitles, the method provided in this embodiment utilizes an image subtitle model to generate subtitles for each frame, encodes the spatiotemporal features of the frame subtitles, and then uses them as a new modal feature. Furthermore, compared to traditional region selection based on the confidence of target detection results, this method proposes a method based on region clustering selection: first, the nouns used to describe objects in the frame subtitles are extracted, then clustered based on the cosine similarity of their feature vectors, filtering out the top k clusters with the largest number of samples within the class, then filtering out the regions corresponding to the output labels of the target detectors belonging to these top k clusters, and finally encoding the spatiotemporal features of these regions. Therefore, the video subtitle generation method provided in this embodiment can effectively improve the performance of existing models (see the Appendix for details) and enhance the accuracy and effectiveness of video subtitle generation.

[0079] Furthermore, this embodiment provides a video subtitle generation device comprising a memory, a processor, and a program stored in the memory, wherein the processor implements the aforementioned method when executing the program. The device processor includes a central processing unit (CPU), which can perform various appropriate actions and processes based on computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). The RAM can also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. An input / output (I / O) interface is also connected to the bus. Multiple components in the device are connected to the I / O interface, including: input units such as a keyboard and mouse; output units such as various types of displays and speakers; storage units such as magnetic disks and optical disks; and communication units such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunications networks. The processing unit performs the various methods and processes described above, such as the aforementioned method. For example, in some embodiments, the aforementioned method can be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps in the aforementioned method can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the aforementioned method in any other appropriate manner (for example, by means of firmware). The functions described above can be performed at least in part by one or more hardware logic components. For example, without limitation, the hardware logic components of the exemplary types that can be used include: field programmable gate array (FPGA), application specific integrated circuit (ASIC), application specific standard product (ASSP), system on chip (SOC), complex programmable logic device (CPLD) and the like.

[0080] Furthermore, this embodiment also provides a storage medium having a program stored thereon, which, when executed, implements the aforementioned method. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server. In the context of the present invention, a computer-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0081] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for generating video subtitles, characterized in that: The following steps are involved: Get the original video data and sample it to get the video frame; Based on the video frames, using a trained image caption model to generate frame captions, using a trained object detection model to obtain the object detection area and corresponding labels of each video frame, and using a video encoder model to extract video embedding features; Based on the frame subtitles, a subgraph is derived from a pre-acquired knowledge graph as an additional input feature, and a subtitle temporal encoder is used to encode the temporal features of each frame subtitle to obtain corresponding frame subtitle embedding features; Clustering and counting are performed based on the frame subtitles, the target detection regions, and corresponding labels, and regional spatiotemporal encoders are used to encode regional features obtained through clustering and counting to obtain regional embedding features obtained through clustering and counting; The video embedding features, the additional input features, the frame subtitle embedding features, and the clustered and counted region embedding features are input into a preset multimodal decoder model to obtain final video subtitles.

2. The video subtitle generation method according to claim 1, wherein: The specific process of clustering and counting based on the frame subtitles, the target detection area and the corresponding labels includes: Extract all nouns in the frame captions, filter the words used to describe objects and obtain the corresponding feature vectors, and calculate the cosine similarity between the words; Based on a pre-set similarity threshold, the words are clustered and the top k clusters with the largest number of samples are selected; Extract the feature vector of the output label of the target detection model, and determine whether each label belongs to the first k clusters. If yes, retain the target detection area corresponding to the label; otherwise, filter the target detection area corresponding to the label.

3. The video subtitle generation method according to claim 2, wherein: The specific process of determining whether each label belongs to the first k clusters includes: Obtain the average characteristics of samples in each cluster of the first k clusters; The cosine similarity between the current tag and the average feature of the samples in each cluster is calculated. When the cosine similarity exceeds the preset similarity threshold, the current tag belongs to the corresponding cluster.

4. The video subtitle generation method according to claim 2, wherein: The cosine similarity between the words is expressed as: Where, Represents similarity calculation, X and Y represent feature vectors of different words.

5. The video subtitle generation method according to claim 1, wherein: The multimodal decoder model includes a multimodal BERT model.

6. The video subtitle generation method according to claim 1, wherein: The regional spatiotemporal encoder is used to encode the clustered and counted regional features. The specific process includes: Where, represents the i-th region feature ( ), It represents the regional features with added position information, rf represents the overall features obtained after fusing all regional features, the Flatten operation flattens the features into one dimension, the ResNet operation refers to the use of the ResNet model to extract regional visualization features, the Concat operation is used for the concatenation and splicing of features, Norm represents the normalization operation, bboxi represents the i-th bounding box, MHA represents the calculation of the multi-head attention mechanism, and TE is the temporal encoding.

7. The video subtitle generation method according to claim 1, characterized in that: The image caption model includes an LLaVA model.

8. The video subtitle generation method according to claim 1, wherein: The target detection model includes a Faster R-CNN model.

9. A video subtitle generation device, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images