Video annotation generation method and device and related equipment

Abstract

The invention provides a video annotation generation method and device and related equipment. The method comprises the following steps: inputting a plurality of obtained original videos into a trained video annotation generation model; acquiring visual features of each original video; obtaining entity characterization, predicate characterization and sentence characterization based on the visual features; dividing the acquired sentence representations into even video frame sentence representations and odd video frame sentence representations, and performing similarity matrix supervision by adopting a comparative learning mode according to the even video frame sentence representations and the odd video frame sentence representations; and integrating the output features of the original videos and obtaining aggregation features for generating subtitles. Three video encoders are designed to model time sequence details and spatial information, which is helpful for capturing features of video frames from different angles and granularities, enhancing extraction of inter-frame information, and avoiding insufficient extraction of inter-frame information content.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video annotation generation method, apparatus and related equipment. Background Art

[0002] Video annotation generation (i.e., video subtitle generation) is an important research direction in the intersection of computer vision and natural language processing, which aims to automatically generate accurate and comprehensive video content descriptions and has a wide range of application value. Although relevant methods have made significant progress in video annotation generation tasks, there are still limitations such as insufficient extraction of inter-frame information content.

[0003] Based on this, the present application provides a video annotation generation method, apparatus and related equipment. Summary of the invention

[0004] In view of the problems existing in the prior art, the present application provides a video annotation generation method, device and related equipment.

[0005] The purpose of this application is achieved by the following technical solutions:

[0006] In a first aspect, the present application provides a method for generating video annotations, the method comprising:

[0007] Constructing and training a video annotation generation model, and inputting the obtained multiple original videos into the trained video annotation generation model; the original video refers to a given untrimmed video;

[0008] For each of the original videos, even frames and odd frames are uniformly sampled to obtain even video frames and odd video frames, and then visual features thereof are obtained, wherein the visual features include object features, context features, and motion features;

[0009] Based on the visual features, entity representation, predicate representation and sentence representation are obtained; the obtained sentence representation is divided into an even video frame sentence representation and an odd video frame sentence representation, and similarity matrix supervision is performed by contrastive learning based on the even video frame sentence representation and the odd video frame sentence representation;

[0010] The output features of each of the original videos are integrated to obtain aggregate features for generating subtitles, wherein the output features include entity representation, predicate representation, and sentence representation.

[0011] In some possible implementations, the methods of obtaining visual features include:

[0012] Based on the object feature, the context feature and the motion feature, using an entity module to obtain an entity representation;

[0013] Combining the motion feature and the entity feature, using a predicate module to obtain a predicate representation;

[0014] The entity representation and the predicate representation are used as prior information, and a sentence module is used to perform prediction according to the visual feature input to obtain an even video frame sentence representation corresponding to the even video frame and an odd video frame sentence representation corresponding to the odd video frame.

[0015] In some possible implementations, methods of obtaining entity representation using the entity module include:

[0016] The DETR model based on the Transformer architecture is used as the basis of the entity module, and object features, context features and motion features are combined to extract entity representations.

[0017] In some possible implementations, combining the motion feature and the entity feature and using a predicate module to obtain a predicate representation includes:

[0018] Learning the dependency relationship between the motion feature and the entity feature;

[0019] Based on the BiLSTM model, motion features and aggregated entity features are combined to obtain predicate representation through attention mechanism and linear transformation, where the attention mechanism is used to adaptively select the most relevant input features.

[0020] In some possible implementations, after obtaining the aggregated features, the method further includes:

[0021] In the initial stage of subtitle generation, average pooling features are introduced to generate the initial vocabulary sequence in the initial stage;

[0022] For a given initial vocabulary sequence in an initial stage, an external language model is used to obtain a generation probability of the next word; the external language model is obtained by pre-training with a monolingual data set;

[0023] KL divergence is used to calculate the loss, and the cross entropy loss is used to train the model to reduce the difference between the generated predictions and the true labels and optimize the subtitle generation effect.

[0024] In some possible implementations, during the training process of the video annotation generation model, all loss function values ​​are added together and training is performed in an end-to-end manner.

[0025] In a second aspect, the present application also provides a video annotation generation device, including:

[0026] Constructing a training module, the training module is used to construct and train a video annotation generation model, and inputting the obtained multiple original videos into the trained video annotation generation model; the original video refers to a given untrimmed video;

[0027] A frame sampling module, wherein the frame sampling module is used to uniformly sample even frames and odd frames for each of the original videos and obtain even video frames and odd video frames, and then obtain visual features thereof, wherein the visual features include object features, context features, and motion features;

[0028] An inference module, the inference module is used to obtain entity representation, predicate representation and sentence representation based on the visual features, wherein the sentence representation is divided into an even video frame sentence representation and an odd video frame sentence representation;

[0029] A similarity matrix module, wherein the similarity matrix module is used to perform similarity matrix supervision by contrastive learning based on the sentence representations of the even video frames and the sentence representations of the odd video frames;

[0030] A decoding module is used to integrate the output features of each of the original videos and obtain aggregate features for generating subtitles, wherein the output features include entity representation, predicate representation and sentence representation.

[0031] In a third aspect, the present application further provides an electronic device, comprising a memory and at least one processor, wherein the memory stores a computer program, and the at least one processor is configured to implement the method as described in any one of the first aspects when executing the computer program.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having instructions stored thereon, wherein when the instructions are executed by a processor, the processor implements the method as described in any one of the first aspects.

[0033] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by at least one processor, the steps of any one of the methods described in the first aspect are implemented.

[0034] In combination with the above-mentioned technical solutions and the technical problems solved, the technical solutions to be protected by the present invention and the positive effects it possesses are: on the one hand, through the interaction of part-of-speech semantic supervision, the potential relationship in semantics can be better captured, and the model's multi-level understanding performance of semantic information can be improved. On the other hand, the use of similarity matrix supervision, combined with contrastive learning, effectively improves the model's attention to the unique content and detailed features of the video. On the other hand, three video encoders are designed to model temporal details and spatial information, which helps to capture the features of video frames from different angles and granularities, enhance the extraction of inter-frame information, and avoid insufficient extraction of inter-frame information content. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present application is further described below in conjunction with the accompanying drawings and implementation methods.

[0036] Figure 1 It is a flowchart of a video annotation generation method provided in an embodiment of the present application.

[0037] Figure 2 It is a schematic diagram of a framework of a video annotation generation model provided in an embodiment of the present application.

[0038] Figure 3 It is a schematic diagram of a flow chart for obtaining visual features provided in an embodiment of the present application.

[0039] Figure 4 It is a visual comparison diagram of the experiments of the invention method provided in the examples of the present application and other methods.

[0040] Figure 5 It is a structural schematic diagram of a video annotation generation device provided in an embodiment of the present application.

[0041] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] Below, in conjunction with the accompanying drawings and specific implementation methods, the present application is further described. It should be noted that, under the premise of no conflict, the various embodiments described below or the various technical features can be arbitrarily combined to form a new embodiment. The implementation procedures of the present application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation procedures, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0043] The following briefly describes an application scenario of an embodiment of the present application to facilitate understanding by those skilled in the art.

[0044] In terms of video content understanding, automatic subtitle generation can help the visually impaired better understand video content; in the field of video retrieval, accurate text descriptions can improve the efficiency and accuracy of video search; on social media platforms, automatically generated video descriptions can help content creators quickly publish short video content containing text descriptions; in the direction of medical testing, video annotation generation tasks can automatically generate key lesion descriptions based on medical video information, providing auxiliary reference for doctors' diagnosis.

[0045] Therefore, the task of video annotation generation has broad application prospects. However, how to introduce effective text supervision in the process of video feature extraction and how to better capture the temporal details and spatial information of the video have become urgent problems to be solved in this field.

[0046] Related video annotation generation methods can be divided into two directions:

[0047] The first direction focuses on designing powerful video encoders to learn representative video features. For example, graph convolutional networks (GCNs) are used to build relationship graphs to capture spatial and temporal interactions between video objects. As Transformer has shown superior performance in computer vision tasks, researchers designed the VideoBERT model, which uses BERT to process pre-computed CNN features. The SwinBERT model based on this directly learns visual features from raw video frames and adopts a window sliding strategy to achieve end-to-end optimization.

[0048] The second research direction is dedicated to narrowing the semantic gap between video representation and language description. Based on the grammatical structure of the generated sentence, fine-grained semantic features are constructed from local information to global information through text supervision, such as from predicting the subject, predicate, and object to combining the above information to construct a sentence.

[0049] The above directions still have the following problems: 1. Video encoders are deficient in processing inter-frame information. Current encoders mainly extract and aggregate frame-level representations for the same video, and lack modeling of frame-level information between different videos. This results in the model generating similar or even identical description outputs when processing videos of different themes due to insufficient extraction of inter-frame information, which cannot accurately reflect the unique content and detailed features of the video; 2. Existing work focuses too much on feature alignment at the same granularity level. These methods mainly use the same-granularity information of real words and predicted words to align, while ignoring the importance of cross-granularity alignment. This single alignment strategy limits the model's multi-level understanding of video content, thereby affecting the accuracy and completeness of the generated description.

[0050] In response to the limitations of insufficient inter-frame information content extraction raised in the background technology, and the fact that video encoders in related technologies are deficient in processing inter-frame information and ignore the importance of cross-granularity alignment, this application proposes a representation alignment framework based on contrastive learning. Specifically, three video encoders are designed to model temporal details and spatial information, and a multi-granularity contrastive learning mechanism is introduced to promote semantic consistency between different granularity levels, thereby improving the accuracy and completeness of the generated description, and helping to capture subtle changes between video frames and enhance the model's understanding of inter-frame information. The following first describes the method, and then describes the apparatus, equipment, etc.

[0051] Method embodiment.

[0052] See also Figure 1 and Figure 2 , Figure 1 is a flow chart of a video annotation generation method provided in an embodiment of the present application. Figure 2 It is a schematic diagram of a framework of a video annotation generation model provided in an embodiment of the present application.

[0053] The video annotation generation method includes:

[0054] S101, constructing and training a video annotation generation model, inputting a plurality of acquired original videos into the trained video annotation generation model; the original video refers to a given untrimmed video;

[0055] S102, for each of the original videos, uniformly sampling even frames and odd frames and obtaining even video frames and odd video frames, and then obtaining visual features thereof; the visual features include object features, context features, and motion features;

[0056] S103, based on the visual features, obtaining entity representation, predicate representation and sentence representation; dividing the obtained sentence representation into even video frame sentence representation and odd video frame sentence representation, and performing similarity matrix supervision by contrastive learning based on the even video frame sentence representation and the odd video frame sentence representation;

[0057] S104, integrating the output features of each of the original videos and acquiring aggregate features for generating subtitles, the output features including entity representation, predicate representation, and sentence representation.

[0058] The technical solution provided in this embodiment uses a video annotation generation model to uniformly sample each input original video, obtain even frames and odd frames respectively, and retain the timing information of the video. The visual features of the even frames and odd frames are extracted, and the extracted visual features are used to further obtain entity representation (used to identify the main objects in the video), predicate representation (used to describe the action or state of the object) and sentence representation (used to form a complete sentence description). The sentence representation is divided into sentence representations of even frames and odd frames, and similarity matrix supervision is performed using contrastive learning to ensure that the model can capture fine-grained information between frames and maintain semantic consistency. Then, the model integrates the output features of each video (including entity representation, predicate representation, and sentence representation) to obtain aggregate features for generating subtitles. Aggregate features contain key information about the video content, and aggregate features can be used to generate accurate and comprehensive video subtitles.

[0059] Therefore, through the multi-granularity contrast learning mechanism and similarity matrix supervision, the model can generate more accurate and comprehensive subtitles that better reflect the video content. The method of uniformly sampling even and odd frames helps the model capture the timing details of the video and enhances its understanding of the dynamic changes of the video. By uniformly sampling frames instead of processing the entire video, the model can use computing resources more efficiently and increase processing speed. The multi-granularity contrast learning mechanism promotes semantic consistency between different granularity levels, thereby improving the accuracy and completeness of the generated description, and helps capture subtle changes between video frames, enhancing the model's understanding of inter-frame information.

[0060] See also Figure 3 , Figure 3 It is a schematic diagram of a flow chart for obtaining visual features provided in an embodiment of the present application.

[0061] In an exemplary embodiment, the method of obtaining the visual feature includes:

[0062] S201, acquiring entity representation using an entity module based on the object feature, the context feature and the motion feature;

[0063] S202, combining the motion feature and the entity feature, and obtaining a predicate representation using a predicate module;

[0064] S203, using the entity representation and the predicate representation as prior information, using a sentence module to perform prediction according to the visual feature input, to obtain even video frame sentence representations corresponding to even video frames and odd video frame sentence representations corresponding to odd video frames.

[0065] The technical solution provided in this embodiment uses an entity module to extract entity representations in combination with object features, context features and motion features in the video. It can be considered that object features may include the shape, size and color of the object; context features involve environmental information around the object; and motion features describe the motion state of the object, such as speed and direction. The role of the entity module is to identify key objects in the video and convert them into a representation form that can be used for further processing. Then, the predicate module is used to obtain the predicate representation in combination with the motion features and entity features. The predicate representation is used to understand and express action events in the video. The entity representation and predicate representation are input into the sentence module as prior information, and combined with the visual feature input, the sentence representations corresponding to the even video frames and the odd video frames are predicted and generated.

[0066] Therefore, by integrating object features, context features, and motion features, the model can more comprehensively understand the video content and improve the accuracy of video annotation. Using entity representation and predicate representation as prior information, the subtitles generated by the model are more relevant and specific, and can accurately reflect the events and actions in the video. The prediction mechanism of the sentence module helps to generate coherent sentence representations, making the final subtitles more semantically fluent and natural.

[0067] In an exemplary embodiment, the method of obtaining entity representation by using the entity module includes:

[0068] The DETR model based on the Transformer architecture is used as the basis of the entity module, and object features, context features and motion features are combined to extract entity representations.

[0069] The technical solution provided by this embodiment is that the DETR model can process image sequences through the self-attention mechanism to achieve object detection and representation learning. The encoder-decoder architecture in the DETR model allows the model to learn global dependencies in image sequences and generate entity representations through the decoder.

[0070] In an exemplary embodiment, combining the motion feature and the entity feature and obtaining a predicate representation using a predicate module includes:

[0071] Learning the dependency relationship between the motion feature and the entity feature;

[0072] Based on the BiLSTM model, motion features and aggregated entity features are combined to obtain predicate representation through attention mechanism and linear transformation, where the attention mechanism is used to adaptively select the most relevant input features.

[0073] In the technical solution provided by this embodiment, the predicate module first learns the dependency between motion features and entity features. Motion features describe the motion state of objects in the video, while entity features describe the properties of the object itself. By learning the dependency between the two features, we can understand how the movement of the object depends on its properties and the surrounding environment. Then, the predicate module aggregates the entity features. A bidirectional long short-term memory network (BiLSTM) model is used to process the aggregated entity features and motion features. BiLSTM can capture long-term dependencies in time series data. The features that are most relevant to the current task are adaptively selected through the attention mechanism. In specific applications, the attention-weighted features can be linearly transformed to generate the final predicate representation.

[0074] Therefore, by learning the dependencies between motion features and entity features, the model can more accurately understand the actions and events in the video. The use of BiLSTM enhances the model's ability to process time series data and helps capture the context of actions in the video. The attention mechanism enables the model to automatically focus on the most important features, improving the information quality and relevance of the predicate representation.

[0075] In an exemplary embodiment, after obtaining the aggregated features, the method further includes:

[0076] In the initial stage of subtitle generation, average pooling features are introduced to generate the initial vocabulary sequence in the initial stage;

[0077] For a given initial vocabulary sequence in an initial stage, an external language model is used to obtain a generation probability of the next word; the external language model is obtained by pre-training with a monolingual data set;

[0078] KL divergence is used to calculate the loss, and the cross entropy loss is used to train the model to reduce the difference between the generated predictions and the true labels and optimize the subtitle generation effect.

[0079] The introduction of average pooling features helps the model start quickly in the early stages of caption generation and generate reasonable initial vocabulary sequences even in the absence of contextual information.

[0080] In an exemplary embodiment, during the training process of the video annotation generation model, all loss function values ​​are added together and training is performed in an end-to-end manner.

[0081] As an example, a video annotation generation method is provided, including:

[0082] Step 1: Video frame sampling and feature extraction:

[0083] The datasets used in this step are MSVD and MSRVTT, which contain multiple types of video data. First, for each given video that has never been trimmed, even T frames and odd T frames are uniformly sampled, with T taking a value of 15. This process ensures that the temporal information of the video is effectively preserved while avoiding processing difficulties when the video length is too long. Uniform sampling helps capture key information in different time periods and ensures the representativeness of each frame. This example uses Faster-RCNN (Ren et al., 2015) for object detection and extracts object features in the video. The object feature dimension is d v =512, indicating the dimension of the embedding vector for each object. Next, we use InceptionResNetV2 (Szegedy et al., 2017) to extract contextual features from each frame. C3D (3D Convolutional Network) (Hara et al., 2018) is a deep learning model designed specifically for video analysis that can capture dynamic information in videos in the temporal dimension.

[0084] Step 2: Entity noun, predicate, and sentence representation extraction:

[0085] This step uses DETR (Deformable DETR, Zhu et al., 2020) as the basis of the entity module, which uses the Transformer architecture to solve the object detection problem. This module combines object features O, context features C, and motion features M to extract entity representation E. This module includes L = 6 learnable queries for aligning the main objects in the video.

[0086] E=DETR(O,C,M)

[0087] Secondly, this step designs a predicate module to generate predicate representation. The construction of a sentence usually involves entities and related actions, in which the predicate plays a key connecting role. Therefore, the method first learns the dependency between motion features and entity features, and then, considering the long-term dependency of the video, the method adopts BiLSTM, which is widely used due to its high efficiency in processing time series information. Through BiLSTM, the method combines motion features and aggregated entity features to obtain the predicate representation P. The methods provided in the related art mainly focus on comparing word by word with the true label, ignoring the complex relationships between words, making it difficult to construct fine-grained semantic features. Therefore, the sentence module designed in this example, in addition to inputting visual features, also uses the previously extracted entity representation and predicate representation as prior information to predict the final sentence representation. The sentence module design of this example adopts the same method as the predicate, and finally obtains the sentence representation S.

[0088] P=Linear(BILSTM(Attention(E,M)))

[0089] S=Linear(BiLSTM(Attention(E,C,M)),P)

[0090] Among them, the attention mechanism Attention method is used to adaptively select the most relevant input features. Take the formula with two features as input as an example:

[0091] Attention(E,M)=Softmax(w p tanh(W p M+U p E+b p ))

[0092] w p , W p , U p and b p are all learnable parameters.

[0093] In summary, this example has generated three different types of semantic predictions. To guide the optimization of the corresponding prediction G, so as to achieve correct alignment in the training stage, a cosine similarity loss function is further designed.

[0094]

[0095] Step 3: Similarity matrix supervision:

[0096] The present application believes that the descriptions generated by the existing methods for two different videos are often too general, resulting in almost identical descriptions. This is mainly because the network lacks sufficient understanding of the interaction of video frames. For the same video, the trained network should be sufficiently robust, that is, after modifying some frames after extraction, similar descriptions can still be generated. On the other hand, each video should have its own unique description. Even if some video frames are similar, the network should fully consider the overall video frames to infer the description of the video. In order to realize this idea, the present invention mainly processes video frames. In order to distinguish adjacent frames, the frame extraction method of the present invention adopts odd frames and even frames to obtain two types of video frames of the same video, and inputs them into the model, and outputs sentence representations at the sentence module, that is, M videos obtain 2M sentence representations S. For these sentence representations, the present invention designs a similarity matrix, adopts a contrastive learning method, and promotes the model to pay attention to the fine-grained information between frames through the calculation method of MSE loss. The specific formula is as follows:

[0097]

[0098] in, It is the similarity matrix obtained by calculating the cosine similarity of the real sentence representation of the video.

[0099] Step 4: Video decoding module:

[0100] This module aims to integrate the output features to obtain the aggregated feature F, which contains noun, predicate and sentence information. This approach enables the decoding module to promote vocabulary generation from a global perspective, which not only improves the accuracy of generation, but also enhances the consistency and richness of semantics.

[0101] F=Linear(Cat(E,P,S))

[0102] In the technical solution provided by the related art, the transformer-based decoder generally includes the following steps:

[0103] H=SA(Embed(source+Pos)),

[0104] P=LMH(CA(H,F,F))

[0105] Among them, source represents the query vector of the target word, Pos represents the sinusoidal position encoding, SA and CA represent the self-attention module and the cross-attention module respectively. In addition, LMH (i.e., language model head) combines a multi-layer perceptron (MLP) and a normalization layer to project features onto the output text probability P.

[0106] However, unlike the next word prediction task in large-scale language models, no vocabulary has been generated in the initial stage of video subtitle decoding. In other words, the feature embedding consists only of randomly initialized sources and fixed position information encoding. In this setting, it becomes challenging to generate accurate vocabulary due to the lack of useful information references.

[0107] To solve this problem, this example introduces the average pooling feature F in the embedding feature calculation stage, thereby providing effective information in the initial generation stage.

[0108]

[0109] P'=LMH(CA(H',F,F))

[0110] Step 5: Subtitle generation:

[0111] This example believes that the long tail problem mainly stems from the insufficient number of samples of certain words in the dataset, which makes it difficult for the model to effectively learn these rare words. Therefore, in the subtitle generation stage, this example introduces an external language model (ELM). The external language model is pre-trained on a large-scale monolingual dataset, so that it can transfer rich linguistic knowledge and help the model better generate rare words. Specifically, given the previous series of words w <n , the probability of ELM generating the nth word is:

[0112] Q n =ELM(w <n ∣Θ ELM )

[0113] Next, this example uses KL divergence to calculate the loss. The specific formula is as follows:

[0114]

[0115] Finally, to further optimize the subtitle generation effect, this example trains the model by calculating the cross entropy loss. The loss function helps reduce the difference between the generated predictions and the true labels during the training process, thereby ensuring that the generated subtitles are more consistent with the true semantics. The specific formula is as follows:

[0116]

[0117] Step 6: Model training:

[0118] In this step, all losses are added together and trained in an end-to-end manner. The specific formula is as follows:

[0119] L=ω G L G +ω M L M +ω soft L soft +ω XE L XE ,

[0120] Among them, ω G ,ω M ,ω soft And ω XE They are training hyperparameters respectively.

[0121] The beneficial effects of this example are: on the one hand, through the interaction of part-of-speech semantic supervision, the potential relationship in semantics is better captured, and the model's multi-level understanding performance of semantic information is improved. On the other hand, the use of similarity matrix supervision and combined with contrastive learning effectively improves the model's attention to the unique content and detailed features of the video. On the other hand, three video encoders are designed to model temporal details and spatial information, which helps to capture the features of video frames from different angles and granularities, enhance the extraction of inter-frame information, and avoid insufficient extraction of inter-frame information content.

[0122] As another example, the experimental process and results of using the video annotation generation model provided by this application are provided.

[0123] In terms of datasets, this example uses two datasets to verify the effectiveness. The first is MSVD (Microsoft Video Description Dataset). The MSVD dataset contains about 2,000 short videos, each 10 seconds long, covering a wide range of daily scenes. Each video is accompanied by multiple manually annotated subtitle descriptions. This dataset is widely used in video subtitle generation tasks and is suitable for evaluating the performance of models in generating accurate and diverse subtitles. The second is MSR-VTT (Microsoft Research Video to Text). The MSR-VTT dataset is a large-scale video-text dataset containing more than 10,000 videos covering 20 different categories of topics, such as sports, entertainment, travel, etc. Each video has subtitle descriptions from multiple annotators, which is suitable for evaluating the performance of models in complex scenarios.

[0124] In terms of evaluation indicators, in order to comprehensively evaluate the performance of the model, this example uses the following four commonly used evaluation indicators:

[0125] BLEU, BLEU is an n-gram-based evaluation metric commonly used in machine translation and text generation tasks. It measures the overlap between the generated text and the reference text on n-grams. BLEU-1, BLEU-2, BLEU-3, and BLEU-4 are calculated, corresponding to the matching of 1-gram, 2-gram, 3-gram, and 4-gram, respectively. Consistent with existing methods, this example uses BLEU-4 (B@4) as the final metric.

[0126] METEOR(M),METEOR takes into account factors such as precision, recall, synonym matching and word order, aiming to solve the problem that the BLEU indicator only focuses on n-gram matching.

[0127] CIDEr (C), CIDEr is an evaluation metric designed for image description and video caption generation. It evaluates the generation quality by calculating the consensus between the generated description and the reference description, focusing on semantic consistency and diversity.

[0128] ROUGE(R), the ROUGE indicator is mainly used for automatic summarization and text generation tasks. It evaluates the overlap between the generated text and the reference text, focusing on recall and coverage, which is particularly important in multi-round dialogues or long text generation.

[0129] In terms of experimental settings, this example uses the Adam optimizer for model optimization, the learning rate is set to 1e-4, the total number of training cycles is 20, and the batch size is set to 64. In terms of text processing, this example uses the syntactic analysis tool in AllenNLP, which can efficiently extract nouns and predicates in subtitles. In this way, this example can identify key information in the video description, thereby providing support for subsequent semantic modeling. Subsequently, the SBERT model is used to process the real-label video description, entity, predicate, and corpus text to generate the corresponding embedding vector, and the dimension of the embedding vector is 768. In the inference stage, this example uses the beam search algorithm and sets the beam width to 5 to generate the final subtitles. All experiments are performed on a single graphics processing unit (GPU) to ensure the efficiency and repeatability of the experiments.

[0130] In terms of benchmark models, the video annotation generation model provided in this application is compared with the following models:

[0131] GRU-EVE: Extends the traditional GRU (Gated Recurrent Unit) network by introducing event embeddings, thereby enhancing the ability to model event and temporal relationships in video caption generation;

[0132] ORG-TRL: In order to solve the lack of sufficient visual representation of interactions between objects, an object relationship graph is introduced, and a teacher recommendation learning method is used to improve the accuracy of prediction results;

[0133] MGRMP: It improves the semantic accuracy of generated subtitles by integrating multimodal information, especially for long videos and multi-view video subtitle generation tasks.

[0134] HMN: It uses three different levels of memory networks to capture video representation and text semantic information, showing excellent performance;

[0135] RSFD: Utilizes frequency-aware diffusion module and semantic supervision module to address the long-tail problem in video captioning tasks.

[0136] Experimental results and analysis: The experimental results are shown in Tables 1 and 2 below:

[0137] Table 1 Experimental comparison results of different methods on MSVD dataset

[0138]

[0139] Table 2 Experimental comparison results of different methods on the MSRVTT dataset

[0140]

[0141]

[0142] Experimental results analysis:

[0143] Table 1 shows the experimental results of different methods on the MSVD dataset. The method provided in this application is competitive in multiple evaluation indicators. On the METEOR indicator, this method achieved a score of 38.0, which is the best performance compared to all other methods. CIDEr is the main evaluation indicator in video subtitles, and its value is related to whether the generated subtitles contain the main elements of the video. The method provided in this application exceeds the best existing method by 5.4%, which proves that focusing on inter-frame information can effectively mine potential semantic information, thereby generating more expressive subtitle content. On the other hand, Table 2 shows the experimental results of this method on the MSRVTT dataset, which is consistent with the performance of the MSVD dataset. The method provided in this application is 1.0%, 0.6% and 2.0% higher than the most advanced HMN in the three indicators of B@4, M, and C, respectively. The results obtained on a larger dataset prove that the method provided in this application can effectively improve the semantic consistency and expressiveness of the generated subtitles.

[0144] For experimental visualization, see Figure 4 , Figure 4 It is a visual comparison diagram of the experiment of the inventive method provided in the embodiment of the present application and other methods, which is used to show the visual comparison of the baseline model (Baseline), the model of the present invention (Ours) and the real annotation (GT, Ground Truth).

[0145] Obviously, the method provided in this application visualizes some videos on the MSVD dataset, and it can be concluded that it can generate descriptions related to the video content more accurately. For example, when processing videos involving complex actions (such as slicing and stirring), the model of this method can more clearly capture the semantics of each action, thereby generating a more fine-grained description, while the baseline model may miss these details. Moreover, through the comparison of inter-frame information, the model of this method has a higher ability to capture details. For example, in multi-step cooking scenes involving complex operations, the method can accurately describe the objects and operations involved in each step, such as "stirring the mixture", "cutting baguettes", etc. The baseline model may generate more general or universal descriptions that are difficult to reflect the uniqueness of the video.

[0146] Device Embodiments.

[0147] The embodiment of the present application provides a video annotation generation device, and its specific implementation method is consistent with the implementation method and technical effects recorded in the above-mentioned system implementation method, and some contents are not repeated here.

[0148] See also Figure 2 and Figure 5 , Figure 5 It is a structural schematic diagram of a video annotation generation device provided in an embodiment of the present application. In the video annotation generation model, input video 1, input video 2 to input video T are input to the frame sampling (module), and then each input video is separately input into the reasoning module, and each reasoning module includes an entity module, a predicate module and a sentence module. The output of the sentence module is connected to the decoding module and the similarity matrix module, and the similarity matrix module is also used to access the true value matrix. The decoding module outputs the video annotation "a man is playing baseball on the playground". The true value matrix is ​​used to represent the correspondence or matching degree between the video frame and the generated sentence, and is used to supervise the learning process.

[0149] The video annotation generating device comprises:

[0150] Constructing a training module, the training module is used to construct and train a video annotation generation model, and inputting the obtained multiple original videos into the trained video annotation generation model; the original video refers to a given untrimmed video;

[0151] A frame sampling module, wherein the frame sampling module is used to uniformly sample even frames and odd frames for each of the original videos and obtain even video frames and odd video frames, and then obtain visual features thereof, wherein the visual features include object features, context features, and motion features;

[0152] An inference module, the inference module is used to obtain entity representation, predicate representation and sentence representation based on the visual features, wherein the sentence representation is divided into an even video frame sentence representation and an odd video frame sentence representation;

[0153] A similarity matrix module, wherein the similarity matrix module is used to perform similarity matrix supervision by contrastive learning based on the sentence representations of the even video frames and the sentence representations of the odd video frames;

[0154] A decoding module is used to integrate the output features of each of the original videos and obtain aggregate features for generating subtitles, wherein the output features include entity representation, predicate representation and sentence representation.

[0155] In an exemplary embodiment, the method of obtaining the visual feature includes:

[0156] Based on the object feature, the context feature and the motion feature, using an entity module to obtain an entity representation;

[0157] Combining the motion feature and the entity feature, using a predicate module to obtain a predicate representation;

[0158] The entity representation and the predicate representation are used as prior information, and a sentence module is used to perform prediction according to the visual feature input to obtain an even video frame sentence representation corresponding to the even video frame and an odd video frame sentence representation corresponding to the odd video frame.

[0159] In an exemplary embodiment, the method of obtaining entity representation by using the entity module includes:

[0160] The DETR model based on the Transformer architecture is used as the basis of the entity module, and object features, context features and motion features are combined to extract entity representations.

[0161] In an exemplary embodiment, combining the motion feature and the entity feature and obtaining a predicate representation using a predicate module includes:

[0162] Learning the dependency relationship between the motion feature and the entity feature;

[0163] Based on the BiLSTM model, motion features and aggregated entity features are combined to obtain predicate representation through attention mechanism and linear transformation, where the attention mechanism is used to adaptively select the most relevant input features.

[0164] In an exemplary embodiment, after the decoding module, the following is further included:

[0165] The subtitle generation module is used to introduce average pooling features in the initial stage of subtitle generation to generate an initial vocabulary sequence in the initial stage; for a given initial vocabulary sequence in the initial stage, an external language model is used to obtain the generation probability of the next word; the external language model is obtained after pre-training using a monolingual data set; KL divergence is used to calculate the loss, and the cross entropy loss is used to train the model to reduce the difference between the generated prediction and the true label, thereby optimizing the subtitle generation effect.

[0166] In an exemplary embodiment, during the training process of the video annotation generation model, all loss function values ​​are added together and training is performed in an end-to-end manner.

[0167] Device Embodiments.

[0168] This embodiment provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the steps of the method described in any one of the method embodiments are implemented.

[0169] See also Figure 6 , Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0170] The electronic device 10 may include, for example, at least one memory 11 , at least one processor 12 , and a bus 13 connecting different platform systems.

[0171] The memory 11 may include a (computer) readable medium in the form of a volatile memory, such as a random access memory (RAM) 111 and / or a cache memory 112 , and may further include a read-only memory (ROM) 113 .

[0172] The memory 11 also stores a computer program, which can be executed by the processor 12 so that the processor 12 implements the steps of any of the above methods.

[0173] The memory 11 may also include a utility 114 having at least one program module 115, such program module 115 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0174] Accordingly, the processor 12 may execute the above-mentioned computer program, and may execute the utility 114 .

[0175] The processor 12 may adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0176] The bus 13 may be a local bus representing one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or any of a variety of bus architectures.

[0177] The electronic device 10 may also communicate with one or more external devices such as a keyboard, a pointing device, a Bluetooth device, etc., and may also communicate with one or more devices that can interact with the electronic device 10, and / or communicate with any device (such as a router, a modem, etc.) that enables the electronic device 10 to communicate with one or more other computing devices. Such communication may be performed through an input / output interface 14. In addition, the electronic device 10 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 15. The network adapter 15 may communicate with other modules of the electronic device 10 through the bus 13. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 10 in actual applications, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0178] Storage Medium Embodiments.

[0179] The embodiment of the present application also provides a computer-readable storage medium, the specific embodiment of which is consistent with the embodiment recorded in the above method embodiment and the technical effects achieved, and some contents will not be repeated here.

[0180] The computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the steps of any of the above methods or the functions of any of the above electronic devices are implemented.

[0181] Computer readable medium can be a computer readable signal medium or a computer readable storage medium. In an embodiment of the present application, a computer readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. Computer readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0182] The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable storage medium may also be any computer-readable medium that can send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the computer-readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. The program code for performing the operation of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as C language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device, partially on a remote computing device, or entirely on a remote computing device or server. Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0183] Program Product Embodiments.

[0184] The embodiment of the present application also provides a computer program product, the specific embodiments of which are consistent with the embodiments recorded in the above method embodiments and the technical effects achieved, and some contents will not be repeated here.

[0185] The present application provides a computer program product, which includes a computer program. When the computer program is executed by at least one processor, the steps of any of the above methods or the functions of any of the above electronic devices are implemented.

[0186] The computer program product is used to implement the steps of any of the above methods or to implement the functions of any of the above electronic devices. The computer program product may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer program product of the present invention is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0187] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b and c can be single or multiple. It is worth noting that "at least one" can also be interpreted as "one or more items".

[0188] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are configured to distinguish similar objects, and do not have to be configured to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0189] This application is explained from the perspectives of purpose of use, effectiveness, progress and novelty, and has met the functional enhancement and usage requirements emphasized by the Patent Law. The above description and drawings of this application are only the preferred embodiments of this application, and are not intended to limit this application. Therefore, all structures, devices, features, etc. that are similar or identical to this application, that is, all equivalent replacements or modifications made in accordance with the scope of the patent application of this application, should fall within the scope of protection of the patent application of this application.

Claims

1. A video annotation generation method, characterized in that: The method comprises: Constructing and training a video annotation generation model, and inputting the obtained multiple original videos into the trained video annotation generation model; the original video refers to a given untrimmed video; For each of the original videos, even frames and odd frames are uniformly sampled to obtain even video frames and odd video frames, and then visual features thereof are obtained, wherein the visual features include object features, context features, and motion features; Based on the visual features, entity representation, predicate representation and sentence representation are obtained; the obtained sentence representation is divided into an even video frame sentence representation and an odd video frame sentence representation, and similarity matrix supervision is performed by contrastive learning based on the even video frame sentence representation and the odd video frame sentence representation; The output features of each of the original videos are integrated to obtain aggregate features for generating subtitles, wherein the output features include entity representation, predicate representation, and sentence representation.

2. The video annotation generation method according to claim 1, characterized in that: Ways to obtain visual features include: Based on the object feature, the context feature and the motion feature, using an entity module to obtain an entity representation; Combining the motion feature and the entity feature, using a predicate module to obtain a predicate representation; The entity representation and the predicate representation are used as prior information, and a sentence module is used to perform prediction according to the visual feature input to obtain an even video frame sentence representation corresponding to the even video frame and an odd video frame sentence representation corresponding to the odd video frame.

3. The video annotation generation method according to claim 2, characterized in that: Ways to use the entity module to obtain entity representation include: The DETR model based on the Transformer architecture is used as the basis of the entity module, and object features, context features and motion features are combined to extract entity representations.

4. The video annotation generation method according to claim 2, characterized in that: The combining the motion feature and the entity feature and obtaining a predicate representation by using a predicate module includes: Learning the dependency relationship between the motion feature and the entity feature; Based on the BiLSTM model, motion features and aggregated entity features are combined to obtain predicate representation through attention mechanism and linear transformation, where the attention mechanism is used to adaptively select the most relevant input features.

5. The video annotation generation method according to claim 2, characterized in that: After obtaining the aggregated features, the method further includes: In the initial stage of subtitle generation, average pooling features are introduced to generate the initial vocabulary sequence in the initial stage; For a given initial vocabulary sequence in an initial stage, an external language model is used to obtain a generation probability of the next word; the external language model is obtained by pre-training with a monolingual data set; The KL divergence is used to calculate the loss, and the cross entropy loss is used to train the model to reduce the difference between the generated predictions and the true labels.

6. The video annotation generation method according to claim 5, characterized in that: During the training process of the video annotation generation model, all loss function values ​​are added together and trained in an end-to-end manner.

7. A video annotation generation device, characterized in that: include: Constructing a training module, the training module is used to construct and train a video annotation generation model, and inputting the obtained multiple original videos into the trained video annotation generation model; the original video refers to a given untrimmed video; A frame sampling module, wherein the frame sampling module is used to uniformly sample even frames and odd frames for each of the original videos and obtain even video frames and odd video frames, and then obtain visual features thereof, wherein the visual features include object features, context features, and motion features; An inference module, the inference module is used to obtain entity representation, predicate representation and sentence representation based on the visual features, wherein the sentence representation is divided into an even video frame sentence representation and an odd video frame sentence representation; A similarity matrix module, wherein the similarity matrix module is used to perform similarity matrix supervision by contrastive learning based on the sentence representations of the even video frames and the sentence representations of the odd video frames; A decoding module is used to integrate the output features of each of the original videos and obtain aggregate features for generating subtitles, wherein the output features include entity representation, predicate representation and sentence representation.

8. An electronic device, characterized in that: The electronic device comprises a memory and at least one processor, wherein the memory stores a computer program, and the at least one processor is configured to implement the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: Instructions are stored thereon, and when the instructions are executed by a processor, the processor implements the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by at least one processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images