A video description generation method and device and a storage medium

Abstract

The application relates to the technical field of video understanding, in particular to a video description generation method and device and a computer storage medium. The video description generation method provided by the application proposes time attention and fusion attention, and adds a mark to each frame feature online to represent global features, and the marks interact with each other to transfer and interact information between frames. First, all the marks are interacted by using time attention to learn the time relationship between the video frames of the input video. Next, the marks processed by the time attention are concatenated with each feature vector in the final feature, and then input into the fusion attention. The fusion attention makes all the features interact with each other, so that the features output after the fusion attention contain the time relationship between the video frames. The spatial features and time sequence information obtained by the pre-training model are combined in the application, the time clues between the video frames are captured, and finally more sufficient feature representation can be obtained, and the prediction accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video understanding, and particularly to a video description generation method and device and computer storage medium. BACKGROUND

[0002] In the prior art, video captioning technology aims to automatically generate natural language descriptions from videos, and plays an important role in helping the visually impaired, human-computer interaction, video retrieval and many other applications. Early video captioning generation methods usually first utilize visual information detected in the video, and then generate sentences through pre-set templates manually set by humans. However, this method highly depends on a fixed set of templates, and can only generate sentences with fixed syntax structures, resulting in unsatisfactory final presentation.

[0003] To solve the above problems, a video description generation system based on a graph convolution network is disclosed in Chinese Patent No. CN202010233049.7, which includes a video feature extraction network, a graph convolution network, a visual attention network and a sentence description generation network. In operation, the video is first sampled and processed by the video feature extraction network to obtain video features, which are output to the graph convolution network; then the graph convolution network is used to reconstruct the video features according to the semantic relationship, and input to the sentence description generation recurrent neural network; finally, the sentence description generation network is used to generate sentences according to the reconstructed features of the video. This system uses graph convolution to reconstruct the frame-level sequence and target-level sequence features in the video, and fully utilizes the temporal information and semantic information within the video when generating the description sentences, making the generation more accurate. However, this system is a video encoder built on the basis of a CNN model, and only uses information from the visual aspect, lacking sufficient language representation. For example, Chinese Patent No. CN202111060036.5 discloses a system and method for generating video description text, which includes a visual encoding module, a caption generation module and a cross-modal information flow alignment module. In operation, the visual encoding module is first used to globally encode the video to be processed to obtain global video features, and the video segmentation information and global video features are used to locally encode the corresponding video segments to obtain visual modal representations of each video frame in the video segments; then in the caption generation module, the visual modal representations are embedded to obtain text modal representations, and description text is generated based on the text modal representations. The description text generated by this system not only has visual accuracy and visual consistency, but also has more coherent front and back texts. However, video data usually contains information about the transformation of objects over time, and this system does not effectively utilize the temporal clues within the video segments. SUMMARY

[0004] Therefore, the technical problem to be solved by the present application is to overcome the problem that the prior art does not effectively utilize the temporal clues within the video segments.

[0005] To solve the above technical problems, the present application provides a video description generation method, comprising:

[0006] inputting a video segment into a trained video description generation model;

[0007] extracting a plurality of video frame features using a pre-trained video encoder;

[0008] averaging and pooling the plurality of video frame features to obtain average pooling features, and concatenating the average pooling features with the plurality of video frame features respectively to obtain a plurality of final frame features;

[0009] adding a marker to the plurality of final frame features online, the marker being a randomly generated feature vector, using time attention to interact all markers, concatenating the plurality of final frame features with the corresponding time attention processed markers respectively, and interacting the plurality of final frame features concatenated with the markers through fusion attention to obtain a plurality of time frame features;

[0010] inputting the plurality of time frame features after discarding the markers into a first Transformer encoder-decoder module to predict and generate a first caption.

[0011] Preferably, after obtaining the plurality of time frame features, the method further comprises:

[0012] inputting the plurality of time frame features after discarding the markers into a multi-layer feedforward neural network to strengthen feature representation.

[0013] Preferably, after inputting the plurality of time frame features after discarding the markers into the first Transformer encoder-decoder module to predict and generate the first caption, the method further comprises:

[0014] converting the generated first caption into a plurality of word features using a text encoder, concatenating the plurality of word features with the plurality of time frame features, and inputting the result into a second Transformer encoder-decoder module to predict and generate a second caption.

[0015] Preferably, before converting the generated first caption into a plurality of word features using the text encoder and inputting the result into the second Transformer encoder-decoder module to predict and generate the second caption, the method further comprises:

[0016] adding a marker to the first caption online, the marker being a randomly generated feature vector, and obtaining a confidence score through a multi-layer feedforward neural network for the added marker;

[0017] if the first confidence score is not lower than a preset threshold, taking the first caption as a final prediction result;

[0018] If the first confidence score is lower than the preset threshold, continue to predict the second subtitle, and take the second subtitle as a final prediction result.

[0019] Preferably, the training method of the video description generation model comprises:

[0020] A training set is constructed, and one training sample in the training set is one video segment and its corresponding multiple sentences of annotated text;

[0021] The video description generation model is trained using the training set, and the model parameters are adjusted until the total loss converges;

[0022] For any training sample, the calculation process of the total loss is as follows:

[0023] The video segment of the training sample is input into a pre-trained video encoder to extract multiple video frame training features;

[0024] The multiple sentences of annotated text of the training sample are input into a text encoder to extract multiple word training features and multiple sentence training features;

[0025] A first contrastive loss is calculated according to the multiple video frame training features and the multiple word training features;

[0026] The multiple video frame training features are average-pooled to obtain average-pooled training features, and the average-pooled training features are respectively concatenated with the multiple video frame training features to obtain multiple final frame training features;

[0027] The multiple final frame training features are average-pooled to obtain a global video training feature;

[0028] A second contrastive loss is calculated according to the multiple sentence training features and the global video training feature;

[0029] A symmetric cross-entropy loss of the video description generation model is calculated according to the training sample;

[0030] A total loss is calculated according to the first contrastive loss, the second contrastive loss, and the symmetric cross-entropy loss.

[0031] Preferably, the first contrastive loss L f-ω is a function of:

[0032]

[0033]

[0034]

[0035] where s(fj , ω i ) is a similarity function between word item ω i and frame feature item f j , s(f i , ω i ) is a similarity function between word item ω and frame feature item f i , s(f i , ω j ) is a similarity function between word item ω j and frame feature item f j , N is the number of words in a sentence, L f2ω and L ω2f represent frame-to-word and word-to-frame loss terms.

[0036] Preferably, the function of the second contrastive loss L v-t is:

[0037]

[0038]

[0039]

[0040] where s(v j , t i ) is a similarity function between text item t i and video item v j , s(v i , t i ) is a similarity function between text item t i and video item v i , s(v i , t j ) is a similarity function between text item t j and video item v i , and B is the number of sentences in a batch, L v2t and L t2v represent video-to-text and text-to-video loss terms.

[0041] Preferably, the function of the symmetric cross-entropy loss L sl is:

[0042] L sl = λ1L ce + (1- λ1)L rce

[0043]

[0044]

[0045] where P and Q are the predicted and actual outputs, respectively, and L ce is the normal cross-entropy loss, L rce is the label-swapped cross-entropy loss, and λ1 is a hyperparameter.

[0046] The application also provides a video description generation device, comprising:

[0047] an input module configured to input a video segment into a trained video description generation model;

[0048] a video frame feature extraction module configured to extract a plurality of video frame features using a pre-trained video encoder;

[0049] a final frame feature acquisition module configured to perform average pooling on the plurality of video frame features to obtain average pooling features, and concatenate the average pooling features with the plurality of video frame features, respectively, to obtain a plurality of final frame features;

[0050] a time converter module configured to add a mark to the plurality of final frame features online, the mark being a randomly generated feature vector, to interact with all marks using time attention, to concatenate the plurality of final frame features with corresponding marks processed by time attention, respectively, and to interact with the plurality of final frame features concatenated with the marks through fusion attention to obtain a plurality of time frame features;

[0051] a subtitle prediction module configured to input the plurality of time frame features after discarding the marks into a first Transformer encoder-decoder module to predict and generate a first subtitle.

[0052] The application also provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the above-mentioned video description generation method.

[0053] The above technical solutions of the application have the following advantages over the prior art:

[0054] In order to realize cross-frame information exchange, the video description generation method provided in the application proposes time attention and fusion attention, and adds a mark to each frame feature online to represent global features, and the marks are interacted to realize information transmission and interaction between frames, first, all the marks are interacted using time attention, the time relationship between the video frames of the input video is learned, next, the mark processed by the time attention is concatenated with each feature vector in the final feature, and then input to the fusion attention, the fusion attention makes all the features interact, so that the feature output after the fusion attention contains the time relationship between the video frames; the application combines the spatial features and time sequence information obtained by the pre-training model to capture the time clues between the video frames, and finally can obtain more sufficient feature representation, and improve the prediction accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to make the content of the application more easily understood, the application will be further described in detail below according to the specific embodiments of the application and in conjunction with the drawings, in which:

[0056] Figure 1 The implementation flowchart of the video description generation method provided by the application;

[0057] Figure 2 The network structure diagram of the video description generation model;

[0058] Figure 3 The implementation flowchart of the time frame feature acquisition;

[0059] Figure 4 The flowchart for calculating the first contrast loss and the second contrast loss;

[0060] Figure 5 The structure block diagram of the video description generation device provided by the embodiment of the application. DETAILED DESCRIPTION

[0061] The core of the application is to provide a video description generation method, device and computer storage medium, which can obtain more sufficient feature representation and improve the prediction accuracy.

[0062] In order to make the person in the technical field better understand the application scheme, the application will be further described in detail below in conjunction with the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by the person skilled in the art without creative labor belong to the protection scope of the application.

[0063] Please refer to Figure 1 and Figure 2 , Figure 1The implementation flowchart of the video description generation method provided by the application, Figure 2 The network structure diagram of the video description generation model; the specific operation steps are as follows:

[0064] S101: inputting a video segment into a trained video description generation model;

[0065] S102: extracting a plurality of video frame features by using a pre-trained video encoder;

[0066] In an embodiment, we first extract video features through the pre-trained video encoder of CLIP4Clip, extract 12 frames from each video, and each frame is a 512-dimensional vector feature.

[0067] S103: performing average pooling on the plurality of video frame features to obtain average pooling features, and concatenating the average pooling features with the plurality of video frame features respectively to obtain a plurality of final frame features;

[0068] The video features extracted from the CLIP4Clip model are subjected to average pooling to obtain average pooling features, which are then concatenated with the previous 12 frame features to obtain final frame features.

[0069] As Figure 3 shown:

[0070] S104: adding an online mark to the plurality of final frame features, the mark being a randomly generated feature vector, using time attention to interact all the marks, concatenating the plurality of final frame features with the corresponding time attention processed marks respectively, and interacting the plurality of final frame features concatenated with the marks through fusion attention to obtain a plurality of time frame features;

[0071] The plurality of time frame features after discarding the marks are input into a multi-layer feedforward neural network to strengthen the feature representation, and the added marks are discarded before passing through the multi-layer feedforward neural network, because the marks are generated online and are only used for interaction between video frames in the time module, and are invalid in the subsequent modules.

[0072] S105: inputting the plurality of time frame features after discarding the marks into a first Transformer encoder-decoder module to predict and generate a first caption.

[0073] The generated first caption is converted into a plurality of word features by using a CLIP text encoder, and the plurality of word features are concatenated with the plurality of time frame features and input into a second Transformer encoder-decoder module to predict and generate a second caption.

[0074] Since the second-level network for generating the second subtitle adds an encoder-decoder structure of a Transformer to the first-level network for generating the first subtitle, the computational load is greatly improved, therefore, we design an adaptive gate module after the encoder-decoder of the Transformer to reduce the computational load of the model, as follows:

[0075] adding a mark to the first subtitle, the mark being a randomly generated feature vector, passing the added mark through a multi-layer feedforward neural network to obtain a confidence score;

[0076] If the first confidence score is not lower than a preset threshold T [0, 1], it indicates that the predicted subtitle result is good, and the first subtitle is taken as the final prediction result;

[0077] If the first confidence score is lower than the preset threshold, it indicates that the predicted subtitle result of the first-level network is not good and needs to be processed by the second-level network, then the second subtitle is predicted and taken as the final prediction result.

[0078] In order to realize cross-frame information exchange, we propose time attention and fusion attention, and add a mark to each frame feature online to represent the global feature, and let these marks interact with each other to transfer and interact information between frames. First, all the marks are interacted using time attention to learn the time relationship between the video frames of the input video. Next, the marks processed by the time attention are concatenated with each feature vector in the final feature, and then input to the fusion attention. The fusion attention makes all the features interact with each other, so that the output features after the fusion attention contain the time relationship between the video frames. The pre-trained model combines spatial features and time sequence information to capture the time clues between video frames, and finally obtains more sufficient feature representation, improving the prediction accuracy. The adaptive two-layer network structure saves the good subtitle prediction effect through the adaptive gate with a certain threshold, and the bad subtitle prediction result enters the second layer network for further prediction. Finally, more accurate subtitles are obtained while reducing the computational load.

[0079] As shown in Figure 4 , based on the above embodiment, the training method of the video description generation model is described:

[0080] A training set is constructed, and a training sample in the training set is a video segment and its corresponding multiple annotated texts;

[0081] The video description generation model is trained using the training set, and the model parameters are adjusted until the total loss converges;

[0082] wherein, for any training sample, the total loss is calculated as follows:

[0083] inputting the video clip of the training sample into the pre-trained video encoder to extract a plurality of video frame training features;

[0084] inputting the multiple sentences of labeled text of the training sample into the text encoder to extract a plurality of word training features and a plurality of sentence training features;

[0085] calculating a first contrastive loss L f-ω from the plurality of video frame training features and the plurality of word training features:

[0086]

[0087]

[0088]

[0089] wherein, s(f j ,ω i ) is a similarity function between the word item ω i and the frame feature item f j , s(f i ,ω i ) is a similarity function between the word item ω i and the frame feature item f i , s(f i ,ω j ) is a similarity function between the word item ω j and the frame feature item f j , N is the number of words in a sentence, L f2ω and L ω2f represent frame-to-word and word-to-frame loss terms;

[0090] performing average pooling on the plurality of video frame training features to obtain an average-pooled training feature, and concatenating the average-pooled training feature with the plurality of video frame training features respectively to obtain a plurality of final frame training features;

[0091] performing average pooling on the plurality of final frame training features to obtain a global video training feature;

[0092] calculating a second contrastive loss L v-t from the plurality of sentence training features and the global video training feature:

[0093] The function of the second contrastive loss is L v-t :

[0094]

[0095]

[0096]

[0097] where s(v j ,t i ) is the similarity function between text item t i and video item v j , s(v i ,t i ) is the similarity function between text item t i and video item v i , s(v i ,t j ) is the similarity function between text item t j and video item v i , and B is the number of sentences in a batch, L v2t and L t2v denote the video-to-text and text-to-video loss terms, respectively.

[0098] The symmetric cross-entropy loss L sl of the video description generation model is calculated according to the training sample:

[0099] L sl = λ1L ce + (1- λ1)L rce

[0100]

[0101]

[0102] Since the extracted video features are usually noisy and ambiguous, we use the symmetric cross-entropy (SCE) loss instead of the original cross-entropy (CE) loss, and use a regularization term to mitigate the overfitting and adversarial noise problems. We use the SCE loss to relax their strict binary labels. To smooth the strict temporal constraint cross-entropy, we slightly decrease the label value from 1 and increase the other type of value from 0. The SCE loss is a combination of the cross-entropy (CE) loss and the reverse cross-entropy (RCE) loss. Where P and Q are the predicted output and the actual output, respectively, L ce is the normal cross-entropy loss, L rce is the cross-entropy loss with the labels exchanged, and λ1 is a hyperparameter.

[0103] The total loss L fl is calculated according to the first contrastive loss, the second contrastive loss, and the symmetric cross-entropy loss: sl + λ3L v-t + λ4L f-ωwhere λ2, λ3, λ4are hyperparameters.

[0104] Since most of the existing methods are difficult to make up for the huge difference between the video appearance and the language clues, in order to solve the inconsistency between the video appearance and the language clues in the video generated caption, the present application compares the calculation of the contrast loss between the sentence features and the global video features, the word features and the frame features, and finally maximizes the similarity between the paired video-text and minimizes the similarity between the non-paired video-text, so as to realize the effect of aligning the video features and the text features at multiple granularities, provide the text-related visual features for accurate video caption, and make up for the gap between the video appearance and the language.

[0105] Based on the above embodiments, the present embodiment verifies the effect of our model on the MSVD and MSR-VTT data sets, as follows:

[0106] The MSVD data set consists of 1970 YouTube video clips, each of which is between 10 and 25 seconds long, and is annotated with about 40 English sentences. The MSR-VTT data set is a large data set of open-domain video captions. It contains 10000 video clips, each of which has about 20 English annotations. The commonly used evaluation indexes of MSVD and MSR-VTT are BLUE@4, METEOR, ROUGE-L and CIDEr.

[0107] No matter whether each module is added continuously on the MSVD or the MSR-VTT test set, the experimental performance is improved, and the effect is best when all the modules are added. It can be seen that, compared with the original network, the adaptive two-level network has greatly improved in performance, and is the best method on the MSVD and MSR-VTT benchmarks. First, we compare it with 14 kinds of most advanced methods, and find that on the two data sets, the proposed method almost defeats other methods in all evaluation indexes. In addition, our method does not add extra target features obtained by a target detection model. These all verify the advantages and superiority of the proposed method over the existing methods.

[0108] Then, the corresponding ablation analysis (a total of eight groups) after adding each module (comparison module, time module and adaptive module) on the test set of MSVD and MSR-VTT is carried out, and it can be seen from Table 1 and Table 2 that when each module is added continuously, the experimental performance is improved, and the effect is best when all the modules are added:

[0109] Table 1 results of ablation experiments of each module on the test set of MSVD

[0110]

[0111] Table 2 results of ablation experiments on each module on the test set of MSR-VTT

[0112]

[0113] Finally, analyze the change of model calculation amount after adding each module, from Table 3, it can be seen that with the increase of modules, the calculation amount of the model gradually increases, and compared with the two-level network, the calculation burden of the adaptive two-level network is reduced by about 12%. At the same time, the size of the model is almost unchanged.

[0114] Table 3 analysis results of model calculation amount after adding each module

[0115]

[0116] Please refer to Figure 5 , Figure 5 The structure block diagram of a video description generation device provided by the embodiment of the application; the specific device can include:

[0117] The input module 100 is used for inputting the video segment into the trained video description generation model;

[0118] The video frame feature extraction module 200 is used for extracting a plurality of video frame features by using a pre-trained video encoder;

[0119] The final frame feature acquisition module 300 is used for performing average pooling on the plurality of video frame features to obtain average pooling features, and performing concatenation of the average pooling features and the plurality of video frame features respectively to obtain a plurality of final frame features;

[0120] The time converter module 400 is used for online adding a mark to the plurality of final frame features, the mark being a randomly generated feature vector, using time attention to interact all marks, concatenating the plurality of final frame features and corresponding time attention processed marks respectively, and interacting the plurality of final frame features concatenated with the marks through fusion attention to obtain a plurality of time frame features;

[0121] The subtitle prediction module 500 is used for inputting the plurality of time frame features after discarding the marks into a first Transformer encoder-decoder module to predict and generate a first subtitle.

[0122] The video description generation apparatus of the embodiment is used to implement the foregoing video description generation method, and therefore the specific implementation of the video description generation apparatus can be seen from the foregoing embodiment part of the video description generation method. For example, the input module 100, the video frame feature extraction module 200, the final frame feature acquisition module 300, the time converter module 400, and the subtitle prediction module 500 are respectively used to implement the steps S101, S102, S103, S104, and S105 in the foregoing video description generation method, and therefore the specific implementation can be referred to the description of the corresponding respective part embodiments, which will not be described herein again.

[0123] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the foregoing video description generation method.

[0124] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product implemented on one or more computer readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes.

[0125] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0126] These computer program instructions can also be stored in a computer readable memory that can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0127] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide steps for implementing the function specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0128] Obviously, the above embodiments are only examples for clearly illustrating the present application, and are not intended to limit the embodiments. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the embodiments are not required to be exhausted, and the obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A video description generation method, characterized in that, include: Input the video clips into the trained video description generation model; The training method for the video description generation model includes: constructing a training set, where a training sample in the training set consists of a video clip and its corresponding multi-sentence annotated text; training the video description generation model using the training set, adjusting the model parameters until the total loss converges; wherein, for any training sample, the calculation process of the total loss is as follows: inputting the video clip of the training sample into a pre-trained video encoder to extract multiple video frame training features; inputting the multi-sentence annotated text of the training sample into a text encoder to extract multiple word training features and multiple sentence training features; calculating a first contrastive loss based on the multiple video frame training features and multiple word training features; performing average pooling on the multiple video frame training features to obtain average pooled training features, and concatenating the average pooled training features with the multiple video frame training features to obtain multiple final frame training features; performing average pooling on the multiple final frame training features to obtain global video training features; calculating a second contrastive loss based on the multiple sentence training features and global video training features; calculating the symmetric cross-entropy loss of the video description generation model based on the training sample; and calculating the total loss based on the first contrastive loss, the second contrastive loss, and the symmetric cross-entropy loss. First comparison loss The function is defined as: ； ； ； in, It is a word item and frame feature terms Similarity function between them It is a word item and frame feature terms Similarity function between them It is a word item and frame feature terms Similarity function between them It is the number of words in a sentence. and This represents the loss terms for frame-to-word and word-to-frame. Second contrast loss The function is defined as: ； ； ； in, It is a text item and video items Similarity function between them It is a text item and video items Similarity function between them It is a text item and video items The similarity function between them, and It is the number of sentences in a batch. and This represents the loss terms for video-to-text and text-to-video conversions. Extract features from multiple video frames using a pre-trained video encoder; Average pooling is performed on the multiple video frame features to obtain average pooled features, and the average pooled features are concatenated with the multiple video frame features to obtain multiple final frame features; A label is added online to the multiple final frame features. The label is a randomly generated feature vector. All labels are interacted using temporal attention. The multiple final frame features are concatenated with the corresponding temporally attention-processed labels. The multiple final frame features concatenated with the labels are then interacted with through fusion attention to obtain multiple temporal frame features. The features of multiple time frames after discarding the markers are input into the first Transformer encoder-decoder module to predict and generate the first subtitle.

2. The video description generation method according to claim 1, characterized in that, After obtaining multiple time frame features, the process also includes: Features from multiple time frames after the labels have been discarded are input into a multilayer feedforward neural network to enhance feature representation.

3. The video description generation method according to claim 1, characterized in that, The step of inputting the features of multiple time frames after discarding the markers into the first Transformer encoder-decoder module to predict and generate the first subtitle also includes: The generated first subtitle is converted into multiple word features using a text encoder, and then concatenated with the multiple time frame features before being input into a second Transformer encoder-decoder module to predict and generate a second subtitle.

4. The video description generation method according to claim 3, characterized in that, The process of converting the generated first subtitle into multiple word features using a text encoder, concatenating them with the multiple time frame features, and then inputting them into a second Transformer encoder-decoder module to predict and generate the second subtitle includes: A tag is added online to the first subtitle. The tag is a randomly generated feature vector. The added tag is then passed through a multi-layer feedforward neural network to obtain a confidence score. If the confidence score is not lower than the preset threshold, the first subtitle will be used as the final prediction result; If the confidence score is lower than the preset threshold, the prediction of the second subtitle continues, and the second subtitle is taken as the final prediction result.

5. The video description generation method according to claim 1, characterized in that, The symmetric cross-entropy loss The function is defined as: ； ； ； in, and These are the predicted output and the actual output, respectively. This is normal cross-entropy loss. It is the cross-entropy loss with swapped labels. It's a hyperparameter.

6. A video description generation apparatus, characterized in that, include: The input module is used to input video clips into the trained video description generation model; The training method for the video description generation model includes: constructing a training set, where a training sample in the training set consists of a video clip and its corresponding multi-sentence annotated text; training the video description generation model using the training set, adjusting the model parameters until the total loss converges; wherein, for any training sample, the calculation process of the total loss is as follows: inputting the video clip of the training sample into a pre-trained video encoder to extract multiple video frame training features; inputting the multi-sentence annotated text of the training sample into a text encoder to extract multiple word training features and multiple sentence training features; calculating a first contrastive loss based on the multiple video frame training features and multiple word training features; performing average pooling on the multiple video frame training features to obtain average pooled training features, and concatenating the average pooled training features with the multiple video frame training features to obtain multiple final frame training features; performing average pooling on the multiple final frame training features to obtain global video training features; calculating a second contrastive loss based on the multiple sentence training features and global video training features; calculating the symmetric cross-entropy loss of the video description generation model based on the training sample; and calculating the total loss based on the first contrastive loss, the second contrastive loss, and the symmetric cross-entropy loss. First comparison loss The function is defined as: ； ； ； in, It is a word item and frame feature terms Similarity function between them It is a word item and frame feature terms Similarity function between them It is a word item and frame feature terms Similarity function between them It is the number of words in a sentence. and This represents the loss terms for frame-to-word and word-to-frame. Second contrast loss The function is defined as: ； ； ； in, It is a text item and video items Similarity function between them It is a text item and video items Similarity function between them It is a text item and video items The similarity function between them, and It is the number of sentences in a batch. and This represents the loss terms for video-to-text and text-to-video conversions. The video frame feature extraction module is used to extract features from multiple video frames using a pre-trained video encoder. The final frame feature acquisition module is used to perform average pooling on the multiple video frame features to obtain average pooled features, and to concatenate the average pooled features with the multiple video frame features to obtain multiple final frame features. The time converter module is used to add a label to the multiple final frame features online. The label is a randomly generated feature vector. All labels are interacted with using time attention. The multiple final frame features are concatenated with the corresponding time attention-processed labels respectively. The multiple final frame features concatenated with the labels are interacted with through fusion attention to obtain multiple time frame features. The subtitle prediction module is used to input the features of multiple time frames after discarding the markers into the first Transformer encoder-decoder module to predict and generate the first subtitle.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the video description generation method as described in any one of claims 1 to 5.

Images