A multimodal inference and iterative optimization video description generation model and method

Abstract

This invention provides a multimodal reasoning and iterative optimization method for video description generation, comprising the following steps: obtaining target features, appearance features, and motion features based on sampled video frames; calculating reasoning features for three types of words using the three features and context-guided features G; selecting one of these features as an input to the basic decoder; using the hidden state output by the basic decoder as an input to the iterator; calculating the probability distribution of the descriptive words; and generating descriptive sentences. The invention also includes a video description generation model, comprising an encoder and a decoder, wherein the decoder comprises a basic decoder and an iterator connected sequentially, the basic decoder and the iterator being constructed from an LSTM network. The fully dynamic context-guided features and iterator of this invention significantly improve the quality of descriptive sentences, enabling the model to generate more accurate, detailed, comprehensive, and expressive descriptive sentences for video data.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a multimodal reasoning and iterative optimization video description generation model and method. Background Technology

[0002] Video is one of the most important media for recording, presenting, and disseminating information in the internet age, profoundly impacting all aspects of human society's production and life. How to quickly and accurately extract valuable information from large-scale video data and achieve efficient video analysis and understanding has become a critical problem urgently needing to be solved in the field of artificial intelligence. As an important research task in video analysis and understanding, video captioning mainly refers to generating natural language descriptions for corresponding videos using models and computation as input. It has already been applied in fields such as automatic narration, navigation assistance, and human-computer collaboration, and has significant implications for promoting the development of medicine, education, and computer science itself. Video captioning is an extremely challenging task. It not only requires detecting static elements such as objects, scenes, and people in the spatial domain, but also recognizing dynamic elements such as actions and events in the temporal domain, perceiving the dynamic interaction relationships and spatiotemporal evolution between various visual elements, and generating appropriate descriptive vocabulary through suitable language models, ultimately forming accurate, coherent descriptive sentences that conform to people's expression habits. Because it involves information from two different modalities—visual and linguistic—video captioning has attracted the joint attention of researchers in both computer vision and natural language processing, becoming a recent hot research topic.

[0003] In recent years, researchers have made significant progress in designing and implementing various video description generation methods based on the classic encoder-decoder framework in sequence learning. Generally, an encoder composed of Convolutional Neural Networks (CNNs) extracts various visual features from the video input. These extracted visual features are then fused or simply concatenated using an attention mechanism before being input into a decoder composed of Recurrent Neural Networks (RNNs) to generate descriptive sentences, thus converting the video frame image input sequence into a language text output sequence. Although the attention mechanism allows the model to allocate different levels of attention to visual features when generating descriptive words, this still has certain limitations. On the one hand, the model directly inputs the visual features extracted by the encoder into the decoder without any differentiation, failing to perform necessary visual reasoning on these visual features during the encoding stage. On the other hand, the decoding network neglects further optimization of visual features during the decoding process. Visual reasoning methods are commonly found in video question answering and visual grounding tasks. They achieve step-by-step reasoning by designing discretely combinable neural network modules with different structures to form specific reasoning structures. This method has recently been applied to video description generation tasks. However, such methods still suffer from a critical problem in visual reasoning: the encoder's visual reasoning process lacks effective guidance, and the decoder does not consider further optimization of the visual reasoning results. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a multimodal reasoning and iterative optimization video description generation model and method. The video description generation model consists of two main parts: an encoder with reasoning capabilities and a decoder with iterative optimization capabilities, generating more accurate, comprehensive, detailed and diverse video description statements for the video data input to the model.

[0005] The technical solution adopted in this invention is: a multimodal reasoning and iterative optimization video description generation method, comprising the following steps:

[0006] Step 1: Sample the input video at equal intervals to obtain a sequence of video frame images;

[0007] Step 2: Input the video frame image sequence into the feature inference module to calculate the target feature Vo, appearance feature Va, and motion feature Vm;

[0008] Step 3: Input the target feature Vo, appearance feature Va, motion feature Vm, and context-guided feature G into the visual reasoning module to calculate the reasoning feature V.t L V t R and V t F V t L Used to generate nouns and adjectives, V t R Used to generate verbs, V t F Used to generate functional words and other parts of speech.

[0009] The formula for calculating the context-guided feature G is:

[0010]

[0011] Where [;] denotes the concatenation operation, E[·] denotes the word embedding matrix, and y t-1 This refers to the descriptive words generated at time t-1. This represents the hidden state of the basic decoder. Indicates the hidden state of the iterator;

[0012] Step 4: Infer the inference feature V t L V t R and V t F The context-guided feature G is input into the score function Score(·,·) to calculate V. t L V t R and V t F The inference feature with the highest score is selected as the initial visual inference feature V. t ;

[0013]

[0014] in, This represents the calculated fraction. The inference features are represented by *∈{L, R, F}, fc(·) represents a fully connected layer, and tanh(·) represents an activation function.

[0015] Step 5: Calculate the initial visual inference features V at time t. t The hidden state of the iterator at time t-1 And the descriptive word y generated at time t-1 t-1 Input the base decoder to obtain the hidden state output by the base decoder at time t. and cell state The basic decoder is composed of an LSTM network;

[0016] Step 6: Calculate the initial visual inference features V at time t. t The hidden state of the basic decoder at time t And the descriptive word y generated at time t-1 t-1 Input an iterator and obtain the hidden state output by the iterator at time t. and cell state The iteration results are used to generate the descriptive words for the current time step and determine the descriptive words for that time step. The iterator is composed of an LSTM network.

[0017] Step 7: Repeat steps 3-6 until the end marker of the output description statement or the maximum length of the specified description statement is reached.

[0018] Furthermore, in step 1, sampling is performed every 9-12 frames.

[0019] Furthermore, in step 2, the target feature Vo is formed by splicing the target location information and the target region features.

[0020] Furthermore, in step 3, the additive attention calculation process is shown in the following formula:

[0021] Additive_Attention(V, q)=soffmax(score(V, q))V

[0022] =soffmax(w T tanh(WV+U1 T q))V

[0023] Where V and q represent the attention value and query, respectively, w, W and U are learnable parameters, and 1 represents a vector of all 1s;

[0024] The calculation process for spatial attention SA(V,q) and temporal attention TA(V,q) is the same as that for additive attention.

[0025] V t L =LOC(V a V o ,G)=TA([SA(V o ,G);V a ],G);

[0026] V t R =REL(V m V o ,G)=TA(Pair[Cate i ;Cate j],G),

[0027] Among them, Pair[Cate i ;Cate j This indicates the operation of concatenating and pairing all previously obtained feature concatenation results, Cate, together. Cate = [SA(V o ,G);V m ];

[0028]

[0029] in, This represents the historical cell state sequence of the basic decoder. This represents the historical cell state sequence of the iterator.

[0030] Furthermore, in step 4,

[0031] in, S represents the inner product operation. t This indicates a one-hot selected vector, and... The maximum value corresponds to 1, while the other two values ​​are 0.

[0032] Furthermore, in step 6, the descriptive word y at time t is generated using the softmax function. t probability distribution:

[0033]

[0034] The beam search method is used based on the description word y. t The probability distribution determines the descriptive word for that time step.

[0035] The technical solution adopted in this invention is: a video description generation model using the above-mentioned multimodal reasoning and iterative optimization video description generation method, comprising an encoder part and a decoder part.

[0036] The encoder section includes a feature inference module, a visual inference module, and a module selector connected in sequence; the decoder section includes a basic decoder and an iterator connected in sequence.

[0037] The feature inference module is divided into an appearance feature inference module, a motion feature inference module, and a target feature inference module. The appearance feature inference module is used to calculate appearance feature Va based on the video frame image sequence. The motion feature inference module is used to calculate motion feature Vm based on the video frame image sequence. The target feature inference module is used to calculate target feature Vo based on the video frame image sequence.

[0038] The visual reasoning module is divided into an LOC visual reasoning module, a REL visual reasoning module, and a FUN visual reasoning module. The LOC visual reasoning module is used to calculate the reasoning feature V based on the target feature Vo, the appearance feature Va, and the contextual guidance feature G. t L The REL visual reasoning module is used to calculate the reasoning feature V based on the target feature Vo, motion feature Vm, and contextual guidance feature G. t R The FUN visual reasoning module is used to determine the historical cell state sequence based on the basic decoder. Iterator's historical cell state sequence The inference feature V is calculated from the context-guided feature G. t F ,

[0039] The module selector is used to select inference feature V. t L V t R and V t F One of them is used as the initial visual reasoning feature V t The input is fed into the underlying decoder.

[0040] Furthermore, the appearance feature inference module is a two-dimensional convolutional neural network using the InceptionResNetV2 network, the motion feature inference module is a three-dimensional convolutional neural network using the I3D motion feature encoding network, and the appearance feature inference module uses the Faster R-CNN object detection network.

[0041] Furthermore, both the basic decoder and the iterator are composed of LSTM networks.

[0042] Furthermore, the total loss function L of the model is:

[0043] L = L ce +λL kl

[0044] Where λ is the proportional parameter balancing the two losses, L ce L represents the cross-entropy loss. kl This represents the KL divergence loss.

[0045] Working principle: The decoder iterator iterates and optimizes repeatedly: The initial visual inference result of the network visual inference module at the encoder end is input to the basic decoder and the iterator in a way similar to residual connection. The iterator optimizes the initial visual inference result again according to the hidden state of the basic decoder at the current time, and uses the optimized visual features together with the hidden states of the basic decoder and the iterator to decode and generate a higher quality description statement.

[0046] Dynamic context-guided multimodal visual reasoning at the encoder end: The richer and more precise contextual information generated by the decoding network after introducing iterators constitutes a fully dynamic context-guided feature. This feature serves as the basis for querying the attention mechanism and selecting modules in the visual reasoning module of the encoder end network, guiding the visual reasoning process and making the model's visual reasoning process more accurate. This achieves the goal of multimodal visual reasoning that fuses contextual textual semantic information and visual information. Finally, the model alternates between dynamically context-guided multimodal visual reasoning and iterative optimization of the initial visual reasoning results, mutually promoting each other and jointly enhancing the model's reasoning ability.

[0047] Compared with the prior art, the beneficial effects of this invention are:

[0048] 1. This invention proposes a fully dynamic context-guided feature G. Compared with using static global visual features after average pooling as guide features, the context-guided feature G not only contains richer content and more accurate semantic information, but also can be dynamically adjusted and updated over time, thereby improving the accuracy of the model's visual reasoning process and further enhancing the model's reasoning ability.

[0049] 2. The decoder part of the present invention adopts a basic decoder and an iterator. The initial visual inference result of the network visual inference module at the encoding end is input to the basic decoder and the iterator in a manner similar to residual connection. The iterator optimizes the initial visual inference result again according to the hidden state of the basic decoder at the current moment, and uses the optimized visual features together with the hidden states of the basic decoder and the iterator to decode and generate higher quality description statements.

[0050] 3. The fully dynamic context-guided features and iterators employed in this invention significantly improve the quality of descriptive statements. The model is able to generate more accurate, detailed, comprehensive, and expressive descriptive statements for video data, even better than the reference descriptive statements. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the video description generation model according to an embodiment of the present invention;

[0052] Figure 2This is a schematic diagram of the LSTM structure according to an embodiment of the present invention;

[0053] Figure 3 This is a flowchart of a method according to an embodiment of the present invention;

[0054] Figure 4 This is a visual example of generating description statements according to an embodiment of the present invention. Detailed Implementation

[0055] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Embodiments of the present invention provide a multimodal reasoning and iterative optimization video description generation model, such as... Figure 1 As shown, the design is based on an encoder-decoder framework, which includes an encoder part and a decoder part. The encoder part includes a feature inference module, a visual inference module, and a module selector connected in sequence, and the decoder part includes a base decoder and an iterator connected in sequence.

[0057] The feature reasoning module is divided into an appearance feature reasoning module, a motion feature reasoning module, and a target feature reasoning module.

[0058] The appearance feature inference module, motion feature inference module, and target feature inference module calculate appearance feature Va, motion feature Vm, and target feature Vo based on the video frame image sequence, respectively. The appearance feature inference module employs a two-dimensional convolutional neural network; specifically, in this embodiment, the InceptionResNetV2 network is used. The motion feature inference module employs a three-dimensional convolutional neural network; specifically, in this embodiment, the I3D motion feature encoding network is used. The appearance feature inference module uses the Faster R-CNN target detection network.

[0059] The visual reasoning module is divided into the LOC visual reasoning module, the REL visual reasoning module, and the FUN visual reasoning module. The LOC visual reasoning module is used to calculate the reasoning feature V based on the target feature Vo, the appearance feature Va, and the fully dynamic context-guided feature G. t L The REL visual reasoning module is used to calculate the reasoning feature V based on the target feature Vo, the motion feature Vm, and the fully dynamic context-guided feature G. t R The FUN visual reasoning module is used to determine the historical cell state sequence based on the basic decoder. Iterator's historical cell state sequence The inference feature V is calculated from the fully dynamic context-guided feature G. tF .

[0060] The formula for calculating the context-guided feature G is:

[0061]

[0062] Where [;] denotes the concatenation operation, E[·] denotes the word embedding matrix, and y t-1 This refers to the descriptive words generated at time t-1. This represents the hidden state of the basic decoder. This represents the hidden state of the iterator.

[0063] The module selector is used to select inference feature V t L V t R and V t F Calculate scores for each context-guided feature G, and select the inference feature with the highest score as the initial visual inference feature V. t The input is fed into the basic decoder.

[0064] Both the basic decoder and iterator are composed of LSTM networks, with the following structure: Figure 2 As shown, at time t, its network input is x. t The hidden state is h t Memory cell state C t The hidden state h at time t-1 t-1 and memory cell state C t-1 The information status transmission process is shown in the following formula:

[0065] i T =σ(W xi x t +W hi h t-1 +b i )

[0066] f t =σ(W xf x t +W hf h t-1 +b f )

[0067] c t =f t ⊙c t-1 +i t ⊙tanh(W xc x t +W hc h t-1 +b c)

[0068] o t =σ(W xo x t +W ho h t-1 +b o )

[0069] h t =o t ⊙tanh(c t )

[0070] Where σ represents the sigmoid function, ⊙ represents element-wise multiplication, and tanh is the activation function. Input gate i t Controlling the transition to memory cell state C t Information added to the middle. Forget Gate f t Controlling the memory cell state C from time t-1 t-1 The information removed can effectively prevent the network from developing vanishing or exploding gradient problems. Output gate o t Control from memory cell state C t The information output.

[0071] The hidden state output by the basic decoder is used as the input to the iterator.

[0072] This embodiment uses two loss functions, cross-entropy loss and KL divergence loss, to train the proposed model end-to-end.

[0073] Cross-entropy loss is often used to measure the accuracy of the descriptive words predicted by a video description generation model. If the video data training set D contains N video data samples, and the reference descriptive statement corresponding to video data sample i... As shown in the following formula:

[0074]

[0075] Where, i∈{1,2,…,N}, l i Let L represent the length of the reference description statement corresponding to video data sample i. Then, the cross-entropy loss L for the entire video data training set D is... ce The calculation can be performed using the following formula:

[0076]

[0077] Among them, y it The model represents the descriptive words predicted and generated for video data sample i at time t. The reference descriptive word represents video data sample i at time t.

[0078] To ensure the model accurately selects the visual inference module when generating descriptive words with different parts of speech, and to guarantee that the three visual inference modules can specifically infer the corresponding features required by the model, as shown in the following formula, this embodiment uses the Spacy Tagging Tool to generate the true descriptive statement for video data sample i. Part-of-speech tags Supervised model inference process.

[0079]

[0080] This embodiment uses the KL divergence loss L kl Force one-hot selection of vector S it Part-of-speech tags with one-hot encoding The distribution of visual features is made as close as possible to enhance their semantic relevance and ensure the correctness of the generated descriptive statement's grammatical structure. The calculation method is shown in the following formula:

[0081]

[0082] Combining the two types of losses mentioned above, the total loss function L of the model is:

[0083] L = L ce +λL kl

[0084] Where λ is the proportional parameter balancing the two losses, which is set to 0.1 in this embodiment. ce L represents the cross-entropy loss. kl This represents the KL divergence loss.

[0085] This embodiment also provides a multimodal reasoning and iterative optimization video description generation method based on the above model, such as... Figure 3 As shown, it includes the following steps:

[0086] Step 1: Sample the input video at equal intervals, once every 9-12 frames, to obtain a video frame image sequence. The frame rate of video data is generally maintained at 25 or 30 frames per second. Video data has highly redundant characteristics; often, two or even more consecutive video frames are extremely similar, containing virtually no change in visual content. Therefore, it is necessary to sample the video data.

[0087] Step 2: Input the video frame image sequence into the pre-trained appearance feature inference module, motion feature inference module, and target feature inference module respectively, and calculate the appearance feature Va, motion feature Vm, and target feature Vo respectively. Among them, the target feature inference module identifies multiple targets from each video frame image in the video frame image sequence, extracts the features of each target, and concatenates them with the corresponding position information to finally form the target feature Vo of the video frame image sequence.

[0088] Step 3: Input the target feature Vo, appearance feature Va, motion feature Vm, and context-guided feature G into the visual reasoning module.

[0089] The LOC visual reasoning module calculates the reasoning feature V based on the target feature Vo, appearance feature Va, and fully dynamic context-guided feature G. t L V t L Used to generate nouns and adjectives such as "man", "cat", and "little".

[0090] The REL visual reasoning module calculates the reasoning feature V based on the target feature Vo, motion feature Vm, and fully dynamic contextual guidance feature G. t R V t R Used to generate verbs such as “ride”, “eat”, and “play”.

[0091] The FUN visual reasoning module is based on the historical cell state sequence of the basic decoder. Iterator's historical cell state sequence The inference feature V is calculated from the fully dynamic context-guided feature G. t F V t F Used to generate functional words as well as other parts of speech such as "a", "an", and "the".

[0092] In this embodiment, additive attention is used to perform visual reasoning on the features of the input module in the three inference modules. The additive attention calculation process is shown in the following formula:

[0093] Additive_Attention(V, q)=softmax(score(V, q))V

[0094] =softmax(w T tanh(WV+U1 T q))V

[0095] Where V and q represent the attention value and query, respectively, w, W and U are learnable parameters, and 1 represents a vector of all 1s.

[0096] When the additive attention mechanism is applied to the spatial or temporal dimensions of the module input features, the corresponding spatial and temporal features can be calculated using the input features as values ​​and the fully dynamic context-guided features G as queries. In this embodiment, these are represented as spatial attention SA(V, q) and temporal attention TA(V, q), respectively. The calculation process of both is the same as that of additive attention.

[0097] Internal reasoning details of the LOC visual reasoning module: When generating visual words, the LOC visual reasoning module needs to pay more attention to the spatiotemporal information of specific target regions. First, the target feature Vo is input into the spatial attention SA, and then the output result is concatenated with the appearance feature Va and input into the temporal attention TA. Finally, the inference result V of the LOC visual reasoning module is output. t L The calculation formula is:

[0098] V t L =LOC(V a V o ,G)=TA([SA(V o ,G);V a ], G).

[0099] The internal reasoning details of the REL visual inference module are as follows: The target feature Vo is first processed through spatial attention (SA), then concatenated with motion features Vm representing the dynamic changes in a continuous video frame image sequence. The concatenated features are then paired and input into temporal attention (TA) to obtain the inference result V from the REL visual inference module. t R The calculation formula is:

[0100] V t R =REL(V m V o ,G)=TA(Pair[Cate i ;Cate j ],G),

[0101] Cate = [SA(V) o ,G);V m ]

[0102] Among them, Pair[Cate i ;Cate jThe [] symbol represents the operation of concatenating and pairing all previously obtained feature concatenation results, Cate, together. This is done to enable the model to model the relationship between actions at two different times when generating verbs, thereby inferring the corresponding verbs. For example, if the initial sequence of video frames shows "a man is holding a cup of coffee," and the subsequent sequence of video frames shows "a cup of coffee is moving to a mouth," then by combining the two actions "holding" and "moving" with the feature information of the relevant target region, the word "drinking" can be inferred.

[0103] Internal reasoning details of the FUN visual reasoning module:

[0104] Historical cell state sequence of the basic decoder and the historical cell state sequence of the iterator After concatenation, the data is input into the temporal attention (TA) module so that the model can generate functional words or other parts of speech besides nouns, adjectives, and verbs. These words can connect the content generated by the LOC visual reasoning module and the REL visual reasoning module to form a complete descriptive statement.

[0105]

[0106]

[0107]

[0108] Step 4: Infer the inference feature V t L V t R and V t F The context-guided feature G is input into the module selector, which is then calculated using the score function Score(·,·) to obtain V. t L V t R and V t F The corresponding score represents the probability that the corresponding inference feature is selected. The score function is calculated as shown in the following formula:

[0109]

[0110] in, This represents the calculated fraction. The inference features are represented by *∈{L, R, F}, fc(·) represents a fully connected layer, and tanh(·) represents an activation function.

[0111] Since directly and discretely selecting the module with the highest score would prevent gradient calculation and optimization during backpropagation, the Gumbel Softmax reparameterization method was used during backpropagation. During forward propagation, the model is analyzed from three inference features V... t L V t R and V t F The initial visual reasoning feature V was finally selected. t The vector S can be selected by one-hot encoding. t The inner product of the three inference results is obtained, and the calculation method is shown in the following formula:

[0112]

[0113] in, S represents the inner product operation. t This indicates a one-hot selected vector, and... The maximum value corresponds to 1, while the other two values ​​are 0.

[0114] In fact, since the three inference modules are each responsible for generating descriptive words with different parts of speech, the one-hot selected vector S can also be used. t This can be understood as the prediction result of the part-of-speech information of the words generated at time t. For example, when S... t When the coordinates are [0, 1, 0], the inference result V of the REL visual inference module is... t R The selected input will be fed into the decoder, which means that the model will predict the part-of-speech information of the word generated at the current moment as a verb.

[0115] Step 5: Calculate the initial visual inference features V at time t. t The hidden state of the iterator at time t-1 And the descriptive word y generated at time t-1 t-1 Input the base decoder to obtain the hidden state output by the base decoder at time t. and cell state The basic decoder is represented as deLSTM, and its calculation formula is:

[0116]

[0117] Step 6: Calculate the initial visual inference features V at time t. t The hidden state of the basic decoder at time t And the descriptive word y generated at time t-1 t-1 Input an iterator and obtain the hidden state output by the iterator at time t. and cell state The basic decoder is represented as reLSTM, and its calculation formula is:

[0118]

[0119] Initial visual inference features V input from the encoder end t It is then fed back into the iterator reLSTM, and adjusted according to the hidden state of the base decoder at the current time step. Further optimization can be achieved. It is worth noting that the initial visual inference feature V... t By sequentially passing the basic decoder deLSTM and the iterator reLSTM in an approximate residual connection manner, the video description generation model in this embodiment possesses iterative capability based on progressive visual reasoning, forming an understanding of the initial visual reasoning features V. t The optimization mechanism. Simultaneously, the hidden state output by the iterator. and cell state It is also more accurate and contains richer historical context information.

[0120] The descriptive word y at time t is generated using the softmax function. t probability distribution:

[0121]

[0122] Where W represents a learnable parameter. A beam search method is used based on the description word y. t The probability distribution determines the descriptive word for that time step. Additionally, the cell state of the basic decoder... and the cell state of the iterator The historical state sequences will be added to the input of the FUN visual reasoning module at time t+1. and In the middle. The hidden state of the basic decoder. Hidden state of iterators The word y generated at time t t This will also constitute a new contextual guidance feature G for the next moment, which can continue to guide the model's encoding network to better perform relevant inference operations at time t+1.

[0123] Step 7: Repeat steps 3-6 until the end marker of the output description statement or the maximum length of the specified description statement is reached. The final description statement is obtained using a beam search method based on the description words and their probabilities.

[0124] The video description generation model (BiReNet model) in this embodiment was tested using two datasets: Microsoft Research Video Description (released in 2011) and Microsoft Research-Video to Text (released in 2016), abbreviated as MSVD and MSR-VTT. These are two widely used public datasets in video description generation tasks.

[0125] The MSVD dataset, also known as the YouTube2Text dataset, originally contained 2089 video clips downloaded from the YouTube website, along with approximately 122,000 descriptive sentences in 35 different languages. These video clips ranged in length from 10 to 25 seconds, and each clip typically depicted a single action or event. Current research based on this dataset primarily uses 1970 video clips and their corresponding descriptive sentences from the original dataset. The first experimental dataset in this embodiment consists of these 1970 video clips and their corresponding 80,000+ English descriptive sentences, averaging approximately 41 English descriptive sentences per video clip. Consistent with previous work, the video data samples in the MSVD dataset were divided into three parts: 1200, 100, and 670, serving as the training, validation, and test sets for the model, respectively.

[0126] MSR-VTT version 1.0 includes 10,000 video clips containing audio and their corresponding 200,000 English descriptive sentences, meaning each video clip corresponds to approximately 20 English descriptive sentences. These video clips were obtained from 7,180 videos retrieved from a commercial video search engine. Their duration ranges from 10 to 30 seconds and they are categorized into 20 different categories based on their content, such as news, sports, and education. The second experimental dataset in this embodiment is MSR-VTT version 1.0. Following the same partitioning method used in previous research, 6,513 video data samples were assigned to the training set, 497 video data samples to the validation set, and the remaining 2,990 video data samples to the test set.

[0127] This embodiment utilizes four of the most commonly used automatic evaluation metrics in video description generation methods: BLEU, METEOR, ROUGE, and CIDEr, to systematically evaluate the model's performance. It should be noted that the first three metrics, BLEU, METEOR, and ROUGE, were designed for machine translation tasks, while the last metric, CIDEr, was originally designed for image description generation tasks. Currently, they are all used to evaluate the performance of video description generation models. Higher metric values ​​indicate higher quality generated descriptions.

[0128] In this embodiment, the number of hidden units in the LSTM network of the model is set to 512 and 1300 respectively for the MSVD and MSR-VTT datasets, and both are initialized with all zeros. The batch size for both datasets is set to 8. The optimizer used in the experiment is Adam, with an initial learning rate of 0.0001 during model training. The model training process involves 20 iterations on both datasets. For the MSVD dataset, the learning rate is reduced by a factor of 10 every 10 iterations. For the MSR-VTT dataset, the learning rate is reduced by a factor of 3 every 5 iterations. In this embodiment, a beam width of 2 is used during model testing.

[0129] Video description generation task based on MSVD dataset test

[0130] As shown in Table 1, the BiReNet model demonstrates superior performance on the MSVD dataset, outperforming the state-of-the-art models listed in the table across multiple metrics. This is primarily due to the combined effect of its fully dynamic context-guided features and iterative mechanism based on progressive visual reasoning. Compared to the baseline model, BiReNet not only outperforms it by 0.6% on both METEOR and ROUGE-L metrics but also achieves a significant 3.7% improvement on the CIDEr metric. It's important to note that among the four evaluation metrics, the CIDEr metric is specifically designed for visual description tasks and is considered a more consistent metric with human evaluation. On the BLEU@4 metric, BiReNet's performance is on par with the baseline model. The BLEU@4 metric measures the degree of matching between the model-generated description and the actual reference description. BiReNet, with its fully dynamic context features and sophisticated iterator, can generate more expressive and detailed descriptions than the actual reference description, so it's reasonable that BiReNet's performance on this metric is on par with the baseline model. Overall, the experimental results in the table clearly demonstrate the performance advantages of the BiReNet model proposed in this paper.

[0131] Table 1. Comparison of experimental results (%) of BiReNet model and different models on MSVD dataset.

[0132]

[0133] Video description generation task test based on MSR-VTT dataset

[0134] The comparative experimental results on the MSR-VTT dataset are shown in Table 2. The BLEU@4, METEOR, ROUGE-L, and CIDEr scores of the BiReNet model are improved by 0.9%, 0.1%, 0.8%, and 0.7% respectively compared with the corresponding scores of the baseline model reproduced in this paper. It is worth noting that the BiReNet model outperforms the Mixture model in all metrics on the MSVD dataset, but its performance on the MSR-VTT dataset is slightly lower than that of the Mixture model.

[0135] Table 2 compares the experimental results (%) of the BiReNet model with different models on the MSR-VTT dataset.

[0136]

[0137] In order to find the reasons behind this phenomenon, after in-depth analysis and investigation of the similarities and differences between the two datasets and the characteristics of the Mixture model, it was found that: (1) Compared with the MSVD dataset, the MSR-VTT dataset is larger in scale and contains more complex content. On the one hand, the MSR-VTT dataset contains more video data samples than the MSVD dataset. On the other hand, the length of the description sentence corresponding to each video data sample in the MSR-VTT dataset is longer than that in the MSVD dataset, and the visual words and verbs are richer and more varied. In addition to visual words and verbs, the vocabulary of the MSR-VTT dataset contains more words of other parts of speech; (2) The Mixture model uses 24 parts of speech tags during training, while this patent divides all parts of speech tags into three categories: nouns and adjectives, verbs and other parts of speech, and only uses these parts of speech tags to help the model generate grammatically correct description sentences. Unlike the Mixture model, which focuses on fine-grained use of part-of-speech information when generating descriptive sentences, the part-of-speech tags in this patent only play an auxiliary role in model training; (3) In the model testing phase, the bundle width of the Mixture model is set to 5, while in order to make a relatively fair comparison with the benchmark model, the bundle width in this patent is set to 2, consistent with the bundle width in the benchmark model. The values ​​of various indicators usually fluctuate with the change in the bundle width. This patent believes that the above findings may be the fundamental reason for the significant difference in performance between the BiReNet model and the Mixture model on the two datasets. In particular, due to the limitation of the computing power of the hardware computing device, the batch size (BatchSize) set in the training process of this patent model is 8 instead of 48 in the benchmark model. Nevertheless, the BiReNet model still achieved quite competitive experimental results on the MSR-VTT dataset, and the indicator values ​​on various experimental indicators are close to those of state-of-the-art methods.

[0138] Ablation experiments based on the MSVD dataset

[0139] To verify the effectiveness of the fully dynamic context-guided features and iterators proposed in this patent, ablation experiments were conducted on the MSVD dataset using models with different configurations. These different configurations are shown in Table 3. When the fully dynamic context-guided features and iterators are not added, the BiReNet model degenerates into the baseline model RMN of this patent, which is represented as Enc(RS)-Dec in Table 3, where RS represents the inference module containing averaged static global visual features.

[0140] To demonstrate that fully dynamic context-guided features are more effective than static global visual features after average pooling in guiding the model's inference process, Enc(RS) *The Enc(RS) module concatenates the hidden state of the decoder from the previous time step with the generated descriptive words into a fully dynamic context-guided feature. This feature replaces the global visual features as the query for spatial and temporal attention in the current inference module, thus guiding the model's inference. As shown in Table 3, Enc(RS) * The -Dec model improves the METEOR and CIDEr metrics by 0.3% and 0.6% respectively compared to the baseline model, demonstrating the advantages and potential of using fully dynamic contextual features to guide model inference. It should be noted that the decoder here consists of only a single LSTM network; that is, the iterator proposed in this patent has not yet been added to the decoder network. Therefore, the fully dynamic contextual guidance features here do not include the hidden states of the iterator. In other words, although the contextual guidance features here are fully dynamic, they still differ somewhat from the contextual guidance features in the BiReNet model proposed in this patent.

[0141] Table 3. Comparison of ablation experimental results of the BiReNet model on the MSVD dataset (%)

[0142]

[0143] Enc(RS)-Dec(RC) incorporates an iterator designed in this patented patent into the baseline model to further optimize the initial inference results generated at the encoder end. As shown in Table 3, after introducing the iterator, Enc(RS)-Dec(RC) outperforms the baseline model in both the ROUGE-L and CIDEr metrics. Specifically, in the CIDEr metric, its performance is significantly improved by 2.8% compared to the baseline model, demonstrating the necessity of iterative optimization of the initial inference results and indicating that the iterator helps to further improve the model's performance. **Enc(RS)-Dec(RC) is the BiReNet model proposed in this patent, which incorporates fully dynamic context-guided features and iterators into the baseline model. Among them, the fully dynamic context-guided features not only include the hidden state of the base decoder at the previous time step and the generated descriptive words in the decoder, but also include the hidden state of the iterator that incorporates more historical context information and is more accurate. As shown in Table 3, compared with Enc(RS)-Dec(RC), the BiReNet model has further improved its performance on various indicators and reached a new state-of-the-art level. This patent believes that the BiReNet model's outstanding performance is mainly due to the following three points: (1) The BiReNet model can use iterators to finely optimize the initial visual reasoning results input at the encoder end, thereby generating more expressive descriptive sentences; (2) The fully dynamic context features generated during the iterative optimization stage of the BiReNet model can guide each visual reasoning module at the encoder end to reason out more reasonable and accurate initial reasoning results; (3) The BiReNet model has the ability to alternately execute visual reasoning and iterative optimization, and the two continuously promote each other, thereby improving the overall performance of the model.

[0144] Qualitative analysis test

[0145] To provide a more intuitive understanding of the high-quality descriptive statements generated by the BiReNet model proposed in this patent, this patent demonstrates the descriptive statements actually generated by BiReNet for some video data samples during the model testing phase. For example... Figure 4 As shown in (a), the actual description of the video data sample is "an elephant is eating." The baseline model generated the incorrect action word "walking," while the BiReNet model proposed in this patent correctly inferred the verb "eating." Similarly, BiReNet... Figure 4 (b) accurately generated "two men," while the baseline model incorrectly generated "a man." This reflects that through stepwise visual reasoning and iterative optimization mechanisms, the model-generated descriptive statements can be more accurate. It is worth noting that in... Figure 4 In (c), the base model generated a completely incorrect description, but the BiReNet model proposed in this patent was able to appropriately describe the content in the video. This indicates that the model does indeed need to re-verify and optimize the initial visual inference results to ensure their correctness; otherwise, incorrect inference at the current moment could lead to a disordered description. Figure 4In (d), although the base model generated "aman is riding a motorcycle", this is not the core content expressed in this video data sample. Instead, the BiReNet model is able to accurately identify that the person in the video is performing motorcycle stunts, rather than simply "riding a motocycle".

[0146] Furthermore, it's easy to see from the visualization examples that the descriptions generated by the BiReNet model are more detailed than those generated by the baseline model. For example, in Figure 4 In (e), the baseline model generated a coarse "a man is cooking," while the BiReNet model generated a more detailed "a man is stirring some sauce in a bowl." This patent also uncovered some more interesting phenomena. Figure 4 In (f), neither the reference description nor the description generated by the baseline model included a description of the panda's subtle action of "looking around" in the video. However, the BiReNet model not only recognized the panda's action of lying on the ground in the video, but also paid attention to the movement of the panda's eyes and head, thus generating a more comprehensive "a panda is lying on the ground and looking around". Figure 4 In (g), the baseline model generated "a person is holding a piece of food," a description containing a clear logical error. In reality, the video shows a hamster holding food, while a person is cradling the hamster in their hands. Furthermore, the core content of the video is the person interacting with the hamster. The BiReNet model correctly generated the key information conveyed by the video. Figure 4 In (h), the baseline model generated a rough "a man is removing something," even though the reference description is more detailed. The person in the video is actually explaining how to make a drink. The reference description is too simplistic; in contrast, the description generated by the BiReNet model is more accurate.

[0147] Overall, the above visualization examples clearly demonstrate that the BiReNet model proposed in this paper can generate more accurate, detailed, comprehensive, and expressive descriptive statements for video data. The quality of the descriptive statements generated by the BiReNet model is significantly higher than that generated by the benchmark model, and even better than the reference descriptive statements. Furthermore, these examples also confirm that the fully dynamic context-guided features and iterators proposed in this patent do indeed greatly improve the quality of the descriptive statements.

[0148] The present invention has been described in detail above through embodiments, but the content described is only an exemplary embodiment of the present invention and should not be considered as limiting the scope of the present invention. The scope of protection of the present invention is defined by the claims. Any technical solutions designed by those skilled in the art using the technical solutions described in the present invention, or designed by those skilled in the art under the inspiration of the technical solutions of the present invention, within the substance and protection scope of the present invention, to achieve the above-mentioned technical effects, or any equivalent changes and improvements made to the scope of the application, should still fall within the patent protection scope of the present invention.

Claims

1. A multi-modal inference and iterative optimization video description generation method, characterized in that: Includes the following steps: Step 1: Sample the input video at equal intervals to obtain a sequence of video frame images; Step 2: Input the video frame image sequence into the feature inference module to calculate the target feature Vo, appearance feature Va, and motion feature Vm; Step 3: input the target feature Vo, the appearance feature Va, the motion feature Vm and the context guiding feature G into the visual reasoning module to calculate the reasoning feature 、 and , for generating nouns and adjectives, for generating verbs, for generating functional words and other words of other parts of speech, The formula for calculating the context-guided feature G is: ； Where [;] represents the concatenation operation, and 𝐸[·] represents the word embedding matrix. This refers to the descriptive words generated at time t-1. This represents the hidden state of the basic decoder. Indicates the hidden state of the iterator; Step 4: Determine the inference features , and The context-guided feature G is input into the fractional function respectively. In the middle, the calculation yielded ,and The inference feature with the highest score is selected as the initial visual inference feature. ; ； in, This represents the calculated fraction. Indicates the characteristics of reasoning, fc(·) represents a fully connected layer, and tanh(·) represents an activation function; Step 5: Extract the initial visual inference features at time t. The hidden state of the iterator at time t-1 and the descriptive words generated at time t-1 Input the base decoder to obtain the hidden state output by the base decoder at time t. and cell state The basic decoder is composed of an LSTM network; Step 6: Calculate the initial visual inference features at time t. The hidden state of the basic decoder at time t and the descriptive words generated at time t-1 Input an iterator and obtain the hidden state output by the iterator at time t. and cell state The iterator is composed of an LSTM network and determines the descriptive word for the current time step. Step 7: Repeat steps 3-6 until the end marker of the output description statement or the maximum length of the specified description statement is reached.

2. The multimodal reasoning and iterative optimization video description generation method as described in claim 1, characterized in that: In step 1, sampling is performed every 9-12 frames.

3. The multimodal reasoning and iterative optimization video description generation method as described in claim 1, characterized in that: In step 2, the target feature Vo is formed by splicing the target location information and the target region features.

4. The multimodal reasoning and iterative optimization video description generation method as described in claim 1, characterized in that: In step 3, the visual reasoning module uses an additive attention mechanism for visual reasoning. The additive attention calculation process is shown in the following formula. ； Where V and q represent the attention value and query, respectively, w, W and U are learnable parameters, and 1 represents a vector of all 1s; Spatial attention and temporal attention The calculation process is consistent with the calculation process of additive attention; ； ； in, This indicates an operation that concatenates and pairs all the previously obtained feature concatenation results (Cate) together. , ； in, This represents the historical cell state sequence of the basic decoder. This represents the historical cell state sequence of the iterator.

5. The multimodal reasoning and iterative optimization video description generation method as described in claim 1, characterized in that: In step 4, , in, This represents the inner product operation. This indicates a one-hot selected vector, and... The maximum value corresponds to 1, while the other two values ​​are 0.

6. The multimodal reasoning and iterative optimization video description generation method as described in claim 1, characterized in that: In step 6, the descriptive words at time t are generated using the softmax function. probability distribution: ； Using the beam search method based on description words The probability distribution determines the descriptive word for the current time step.

7. A video description generation model using the multimodal reasoning and iterative optimization video description generation method according to any one of claims 1-6, characterized in that: Includes the encoder and decoder sections. The encoder section includes a feature inference module, a visual inference module, and a module selector connected in sequence; the decoder section includes a basic decoder and an iterator connected in sequence. The feature inference module is divided into an appearance feature inference module, a motion feature inference module, and a target feature inference module. The appearance feature inference module is used to calculate appearance feature Va based on the video frame image sequence. The motion feature inference module is used to calculate motion feature Vm based on the video frame image sequence. The target feature inference module is used to calculate target feature Vo based on the video frame image sequence. The visual reasoning module is divided into an LOC visual reasoning module, a REL visual reasoning module, and a FUN visual reasoning module. The LOC visual reasoning module is used to calculate reasoning features based on target features Vo, appearance features Va, and context-guided features G. The REL visual reasoning module is used to calculate reasoning features based on target features Vo, motion features Vm, and context-guided features G. The FUN visual reasoning module is used to determine the historical cell state sequence based on the basic decoder. The historical cell state sequence of the iterator The inference features are calculated from the context-guided features G. , The module selector is used to select reasoning features. , and One of them serves as an initial visual reasoning feature. The input is fed into the underlying decoder.

8. The video description generation model as described in claim 7, characterized in that: The appearance feature inference module is a two-dimensional convolutional neural network, the motion feature inference module is a three-dimensional convolutional neural network, and the appearance feature inference module uses the Faster R-CNN object detection network.

9. The video description generation model as described in claim 7, characterized in that: Both the basic decoder and iterator are composed of LSTM networks.

10. The video description generation model as described in claim 7, characterized in that: The total loss function L of the model is: ； in, It is a proportional parameter that balances the two losses. Represents cross-entropy loss, This represents the KL divergence loss.

Images