Video description generation method based on perceptual grammar knowledge

Abstract

This invention belongs to the field of computer vision technology, specifically a tree-structured video description generation method based on perceptual grammar knowledge. This invention explicitly utilizes semantic information inherent in language, using dependency structure analysis tools to transform sequential sentences into a syntax tree structure. By analyzing the connections between parent and child nodes in the tree, the dependency grammar structure within the sentence is explicitly modeled. A perceptual context attention network is used to model the contextual information generated along different paths during the generation process. Simultaneously, reinforcement learning and iterative generation training methods are introduced during the training phase to further improve model performance. Qualitative and quantitative experiments demonstrate that the model has the ability to generate more accurate and semantically richer descriptions.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a video description generation method based on perceptual grammar knowledge. Background Technology

[0002] With the advancement of technology, the difficulty of shooting videos is constantly decreasing. Therefore, as the number of videos increases, understanding video content through technology becomes increasingly important. Video description generation aims to provide a concise description of video content in a single sentence. Video description generation is a cross-modal task combining computer vision and natural language processing, requiring computers not only to understand the interactions between objects and other objects in the video, but also to describe them using a form that conforms to natural language grammar. Current mainstream research generally uses an encoder-decoder structure, utilizing the encoder to obtain the video's feature representation and the decoder to generate sentences. Inspired by the improvement in model performance achieved by explicitly introducing semantic knowledge in natural language processing tasks, this invention proposes a grammatical structure-based video description method: a tree-structured generative network that perceives grammatical knowledge. Summary of the Invention

[0003] To address the problems of existing video description generation algorithms neglecting the complex grammatical knowledge and sentence structure in natural language, and the exposure bias in generation models, this invention proposes a video description algorithm based on perceptual grammar knowledge.

[0004] This invention leverages grammatical knowledge to aid in video description generation, discusses how to utilize grammatical information to improve model performance during the generation process, and proposes a video description generation network that perceives grammatical knowledge. It uses an explicit approach to model textual grammatical knowledge and employs an attention computation process to enable the model to learn the grammatical relationships between words in sentences through training.

[0005] To leverage the complex grammatical knowledge and sentence structures inherent in natural language, a tree-structured generative network that perceives grammatical knowledge models the contextual information of different paths generated during the generation process, and processes multimodal features through an attention network. Furthermore, to avoid the vanishing gradient problem during model convergence and the exposure bias of the generative model caused by overly complex model design, reinforcement learning and iterative generation training methods are introduced during the training phase to further improve model performance.

[0006] In the experiments, qualitative and quantitative tests were conducted on two commonly used datasets for video description generation tasks. By comparing the performance metrics of the constructed model with those of classic models in recent years for video description generation tasks, and by visualizing the model's generation results, the performance improvement brought about by the use of grammatical knowledge was demonstrated. The qualitative and quantitative experiments showcased the model's ability to generate more accurate and semantically richer descriptions.

[0007] The video description method based on perceptual grammar knowledge provided by this invention includes four steps: video feature extraction, tree structure representation extraction, topology prediction, and sentence generation, wherein:

[0008] Step 1, Video Feature Extraction: Encode the video, perform fixed sampling for each video, and encode the sampled video frames and the consecutive time periods centered on the frame using deep learning 2D convolutional models and 3D convolutional models respectively, producing 2D convolutional features and 3D convolutional features respectively.

[0009] Step 2, Tree Structure Representation Extraction: Extract the grammatical structure information contained in the sentence, and generate a dependency tree for each sentence using an existing parsing tool (nltk toolkit); then preprocess the constructed dependency tree (adding and deleting some information); finally, modify the tree structure using dependency tree features (parent-child, sibling relationships of nodes) and store it in a table format for easy use by subsequent models;

[0010] Step 3, Topology prediction: After extracting video features and tree structure information in steps 1 and 2, topology prediction is performed using an attention mechanism based on the dependency order of nodes in the tree structure, utilizing two different context paths (sibling node path and parent node path).

[0011] Step 4, Sentence Generation: Explicitly collect and use historical information from the generation process, and use an attention mechanism to fuse information from two different context paths (information from all sibling nodes of the current node and information from the parent node of the current node) and decode to generate video descriptions.

[0012] The video feature extraction process described in step 1 is as follows:

[0013] For each video, a fixed sampling method (fixed sampling means sampling a fixed number of frames per video, where the interval needs to be determined based on the number of frames) is used to select 28 frames as keyframes. For 2D convolutional features, the InceptionResNet2 2D convolutional model pre-trained on the ImageNet dataset is used to extract 1,536-dimensional features from each keyframe, resulting in 2D convolutional features, which are then expressed using V... a The representation indicates that for 3D convolutional features, keyframes are used as center frames, and consecutive time segments are extracted to form 28 short segments. Using the pre-trained 3D convolutional model C3D on the Kinetics-400 dataset, 2,048-dimensional features are extracted from each segment to obtain the 3D convolutional features. V is then used... mThe final step involves concatenating the 1,536-dimensional two-dimensional convolutional features and the 2,048-dimensional three-dimensional convolutional features of each keyframe in the video as the action feature, as shown in the formula:

[0014] V i =W v [V m,i V a,i ], (1)

[0015] Where [;] represents the data concatenation operation; W v This represents the learnable parameters in the model, and the subscript i indicates which frame in the 28 frames of the video.

[0016] Step 2, the tree structure representation extraction, specifically includes:

[0017] Sentences, as we generally perceive them, consist of words arranged in sequence. Inspired by this phenomenon, most generative models employ recurrent neural networks to construct sentences by generating words sequentially. However, the grammatical structure of sentences is complex, and higher-level representations often contain richer grammatical knowledge and sentence structures. Therefore, to explicitly model this semantic knowledge, this invention uses a tree-structured recurrent neural network as the decoder generation model and employs long-distance dependency relationships between words as semantic knowledge representation. The process is described in [link to process details]. Figure 2 The above;

[0018] Among them, the processing of video and descriptive statements, such as Figure 2 As shown in (a). The processing of descriptive statements involves the following steps:

[0019] (1) First, construct the dependency tree: Use existing parsing tools (nltktoolkit) to extract relation triples: subject, relation, object. Then, based on the relation triples, with the subject as the parent node and the object as the child node, use an edge with a relation pointing from the parent node to the child node to represent the dependency relation and link the parent and child nodes. By doing this, a dependency tree can be constructed, also called a dependency relation tree, such as... Figure 2 As shown in (b);

[0020] (2) Secondly, the constructed dependency tree needs to be preprocessed; the dependency tree needs to be preprocessed after the model, the links between the left and right children of the parent node are deleted, the edge information is discarded, and the left child of the parent node is linked with a relational edge, and the right child is linked with a relational edge, such as Figure 2 As shown in (c);

[0021] (3) Then, the tree structure information is stored in a table format; 0 and 1 are used to indicate whether the current node has a sibling node, a left child node, and a right child node. The binary representations of these three node types are then converted to decimal status to represent the parent node's status, such as... Figure 2 As shown in (d), the dependency tree does reflect the relative importance relationships between words to a certain extent, and the length of the generated sequence is significantly shorter than the length of the sentence itself. To better utilize the semantic knowledge in the dependency tree, the model removes the dependency types from the edges of the original dependency tree and adds sibling node relationships, that is, adds an edge with a relationship between child nodes that share the same parent node, such as... Figure 2 As shown in (c). Furthermore, considering the temporal order of words in a sentence, for all child nodes of the same node, the model further distinguishes them into left and right child nodes based on their relative order within the sentence. For example, for the node "rubbing," "person" and "is" are its left child nodes, while "toad" and "finger" are its right child nodes. At the same time, a sibling connection is not needed between "is" and "toad." Finally, to enable the model to directly use dependency relationships through a recurrent neural network, such as... Figure 2 As shown in (d), the model converts the tree structure into a table format, using the attributes in the table to represent the dependencies between nodes in the tree. Specifically, "Sibling," "Left Child," and "Right Child" use binary data to indicate whether the current node has this relationship. Furthermore, to indicate that the relationships are not independent, the model treats these three relationships as consecutive binary representations and converts them into a "Status" attribute. For example, for the node "rubbing," its binary characteristic is represented as "011," indicating that it has no sibling nodes but has a left child node and a right child node; therefore, its "Status" attribute is represented as "3."

[0022] Thus, the model successfully models and represents semantic knowledge in dependency forms and transforms it into a way that recurrent neural networks can directly model and process.

[0023] The topology prediction in step 3 involves the following specific processing steps:

[0024] (1) Specific modeling of dependency tree connection information:

[0025] By definition, given a dependency tree, the connection information it contains can be modeled and represented as follows: in Let σ represent the set of nodes, σ represent the set of edges, and π represent the vocabulary. In the model... In, the set of nodes Let represent the words in the sentence, σ represent the state information of each node, and π be a vocabulary list containing all words constructed based on the data. For each node i∈π and its corresponding parent node p(i) and sibling node s(i), the model constructs its information update path, and its data processing can be represented as follows:

[0026]

[0027]

[0028] Among them, GRU w and GRU d Represent the dependency tree structure network on the paths of sibling nodes and the parent node, respectively; I w and I d These represent the input information passed from the sibling node and the parent node, respectively. and These represent the hidden layer information the current node receives from its sibling and parent nodes, respectively. It's important to note that the updates to the two paths are not simultaneous in the model; the information computation on the parent node's path depends on the results on the sibling node's path. This is because in a dependency tree, the dependencies between parent nodes are real, while the connections between sibling nodes are artificially added during the tree structure representation phase to make the dependency tree easier for the model to process. Therefore, in the actual implementation, the information processing of the parent node is more crucial. The results on the parent node's path... This includes not only the parent node information of node i, but also its sibling node information. Furthermore, since child nodes are explicitly distinguished as left and right children based on the relative positions of words during the tree structure representation stage, the parent node update network also needs to be specially partitioned into a GRU (Generic Randomization) architecture. dl and GRU dr These are used to process information about the left and right child nodes, respectively.

[0029] For traditional models using recurrent neural networks as decoders, a special symbol is typically added to the vocabulary and appended to the end of all sentences to enable the model to learn when to stop sentence generation. This prevents the model from generating further sentences after the symbol is generated. However, for the dependency tree structure generative network used in this model, the update path is divided into vertical parent-child node information updates and horizontal sibling node information updates. Therefore, using this special symbol to mark the end of sequence generation requires adding several stop symbols to the dependency tree, significantly increasing the tree size and computational cost. Furthermore, the traditional method of using special symbols to mark the end increases sequence length, potentially exacerbating gradient vanishing and exploding problems. Moreover, using special symbols to mark the end of sentences does not align with human habitual description in everyday generation. Therefore, this invention employs an alternative method to stop tree generation during the process: explicit prediction of the model's topological information. The model uses a tree structure network. To generate node state information:

[0030]

[0031] Among them, w topo p represents the learnable parameters in the model. t ∈[0,7] represents the state information of a node, indicating whether the current node has a sibling node, a left child node, and a right child node. Where n represents the number of categories, and i represents the category number. Once the model predicts a topology indicating that a node has no sibling, left child, or right child node to generate, the model stops information transmission along the corresponding path. Once all nodes have completed their information updates and no new nodes need to be processed, the dependency tree is considered complete. Therefore, explicitly predicting the topology does not increase the size of the tree or the computational overhead of the network, and it also allows the model to utilize its topology information during the sentence generation phase, explicitly implementing the use of semantic knowledge in the generation process.

[0032] The statement generation described in step 4 specifically includes:

[0033] The sentence generation network consists of a Gate Recurrent Unit (GRU), a Context-Aware Attention (CAA) module, and a Gated Linear Unit (GLU). The specific functions of each module and their respective data processing procedures are as follows:

[0034] (1) First, the hierarchical tree structure contains two layers of gated loop units. The global information of the model needs to be processed first, and the processing can be expressed as follows:

[0035]

[0036]

[0037] in, This represents the output of the previous node, which can be categorized into information obtained from the parent node or sibling node depending on the update path; W e Indicates learnable parameters; w t-1 This represents a single valid code for the word in the previous node; This indicates the mean characteristic of the video. This indicates the output of the first-level gated loop unit.

[0038] (2) The result of the hidden layer of global feature information of the given processing model The generative model is then fed into the attention calculation module of the perceptual context to obtain local information at the current time step. This perceptual context attention calculation module comprises four attention calculation processes, all using an additive approach, but each with different calculation parameters and requiring attention calculations for different features:

[0039] (2.1) First, the hidden layer results of global feature information In global feature {V i Attention calculation is performed to determine which specific information segment is important for the generation of the current time step. The attention calculation process can be represented as follows:

[0040]

[0041] (2.2) Secondly, given the context c of the previous node. t-1 This indicates the result of the attention calculation module of its parent or sibling node, using... The result of the current attention calculation on global features is a t and historical information of attention c t -1 Attention is selected based on contextual information to determine whether it is helpful for the generation of the current time step. The calculation process can be represented as follows:

[0042]

[0043] (2.3) Then, after obtaining the result feature o tThen, the perception context module fuses it with global features to obtain the global feature information {V} of the perception context. c,i The calculation process can be expressed as follows:

[0044] V c,i =V i -o t (9)

[0045] (2.4) Subsequently, using Attention is calculated on the global feature information of the perceptual context to obtain the local feature information of the perceptual context. The calculation process can be represented as follows:

[0046]

[0047] (2.5) Finally, after obtaining the local feature information a t and local feature information of the perceptual context a c,t In this case, use Make a decision, determine whether contextual information is needed at the current moment, and finally obtain the output c of the attention calculation module for perceiving the context at the current moment. t The calculation process can be expressed as follows:

[0048]

[0049] Among them, f att This refers to the attention calculation process.

[0050] (3) The result c of the attention calculation module obtained in the perceptual context t In this case, the model additionally introduces gated linear units to generate the hidden values ​​of the language model. The gated linear unit generates the information vector and the attention gate through two linear layers, respectively. Its computation process can be represented as follows:

[0051]

[0052]

[0053] in, b i and b g This represents the learnable parameters and biases in the model. After obtaining the information vector i... t and attention gate g t Then, the hidden values ​​of the language model can be calculated:

[0054]

[0055] Here, ⊙ represents the dot product operation between vectors. Simultaneously, according to formula (4), the topological structure information p of the node can be obtained.t .

[0056] Common sense tells us that grammatical information about words helps in word prediction. For example, if a word has child nodes, it cannot be a definite article like "a". Word prediction requires consideration not only of the hidden layer results of the language model. The topological structure information of words also needs to be considered. Therefore, the word prediction process that considers the topological prediction results can be represented as:

[0057]

[0058] Where, p topo Represents the probability distribution p of the topological state t The results obtained through sampling are used for prediction during model training with the correct topology of the nodes, and the sampling results are used during the prediction phase; W p The learnable parameter matrix represents the state information, from which the state representation of the topological information is obtained; This represents the language hidden layer information of the current node; w p This represents the learnable parameters in the model. t This represents the word predicted at time t. The words predicted at all times together form the final video description.

[0059] The advantages of this invention include:

[0060] First, the decoder is extended into a tree architecture, and historical information from sibling and parent nodes is explicitly used to help the model mitigate long-distance dependencies during generation. The model also employs iterative generation and reinforcement learning training methods to alleviate gradient vanishing and exposure bias issues. Finally, this invention achieves performance exceeding current mainstream methods on the publicly available datasets MSR-VTT and MSVD. Attached Figure Description

[0061] Figure 1 This is a framework diagram of the present invention.

[0062] Figure 2 This is a diagram illustrating the extraction of grammatical structure information in this invention. (a) shows the video and its corresponding description; (b) shows the dependency tree extracted using an existing tool (nltk toolkit); (c) shows the preprocessed dependency tree; and (d) shows the binary representation of the three states of nodes in the preprocessed dependency tree and their conversion to decimal status.

[0063] Figure 3 These are renderings of the invention. (a) shows an example effect. Figure 1 (b) shows the example effect. Figure 2 (c) is an example effect. Figure 3 (d) is the example effect diagram 4. Each example effect diagram can be divided into 3 parts (left, middle, right). The left part represents the video and description corresponding to the example, the middle part represents the node state representation of the dependency tree corresponding to the description of the example after processing, and the right part represents the dependency tree corresponding to the example and the dependency relationship between each node (using weight to represent the degree of importance). Detailed Implementation

[0064] As the background technology reveals, existing video caption generation algorithms suffer from three main drawbacks. First, most algorithms neglect the use of complex grammatical knowledge and sentence structures. Second, most models require attention calculations on global features at every time step, thus ignoring the use of attention results related to specific words to obtain better attention features. Third, generative models suffer from exposure bias; that is, they typically use the prediction result of the previous word as the model input for the current time step. Therefore, when a misprediction occurs, the error accumulates and affects all subsequent generation time steps.

[0065] For each video, 28 frames were selected as keyframes using a fixed sampling method. For 2D convolutional features, the InceptionResNet2 network pre-trained on the ImageNet dataset was used to extract 1,536-dimensional features from each keyframe. For 3D convolutional features, 28 short segments were formed by taking consecutive time intervals from the keyframes as center frames, and the C3D network pre-trained on the Kinetics-400 dataset was used to extract 2,048-dimensional features from each segment. Finally, for each keyframe in the video, its 1,536-dimensional 2D convolutional features and 2,048-dimensional 3D convolutional features were directly concatenated as action features, as shown below:

[0066] V i =W v [V m,i V a,i (1)

[0067] Where [;] represents the data concatenation operation; W v This represents the learnable parameters in the model.

[0068] Given a dependency tree, the connection information it contains can be modeled as follows: in Let σ represent the set of nodes, σ represent the set of edges, and π represent the vocabulary. In the model, the set of nodes represents the words in the sentence, the set of edges represents the state information of each node, and the vocabulary is a list of all words constructed based on the data. For each node i∈π and its corresponding parent node p(i) and sibling node s(i), the model constructs its information update path, and its data processing can be represented as follows:

[0069]

[0070]

[0071] Among them, GRU w and GRU d Represent the tree structure network along the paths of sibling nodes and the parent node, respectively; I w and I d These represent the input information passed from the sibling node and the parent node, respectively. and These represent the hidden layer information that the current node obtains from its sibling node and parent node, respectively.

[0072] Subsequently, the model uses the results of a tree-structured network. To generate node state information:

[0073]

[0074] Among them, w topo p represents the learnable parameters in the model. t ∈[0,7] represents the state information of a node, which indicates whether the current node has a sibling node, a left child node, and a right child node.

[0075] When generating statements, the global information of the model needs to be processed first. The processing can be represented as follows:

[0076]

[0077]

[0078] in, This represents the output of the previous node, which can be categorized into information obtained from the parent node or sibling node depending on the update path; W e Represents the word embedding matrix; w t-1 This represents a single valid code for the word in the previous node; This indicates the mean characteristic of the video. This indicates the output of the first-level gated loop unit.

[0079] Subsequently, the hidden layer results of global feature information With global features {Vi Attention calculation is performed to determine which specific information segment is important for the generation of the current time step. The attention calculation process can be represented as follows:

[0080]

[0081] Secondly, given the context c of the previous node t-1 ,use The result of the current attention calculation on global features is a t and historical information of attention c t-1 The selection of attention can be calculated as follows:

[0082]

[0083] Subsequently, after obtaining the result feature o t Then, the perceptual context module fuses it with the global features to obtain the global feature information {V} of the perceptual context. c,i The calculation process can be expressed as follows:

[0084] V c,i =V i -o t (9)

[0085] Subsequently, using Attention is calculated on the global feature information of the perceptual context to obtain the local feature information of the perceptual context. The calculation process can be represented as follows:

[0086]

[0087] Finally, after obtaining local feature information a t and local feature information of the perceptual context a c,t In the case of, use It determines whether contextual information needs to be utilized at the current moment, and ultimately obtains the output c of the attention calculation module for perceiving the context at the current moment. t The calculation process can be expressed as follows:

[0088]

[0089] Among them, f att This refers to the attention calculation process.

[0090] Finally, the result c of the attention calculation module for the perceptual context is obtained. t In this case, the model additionally introduces gated linear units to generate the hidden values ​​of the language model. The calculation process can be represented as follows:

[0091]

[0092]

[0093] in, b i and b g This represents the learnable parameters and biases in the model. After obtaining the information vector i... t and attention gate g t Then, the hidden values ​​of the language model can be calculated:

[0094]

[0095] Here, ⊙ represents the dot product operation between vectors. Meanwhile, according to formula (4), the topological structure information p of the node can be obtained. topo The word prediction process considering topological prediction results can be represented as follows:

[0096]

[0097] Where, p topo Represents the probability distribution p of the topological state t The results obtained through sampling are used for prediction during model training with the correct topology of the nodes, and the sampling results are used during the prediction phase; W p The embedding matrix represents the state information, from which the state representation of the topological information is obtained; This represents the language hidden layer information of the current node; w p This represents the learnable parameters in the model.

[0098] Then, the loss is calculated, and the loss function is minimized through backpropagation, thereby updating the network parameters. The specific process is as follows:

[0099] First, training is performed during the cross-entropy phase, given the correct descriptions of manually annotated videos. Correct state information of the dependency tree The model simultaneously calculates the loss functions for both distributions using cross-entropy and combines them by summation. The specific loss function is defined as follows:

[0100]

[0101] Where θ is the model parameter, T is the correct description length of the manual annotation, and V is the video feature.

[0102] Secondly, reinforcement learning training is performed. After pre-training the model using the cross-entropy loss function for a period of time, the model is further optimized using reinforcement learning methods and natural language evaluation metrics. Similar to the cross-entropy loss function stage, the reinforcement learning stage still needs to consider loss functions for both topological structure and word distributions. Therefore, the model's expectation can be expressed as:

[0103] L(θ)=-Ε[r(y s ,p s )] , (17)

[0104] Among them, y s and p s represents the sampling results for words and state information, respectively; r() represents the reward value calculation, i.e., the evaluation metric for natural language processing. Considering that the calculation process of the evaluation metric is usually non-differentiable, it cannot be directly used as the objective function to optimize the model. Here, a reinforcement algorithm is used to estimate the expected gradient of the model, which is expressed as:

[0105]

[0106] Considering the impact of biases in the gradient estimation process on the convergence process, the model employs a self-supervised reinforcement learning method. An additional baseline reward value is introduced, and the model itself obtains the result during the testing phase through maximum sampling rather than probability sampling, calculating the difference between these reward values ​​for gradient estimation. The self-supervised gradient estimation process can be expressed as: where b is a hyperparameter controlling the penalty applied to the model.

[0107]

[0108] By introducing a baseline reward value, the model achieves better results under probability sampling than those under maximum sampling, thus enabling the model to be optimized directly at the sentence level in terms of evaluation metrics.

[0109] In this embodiment, two of the most widely used datasets are selected to validate the algorithm: MSVD and MSR-VTT. The MSVD dataset consists of 1,967 short videos, each containing approximately 40 manually annotated English descriptive sentences. In current experiments, following the officially defined data partitioning, 1,200 videos are used as the training set, 100 videos as the validation set, and 670 videos as the test set. The MSR-VTT dataset is a large video benchmark dataset released by Microsoft in 2016, containing 10,000 video segments, each containing approximately 20 manually annotated English descriptive sentences. Currently, 6,513 videos are typically used as the training set, 497 videos as the validation set, and 2,990 videos as the test set.

[0110] This embodiment uses four objective evaluation metrics—BLEU, METEOR, ROUGE, and CIDEr—to evaluate the performance of video content description. Experimental results show that this embodiment outperforms other methods in retrieval accuracy on both datasets. On two commonly used video description datasets, MSVD and MSR-VTT, the model is compared with classic models based on commonly used evaluation metrics BLEU-4, METEOR, ROUGE, and CIDEr. The relevant experimental results are shown in Tables 1 and 2. As can be seen from the tables, for the commonly used CIDEr metric in video description generation tasks, the stacked attention network with perceptual context achieves the best results compared to other methods, especially on the MSR-VTT dataset. Similarly, the model still achieves good results for the ROUGE and METEOR metrics. To address the exposure bias problem, reinforcement learning is introduced during the training phase, selecting CIDEr as the reward value for model optimization. Therefore, the model shows a significant improvement in the CIDEr metric. A similar phenomenon can be seen in the MARN model, which uses reinforcement learning to optimize the model; the MARN model also significantly outperforms other models in terms of the CIDEr metric.

[0111] Table 1: Comparison results with other existing methods on the MSVD dataset

[0112]

[0113] Table 2: Comparison results with other existing methods on the MSR-VTT dataset

[0114]

Claims

1. A video description generation method based on perceptual grammar knowledge, characterized in that, This method explicitly utilizes semantic information inherent in the language, employing dependency structure analysis tools to transform sequential sentences into syntax trees. It explicitly models the dependency syntax structure of sentences through the connections between parent and child nodes within the trees. Furthermore, it models the contextual information generated during the generation process using a context-aware attention network. Simultaneously, it introduces reinforcement learning and iterative generation training methods during the training phase to further improve model performance. The specific steps are as follows: Step 1, Video Feature Extraction: Encode the video. For each video frame, use a deep learning 2D convolutional model and a 3D convolutional model to encode it, and produce 2D convolutional features and 3D convolutional features respectively. Step 2, Tree structure representation extraction: Extract the grammatical structure information contained in the sentence, and use the existing parsing tool nltk toolkit to explicitly model the grammatical knowledge for each sentence; Step 3, Topology prediction: After extracting video features and tree structure information in steps 1 and 2, an attention mechanism is used to predict the topology using two different contextual paths: the sibling node path and the parent node path. Step 4, Sentence Generation: Explicitly collect and use historical information from the generation process, utilize attention mechanisms to fuse information from two different contexts, and decode to generate video descriptions; The topology prediction in step 3 is specifically as follows: (1) Specific modeling of dependency tree connection information: Given a dependency tree, its connection information is modeled as Γ={θ,σ,π}, where θ represents the set of nodes, σ represents the set of edges, and π represents the vocabulary. In the model Γ={θ,σ,π}, the node set θ represents the words in the sentence, the edge set σ represents the state information of each node, and the vocabulary π is a vocabulary containing all words constructed based on the data. For each node i∈π and its corresponding parent node p(i) and sibling node s(i), the path for updating its information is constructed, and its data processing is represented as follows: Among them, GRU w and GRU d Represent the dependency tree structure network on the paths of sibling nodes and the parent node, respectively; I w and I d These represent the input information passed from the sibling node and the parent node, respectively. and These represent the hidden layer information that the current node receives from its sibling node and parent node, respectively. The updates to these two paths are not performed simultaneously in the model; the information computation on the parent node's path depends on the results on the sibling node's path, meaning the parent node's information processing is more crucial. The results on the parent node's path... It includes not only the parent node information of node i, but also its sibling node information; furthermore, since child nodes are explicitly distinguished as left and right child nodes based on the relative positions of words during the tree structure representation stage, the parent node update network is also divided into GRU. dl and GRU dr These are used to process the information of the left child node and the right child node, respectively; A special approach is used to generate stopping trees, namely, to explicitly predict the topological information of the model, specifically using the results of tree-structured networks. To generate node state information: Among them, w topo p represents the learnable parameters in the model. t ∈[0,7] represents the state information of a node, indicating whether the current node has a sibling node, a left child node, and a right child node. Where n represents the number of categories and i represents the category number; once the topological structure information predicted by the model indicates that this node has no sibling node, left child node, or right child node to be generated, the model stops the information transmission on the corresponding path; when all nodes have completed information updates and there are no new nodes to be processed, the dependency tree is considered to have been generated.

2. The video description generation method based on perceptual grammar knowledge according to claim 1, characterized in that, The video feature extraction process described in step 1 is as follows: For each video, 28 frames were selected as keyframes using a fixed sampling method. The InceptionResNet2 2D convolutional model, pre-trained on the ImageNet dataset, was used to extract 1,536-dimensional features from each keyframe, resulting in 2D convolutional features. V... a This method involves using keyframes as center frames and extracting 28 short segments from consecutive time intervals. A pre-trained 3D convolutional model, C3D, on the Kinetics-400 dataset is used to extract 2,048-dimensional features from each segment, resulting in 3D convolutional features. The V... m Finally, for each keyframe in the video, its 1,536-dimensional two-dimensional convolutional features and 2,048-dimensional three-dimensional convolutional features are directly concatenated as action features, represented as follows: V i =W v [V m,i ;V a,i ], (1) Where [;] represents a data concatenation operation; W v This represents the learnable parameters in the model, and the subscript i indicates which frame in the 28 frames of the video.

3. The video description generation method based on perceptual grammar knowledge according to claim 2, characterized in that, Step 2 describes the extraction of tree structure representation. First, a recurrent neural network with a tree structure is selected as the decoder generation model, and long-distance dependency relationships between words are used as semantic knowledge representation. (1) Constructing a dependency tree: Use parsing tools to extract relation triples: subject, relation, object. Then, based on the relation triples, use the subject as the parent node and the object as the child node. Use an edge with a relation from the parent node to the child node to represent the dependency relation and link the parent and child nodes. In this way, a dependency tree is constructed, which can also be called a dependency tree. (2) Preprocess the constructed dependency tree; delete the links between the left and right children of the parent node, discard the edge information, and link the left child of the parent node with the edge with the relationship, and link the right child with the edge with the relationship. (3) Store tree structure information in tabular form; Use 0 and 1 to indicate whether the current node has a sibling node, a left child node, and a right child node. Then convert the binary representation of the three types of nodes into a decimal status to represent the status of the parent node. Remove the dependency types from the edges of the original dependency tree and add sibling relationships, that is, add an edge with a relationship between child nodes that share the same parent node. Considering the chronological order of words in a sentence, for all child nodes of the same node, they are further divided into left child nodes and right child nodes according to their relative order in the sentence; Finally, the tree structure is converted into a table, and the attributes in the table are used to represent the dependencies of nodes in the tree. Among them, "Sibling", "Left Child" and "Right Child" use binary data to indicate whether the current node has such a relationship; at the same time, in order to indicate that the relationships do not exist independently, these three relationships are further regarded as continuous binary representations and converted into "Status" attributes; Therefore, semantic knowledge of dependency forms is modeled and represented, and then converted into a way that can be directly modeled and processed by recurrent neural networks.

4. The video description generation method based on perceptual grammar knowledge according to claim 3, characterized in that, In step 4, the sentence generation model comprises three modules: a gated recurrent unit (GRU), a context-aware attention computation module (CAA), and a gated linear unit (GLU). The function of each module and its respective data processing procedure are as follows: (1) First, the hierarchical tree structure contains two levels of gated loop units. The global information of the model is processed first, and the processing procedure is as follows: in, This represents the output of the previous node, which, depending on the update path, is information obtained from either the parent node or the sibling node; W e Indicates learnable parameters; w t-1 This represents a single valid code for the word in the previous node; This indicates the mean characteristic of the video. This indicates the output of the first-level gated loop unit; (2) The result of the hidden layer of global feature information of the given processing model The generative model is then fed into the attention calculation module of the perceptual context to obtain local information at the current time step. This module comprises four attention calculation processes, all using an additive approach, but each with different calculation parameters and performing attention calculations on different features: (2.1) First, the hidden layer results of global feature information In global features {V i Attention is calculated to determine which specific information segment is important for the generation of the current time step. The attention calculation process is represented as follows: (2.2) Secondly, given the context c of the previous node. t-1 This indicates the result of the attention calculation module of its parent or sibling node, using... The result of the current attention calculation on global features is a t and historical information of attention c t-1 Attention is selected to determine whether contextual information is helpful for the generation of the current time step. The calculation process is as follows: (2.3) Then, after obtaining the result feature o t Then, the perception context module fuses it with global features to obtain the global feature information {V} of the perception context. c,i The calculation process is expressed as follows: V c,i =V i -o t , (9) (2.4) Subsequently, using Attention is calculated on the global feature information of the perceptual context to obtain the local feature information of the perceptual context. The calculation process is as follows: (2.5) Finally, after obtaining the local feature information a t and local feature information of the perceptual context a c,t In this case, use Make a decision, determine whether contextual information is needed at the current moment, and finally obtain the output c of the attention calculation module for perceiving the context at the current moment. t The calculation process is expressed as follows: Among them, f att This refers to the attention calculation process; (3) The result c of the attention calculation module obtained in the perceptual context t In this case, the model additionally introduces gated linear units to generate the hidden values ​​of the language model; the gated linear unit generates the information vector and the attention gate through two linear layers, respectively, and its calculation process is expressed as follows: in, b i and b g This represents the learnable parameters and biases in the model; after obtaining the information vector i t and attention gate g t Then, the hidden values ​​of the language model can be calculated: Where ⊙ represents the dot product operation between vectors; according to formula (4), the topological structure information p of the node is obtained. t ; In word prediction, the results of the hidden layer of the language model should be considered. We also need to consider the topological structure information of words; therefore, the word prediction process considering the topological prediction results is expressed as follows: Where, p topo Represents the probability distribution p of the topological state t The results are obtained through sampling; the correct topology of the nodes is used for prediction during model training, and the sampling results are used during the prediction phase; W p The learnable parameter matrix represents the state information, from which the state representation of the topological information is obtained; This represents the language hidden layer information of the current node; w p This represents the learnable parameters in the model.

5. The video description generation method based on perceptual grammar knowledge according to claim 4, characterized in that, During model training, the loss is calculated by minimizing the loss function using backpropagation, and the network parameters are updated accordingly. The specific process is as follows: First, training is performed during the cross-entropy phase, given the correct descriptions of manually annotated videos. Correct state information of the dependency tree The model simultaneously calculates the loss functions for both distributions using cross-entropy and combines them by summation; the specific loss function is defined as: Where θ is the model parameter, T is the correct description length of the manual annotation, and V is the video feature; Secondly, reinforcement learning training is performed. After the model has been pre-trained using the cross-entropy loss function for a period of time, reinforcement learning is sampled to further optimize the model on natural language evaluation metrics. During the reinforcement learning phase, the model still needs to consider loss functions for both topological structure and word distributions; therefore, the model's expectation is expressed as: L(θ)=-Ε[r(y s ,p s )], (17) Among them, y s and p s Let r represent the sampling results for words and state information, respectively; r() represents the reward value calculation, i.e., the evaluation index of natural language; considering that the calculation process of the evaluation index is not differentiable, it cannot be directly used as the objective function to optimize the model. Here, a reinforcement algorithm is used to estimate the expected gradient of the model, and its gradient is expressed as: Considering the impact of biases in the gradient estimation process on the convergence process, a self-supervised reinforcement learning method is adopted. An additional baseline reward value is introduced, obtained by the model itself during the testing phase using maximum sampling, and the difference between these reward values ​​is calculated for gradient estimation. The self-supervised gradient estimation process is expressed as follows: Where b is a hyperparameter that controls the penalty applied to the model.

Images