Grammar controllable video description generation method and system based on dynamic gating decoding

Abstract

The invention belongs to the technical field of computer vision and natural language processing, and provides a grammar controllable video description generation method and system based on dynamic gating decoding, and the method comprises the steps: obtaining a to-be-described video and an example statement; preprocessing the to-be-described video and the example statement; inputting the preprocessed to-be-described video and the sample statement into a pre-constructed grammar-controllable video description model to obtain a final description; wherein the grammar-controllable video description model is composed of a video encoder, a hierarchical graph grammar encoder and a dynamic gating decoder; a hierarchical graph grammar encoder is arranged, the hierarchical graph grammar encoder can model a grammar structure from a part-of-speech level to a phrase level and then to a sentence level step by step, the dependency relationship in a sentence is accurately captured, meanwhile, a video encoder is arranged, the spatial-temporal characteristics of a video can be effectively extracted and converted into high-level semantic representation, and the semantic representation is more accurate. And the high consistency of the generated description and the video content is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and natural language processing, and in particular relates to a method and system for generating grammatically controllable video description based on dynamic gated decoding. Background Art

[0002] Video description generation is a technology that automatically generates semantically and grammatically correct natural language descriptions by analyzing video content and combining it with grammatical structure.

[0003] Most video description generation methods can only generate fixed sentences based on fully trained model parameters, making the grammar and style of the generated sentences difficult to control and unable to flexibly adapt to different practical needs. Therefore, the task of grammatically controllable video description generation has emerged. Its goal is to reflect the grammatical structure of the example sentence while generating the description. Existing grammatically controllable video description generation technologies mainly include the following different model architectures:

[0004] 1. Encoding-decoding model based on long short-term memory network: By embedding sentences into part-of-speech sequences and using them as gating guides to control description decoding. This method oversimplifies the complexity of grammatical structure in language.

[0005] 2. Linear Parse Tree Model: This method converts the input sentence into a linear parse tree and generates grammatical control signals through linear encoding. However, this method inevitably loses important information in the language structure and cannot guarantee that the generated description is consistent with the expected grammatical rules. Summary of the Invention

[0006] In response to the problems in the background technology, the present invention proposes a grammatically controllable video description generation method and system based on dynamic gated decoding to generate a video description that conforms to grammatical rules and accurately reflects the main content and details of the video.

[0007] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0008] A method for generating grammatically controllable video description based on dynamic gated decoding, comprising:

[0009] Get the video to be described and the sample sentence;

[0010] Preprocessing the video to be described and the example sentences;

[0011] The pre-processed video to be described and the example sentence are input into a pre-built grammatically controllable video description model to obtain a final description; wherein the grammatically controllable video description model is composed of a video encoder, a layered graph grammar encoder and a dynamic gated decoder.

[0012] Preferably, constructing a grammatically controllable video description model includes:

[0013] Obtaining an original video and a description text of the original video;

[0014] Preprocessing the original video to obtain spatiotemporal features and visual features of each frame of the original video, and preprocessing the description text;

[0015] Allocating a number of pre-processed description texts to each of the original videos; wherein the pre-processed description texts serve as grammatical templates;

[0016] Based on a predefined loss function, the grammatical template and the spatiotemporal features and the visual features are used to train a controllable video description model to obtain a constructed grammatical controllable video description model.

[0017] Preferably, the grammatical template, the spatiotemporal features and the visual features are input into a controllable video description model to obtain a final video text;

[0018] The expression of the loss function is:

[0019]

[0020] Where, represents the description generation loss, represents the grammatical attention loss, represents the grammar node classification loss, represents the video reconstruction loss, λ1, λ2, λ3, and λ4 are weight coefficients respectively;

[0021] The description generation loss is used to calculate the difference between the final video text and the grammatical template;

[0022] The grammatical attention loss is used to calculate the difference between the dynamic gated decoder and the dependency tree of the grammatical template;

[0023] The grammar node classification loss is used to calculate the loss of the part-of-speech sequence of the grammar reconstructor reconstructing the grammar template based on the final video text;

[0024] The video reconstruction loss is used to calculate the loss of the video features of the original video reconstructed by the video reconstructor based on the fusion feature representation output by the dynamic gating decoder.

[0025] Preferably, the method of preprocessing the video to be described is the same as the method of preprocessing the original video;

[0026] The preprocessing of the original video to obtain the spatiotemporal features and visual features of each frame of the original video includes:

[0027] Determine whether the duration of the original video is less than a preset duration; if so, extend the original video to the preset duration by a loop filling method; if not, truncate the original video to the preset duration;

[0028] Sampling the original video of a predetermined duration according to a predetermined interval;

[0029] A pre-trained ResNet-101 is used to extract visual features of each frame of the sampled original video and normalize the visual features;

[0030] A pre-trained 3D-CNN is used to extract the spatiotemporal features of each frame of the sampled original video.

[0031] Preferably, the method of preprocessing the description text is the same as the method of preprocessing the example sentence;

[0032] The preprocessing of the description text includes:

[0033] Filtering the description text according to a preset sentence length;

[0034] Perform data cleaning on the filtered description text.

[0035] Preferably, the method for normalizing the visual features is L2 normalization.

[0036] Preferably, the step of inputting the pre-processed video to be described and the sample sentence into a pre-trained grammar-controllable video description model to obtain a final description of the video includes:

[0037] Parsing the preprocessed example sentences to obtain a dependency tree, converting the dependency tree into a directed graph, and constructing an arc-out mask matrix, an arc-in mask matrix, a dependency distance matrix, and a node feature matrix based on the directed graph;

[0038] Inputting the spatiotemporal features into a video encoder to obtain a video code; wherein the video encoder is composed of a multi-layer stacked Transformer encoding layer, a multi-head self-attention mechanism, and a feedforward network layer;

[0039] Inputting the node feature matrix into a hierarchical graph grammar encoder, combining the outgoing arc mask matrix, the incoming arc mask matrix, and the dependency distance matrix to obtain a sentence-level grammatical representation; wherein the hierarchical graph grammar encoder is composed of a hierarchical graph attention network and a dynamic tree position code, and the hierarchical graph attention network is composed of a bottom-level code, a middle-level code, and a top-level code;

[0040] The sentence-level grammatical representation and the video code are input into a dynamic gated decoder to obtain a final description of the video; wherein the dynamic gated decoder is composed of a gate unit, a Transformer decoding layer and a feedforward network layer.

[0041] Preferably, the node feature matrix is input into a hierarchical graph grammar encoder, and the outgoing arc mask matrix, the incoming arc mask matrix and the dependency distance matrix are combined to obtain a sentence-level grammar representation, including:

[0042] The node feature matrix is input into the underlying code, the connection relationship between the nodes is defined using the outgoing arc mask matrix and the incoming arc mask matrix, and the attention coefficient is calculated and the node features are updated through the multi-head graph attention mechanism to obtain the first feature matrix;

[0043] Inputting the first feature matrix into the middle-level encoding to perform phrase node aggregation and generate phrase-level features;

[0044] The phrase-level features are input into the top-level encoding so that the top-level encoding calculates the attention weights between all nodes in the phrase-level features through the global attention mechanism, and adjusts the attention weights in combination with the dynamic tree position encoding and the dependency distance matrix to output the sentence-level grammatical representation.

[0045] Preferably, the step of inputting the sentence-level grammatical representation and the video code into a dynamic gated decoder to obtain a final description of the video comprises:

[0046] Generate context-aware features through the self-attention mechanism of the Transformer decoding layer;

[0047] Compute grammatical cross-attention based on context-aware features and sentence-level grammatical representation;

[0048] Compute visual cross-attention based on feature vectors and video encoding;

[0049] The gate value is dynamically calculated through the Sigmoid function to balance the weights of visual cross-attention and grammatical cross-attention;

[0050] The visual cross attention and the grammatical cross attention are weightedly fused based on the gating value through the gating unit, and the fused feature representation is output;

[0051] The fused feature representation is mapped to the vocabulary space through a feedforward network to generate the final description of the video.

[0052] The present invention also provides a grammatically controllable video description generation system based on dynamic gated decoding, which is applicable to the above-mentioned grammatically controllable video description generation method based on dynamic gated decoding, including:

[0053] A data acquisition module is used to obtain the video to be described and sample sentences;

[0054] A data preprocessing module, used for preprocessing the video to be described and the example sentences;

[0055] A generation module is used to input the preprocessed video to be described and the example sentence into a pre-built grammatically controllable video description model to obtain a final description; wherein the grammatically controllable video description model is composed of a video encoder, a layered graph grammar encoder and a dynamic gated decoder.

[0056] Beneficial effects of the present invention:

[0057] The method of the present invention sets a hierarchical graph grammar encoder, which can gradually model the grammatical structure from the part-of-speech level to the phrase level and then to the sentence level, accurately capturing the dependency relationship within the sentence. At the same time, by setting a video encoder, it can effectively extract the spatiotemporal features of the video and convert them into high-level semantic representations, ensuring a high degree of consistency between the generated description and the video content, thereby achieving the capture of complex visual information and enhancing semantic consistency; in addition, by setting a dynamic gated decoder, it can dynamically balance the weights of visual cross-attention and grammatical cross-attention, adjust the attention allocation in the generation process according to context-aware features, sentence-level grammatical representations and video coding features, thereby achieving the optimization of the semantic coherence and diversity of the generated description, ensuring that the final generated description not only conforms to grammatical rules but also accurately reflects the main content and details of the video.

[0058] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 The flowchart of the method for generating grammatically controllable video description based on dynamic gated decoding of the present invention is shown.

[0061] Figure 2 The following figure shows the comparison of the exemplary video description generation effect of the present invention. (a) shows the comparison of the exemplary video description generation effect of the present invention. Figure 1, (b) shows the comparison of the exemplary video description generation effect of the present invention Figure 2 .

[0062] Figure 3 The diagram shows the architecture of the grammar-controllable video description generation of the present invention.

[0063] Figure 4 Shown is a device framework diagram of the present invention.

[0064] Figure 5 Shown is a schematic diagram of the device structure of the present invention. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0066] Reference Figure 1 As shown, the present disclosure proposes a method for generating grammatically controllable video description based on dynamic gated decoding, which specifically includes the following steps:

[0067] S1. Obtain the video to be described and the sample sentence;

[0068] S2, preprocessing the video to be described and the sample sentences;

[0069] S3. Input the preprocessed video to be described and the example sentence into a pre-built grammatically controllable video description model to obtain the final description; wherein the grammatically controllable video description model consists of a video encoder, a layered graph grammar encoder and a dynamic gated decoder.

[0070] In the above step S3, the method for constructing a syntax-controllable video description model based on dynamic gated decoding specifically includes the following steps:

[0071] S311, obtaining the original video, the description text of the original video, and the sample sentence;

[0072] Specifically, the description text of the original video can be downloaded from the Shutterstock corpus, which covers a variety of scenes (such as daily activities, sports, natural landscapes, etc.).

[0073] S312, preprocessing the original video to obtain the spatiotemporal features and visual features of each frame of the original video, and preprocessing the description text;

[0074] In the above steps S2 and S312, the method for preprocessing the video to be described is the same as the method for preprocessing the original video. In the above step S312, the method for preprocessing the original video to obtain the spatiotemporal features and visual features of each frame of the original video specifically includes the following steps:

[0075] S3121, determining whether the duration of the original video is less than a preset duration;

[0076] If so, the original video is extended to a preset duration by a loop filling method;

[0077] If not, the original video will be truncated to the preset length;

[0078] S3122. Sampling the original video according to a preset interval to ensure coverage of key actions and scene changes;

[0079] S3123. Use the pre-trained ResNet-101 to extract the visual features of each frame of the sampled original video and normalize the visual features.

[0080] Among them, ResNet-101 (deep learning network) is trained through the existing ILSVRC-2012-CLS dataset to obtain the trained ResNet-101.

[0081] Specifically, a pre-trained ResNet-101 is used to extract 2048-dimensional visual features of each frame of the sampled original video. In this embodiment, the number of sampled video frames is 30.

[0082] Furthermore, the method for normalizing the visual features is L2 normalization to ensure numerical stability.

[0083] S3124. Use the pre-trained 3D-CNN to extract the spatiotemporal features of each frame of the sampled original video.

[0084] Specifically, the 3D-CNN (3D convolutional neural network) is a SlowFast network, that is, the spatiotemporal features of each frame of the sampled original video are extracted through the SlowFast network, and the output dimension is T×512, where T is the time step, which is consistent with the number of video frames. In this embodiment, T=30.

[0085] For example, the preset duration is 30 seconds. Videos shorter than 30 seconds are extended to 30 seconds through loop filling, and videos to be described that are longer than 30 seconds are truncated to 30 seconds to ensure input consistency. The preset interval is 1 frame per second, that is, 30 frames are sampled. It should be further explained that the preset duration can be adaptively adjusted according to actual conditions. The 30 seconds and 1 frame per second in the example are only for illustrative purposes and are not meant to be limited to this specific value. In actual applications, the preset duration and preset interval can be appropriately increased or decreased according to the characteristics of the video content to be described, the requirements of the application scenario, or the status of computing resources to achieve optimal processing effect and efficiency. For example, when processing short video content, it may be necessary to set the preset duration and preset interval to be shorter to improve the accuracy of the description; when analyzing longer continuous shots, it may be necessary to select a longer preset duration and preset interval to capture more contextual information.

[0086] In the above step S312 and step S2, the method for preprocessing the description text is the same as the method for preprocessing the example sentence. The method for preprocessing the description text specifically includes the following steps:

[0087] (a) Filtering description texts according to preset sentence lengths;

[0088] Exemplarily, the preset sentence length is 8-30 words, that is, description texts with a sentence length of 8-30 words are retained, description texts with less than 8 words are filtered, and description texts with more than 30 words are trimmed to the first 30 words to ensure uniform length of the input sequence.

[0089] (b) Perform data cleaning on the filtered description text.

[0090] Specifically, the filtered description text is deduplicated using text hash values to remove duplicate descriptions, and the SymSpell tool is used to correct spelling errors to improve text quality.

[0091] As an exemplary embodiment of the present disclosure, in the above step S311, 2,322,628 description texts of original videos are downloaded from the Shutterstock corpus. After preprocessing the description texts, 761,582 description texts are finally retained.

[0092] S313, assigning a number of description texts to each original video; wherein the description texts serve as grammatical templates;

[0093] Specifically, a number of description texts are randomly assigned to each original video.

[0094] S314: Based on a predefined loss function, a controllable video description model is trained using a grammatical template and spatiotemporal features and visual features to obtain a constructed grammatical controllable video description model, which is expressed as:

[0095]

[0096] Where, represents the description generation loss, represents the grammatical attention loss, represents the grammar node classification loss, represents the video reconstruction loss, λ1, λ2, λ3, and λ4 are weight coefficients respectively;

[0097] The description generation loss is used to calculate the difference between the final video text and the grammatical template. Where y t represents the word at time step t, V represents the visual feature, T represents the grammatical template, p(y t |y 1:t-1 ,V,T) represents the current word y t probability.

[0098] The grammatical attention loss is used to calculate the difference between the dependency tree of the dynamic gated decoder and the grammatical template. Where A t represents the average multi-head attention matrix, A represents the dependency tree of the grammatical template, and MSE represents the mean square error.

[0099] The grammar node classification loss is used to calculate the loss of the part-of-speech sequence of the grammar reconstructor to reconstruct the grammar template based on the final video text. Where z t represents the part-of-speech tag at time step t, p(z t |z 1:t-1 ,V,T) represents the current part-of-speech tag z t probability.

[0100] The video reconstruction loss is used to calculate the loss of the video features of the original video reconstructed by the video reconstructor based on the fusion feature representation output by the dynamic gated decoder. Where, v i represents the i-th feature vector of the original video, m represents the number of feature vectors, Euclidean represents the Euclidean distance, represents the i-th video feature vector reconstructed based on the fusion feature representation output by the dynamic gating decoder.

[0101] Through a multi-task loss function (description generation loss, grammatical node classification loss, video reconstruction loss, and grammatical attention loss), the description generation loss is used to measure the difference between the generated description and the true description, ensuring that the output text has high semantic accuracy; the grammatical node classification loss focuses on optimizing the syntactic similarity between the generated description and the example sentence, thereby enhancing the correctness of the grammatical structure. At the same time, the video reconstruction loss aims to ensure that the generated description can accurately reflect the video content and improve the semantic consistency of the description. Finally, the grammatical attention loss is used to adjust the model's attention to grammatical structure, further ensuring that the generated description not only conforms to grammatical rules but also captures key details and actions in the original video. Through this comprehensive optimization method, the grammatically controllable video description generation model is able to generate high-quality and semantically coherent video descriptions while maintaining grammatical correctness.

[0102] Furthermore, in the above step S3, the pre-processed video to be described and the sample sentence are input into a pre-trained grammar-controllable video description model to obtain the final description of the video, specifically including:

[0103] S321. Use the Spacy toolkit to parse the grammar template to obtain a dependency tree, convert the dependency tree into a directed graph, and construct an arc mask matrix, an incoming arc mask matrix, a dependency distance matrix, and a node feature matrix based on the directed graph;

[0104] The dependency tree consists of nodes and edges. Nodes contain information such as part-of-speech tags (e.g., noun, verb) and phrase components (e.g., NP, VP). Edges represent dependencies between words, running from the central word to the dependent word. Each edge has a weight, which represents the dependency probability. When the dependency tree is converted into a directed graph, the nodes in the graph represent words, the edges represent dependency relationships, and the node features represent part-of-speech tags.

[0105] Dependency distance matrix D∈R N×N , D ij It represents the shortest path length from node i to node j and is used to capture long-distance dependencies (such as clause structure). R is a real number space, indicating that the elements in the matrix are real numbers.

[0106] Arc mask matrix A out ∈{0,1} N×N , the elements in the matrix are binary values, indicating whether there is an outgoing arc from node i to node j (i.e., whether node i points to node j).

[0107] Arc mask matrix A in ∈{0,1} N×N, N is the number of words in the sentence (number of nodes), and the elements in the matrix are binary values, indicating whether there is an incoming arc from node j to node i (i.e., whether node j points to node i).

[0108] When constructing the outgoing arc mask matrix and the incoming arc mask matrix, only the existence of the dependency relationship is retained (binarization) and the specific type (such as subject-predicate, verb-object) is ignored to avoid the propagation of incorrect labels.

[0109] S322, inputting the spatiotemporal features into a video encoder to obtain a video code; wherein the video encoder is composed of 6 stacked Transformer encoding layers, a multi-head self-attention mechanism, and a feedforward network layer;

[0110] Specifically, the spatiotemporal features are linearly transformed to obtain the Q, K, V matrix, and then the attention score of the Q, K, V matrix is calculated, which is expressed as: Where Q, K, and V represent query, key, and value matrices respectively. k Represents the dimension of the key vector; the multi-head self-attention mechanism sets 8 attention heads, and the dimension of each head is 64. The outputs of the 8 heads are spliced on the feature dimension to obtain a 768-dimensional output. Then the multi-head attention output is input into the feedforward network layer for processing, and the final output is the video code H v ∈R T×768 ; Among them, the hidden layer dimension of the feedforward network layer is 2048, and the activation function is the ReLU function.

[0111] S323. Input the node feature matrix into a hierarchical graph grammar encoder to obtain a sentence-level grammatical representation; wherein the hierarchical graph grammar encoder is composed of a hierarchical graph attention network and a dynamic tree position encoding, and the hierarchical graph attention network is composed of a bottom layer encoding, a middle layer encoding, and a top layer encoding;

[0112] Specifically, first, the node feature matrix H (0) ∈R N×d Input into the underlying code, use the arc mask matrix and arc mask matrix to define the connection relationship between nodes, calculate the attention coefficient through the multi-head graph attention mechanism (4 heads) and update the node features to obtain the first feature matrix H (1) ∈R N×d , where H (1) is the feature matrix of the bottom-level encoding output, R is a set of real numbers, d'=4×d represents the output feature dimension of each node, d represents the initial feature dimension of each node, and N represents the number of nodes;

[0113] Among them, the expression for calculating the attention coefficient is:

[0114] α ij=Softmax(e ij )

[0115] Where, e ij =LeakyReLU(W a [h i ||h j ]) represents the unnormalized value of the attention coefficient between node i and node j, W a is a learnable parameter, h i represents the characteristics of node i, h j represents the feature of node j, || represents the splicing operation, α ij represents the normalized attention weight.

[0116] Among them, the expression for updating node characteristics is: Where, represents the output of 4 heads concatenated along the feature dimension (output dimension d'=4×d), represents the normalized weight of the kth attention head after normalization, W k ∈R 64*d represents the learnable weight matrix, represents the initial features of node j, N(i) represents the neighbor set of node i, represents the updated node features, and δ represents the RELU activation function.

[0117] Then the first feature matrix H of the bottom layer encoding output (1) ∈R N×d Input into the middle layer code to aggregate phrase nodes and generate phrase-level features. For each phrase (such as a noun phrase NP), the feature vectors of all word nodes contained in the phrase are average pooled to obtain the phrase-level feature vector. Where, is the updated feature of node i, i∈NP means that node i belongs to the noun phrase NP, MeanPool represents the mean pooling function, and updates the arc mask matrix and the arc mask matrix, and then combines the feature vectors of all phrases into a phrase-level

[0118] Feature matrix H (2) ∈R M×d , where M is the number of phrase nodes.

[0119] Finally, the phrase-level feature matrix H (2)The input is sent to the top-level encoder, which fuses phrase-level features through global attention, combines dynamic position encoding and dependency distance weighting, and outputs a sentence-level grammatical representation containing part-of-speech, phrase, and sentence-level grammatical information. Specifically, the top-level encoder calculates the attention weights between all nodes in the phrase-level features through the global attention mechanism, and then outputs the sentence-level grammatical representation. The expression for calculating the attention weights between all nodes in the phrase-level features through the global attention mechanism is: Where W q and W k represents the learnable query,key matrix, T represents the transpose symbol, is the scaling factor, and Softmax is the normalization function. At the same time, combined with dynamic tree position encoding, according to the depth P of the node in the syntax tree i Generate hierarchical embeddings i ), calculate the relative position offset Δp for each node (i, j) ij =p i -p j , and mapped into an embedding vector, fusing the positional encoding with the grammatical features into And adjust the attention weight α through the dependency distance matrix D ij =α ij ·exp(-λD ij ), where λ is the attenuation coefficient. Finally, the final sentence-level grammatical representation H is output (F) =H (3) ∈R N×d′ , which contains part-of-speech level features, phrase-level structure and sentence-level grammatical pattern information.

[0120] S324. Input the sentence-level grammatical representation and video encoding into a dynamic gated decoder to obtain a final description of the video; wherein the dynamic gated decoder is composed of a gate unit, a Transformer decoding layer, and a linear projection layer.

[0121] Specifically, S3241 generates context-aware features h through the self-attention mechanism y , whose expression is:

[0122] h y =SelfAtt(Q,K,V←W)

[0123] In the formula, the input matrix is the real word embedding sequence, n is the sequence length, SelfAtt() is the multi-head self-attention mechanism, where Q / K / V are homology projections: Q = W × W Q , K=W×W K , V=W×W V , the projection matrix W can be learnedQ , W K ,

[0124] S3242, based on context-aware feature h y and sentence-level grammatical representation H (F) Computational Grammar Cross-Attention Its expression is:

[0125]

[0126] The output h of the decoder self-attention layer y The query is input to Q and the sentence-level grammatical representation is input to K and V.

[0127] S3243, according to the characteristic vector h y Computational visual cross attention with video encoding Its expression is:

[0128]

[0129] S3244. Dynamically calculate the gate value g∈[0,1] through the Sigmoid function to balance the weights of visual cross-attention and grammatical cross-attention. The expression is:

[0130]

[0131] Where σ is the Sigmoid function, W g is the gating weight matrix, It is to splice the output of visual cross attention and the output of grammatical cross attention;

[0132] S3245, the visual cross attention and the grammatical cross attention are weighted fused based on the gate value, and the fusion feature representation H is output. fusion , whose expression is:

[0133]

[0134] S3246, through the feedforward network to represent the fusion feature H fusion Mapped to the word table space, the final description of the video is generated, and its expression is:

[0135] P(w t )=Softmax(W o H fusion +b o )

[0136] Where, the bias vector and the projection matrix (V is the vocabulary size, d = 256) are all learnable parameters, and the Softmax normalized exponential function is applied to generate the vocabulary probability distribution.

[0137] By setting up a hierarchical graph grammar encoder, the hierarchical graph grammar encoder can gradually model the grammatical structure from the part-of-speech level to the phrase level and then to the sentence level, accurately capturing the dependencies within the sentence. At the same time, by setting up a video encoder, it can effectively extract the spatiotemporal features of the video and convert them into high-level semantic representations, ensuring a high degree of consistency between the generated description and the video content, thereby achieving the capture of complex visual information and enhancing semantic consistency. In addition, by setting up a dynamic gated decoder, it can dynamically balance the weights of visual cross-attention and grammatical cross-attention, and adjust the attention allocation in the generation process according to context-aware features, sentence-level grammatical representations and video coding features, thereby optimizing the semantic coherence and diversity of the generated description, ensuring that the final generated description not only conforms to grammatical rules but also accurately reflects the main content and details of the video.

[0138] This grammatically controllable video description generation model can generate grammatically standardized, easy-to-understand video descriptions for hearing-impaired users, improving information acquisition efficiency. It also supports on-demand generation of descriptions in specific styles (such as passive voice and interrogative sentences), enhancing the expressiveness and professionalism of the video. Furthermore, by replacing grammatical templates, it can quickly adapt to the syntactic structures of different languages, reducing the cost of cross-language content generation.

[0139] Reference Figure 2 As shown in (a), Figure 2 The top of (a) shows six consecutive video frames showing different expressions and movements of the animated character SpongeBob.

[0140] Groundtruth caption: The true description is "A song is playing and SpongeBob is playing in the background."

[0141] Seq2Seq generated caption: The description generated using the Seq2Seq model is "A cartoon is singing".

[0142] Exemplar Sentence 1: The example sentence 1 is “Cute girl with skateboard in the city park”.

[0143] Ours syntax customized caption1: Ours syntax customized caption1 is “SpongeBob SquarePants from the animated show”.

[0144] Exemplar Sentence2: Exemplar sentence 2 is “A group of fresh carrots in the fruit market.”

[0145] Ours syntax customized caption2: Syntax customized caption 2 is "Cartoon with cartoon character SpongeBob SquarePants"

[0146] Reference Figure 2 As shown in (b), Figure 2 The top portion of (b) in FIG. 5 shows a plurality of consecutive video frames showing a group of children and teenagers playing football on a grass field.

[0147] Groundtruth Caption: This is a manually annotated true description that accurately reflects the video content.

[0148] Seq2Seq Generated Caption: This is a description generated by a standard Seq2Seq model. Although it is generally correct, it is not specific enough.

[0149] Exemplar Sentences: These example sentences provide different types of contextual information to guide the model to generate more specific descriptions.

[0150] Syntax Customized Captions: These are grammatically customized captions designed to combine the grammatical structure and vocabulary in the example sentences to produce more detailed and contextual descriptions.

[0151] It can be seen that compared with Groundtruth Caption, Seq2Seq Generated Caption is more generalized and Groundtruth Caption is more specific.

[0152] Syntax Customized Captions attempts to combine the grammatical structure and vocabulary of the example sentences to generate more specific and contextual descriptions. For example, "Several young boys practicing different soccer by kicking in soccer in a grassy field" and "Young boy in soccer uniforms run around outside field" both attempt to provide more detailed descriptions and incorporate innovative grammar and vocabulary.

[0153] Reference Figure 4 As shown, based on the same inventive concept as the above method, the present disclosure also proposes a syntax-controllable video description generation system based on dynamic gated decoding, comprising:

[0154] A data acquisition module 110 is used to acquire a video to be described and an example sentence;

[0155] A data preprocessing module 120 is used to preprocess the video to be described and the example sentences;

[0156] The generation module 130 is used to input the pre-processed video to be described and the sample sentence into the pre-built grammatically controllable video description model to obtain the final description; wherein the grammatically controllable video description model is composed of a video encoder, a layered graph grammar encoder and a dynamic gated decoder.

[0157] Reference Figure 5 As shown, based on the same inventive concept as the above method, the present disclosure also proposes a device, including a memory and a processor, wherein the memory stores computer instructions that can be run on the processor, and when the processor runs the computer instructions, it executes the above-mentioned method for generating grammatically controllable video description based on dynamic gated decoding.

[0158] Based on the same inventive concept as the above method, the present disclosure also proposes a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed, the above-mentioned method for generating a grammatically controllable video description based on dynamic gated decoding can be implemented.

[0159] Any reference to memory, storage, database, or other media used in the embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0160] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or elements that are inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0161] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating grammatically controllable video description based on dynamic gated decoding, characterized in that: include: Get the video to be described and the sample sentence; Preprocessing the video to be described and the example sentences; The pre-processed video to be described and the example sentence are input into a pre-built grammatically controllable video description model to obtain a final description; wherein the grammatically controllable video description model is composed of a video encoder, a layered graph grammar encoder and a dynamic gated decoder.

2. The method for generating grammatically controllable video description based on dynamic gated decoding according to claim 1, characterized in that: Build a grammatically controllable video description model, including: Obtaining an original video and a description text of the original video; Preprocessing the original video to obtain spatiotemporal features and visual features of each frame of the original video, and preprocessing the description text; Allocating a number of pre-processed description texts to each of the original videos; wherein the pre-processed description texts serve as grammatical templates; Based on a predefined loss function, the grammatical template and the spatiotemporal features and the visual features are used to train a controllable video description model to obtain a constructed grammatical controllable video description model.

3. The method for generating grammatically controllable video description based on dynamic gated decoding according to claim 2, characterized in that: Inputting the grammatical template, spatiotemporal features and visual features into a controllable video description model to obtain a final video text; The expression of the loss function is: Where, represents the description generation loss, represents the grammatical attention loss, represents the grammar node classification loss, represents the video reconstruction loss, λ1, λ2, λ3, and λ4 are weight coefficients respectively; The description generation loss is used to calculate the difference between the final video text and the grammatical template; The grammatical attention loss is used to calculate the difference between the dynamic gated decoder and the dependency tree of the grammatical template; The grammar node classification loss is used to calculate the loss of the part-of-speech sequence of the grammar reconstructor reconstructing the grammar template based on the final video text; The video reconstruction loss is used to calculate the loss of the video features of the original video reconstructed by the video reconstructor based on the fusion feature representation output by the dynamic gating decoder.

4. The method for generating grammatically controllable video description based on dynamic gated decoding according to claim 2, wherein: The method of preprocessing the video to be described is the same as the method of preprocessing the original video; The preprocessing of the original video to obtain the spatiotemporal features and visual features of each frame of the original video includes: Determine whether the duration of the original video is less than a preset duration; if so, extend the original video to the preset duration by a loop filling method; if not, truncate the original video to the preset duration; Sampling the original video of a predetermined duration according to a predetermined interval; A pre-trained ResNet-101 is used to extract visual features of each frame of the sampled original video and normalize the visual features; A pre-trained 3D-CNN is used to extract the spatiotemporal features of each frame of the sampled original video.

5. The method for generating grammatically controllable video description based on dynamic gated decoding according to claim 2, characterized in that: The method of preprocessing the description text is the same as that of preprocessing the sample sentence; The preprocessing of the description text includes: Filtering the description text according to a preset sentence length; Perform data cleaning on the filtered description text.

6. The method for generating grammatically controllable video description based on dynamic gated decoding according to claim 4, characterized in that: The method for normalizing the visual features is L2 normalization.

7. The method for generating grammatically controllable video description based on dynamic gated decoding according to claim 1, characterized in that: The pre-processed video to be described and the example sentence are input into a pre-trained grammar-controllable video description model to obtain a final description of the video, including: Parsing the preprocessed example sentences to obtain a dependency tree, converting the dependency tree into a directed graph, and constructing an arc-out mask matrix, an arc-in mask matrix, a dependency distance matrix, and a node feature matrix based on the directed graph; Inputting the spatiotemporal features into a video encoder to obtain a video code; wherein the video encoder is composed of a multi-layer stacked Transformer encoding layer, a multi-head self-attention mechanism, and a feedforward network layer; Inputting the node feature matrix into a hierarchical graph grammar encoder, combining the outgoing arc mask matrix, the incoming arc mask matrix, and the dependency distance matrix to obtain a sentence-level grammatical representation; wherein the hierarchical graph grammar encoder is composed of a hierarchical graph attention network and a dynamic tree position code, and the hierarchical graph attention network is composed of a bottom-level code, a middle-level code, and a top-level code; The sentence-level grammatical representation and the video code are input into a dynamic gated decoder to obtain a final description of the video; wherein the dynamic gated decoder is composed of a gate unit, a Transformer decoding layer and a feedforward network layer.

8. The method for generating grammatically controllable video description based on dynamic gated decoding according to claim 7, characterized in that: The node feature matrix is input into the hierarchical graph grammar encoder, and the outgoing arc mask matrix, the incoming arc mask matrix and the dependency distance matrix are combined to obtain a sentence-level grammar representation, including: The node feature matrix is input into the underlying code, the connection relationship between the nodes is defined using the outgoing arc mask matrix and the incoming arc mask matrix, and the attention coefficient is calculated and the node features are updated through the multi-head graph attention mechanism to obtain the first feature matrix; Inputting the first feature matrix into the middle-level encoding to perform phrase node aggregation and generate phrase-level features; The phrase-level features are input into the top-level encoding so that the top-level encoding calculates the attention weights between all nodes in the phrase-level features through the global attention mechanism, and adjusts the attention weights in combination with the dynamic tree position encoding and the dependency distance matrix to output the sentence-level grammatical representation.

9. The method for generating grammatically controllable video description based on dynamic gated decoding according to claim 7, characterized in that: The step of inputting the sentence-level grammatical representation and the video code into a dynamic gated decoder to obtain a final description of the video includes: Generate context-aware features through the self-attention mechanism of the Transformer decoding layer; Compute syntactic cross-attention based on context-aware features and sentence-level grammatical representations; Compute visual cross-attention based on feature vectors and video encoding; The gate value is dynamically calculated through the Sigmoid function to balance the weights of visual cross-attention and grammatical cross-attention; The visual cross attention and the grammatical cross attention are weightedly fused based on the gating value through the gating unit, and the fused feature representation is output; The fused feature representation is mapped to the vocabulary space through a feedforward network to generate the final description of the video.

10. A system for generating grammatically controllable video description based on dynamic gated decoding, applicable to the method for generating grammatically controllable video description based on dynamic gated decoding according to any one of claims 1 to 9, characterized in that: include: A data acquisition module is used to obtain the video to be described and sample sentences; A data preprocessing module, used for preprocessing the video to be described and the example sentences; A generation module is used to input the preprocessed video to be described and the example sentence into a pre-built grammatically controllable video description model to obtain a final description; wherein the grammatically controllable video description model is composed of a video encoder, a layered graph grammar encoder and a dynamic gated decoder.

Images