Methods, devices, electronic devices, and storage media for video content understanding based on structured grammar information.

Abstract

This application, in the field of computer application technology, discloses a method, apparatus, electronic device, and storage medium for video content understanding based on structured grammatical information. The method includes: inputting training samples into a target model for content understanding processing to obtain predicted text, and constructing a syntax tree corresponding to the predicted text; calculating the syntax tree edit distance between the syntax tree and the syntax tree of the reference text; calculating the language structure loss using the syntax tree edit distance, and updating the model parameters of the target model using the language structure loss; acquiring the target video after training the model parameters; and inputting the target video into the target model for processing to obtain the content text of the target video. In this application, the target model is trained based on the language structure loss calculated from the syntax tree, enabling it to understand the target video and thus obtain content text that is grammatically accurate, has a reasonable sentence structure, and is natural.

Description

Technical Field

[0001] This application relates to the field of computer application technology, and in particular to methods, apparatus, electronic devices and storage media for video content understanding based on structured grammar information. Background Technology

[0002] Video content understanding refers to the technology of automatically converting video content into natural language text, thereby enabling the automatic generation of subtitles based on the natural language text.

[0003] However, current methods for converting video content into natural language text often result in grammatical errors, loose sentence structures, and unnatural text descriptions, leading to subtitle generation that is ineffective.

[0004] In conclusion, how to effectively solve the problem of content understanding of video content and obtain content text that meets expectations is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for video content understanding based on structured grammatical information, so as to obtain content text that conforms to the grammatical structure when performing content understanding on video.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0007] A video content understanding method based on structured grammatical information includes:

[0008] The training samples are input into the target model for content understanding processing to obtain the predicted text, and the syntax tree corresponding to the predicted text is constructed.

[0009] Calculate the syntactic tree edit distance between the syntactic tree and the syntactic tree of the reference text;

[0010] The language structure loss is calculated using the edit distance of the syntactic tree, and the model parameters of the target model are updated using the language structure loss.

[0011] After training the model parameters of the target model, acquire the target video;

[0012] The target video is input into the target model for processing to obtain the content text of the target video.

[0013] Preferably, constructing the syntax tree corresponding to the predicted text includes:

[0014] A grammatical structure representation is constructed using the grammatical components of the predicted text and the grammatical relationships between these components;

[0015] Based on the grammatical structure representation, a tree structure with words as nodes and grammatical relations as edges is constructed to obtain the syntax tree; wherein, the syntax tree includes annotations of the grammatical roles and hierarchical relationships of each node.

[0016] Preferably, calculating the syntactic tree edit distance between the syntactic tree and the syntactic tree of the reference text includes:

[0017] Align the nodes of two syntax trees and calculate the minimum cost of editing operations required to make the two trees identical, wherein the editing operations include at least one of node replacement, deletion and insertion.

[0018] The minimum edit operation cost is determined as the syntax tree edit distance.

[0019] Preferably, the target model includes an audio encoder, a visual encoder, a cross-modal fusion module, a structure-aware encoder, and a decoder. The target video is input into the target model for processing to obtain the content text of the target video, including:

[0020] The audio features of the target video are extracted using the audio encoder, and the visual features of the target video are extracted using the visual encoder in the target model.

[0021] The cross-modal fusion module is used to fuse the audio features and the visual features to obtain grammar-aware multimodal features;

[0022] The structure-aware encoder is used to encode the multimodal features to obtain a syntax-aligned cross-modal joint representation;

[0023] The content text is obtained by decoding the cross-modal joint representation using the decoder.

[0024] Preferably, after obtaining the content text of the target video, the method further includes:

[0025] The content text and the target video are combined to obtain an output video with subtitles.

[0026] Preferably, updating the model parameters of the target model using the language structure loss includes:

[0027] Calculate the autoregressive language modeling loss corresponding to the predicted text and the reference text;

[0028] Calculate the masked language modeling loss corresponding to the predicted text and the reference text;

[0029] The model parameters are adjusted by combining the autoregressive language modeling loss, the masked language modeling loss, and the language structure loss.

[0030] Preferably, calculating the autoregressive language modeling loss corresponding to the predicted text and the reference text includes:

[0031] In the process of generating the predicted text autoregressively using the decoder, the masked language modeling loss between the predicted word and the real word is calculated at each step; wherein, the predicted word is the word in the predicted text, and the real word is the word in the reference text.

[0032] A video content understanding device based on structured grammatical information includes:

[0033] The model training module is used to input training samples into the target model for content understanding processing to obtain predicted text and construct the syntax tree corresponding to the predicted text; calculate the syntax tree edit distance between the syntax tree and the syntax tree of the reference text; calculate the language structure loss using the syntax tree edit distance; and update the model parameters of the target model using the language structure loss.

[0034] The video acquisition module is used to acquire the target video after the model parameters of the target model have been trained.

[0035] The prediction application module is used to input the target video into the target model for processing to obtain the content text of the target video.

[0036] An electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is used to implement the steps of the video content understanding method based on structured grammar information described above when executing the computer program.

[0039] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video content understanding method based on structured grammar information described above.

[0040] Applying the method provided in the embodiments of this application, training samples are input into the target model for content understanding processing to obtain predicted text, and a syntax tree corresponding to the predicted text is constructed; the syntax tree edit distance between the syntax tree and the syntax tree of the reference text is calculated; the language structure loss is calculated using the syntax tree edit distance, and the model parameters of the target model are updated using the language structure loss; with the model parameters of the target model trained, the target video is acquired; the target video is input into the target model for processing to obtain the content text of the target video.

[0041] This application aims to enhance video content understanding and natural language description generation capabilities by leveraging syntactic structure information, thereby obtaining a multimodal content understanding model with accurate semantic expression, reasonable grammatical structure, and coherent content logic. First, the target model is trained based on training samples. That is, the target model is used to perform content understanding on the training samples to obtain predicted text. To enable the target model to output text that better conforms to grammatical structure, this application constructs a corresponding syntax tree for the predicted text. The language structure loss is determined based on the syntactic tree edit distance between the syntax trees corresponding to the predicted text and the reference text. The model reference of the target model is adjusted based on the language structure loss, enabling the target model, after training, to perform content recognition on input videos and output content text that conforms to grammatical structure. After training, the target video for which content understanding is required can be obtained and input into the target model for content understanding, thus obtaining content text that conforms to grammatical structure. Because the target model is trained using the language structure loss calculated based on the syntax tree during model training, the resulting content text possesses the characteristics of grammatical accuracy, reasonable sentence structure, and naturalness.

[0042] Accordingly, embodiments of this application also provide a video content understanding apparatus, device, and readable storage medium based on structured grammar information, corresponding to the above-described video content understanding method based on structured grammar information, which have the aforementioned technical effects and will not be elaborated further here. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating an implementation of a video content understanding method based on structured grammatical information in this application.

[0045] Figure 2 This is a schematic diagram of the structure of a video content understanding device based on structured grammar information in an embodiment of this application;

[0046] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0047] Figure 4 This is a schematic diagram of the specific structure of an electronic device in an embodiment of this application. Detailed Implementation

[0048] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] Please refer to Figure 1 , Figure 1 This is a flowchart of a video content understanding method based on structured grammatical information according to an embodiment of this application. The method includes the following steps.

[0050] S101. Input the training samples into the target model for content understanding processing to obtain the predicted text, and construct the syntax tree corresponding to the predicted text.

[0051] The training samples can be videos with reference text, which is the textual expression of the content corresponding to the video. The video can be a video consisting of only multiple frames of images, or an audio-visual video consisting of multiple frames of images and audio.

[0052] The target model is one that can understand the content of an input video and output the content text. For clarity, in this embodiment, the content text output by the model during the training phase is referred to as the predicted text.

[0053] Once the predicted text is obtained, the corresponding syntax tree can be constructed, which facilitates the analysis of the syntactic structure of the predicted text based on the syntax tree.

[0054] In one specific embodiment of this application, constructing a syntax tree corresponding to the predicted text includes: constructing a grammatical structure representation using the grammatical components of the predicted text and the grammatical relationships between the grammatical components; constructing a tree structure with words as nodes and grammatical relationships as edges based on the grammatical structure representation to obtain the syntax tree; wherein, the syntax tree includes annotations of the grammatical roles and hierarchical relationships of each node.

[0055] In other words, a grammatical structure representation can be constructed based on the grammatical components of the predicted text and the grammatical relationships between them, such as dependencies. Then, a tree structure with words as nodes and grammatical relationships as edges is constructed based on the grammatical structure representation. This tree structure is the syntax tree. In particular, in this embodiment, the syntax tree also has annotations for the grammatical roles and hierarchical relationships of each node. For example, a grammatical role can be any of the roles in subject, predicate, object, attributive, adverbial, and complement. The hierarchical relationship can correspond to the following layers: morpheme layer, the smallest grammatical / semantic unit, including free morphemes and attached morphemes (such as prefixes and suffixes); word layer, units composed of words, such as nouns and verbs, with independent meaning and grammatical function; phrase layer, composed of words combined through grammatical relationships; clause layer, linguistic units expressing relatively complete meaning, such as absolute constructions; and sentence layer, the highest grammatical unit, composed of clauses, expressing complete ideas.

[0056] S102. Calculate the syntax tree edit distance between the syntax tree and the syntax tree of the reference text.

[0057] The syntax tree of the reference text is constructed in the same way as the syntax tree of the predicted text. The syntax tree of the predicted text can be constructed in advance or during training.

[0058] By comparing and calculating the syntax trees of the predicted text and the reference text, the syntax tree edit distance between the two syntax trees can be determined. The syntax tree edit distance refers to the distance of the editing operations required to edit the two syntax trees to have the same structure.

[0059] In one specific embodiment of this application, calculating the syntactic tree edit distance between the syntactic tree and the syntactic tree of the reference text includes: aligning the nodes of the two syntactic trees and calculating the minimum edit operation cost required to make the two trees identical, wherein the edit operation includes at least one of node replacement, deletion and insertion; and determining the minimum edit operation cost as the syntactic tree edit distance.

[0060] In this embodiment, the nodes of the two syntax trees can be aligned first, and then the minimum edit operation cost required to make the two syntax trees identical can be calculated. Editing operations can include at least one of replacing, deleting, or inserting nodes. This minimum edit operation cost is the syntax tree edit distance. Specifically, the corresponding operation cost can be determined for different editing operations, and then the sum of the operation costs corresponding to different editing schemes can be traversed. The editing scheme with the minimum total cost is taken as the optimal scheme for syntax tree editing, and this minimum total cost can be used as the syntax tree edit distance.

[0061] S103. Calculate the language structure loss using the edit distance of the syntactic tree, and update the model parameters of the target model using the language structure loss.

[0062] After obtaining the edit distance of the syntax tree, the language structure loss can be calculated based on the edit distance of the syntax tree.

[0063] Specifically, the generated predicted text can be parsed using a parser. and reference text Given two syntax trees, perform node alignment on each tree and calculate the minimum edit cost required to make them identical. Edit operations must include at least node replacement, deletion, and insertion. The minimum edit cost is the edit distance between the two syntax trees. In summary, the formula for calculating language structure loss is: ,in For weight parameters, This represents the maximum number of nodes for the generated text and the reference text.

[0064] In one specific embodiment of this application, updating the model parameters of the target model using language structure loss includes: calculating the autoregressive language modeling loss corresponding to the predicted text and the reference text; calculating the masked language modeling loss corresponding to the predicted text and the reference text; and adjusting the model parameters by jointly using the autoregressive language modeling loss, the masked language modeling loss, and the language structure loss.

[0065] The calculation of the autoregressive language modeling loss between the predicted text and the reference text includes: calculating the masked language modeling loss between the predicted word and the real word at each step during the process of generating the predicted text autoregressively using the decoder; where the predicted word is the word in the predicted text and the real word is the word in the reference text.

[0066] Specifically, language modeling loss (including autoregressive language modeling loss and masked language modeling loss) can be introduced to optimize the global fluency and local accuracy of text generation. During the decoder's autoregressive generation of the target text, the cross-entropy loss between the predicted word and the real word at each step can be calculated, i.e., the autoregressive language modeling loss. Based on the masking mechanism, the random mask positions in the predicted text are predicted, and the difference between the predicted result and the real words is calculated, which is the masked language modeling loss. The language structure loss is introduced, which explicitly optimizes the syntactic structure of the generated text through syntactic tree edit distance, significantly improving syntactic compliance and cross-modal alignment capabilities. A syntactic tree is constructed based on the reference text. Based on the syntactic structure between the predicted text and the reference text, the syntactic tree edit distance is calculated to further obtain the language structure loss. The language modeling loss and language structure loss are combined in a weighted manner as the total loss function. : This enables the collaborative optimization of driving model parameters, where This is a hyperparameter.

[0067] S104. With the model parameters of the target model trained, acquire the target video.

[0068] After training is complete, the target video can be obtained. This target video is the video for which content recognition needs to be performed (such as a video for which subtitles need to be automatically generated). This video can be a video composed of multiple frames of images, or an audio-visual video composed of multiple frames of images and audio.

[0069] S105. Input the target video into the target model for processing to obtain the content text of the target video.

[0070] Since the trained target model can perform content recognition on the target video and output the corresponding text, inputting the target video into the target model for processing can yield the content text of the target video.

[0071] In one specific embodiment of this application, the target model includes an audio encoder, a visual encoder, a cross-modal fusion module, a structure-aware encoder, and a decoder. The target video is input into the target model for processing to obtain the content text of the target video. This process includes: extracting audio features from the target video using the audio encoder; extracting visual features from the target video using the visual encoder in the target model; fusing the audio and visual features using the cross-modal fusion module to obtain grammar-aware multimodal features; encoding the multimodal features using the structure-aware encoder to obtain a grammar-aligned cross-modal joint representation; and decoding the cross-modal joint representation using the decoder to obtain the content text.

[0072] Specifically, in this embodiment, the target model performs content recognition on the target video, which may include extracting audio features from the target video using an audio encoder and extracting visual features using a visual encoder. Then, the audio and visual features are fused using a cross-modal fusion module to obtain multimodal features. Encoding these multimodal features using a structural encoder yields a grammatically aligned cross-modal joint representation. Finally, the content text is obtained by decoding this cross-modal joint representation using a decoder.

[0073] In other words, when the target video is input into the target model, the target model can extract the visual and audio features of the input audio and video data; input the visual and audio features into the structure-aware encoder of the trained and saved model to generate a grammatically aligned cross-modal joint representation; input the joint representation into the decoder, and generate predicted text word by word through autoregression based on the parameters optimized during the training phase; output fine-grained paragraph-level audio and video content descriptions. Due to the optimization of the added syntactic tree supervision signal, the grammatical regularity and semantic completeness of the generated text are significantly improved.

[0074] Correspondingly, during the training of the target model, the process of performing content understanding on the input training samples to obtain the predicted text is the same as the process of acquiring the content text. Specifically, after the training samples are input into the target model, the visual and audio features of the input audio and video data are first extracted; guided by the reference text features, the visual and audio features are fused across modally through a dynamic masking mechanism to generate grammar-aware multimodal features; the multimodal features are input into the structure-aware encoder to generate a grammar-aligned cross-modal joint representation; the joint representation is input into the decoder to generate the predicted text through an autoregressive approach; based on the predicted text and the reference text, the language modeling loss and language structure loss are calculated, where the language structure loss is measured by the syntactic tree edit distance to measure the grammatical difference between the generated text and the reference text; the above loss functions are jointly optimized, and the model parameters are updated, including the parameters of the visual encoder, audio encoder, cross-modal fusion module, structure-aware encoder, and decoder; the optimized model and model parameters are saved.

[0075] Once the content text is obtained, it can be used for subsequent applications such as assisting hearing-impaired users, video retrieval, content summarization, and cross-modal translation.

[0076] In one specific embodiment of this application, after obtaining the content text of the target video, the method further includes: fusing the content text with the target video to obtain an output video with subtitles. Some multimodal subtitle generation methods primarily rely on language models that implicitly learn language rules, but lack explicit modeling of the syntactic structure of the generated text. While this method improves the semantic accuracy of subtitle generation to some extent, it still has significant shortcomings in terms of the rationality of language structure. This leads to potential grammatical errors, loose sentence structures, and unnaturalness in the generated natural language text descriptions, especially when modeling complex language phenomena (such as clauses, parenthetical phrases, etc.), where the effect is poor. Furthermore, there is a lack of effective language structure feedback mechanisms, making it impossible to evaluate the differences in language structure between the generated text and the reference text during training.

[0077] To address these issues, the widespread application of structured language information in natural language processing has led to new technological approaches for understanding language structure. Among various methods of representing structured language information, common forms include syntax trees, dependency graphs, semantic role labeling, and component structure representation. Syntax trees, due to their unique advantages, have become one of the most commonly used representation methods. By decomposing sentences into hierarchical grammatical components through a tree-like data structure, they can not only clearly present the modification and dependency relationships between syntactic components, such as core grammatical relationships like subject-verb and verb-object, but also effectively capture nested structures and long-distance dependencies in complex sentences. This application finds that using syntax trees can significantly improve the grammatical compliance of generated text while maintaining semantic completeness, a characteristic that makes it a promising application in scenarios requiring high-quality text output, such as audio and video content understanding.

[0078] Based on the aforementioned technological advancements, a joint optimization method can be employed. This method involves simultaneously optimizing both autoregressive language modeling loss and language structure loss. The autoregressive language modeling loss ensures the semantic accuracy of the text, while the language structure loss guarantees grammatical compliance through syntactic tree edit distance. The weighted combination of these two methods collaboratively optimizes the model, significantly improving grammatical conformity while maintaining semantic accuracy. The video content understanding method based on structured grammatical information provided in this application represents a paradigm shift from implicit statistical generation to explicit, structure-controlled generation.

[0079] Applying the method provided in the embodiments of this application, training samples are input into the target model for content understanding processing to obtain predicted text, and a syntax tree corresponding to the predicted text is constructed; the syntax tree edit distance between the syntax tree and the syntax tree of the reference text is calculated; the language structure loss is calculated using the syntax tree edit distance, and the model parameters of the target model are updated using the language structure loss; with the model parameters of the target model trained, the target video is acquired; the target video is input into the target model for processing to obtain the content text of the target video.

[0080] This application aims to enhance video content understanding and natural language description generation capabilities by leveraging syntactic structure information, thereby obtaining a multimodal content understanding model with accurate semantic expression, reasonable grammatical structure, and coherent content logic. First, the target model is trained based on training samples. That is, the target model is used to perform content understanding on the training samples to obtain predicted text. To enable the target model to output text that better conforms to grammatical structure, this application constructs a corresponding syntax tree for the predicted text. The language structure loss is determined based on the syntactic tree edit distance between the syntax trees corresponding to the predicted text and the reference text. The model reference of the target model is adjusted based on the language structure loss, enabling the target model, after training, to perform content recognition on input videos and output content text that conforms to grammatical structure. After training, the target video for which content understanding is required can be obtained and input into the target model for content understanding, thus obtaining content text that conforms to grammatical structure. Because the target model is trained using the language structure loss calculated based on the syntax tree during model training, the resulting content text possesses the characteristics of grammatical accuracy, reasonable sentence structure, and naturalness.

[0081] To help those skilled in the art better understand the methods provided in the embodiments of this application, the different stages of the implementation of the method are described in detail below.

[0082] In the specific implementation of multimodal feature extraction during the training and inference phases, visual feature extraction employs a spatiotemporal modeling network based on a hierarchical window mechanism, extracting spatiotemporal features from videos through a multi-level window attention mechanism. The network uses a progressive downsampling strategy to gradually expand the receptive field, capturing hierarchical representations from local motion features to global scene features. Audio feature extraction uses an encoding network based on spectrogram patch partitioning, processing the temporal spectrogram through multi-scale convolutional kernel groups, and combining a self-attention mechanism to model the temporal dependencies of sound events, effectively capturing features from transient sounds to continuous audio patterns.

[0083] The specific implementation of cross-modal feature fusion during the training phase is guided by reference text features and employs a dynamic attention masking mechanism to achieve intelligent fusion of visual and audio features. First, a cross-modal semantic association matrix is ​​established. Then, the retention ratio of each modality feature is automatically adjusted based on the association strength. Finally, feature fusion is completed through a learnable gating mechanism. Specifically, residual connections are used to preserve the syntactic structure information of the original text.

[0084] The specific implementation of generating cross-modal joint representations during the training phase involves injecting a grammatical constraint matrix into the encoder, which encodes the strength distribution of core grammatical relations such as subject-verb and verb-object. A multi-level attention mechanism is used to progressively refine the grammatical structure: the first layer models phrase-level relations, the middle layer processes clause structures, the top layer establishes paragraph-level coherence, and finally, a grammatically aligned joint representation is output.

[0085] The specific implementation of generating predicted text by the decoder during the training phase includes the following methods for generating grammatically aligned cross-modal joint representations during the training phase: semantically aligning visual and audio features based on reference text to obtain multimodal features corresponding to the reference text in semantic segments; concatenating the aligned visual features, audio features, and corresponding reference text features in a predetermined order along the channel dimension to construct a fused cross-modal feature representation; and inputting the fused feature representation into the subsequent encoding module to obtain a cross-modal joint representation in a unified grammatical space.

[0086] The specific implementation of generating cross-modal joint representations during the training phase involves generating a dynamic attention mask guided by reference text features to adjust the retention ratio of visual and audio features; based on the attention mask, the visual and audio features are weighted to obtain adjusted multimodal feature representations; the adjusted visual and audio features are concatenated with the reference text features along the channel dimension to form fused cross-modal joint features; and the fused cross-modal joint features are further modeled through a learnable gating mechanism to obtain cross-modal joint representations with a unified grammatical space distribution.

[0087] During the training phase, grammatically aligned cross-modal joint representations are generated. To ensure that the feature representations conform to grammatical rules, the concatenated and fused multimodal features undergo grammatically guided encoding. A hierarchical attention mechanism is used to refine the modeling of the fused features layer by layer. Transform into syntax alignment representation .

[0088] The specific implementation of the joint optimization loss function significantly improves the grammatical compliance and semantic completeness of the text generated by the audio and video content understanding model by syntactically optimizing the autoregressive language modeling loss and language structure loss. Specifically, the autoregressive language modeling loss ( The negative log-likelihood function is used to calculate the masked language modeling loss for each time step during the autoregressive generation process of the decoder. This loss function ensures that the model learns the local word-level accuracy and global sequence fluency of text generation. Masked language modeling loss ( This paper proposes a loss function that randomly masks a certain proportion of words in the input text, requiring the model to predict the masked content based on context and cross-modal information. This loss function enhances the model's understanding of semantic context, which is particularly beneficial for handling semantic ambiguity in audio and video content. This application introduces a language structure loss based on syntactic tree edit distance: a syntactic analyzer is used to construct syntactic trees from the predicted and reference texts. First, the text needs to be preprocessed (including sentence boundary detection, word segmentation, lexical reconstruction, and stemming); then, part-of-speech tagging and grammatical analysis are performed (including decomposing text sentences into phrase structures and analyzing the dependencies between words); finally, the tree is formatted.

[0089] Using dynamic programming, recursively compute the results for two given trees. Calculate the minimum edit operation cost required to make them the same, i.e., their tree edit distance. For each pair of tokens, i.e., nodes There are three types of operations and their costs.

[0090] Firstly, the cost of node replacement: .

[0091] Secondly, the cost of node deletion: .

[0092] Third, node insertion cost: .

[0093] in , representing node similarity.

[0094] Ultimately, we obtain the language structure loss. Calculation formula: ,in For weight parameters, This represents the maximum number of nodes for the generated text and the reference text.

[0095] Joint optimization mechanism: The above loss functions are combined in a weighted manner to form the final optimization objective. ,in This is a hyperparameter.

[0096] During backpropagation, the joint loss function simultaneously optimizes the following aspects: enhancing the semantic understanding ability of the model through masked language modeling loss, ensuring the fluency of text generation through autoregressive language modeling loss, and enforcing the compliance of grammatical structures through language structure loss.

[0097] Save the trained target model, that is, save the complete model parameter system, including all trainable parameters of the visual encoder, audio encoder, cross-modal fusion module, structure-aware encoder and decoder, to ensure that the various capabilities obtained during training can be fully reproduced during the inference stage.

[0098] The specific implementation of obtaining cross-modal joint representations during the inference phase is based on generating cross-modal joint representations using fixed model parameters after training. The core differences from the training phase are: feature extraction is performed using an encoder with frozen parameters, and cross-modal fusion is achieved by directly applying the optimized attention weight matrix; gradient calculation and parameter updates are eliminated, retaining only the forward propagation process; and the strength of the grammatical constraints is fixed to the optimal value obtained during training. This inherits the multimodal understanding capabilities acquired during training, achieving efficient and stable inference computation.

[0099] In the inference phase, the decoder generates the predicted text. After loading the optimized model parameters from training, the decoder uses grammatically aligned cross-modal joint representations as initial context input and generates a text sequence word by word through an autoregressive approach. Specifically, based on the parameters jointly optimized during training (including word prediction ability optimized by autoregressive language modeling loss and syntactic constraints reinforced by language structure loss), the model dynamically calculates attention weights when generating each word, ensuring that the output text maintains both semantic accuracy and grammatical conformity. This process is entirely based on forward computation with fixed parameters, requiring no gradient updates.

[0100] To facilitate those skilled in the art to better understand and implement the video content understanding method based on structured grammar information provided in the embodiments of this application, the following detailed explanation of the video content understanding method based on structured grammar information is provided in conjunction with specific examples.

[0101] As mentioned above, this application focuses on the task of audio and video content understanding. However, current audio and video content understanding mainly relies on end-to-end deep learning methods. Although these methods can capture semantic relationships, they lack explicit modeling of language structure, leading to problems such as sentence confusion and grammatical errors in the generated text, which seriously restricts the effectiveness of practical applications.

[0102] To address existing problems, this application proposes an audio-visual content understanding method based on structured grammatical information enhancement. During the training phase, the model first extracts multimodal features through visual and audio encoders. The visual encoder employs a hierarchical spatiotemporal attention mechanism to progressively capture semantic information from local motion to the global scene; the audio encoder combines convolution and temporal modeling to extract spectral features of the sound signal. Subsequently, the cross-modal fusion module, guided by reference text, achieves adaptive feature fusion through dynamic attention weights while preserving the original grammatical structure information. The structure-aware encoder injects grammatical constraints into the standard converter architecture, guiding the feature encoding process through a predefined grammatical rule matrix. The decoder employs a joint optimization strategy, simultaneously minimizing autoregressive language modeling loss and language structure loss based on syntactic tree edit distance, ensuring that the model parameters possess both semantic understanding capabilities and internalized grammatical rule knowledge.

[0103] During the inference phase, pre-trained model parameters are directly loaded and processed. Input audio and video first undergo feature extraction and cross-modal fusion to generate a grammar-aware joint representation. When the decoder generates text, grammatical constraints are injected during decoding state initialization, and global grammatical consistency checks ensure grammatical correctness. The entire process is based entirely on the trained parameters for forward computation, requiring no additional correction.

[0104] This method achieves grammatically controllable audio and video content understanding through explicit learning of grammatical rules during the training phase and constrained generation during the inference phase, significantly improving the semantic completeness and grammatical compliance of the generated text.

[0105] Specifically, the implementation process of this method includes the following steps:

[0106] During the training and inference phases, visual and audio features of the video are extracted.

[0107] The input audio and video data includes: Video data: frame sequence ,in For frame number, Here, H represents the spatial resolution, and W represents the width. Represents the RGB three-channel color information of each video frame, where R is the pixel value (ranging from -255 to +255); Audio data: Mel spectrogram. F represents the frequency dimension, and M represents the number of time frames; text data: the corresponding text description. L represents the text length.

[0108] To capture multi-layered spatiotemporal information in the video, when extracting visual features, a spatiotemporal modeling network is used to process the original video frame sequence V. A hierarchical window attention mechanism is employed to extract local motion features and global scene features at different scales, thus integrating spatiotemporal features. As the output, d represents the feature dimension.

[0109] To extract temporally correlated audio feature representations, the audio spectrum A is processed using an audio coding network based on spectrogram partitioning and a multi-scale perception mechanism. Multi-scale convolutional kernel groups are used to extract frequency domain features, and self-attention is combined to model temporal dependencies, outputting the audio features. .

[0110] In the process of acquiring multimodal features during the training phase, dynamic attention mask fusion is performed to achieve feature alignment and information complementarity between visual, audio, and textual elements. and text features Calculate cross-modal attention weights, generate dynamic mask coefficients, and weightedly fuse multimodal features to finally output the fused features. .

[0111] During the training phase, grammatically aligned cross-modal joint representations are generated. To ensure that the feature representations conform to grammatical rules, grammatically constrained feature encodings are combined and injected into the grammatical constraint matrix. The grammatical structure is refined through hierarchical attention, and the fused features are then integrated. Transform into syntax alignment representation .

[0112] During the training phase, the decoder generates predicted text. The decoder is implemented using a multi-layer transformer architecture, with each layer containing three core sub-modules: a self-attention mechanism to process the generated sequence. Establish dependencies between words; cross-attention mechanism will Alignment is performed to obtain content-related contextual information; a feedforward neural network performs feature transformation. At each time step t, the decoder receives the embedding representations of the first t-1 words, and then... The current hidden state is obtained through interactive computation, and the probability distribution on the vocabulary is finally obtained through output layer transformation. The corresponding predicted text sequence is generated.

[0113] The autoregressive language modeling loss and language structure loss are combined to update the model parameters. During the decoder's autoregressive generation of the target text, the mask language modeling loss between the predicted word and the real word is calculated at each step, i.e., the autoregressive language modeling loss. Based on the masking mechanism, the random mask positions in the predicted text are predicted, and the difference between the predicted result and the real words is calculated, which is the masked language modeling loss. ; Construct its syntax tree based on the reference text; Calculate the edit distance of the syntax tree based on the syntax structure between the predicted text and the reference text, and further obtain the language structure loss. The autoregressive language modeling loss and the language structure loss are combined in a weighted manner as the total loss function L: This enables the collaborative optimization of driving model parameters, where This is a hyperparameter.

[0114] Save the trained model, that is, save the complete model parameter system, including all trainable parameters of the visual encoder, audio encoder, cross-modal fusion module, structure-aware encoder and decoder, to ensure that the various capabilities obtained during training can be fully reproduced during the inference stage.

[0115] During the inference phase, cross-modal representations are acquired, and the attention weights of the trained encoder are applied to fuse audio and video features. Transform into syntax alignment representation .

[0116] During the inference phase, the decoder generates predicted text. Given He as input, the decoder generates a text sequence based on the joint loss optimization parameters obtained during the training phase. .

[0117] To verify the effectiveness of the Tree Edit Distance (TED) optimization mechanism introduced in this application in the task of multi-granularity description of audio and video content, a comparative analysis of the performance difference of the model before and after adding the TED structure optimization can be performed based on the publicly available benchmark dataset FAVDBench. The FAVDBench dataset contains 11,424 video clips with detailed audio and video descriptions, including 71 fine-grained semantic tags across 5 categories, covering five dimensions: people, animals, musical instruments, vehicles, and common activities, and provides bilingual (Chinese and English) annotations. This dataset emphasizes semantic hierarchy and structural integrity, making it suitable for evaluating the comprehensive performance of grammatical structure modeling and natural language generation systems.

[0118] The experimental procedure includes the following steps.

[0119] Step 1: System setup and training process.

[0120] 1. Model framework: It adopts a multimodal converter structure. The input includes visual, auditory and text features, and the intermodal fusion is achieved through cross attention.

[0121] 2. Experimental variable control: Scheme A (without TED optimization): training is performed using only autoregressive language modeling loss; Scheme B (with TED optimization): based on Scheme A, a syntax tree edit distance loss term is introduced to constrain the consistency between the syntactic structure of the generated sentence and the reference syntax tree, forming a joint loss function.

[0122] 3. Training data splitting: 7,500 videos from FAVDBench were used as the training set, 1,500 videos as the validation set, and 1,000 videos as the test set for hyperparameter tuning and final evaluation, respectively.

[0123] Step 2: Indicator comparison and effect evaluation.

[0124] The performance of the model before and after structural optimization was compared using four automated evaluation metrics—BLEU-4, METEOR, ROUGE-L, and CIDEr—on the test set. In audio and video content understanding tasks, BLEU-4, ROUGE, METEOR, and CIDEr are four core metrics for evaluating the quality of generated text, testing the model's performance from different perspectives.

[0125] Among them: BLEU-4 evaluates phrase-level accuracy through 4-gram phrase exact matching, reflecting the model's accuracy in describing key actions and objects; ROUGE measures the ability of generated text to cover the core content of reference text based on recall, reflecting the model's ability to capture key information; METEOR introduces synonym and stemming analysis to comprehensively evaluate semantic similarity, reflecting the fluency and diversity of language expression; CIDEr adopts weighted multi-gram phrase similarity, emphasizing the consistency between generated text and human consensus, highlighting the richness of descriptive details.

[0126] The statistics are shown in Table 1 below.

[0127] Table 1 shows the implementation statistical analysis.

[0128]

[0129] As can be seen from the results of this embodiment, when the above indicators improve synergistically, it shows that the model is simultaneously optimized in three key dimensions: semantic accuracy (BLEU-4, ROUGE), language standardization (METEOR), and descriptive richness (CIDEr). In particular, after introducing structured grammatical information, it not only directly improves the grammatical compliance rate through syntactic tree supervision, but also indirectly enhances the semantic expressive ability—the improvement in BLEU-4 indicates more accurate generation of key phrases, the increase in ROUGE indicates more comprehensive coverage of core content, the improvement in METEOR reflects a more natural and fluent expression, and the improvement in CIDEr reflects a more vivid and rich description of details. This multi-indicator synergistic optimization verifies that while maintaining semantic completeness, the grammatical enhancement method significantly improves the modeling ability of the multimodal video content description system at the grammatical level due to the tree edit distance structure optimization mechanism introduced in this invention, enhancing the logic and standardization of language expression, and has important practical value and promotion significance.

[0130] Corresponding to the above method embodiments, this application also provides a video content understanding device based on structured grammatical information. The video content understanding device based on structured grammatical information described below can be referred to in correspondence with the video content understanding method based on structured grammatical information described above.

[0131] See Figure 2 As shown, the device includes the following modules:

[0132] The model training module 101 is used to input training samples into the target model for content understanding processing to obtain predicted text and construct the syntax tree corresponding to the predicted text; calculate the syntax tree edit distance between the syntax tree and the syntax tree of the reference text; calculate the language structure loss using the syntax tree edit distance; and update the model parameters of the target model using the language structure loss.

[0133] The video acquisition module 102 is used to acquire the target video after the model parameters of the target model have been trained.

[0134] The prediction application module 103 is used to input the target video into the target model for processing to obtain the content text of the target video.

[0135] Using the apparatus provided in the embodiments of this application, training samples are input into the target model for content understanding processing to obtain predicted text, and a syntax tree corresponding to the predicted text is constructed; the syntax tree edit distance between the syntax tree and the syntax tree of the reference text is calculated; the language structure loss is calculated using the syntax tree edit distance, and the model parameters of the target model are updated using the language structure loss; after the model parameters of the target model are trained, the target video is acquired; the target video is input into the target model for processing to obtain the content text of the target video.

[0136] This application aims to enhance video content understanding and natural language description generation capabilities by leveraging syntactic structure information, thereby obtaining a multimodal content understanding model with accurate semantic expression, reasonable grammatical structure, and coherent content logic. First, the target model is trained based on training samples. That is, the target model is used to perform content understanding on the training samples to obtain predicted text. To enable the target model to output text that better conforms to grammatical structure, this application constructs a corresponding syntax tree for the predicted text. The language structure loss is determined based on the syntactic tree edit distance between the syntax trees corresponding to the predicted text and the reference text. The model reference of the target model is adjusted based on the language structure loss, enabling the target model, after training, to perform content recognition on input videos and output content text that conforms to grammatical structure. After training, the target video for which content understanding is required can be obtained and input into the target model for content understanding, thus obtaining content text that conforms to grammatical structure. Because the target model is trained using the language structure loss calculated based on the syntax tree during model training, the resulting content text possesses the characteristics of grammatical accuracy, reasonable sentence structure, and naturalness.

[0137] In one specific embodiment of this application, the model training module is specifically used to construct a grammatical structure representation using the grammatical components of the predicted text and the grammatical relationships between the grammatical components; and to construct a tree structure with words as nodes and grammatical relationships as edges based on the grammatical structure representation to obtain a syntax tree; wherein, the syntax tree includes annotations of the grammatical roles and hierarchical relationships of each node.

[0138] In one specific embodiment of this application, the model training module is specifically used to align the nodes of two syntax trees and calculate the minimum edit operation cost required to make the two trees identical. The edit operation includes at least one of node replacement, deletion and insertion. The minimum edit operation cost is determined as the syntax tree edit distance.

[0139] In one specific embodiment of this application, the target model includes an audio encoder, a visual encoder, a cross-modal fusion module, a structure-aware encoder, and a decoder. A prediction module is used to extract audio features from the target video using the audio encoder and visual features from the target video using the visual encoder in the target model. The cross-modal fusion module fuses the audio and visual features to obtain grammar-aware multimodal features. The structure-aware encoder encodes the multimodal features to obtain a grammar-aligned cross-modal joint representation. The decoder decodes the cross-modal joint representation to obtain the content text.

[0140] In one specific embodiment of this application, it further includes:

[0141] The automatic subtitle generation module is used to merge the content text with the target video to obtain an output video with subtitles.

[0142] In one specific embodiment of this application, the model training module is specifically used to calculate the autoregressive language modeling loss corresponding to the predicted text and the reference text; calculate the masked language modeling loss corresponding to the predicted text and the reference text; and adjust the model parameters by jointly using the autoregressive language modeling loss, the masked language modeling loss, and the language structure loss.

[0143] In one specific embodiment of this application, the model training module is specifically used to calculate the masked language modeling loss between the predicted word and the real word at each step in the process of generating predicted text autoregressively using the decoder; wherein, the predicted word is a word in the predicted text, and the real word is a word in the reference text.

[0144] Furthermore, in practical applications, the steps of this method can be implemented within the system. In practical applications, the system implementing this video content understanding method based on structured grammatical information—namely, an audio-visual content understanding system enhanced by structured grammatical information—is composed of a hierarchical feature extraction module, a grammar-aware fusion module, a joint representation encoding module, a constraint decoding module, and a multi-objective optimization module. The hierarchical feature extraction module extracts spatiotemporal visual features of the video and multi-scale spectral features of the audio through a spatiotemporal modeling network and an audio encoding network, respectively. The grammar-aware fusion module utilizes a cross-attention mechanism to achieve cross-modal alignment between visual-text and audio-text, and injects syntactic tree structure information from the reference text. The joint representation encoding module uses a structure-guided encoder to generate multimodal joint representations under grammatical constraints. The constraint decoding module dynamically adjusts the syntactic rationality of the generated sequence by combining autoregressive generation and real-time grammar verification mechanisms. The multi-objective optimization module drives model training through a weighted combination of autoregressive language modeling loss and language structure loss. Through an end-to-end collaborative mechanism, each module significantly improves the grammatical compliance of the generated text while maintaining semantic completeness. The innovative design of the constraint decoding module effectively addresses the shortcomings of traditional methods in terms of grammatical standardization.

[0145] Corresponding to the above method embodiments, this application also provides an electronic device. The electronic device described below and the video content understanding method based on structured grammatical information described above can be referred to and correspond to each other.

[0146] See Figure 3 As shown, the electronic device includes:

[0147] Memory 332 is used to store computer programs;

[0148] The processor 322 is used to implement the steps of the video content understanding method based on structured grammar information in the above method embodiments when executing a computer program.

[0149] For details, please refer to Figure 4 , Figure 4 This is a schematic diagram of the specific structure of an electronic device provided in this embodiment. The electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the electronic device 301.

[0150] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0151] The steps in the video content understanding method based on structured grammatical information described above can be implemented by the structure of electronic devices.

[0152] Corresponding to the above method embodiments, this application also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the video content understanding method based on structured grammatical information described above.

[0153] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video content understanding method based on structured grammar information described in the above method embodiments.

[0154] The readable storage medium can specifically be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other readable storage medium capable of storing program code.

[0155] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0156] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0157] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0158] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0159] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video content understanding method based on structured grammatical information, characterized in that, include: The training samples are input into the target model for content understanding processing to obtain the predicted text, and the syntax tree corresponding to the predicted text is constructed. Calculate the syntactic tree edit distance between the syntactic tree and the syntactic tree of the reference text; use a syntactic analyzer to parse the generated predicted text respectively. and reference text Given two syntax trees, perform node alignment on the two syntax trees and calculate the minimum edit operation cost required to make the two trees identical; determine the minimum edit operation cost as the edit distance of the syntax trees. ; The language structure loss is calculated using the edit distance of the syntactic tree, and the model parameters of the target model are updated using the language structure loss; wherein, the formula for calculating the language structure loss is: ,in For weight parameters, This represents the maximum number of nodes in the generated text and the reference text. After training the model parameters of the target model, acquire the target video; The target video is input into the target model for processing to obtain the content text of the target video; The process of updating the model parameters of the target model using the language structure loss includes: Calculate the autoregressive language modeling loss corresponding to the predicted text and the reference text; Calculate the masked language modeling loss corresponding to the predicted text and the reference text; The model parameters are adjusted by combining the autoregressive language modeling loss, the masked language modeling loss, and the language structure loss. The calculation of the autoregressive language modeling loss corresponding to the predicted text and the reference text includes: In the process of generating the predicted text using an autoregressive decoder, the masked language modeling loss for each predicted word and the real word is calculated; where the predicted word is a word in the predicted text, and the real word is a word in the reference text; based on the masking mechanism, random mask positions in the predicted text are predicted, and the difference between the prediction result and the real word is calculated to obtain the masked language modeling loss. .

2. The method according to claim 1, characterized in that, Constructing the syntax tree corresponding to the predicted text includes: A grammatical structure representation is constructed using the grammatical components of the predicted text and the grammatical relationships between these components; Based on the grammatical structure representation, a tree structure with words as nodes and grammatical relations as edges is constructed to obtain the syntax tree; wherein, the syntax tree includes annotations of the grammatical roles and hierarchical relationships of each node.

3. The method according to claim 1, characterized in that, The editing operations include at least one of node replacement, deletion, and insertion.

4. The method according to claim 1, characterized in that, The target model includes an audio encoder, a visual encoder, a cross-modal fusion module, a structure-aware encoder, and a decoder. The target video is input into the target model for processing to obtain the content text of the target video, including: The audio features of the target video are extracted using the audio encoder, and the visual features of the target video are extracted using the visual encoder in the target model. The cross-modal fusion module is used to fuse the audio features and the visual features to obtain grammar-aware multimodal features; The structure-aware encoder is used to encode the multimodal features to obtain a syntax-aligned cross-modal joint representation; The content text is obtained by decoding the cross-modal joint representation using the decoder.

5. The method according to claim 1, characterized in that, After obtaining the content text of the target video, the method further includes: The content text and the target video are combined to obtain an output video with subtitles.

6. A video content understanding device based on structured grammatical information, characterized in that, include: The model training module is used to input training samples into the target model for content understanding processing, obtain predicted text, and construct the syntax tree corresponding to the predicted text; Calculate the syntactic tree edit distance between the syntactic tree and the syntactic tree of the reference text; use a syntactic analyzer to parse the generated predicted text respectively. and reference text Given two syntax trees, perform node alignment on the two syntax trees and calculate the minimum edit operation cost required to make the two trees identical; determine the minimum edit operation cost as the edit distance of the syntax trees. ; The language structure loss is calculated using the edit distance of the syntactic tree, and the model parameters of the target model are updated using the language structure loss; wherein, the formula for calculating the language structure loss is: ,in For weight parameters, This represents the maximum number of nodes in the generated text and the reference text. The video acquisition module is used to acquire the target video after the model parameters of the target model have been trained. The prediction application module is used to input the target video into the target model for processing to obtain the content text of the target video; The model training module is specifically used to calculate the autoregressive language modeling loss corresponding to the predicted text and the reference text; calculate the masked language modeling loss corresponding to the predicted text and the reference text; and adjust the model parameters by combining the autoregressive language modeling loss, the masked language modeling loss, and the language structure loss. Calculating the autoregressive language modeling loss corresponding to the predicted text and the reference text includes: calculating the masked language modeling loss between the predicted word and the real word at each step during the autoregressive generation of the predicted text using the decoder; wherein the predicted word is a word in the predicted text, the real word is a word in the reference text, and the random mask positions in the predicted text are predicted based on a masking mechanism; the difference between the prediction result and the real word is calculated to obtain the masked language modeling loss. .

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the video content understanding method based on structured grammar information as described in any one of claims 1 to 5 when executing the computer program.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the video content understanding method based on structured grammatical information as described in any one of claims 1 to 5.

Images