Video content understanding method and device based on structured grammar information, electronic equipment and storage medium

Abstract

The invention belongs to the technical field of computer application, and discloses a video content understanding method and device based on structured grammar information, electronic equipment and a storage medium, and the method comprises the steps: inputting a training sample into a target model for content understanding processing, obtaining a prediction text, and constructing a syntactic tree corresponding to the prediction text; calculating a syntactic tree editing distance between the syntactic tree and the syntactic tree of the reference text; calculating language structure loss by using the syntactic tree editing distance, and updating model parameters of the target model by using the language structure loss; under the condition that the model parameters of the target model are trained, obtaining a target video; and inputting the target video into the target model for processing to obtain a content text of the target video. In the application, the target model is trained based on the language structure loss calculated based on the syntax tree, and the target video can be understood, so that the content text with accurate grammar and reasonable and natural sentence structure is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer application, in particular to a video content understanding method and device based on structured syntax information, an electronic device and a storage medium. BACKGROUND

[0002] Video content understanding refers to a technology of automatically paraphrasing video content into natural language text, so as to automatically generate subtitles based on the natural language text.

[0003] However, when paraphrasing video content into natural language text, the generated natural language text description often has problems such as grammatical errors, loose sentence structure, and unnaturalness, which makes the automatic generation of subtitles ineffective.

[0004] To sum up, how to effectively solve the problems of content understanding of video content and obtaining content text meeting expectations is a technical problem that needs to be solved by the technical personnel in the field at present. SUMMARY

[0005] The purpose of the present application is to provide a video content understanding method and device based on structured syntax information, an electronic device and a storage medium, so as to obtain content text meeting the grammatical structure when understanding the content of the video.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] A video content understanding method based on structured syntax information, comprising:

[0008] inputting a training sample into a target model for content understanding processing to obtain a predicted text, and constructing a syntax tree corresponding to the predicted text;

[0009] calculating a syntax tree edit distance between the syntax tree and a syntax tree of a reference text;

[0010] calculating a language structure loss using the syntax tree edit distance, and updating model parameters of the target model using the language structure loss;

[0011] under the condition that the model parameters of the target model are trained, obtaining a target video;

[0012] inputting the target video into the target model for processing to obtain a content text of the target video.

[0013] Preferably, constructing the syntax tree corresponding to the predicted text comprises:

[0014] constructing a syntax structure representation using the grammatical components of the predicted text and the grammatical relationships between the grammatical components;

[0015] construct a tree structure with words as nodes and grammatical relations as edges based on the syntax structure representation, to obtain a syntax tree; wherein the syntax tree comprises annotations of grammatical roles and hierarchical relations of each node.

[0016] Preferably, the syntax tree edit distance between the syntax tree and the syntax tree of the reference text is calculated, comprising:

[0017] align the nodes of the two syntax trees and calculate the minimum editing operation cost required to make the two trees identical, the editing operation including at least one of node replacement, deletion and insertion;

[0018] determine the minimum editing operation cost as the syntax tree edit distance.

[0019] Preferably, the target model comprises an audio encoder, a visual encoder, a cross-modal fusion module, a structure-aware encoder and a decoder, and the target video is input into the target model for processing to obtain the content text of the target video, comprising:

[0020] extract audio features of the target video using the audio encoder, and extract visual features of the target video using the visual encoder in the target model;

[0021] fuse the audio features and the visual features using the cross-modal fusion module to obtain syntax-aware multi-modal features;

[0022] encode the multi-modal features using the structure-aware encoder to obtain syntax-aligned cross-modal joint representation;

[0023] decode the cross-modal joint representation using the decoder to obtain the content text.

[0024] Preferably, after obtaining the content text of the target video, further comprising:

[0025] fuse the content text and the target video to obtain an output video with subtitles.

[0026] Preferably, the model parameters of the target model are updated using the language structure loss, comprising:

[0027] calculate the autoregressive language modeling loss corresponding to the predicted text and the reference text;

[0028] calculate the masked language modeling loss corresponding to the predicted text and the reference text;

[0029] adjust the model parameters jointly with the autoregressive language modeling loss, the masked language modeling loss and the language structure loss.

[0030] Preferably, the autoregressive language modeling loss corresponding to the predicted text and the reference text is calculated, comprising:

[0031] In the process of generating the predicted text autoregressively by the decoder, a masked language modeling loss of each predicted word and a real word is calculated; wherein the predicted word is a word in the predicted text, and the real word is a word in the reference text.

[0032] A video content understanding device based on structured syntax information, comprising:

[0033] A model training module configured to input a training sample into a target model for content understanding processing to obtain a predicted text, and to construct a syntax tree corresponding to the predicted text; calculate a syntax tree edit distance between the syntax tree and a syntax tree of a reference text; calculate a language structure loss by using the syntax tree edit distance, and update model parameters of the target model by using the language structure loss;

[0034] A video acquisition module configured to acquire a target video in a case where the model parameters of the target model are trained.

[0035] A prediction application module configured to input the target video into the target model for processing to obtain a content text of the target video.

[0036] An electronic device, comprising:

[0037] A memory configured to store a computer program;

[0038] A processor configured to implement the steps of the above-mentioned video content understanding method based on structured syntax information when executing the computer program.

[0039] A readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the above-mentioned video content understanding method based on structured syntax information.

[0040] According to the method provided in the embodiments of the present application, a training sample is input into a target model for content understanding processing to obtain a predicted text, and a syntax tree corresponding to the predicted text is constructed; a syntax tree edit distance between the syntax tree and a syntax tree of a reference text is calculated; a language structure loss is calculated by using the syntax tree edit distance, and model parameters of the target model are updated by using the language structure loss; a target video is acquired in a case where the model parameters of the target model are trained; and the target video is input into the target model for processing to obtain a content text of the target video.

[0041] The application aims to enhance the ability of video content understanding and natural language description generation by using syntactic structure information, so as to obtain a multi-modal content understanding model with accurate semantic expression, reasonable grammar structure and logical coherent content. First, the target model is trained based on training samples. That is, the target model is used to understand the content of the training samples, so as to obtain predicted text. In order to make the target model output text that conforms to the grammar structure, in the application, a corresponding grammar tree is constructed for the predicted text, and the language structure loss is determined based on the syntactic tree edit distance between the grammar trees corresponding to the predicted text and the reference text. Adjusting the model reference of the target model based on the language structure loss can make the target model, after training, be able to recognize the content of the input video and output content text conforming to the grammar structure. After training, the target video that needs to be understood can be obtained, and the target video is input into the target model for content understanding, so as to obtain content text conforming to the grammar structure. Since the language structure loss calculated based on the grammar tree is used to train the target model during the model training process, the content text has the characteristics of accurate grammar, reasonable sentence structure and naturalness.

[0042] Correspondingly, the application also provides a video content understanding device based on structured syntax information, a device based on structured syntax information and a readable storage medium corresponding to the above-mentioned video content understanding method based on structured syntax information, which have the above-mentioned technical effects, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiment or related art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0044] Figure 1 The implementation flowchart of the video content understanding method based on structured syntax information in the embodiment of the application;

[0045] Figure 2 The structural schematic diagram of the video content understanding device based on structured syntax information in the embodiment of the application;

[0046] Figure 3 The structural schematic diagram of the electronic device in the embodiment of the application;

[0047] Figure 4 The specific structural schematic diagram of the electronic device in the embodiment of the application. DETAILED DESCRIPTION

[0048] For those skilled in the art, better understand the present application, the following will be combined with the drawings and specific embodiments of the application is further described in detail. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without doing creative work, within the scope of the present application.

[0049] Please refer to Figure 1 , Figure 1 For the flow chart of a video content understanding method based on structured syntax information in an embodiment of the present application, the method comprises the following steps.

[0050] S101, input the training sample into the target model for content understanding processing, obtain the predicted text, and construct the syntax tree corresponding to the predicted text.

[0051] Among them, the training sample can be a video with reference text, that is, the text expression of the content corresponding to the video. The video can be a video composed of multiple frames of images, or an audio and video including multiple frames of images and audio.

[0052] The target model is a model that can understand the content of the input video and output the content text. In order to facilitate the distinction, in this embodiment, the content text output by the model in the model training stage is called predicted text.

[0053] After obtaining the predicted text, the syntax tree corresponding to the predicted text can be constructed to facilitate the analysis of the syntax structure of the predicted text based on the syntax tree.

[0054] In a specific embodiment of the present application, the syntax tree corresponding to the predicted text is constructed, comprising: using the syntax components of the predicted text and the syntax relationship between the syntax components to construct the syntax structure representation; based on the syntax structure representation, a tree structure with words as nodes and syntax relationship as edges is constructed to obtain the syntax tree; wherein the syntax tree comprises the annotation of the syntax role and hierarchical relationship of each node.

[0055] That is, a syntax structure representation can be constructed based on the syntax components of the predicted text and the syntax relationship between the syntax components, such as dependency relationship and the like. Then, a tree structure with words as nodes and syntax relationships as edges is constructed based on the syntax structure representation. The tree structure is a syntax tree. In particular, the syntax tree also has annotations of syntax roles and hierarchical relationships of nodes in the embodiment. For example, the syntax role can be any of subject-predicate-object-determinative-complement, and the corresponding layer of the hierarchical relationship can have the following layers: morpheme layer, the smallest syntax / semantic unit, including free morphemes and affixes (such as prefixes and suffixes); word layer, a unit composed of words, such as nouns, verbs, etc., having independent meaning and syntax function; phrase layer, a unit composed of words combined by syntax relationship; clause layer, a language unit expressing relatively complete meaning, such as independent nominative structure; sentence layer, the highest syntax unit, composed of clauses, expressing complete thought.

[0056] S102, calculate the syntax tree edit distance between the syntax tree of the predicted text and the syntax tree of the reference text.

[0057] The syntax tree of the reference text is constructed in the same way as the syntax tree of the predicted text, and the syntax tree of the predicted text can be constructed in advance or during the training process.

[0058] The syntax tree edit distance between the two syntax trees can be determined by comparing and calculating the syntax trees of the predicted text and the reference text. The syntax tree edit distance refers to the distance corresponding to the editing operation required to edit the two syntax trees into the same structure.

[0059] In a specific embodiment of the present application, the syntax tree edit distance between the syntax tree and the syntax tree of the reference text is calculated, including: aligning the nodes of the two syntax trees and calculating the minimum editing operation cost required to make the two trees identical, the editing operation including at least one of node replacement, deletion and insertion; determining the minimum editing operation cost as the syntax tree edit distance.

[0060] In the embodiment, the nodes of the two syntax trees can be aligned first, and then the minimum editing operation cost required to make the two syntax trees identical is calculated. The editing operation can include at least one of replacing the node, deleting the node, and inserting the node. The minimum editing operation cost is the syntax tree edit distance. Specifically, the corresponding operation cost of different editing operations can be determined, and then the operation cost sum corresponding to the editing operation of different editing schemes is traversed. The editing scheme corresponding to the minimum cost sum is taken as the best scheme of syntax tree editing, and the minimum cost sum can be taken as the syntax tree edit distance.

[0061] S103, calculate a language structure loss by using the syntax tree edit distance, and update the model parameters of the target model by using the language structure loss.

[0062] After obtaining the syntax tree edit distance, the language structure loss can be calculated based on the syntax tree edit distance.

[0063] Specifically, the syntax analyzer can be used to parse the generated predicted text and the syntax tree of the reference text respectively; the node alignment of the two syntax trees is performed and the minimum editing operation cost required to make the two trees identical is calculated, the editing operation at least including node replacement, deletion and insertion, and the obtained minimum editing operation cost is the edit distance of the two syntax trees ; in summary, the language structure loss calculation formula is: wherein is a weight parameter, is the maximum value of the node number of the generated text and the reference text.

[0064] In a specific embodiment of the present application, the model parameters of the target model are updated by using the language structure loss, including: calculating an autoregressive language modeling loss corresponding to the predicted text and the reference text; calculating a masked language modeling loss corresponding to the predicted text and the reference text; adjusting the model parameters by combining the autoregressive language modeling loss, the masked language modeling loss and the language structure loss.

[0065] Wherein, the autoregressive language modeling loss corresponding to the predicted text and the reference text is calculated, including: calculating the masked language modeling loss of each step of the predicted word and the real word in the process of generating the predicted text by the decoder autoregressively; wherein the predicted word is the word in the predicted text, and the real word is the word in the reference text.

[0066] Specifically, the language modeling loss (including the autoregressive language modeling loss and the masked language modeling loss) can be introduced to optimize the global fluency and local accuracy of text generation. In the process of generating the target text by the decoder autoregressively, the cross-entropy loss of each step of the predicted word and the real word is calculated, i.e. the autoregressive language modeling loss ; the difference between the predicted result and the real word is calculated based on the prediction of the random mask position in the predicted text by the mask mechanism, i.e. the masked language modeling loss ; the language structure loss is introduced, which explicitly optimizes the grammatical structure of the generated text by the syntax tree edit distance, significantly improves the syntax compliance and cross-modal alignment capability, and constructs a syntax tree based on the reference text; the syntax tree edit distance is calculated based on the syntax structure between the predicted text and the reference text, and the language structure loss is further obtained ; the language modeling loss and the language structure loss are combined in a weighted form as the total loss function : to realize collaborative optimization of the driving model parameters, wherein are hyperparameters.

[0067] S104, acquire a target video in a case where the model parameters of the target model are trained.

[0068] After the training is completed, the target video can be acquired. The target video is a video (such as a video that needs to automatically generate subtitles) that needs to be content-recognized, and the video can be a video composed of multiple image frames or an audio-video composed of multiple image frames and audio.

[0069] S105, input the target video into the target model for processing to obtain content text of the target video.

[0070] Since the target model after the training can perform content recognition on the target video and can output corresponding text, inputting the target video into the target model for processing can obtain content text of the target video.

[0071] In a specific embodiment of the present application, the target model includes an audio encoder, a visual encoder, a cross-modal fusion module, a structure perception encoder, and a decoder. Inputting the target video into the target model for processing to obtain content text of the target video includes: extracting audio features of the target video by using the audio encoder, and extracting visual features of the target video by using the visual encoder in the target model; fusing the audio features and the visual features by using the cross-modal fusion module to obtain syntax-aware multi-modal features; encoding the multi-modal features by using the structure perception encoder to obtain syntax-aligned cross-modal joint representations; and decoding the cross-modal joint representations by using the decoder to obtain the content text.

[0072] Specifically, in the embodiment, the target model performs content recognition on the target video, which can specifically include extracting audio features of the target video based on the audio encoder, and extracting visual features of the target video by using the visual encoder. Then, the audio features and the visual features are fused by means of the cross-modal fusion module to obtain multi-modal features. The multi-modal features are encoded by using the structure encoder to obtain syntax-aligned cross-modal joint representations. Finally, the cross-modal joint representations are decoded based on the decoder, and the content text can be obtained.

[0073] That is, after the target video is input into the target model, the target model can extract visual features and audio features of the input audio-video data; input the visual features and the audio features into a structure-aware encoder of the saved model after training to generate syntax-aligned cross-modal joint representations; input the joint representations into a decoder to generate predicted text word by word in a self-recurrent manner based on the parameters optimized in the training stage; and output fine-grained paragraph-level audio-video content description. Due to the optimization of the syntax tree supervision signal, the grammatical normativity and semantic completeness of the generated text are significantly improved.

[0074] Correspondingly, in the training process of the target model, the process of obtaining the predicted text by content understanding of the input training sample is also the process of obtaining the content text. Specifically, after the training sample is input into the target model, first, the visual features and the audio features of the input audio-video data are extracted; the visual features and the audio features are cross-modally fused through a dynamic mask mechanism guided by the reference text features to generate syntax-aware multi-modal features; the multi-modal features are input into a structure-aware encoder to generate syntax-aligned cross-modal joint representations; the joint representations are input into a decoder to generate predicted text in a self-recurrent manner; and based on the predicted text and the reference text, a language modeling loss and a language structure loss are calculated, where the language structure loss measures the grammatical difference between the generated text and the reference text through a syntax tree edit distance; the above loss functions are jointly optimized to update the model parameters, including the parameters of the visual encoder, the audio encoder, the cross-modal fusion module, the structure-aware encoder, and the decoder; and the optimized model and the model parameters are saved.

[0075] After obtaining the content text, subsequent applications such as hearing-impaired users, video retrieval, content summary, and cross-modal translation can be assisted.

[0076] In a specific embodiment in the present application, after obtaining the content text of the target video, the content text and the target video are fused to obtain an output video with subtitles. Some multi-modal subtitle generation methods mainly rely on language models that implicitly learn language rules, but lack explicit modeling of the syntax structure of the generated text. Although this method improves the semantic accuracy of subtitle generation to some extent, it still has obvious shortcomings in terms of language structure rationality. This leads to problems such as grammatical errors, loose and unnatural sentence structure in the generated natural language text description, especially when modeling complex language phenomena such as clauses and inserted words, the effect is not good. In addition, there is also a lack of effective language structure feedback mechanism, and the difference between the generated text and the reference text in terms of language structure cannot be evaluated in the training process.

[0077] To solve these problems, structured language information is widely used in the field of natural language processing, and a new technical path for understanding language structure has emerged. Among various expression methods of structured language information, common forms also include syntax tree, dependency graph, semantic role labeling, and constituent structure representation, etc. Among them, the syntax tree becomes one of the most commonly used representation methods due to its unique advantages. The syntax tree can clearly present the modification and dependency relationship between the syntactic components, such as the core grammar relationship of subject-predicate and verb-object, and effectively capture the nested structure and long-distance dependency in complex sentence patterns. The present application finds that the representation method using the syntax tree can significantly improve the grammatical compliance of the generated text while maintaining semantic integrity. This feature makes it have important application prospects in scenarios such as audio and video content understanding that require high-quality text output.

[0078] Based on the above technical progress, a joint optimization method can be used. This method refers to a method of simultaneously optimizing the autoregressive language modeling loss and the language structure loss. The autoregressive language modeling loss ensures the semantic accuracy of the text, and the language structure loss ensures the grammatical compliance through the syntax tree edit distance. The two are combined and optimized together, so that the model can significantly improve the grammatical compliance while maintaining semantic accuracy. The video content understanding method based on structured syntax information provided by the embodiments of the present application can solve the above problems, that is, the paradigm transition from implicit statistical generation to explicit structure controllable generation.

[0079] By applying the method provided by the embodiments of the present application, the training sample is input into the target model for content understanding processing to obtain a predicted text, and a syntax tree corresponding to the predicted text is constructed. The syntax tree edit distance between the syntax tree and the syntax tree of the reference text is calculated. The language structure loss is calculated using the syntax tree edit distance, and the model parameters of the target model are updated using the language structure loss. In the case where the model parameters of the target model are trained, the target video is obtained. The target video is input into the target model for processing to obtain the content text of the target video.

[0080] The application aims to enhance the capability of video content understanding and natural language description generation by using syntactic structure information, so as to obtain a multi-modal content understanding model with accurate semantic expression, reasonable grammar structure and logical coherent content. First, the target model is trained based on training samples. That is, the target model is used to perform content understanding on the training samples, so as to obtain predicted text. In order to enable the target model to output text that conforms to the grammar structure, in the application, a corresponding grammar tree is constructed for the predicted text, and the language structure loss is determined based on the syntactic tree edit distance between the grammar trees corresponding to the predicted text and the reference text. The model reference of the target model is adjusted based on the language structure loss, so that after the training is completed, the target model can perform content recognition on the input video and output content text that conforms to the grammar structure. After training, the target video that needs to be understood can be obtained, and the target video is input into the target model for content understanding, so as to obtain content text that conforms to the grammar structure. Since the language structure loss calculated based on the grammar tree is used to train the target model during the model training process, the content text has the characteristics of accurate grammar, reasonable sentence structure and naturalness.

[0081] To make the skilled in the art better understand the method provided by the embodiments of the application, the different steps of the implementation of the method are described in detail below.

[0082] In the training and inference stage, the specific implementation of multi-modal feature extraction is as follows. The visual feature extraction adopts a hierarchical window mechanism-based spatio-temporal modeling network, which extracts spatio-temporal features from the video through a multi-level window attention mechanism. The network adopts a progressive downsampling strategy, gradually expands the receptive field, and captures hierarchical representations from local motion features to global scene features. The audio feature extraction uses an encoding network based on spectral graph block division, which processes the time-frequency spectrum graph through a multi-scale convolution kernel group, combines a self-attention mechanism to model the temporal dependence of sound events, and effectively captures features from transient sounds to continuous audio patterns.

[0083] In the training stage, the specific implementation of cross-modal feature fusion is as follows. The reference text features are used as a guide to realize intelligent fusion of visual and audio features by using a dynamic attention mask mechanism. First, a cross-modal semantic association matrix is established, then the reservation ratio of each modal feature is automatically adjusted based on the association strength, and finally the feature fusion is completed through a learnable gating mechanism. In particular, the original text grammar structure information is reserved through a residual connection.

[0084] In the training stage, the specific implementation of generating cross-modal joint representation is as follows. A grammar constraint matrix is injected into the encoder, which encodes the strength distribution of core grammar relationships such as subject-predicate and verb-object. The grammar structure is gradually refined through a multi-level attention mechanism, the first layer models the phrase-level relationship, the middle layer processes the sentence structure, and the top layer establishes the paragraph-level coherence, and finally outputs the grammar-aligned joint representation.

[0085] In the training phase, the decoder generates predicted text, and the method for generating syntax-aligned cross-modal joint representation in the training phase includes: performing semantic alignment on the visual features and the audio features based on the reference text to obtain multi-modal features corresponding to the reference text in a semantic segment; concatenating the aligned visual features, audio features, and corresponding reference text features in a predetermined order in the channel dimension to construct a fused cross-modal feature representation; and inputting the fused feature representation into a subsequent encoding module to obtain a cross-modal joint representation in a unified syntax space.

[0086] In the training phase, the decoder generates predicted text, and the method for generating syntax-aligned cross-modal joint representation in the training phase includes: performing semantic alignment on the visual features and the audio features based on the reference text to obtain multi-modal features corresponding to the reference text in a semantic segment; concatenating the aligned visual features, audio features, and corresponding reference text features in a predetermined order in the channel dimension to construct a fused cross-modal feature representation; and inputting the fused feature representation into a subsequent encoding module to obtain a cross-modal joint representation in a unified syntax space.

[0087] In the training phase, the decoder generates predicted text, and the method for generating syntax-aligned cross-modal joint representation in the training phase includes: performing semantic alignment on the visual features and the audio features based on the reference text to obtain multi-modal features corresponding to the reference text in a semantic segment; concatenating the aligned visual features, audio features, and corresponding reference text features in a predetermined order in the channel dimension to construct a fused cross-modal feature representation; and inputting the fused feature representation into a subsequent encoding module to obtain a cross-modal joint representation in a unified syntax space.

[0088] In the training phase, the decoder generates predicted text, and the method for generating syntax-aligned cross-modal joint representation in the training phase includes: performing semantic alignment on the visual features and the audio features based on the reference text to obtain multi-modal features corresponding to the reference text in a semantic segment; concatenating the aligned visual features, audio features, and corresponding reference text features in a predetermined order in the channel dimension to construct a fused cross-modal feature representation; and inputting the fused feature representation into a subsequent encoding module to obtain a cross-modal joint representation in a unified syntax space. ​​​This paper proposes a loss function that randomly masks a certain proportion of words in the input text, requiring the model to predict the masked content based on context and cross-modal information. This loss function enhances the model's understanding of semantic context, which is particularly beneficial for handling semantic ambiguity in audio and video content. This application introduces a language structure loss based on syntactic tree edit distance: a syntactic analyzer is used to construct syntactic trees from the predicted and reference texts. First, the text needs to be preprocessed (including sentence boundary detection, word segmentation, lexical reconstruction, and stemming); then, part-of-speech tagging and grammatical analysis are performed (including decomposing text sentences into phrase structures and analyzing the dependencies between words); finally, the tree is formatted.

[0089] Using dynamic programming, recursively compute the results for two given trees. Calculate the minimum edit operation cost required to make them the same, i.e., their tree edit distance. For each pair of tokens, i.e., nodes There are three types of operations and their costs.

[0090] Firstly, the cost of node replacement: .

[0091] Secondly, the cost of node deletion: .

[0092] Third, node insertion cost: .

[0093] in , representing node similarity.

[0094] Ultimately, we obtain the language structure loss. Calculation formula: ,in For weight parameters, This represents the maximum number of nodes for the generated text and the reference text.

[0095] Joint optimization mechanism: The above loss functions are combined in a weighted manner to form the final optimization objective. ,in This is a hyperparameter.

[0096] During backpropagation, the joint loss function simultaneously optimizes the following aspects: enhancing the semantic understanding ability of the model through masked language modeling loss, ensuring the fluency of text generation through autoregressive language modeling loss, and enforcing the compliance of grammatical structures through language structure loss.

[0097] The trained target model is saved, i.e., the complete model parameter system is saved, including all trainable parameters of the visual encoder, the audio encoder, the cross-modal fusion module, the structure perception encoder, and the decoder, to ensure that the inference stage can completely reproduce the capabilities obtained in the training stage.

[0098] The specific implementation of obtaining the cross-modal joint representation in the inference stage generates the cross-modal joint representation based on the fixed model parameters obtained after training. The core difference from the training stage is that the encoder with frozen parameters is used for feature extraction, the optimal attention weight matrix is directly applied to realize cross-modal fusion, gradient calculation and parameter updating are cancelled, only the forward propagation process is retained, and the syntax constraint strength is fixed as the optimal value obtained in the training. Thus, the multi-modal understanding capability obtained in the training is inherited, and efficient and stable inference calculation is realized.

[0099] The specific implementation of the decoder generating the predicted text in the inference stage is that, after loading the model parameters optimized in the training, the decoder takes the syntax-aligned cross-modal joint representation as the initial context input and generates a text sequence word by word in a self-recurrent manner. In the specific implementation, the model is based on the parameters obtained in the joint optimization in the training stage (including the word prediction capability optimized by the self-recurrent language modeling loss and the syntactic constraint strengthened by the language structure loss), dynamically calculates the attention weight when generating each word, and ensures that the output text not only maintains semantic accuracy but also conforms to the syntax specification. This process is completely based on fixed parameters to perform forward calculation, without gradient updating.

[0100] To make the skilled in the art better understand and implement the method for understanding video content based on structured syntax information provided by the embodiments of the present application, the method for understanding video content based on structured syntax information will be described in detail below in conjunction with specific examples.

[0101] From the above, it can be seen that the present application focuses on the task of audio-video content understanding. However, current audio-video content understanding mainly relies on end-to-end deep learning methods, which can capture semantic associations, but lack explicit modeling of language structure, resulting in problems such as chaotic sentence patterns and syntax errors in generated text, which seriously restricts the actual application effect.

[0102] To solve the existing problems, the application provides an audio and video content understanding method based on structured syntax information enhancement. In the training stage, the model first extracts multi-modal features through visual and audio encoders. The visual encoder adopts a hierarchical spatiotemporal attention mechanism to gradually capture semantic information from local motion to global scene; the audio encoder combines convolution and temporal modeling to extract the spectral features of the sound signal. Subsequently, the cross-modal fusion module guides the adaptive fusion of features through dynamic attention weights while preserving the original syntax structure information with reference to the text. The structure perception encoder injects syntax constraints into the standard converter architecture, guiding the feature encoding process through a pre-defined syntax rule matrix. The decoder adopts a joint optimization strategy to minimize the autoregressive language modeling loss and the language structure loss based on the syntax tree edit distance, enabling the model parameters to both grasp semantic understanding and internalize grammar rule knowledge.

[0103] The inference stage directly loads the trained model parameters for processing. The input audio and video first undergo feature extraction and cross-modal fusion to generate syntax-aware joint representations. When generating text, the decoder injects syntax constraint information and global syntax consistency checks to ensure grammatical correctness during the initialization of the decoding state. The entire process is based on forward calculation of the parameters obtained during training, without the need for additional corrections.

[0104] This method realizes syntax-controllable audio and video content understanding through explicit learning of syntax rules in the training stage and constraint generation in the inference stage, significantly improving the semantic completeness and grammatical compliance of the generated text.

[0105] Specifically, the specific implementation process of the method includes the following steps:

[0106] In the training and inference stages, video visual features and audio features are extracted.

[0107] The input audio and video data includes: video data: frame sequence , where is the number of frames, is the spatial resolution, H is high, and W is wide, represents the RGB three-channel color information of each video frame, R is the pixel value (valued between -255 and +255); audio data: mel spectrogram , F is the frequency dimension, and M is the number of time frames; text data: corresponding text description , L is the length of the text.

[0108] To capture multi-level spatiotemporal information in the video, when extracting visual features, the original video frame V is processed through a spatiotemporal modeling network, a hierarchical window attention mechanism is used to extract local motion features and global scene features at different scales, and the spatiotemporal features As an output, d is the feature dimension.

[0109] In order to extract audio feature representation with time sequence correlation, when extracting audio features, the audio spectrum A is processed by an audio coding network based on spectral block division and multi-scale perception mechanism, frequency domain features are extracted using a multi-scale convolution kernel group, time sequence dependence is modeled by combining self-attention, and audio features are output .

[0110] In the process of obtaining multi-modal features in the training stage, in order to realize the feature alignment and information complementarity of vision-audio-text, dynamic attention mask fusion is performed on the and text features , cross-modal attention weights are calculated, dynamic mask coefficients are generated and weighted fusion of multi-modal features is performed, and finally fusion features are output .

[0111] In the training stage, the syntax-aligned cross-modal joint representation is generated, in order to ensure that the feature representation conforms to the syntax specification, the feature encoding combined with the syntax constraint is injected, the syntax structure is refined through hierarchical attention, and the fusion features are converted into syntax-aligned representations .

[0112] In the training stage, the decoder generates predicted text, and the specific implementation of the decoder adopts a multi-layer transformer architecture, each layer containing three core sub-modules: a self-attention mechanism processes the generated sequence , establishing the dependency between words; a cross-attention mechanism aligns to obtain content-related context information; and a feedforward neural network performs feature transformation. At each time step t, the decoder receives the embedding representation of the first t-1 words, calculates the current hidden state through interaction with , and finally obtains the probability distribution on the vocabulary through the output layer transformation , generating the corresponding predicted text sequence.

[0113] The joint autoregressive language modeling loss and language structure loss are used to update the model parameters, and in the process of autoregressively generating the target text by the decoder, the mask language modeling loss between the predicted word and the true word at each step is calculated, that is, the autoregressive language modeling loss ; based on the mask mechanism, the random mask position in the predicted text is predicted, and the difference between the predicted result and the true word is calculated, that is, the mask language modeling loss ; the syntax tree of the reference text is constructed; based on the syntax structure between the predicted text and the reference text, the syntax tree edit distance is calculated, and the language structure loss is further obtained ; the autoregressive language modeling loss and the language structure loss are combined in a weighted form as the total loss function L: , realize the collaborative optimization of the driving model parameters, wherein is a hyperparameter.

[0114] Save the trained model, that is, save the complete model parameter system, including all trainable parameters of the visual encoder, the audio encoder, the cross-modal fusion module, the structure perception encoder, and the decoder, to ensure that the inference stage can completely reproduce the various capabilities obtained in the training stage.

[0115] In the inference stage, cross-modal representations are obtained, and the encoder attention weights obtained in the training are applied to fuse the audio and video features, and the fused features are converted into syntax alignment representations .

[0116] In the inference stage, the decoder generates predicted text, and based on the decoder parameters obtained by optimizing the joint loss in the training stage, the decoder generates a text sequence .

[0117] To verify the effectiveness of the tree edit distance (TED) optimization mechanism introduced in this application in the task of multi-granularity description of audio and video content, the performance difference of the model before and after adding the TED structure optimization can be compared and analyzed based on the public benchmark dataset FAVDBench, thereby verifying it. The FAVDBench dataset contains 11,424 clips of video with detailed audio and video descriptions, including 5 categories and 71 fine-grained semantic labels, covering five dimensions of people, animals, musical instruments, vehicles, and common activities, and providing bilingual annotations in Chinese and English. This dataset focuses on semantic hierarchy and structural integrity, and is suitable for evaluating the comprehensive performance of syntax structure modeling and natural language generation systems.

[0118] The experimental process includes the following steps.

[0119] Step 1: System settings and training process.

[0120] 1. Model framework: adopt a multi-modal transformer structure, input including visual, auditory features and text features, and realize inter-modal fusion through cross-attention.

[0121] 2. Experimental variable control: scheme A (no TED optimization): only use autoregressive language modeling loss for training; scheme B (add TED optimization): introduce a syntax tree edit distance loss term based on scheme A, which is used to constrain the consistency between the syntax structure of the generated sentence and the reference syntax tree, forming a joint loss function.

[0122] 3. Training data division: 7,500 videos in FAVDBench are used as the training set, 1,500 videos are used as the validation set, and 1,000 videos are used as the test set for hyperparameter tuning and final evaluation.

[0123] Step two: index comparison and effect evaluation.

[0124] The BLEU-4, METEOR, ROUGE-L, and CIDEr four automatic evaluation indexes are used on the test set to compare the model performance before and after the structure optimization. In the audio-video content understanding task, BLEU-4, ROUGE, METEOR, and CIDEr are four core indexes for evaluating the quality of generated text. They test the performance of the model from different angles.

[0125] Among them: BLEU-4 evaluates the phrase-level accuracy through 4-gram exact matching, reflecting the accuracy of the model in describing key actions and objects; ROUGE measures the coverage ability of the generated text to the core content of the reference text based on recall, reflecting the ability of the model to capture main information; METEOR introduces synonym and stem analysis to comprehensively evaluate semantic similarity, reflecting the fluency and diversity of language expression; CIDEr uses weighted multi-gram similarity, emphasizing the consistency of the generated text with human consensus, highlighting the richness of the description details.

[0126] Statistics are shown in Table 1.

[0127] Table 1 is a statistical analysis table

[0128]

[0129] As can be seen, based on the results of the present embodiment, when the above indexes are improved simultaneously, it indicates that the model is optimized in three key dimensions of semantic accuracy (BLEU-4, ROUGE), language standardization (METEOR), and description richness (CIDEr). Especially after introducing structured syntax information, not only does it directly improve the syntax compliance rate through syntax tree supervision, but also indirectly enhances the semantic expression ability - the BLEU-4 improvement shows that the key phrase generation is more accurate, the ROUGE growth indicates that the core content coverage is more comprehensive, the METEOR progress reflects that the expression method is more natural and fluent, and the CIDEr improvement embodies that the detail description is more vivid and rich. This multi-index collaborative optimization verifies that the grammar enhancement method maintains semantic completeness while significantly improving the modeling ability of the multi-modal video content description system at the syntax level due to the tree edit distance structure optimization mechanism introduced by the present invention, enhancing the logicality and standardization of language expression, which has important practical value and promotional significance.

[0130] Corresponding to the above method embodiments, the embodiments of the present application also provide a video content understanding apparatus based on structured syntax information. The video content understanding apparatus based on structured syntax information described below can be mutually corresponding with the video content understanding method based on structured syntax information described above.

[0131] Referring to Figure 2 As shown in the figure, the apparatus comprises the following modules:

[0132] The model training module 101 is configured to input a training sample into a target model for content understanding processing, obtain a predicted text, and construct a syntax tree corresponding to the predicted text; calculate a syntax tree edit distance between the syntax tree and a syntax tree of a reference text; calculate a language structure loss by using the syntax tree edit distance, and update model parameters of the target model by using the language structure loss.

[0133] The video acquisition module 102 is configured to acquire a target video in the case that the model parameters of the target model are trained.

[0134] The prediction application module 103 is configured to input the target video into the target model for processing, and obtain a content text of the target video.

[0135] By using the apparatus provided in the embodiments of the present application, a training sample is input into a target model for content understanding processing, a predicted text is obtained, and a syntax tree corresponding to the predicted text is constructed; a syntax tree edit distance between the syntax tree and a syntax tree of a reference text is calculated; a language structure loss is calculated by using the syntax tree edit distance, and model parameters of the target model are updated by using the language structure loss; a target video is acquired in the case that the model parameters of the target model are trained; and the target video is input into the target model for processing, and a content text of the target video is obtained.

[0136] The application aims to enhance the ability of video content understanding and natural language description generation by using syntactic structure information, so as to obtain a multi-modal content understanding model with accurate semantic expression, reasonable grammar structure and logical coherent content. First, the target model is trained based on training samples. That is, the target model is used to understand the content of the training samples, thereby obtaining a predicted text. In order to enable the target model to output a text that conforms to the grammar structure, in the application, a corresponding grammar tree is constructed for the predicted text, and a language structure loss is determined based on the syntactic tree edit distance between the grammar trees corresponding to the predicted text and the reference text. Adjusting the model reference of the target model based on the language structure loss can enable the target model to recognize the content of the input video and output content text conforming to the grammar structure after training. After training, the target video that needs to be understood can be obtained, and the target video is input into the target model for content understanding, thereby obtaining content text conforming to the grammar structure. Since the language structure loss calculated based on the grammar tree is used to train the target model during the model training process, the content text has the characteristics of accurate grammar, reasonable sentence structure and naturalness.

[0137] In an embodiment of the application, the model training module is specifically configured to construct a grammar structure representation using the grammar components of the predicted text and the grammar relationship between the grammar components; construct a tree structure with words as nodes and grammar relationships as edges based on the grammar structure representation to obtain a syntactic tree; wherein the syntactic tree includes annotations of the grammar roles and hierarchical relationships of the nodes.

[0138] In an embodiment of the application, the model training module is specifically configured to align the nodes of the two syntactic trees and calculate the minimum editing operation cost required to make the two trees identical, the editing operation including at least one of node replacement, deletion and insertion; and determine the minimum editing operation cost as the syntactic tree edit distance.

[0139] In an embodiment of the application, the target model includes an audio encoder, a visual encoder, a cross-modal fusion module, a structure perception encoder and a decoder, and the prediction module is configured to extract audio features of the target video using the audio encoder, and extract visual features of the target video using the visual encoder in the target model; fuse the audio features and the visual features using the cross-modal fusion module to obtain grammar-aware multi-modal features; encode the multi-modal features using the structure perception encoder to obtain grammar-aligned cross-modal joint representation; and decode the cross-modal joint representation using the decoder to obtain the content text.

[0140] In an embodiment of the application, it further includes:

[0141] The subtitle automatic generation module is configured to fuse the content text and the target video to obtain an output video with subtitles.

[0142] In an embodiment of the present application, the model training module is specifically configured to calculate an autoregressive language modeling loss corresponding to the predicted text and the reference text; calculate a masked language modeling loss corresponding to the predicted text and the reference text; and adjust the model parameters in combination with the autoregressive language modeling loss, the masked language modeling loss, and the language structure loss.

[0143] In an embodiment of the present application, the model training module is specifically configured to calculate a masked language modeling loss of each predicted word and a real word in the process of autoregressively generating the predicted text by the decoder; wherein the predicted word is a word in the predicted text, and the real word is a word in the reference text.

[0144] In addition, in practical applications, the steps of the method can also be implemented in a system. In practical applications, the system for implementing the video content understanding method based on structured syntax information, that is, an audio-visual content understanding system enhanced based on structured syntax information, is composed of a hierarchical feature extraction module, a syntax-aware fusion module, a joint representation encoding module, a constraint decoding module, and a multi-objective optimization module. The hierarchical feature extraction module extracts the spatiotemporal visual features of the video and the multi-scale spectral features of the audio through a spatiotemporal modeling network and an audio encoding network, respectively; the syntax-aware fusion module realizes cross-modal alignment of visual-text and audio-text using a cross-attention mechanism, and injects the syntax tree structure information of the reference text; the joint representation encoding module generates a multi-modal joint representation under the constraint of syntax using a structure-guided encoder; the constraint decoding module dynamically adjusts the syntax rationality of the generated sequence by combining the autoregressive generation and the real-time syntax verification mechanism; and the multi-objective optimization module drives the model training through the weighted combination of the autoregressive language modeling loss and the language structure loss. Through the end-to-end collaborative working mechanism of the modules, the semantic completeness is maintained while the grammatical compliance of the generated text is significantly improved, and the innovative design of the constraint decoding module effectively solves the shortcomings of traditional methods in terms of grammatical compliance.

[0145] Corresponding to the method embodiments above, the embodiments of the present application also provide an electronic device. The electronic device described below can correspond to the video content understanding method based on structured syntax information described above.

[0146] Referring to Figure 3 As shown in the figure, the electronic device includes:

[0147] The memory 332 is configured to store a computer program.

[0148] A processor 322, which is configured to implement the steps of the method for video content understanding based on structured syntax information in the above method embodiments when executing a computer program.

[0149] Specifically, please refer to Figure 4 , Figure 4 FIG. [FIG. number not provided in the original] is a schematic structural diagram of an electronic device provided in this embodiment. The electronic device may vary greatly due to different configurations or performances, and may include one or more processors (central processing units, CPUs) (for example, one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 may be a transient storage or a persistent storage. The programs stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the data processing device. Further, the processor 322 may be configured to communicate with the memory 332 and execute a series of instruction operations in the memory 332 on the electronic device 301.

[0150] The electronic device 301 may further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0151] The steps in the above-described method for video content understanding based on structured syntax information may be implemented by the structure of the electronic device.

[0152] Corresponding to the above method embodiments, an embodiment of the present application also provides a readable storage medium. The following-described readable storage medium may be mutually corresponding and referred to with the above-described method for video content understanding based on structured syntax information.

[0153] A readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the method for video content understanding based on structured syntax information in the above method embodiments.

[0154] The readable storage medium may specifically be various readable storage media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0155] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0156] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or the combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to functions in the above description. Whether these functions are executed in the form of hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0157] The steps of the methods or algorithms described in combination with the embodiments disclosed in this article can be directly implemented by hardware, software modules executed by a processor, or the combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0158] Finally, it should also be noted that in this article, relationships such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0159] Specific examples are used in this article to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A method for video content understanding based on structured syntax information, characterized in that, The method comprises the following steps: inputting a training sample into a target model for content understanding processing to obtain a predicted text, and constructing a syntax tree corresponding to the predicted text; calculating a syntax tree edit distance between the syntax tree and a syntax tree of a reference text; calculating a language structure loss by using the syntax tree edit distance, and updating model parameters of the target model by using the language structure loss; in a case where the model parameters of the target model are trained, obtaining a target video; inputting the target video into the target model for processing to obtain a content text of the target video.

2. The method of claim 1, wherein, The method for constructing the syntax tree corresponding to the predicted text comprises the following steps: constructing a syntax structure representation by using a grammatical component of the predicted text and a grammatical relationship between the grammatical components; constructing a tree structure with words as nodes and grammatical relationships as edges based on the syntax structure representation to obtain the syntax tree; wherein the syntax tree comprises annotations of grammatical roles and hierarchical relationships of each node.

3. The method of claim 1, wherein, The method for calculating the syntax tree edit distance between the syntax tree and the syntax tree of the reference text comprises the following steps: aligning nodes of the two syntax trees and calculating a minimum edit operation cost required to make the two trees identical, wherein the edit operation comprises at least one of node replacement, deletion and insertion; determining the minimum edit operation cost as the syntax tree edit distance.

4. The method of claim 1, wherein, The target model comprises an audio encoder, a visual encoder, a cross-modal fusion module, a structure perception encoder and a decoder. The method for obtaining the content text of the target video by inputting the target video into the target model for processing comprises the following steps: extracting audio features of the target video by using the audio encoder, and extracting visual features of the target video by using the visual encoder in the target model; fusing the audio features and the visual features by using the cross-modal fusion module to obtain syntax-aware multi-modal features; encoding the multi-modal features by using the structure perception encoder to obtain syntax-aligned cross-modal joint representations; decoding the cross-modal joint representations by using the decoder to obtain the content text.

5. The method of claim 1, wherein, After obtaining the content text of the target video, the method further comprises the following steps: fusing the content text and the target video to obtain an output video with subtitles.

6. The method according to any one of claims 1 to 5, characterized in that, The method for updating the model parameters of the target model by using the language structure loss comprises the following steps: calculating an autoregressive language modeling loss corresponding to the predicted text and the reference text; calculating a masked language modeling loss corresponding to the predicted text and the reference text; adjusting the model parameters by combining the autoregressive language modeling loss, the masked language modeling loss and the language structure loss.

7. The method of claim 6, wherein, The method for calculating the autoregressive language modeling loss corresponding to the predicted text and the reference text comprises the following steps: during autoregressive generation of the predicted text by using the decoder, calculating a masked language modeling loss of each predicted word and a real word; wherein the predicted word is a word in the predicted text, and the real word is a word in the reference text.

8. An apparatus for video content understanding based on structured syntax information, characterized in that, The method comprises the following steps: a model training module configured to input a training sample into a target model for content understanding processing to obtain a predicted text, and construct a syntax tree corresponding to the predicted text; calculating a syntax tree edit distance between the syntax tree and a syntax tree of a reference text; calculating a language structure loss using the syntax tree edit distance, and updating model parameters of the target model using the language structure loss; a video obtaining module, configured to obtain a target video after the model parameters of the target model are trained; a prediction application module, configured to input the target video into the target model for processing to obtain content text of the target video.

9. An electronic device, comprising: comprising: a memory, configured to store a computer program; a processor, configured to implement steps of the method for video content understanding based on structured syntax information according to any one of claims 1 to 7 when the computer program is executed.

10. A readable storage medium, characterized by, The readable storage medium has a computer program stored thereon, and the computer program is executed by the processor to implement steps of the method for video content understanding based on structured syntax information according to any one of claims 1 to 7.

Images