Video content description method based on concept parser and multimodal graph learning

Abstract

The application discloses a video content description method based on a concept resolver and multi-modal graph learning, which simultaneously considers low-level visual information and high-level concept clues and effectively integrates them in an end-to-end manner. Specifically, the low-level visual information and the high-level concept clues are obtained from a VideoSwinTransformer and a concept resolver respectively, and a concept loss is introduced to regulate the learning process of the concept resolver to generate real concept results. In order to fuse multi-level features, the method introduces a CaptionTransformer, which takes visual information and concept clues as input, processes the input information through multi-modal graph learning. The multi-modal graph models the relationship between multi-level features, including text features, visual features and concept features, in an explicit manner, and further refines their importance.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a video content description method based on a concept parser and multimodal graph learning. Background Art

[0002] With the rapid development of information technology and the increasing number of short video platforms, video has gradually become the primary medium for people to share information about their daily lives, learn, socialize, and entertain themselves. Consequently, video description has become a hot topic in the field of vision, with a wide range of applications, including detecting harmful content in videos, extracting valid video segments, monitoring security in surveillance videos, video annotation, and automatic captioning.

[0003] As one of the most popular tasks in cross-modal learning, video captioning aims to leverage both visual and linguistic information to describe the content of a video in natural language. A range of existing methods address this problem by extracting low-level visual cues and additional scene or semantic information to generate captions. While these methods have achieved some success, they are typically staged and utilize off-the-shelf models originally designed for other tasks, such as scene graph generation and natural language translation, for feature extraction. As a result, these features cannot be updated with the specific task, leading to the introduction of significant noise in the subsequent video caption generation. Alternatively, some methods propose an end-to-end framework that incorporates the Transformer for feature extraction and leverages its strengths to mitigate the semantic gap between the visual and caption spaces. While this approach addresses the challenges of these staged models, these methods still lack high-level cues, such as event or content descriptions. In summary, video captioning lacks a unified framework that can capture multi-level cues in an end-to-end manner. Summary of the Invention

[0004] To address the shortcomings of the existing technology, this paper proposes a video content description method (CAT) based on a concept parser and multimodal graph learning. This method addresses the potential for additional noise introduced by staged approaches and the lack of high-level cues in end-to-end approaches. CAT is a novel unified framework that simultaneously considers low-level visual information and high-level conceptual cues, effectively integrating them in an end-to-end manner. Specifically, low-level visual information and high-level conceptual cues are obtained from the Video Swin Transformer and the concept parser, respectively. A concept loss is further introduced to regularize the concept parser's learning process to produce realistic conceptual results. To integrate multi-level features, this method introduces the CaptionTransformer, which takes visual information and conceptual cues as input to generate video descriptions. A key design choice in the CaptionTransformer is to process this input through multimodal graph learning. The multimodal graph explicitly models the relationships between multi-level features—text features, visual features, and conceptual features—and further refines their importance.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] The video content description method based on concept parser and multimodal graph learning has the following steps:

[0007] Step 1. Dataset and concept acquisition: We used three public video description datasets: MSVD, MSR-VTT, and VATEX. We used NLTK to extract nouns and verbs from real video description text as pseudo-real concepts.

[0008] Step 2. Visual feature extraction: sample the input video to obtain video frames, use the video frames as input to VideoSwin Transformer and finally output visual features.

[0009] Step 3. Concept feature parsing: input the visual features into the concept parser to obtain high-level concept features.

[0010] Step 4: Caption generation guided by the multimodal graph. Using the Caption Transformer, the multimodal graph intelligently bridges the gap between description, visual, and conceptual spaces, generating high-quality video captions. The Caption Transformer takes three modal features as input: textual, visual, and conceptual. The textual features are encoded using the BERT model to create masked text. The Caption Transformer then predicts masked words to complete the sentence.

[0011] Step 5. Determine the loss function. The loss function is divided into three parts, including concept loss Multimodal Graph Learning Loss and video description loss Optimize in an end-to-end manner.

[0012] Step 6: Network training and inference. The Video Swin Transformer uses model parameters pre-trained on the video dataset. The Concept Parser and Caption Transformer are trained with random initialization. During network training, the GT video description is masked, and the goal is to correctly predict the masked words. During network inference, the text is simply a masked token, and is generated word by word in an autoregressive manner until the description is complete. The final evaluation metrics are BLEU@4 (B@4), METEOR (M), ROUGE-L (R), and CIDEr (C).

[0013] The visual feature extraction described in step 2 is specifically implemented as follows:

[0014] The input video is densely sampled into a size of T×H×W×3, which includes T frames, each frame includes H×W×3 pixels. Then it is input into Video Swin Transformer and the output is visual features and express them as v i represents the i-th visual feature, N and D represent the total number and dimension of visual features respectively.

[0015] The concept parser described in step 3 is implemented as follows:

[0016] The visual features are input into the concept parser. After pooling and projection operations, the visual features are mapped to the concept space. Then the top k concepts with the highest probability of occurrence are selected and encoded as the input of the subsequent Caption Transformer.

[0017] The specific method for generating descriptions under the guidance of the multimodal graph described in step 4 is as follows:

[0018] Caption Transformer takes the visual features in step 2, the conceptual features in step 3, and the text features ω as input. Through the BERT model, the masked sentence s mask to encode.

[0019] On the Caption Transformer, the relationship between multi-level features, including text features, visual features, and conceptual features, is explicitly modeled through a multimodal graph, and their importance is further refined to enhance the representational capabilities of visual features and conceptual features.

[0020] Step 5. The loss function is divided into three parts, including concept loss Multimodal Graph Learning Loss and video description loss The loss is optimized in an end-to-end manner.

[0021] The total training loss formula is as follows:

[0022]

[0023] Where λ and γ are hyperparameters, and the training goal is to minimize the loss

[0024] Step 6. Network training and inference. During network training, the Video Swin Transformer is initialized using model parameters pre-trained on the Kinetics-600 video dataset. The other two modules, the concept parser and the Caption Transformer, are trained using random initialization. After inputting visual features into the concept parser, the top-k concepts are obtained and compared with the pseudo-real concepts to calculate the concept loss, requiring it to focus on visual features and refine correct, high-quality concepts. A certain percentage of words in the video content description of the corresponding dataset GT in step 1 are then randomly selected and replaced with a special token [MASK], thereby implementing a masking operation. The masked sentence is then encoded and input into the Caption Transformer along with the video features from step 2 and the concept features from step 3. The goal of the Caption Transformer is to predict the real words in the masked sentence to complete the complete sentence s.

[0025]

[0026] Where f in Formula 2 cap It represents Caption Transformer.

[0027] During the network inference process, the masked sentence contains only one [SEP] tag, and the masked sentence is encoded and input into the Caption Transformer at the same time as the video features in step 2 and the concept features in step 3. The video description is output in an autoregressive manner, that is, a word is predicted and generated each time, and then the previously predicted sentence is re-encoded and input into the Caption Transformer again. This process is repeated until the number of words generated reaches the preset maximum value or the pre-defined terminator [EOS] is generated.

[0028] Finally, the evaluation indicators are used for evaluation, including BLEU@4(B@4), METEOR(M), ROUGE-L(R) and CIDEr(C).

[0029] The beneficial effects of the present invention are as follows:

[0030] The present invention provides a video content description method based on concept parser and multimodal graph learning - CAT, to address the problems of staged methods that may introduce additional noise and the lack of high-level clues in end-to-end methods. CAT is a novel unified framework that simultaneously considers low-level visual information and high-level conceptual clues and effectively integrates them in an end-to-end manner. Specifically, low-level visual information and high-level conceptual clues are obtained from Video SwinTransformer and concept parser respectively, and concept loss is further introduced to regularize the learning process of the concept parser to generate realistic conceptual results. In order to fuse multi-level features, this method introduces Caption Transformer, which takes visual information and conceptual clues as input to generate video descriptions. In Caption Transformer, this method makes a key design choice to process these input information through multimodal graph learning. The multimodal graph will explicitly model the relationship between multi-level features, including text features, visual features, and conceptual features, and will further refine their importance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the present invention.

[0032] Figure 2 It is a schematic diagram of the entire model framework in the invention method.

[0033] Figure 3 It is a schematic diagram of the concept parser part of the invention method. DETAILED DESCRIPTION

[0034] The present invention will be further described below with reference to the accompanying drawings and examples.

[0035] The application provides a video content description method based on concept resolver and multi-modal graph learning-CAT to solve the problems of introducing additional noise caused by the phased method and the lack of advanced clues of the end-to-end method. CAT is a novel unified framework that considers low-level visual information and high-level concept clues and effectively integrates them in an end-to-end manner. Specifically, the low-level visual information and the high-level concept clues are obtained from the Video SwinTransformer and the concept resolver respectively, and the concept loss is further introduced to regulate the learning process of the concept resolver to generate real concept results. In order to fuse multi-level features, the method introduces a caption transformer which takes visual information and concept clues as input to generate video descriptions. In the caption transformer, the method makes key design choices to process these input information through multi-modal graph learning. The multi-modal graph models the relationship between multi-level features, including text features, visual features and concept features, in an explicit manner, and further refines their importance.

[0036] As shown in Figure 1 The video content description method based on concept resolver and multi-modal graph learning flow chart specifically includes the following steps:

[0037] Step 1. Data set and concept acquisition, the method uses three public video description data sets, which are MSVD, MSR-VTT and VATEX. Among them, in the division of training, verification and test sets of the three data sets, the official division method of the data set is followed. The MSVD data set includes 1970 short video clips on YouTube, each video has forty English video descriptions. The method divides it into three parts, of which 1200 videos are used for training, 100 videos are used for verification, and 670 videos are used for testing. The MSR-VTT data set includes 10000 open videos, each video includes 20 English descriptions, and the method divides it according to the official division method: 6513 videos are used for training, 497 videos are used for verification, and 2990 videos are used for testing. Finally, the VATEX data set is a large-scale data set, which includes 41269 videos, each video includes 10 longer and higher quality English video descriptions. The method divides it according to the official division method: 25991 videos are used for training, and 6000 videos are used for testing.

[0038] In order to effectively use real video description text without requiring additional manual annotation, the method selects pseudo-real concepts from real video description text. Specifically, the nouns and verbs in the real video description text are extracted as pseudo-real concepts using the NLTK tool.

[0039] Step 2. Visual feature extraction, such as Figure 2 As shown in the overall block diagram of the model, this method uses dense sampling of the input video into a size of T×H×W×3, which includes T frames, each frame includes H×W×3 pixels. Then it is input into Video SwinTransformer and output visual features and express them as v i Denotes the i-th visual feature, N and D denote the total number of visual features and their dimensions, respectively. The output visual features serve as input to the subsequent concept parser and Caption Transformer to provide low-dimensional visual information. In this method, dense sampling of 64 frames is used for processing the original videos of all datasets, and the size of each frame is set to 224×224. That is, T is set to 64, H and W are set to 224, and N is calculated to be 1568. Finally, the dimension of each feature vector is set to 1024.

[0040] Step 3. Concept feature analysis, such as Figure 3 As shown in the diagram of the concept parser, the concept parser consists of a pooling layer and two projection layers. In the pooling layer, N visual features are first averaged to obtain in Represents the representation of the visual level. The subsequent projection layer will As input, the output concept distribution Where K is the length of the entire concept space set. The concept space set consists of nouns and verbs in the GT video content description text in each dataset. This method regards the concept prediction problem as a multi-classification task, where Represents the probability of the output c-th concept appearing in this video sequence. Then all concepts are sorted according to probability, and the top k concepts with the highest probability of occurrence are selected and encoded into concept features. And use it as the input of the subsequent CaptionTransformer. In this method, k is set to 25.

[0041] Step 4. Description generation guided by multimodal graphs, such as Figure 2 As shown in the overall block diagram of the model, CaptionTransformer f cap The visual features in step 2, the conceptual features in step 3, and the text features ω are taken as input. The BERT model will be used to mask the sentence s mask Encode. CaptionTransformer f cap The goal is to predict the real words in the masked sentence to complete the complete sentence s.

[0042] Previous research has revealed that processing a large number of features in the Transformer can lead to a drop in computational efficiency, and that a large number of features can cause the activation function in the Transformer's attention mechanism to affect performance. To alleviate this problem, this method applies multimodal graphs to the Caption Transformer.

[0043] Multimodal graph G = {n a , e a，b} a，b , consisting of (M+N+k) nodes and (M+N+k)×(M+N+k) edges. a Represents the ath node, which can be a text feature, a visual feature, or a conceptual feature. a，b represents the edge between node a and node b. The goal of multimodal graphs is to learn e a，b , so that the connection between multi-level features can be captured efficiently. This method regards such a learning process as a matrix learning problem, and regards the multimodal graph G as a (M+N+k)×(M+N+k) matrix A, where the position of (a, b) is equivalent to e a，b This learnable matrix is ​​then introduced into the attention module to refine the input nodes and improve computational efficiency, where the initial node is The processed node is n′. It can be expressed as follows:

[0044] n=[w;c;v], (Formula 1)

[0045] q, k, v = nW q ,nW k ,nW v , (Formula 2)

[0046] A=λ(1-A), (Formula 3)

[0047]

[0048] Where q, k, and v represent query, key, and value features respectively. q , W k , W v are three learnable matrices with the same dimensions. λ is a hyperparameter, usually set to a large negative number, and is set to -1e8 in this experiment. k is the dimension of k. [;] means concatenating the two features.

[0049] Step 5. Loss function, the loss function is divided into three parts, including concept loss Multimodal Graph Learning Loss and video description loss The loss is optimized in an end-to-end manner. The total training loss is defined as follows:

[0050]

[0051] Where λ and γ are hyperparameters, and the training goal is to minimize the loss In this method, λ and γ are set to 0.5.

[0052] Concept Loss In order to ensure that the features output by the concept parser are concept-related, this method introduces concept loss to make the concept parser output concepts that are as similar as possible to the pseudo-true concepts. The cross entropy function is used to calculate the loss:

[0053]

[0054] in represents the cth pseudo-real concept, p c ∈[0, 1] indicates the possibility of predicting the c-th concept.

[0055] Multimodal Graph Learning Loss

[0056] The matrix A is first initialized with prior knowledge and then trained initialization =1, where a1∈[1,…,M], b1∈[M,…,M+N+k], which means that the generation of video description will be based on visual features and conceptual features, and then the sequence mask is set in the description area, that is, the area consisting of the Mth row and Mth column at the beginning of the matrix A, to achieve sequence-to-sequence description. After that, only the visual and conceptual areas in the matrix will be updated, that is, A'∈ The multimodal graph consists only of visual and conceptual nodes, and the edges connecting them. This design is based on the understanding that while concepts and visual features can provide information to each other, they may also contribute some irrelevant, unique semantics. The multimodal graph designed by this method eliminates these semantic edges as much as possible while retaining the important information.

[0057]

[0058] where A' a'，b' It is equivalent to e in the multimodal graph a'+M，b'+M . Where a'∈[1,…,N+k], b'∈[1,…,N+k];

[0059] Video Description Loss

[0060] As in (Formula 1) s = f cap (s mask As described in

[15] , v, c), this method uses the masked sentence as the input of Caption Transformer, and according to the design of the BERT model, Caption Transformer will predict the true value at the mask position. Video Description Loss Use MLM loss, that is:

[0061]

[0062] where m(x) and \m(x) represent the masked words and the remaining words in the description, respectively.

[0063] Step 6. Network training and inference. During the network training process, Video SwinTransformer is initialized using the model parameters pre-trained on the Kinetics-600 video dataset. The other two modules, including the concept parser and Caption Transformer, are trained using random initialization. After the visual features are input into the concept parser, the top-k concepts are obtained, and the concept loss is calculated with the pseudo-real concepts, which requires it to focus on visual features and refine correct, high-quality concepts. Then, 15% of the words in the video content description of the corresponding dataset GT in step 1 are randomly selected for replacement, and 80% of the words will be replaced with a special tag [MASK], 10% of the words will be replaced with random tags, and the remaining 10% of the words remain unchanged. The masked sentence is then encoded and input into the Caption Transformer along with the video features in step 2 and the concept features in step 3. The goal of the Caption Transformer is to predict the real words in the masked sentence to complete the complete sentence s. That is, by reducing the loss in step 5 Pass the gradient back to Video Swin Transformer, Concept Parser, and Caption Transformer to train the model.

[0064] During network inference, the masked sentence contains only one [SEP] tag. The masked sentence is encoded and input into the Caption Transformer together with the video features in step 2 and the concept features in step 3. The video description is output in an autoregressive manner. That is, a word is predicted each time, and then the previously predicted sentences are concatenated and re-encoded and input into the Caption Transformer again. This process is repeated until the number of generated words reaches the preset maximum value or the pre-defined terminator [EOS] is generated.

[0065] This method undergoes extensive experiments on three public datasets and reports performance on four evaluation metrics: BLEU@4 (B@4), METEOR (M), ROUGE-L (R), and CIDEr (C). Table 1 shows the results of different methods and architectures on the MSVD and MST-VTT datasets, demonstrating the clear advantages of end-to-end architectures. This method outperforms existing methods on both datasets, achieving significant improvements across most metrics. In particular, this method improves the CIDEr metric by 2.3% on the MSVD and 0.7% on the MSR-VTT datasets. The CIDEr metric is specifically designed for video description and shows high consistency with human performance. Table 2 further demonstrates performance on the VATEX dataset, where BLEU@4, METEOR, and ROUGE-L are comparable to state-of-the-art methods. These performance results demonstrate that this method generates richer and more explicit descriptions for the same videos.

[0066] Table 1 Performance comparison of existing methods on MSVD and MSR-VTT datasets

[0067]

[0068] Table 2 Performance comparison on the VATEX public test set

[0069]

[0070] In this method, we propose a unified framework (CAT) consisting of a Video Swin Transformer, a concept parser, and a Caption Transformer. Through supervision based on the generated pseudo-real-world concept loss, our method generates high-level conceptual features in an end-to-end manner. Furthermore, guided by a multimodal graph, our method can better handle the fusion of multi-level features. Extensive experimental results on three benchmark datasets validate our approach.

[0071] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0072] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A video content description method based on concept parser and multimodal graph learning, characterized in that: The steps are as follows: Step 1. Dataset and concept acquisition: We used three public video description datasets: MSVD, MSR-VTT, and VATEX. We used NLTK to extract nouns and verbs from real video description text as pseudo-real concepts. Step 2. Visual feature extraction: sample the input video to obtain video frames, use the video frames as input to Video SwinTransformer, and finally output visual features; Step 3. Concept feature parsing: Input the visual features into the concept parser to obtain high-level concept features. The specific implementation is as follows: The visual features are input into the concept parser. After pooling and projection operations, the visual features are mapped to the concept space. Then, the top k concepts with the highest probability of occurrence are selected and encoded as the input of the subsequent Caption Transformer. Step 4: Caption generation guided by the multimodal graph. Using the Caption Transformer, the multimodal graph guides the gap between description, visual, and conceptual space, thereby outputting high-quality video caption text. The specific method is as follows: Caption Transformer combines the visual features in step 2, the conceptual features in step 3, and the text features As input; the text features The masked sentences are processed by the BERT model. Encoding is performed; In the Caption Transformer, the relationship between multi-level features, including textual features, visual features, and conceptual features, is explicitly modeled through a multimodal graph. Their importance is further refined to enhance the representational capabilities of visual and conceptual features. Step 5. Determine the loss function; Step 6. Network training and inference.

2. The video content description method based on concept parser and multimodal graph learning according to claim 1 is characterized in that The visual feature extraction described in step 2 is specifically implemented as follows: Densely sample the input video into The size of the frame, including T frames, each frame includes pixels; then input it into Video Swin Transformer and output visual features and express them as , Indicates the visual features, and Represent the total number of visual features and dimensions respectively.

3. The video content description method based on concept parser and multimodal graph learning according to claim 2 is characterized in that: Step 5: The loss function is divided into three parts, including concept loss , Multimodal Graph Learning Loss and video description loss , the loss will be optimized in an end-to-end manner; The total training loss formula is as follows: in and is a hyperparameter, and the goal of training is to minimize the loss .

4. The video content description method based on concept parser and multimodal graph learning according to claim 3 is characterized in that: Step 6: Network training and inference. During network training, the Video Swin Transformer is initialized using model parameters pre-trained on the Kinetics-600 video dataset. The other two modules, including the concept parser and the Caption Transformer, are trained using random initialization. After inputting visual features into the concept parser, the top-k concepts are obtained and the concept loss is calculated by comparing them with the pseudo-real concepts, thus requiring it to focus on visual features. Refine the correct and high-quality concepts; then randomly select a certain percentage of words from the video content description of the corresponding dataset GT in step 1 and replace them with a special tag [MASK] to achieve the masking operation; then encode the masked sentence and input it into the Caption Transformer together with the video features in step 2 and the concept features in step 3; the goal of Caption Transformer is to predict the real words in the masked sentence to complete the complete sentence ; Wherein formula 2 It represents Caption Transformer; During network inference, the masked sentence contains only one [SEP] tag. The masked sentence is encoded and fed into the Caption Transformer together with the video features from step 2 and the concept features from step 3. The video description is output in an autoregressive manner. That is, a word is predicted each time, and then the previously predicted sentence is re-encoded and fed into the Caption Transformer again. This process is repeated until the number of generated words reaches the preset maximum or the pre-defined terminator [EOS] is generated. Finally, the evaluation indicators are used for evaluation, including BLEU@4, METEOR, ROUGE-L and CIDEr.

5. The video content description method based on concept parser and multimodal graph learning according to claim 3 is characterized in that: Concept Loss : In order to ensure that the features output by the concept parser are concept-related, a concept loss is introduced to make the concept parser output concepts that are as similar as possible to the pseudo-true concepts. The cross entropy function is used to calculate the loss: in represents the cth pseudo-real concept, Indicates the possibility of predicting the c-th concept.

6. The video content description method based on concept parser and multimodal graph learning according to claim 3 is characterized in that: Multimodal Graph Learning Loss : matrix First, it will be initialized with prior knowledge, and then learn and train ; Initialize edge is 1, where , , which means that the generation of video description will be based on visual features and conceptual features, and then the sequence mask will be set in the description area, that is, the area consisting of the Mth row and Mth column at the beginning of matrix A, to achieve sequence-to-sequence description; after that, only the visual and conceptual areas in the matrix will be updated, that is It consists only of nodes of the visual and conceptual parts, and the edges connecting them; in It is equivalent to the multimodal graph ;in , .

7. The video content description method based on concept parser and multimodal graph learning according to claim 3 is characterized in that: Video Description Loss Use MLM loss, that is: in and Represent the masked words and the remaining words in the description respectively.

Images