Semi-supervised video description generation method enhanced by large language model guided pseudo label

Abstract

The application discloses a large language model guided pseudo-label enhancement semi-supervised video description generation method, comprising the following steps: constructing a double-path collaborative framework, wherein the double-path collaborative framework comprises a video-level coarse-grained pseudo-label generation branch and a high-confidence visual tuple generation branch; inputting a preprocessed video with a base description into the double-path collaborative framework respectively to generate a video-level coarse-grained pseudo-label and a high-confidence visual tuple; inputting the high-confidence visual tuple as a fine-grained visual prompt into an LLaMA large language model together with the video-level coarse-grained pseudo-label, and enhancing the video-level coarse-grained pseudo-label under the guidance of the large language model; and optimizing and generating a final video description through iterative self-training. The method solves the problem that in the prior art, a small amount of base descriptions are not fully assisted by prior knowledge, it is difficult to fully capture the fine-grained differences of structured semantic elements of a video, and the generated video description has semantic deviation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and image processing, and particularly relates to a semi-supervised video description generation method based on large language model guided pseudo-label enhancement. BACKGROUND

[0002] With the rapid development of Internet technology and the wide popularity of social media platforms, videos have gradually become one of the mainstream forms of information dissemination. Whether in social media, e-commerce platforms or education and training fields, the scale of video content generation has increased dramatically. How to quickly obtain the required content in the vast amount of information has become a common problem faced by users. For specific groups (such as visually impaired and cognitively impaired people), there are many obstacles in understanding video content, which further limits their ability to obtain video information. In this context, users have new demands for understanding video content, i.e., how to achieve deep semantic understanding of video data to achieve automatic understanding and efficient dissemination. In order to meet this demand, video description generation methods have emerged.

[0003] Video description generation methods are mainly divided into three categories: fully supervised video description generation method, unsupervised video description generation method and semi-supervised video description generation method. The fully supervised video description generation method relies on a large number of base descriptions and uses an encoder-decoder model to generate natural language descriptions of video content. This method relies on a large number of manually labeled base descriptions, has high labor costs, and has limited applicability. In order to alleviate this problem, further research has been carried out on unsupervised video description generation methods. This method mainly uses semantic matching and reconstruction mechanisms to generate text descriptions of video content. However, due to the lack of base descriptions that align with the semantic content of the video, it is difficult for unsupervised video description generation methods to accurately establish the cross-modal semantic mapping relationship between the video and the text, resulting in the difficulty of generating visual descriptions that are highly consistent with human natural language expressions and video semantic content. At present, semi-supervised video description generation methods mainly use the semantic elements of a small number of base descriptions and the features of a large number of unlabelled data, and use pseudo-label generation methods to achieve accurate video content understanding and rich visual text description, thereby overcoming the above problems. However, the existing semi-supervised video description generation methods do not fully utilize the prior knowledge of the small number of base descriptions, and are difficult to fully capture the fine-grained differences of the structured semantic elements of the video, which may easily lead to semantic bias in the generated video descriptions. SUMMARY

[0004] The purpose of the present application is to provide a large language model guided pseudo label enhanced semi-supervised video description generation method, which solves the problem of insufficient use of prior knowledge in the prior art, which is difficult to fully capture the fine-grained differences of structured semantic elements of the video, resulting in semantic deviation in the generated video description.

[0005] The technical solution adopted by the present application is a large language model guided pseudo label enhanced semi-supervised video description generation method, which specifically includes the following steps:

[0006] Step 1: Construct a dual-path collaborative framework, which includes a video-level coarse-grained pseudo label generation branch and a high-confidence visual tuple generation branch;

[0007] Step 2: Pre-process the video with base description and input it into the dual-path collaborative framework to generate video-level coarse-grained pseudo label and high-confidence visual tuple;

[0008] Step 3: Input the high-confidence visual tuple as a fine-grained visual prompt into the LLaMA large language model together with the video-level coarse-grained pseudo label, and complete the enhancement of the video-level coarse-grained pseudo label under the guidance of the LLaMA large language model;

[0009] Step 4: Optimize and generate the final video description through iterative self-training.

[0010] The present application is also characterized by:

[0011] The video-level coarse-grained pseudo label generation branch in step 1 includes a CoCap network model, which is used to process the video with base description divided into several independent coded images into video-level coarse-grained pseudo label;

[0012] The high-confidence visual tuple generation branch is a VIS-MiniCPM model, which is a MiniCPM-V-2 model output end adding a visual event information selector, the visual event information selector including an LLaMA large language model and a visual tuple generation module; the MiniCPM-V-2 model is used to process the video with base description divided into continuous frames into video frame-level fine-grained visual description; the LLaMA large language model of the visual event information selector is used to process the video frame-level fine-grained visual description into a sentence set, and the visual tuple generation module obtains a weighted score set of each type of word in the sentence set based on the TextRank algorithm and the word frequency algorithm, and selects the highest score word from each type of word to form a high-confidence visual tuple.

[0013] In step 2, the generation process of the video-level coarse-grained pseudo label is as follows:

[0014] The video with base description is pre-processed The continuous frames of the video are divided into several independently coded image groups, as shown in the following formula:

[0015]

[0016] In the formula, represents the first n independent coded image group, Each independent coded image group starts with an I frame, followed by M P frames and B frames.

[0017] The video with base description divided into several independently coded image groups is input into the CoCap network model; the CoCap network model includes an encoder and a decoder, and the processing process is as follows:

[0018] In the encoding stage, the I frame in each GOP is converted into a 768-dimensional feature vector through the I frame encoder, and then output the context semantic features through the 12-layer Transformer encoder; the I frame encoder includes a convolutional layer with a 16x16 convolution kernel, a 16x16 step size, and an output channel number of 768; the P frame or B frame of each GOP is converted into a 192-dimensional feature vector through the motion encoder and output the motion vector through the 2-layer Transformer encoder; the motion encoder includes a convolutional layer with an 8x8 convolution kernel, an 8x8 step size, and an output channel number of 192; at the same time, the P frame or B frame of each GOP is converted into a 768-dimensional feature vector through the residual encoder and output the residual feature through the 2-layer Transformer encoder; the residual encoder includes a convolutional layer with a 64x64 convolution kernel, a 64x64 step size, and an output channel number of 768.

[0019] After the motion feature and the residual feature are processed through dropout, they are combined with the context semantic features of the I frame to form a time sequence feature sequence, which is input into the action encoder; the action encoder first encodes the time sequence feature sequence through position embedding and frame type embedding, and then sequentially performs time sequence modeling through the self-attention module and feature fusion through the cross-attention module, and finally outputs a 512-dimensional fusion feature vector.

[0020] In the decoding stage, the decoder adopts a 2-layer BERT structure, including word embedding, position encoding, and type encoding, and finally generates a video-level coarse-grained pseudo label through linear mapping .

[0021] The loss function of the CoCap network model is shown in the following formula:

[0022] (1)

[0023] In the formula,​ represents a loss function, represents the predicted word sequence length, represents the prediction result of the current t word, represents the real sequence of the first words, represents the probability of predicting the word under the condition of the real sequence of the first words and the video with base description. t

[0024] In step 2, the generation process of high-confidence visual tuples is represented as follows:

[0025] Step 2.1: Sample the video with base description into a sequence at a frame rate T, and input each video frame in the sequence into the high-confidence visual tuple generation branch, i.e., the VIS-MiniCPM model. First, pass through the MiniCPM-V-2 model to generate the initial text description of each video frame, represented as follows:

[0026] (2)

[0027] In the formula, represents fine-grained visual description, represents the text description of the nth video frame generated by MiniCPM-V-2, represents the model parameters of MiniCPM-V-2;

[0028] Step 2.2: Use the LLaMA large language model in the visual event information selector to summarize the fine-grained visual description to generate a summary semantic description for the video segment, represented as follows:

[0029] (3)

[0030] In the formula, represents the summary semantic description of the video segment, represents the LLaMA large language model, is the model parameter of LLaMA;

[0031] Subsequently, merge the video-level fine-grained visual description and the summary semantic description of the corresponding video segment to obtain a sentence set , represented as follows: ​​​

[0032] (4)

[0033] wherein, denotes the connection of and ;

[0034] Step 2.3: input the sentence set to the visual tuple generation module, and obtain the weighted score set of each type of word in the sentence set based on the TextRank algorithm and the word frequency algorithm, and filter out the highest score word from each type of word to form a high-confidence visual tuple.

[0035] Step 2.3 specifically includes the following sub-steps:

[0036] Step 2.3.1: use spaCy to perform dependency parsing on the sentence set to obtain all subject, object, action and environment words in the sentence, and each type of word forms a candidate word set, respectively denoted as , , , ;

[0037] Step 2.3.2: based on the word frequency algorithm, calculate the frequency of each candidate item in the sentence set , denoted as follows:

[0038] (5)

[0039] wherein, denotes the i-th word in the text, ; denotes the frequency function, if then =1, otherwise 0, denotes the word in the sentence set , and m denotes the total number of words in the text;

[0040] At the same time, based on TextRank, the TextRank score of each candidate item in the candidate word set is calculated, denoted as follows:

[0041] (6)

[0042] wherein, denotes the TextRank score of the candidate word , is a damping coefficient, denotes the connection of the candidate word in the sentence set a set of adjacent candidate words with semantic association, representative candidate words and between them as edge weights. for determining whether the candidate words appear simultaneously in the first window, if they appear simultaneously in the first window, the value is 1, otherwise the value is 0.

[0043] Step 2.3.3: Calculate the score of all words in each candidate word set, represented as follows:

[0044] (7)

[0045] In the formula, , denotes the weight coefficient, ;

[0046] Step 2.3.4: Extract high-confidence visual tuples from each candidate word set , where, , , , respectively represent the subject, object, action and environment, represented as follows:

[0047] (8)

[0048] In the formula, denotes the word selected from the candidate word set that maximizes the corresponding score function value.

[0049] The LLaMA large language model includes a byte pair encoding model and a decoder, the decoder is composed of 32 layers of Transformer modules, each Transformer module contains a multi-head self-attention network and a feedforward neural network, and the input of each sub-layer is normalized by the RMSNorm function;

[0050] Step 3 specifically includes the following sub-steps:

[0051] Step 3.1: The byte pair encoding model decomposes the input into subword units, and then maps the subword units to the corresponding Token in the preset vocabulary; then the Token is converted into a word vector to obtain the input matrix X;

[0052] Step 3.2: Input the input matrix X into the decoder for enhancement:

[0053] ​​​During the enhancement process, the multi-head self-attention network in each Transformer module introduces positional information by encoding the query and key through rotation. The multi-head self-attention network also integrates fine-grained visual cues provided by high-confidence visual tuples into the semantic representation of coarse-grained pseudo-labels. After processing by each Transformer module, the output features are passed to the next Transformer module for further feature transformation, until the final Transformer module outputs a feature vector containing both coarse-grained pseudo-labels and fine-grained visual cues. , The size is [1, 4096], and the calculation process is as follows:

[0054] (9)

[0055] (10)

[0056] In the formula, Indicates the first Semantic features extracted by an attention head in a subspace For multi-head self-attention functions, This represents the rotational position encoding applied to the query and key matrices. , and For learnable query, key, and value parameter matrices, Represents matrix multiplication; This indicates that the features from multiple attention heads are concatenated. To output the transformation matrix, This represents a feedforward neural network. This indicates a normalization operation. This represents the output of the 32nd layer Transformer module;

[0057] Feature vectors with coarse-grained pseudo-labels and fine-grained visual cues from the video Vocabulary embedding matrix of the LLaMA large language model Perform a product operation, then normalize using the softmax function to obtain the th... Pace prediction words probability distribution , means as follows:

[0058] (11);

[0059] Step 3.3: Based on the greedy sampling strategy, the first... Predicting words for steps From probability distribution The word with the highest probability is selected, as shown below:

[0060] (12)

[0061] The final enhanced pseudo-label text sequence is generated by iteratively performing single-step prediction , and is represented as follows:

[0062] (13)

[0063] The output probability of the enhanced pseudo-label text sequence is represented as follows:

[0064] (14)

[0065] In the formula, T represents the length of the generated pseudo-label text sequence.

[0066] Step 4: Merge the video data set with base description and the video data set containing the enhanced pseudo-label text sequence as the training data set, denoted as Repeat steps 1-3, and iteratively self-train using the following overall loss function:

[0067] (15)

[0068] In the formula, represents the overall loss function, represents the number of samples in the base description data set, represents the number of samples in the non-base description data set, represents the i-th sample in the base description data set, represents the true label, represents the i-th sample in the non-base description data set, represents the sample d in the non-label data set, represents the semantic enhanced pseudo-label corresponding to the sample in the non-label data set, generically refers to the parameters of the semi-supervised method proposed in the present application, is a hyperparameter used to balance the weight of the labeled data loss and the enhanced pseudo-label data loss in the total loss function, is the number of self-training iterations.

[0069] The beneficial effects of the present application are:

[0070] ​The large language model guided pseudo label enhancement semi-supervised video description generation method of the application, by constructing a double-channel collaborative framework to generate video-level coarse-grained pseudo labels to capture the global semantics of the video, and at the same time generating high-confidence visual tuples to supplement the local visual information of the video, realizing the semantic expression of the video content at different granularities; in the high-confidence visual tuple branch of the double-channel collaborative framework, a VIS-MiniCPM model based on a visual event information selector is constructed. The model extracts high-confidence visual tuples from video frame-level fine-grained descriptions. In this process, the visual event information selector selects visual tuples that meet the video theme and key information, reducing the interference of redundant words in video frame-level fine-grained descriptions; finally, the above high-confidence visual tuples are used as fine-grained visual prompts, and the video-level coarse-grained pseudo labels are input into the LLaMA large language model together, and the enhancement of the video-level coarse-grained pseudo labels is completed under the guidance of the large language model, and the enhanced pseudo labels are obtained. Then, the base description and the enhanced pseudo label are used to generate more fluent and natural video descriptions through self-training in the case of relatively less base description. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is the framework diagram of the large language model guided pseudo label enhancement semi-supervised video description generation method of the application;

[0072] Figure 2 is the training loss decline curve diagram of the large language model guided pseudo label enhancement semi-supervised video description generation method of the application on the MSVD dataset;

[0073] Figure 3 is the training loss decline curve diagram of the large language model guided pseudo label enhancement semi-supervised video description generation method of the application on the MSR-VTT dataset;

[0074] Figure 4 is the visual description diagram of the large language model guided pseudo label enhancement semi-supervised video description generation method of the application on the MSVD dataset;

[0075] Figure 5 is the visual description diagram of the large language model guided pseudo label enhancement semi-supervised video description generation method of the application on the MSR-VTT dataset. DETAILED DESCRIPTION

[0076] The application will be described in detail below in combination with the drawings and specific embodiments.

[0077] The large language model guided pseudo label enhancement semi-supervised video description generation method of the application, as shown in Figure 1 , specifically includes the following steps:

[0078] Step 1: Construct a dual-path collaborative framework, which includes a video-level coarse-grained pseudo-label generation branch and a high-confidence visual tuple generation branch;

[0079] The video-level coarse-grained pseudo-label generation branch includes the CoCap network model, which is used to process videos with base descriptions that are divided into several independent encoded and decoded image groups into video-level coarse-grained pseudo-labels.

[0080] The high-confidence visual tuple generation branch uses the VIS-MiniCPM model, an improvement on the MiniCPM-V-2 model. It adds a visual event information selector to the MiniCPM-V-2 model's output, taking video frames as input and outputting visual tuples. The visual event information selector includes an LLaMA large language model and a visual tuple generation module. The MiniCPM-V-2 model consists of a visual encoder, a compression layer, and a large language model. The visual encoder uses a SigLIPSoViT-400m / 14 and employs adaptive visual coding technology to process high-resolution images with different aspect ratios, slicing them for encoding. The compression layer uses a perceiver resampler structure, compressing 1024 visual tokens per slice into 64. The large language model is based on MiniCPM 2.4B, combining the processed visual tokens with text input to generate fine-grained visual descriptions at the video frame level. The visual event information selector is used to process fine-grained visual descriptions at the video frame level into high-confidence visual tuples. Specifically, the LLaMA large language model is used to process fine-grained visual descriptions at the video frame level into generalized semantic descriptions for video segments. The visual tuple generation module obtains a weighted score set of words in the generalized semantic descriptions of video segments and fine-grained visual descriptions at the video frame level based on the TextRank algorithm and word frequency statistics algorithm. The words with the highest scores in each category are selected to form high-confidence visual tuples.

[0081] Step 2: After preprocessing the video with the base description, input it into the dual-path collaborative framework to generate video-level coarse-grained pseudo-labels and high-confidence visual tuples;

[0082] The process of generating video-level coarse-grained pseudo-tags is as follows:

[0083] Video with base description The consecutive frames are divided into several independently encoded and decoded image groups, as shown below:

[0084]

[0085] In the formula, Indicates the first n A group of images that are independently encoded and decoded. each independently coded image group starts with an I-frame, followed by M P-frames and B-frames;

[0086] a video with base description divided into several independently coded image groups Input the CoCap network model; the CoCap network model is the network model in the paper Shen Y, Gu X, Xu K, et al. Accurate and fast compressed video captioning [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023: 15558-15567. doi: 10.1109 / ICCV51070.2023.01426. The processing process is as follows:

[0087] In the encoding stage, the I-frame in each GOP is converted into a 768-dimensional feature vector by the I-frame encoder, and then passes through the 12-layer Transformer encoder to output the context semantic features; the I-frame encoder includes a convolutional layer with a 16x16 convolution kernel, a 16x16 step size, and an output channel number of 768; the P-frame or B-frame of each GOP is converted into a 192-dimensional feature vector by the motion encoder and passes through the 2-layer Transformer encoder to output the motion features; the motion encoder includes a convolutional layer with an 8x8 convolution kernel, an 8x8 step size, and an output channel number of 192; at the same time, the P-frame or B-frame of each GOP is converted into a 768-dimensional feature vector by the residual encoder and passes through the 2-layer Transformer encoder to output the residual features; the residual encoder includes a convolutional layer with a 64x64 convolution kernel, a 64x64 step size, and an output channel number of 768;

[0088] After the motion features and residual features are processed by dropout, they are combined with the context semantic features of the I-frame to form a time sequence feature sequence, which is input into the action encoder. The action encoder first encodes the time sequence feature sequence through position embedding and frame type embedding, and then sequentially passes the encoded time sequence feature sequence through the self-attention module for time sequence modeling and through the cross-attention module for feature fusion, finally outputting a 512-dimensional fusion feature vector;

[0089] In the decoding stage, the decoder adopts a 2-layer BERT structure, including word embedding, position encoding, and type encoding, and finally generates a video-level coarse-grained pseudo label through linear mapping .

[0090] To prevent overfitting, the dropout rate on the motion feature and residual feature of the CoCap network model is set to 0.2, and the dropout rate on the BERT model is set to 0.1.

[0091] The loss function of the CoCap network model is represented as follows:

[0092] (1)

[0093] wherein, represents the loss function, represents the predicted word sequence length, represents the prediction result of the current t word, represents the real sequence of the first words, represents the probability of predicting the word as under the condition of the given real sequence of the first words t and the video with base description.

[0094] The generation process of the high-confidence visual tuple is represented as follows:

[0095] Step 2.1: The video with base description is sampled into a sequence at a frame rate T, and each video frame in the sequence is input into the high-confidence visual tuple generation branch, i.e., the VIS-MiniCPM model. First, the MiniCPM-V-2 model is used to generate the initial text description of each video frame, represented as follows:

[0096] (2)

[0097] wherein, represents the fine-grained visual description, represents the text description of the nth video frame generated by MiniCPM-V-2, represents the model parameters of MiniCPM-V-2.

[0098] Step 2.2: The LLaMA large language model in the visual event information selector is used to summarize the fine-grained visual description to generate a summary semantic description for the video segment, represented as follows:

[0099] (3)

[0100] wherein, ​a summary semantic description of the video segment, representing a large language model LLaMA, model parameters for LLaMA;

[0101] Subsequently, the video-level fine-grained visual description and the summary semantic description of the corresponding video segment are merged to obtain a sentence set , which represents as follows:

[0102] (4)

[0103] wherein, represents connecting and ;

[0104] Step 2.3: inputting the sentence set into a visual tuple generation module to obtain a weighted score set of each type of word in the sentence set based on a TextRank algorithm and a word frequency algorithm, and filtering out the highest-scored words from each type of word to form a high-confidence visual tuple.

[0105] Since the above process will generate a large number of redundant words and lack of video theme representation, considering that the word frequency algorithm can represent the text theme by measuring the frequency of words and the TextRank can extract the characteristics of keywords by constructing the co-occurrence graph of words, the present application designs a visual tuple generation module based on the word frequency algorithm and the TextRank algorithm to remove the redundant words in the above process and extract the visual tuple. The specific process is as follows:

[0106] Step 2.3.1: performing dependency syntax analysis on the sentence set by means of spaCy to obtain all subject (Subject), object (Object), action (Action) and environment (Environment) words in the sentence, each type of word forming a candidate word set, respectively represented as , , , ;

[0107] Step 2.3.2: based on the word frequency algorithm, calculating the frequency of each candidate word in the candidate word set appearing in the sentence set , represented as follows:

[0108] (5)

[0109] wherein, represents the i-th word in the text, ; denotes the frequency function, if then = 1, otherwise 0, denotes the words in the sentence set denotes the total number of words in the text; m

[0110] Meanwhile, the TextRank score of the candidate item in each candidate word set is calculated based on TextRank, denoted as follows:

[0111] (6)

[0112] In the formula, denotes the TextRank score of the candidate word , is a damping coefficient, denotes the set of adjacent candidate words that have semantic association with the candidate word in the sentence set denotes the edge weight between the candidate word and . is used to determine whether the candidate words and appear simultaneously in the th sliding window, if and appear simultaneously in the th window, then the value is 1, otherwise the value is 0;

[0113] Step 2.3.3: Calculate the score of all words in each candidate word set, denoted as follows:

[0114] (7)

[0115] In the formula, , denotes the weight coefficient, ;

[0116] Step 2.3.4: Extract high-confidence visual tuples from each candidate word set , wherein, , , , respectively represent the subject, object, action and environment, denoted as follows:

[0117] (8)

[0118] In the formula, denotes the word selected from the candidate word set that maximizes the corresponding score function value. ​​​

[0119] Step 3: The high-confidence visual tuple is input into the LLaMA large language model as a fine-grained visual cue together with the video-level coarse-grained pseudo label, and the enhancement of the video-level coarse-grained pseudo label is completed under the guidance of the LLaMA large language model.

[0120] The high-confidence visual tuple obtained in step 2 is input into the LLaMA large language model as a fine-grained visual cue, and the enhancement of the video-level coarse-grained pseudo label is realized under the guidance of the LLaMA large language model. The enhancement of the video-level coarse-grained pseudo label is realized under the guidance of the LLaMA large language model, and the enhanced pseudo label consistent with the video content is generated. Specifically, the following steps are included:

[0121] The LLaMA large language model includes a Byte Pair Encoding (BPE) model and a decoder, and the decoder is composed of 32 layers of Transformer modules, each of which contains a multi-head self-attention network and a feed-forward neural network, and the input of each sub-layer is normalized by an RMSNorm function.

[0122] Step 3.1: The BPE model decomposes the input into subword units, and then maps the subword units to the corresponding Token in the preset vocabulary; then the Token is converted into a word vector to obtain the input matrix X;

[0123] Step 3.2: The input matrix X is input into the decoder for enhancement:

[0124] In the enhancement process, the multi-head self-attention network in each Transformer module introduces position information by rotating position encoding as query and key; through the multi-head self-attention network, the fine-grained visual cue information provided by the high-confidence visual tuple is integrated into the semantic representation of the coarse-grained pseudo label; after each layer of Transformer module processing is completed, the output features are passed to the next layer of Transformer module for further feature transformation, until the last layer of Transformer module outputs the feature vector , with a size of [1, 4096], and the calculation process is represented as follows:

[0125] (9)

[0126] (10)

[0127] In the formula, represents the semantic features extracted by the i-th attention head in the subspace, ​​is a multi-head self-attention function, represents the positional encoding applied to the query and key matrices, , and are learnable query, key, and value parameter matrices, represents matrix multiplication; represents concatenating the features of multiple attention heads, is an output transformation matrix, represents a feed-forward neural network, is a normalization operation, represents the output of the 32nd Transformer module;

[0128] the feature vector with video coarse-grained pseudo-labels and fine-grained visual cues is multiplied by the vocabulary embedding matrix of the LLaMA large language model (dimension ), and then normalized by the softmax function to obtain the probability distribution of the predicted word at step , which is represented as follows:

[0129] (11);

[0130] Step 3.3: Based on the greedy sampling strategy, the predicted word at step is selected from the probability distribution with the highest probability, which is represented as follows:

[0131] (12)

[0132] By iteratively performing single-step prediction, the final enhanced pseudo-label text sequence is generated, which is represented as follows:

[0133] (13)

[0134] The output probability of the enhanced pseudo-label text sequence is represented as follows:

[0135] (14)

[0136] where T represents the length of the generated pseudo-label text sequence.

[0137] Step 4: Through iterative self-training optimization, the final video description is generated.

[0138] the video data set with base descriptions is combined with the enhanced pseudo-label text sequence​ a video data set merge, as a training data set, is expressed as , repeat steps 1~3, continuously optimize model parameters and processing results in the whole process by self-training, reduce errors, improve the accuracy and consistency of the description, until the final video text description is generated.

[0139] The present application uses the following overall loss function for iterative self-training:

[0140] (15)

[0141] In the formula, indicates the overall loss function, indicates the number of samples in the base description data set, indicates the number of samples in the non-base description data set, indicates the i-th sample in the base description data set, indicates the true label, indicates the i-th sample in the non-base description data set, indicates the sample in the non-labeled data set d corresponding to the pseudo label after semantic enhancement, generically refers to the parameters of the semi-supervised method proposed in the present application, is a hyperparameter, used to balance the weight of the labeled data loss and the pseudo label data loss in the total loss function, is the number of self-training iterations. Embodiment 1

[0142] The embodiment provides a semi-supervised video description generation method with large language model guided pseudo label enhancement, which specifically includes the following steps:

[0143] Step 1: build a dual-path collaborative framework, which includes a video-level coarse-grained pseudo label generation branch and a high-confidence visual tuple generation branch;

[0144] Step 2: input the preprocessed video with base description into the dual-path collaborative framework respectively, to generate video-level coarse-grained pseudo label and high-confidence visual tuple;

[0145] Step 3: input the high-confidence visual tuple as a fine-grained visual prompt into the LLaMA large language model together with the video-level coarse-grained pseudo label, and complete the enhancement of the video-level coarse-grained pseudo label under the guidance of the LLaMA large language model;

[0146] Step 4: optimize and generate the final video description through iterative self-training.

[0147] Step 4: optimize and generate the final video description through iterative self-training.

[0148] ​Embodiment 2

[0149] On the basis of Embodiment 1, the video-level coarse-grained pseudo-label generation branch in step 1 includes a CoCap network model, which is used to process the video with base description divided into a plurality of independently coded image groups into a video-level coarse-grained pseudo-label; the high-confidence visual tuple generation branch is a VIS-MiniCPM model, which is a MiniCPM-V-2 model with a visual event information selector added at the output end. The visual event information selector includes an LLaMA large language model and a visual tuple generation module. The MiniCPM-V-2 model is used to process the video with base description divided into continuous frames into a video frame-level fine-grained visual description. The LLaMA large language model of the visual event information selector is used to process the video frame-level fine-grained visual description into a sentence set. The visual tuple generation module obtains a weighted score set of each type of word in the sentence set based on the TextRank algorithm and the word frequency algorithm, and selects the highest scoring word from each type of word to form a high-confidence visual tuple.

[0150] Embodiment 3

[0151] On the basis of Embodiment 2, in step 2, the generation process of the video-level coarse-grained pseudo-label is as follows:

[0152] The video with base description is divided into a plurality of independently coded image groups, which is represented as follows:

[0153]

[0154] In the formula, represents the i-th independently coded image group, n , each independently coded image group takes an I frame as the first frame, followed by M P frames and B frames; The video with base description divided into a plurality of independently coded image groups is input into the CoCap network model; the CoCap network model includes an encoder and a decoder, and the processing process is as follows:

[0155]

[0156] ​​​In the encoding stage, the I frame in each GOP is converted into a 768-dimensional feature vector by an I frame encoder, and then passes through a 12-layer Transformer encoder to output the context semantic features; the I frame encoder includes a convolutional layer with a 16x16 convolution kernel, a 16x16 step size, and an output channel number of 768; the P frame or B frame of each GOP is converted into a 192-dimensional feature vector by a motion encoder and passes through a 2-layer Transformer encoder to output the motion vector; the motion encoder includes a convolutional layer with an 8x8 convolution kernel, an 8x8 step size, and an output channel number of 192; at the same time, the P frame or B frame of each GOP is converted into a 768-dimensional feature vector by a residual encoder and passes through a 2-layer Transformer encoder to output the residual feature; the residual encoder includes a convolutional layer with a 64x64 convolution kernel, a 64x64 step size, and an output channel number of 768;

[0157] The motion feature and the residual feature are respectively subjected to dropout processing, and constitute a time sequence feature sequence with the context semantic feature of the I frame, the time sequence feature sequence is input into the action encoder, the action encoder first encodes the time sequence feature sequence through position embedding and frame type embedding, then sequentially passes the encoded time sequence feature sequence through a self-attention module for time sequence modeling and through a cross-attention module for feature fusion, and finally outputs a 512-dimensional fusion feature vector;

[0158] In the decoding stage, the decoder adopts a 2-layer BERT structure, including word embedding, position encoding and type encoding, and finally generates a video-level coarse-grained pseudo label through linear mapping .

[0159] To prevent overfitting, the dropout rate of the motion feature and the residual feature of the CoCap network model is set to 0.2, and the dropout rate of the BERT model is set to 0.1.

[0160] The loss function of the CoCap network model is represented as follows:

[0161] (1)

[0162] In the formula, represents the loss function, represents the predicted word sequence length, represents the prediction result of the current t word, represents the real sequence of the previous words, represents the prediction of the word under the condition that the real sequence of the previous words and the video with base description are given, represents the prediction of the t word The probability of.

[0163] Example 4

[0164] Based on Example 3, the process of generating high-confidence visual tuples in step 2 is as follows:

[0165] Step 2.1: Transfer the video with base description Sampling is performed according to frame rate T as a sequence. Each video frame in the sequence The high-confidence visual tuple generation branch, namely the VIS-MiniCPM model, is input sequentially. First, it passes through the MiniCPM-V-2 model to generate the initial text description for each video frame, as shown below:

[0166] (2)

[0167] In the formula, This indicates a fine-grained visual description. This refers to the text description of the nth video frame generated by MiniCPM-V-2. This represents the model parameters of MiniCPM-V-2;

[0168] Step 2.2: Utilize the LLaMA large language model in the visual event information selector for fine-grained visual description. In summary, a general semantic description of the video segment is generated, as follows:

[0169] (3)

[0170] In the formula, A general semantic description of a video segment. Represents the LLaMA large language model. These are the model parameters for LLaMA;

[0171] Subsequently, fine-grained visual descriptions at the video level will be implemented. and a general semantic description of the corresponding video clip Merge to obtain a set of sentences , means as follows:

[0172] (4)

[0173] In the formula, Indicates will and connect;

[0174] Step 2.3: Set the sentences The input visual tuple generation module obtains a weighted score set of each type of word in the sentence set based on a TextRank algorithm and a word frequency algorithm, and filters the highest-scored words from each type of word to form a high-confidence visual tuple.

[0175] Step 2.3.1: obtaining all subject, object, action and environment words in the sentence set by means of spaCy , , , ;

[0176] Step 2.3.2: calculating the frequency of each candidate in each candidate word set in the sentence set based on a word frequency algorithm, and representing as follows:

[0177] (5)

[0178] In the formula, represents the i-th word in the text, ; represents a frequency function, if then =1, otherwise 0, represents a word in the sentence set , and m represents the total number of words in the text;

[0179] At the same time, the TextRank score of each candidate in each candidate word set is calculated based on TextRank, and is represented as follows:

[0180] (6)

[0181] In the formula, represents the TextRank score of the candidate word , is a damping coefficient, represents a set of adjacent candidate words that have semantic association with the candidate word in the sentence set , represents the edge weight between the candidate word and ; is used to judge whether the candidate words and appear at the same time in the i-th sliding window, if and appear at the same time in the i-th window, and ​​is 1, otherwise 0;

[0182] Step 2.3.3: Calculate the score of all words in each candidate word set, denoted as follows:

[0183] (7)

[0184] wherein, , denotes the weight coefficient, ; the embodiment sets = 0.5;

[0185] Step 2.3.4: Extract high-confidence visual tuples from each candidate word set wherein, , , , respectively represent the subject, object, action and environment, denoted as follows:

[0186] (8)

[0187] wherein, denotes the word selected from the candidate word set to maximize the corresponding score function value.

[0188] Embodiment 5

[0189] On the basis of Embodiment 4, the LLaMA large language model includes a byte pair encoding model and a decoder, the decoder is composed of 32 layers of Transformer modules, each Transformer module contains a multi-head self-attention network and a feedforward neural network, and the input of each sub-layer is normalized by using the RMSNorm function; the embodiment uses an LLaMA large language model with a parameter amount of 8B.

[0190] Step 3 specifically includes the following sub-steps:

[0191] Step 3.1: The byte pair encoding model decomposes the input into subword units, and then maps the subword units to the corresponding Token in the preset vocabulary; then the Token is converted into a word vector to obtain an input matrix X;

[0192] Step 3.2: Input the input matrix X into the decoder for enhancement:

[0193] During the enhancement process, the multi-head self-attention network in each Transformer module introduces positional information by encoding the query and key through rotation. The multi-head self-attention network also integrates fine-grained visual cues provided by high-confidence visual tuples into the semantic representation of coarse-grained pseudo-labels. After processing by each Transformer module, the output features are passed to the next Transformer module for further feature transformation, until the final Transformer module outputs a feature vector containing both coarse-grained pseudo-labels and fine-grained visual cues. , The size is [1, 4096], and the calculation process is as follows:

[0194] (9)

[0195] (10)

[0196] In the formula, Indicates the first Semantic features extracted by an attention head in a subspace For multi-head self-attention functions, This represents the rotational position encoding applied to the query and key matrices. , and For learnable query, key, and value parameter matrices, Represents matrix multiplication; This indicates that the features from multiple attention heads are concatenated. To output the transformation matrix, This represents a feedforward neural network. This indicates a normalization operation. This represents the output of the 32nd layer Transformer module;

[0197] Feature vectors with coarse-grained pseudo-labels and fine-grained visual cues from the video Vocabulary embedding matrix of the LLaMA large language model Perform a product operation, then normalize using the softmax function to obtain the th... Pace prediction words probability distribution , means as follows:

[0198] (11);

[0199] Step 3.3: Based on the greedy sampling strategy, the first... Predicting words for steps From probability distribution The word with the highest probability is selected, as shown below:

[0200] (12)

[0201] The final enhanced pseudo-label text sequence is generated by iteratively performing single-step prediction , and is represented as follows:

[0202] (13)

[0203] The output probability of the enhanced pseudo-label text sequence is represented as follows:

[0204] (14)

[0205] In the formula, T represents the length of the generated pseudo-label text sequence.

[0206] Embodiment 6

[0207] On the basis of Embodiment 5, in step 4, the video data set with base description is merged with the video data set containing the enhanced pseudo-label text sequence as the training data set, represented as Steps 1-3 are repeated, and the following overall loss function is used for iterative self-training:

[0208] (15)

[0209] In the formula, L represents the overall loss function, N represents the number of samples in the base description data set, M represents the number of samples in the non-base description data set, xi represents the i-th sample in the base description data set, yi represents the true label, xi' represents the i-th sample in the non-base description data set, xi" represents the sample corresponding to the semantic enhanced pseudo-label in the non-label data set, and represents the parameters of the semi-supervised method proposed in the present application. d

[0210] The batch size of the semi-supervised video description generation method proposed in the present embodiment is set to 2. The number of training iterations is set to 20 times, and the learning rate is set to ​​​​​​​​​​​​​​Finally, to avoid overfitting, the regularization rate of the Dropout algorithm is set to 0.2, and the BertAdam optimizer is used for training optimization.

[0211] Simulation experiment

[0212] Two public data sets, MSVD and MSR-VTT, are used for experimental evaluation. Among them, MSVD is derived from YouTube and contains 1970 open field video clips, each with an average of 40 annotations, divided into 1200 training videos, 100 validation videos and 670 test videos. MSR-VTT contains about 10,000 video clips, with an average of 20 annotations per video, divided into 6513 training videos, 497 validation videos and 2990 test videos. In order to evaluate the performance of the present application on different scale data sets, the present application randomly selects 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90% of the subsets from the MSVD and MSR-VTT data sets as labeled data, and the remaining unselected data is used as unlabeled data.

[0213] In order to verify the performance of the present application, the present application uses the standard evaluation index set widely recognized in the field of video description generation, which specifically includes BLEU@4, METEOR, ROUGE-L, and CIDEr and the overall performance index OVERALL.

[0214] The calculation formula (16) of the OVERALL index is shown in the following formula:

[0215] (16)

[0216] In the formula, B-4 represents BLEU@4, C represents CIDEr, M represents METEOR, R represents ROUGE-L, represents the maximum value of a certain index. If the model approaches the maximum value of all model indexes in each index result, the overall score approaches 100. The overall score is 100 only when and only when a model has the most advanced performance on all indexes.

[0217] In addition, the present experiment uses Python 3.8 programming and uses the Pytorch 2.0.0 framework to train in a Linux environment. In the experiment, the GPU is RTX 4090, the memory size is 100GB, the hard disk size is 100GB, and CUDA 11.8 and cuDNN8.6 are used for acceleration.

[0218] Experimental results:

[0219] The present application respectively uses 48000 data and 130260 data in the MSVD and MSR-VTT training set as training data. In order to find the best performance of the model, the learning rate is set to 、 and during the training stage. Taking 80% of the MSVD and MSR-VTT data sets (80% of the data with base description and the remaining 20% of the data without base description) as an example, the experimental results are shown in Figure 2 and Figure 3 . When the learning rate is , the loss of the present application on the two data sets reaches the minimum. This is because the present application uses a dual-path collaborative framework to realize semantic expression of video content at different granularities. At this learning rate, the loss is steadily declining while generating higher quality text. Specifically, as shown in Figure 2 , in the 80% MSVD data set, the model starts to converge smoothly after 20000 iterations; as shown in Figure 3 , in the 80% MSR-VTT data set, the model starts to converge smoothly after 50000 iterations.

[0220] Table 1 shows the comparison of the effect of the large language model guided pseudo label enhanced semi-supervised video description generation method of the present application and the comparative method on the MSVD and MSR-VTT data sets and different fully supervised video description models.

[0221] Table 1

[0222]

[0223] Through the comparison of the data in Table 1, it is found that in the fully supervised state, the present application achieves the best BLEU@4, METEOR and CIDEr indicators on the MSVD data set, and the ROUGE-L indicator is slightly lower than CoCap(ViT / B16). On the MSR-VTT data set, the present application achieves the best METEOR and ROUGE-L indicators, and the BLEU@4 is slightly lower than the performance of the present application on 80% labeled data, and the CIDEr is slightly lower than CoCap(ViT / B16). Although the semi-supervised method of the present application is slightly lower than CoCap(ViT / B16) in some indicators, it still surpasses a variety of fully supervised methods such as S2VT, SAAT, TTA, CSA-SR, GRU-EVE, RecNet, aLSTM, TDDF, PickNet, LEAD, VC-STG and ASGNet.

[0224] In view of the fact that there is no comprehensive evaluation index for text description quality at present, the existing evaluation indexes (such as BLEU@4, METEOR, CIDEr and ROUGE-L) focus on measuring specific aspects of text description. Therefore, the present application uses the overall performance index OVERALL to balance the commonly used indexes for comprehensive evaluation. It can be found through the comparative experiments in Table 1 that the present application achieves the optimal overall performance index OVERALL compared with fully supervised methods such as S2VT, SAAT, TTA, CSA-SR, GRU-EVE, RecNet, aLSTM, TDDF, PickNet, LEAD, VC-STG, CoCap (ViT / B16) and ASGNet on the MSVD and MSR-VTT datasets.

[0225] Although the present application does not achieve the optimal performance of all indexes in the fully supervised scenario, as a semi-supervised learning model, it can still supplement the video fine-grained visual information through the dual-path collaborative framework and generate semantic consistent video descriptions by using the enhancement mechanism, so that the model achieves similar or even better performance than the fully supervised method under semi-supervised conditions. For example, as shown in Table 1, the present application achieves a CIDEr of 108.5 on the MSVD dataset using only 80% of the labeled data, which exceeds a variety of fully supervised methods. In addition, the present application still outperforms a variety of fully supervised methods in terms of the overall performance index OVERALL by using only 80% or even 50% of the labeled data for training. This shows that even in the case of limited base descriptions, the present application can still generate high-quality video descriptions and effectively alleviate the performance bottleneck caused by insufficient base descriptions, and alleviate the high cost problem of manual labeling of base descriptions.

[0226] Table 2 shows the effect comparison of the semi-supervised video description generation method of the present application and the comparative method guided by a large language model and pseudo label enhancement on different semi-supervised video description models on the MSVD and MSR-VTT datasets.

[0227] Table 2

[0228]

[0229] Through the comparative experiments of Table 2, it can be found that on the MSVD and MSR-VTT datasets, the present application and the semi-supervised method LCABM exhibit better performance on the METEOR, ROUGE-L, CIDEr and overall score indicators OVERALL under different base description annotation ratios (30%, 50% and 80%). Compared with LCABM, the present application adopts a dual-path collaborative framework, can fully fuse rich visual information in the video frame when generating coarse-grained descriptions, and further optimizes the visual description through the enhancement module. At the same time, the visual selector effectively reduces the redundant vocabulary of the video frame-level fine-grained description, enhances the ability to capture video details, and makes the generated description more accurately reflect the fine-grained information in the video.

[0230] It should be noted that although the present application is slightly lower than LCABM in the BLEU@4 indicator, this is because BLEU@4 mainly based on n -gram matching, focusing on the accurate overlap between the description and the reference text, while the present application introduces a large language model in the dual-path collaborative framework and the enhancement process. The text generated is more flexible in vocabulary and may contain synonymous replacements, resulting in a decrease in the degree of n -gram matching with the reference description, which does not affect its ability to accurately capture semantic information of the video.

[0231] Finally, the proposed present application is used to generate text descriptions on the MSVD and MSR-VTT test sets. As Figure 4 and Figure 5 experimental results show that this method can fully fuse rich visual information in the video frame when generating coarse-grained descriptions, effectively alleviating the problem of semantic deviation in video description.

[0232] As Figure 4 and Figure 5 show, in order to verify the influence of the dual-path collaborative framework and the visual selector on the performance of the model, the present application conducts experiments under the condition of 80% base description ratio. In the experiment, single-path learning mechanism, dual-path collaborative learning strategy using MiniCPM-V-2 model and dual-path collaborative learning strategy using VIS-MiniCPM model are set respectively to highlight the influence of the dual-path collaborative framework and the VIS-MiniCPM model on the performance of the model. From Figure 4As can be seen, when the single-path learning mechanism is selected, the output descriptions suffer from semantic bias, such as generating "riding" and "horse". When the dual-path learning mechanism using the MiniCPM-V-2 model is selected, the model can basically output complete video descriptions. However, when the dual-path collaborative learning strategy of the MiniCPM-V-2 model is selected, redundant words in the video frame descriptions introduce semantic noise into the model, resulting in semantic bias in the generated sentences, such as obtaining descriptions like "in a park" that do not conform to the video content. On the other hand, when the dual-path collaborative learning strategy of the VIS-MiniCPM model is selected, it can help the model generate more accurate descriptions, such as generating "dogs" and "in a field".

[0233] Depend on Figure 5 It is evident that using a single-path learning mechanism cannot improve the quality of generated sentences. For example, the model may predict incorrect descriptive sentences, such as "A man is riding on the water." When a dual-path collaborative learning strategy using the MiniCPM-V-2 model is selected, semantic noise is introduced, leading to a decrease in the quality of generated descriptions, such as generating "watching" and "child." When a dual-path collaborative learning strategy using the VIS-MiniCPM model is selected, the accuracy of the generated descriptive sentences is significantly improved.

[0234] Ultimately, by Figure 4 and Figure 5 It is evident that when the learning strategy of using the VIS-MiniCPM model in a dual-path collaborative manner is selected, this invention can alleviate the semantic bias problem in video description.

Claims

1. A semi-supervised video description generation method guided by a large language model and augmented with pseudo-labels, characterized in that, Specifically, the steps include the following: Step 1: Construct a dual-path collaborative framework, which includes a video-level coarse-grained pseudo-label generation branch and a high-confidence visual tuple generation branch; Step 2: After preprocessing the video with the base description, input it into the dual-path collaborative framework to generate video-level coarse-grained pseudo-labels and high-confidence visual tuples; Step 3: Input the high-confidence visual tuples as fine-grained visual cues, along with the video-level coarse-grained pseudo-labels, into the LLaMA large language model. Under the guidance of the LLaMA large language model, the video-level coarse-grained pseudo-labels are enhanced. Step 4: Optimize and generate the final video description through iterative self-training.

2. The semi-supervised video description generation method with pseudo-label enhancement guided by a large language model according to claim 1, characterized in that, The video-level coarse-grained pseudo-tag generation branch in step 1 includes the CoCap network model, which is used to process a video with a base description that is divided into several independent encoding and decoding image groups into video-level coarse-grained pseudo-tags. The high-confidence visual tuple generation branch is the VIS-MiniCPM model. The VIS-MiniCPM model adds a visual event information selector to the output of the MiniCPM-V-2 model. The visual event information selector includes an LLaMA large language model and a visual tuple generation module. The MiniCPM-V-2 model is used to process video divided into continuous frames with base descriptions into fine-grained visual descriptions at the video frame level. The LLaMA large language model of the visual event information selector is used to process the fine-grained visual descriptions at the video frame level into a set of sentences. The visual tuple generation module obtains a weighted score set of words of each category in the sentence set based on the TextRank algorithm and word frequency statistics algorithm, and selects the words with the highest scores from each category to form a high-confidence visual tuple.

3. The semi-supervised video description generation method with large language model-guided pseudo-label enhancement according to claim 2, characterized in that, In step 2, the process of generating the video-level coarse-grained pseudo-tags is as follows: Video with base description The consecutive frames are divided into several independently encoded and decoded image groups, as shown below: In the formula, Indicates the first n A group of images that are independently encoded and decoded. Each independently encoded and decoded image group starts with an I-frame, followed by P-frames and B-frames, for a total of M frames. Video with base description, divided into several independently encoded and decoded image groups. Input a CoCap network model; the CoCap network model includes an encoder and a decoder, and the processing procedure is as follows: During the encoding phase, each GOP's I-frame is converted into a 768-dimensional feature vector by an I-frame encoder, and then passed through 12 Transformer encoder layers to output contextual semantic features. The I-frame encoder includes a 16×16 convolutional kernel, a 16×16 stride, and a convolutional layer with 768 output channels. Each GOP's P-frame or B-frame is converted into a 192-dimensional feature vector by a motion encoder and then passed through 2 Transformer encoder layers to output motion vectors. The motion encoder includes an 8×8 convolutional kernel, an 8×8 stride, and a convolutional layer with 192 output channels. Simultaneously, each GOP's P-frame or B-frame is converted into a 768-dimensional feature vector by a residual encoder and then passed through 2 Transformer encoder layers to output residual features. The residual encoder includes a 64×64 convolutional kernel, a 64×64 stride, and a convolutional layer with 768 output channels. After the motion features and residual features are processed by dropout, they are combined with the contextual semantic features of the I-frame to form a temporal feature sequence. The temporal feature sequence is then input into the action encoder. The action encoder first encodes the temporal feature sequence through position embedding and frame type embedding. Then, the encoded temporal feature sequence is sequentially passed through the self-attention module for temporal modeling and through the cross-attention module for feature fusion, and finally outputs a 512-dimensional fused feature vector. During the decoding stage, the decoder employs a two-layer BERT structure, including word embedding, positional encoding, and type encoding, and ultimately generates video-level coarse-grained pseudo-tags through linear mapping. .

4. The semi-supervised video description generation method with pseudo-label enhancement guided by a large language model according to claim 3, characterized in that, The loss function of the CoCap network model is expressed as follows: （1） In the formula, Represents the loss function. Indicates the length of the predicted word sequence. Indicates the current number t The prediction results for each word Indicates the preceding The true sequence of words, Indicates before The true sequence of words and videos with base descriptions Under the conditions, the first t The word prediction is as follows The probability of.

5. The semi-supervised video description generation method with pseudo-label enhancement guided by a large language model according to claim 2, characterized in that, In step 2, the process of generating the high-confidence visual tuple is as follows: Step 2.1: Transfer the video with base description Sampling is performed according to frame rate T as a sequence. Each video frame in the sequence The high-confidence visual tuple generation branch, namely the VIS-MiniCPM model, is input sequentially. First, it passes through the MiniCPM-V-2 model to generate the initial text description for each video frame, as shown below: （2） In the formula, This indicates a fine-grained visual description. This refers to the text description of the nth video frame generated by MiniCPM-V-2. This represents the model parameters of MiniCPM-V-2; Step 2.2: Utilize the LLaMA large language model in the visual event information selector for fine-grained visual description. In summary, a general semantic description of the video segment is generated, as follows: （3） In the formula, A general semantic description of a video segment. Represents the LLaMA large language model. These are the model parameters for LLaMA; Subsequently, fine-grained visual descriptions at the video level will be used. and a general semantic description of the corresponding video clip Merge to obtain a set of sentences , means as follows: （4） In the formula, Indicates will and connect; Step 2.3: Set the sentences The input visual tuple generation module obtains a weighted score set of words of each category in the sentence set based on the TextRank algorithm and word frequency statistics algorithm. It then selects the words with the highest scores from each category to form a high-confidence visual tuple.

6. The semi-supervised video description generation method with pseudo-label enhancement guided by a large language model according to claim 5, characterized in that, Step 2.3 specifically includes the following sub-steps: Step 2.3.1: Use spaCy to process the sentence set Dependency parsing yields all subject, object, action, and context words in the sentence. Each category forms a candidate word set, represented as follows: , , , ; Step 2.3.2: Based on the word frequency algorithm, calculate the candidate words in each candidate word set. In the sentence collection The frequency of occurrence in the text is represented as follows: （5） In the formula, Indicates the first in the text i One word, ; Represents the frequency function, if but =1, otherwise 0 Represents a set of sentences The words in m Indicates the total number of words in the text; Meanwhile, the TextRank scores of the candidates in each candidate word set are calculated based on TextRank, and are represented as follows: （6） In the formula, Indicate candidate words TextRank score, The damping coefficient is... Indicates with candidate words In the sentence collection A set of adjacent candidate words with semantic relationships. Indicate candidate words and Edge weights between them; Used to determine in the first Candidate words in a sliding window and Do they appear simultaneously? If in the first... Simultaneous appearance of multiple windows and The value is 1 if the value is 1, otherwise the value is 0. Step 2.3.3: Calculate the score of all words in each candidate word set, as shown below: （7） In the formula, , Indicates the weighting coefficient. ; Step 2.3.4: Extract high-confidence visual tuples from each candidate word set. ,in, , , , These represent the subject, object, action, and environment, respectively, as follows: （8） In the formula, This indicates selecting the word from the candidate word set that maximizes the corresponding score function value.

7. The semi-supervised video description generation method with pseudo-label enhancement guided by a large language model according to claim 1, characterized in that, The LLaMA large language model includes a byte pair encoding model and a decoder. The decoder consists of 32 Transformer modules. Each Transformer module contains a multi-head self-attention network and a feedforward neural network. The input of each sub-layer is normalized using the RMSNorm function. Step 3 specifically includes the following sub-steps: Step 3.1: The byte-pair encoding model will input... It is broken down into sub-word units, and then the sub-word units are mapped to the corresponding tokens in the preset vocabulary. The tokens are then converted into word vectors to obtain the input matrix X; Step 3.2: Enhance the input matrix X input decoder: During the enhancement process, the multi-head self-attention network in each Transformer module introduces positional information by encoding the rotation position as query and key; A multi-head self-attention network is used to integrate fine-grained visual cues provided by high-confidence visual tuples into the semantic representation of coarse-grained pseudo-labels. After each Transformer module completes its processing, the output features are passed to the next Transformer module for further feature transformation, until the final Transformer module outputs a feature vector containing both coarse-grained pseudo-labels and fine-grained visual cues. , The size is [1, 4096], and the calculation process is as follows: （9） （10） In the formula, Indicates the first Semantic features extracted by an attention head in a subspace For multi-head self-attention functions, This represents the rotational position encoding applied to the query and key matrices. , and For learnable query, key, and value parameter matrices, Represents matrix multiplication; This indicates that the features from multiple attention heads are concatenated. To output the transformation matrix, This represents a feedforward neural network. This indicates a normalization operation. This represents the output of the 32nd layer Transformer module; Feature vectors with coarse-grained pseudo-labels and fine-grained visual cues from the video Vocabulary embedding matrix of the LLaMA large language model Perform a product operation, then normalize using the softmax function to obtain the th... Pace prediction words probability distribution , means as follows: （11）； Step 3.3: Based on the greedy sampling strategy, the first... Predicting words for steps From probability distribution The word with the highest probability is selected, as shown below: （12） By iteratively performing single-step predictions, the final enhanced pseudo-label text sequence is generated. , means as follows: （13） The output probability of the enhanced pseudo-labeled text sequence is represented as follows: （14） In the formula, T represents the length of the generated pseudo-label text sequence.

8. The semi-supervised video description generation method with pseudo-label enhancement guided by a large language model according to claim 7, characterized in that, Step 4, combine the video data set with the base description. With the enhanced pseudo-label text sequence video data set Merging them into a training dataset, represented as Repeat steps 1 through 3, using the following overall loss function for iterative self-training: （15） In the formula, Represents the overall loss function. This indicates the number of samples in a dataset with a basis description. This indicates the number of samples in the dataset that describes the data without a basis. This indicates that the data in the dataset has a basis description. One sample, Indicates the true label, This indicates the first term in the dataset that is described without basis. d One sample, Represents samples in an unlabeled dataset Corresponding to the semantically enhanced pseudo-tags, The parameters represent the semi-supervised method. These are hyperparameters used to balance the weights of labeled data loss and pseudo-labeled data loss in the total loss function. This represents the number of self-training iterations.

Images