A multi-modal description method based on a pre-trained large language model

Abstract

The application belongs to the technical field of multimedia information content processing and video understanding, and discloses a multimodal description method based on a pre-trained large language model. Video features are extracted from a video and mapped into video embeddings, and audio features are extracted from audio and converted into audio text; entity information is extracted from the video and the audio text and converted into entity text; a prompt template is constructed, and the audio text and the entity text are added; the prompt template, vector tokens, video embeddings and a prompt question are input into a large language model to obtain a video description; the vector tokens are obtained through a feedback model output of the video description. Through the deep learning capability of the large language model, the application can recognize and describe named entities such as characters, locations and buildings in the video. The accuracy and relevance of the video description are improved, and the user's understanding and search ability of the video content are enhanced; the application is suitable for various video contents, including movies, television programs and user-generated videos.

Description

Technical Field

[0001] This invention relates to the fields of multimedia information content processing and video understanding technology, and in particular to a multimodal description method based on a pre-trained large-scale language model. Background Technology

[0002] Video captioning refers to the automated generation of textual descriptions for videos, describing the events or scenes that occur within them. This field intersects with multimedia processing, computer vision, and natural language processing, aiming to enable computer systems to understand and generate natural language descriptions of video content.

[0003] Early video descriptions used template-based methods. These methods primarily extracted semantic concepts, such as subject, object, and verbs, from a set of visual classifiers and then filled in placeholders or variables in a template to generate a description. Template-based video description methods are suitable for simple and structured scenarios, but they have limitations when dealing with complex, diverse, and real-time changing video content. In recent years, with the development of deep learning technology, deep learning-based methods have gradually become the mainstream for handling video description tasks, capable of more flexibly learning complex feature representations and semantic information.

[0004] With the development of video captioning, a single video caption can no longer fully describe the information in a video, because a video may contain one or more segments (moments) that can be divided. These segments have low semantic relevance and can form a sentence on their own, which has led to the development of dense video captioning.

[0005] With the rise of large-scale language models and the widespread application of prompt learning on large language models, powerful large-scale language models have made zero-shot dense video captioning possible. The paper "Zero-Shot Dense Video Captioning by Jointly Optimizing Text and Moment. Jo Yongrae.alArxiv. 2023" proposed the ZeroTA model, which achieves zero-shot learning for dense video captioning. However, this type of technique has certain limitations; it cannot completely and finely describe image semantics and lacks supplementary background knowledge. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a multimodal description method based on a pre-trained large-scale language model, which can describe multimodal video descriptions of background knowledge such as public figures, company logos, and locations.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A multimodal description method based on a pre-trained large-scale language model includes the following steps; Step 1: Extract video features from the video and map them to video embeddings; extract audio features from the audio and convert them into audio text. Step 2: Extract entity information from video and audio text and convert it into entity text; Step 3: Build the prompt template, adding audio text and physical text; Step 4: Input the prompt template, vector tokens, video embedding, and prompt question into the large language model to obtain the video description; the vector tokens are obtained by outputting the video description through the feedback model.

[0008] After the video is frame-stripped, a discrete video frame sequence is obtained. Image features are extracted from the video frame sequence using the image encoder CLIP. The last dimension of the image features is normalized. The dot product of the normalized image features is inverted to form a similarity matrix. Image features are filtered using a similarity matrix; a similarity filtering threshold is defined; the image features are divided into multiple groups with continuous image features, and the similarity of the image features within each group is greater than the similarity filtering threshold; the similarity filtering similarity between the last image feature of the first group and the first image feature of the second group in adjacent groups is less than the threshold; one image feature is taken from each group to form the filtered video features; The audio was transcribed into audio text using the Whisper speech recognition model.

[0009] The video features are hidden states composed of vectors. The video features are then mapped to video embeddings through a mapping layer. The mapping layer consists of simple fully connected layers, and the entire process is represented by the following formula: Where X represents the processed video features, W represents the weights of the mapping layer, and T represents the video embedding.

[0010] The second step is as follows: First, representative video frames from the discrete video frame sequence are input into the object detection API to obtain entity information contained in the video. Second, for entity information in the audio text, a named entity recognition method is used to obtain entity information based on the BERT deep learning model. Finally, the entity information obtained from the video frames and the entity information obtained from the audio text are concatenated to form entity text. For audio text with an empty text sequence, the audio entity information is set to empty and is not concatenated with the entity information list E obtained from the video frames; for audio text that is not empty, entities are extracted, and the obtained entities are concatenated with list E to form a new entity list. The entities extracted from the audio are to .

[0011] The third step specifically involves: The prompt template includes: Basic task setting: Assign a large model to act as a video description assistant, answering human questions; Role setting: Designate a large model to play the role of an assistant, and designate a human to play the role of a user; Spacing settings: Set a separator word between the user's question and the assistant's answer; Entity information setting: Specify the entity text obtained in step two; Audio text settings: Specify the audio text extracted in the first step; Build prompt templates include: The template above is set up by concatenating the basic task, audio text, entity information, user questions, intervals, and assistant answers in that order, and then converting it into text format.

[0012] The fourth step is specifically as follows: The prompt template, vector tokens, video embedding, and prompt question are concatenated and then input into the large language model. Among them, the vector tokens are a vector generated by the feedback model in response to the video description, which is used to guide the output of the large language model; The prompt question asks a large language model to describe the input video embedding.

[0013] The feedback model includes an initial module and an iterative module. During the first video description, the initial module takes the video embedding obtained in the first step as input and outputs vector tokens. During the iterative video description process, the iterative module takes the video description of the previous generation as input, as well as the entity information and audio text obtained in the second step, and outputs vector tokens.

[0014] The iterative module outputs the following vector tokens: entity information and audio text are processed by a text encoder to obtain vectors. The video description is processed by a text encoder to obtain a vector. For vectors and To achieve common attention and ,Will and splicing ; Next, the features with dimensions of 1×20×4096 are reshaped into vectors with dimensions of 1×81920, and then passed through a connection layer to form a vector with dimensions of 1×15360. This is then reshaped into a vector V with dimensions of 20×768. Then, the vectors V and V are further processed... Perform common attention to obtain vectors and splicing and Then reshape it into a vector of tokens with dimensions of 1×20×4096.

[0015] The beneficial effects of this invention are as follows: This invention relates to a multimodal description method based on a pre-trained large-scale language model, aiming to improve the accuracy and relevance of video descriptions through the large-scale language model. Existing video description systems often lack the ability to identify and describe specific named entities appearing in videos, resulting in incomplete descriptions or a lack of background knowledge. This invention, by combining the deep learning capabilities of a large-scale language model, provides a system and method capable of identifying and describing named entities such as people, locations, and buildings in videos. This improves the accuracy and relevance of video descriptions, especially when named entities are present. It enhances users' understanding and search capabilities for video content; it is applicable to various video content, including movies, television programs, and user-generated videos. Attached Figure Description

[0016] Figure 1 This is a video description flowchart of the present invention; Figure 2 This is a flowchart of a multimodal description method based on a pre-trained large-scale language model according to the present invention; Figure 3 This is a structural diagram of the mapping layer of the present invention; Figure 4 A schematic diagram for generating vector tokens.

[0017] Figure 5 This is a structural diagram of the iterative module in the feedback model. Detailed Implementation

[0018] The technical methods mentioned in the above steps are described in detail below: like Figure 1 As shown, the video description of this invention is mainly divided into four steps: Step 1: Extract video features from the video and map them to video embeddings; extract audio features from the audio and convert them into audio text. Step 2: Extract entity information from video and audio text and convert it into entity text; Step 3: Build the prompt template and add audio text and physical text; Step 4: Input the prompt template, vector tokens, video embedding, and prompt question into the large language model to obtain the video description; the vector tokens are obtained by outputting the video description through the feedback model.

[0019] The second step includes: First, the discrete video frame sequence obtained in the first step is used. Representative video frames are input into the object detection API provided by Baidu to obtain entity information contained in the video, which is then converted into text and saved. Second, for entity information in the audio text, named entity recognition is used, based on the BERT deep learning model to obtain entity information. The entity information obtained from the video frames and the entity information obtained from the audio text are then concatenated to form entity text.

[0020] The prompt template in the third step refers to setting a specific template for the large language model, clarifying the role the model will play (e.g., assistant, expert, storyteller), describing the specific tasks expected of the large language model, specifying any limitations or scope of the large language model, and defining the posture and tone the large language model should exhibit when answering. The large language model is set to act as a question-and-answer assistant, answering questions posed by humans and completing the tasks specified in the video description within the template.

[0021] The templates for the specified video description task mentioned above include: Basic task setting: Assign a large model to act as a helpful, respectful, honest and trustworthy video description assistant, answering human questions.

[0022] Role setting: Designate a large model to play the role of an assistant, and designate a human to play the role of a user; Spacing settings: Set a separator word between the user's question and the assistant's answer; Other information settings: Concatenate the entity text obtained in step 2 and the audio text obtained in step 1 to form a text, and add it.

[0023] In the fourth step, the vector tokens are a segment of vectors generated by the feedback model to guide the output of the large language model. Unlike the discrete word embeddings in the large language model's tokenizer, where each word embedding represents a specific word in the vocabulary, these vectors are continuous, representing a continuous token space. This means that the vector tokens can be smoothly interpolated within the token space, thereby generating an infinite number of new and previously unseen expressions to guide the large language model.

[0024] Furthermore, when using a large language model, a cue is a set of text or instructions provided to the model to guide it in generating specific outputs. In the video description task, the cue is set to ask the large language model to describe the input video embedding.

[0025] The above prompts include: Provide clues based on characters, locations, and background: 1) Characters: Describe the main characters appearing in the video, including their physical features, expressions, actions, and roles.

[0026] 2) Location: Indicate the location where the video was filmed, such as city, countryside, indoors or outdoors. Describe the characteristics of the location, such as architectural style, natural landscape or specific landmarks.

[0027] Background: Describes background information about the video, such as events, themes, or plot.

[0028] A specific implementation method is provided for the above specific steps.

[0029] The first step includes: The first step is video feature processing. When processing video content, the `cv.VideoCapture` function from the OpenCV library is used to read the video from the video file to obtain a video object. Next, the `cv2.CAP_PROP_FPS` property is called to obtain the video's frame rate, thus determining the number of frames per second. Then, each frame in the video is sampled, and the sampling step size is determined by dividing the number of frames per second by 3. Based on this step size, the video is sampled, and the sampled frames are converted from BGR format to RGB format for subsequent processing.

[0030] After the conversion is complete, the converted frames are converted from OpenCV format to PIL image format, and all sampled PIL images are compiled into a list. Next, this list is processed by the preprocessor provided by the CLIP model, converting it into tensor vectors. Its dimensions are n×3×224×224, where n is the total number of frames after sampling.

[0031] During the conversion process, we utilize the CLIP image encoder to transform tensor vectors into image features with dimensions n×512. To improve processing accuracy, we normalize the last dimension of the image features. Normalized image features better reflect the similarity between different frames. Then, we invert the dot product of the normalized image features to construct a similarity matrix.

[0032] Next, image features are filtered. First, a similarity filtering threshold is defined, with a default setting of 0.85. List 1 is initialized to store groups of similar frames after filtering. List 2 is initialized to temporarily store the currently processed group of similar frames. The rows of the similarity matrix are traversed, i.e., each frame is traversed. If list 2 is empty, the index of the current frame is added to list 2. If the current frame is the second to last frame and is recorded in list 2, list 2 is added to list 1. If the similarity between the current frame and the next frame is greater than the threshold, the index of the next frame is added to list 2; otherwise, list 2 is added to list 1, and list 2 is cleared to start a new grouping. List 1 is traversed, and the feature vector of the first frame of each group is added to the final returned list, along with the corresponding index. The feature vectors in the final returned list are stacked into a tensor using `torch.stack`. The filtered feature vectors are returned. and the corresponding index list , where j is the number of video frames corresponding to the filtered feature vector.

[0033] Next is audio feature processing.

[0034] The Whisper model is used to process the input video, parsing the audio and then transcribing it into text. Next, preliminary syntactic and semantic analysis is performed on the transcribed text to remove obvious irrelevant information, such as music and background noise, ensuring that the extracted text content focuses on the spoken words. Audio without specific semantic meaning is left blank. After this processing, an appendix is ​​obtained. Figure 1 The audio text is either a text sequence or null.

[0035] As shown in step two: Video entity extraction: During video content analysis, as described in the previous step, video frames are first read from the video file, and image processing techniques are used to extract the feature vector F for each frame. Simultaneously, to ensure accurate positioning of each frame, its corresponding index I in the sampled frame sequence is recorded.

[0036] Next, we iterate through these indices I and use Baidu's object detection API to process the video frames corresponding to each index. Through the Baidu API, we accurately identify entities in the video frames and convert these entities into text. These entity texts include people, locations, buildings, etc., appearing in the video; they are important components of the video content and crucial for generating accurate descriptions.

[0037] Next, the entity text generated from all frames is concatenated into a list, i.e. , where i represents the total number of entities extracted from the video for further processing and analysis. During this process, duplicate entity text is removed from the list to ensure that the final entity information is unique and that repetition does not affect the accuracy of the description.

[0038] After the above steps, entity information was successfully extracted from the video.

[0039] Audio entity extraction: In the first step of audio extraction, audio text is obtained. For audio text with an empty text sequence, the entity information is set to empty and is not concatenated with the video entity information list E; for audio text that is not empty, entity extraction is performed, and the obtained entities are concatenated with list E to form a new entity list. The entities extracted from the audio are to .

[0040] In the process of entity extraction from text sequences, a finely tuned BERT model was used. This model is based on a pre-trained BERT model with parameter version bert-base-chinese. The BERT model consists of a 12-layer Transformer encoder.

[0041] Let's take the text "A child is playing with building blocks" as an example to explain the entity extraction process in detail. In the BERT tokenizer, this text sequence has 10 text elements. Adding the identifiers [CLS], [SEP], and the pre-labeled entity tokens "child" and "building blocks," the total sequence length is 18. First, this 18-length sequence is encoded and embedded, then padded to a sequence of length 100 to facilitate processing by the BERT model. Since the BERT vocabulary size is 21128, the dimension of the token encoding embedding is 18 × 21128. The dimension of the segment embedding is the same as the encoding embedding, also 18 × 21128.

[0042] [CLS], the tokens of the audio text, and the segmentation embedding of the first [SEP] are marked as A, while the segmentation embeddings of [entity], [substance], [have], and the second [SEP] are marked as B. The word types of the Bert model are divided into two categories, 0 and 1. The dimension of the token with segmentation embedding A is set to 1×21128, and the value of each element is 0. Similarly, the dimension of the token with segmentation embedding B is set to 1×21128, and the value of each element is 1. The dimension of the position embedding is 1×21128. The position embedding of [CLS] is [1, 0, …, 0], the position embedding of '儿' is [0, 1, 0, …, 0], the position embedding of '童' is [0, 0, 1, 0, …, 0], and the position embeddings of other tokens are analogized according to their positions. Finally, the token embedding, segmentation embedding, and position embedding are added together to obtain the input of the Bert model.

[0043] Next, the Bert model is fine-tuned on a private dataset collected for this video description task. The data is in the form of: {"text", {entity list}}. During training, 15% of the words in the entity list are randomly transformed. Among the transformed words, 85% of the words are replaced with the special token [MASK], and 15% of the words are randomly replaced with another word. The Adam optimizer is used with a learning rate of 0.0001, and the cross-entropy loss function is used to optimize the model parameters.

[0044] Among them, L represents the cross-entropy loss; N represents the number of masked word tokens; C is the vocabulary size; is an indicator function. If the true category of the word token is c, then , otherwise it is 0; is the probability that the model predicts that the word token is equal to c.

[0045] Construction of the text: In the extraction of entity information from videos and audio, the corresponding entity information texts are obtained, which are mainly divided into two categories. One category is that the audio text extracted from the audio is empty, and the other category is non-empty. Its specific composition can be shown in Table 1.

[0046] Table 1 Types of texts

[0047] In the case where the audio text is empty, the text is only composed of video entities; in the case where the audio text is not empty, the text is composed of the audio text, the video entity list, and the audio entity list.

[0048] As shown in the third step: Different prompt templates are constructed based on different large-scale language models. The following examples illustrate the template construction for the llava-v1, llava-llama-2, and llama-v2 large-scale language models: Llava-v1: Basic task setting: A curious human chats with an AI assistant. The AI ​​assistant provides helpful, detailed, and polite answers to the human's questions; Roles: User, Assistant; Other information settings: constructed from the text obtained in step two; Segmentation words: “”, “”; llava-llama-2: Basic task setup: You are a powerful assistant in terms of language and vision. You can understand the visual content provided by the user and use natural language to assist the user in completing various tasks; Roles: User, Assistant; Other information settings: constructed from the text obtained in step two; Segmentation setting: <s>"、”< / s> ”; llama-v2: Basic task setting: You are a helpful, respectful, and honest assistant. Answer questions as best you can, while ensuring your safety. Please ensure your answers are free of social bias and have positive implications. If a question is meaningless or inaccurate, explain why; do not answer incorrect questions. If you do not know the answer, please do not share false information. Roles: User, Assistant; Other information settings: constructed from the text obtained in step two; Segmentation setting: <s>"、”< / s> ”; The process of constructing the other information mentioned above: First, the prompt template for the large language model is set as an object. Then, during the creation of this object, the text obtained in step two is saved as a string into the template object's properties. Other roles, basic tasks, and segmentation word settings are pre-saved in their respective configuration files and are automatically loaded during object initialization.

[0049] As shown in step four: The prompt template, vector tokens, video embedding, and prompt question are input into a large language model to obtain a video description. In this step, the prompt template, vector tokens, video embedding, and prompt question need to be processed to transform them into input acceptable to the large language model, outputting the video description.

[0050] In the first step described above, video embedding is achieved by mapping the extracted feature vector F from the video using a mapping layer. The structure of the mapping layer is as follows: Figure 3 As shown. The input dimension of this mapping layer is 512, and the output dimension is 20×4096. Here, the input dimension is the dimension of the hidden states of the CLIP encoder, the output dimension of 4096 refers to the dimension of the hidden states of the large language model, and 20 specifies that the video tokens are of a pre-defined length of 20. This can be expressed by the following formula: Where H is the mapped tensor with a size of 1×20×4096, F is the video feature vector with a dimension of j×512, Projection represents the mapping layer, and Pre represents the input that the mapping layer can accept after processing F.

[0051] Pre represents the preprocessing of the feature vector F. Specifically, the process is as follows: First, the video feature vector with dimension j×512 is augmented with a new dimension, becoming 1×j×512. Next, the mean of the second dimension j is calculated to construct a new tensor with dimension 1×512.

[0052] The processed video feature vectors are passed through a mapping layer to obtain a tensor with a dimension of 1×(20×4096). This tensor is then reshaped into a new tensor with a dimension of 1×20×4096, and finally formed into a tensor form that is input into a large language model.

[0053] The prompt questions are drawn from a pre-defined list of texts. One text is randomly selected as the prompt question during each video description generation process. Seven examples of prompt questions are given below, as shown in Table 2.

[0054] Table 2 List of Problems Prompt

[0055] Vector tokens are generated by a feedback model. The feedback model consists of an initialization module and an iteration module.

[0056] like Figure 4 As shown, the initial module of the feedback model encodes the initial video embedding and outputs a vector of tokens. The components of this module are as follows: First, the video embedding with dimensions of 1×20×4096 is flattened into a vector with dimensions of 1×81,920; Next, the vector with a dimension of 1×81,920 is input into a linear layer with an input of 81920 and an output of 1024; The vector after the fully connected layer has a dimension of 1×1024. This vector is then input into a dropout layer with a ratio of 0.5. The dropped-out vector is then input into a linear layer with an input of 1027 and an output of 81920, resulting in a vector with a dimension of 1×81,920. Finally, the tokens obtained in the previous step are reshaped to a vector of dimensions 1×20×4096.

[0057] Iterative modules such as Figure 5 As shown.

[0058] The entity information and audio text are processed by a text encoder to obtain a vector of dimension t×768. Where t is the sequence length of entity information and audio text; similarly, the video description is encoded by a text encoder to obtain a vector of size k×768. , where k is the sequence length of the video description. For vectors and To achieve common attention and ,Will and splicing Its dimension is (t + k)×768.

[0059] It can be expressed by the following formula: Where A is the attention matrix and T represents the transpose.

[0060] For vectors Attention score For vectors The attention score is obtained by multiplying the attention score by the corresponding vector. and ,Will and splicing .

[0061] Next, the features with dimensions of 1×20×4096 are reshaped into vectors with dimensions of 1×81920, and then passed through a connection layer with an input of 1×81920 and an output of 1×15360 to form a vector with dimensions of 1×15360. This vector is then reshaped into a vector V with dimensions of 20×768. Then, the vectors V and V are further processed... Perform common attention to obtain vectors and They are concatenated into a vector of size (20 + t + k) × 768, and then reshaped into a vector of size 1 × 20 × 4096 called tokens.

[0062] The initial module of the feedback model is trained using private data. The loss function is cross-entropy loss, and the trainer is Adam. The specific training process involves fixing a large language model while simultaneously training the initial modules of the mapping layer and the feedback model.

[0063] The iterative module of the feedback model is a self-optimizing process, continuously optimizing as it iterates to generate video descriptions. The trainer is Adam, and the loss function is the CLIP similarity score, which is the similarity between the generated video description and the video features each time.

[0064] The above examples yielded tensor-form vector tokens, video embeddings, object-form hint templates, and text-form hint questions. Next, the object-form hint templates and text-form hint questions will be converted into tensor forms.

[0065] First, the prompt template and text are concatenated as strings. Using llama-v2 as an example, the string is: <s> You are a helpful, respectful, and honest assistant. Answer questions as best you can, while ensuring your safety. Please ensure your answers are free of social bias and have positive implications. If a question is meaningless or inaccurate, explain why; do not answer incorrect questions. If you do not know the answer, please do not share false information.< / s> <s> Now you have some basic information: text< / s> The string above is encoded to form a tensor, and this tensor is concatenated with the vector tokens and the video embedding to form a new tensor. Next, the prompt questions sampled from the text list are also encoded to form a tensor and concatenated with the new tensor. Finally, a tensor with the form {prompt template, vector tokens, video embedding, prompt question} is obtained.

[0066] The tensor obtained by concatenating the above prompt template, vector tokens, video embedding, and prompt question is input into a large language model to obtain the final video description.

Claims

1. A multimodal description method based on a pre-trained large-scale language model, characterized in that, The specific steps are as follows; Step 1: Extract video features from the video and map them to video embeddings; extract audio features from the audio and convert them into audio text. Step 2: Extract entity information from video and audio text and convert it into entity text; Step 3: Build the prompt template, adding audio text and physical text; Step 4: Input the prompt template, vector tokens, video embedding, and prompt question into the large language model to obtain the video description; the vector tokens are obtained by outputting the video description through the feedback model. The prompt template, vector tokens, video embedding, and prompt question are concatenated and then input into the large language model. Among them, the vector tokens are a vector generated by the feedback model in response to the video description, which is used to guide the output of the large language model; The prompt question asks a large language model to describe the input video embedding; The feedback model includes an initial module and an iterative module. During the first video description, the initial module takes the video embedding obtained in the first step as input and outputs vector tokens. During the iterative video description process, the iterative module takes the video description of the previous generation as input, as well as the entity information and audio text obtained in the second step, and outputs vector tokens.

2. The multimodal description method based on a pre-trained large-scale language model according to claim 1, characterized in that, After the video is frame-stripped, a discrete video frame sequence is obtained. Image features are extracted from the video frame sequence using the image encoder CLIP. The last dimension of the image features is normalized. The dot product of the normalized image features is inverted to form a similarity matrix. Image features are filtered using a similarity matrix; a similarity filtering threshold is defined; the image features are divided into multiple groups with continuous image features, and the similarity of the image features within each group is greater than the similarity filtering threshold; the similarity filtering similarity between the last image feature of the first group and the first image feature of the second group in adjacent groups is less than the threshold; one image feature is taken from each group to form the filtered video features; The audio was transcribed into audio text using the Whisper speech recognition model.

3. The multimodal description method based on a pre-trained large-scale language model according to claim 1 or 2, characterized in that, The video features are hidden states composed of vectors. The video features are then mapped to video embeddings through a mapping layer. The mapping layer consists of simple fully connected layers, and the entire process is represented by the following formula: Where X represents the processed video features, W represents the weights of the mapping layer, and T represents the video embedding.

4. The multimodal description method based on a pre-trained large-scale language model according to claim 3, characterized in that, The second step is as follows: First, representative video frames from the discrete video frame sequence are input into the object detection API to obtain entity information contained in the video. Second, for entity information in the audio text, a named entity recognition method is used to obtain entity information based on the BERT deep learning model. Finally, the entity information obtained from the video frames and the entity information obtained from the audio text are concatenated to form entity text. For audio text with an empty text sequence, the audio entity information is set to empty and is not concatenated with the entity information list E obtained from the video frames; for audio text that is not empty, entities are extracted, and the obtained entities are concatenated with list E to form a new entity list. The entities extracted from the audio are to .

5. The multimodal description method based on a pre-trained large-scale language model according to claim 4, characterized in that, The third step specifically involves: The prompt template includes: Basic task setting: Assign a large model to act as a video description assistant, answering human questions; Role setting: Designate a large model to play the role of an assistant, and designate a human to play the role of a user; Spacing settings: Set a separator word between the user's question and the assistant's answer; Entity information setting: Specify the entity text obtained in step two; Audio text settings: Specify the audio text extracted in the first step; Build prompt templates include: The template above is set up by concatenating the basic task, audio text, entity information, user questions, intervals, and assistant answers in that order, and then converting it into text format.

6. The multimodal description method based on a pre-trained large-scale language model according to claim 1, characterized in that, The iterative module outputs the following vector tokens: entity information and audio text are processed by a text encoder to obtain vectors. The video description is processed by a text encoder to obtain a vector. For vectors and To achieve common attention and ,Will and splicing ; Next, the features with dimensions of 1×20×4096 are reshaped into vectors with dimensions of 1×81920, and then passed through a connection layer to form a vector with dimensions of 1×15360. This is then reshaped into a vector V with dimensions of 20×768. Then, the vectors V and V are further processed... Perform common attention to obtain vectors and splicing and Then reshape it into a vector of tokens with dimensions of 1×20×4096.

Images