Multi-modal video question answering method and device based on large language model optimization

Abstract

The invention discloses a multi-modal video question-answering method and device based on large language model optimization, and the method employs a CoVGT basic framework, optimizes video feature extraction and text input, and adds a video question-answering model output optimizer. The method comprises the steps of video feature extraction, text feature extraction, feature fusion, preliminary answer prediction, related sample generation, Prompt text generation and final answer generation. According to the method, the answer optimizer is designed by using the learning characteristics in GPT-3 analogy, related sample generation, Prompt text generation and final answer generation, and compared with a basic model, the model accuracy is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal recognition and text generation, and in particular to a multimodal video question-answering method and device based on large language model optimization. Background Art

[0002] In recent years, with the rapid development of artificial intelligence (AI), AI automatic answering systems have received more and more attention. The artificial intelligence answering system is a technology that applies human intelligence to information acquisition and organically combines the ability to analyze problems with the ability to seek answers. It is an intelligent question-answering system that integrates language processing and multimodal intelligent computing. It can process voice, image and video content information in real time and can automatically answer questions. The automatic answering system can not only understand natural language, but also integrate multimodal content and extract key information to generate answers. It not only needs to have knowledge of language expression and data processing, but also needs to be able to grasp more specialized information, summarize and organize it, so as to provide accurate answers.

[0003] Video Question Answering (VideoQA) is an interdisciplinary subject involving computer vision and natural language processing. It aims to solve the problem of how to map natural language questions to videos and extract answers from videos. With the development of deep learning and neural networks, video question answering has received more and more attention. In video question answering, a natural language question and a short video are input into a deep neural network, which outputs an answer related to the question by learning the correlation between the two inputs. The implementation of video question answering requires combining video information with natural language processing, and realizing cross-modal question answering through multimodal in-depth understanding of video and language. In recent years, many video question answering models based on deep learning have been proposed. These models use different architectures and technologies, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), graph neural networks (GNNs), attention mechanisms (Transformers), etc., to improve the accuracy and robustness of video question answering models. At the same time, video question answering systems have been widely used in many fields, such as intelligent question answering, intelligent customer service, autonomous driving, medical diagnosis, etc. In short, video question answering is an important interdisciplinary subject with broad application prospects. Further improving the performance of the video question-answering model can enable more extensive and effective applications in more fields.

[0004] At present, the existing video question answering models still have some shortcomings in performance, especially when dealing with complex video scenes, long video clips or multimodal information, the model's prediction accuracy and robustness are relatively lacking. In addition, the existing technology has limited capabilities in question understanding and video content association processing, which often leads to the model's inability to accurately give the best answer, although the correct answer is usually among the first few high-confidence prediction results of the model. Therefore, by re-optimizing the answer selection process and further improving the reasoning ability of the video question answering model, its performance in complex scenarios can be significantly improved, better meeting the needs of actual applications, and achieving widespread and effective applications in more fields. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies in the prior art and to propose a multimodal video question answering method and device based on large language model optimization.

[0006] The objective of the present invention is achieved through the following technical solution: A multimodal video question answering method based on large language model optimization comprises the following steps:

[0007] S1. Generate video description and frame description for the input video using the pre-trained video summarization model.

[0008] S2. Video feature extraction: Use the pre-trained object detector to extract the spatial position features of the object features in the video, use CLIP to extract the image features in the video, and fuse the extracted features;

[0009] S3, text feature extraction: the video description text and the input text are concatenated and input into the language model, and the token output of the language model is linearly projected to obtain text features;

[0010] S4, performing cross-modal interaction between video features and text features to obtain video features related to the text query;

[0011] S5. Predict the answer based on the video features and the preliminary candidate answer pool to obtain a preliminary answer prediction result;

[0012] S6. Generate relevant samples, extract the preliminary answer prediction results and relevant samples as the final candidate answer pool according to the similarity between the text representation and the video representation, convert the similarity into a confidence value, splice it with the video description and the frame description and input it into the large model;

[0013] S7. Use a multi-query integration strategy to count the answers generated by the large model and return the answer with the highest count as the final predicted answer.

[0014] Furthermore, the generating of the video description and the frame description specifically includes:

[0015] Cvideo =M V (v)

[0016] Among them, M V is a pre-trained video summarization model, v is a video;

[0017] Extract the first frame F from the video v f , the middle frame F m and the last frame F l ; Use the pre-trained Image Caption model to generate an image description for each extracted frame, expressed as:

[0018] C frame-f =M I (F f )

[0019] C frame-m =M I (F m )

[0020] C frame-l =M I (F l )

[0021] Among them, M I It is a pre-trained Image Caption model.

[0022] Furthermore, the video feature extraction specifically includes:

[0023] After sparse sampling of the video v, we get l video Frame, video The frame is evenly divided into k segments, each segment contains Frame; for each frame, use the pre-trained object detector to extract n RoI-aligned features to obtain the object features in the image and spatial location characteristics r i Represents the i-th target area in a frame; image features are obtained by the CLIP model I t ∈{1,2…l video}; Input the above-mentioned features at all levels into the DGT module to obtain the video fusion feature F DGT , expressed as:

[0024]

[0025] The symbol [;] indicates series connection, and d is 512.

[0026] Furthermore, the text feature extraction is specifically as follows: text feature X qIt is obtained by linearly projecting the token output of the language model and is expressed as:

[0027] X q =W q (F lang (Q+C video ))

[0028] Where W q ∈R 768×d , F lang is the language model RoBERTa, Q is the question text, C video It is the video description text, and the symbol [+] represents the text concatenation operation.

[0029] Furthermore, the cross-modal interaction mode is expressed as:

[0030]

[0031] Where β=σ((F DGT ) T X q ), σ is the Softmax operation for normalizing each row; M represents the number of tokens in the text query; after a Transformer module, the global representation f of the video feature is finally obtained qv ∈R d , expressed as:

[0032] f qv =MPool(MHSA (H) (F′ DGT ))

[0033] Among them, MPool represents a simple mean pooling operation, MHSA is a multi-head self-attention mechanism, the Transformer module uses MHSA to mix token information, and H is the number of layers.

[0034] Furthermore, the answer prediction in S5 is specifically as follows: firstly, a global representation of a specific candidate answer is calculated, which is expressed as:

[0035] f A =MPool(W q (F laag (Answer)))

[0036] Among them, Answer represents the answer text in the candidate answer pool, where the candidate answer pool is provided by the data set; the final predicted answer is expressed as a joint decision as follows:

[0037] a * =argMax((f qv ) T FA ⊙(X q ) T F A ))

[0038] in |Answer| represents the number of answer candidates, and ⊙ is the element-wise product.

[0039] Furthermore, the related sample generation is specifically as follows:

[0040] Get the input in the training set from the open source dataset (v i ,q i ), by calculating the fusion feature f obtained by the original video input in the original video question answering model qv And the fusion feature f obtained from each training input i qv The cosine similarity between them is used to select the first N training examples with the largest similarity value as the relevant examples. ε is expressed as:

[0041]

[0042] Among them I r is the index set of the first N samples with the largest similarity in the training set, and M is the number of training sets;

[0043] ε={(v i ,q i ,a i )|i∈I r}

[0044] Among them, a is the answer word corresponding to the training set example.

[0045] Furthermore, the extraction of the preliminary answer prediction results and related samples as the final candidate answer pool according to the similarity between the text representation and the video representation is specifically as follows: taking the top K answers with the largest similarity between the text representation and the video representation obtained by the related samples and the original input as the final candidate answer pool:

[0046] a topK =argTopK((f qv ) T F A ⊙(X q ) T F A ))

[0047] where a topK The value in is the similarity value, f qv The global representation of video features is obtained for video feature extraction, X q It is the text representation obtained from the language model.

[0048] On the other hand, the present invention specification also provides a multimodal video question-answering device based on large language model optimization, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the multimodal video question-answering method based on large language model optimization.

[0049] On the other hand, the specification of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the multimodal video question-answering method based on large language model optimization is implemented.

[0050] Beneficial effects of the present invention:

[0051] Based on the original video question-answering model, the present invention combines the reasoning optimization capabilities of the large language model, performs secondary optimization based on the predicted answers provided by the original model, and selects a better answer. This optimization method effectively solves the problem of insufficient answer accuracy in complex scenarios in the existing technology, significantly improves the overall performance of the model, enables it to better cope with diverse video question-answering applications, and provide higher quality question-answering results.

[0052] In addition, the present invention is not a module embedded in the original video question-answering model, but an independent module connected to the original video question-answering model. This means that our method does not rely on a specific basic video question-answering model (such as CoVGT), but can be flexibly used in combination with a variety of basic video question-answering models. As a pluggable module in the reasoning optimization process, this method realizes seamless integration between models and truly achieves a "plug and play" effect. This flexibility enables the method to adapt to different model structures and significantly improves the versatility and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 An overall model framework diagram of a multimodal video question answering method based on large language model optimization provided in an embodiment of the present invention.

[0054] Figure 2 A schematic diagram of the result accuracy of the multimodal video question answering method based on large language model optimization provided in an embodiment of the present invention.

[0055] Figure 3 A structural diagram of a multimodal video question-answering device based on large language model optimization provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0057] like Figure 1 As shown, a multimodal video question answering method based on large language model optimization provided by the present invention adopts the video question answering model CoVGT as the basic framework.

[0058] The original video question answering model (CoVGT) is as follows Figure 1 As shown in (a) of FIG. 1 , it is divided into three parts: video feature extraction, text feature extraction, and feature fusion. The present invention optimizes video feature extraction and text input respectively. The designed video question answering model output optimizer is shown in FIG. Figure 1 As shown in (b) in the figure, it is divided into three parts: relevant sample generator, prompt text generator and answer generator.

[0059] Specifically, the method comprises the following steps:

[0060] S1. Generate video description and frame description for the input video using the pre-trained video summarization model.

[0061] The present invention extracts the video description and the three frame descriptions corresponding to each video. The video description is generated for the video using the pre-trained video caption model, which is expressed as:

[0062] C video =M V (v)#(1)

[0063] Among them, M V It is a pre-trained video caption model. The mPLUG-2 model is used in the present invention, and v is a video.

[0064] Extract the first frame F from the video v f , the middle frame F m and the last frame F l . Use the pre-trained Image Caption model to generate an image description for each extracted frame, expressed as:

[0065] C frame-f =M I (F f )#(2)

[0066] C frame-m =M I (F m )#(3)

[0067] C frame-l =M I (F l )#(4)

[0068] Among them, M IIt is a pre-trained Image Caption model. The Blip-Image-Captioning-Large model is used in the present invention.

[0069] S2. Video feature extraction: Use the pre-trained object detector to extract the spatial position features of the object features in the video, use CLIP to extract the image features in the video, and fuse the extracted features;

[0070] Specifically, the video v is sparsely sampled to obtain l video (l video =32) frames, video The frame is evenly divided into k (k = 8) segments, each segment contains For each frame, a pre-trained object detector (Faster R-CNN) is used to extract n (n = 10) RoI-aligned features to obtain the object features in the image. and spatial location characteristics r i represents the i-th target region in a frame. The frame-level features obtained by the image classification model (ResNet-101) in the original model are replaced by those generated by the CLIP model. Image features are obtained I t ∈{1,2…l video Input the above-mentioned features at all levels into the DGT (Dynamic Graph Transformer) module to obtain the initial fusion feature F of the video DGT , expressed as:

[0071]

[0072] The symbol [;] represents concatenation, where d represents the hidden layer dimension of the model, which is 512.

[0073] S3, text feature extraction: the video description text and the input text are concatenated and input into the language model, and the token output of the language model is linearly projected to obtain text features;

[0074] For text node X q , obtained by a simple linear projection of the token output of the language model, is expressed as:

[0075] X q =W q (F lang (Q+C video ))#(6)

[0076] Where W q is the language model training parameter, W q ∈R768×d , F lang is the language model RoBERTa, Q is the question word, C video is the video description text generated in formula (1), and the symbol [+] is a simple text concatenation operation.

[0077] The original video question-answering model output is the answer prediction. First, the global representation of a specific candidate answer is obtained by the mean pooling operation, which is expressed as:

[0078] f A =MPool(W q (F lang (Answer)))#(7)

[0079] Among them, Answer represents the answer text in the candidate answer pool, where the candidate answers in the candidate answer pool are pre-given in the open source dataset.

[0080] S4. Perform cross-modal interaction between video features and text features to obtain video features related to text queries; cross-modal interaction between visual nodes and text nodes can find visual information content related to specific text queries. The interaction method is expressed as:

[0081]

[0082] Where β=σ((F DGT ) T X q ), σ is the Softmax operation for normalizing each row; M represents the number of tokens in the text query. After a Transformer module, the global representation f of the video feature is finally obtained. qv ∈R d , expressed as:

[0083] f qv =MPool(MHSA (H) (F′ 1GT ))#(9)

[0084] Among them, MPool represents a simple mean pooling operation, MHSA is a multi-head self-attention mechanism, the Transformer module uses MHSA to mix token information, and H is the number of layers.

[0085] The final predicted answer can be expressed as a joint decision as follows:

[0086] a * =argMax((f qv ) T F A ⊙(X q )T F A ))#(10)

[0087] in |Answer| represents the number of answer candidates, ⊙ is the element-wise product, a * Contains the predicted answer text and the similarity value between the corresponding text representation and the video representation.

[0088] Generate relevant samples, extract preliminary answer prediction results based on similarity as the final candidate answer pool, convert similarity into confidence value, splice with video description and frame description and input into the large model;

[0089] Specifically include: getting the input in the training set from the open source dataset (v i ,q i ), by calculating the fusion feature f obtained by the original video input in the original video question answering model qv And the fusion feature f obtained from each training input i qv The cosine similarity between them is used to select the first N (N=10) training examples with the largest similarity value as the relevant examples ε. It can be expressed as:

[0090]

[0091] Among them I r It is the index set of the first N samples with the largest similarity in the training set, and M is the number of training sets.

[0092] ε={(v i ,q i ,a i )|i∈I r}#(12)

[0093] Among them, a is the answer word corresponding to the training set example.

[0094] The top K (K=3) answers with the greatest similarity between the trained text representation and the video representation input into the original model are taken as the final candidate answer pool.

[0095] It is expressed as:

[0096] a topK =argTopK((f qv ) T F A ⊙(x q ) T F A ))#(13)

[0097] where a topKThe value in is the similarity value. Applying the Softmax() function converts it into a confidence value between 0 and 1, which is helpful for the large language model to understand the semantics, expressed as:

[0098] a-conf topK =Softmax(a topK )#(14)

[0099] In order to better utilize the excellent context learning ability of the large language model (GPT-3), we concatenate and generate Prompt-Context text. The concatenation method is as follows:

[0100]

[0101] The symbol [+] is a simple text concatenation, c video 、c frame The video description and frame description text generated by S1, w jk For candidate answers, is the corresponding confidence value, Line-Prefix is ​​the text "===\n", "\n" is the line break character. i∈(1,2…N)

[0102] S7. Use a multi-query integration strategy to count the answers generated by the large model and return the answer with the highest count as the final predicted answer.

[0103] In order to make better use of the examples in the available training set, the present invention uses a multi-query integration strategy, where T is the number of queries (T=5), and each query still contains N relevant cases. The Prompt-Text of a single query is represented as:

[0104]

[0105] The Prompt-Head is the query prompt header: "Please answer the question according to the context and the answer candidates. Each answer candidate is associated with a confidence score within a bracket. The true answer may not be included in the candidates". The complete query text is input into the large language model (GPT-3), and a majority vote is performed on the T prediction results to determine the final answer pre-answer, which is expressed as:

[0106]

[0107] Among them, argMaxCount() counts the generated answers and returns the answer with the highest count.

[0108] like Figure 2 The figure shows the model accuracy. After optimizing the feature extraction and text representation parts of the original model and adding the answer optimizer, experiments were conducted on the common video question answering datasets MSRVTT-QA and MSVD-QA. The proposed method improved the model accuracy by 7.84% and 4.38% respectively compared with the basic model.

[0109] Corresponding to the aforementioned embodiment of the multimodal video question-answering method based on large language model optimization, the present invention also provides an embodiment of a multimodal video question-answering device based on large language model optimization.

[0110] See also Figure 3 A multimodal video question-answering device based on large language model optimization provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a multimodal video question-answering method based on large language model optimization in the above embodiment.

[0111] The embodiment of a multimodal video question-answering device based on large language model optimization provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 3 As shown, it is a hardware structure diagram of any device with data processing capability in which a multimodal video question-answering device based on large language model optimization provided by the present invention is located, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0112] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0113] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0114] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a multimodal video question-answering method based on large language model optimization in the above embodiment is implemented.

[0115] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0116] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the multimodal video question-answering method based on large language model optimization.

[0117] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0118] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A multimodal video question answering method based on large language model optimization, characterized in that: The steps include: S1. Generate video description and frame description for the input video using the pre-trained video summarization model. S2. Video feature extraction: Use the pre-trained object detector to extract the spatial position features of the object features in the video, use CLIP to extract the image features in the video, and fuse the extracted features; S3, text feature extraction: the video description text and the input text are concatenated and input into the language model, and the token output of the language model is linearly projected to obtain text features; S4, performing cross-modal interaction between video features and text features to obtain video features related to the text query; S5. Predict the answer based on the video features and the preliminary candidate answer pool to obtain a preliminary answer prediction result; S6. Generate relevant samples, extract the preliminary answer prediction results and relevant samples as the final candidate answer pool according to the similarity between the text representation and the video representation, convert the similarity into a confidence value, splice it with the video description and the frame description and input it into the large model; S7. Use a multi-query integration strategy to count the answers generated by the large model and return the answer with the highest count as the final predicted answer.

2. According to the multimodal video question answering method based on large language model optimization according to claim 1, it is characterized in that: The generating of the video description and the frame description specifically comprises: C video =M V (v) Among them, M V is a pre-trained video summarization model, v is a video; Extract the first frame F from the video v f , the middle frame F m and the last frame F l ; Use the pre-trained Image Caption model to generate an image description for each extracted frame, expressed as: C frame-f =M I (F f ) C frame-m =M I (F m ) C frame-l =M I (F l ) Among them, M I It is a pre-trained Image Caption model.

3. The multimodal video question answering method based on large language model optimization according to claim 1, characterized in that: The video feature extraction specifically includes: After sparse sampling of the video v, we get l video Frame, video The frame is evenly divided into k segments, each segment contains Frame; for each frame, use the pre-trained object detector to extract n RoI-aligned features to obtain the object features in the image and spatial location characteristics r i Represents the i-th target area in a frame; image features are obtained by the CLIP model I t ∈{1,2…l video }; Input the above-mentioned features at all levels into the DGT module to obtain the video fusion feature F DGT , expressed as: The symbol [;] indicates series connection, and d is 512.

4. The multimodal video question answering method based on large language model optimization according to claim 1, characterized in that: The text feature extraction is specifically: text feature X q It is obtained by linearly projecting the token output of the language model and is expressed as: X q =W q (F lang (Q+C video )) Where W q ∈R 768×d , F lang is the language model RoBERTa, Q is the question text, C video It is the video description text, and the symbol [+] represents the text concatenation operation.

5. The multimodal video question answering method based on large language model optimization according to claim 3, characterized in that: The cross-modal interaction method is expressed as: Where β=σ((F DGT ) T X q ), σ is the Softmax operation for normalizing each row; M represents the number of tokens in the text query; after a Transformer module, the global representation f of the video feature is finally obtained qv ∈R d , expressed as: f qv =MPool(MHSA (H) (F′ DGT )) Among them, MPool represents a simple mean pooling operation, MHSA is a multi-head self-attention mechanism, the Transformer module uses MHSA to mix token information, and H is the number of layers.

6. The multimodal video question answering method based on large language model optimization according to claim 1, characterized in that: The answer prediction in S5 is specifically as follows: firstly, the global representation of a specific candidate answer is calculated, which is expressed as: f A =MPool(W q (F lang (Answer))) Among them, Answer represents the answer text in the candidate answer pool, where the candidate answer pool is provided by the data set; the final predicted answer is expressed as a joint decision as follows: a * =argMax((f qv ) T F A ⊙(X q ) T F A )) in |Answer| represents the number of answer candidates, and ⊙ is the element-wise product.

7. The multimodal video question answering method based on large language model optimization according to claim 1, characterized in that: The relevant sample generation is specifically as follows: Get the input in the training set from the open source dataset (v i ,q i ), by calculating the fusion feature f obtained by the original video input in the original video question answering model qv And the fusion features obtained from each training input The cosine similarity between them is used to select the first N training examples with the largest similarity value as the relevant examples. ε is expressed as: Among them I r is the index set of the first N samples with the largest similarity in the training set, and M is the number of training sets; ε={(v i ,q i ,a i )|i∈I r } Among them, a is the answer word corresponding to the training set example.

8. The multimodal video question answering method based on large language model optimization according to claim 6, characterized in that: The method of extracting the preliminary answer prediction results and related samples as the final candidate answer pool according to the similarity between the text representation and the video representation is as follows: taking the top K answers with the largest similarity between the text representation and the video representation obtained from the related samples and the original input as the final candidate answer pool: a topK =argTopK((f qv ) T F A ⊙(X q ) T F A )) where a topK The value in is the similarity value, f qv The global representation of video features is obtained for video feature extraction, X q It is the text representation obtained from the language model.

9. A multimodal video question-answering device based on large language model optimization, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a multimodal video question answering method based on large language model optimization as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a multimodal video question answering method based on large language model optimization as described in any one of claims 1 to 8 is implemented.

Images