Long video question and answer processing system based on large language model

By designing a long video question and answer processing system based on large language models, the problem of low long video processing efficiency and high computing resource consumption in existing technology, and realized the efficient Q & A processing and accurate information extraction of long videos.

CN119377440APending Publication Date: 2025-01-28LINKER
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411408884.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-10
Publication Date
2025-01-28

AI Technical Summary

Technical Problem

When processing long videos, existing long video processing systems have problems such as high computing resource consumption, low recognition efficiency, inaccurate subtitles generation and inability to locate Q&A information when processing long videos.

Method used

Design a long video question and answer processing system based on a large language model. The long video is separated into visual and audio parts information through the pre -processing module. Questions and answer processing of type questions.

Benefits of technology

The system significantly improves the performance and efficiency of long video Q&A, can handle longer videos, reduce the consumption of computing resources, and improve the accuracy and operability of the question and answer.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119377440A_ABST
    Figure CN119377440A_ABST
Patent Text Reader

Abstract

The invention discloses a long video question and answer processing system based on a large language model, and the system comprises a preprocessing module which is used for separating a long video into visual part information and audio part information; the question classification module is used for classifying input questions, specifically, the input questions are divided into common video questions, video time related questions, video vision related questions and video audio related questions; and the question processing flow module is in communication connection with the preprocessing module and the question classification module, receives the four different questions classified by the question classification module, and then enters different processing flows in combination with the video part information and the audio part information of the preprocessing module to complete question and answer processing of the long video. According to the long video question and answer processing system based on the large language model, the model design is simplified, the performance and efficiency of long video question and answer are remarkably improved, and a solution which is more operable and efficient is provided for practical application.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a long video processing system, and more particularly to a long video processing system based on a large language model. Background Art

[0002] With the explosive growth of online videos, video recognition has attracted a lot of attention due to its wide range of application scenarios, such as video recommendation, video retrieval, and network security. However, while classic video recognition models have impressive performance, the excessive complexity and computational cost of these models restrict their deployment in many resource-limited applications, such as edge computing, online vision, etc. Therefore, it becomes crucial to design efficient video recognition models to maintain recognition accuracy while minimizing computational cost, especially for untrimmed long videos.

[0003] Currently, the existing systems for processing long videos do not take into account the efficiency of recognition and the consumption of computing power. Therefore, in the process of processing long videos, the existing systems have problems such as long-term reliance on complex relationship modeling, high consumption of computing resources, inaccurate subtitle generation, and inability to locate question and answer information. Summary of the invention

[0004] In view of the deficiencies in the prior art, the object of the present invention is to provide a long video processing system based on a large language model that can solve one or more problems.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solution: a long video question-answering processing system based on a large language model, characterized in that it includes:

[0006] A preprocessing module is used to separate long videos into visual part information and audio part information;

[0007] The question classification module is used to classify the input questions into common video questions, video time-related questions, video vision-related questions, and video audio-related questions;

[0008] The question processing flow module is connected to the preprocessing module and the question classification module, receives four different questions classified by the question classification module, and then enters different processing flows in combination with the video part information and the audio part information of the preprocessing module to complete the question-answering processing of the long video. The processing flow is as follows:

[0009] For common video questions, the visual description C and audio description T of the entire long video, as well as the question Q and option A, will be input into the prompt words for answering the question;

[0010] For vision-related questions, the visual description C is used to first obtain a summary SC based on the question Q, and then the question is answered based on the summary SC;

[0011] For audio-related questions, the audio text description T is used to first obtain a summary ST based on the question Q, and then the question and answer are performed based on the summary ST;

[0012] For questions related to video time, the visual description C and audio description T of the entire long video are directly used for question answering.

[0013] As a further improvement of the present invention, after the preprocessing module separates the long video into visual part information and audio part information, it extracts frames of the video at a certain frequency P, such as fps0.5, for the video part information. The extracted video frames will be sent to the large model to generate a visual description of the video in a language of less than 20 words:

[0014] C[c1,c2,c3,......c n ];

[0015] Among them, C is the description of the entire video, c n is the text description of the nth frame, c is Deposit it in c in the form of

[0016] At the same time, the audio information is segmented at the same frequency and converted into text through the ASR model to generate speech text:

[0017] t[t1, t2, t3, ... t n ];

[0018] Among them, T is the description of the entire audio part, t n is the audio description of the nth segment, t is In the form of T, the duration D of the video and the frame rate P of the video will also be saved for subsequent processes. As a further improvement of the present invention, the question classification module classifies the input questions in a specific way by using large model prompts for classification. First, the question Q and option A [a1, a2, a3, a4] are input, and then the following large model prompts are used: "Please judge which category the question belongs to based on the questions and options of the following video: ["Video vision related questions", "Video audio related questions", "Video time related questions", "Ordinary video questions"], \nQuestion: Q\nOptions: (A) a1; (B) a2; (C) a3; (D) a4\nPlease read the questions and options carefully and judge the type of question. Your answer only includes the type of question", and the input question Q is classified according to the user's answer.

[0019] As a further improvement of the present invention, in the question processing flow module, the visual description C and audio description T of the entire long video, as well as the question Q and the option A [a1, a2, a3, a4] are input together with prompt words to answer the question, wherein the prompt word is "Please provide a single letter answer (A, B, C, D) for the following multiple choice questions, and the answer must be one of the letters (A, B, C or D). You may not provide any other answers or explanations. If you are not sure, please answer the most likely answer. I will give you some language descriptions of the video content from a first-person perspective. The video is 1fps and is described as letters for every two frames. The following is the description: C+T\nQuestion: Q\nOption:\n(A): a1\n(B): a2\n(C): a3\n(D): a4\n", and then obtain the answer to the above prompt word to complete the processing of the common video question;

[0020] The visual description C is the frame image description C[c1, c2, c3, ... c n ], the audio description T is the speech text T[t1, t2, t3, ... t n ].

[0021] As a further improvement of the present invention, the question processing flow module uses the visual description C to first obtain the summary SC based on the question Q, and then conducts question and answer based on the summary SC in the specific manner: first input the visual description C, question Q and option A[a1, a2, a3, a4] as well as the basic information of the video, the video length D and the video frame frequency P into the prompt word for question answering, wherein the prompt word is "give you some language descriptions of the visual content of the video. The video length is D seconds. Each sentence describes a {1 / P}s segment. These descriptions are continuous, without overlap, and accurately cover the entire video. The following is the description: C. When summarizing, please remember that your summary will be used to answer this multiple-choice question: Q" and then based on the above prompt Get the summary SC based on the answer of the word, and then enter the summary SC into the following prompt word "Please provide a single letter answer (A, B, C, D) to the following multiple-choice questions, and the answer must be one of the letters (A, B, C or D). You may not provide any other answers or explanations. I will give you some language descriptions of the visual content of a first-person perspective video. The video length is D seconds. Here is the description: SC You will answer a choice based on these descriptions. Your answer should be a letter selected from the options: Q\nThese are the option choices:\n(A): a1\n(B): a2\n(C): a3\n(D): a4\n", and then get the answer to the above prompt word to complete the processing of visual-related questions.

[0022] As a further improvement of the present invention, the question processing flow module uses the audio text description T to first obtain a summary ST based on the question Q, and then conducts question and answer based on the summary ST in the specific manner: first input the audio description T, the question Q and the options A[a1, a2, a3, a4] as well as the basic information of the video, the video duration D and the video frame frequency P into the prompt word for question answering, wherein the prompt word is "give you some language descriptions of the visual content of the video. The video length is D seconds. Each sentence describes a {1 / P}s segment. These descriptions are continuous, without overlap, and accurately cover the entire video. The following is the description: T. When summarizing, please remember that your summary will be used to answer this multiple-choice question: Q". Then based on the above prompt word Get the summary SC based on the answer to the prompt words, and then input the summary SC into the following prompt words "Please provide a single letter answer (A, B, C, D) to the following multiple-choice questions, and the answer must be one of the letters (A, B, C or D). You may not provide any other answers or explanations. I will give you some language descriptions of the visual content of the first-person perspective video. The video length is D seconds. Here is the description: ST You will answer a choice based on these descriptions, and your answer should be a letter selected from the options: Q\nThese are the option choices:\n(A): a1\n(B): a2\n(C): a3\n(D): a4\n", and then get the answer to the above prompt words to complete the processing of audio-related questions. As a further improvement of the present invention, the question processing flow module uses the visual description C and audio description T of the entire long video for question and answering in the following specific manner: first, the visual description C, the audio description T, the question Q and the option A[a1, a2, a3, a4] as well as the basic information of the video, the video duration D and the video frame extraction frequency P are input into the prompt word for question answering, wherein the prompt word is "I will provide a video description and a question about the video. The video is 1FPS and is described as a video description of every {1 / P} frame. To answer this question, what is the minimum frame interval that needs to be confirmed first? {frame_start_index, frame_end_index}, here is the description: \nC+T\here is the question: Q\nPlease follow the following output format for the output: \n#Example 2: [30,60]\n#Example 3: [1,10]and[50,60]", then obtain the answer to the above prompt word, and then convert the time through the frame label to complete the processing of video time-related problems.

[0023] The beneficial effect of the present invention is that, through the setting of the preprocessing module, the long video can be effectively preprocessed to separate the visual part information and the audio part information, and then the input questions can be classified through the question classification module to divide them into common video questions, video time-related questions, video vision-related questions and video audio-related questions. After that, through the function of the question processing flow module, question-answering processing for different questions can be realized. Compared with the methods in the prior art, this simplifies the model design and significantly improves the performance and efficiency of long video question-answering, providing a more operational and efficient solution for practical applications; more comprehensive processing of long videos can process longer videos, while traditional video question-answering processing can only This process can be used to process videos of dozens of minutes to hours, and can be used to process videos of dozens of minutes to hours. Questions are classified according to question and answer types, and different processing processes are used to handle the questions, which strengthens the model's understanding of the questions and allows the question-answering model to focus on the key information of the question in the entire long video, improving the question-answering performance. The use of specific prompt words reduces resource consumption, allowing only relevant information to be focused on for different questions, and the use of tokens is significantly reduced compared to other question-answering processes. Very difficult time-related questions are converted into simple query questions in a simple way, the video is segmented into frames at a specific frequency, and frame labels and related frequency information are input into the model so that the model can understand the video timing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a processing flow chart of the long video question answering processing system based on a large language model of the present invention;

[0025] Figure 2 A flowchart for preprocessing for the preprocessing module;

[0026] Figure 3 Flowchart for problem classification module;

[0027] Figure 4 This is a flowchart of the problem handling process module for common video problems;

[0028] Figure 5 This is a flowchart of the problem handling process module for visual related problems;

[0029] Figure 6 This is a flowchart of the problem handling process module for audio-related problems;

[0030] Figure 7 This is a flowchart of how the problem handling process module handles video time-related issues. DETAILED DESCRIPTION

[0031] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0032] Reference Figure 1 As shown, a long video question-answering processing system based on a large language model in this embodiment is characterized by comprising:

[0033] A preprocessing module is used to separate long videos into visual part information and audio part information;

[0034] The question classification module is used to classify the input questions into common video questions, video time-related questions, video vision-related questions, and video audio-related questions;

[0035] The question processing flow module is connected to the preprocessing module and the question classification module, receives four different questions classified by the question classification module, and then enters different processing flows in combination with the video part information and the audio part information of the preprocessing module to complete the question-answering processing of the long video. The processing flow is as follows:

[0036] For common video questions, the visual description C and audio description T of the entire long video, as well as the question Q and option A, will be input into the prompt words for answering the question;

[0037] For vision-related questions, the visual description C is used to first obtain a summary SC based on the question Q, and then the question is answered based on the summary SC;

[0038] For audio-related questions, the audio text description T is used to first obtain a summary ST based on the question Q, and then the question and answer are performed based on the summary ST;

[0039] For questions related to video time, the visual description C and audio description T of the entire long video will be directly used for question answering. In the process of using the question answering processing system of this embodiment, it is only necessary to input the long video and the question into the preprocessing module for preprocessing, and then input the preprocessed visual part information and audio part information into the question classification module, classify them through the question classification module, and then perform question answering processing on the classified questions through the question processing flow module. Compared with the method in the prior art, the questions are classified according to the question and answer type, and the questions are processed with different processing flows, which strengthens the model's understanding of the questions, so that the question answering focuses on the key information of the questions in the entire long video, and improves the question answering performance; the use of specific prompt word design reduces resource consumption, so that only relevant information is focused on for different questions, and the use of tokens is significantly reduced compared with other question answering processes; a very difficult time-related question is converted into a simple query question in a simple way, the video is framed and segmented at a specific frequency, and the frame labels and related frequency information are input into the model, so that the model can understand the video timing.

[0040] Reference Figure 2 As shown in the figure, as a specific implementation of the improvement, after the preprocessing module separates the long video into visual part information and audio part information, it extracts frames of the video at a certain frequency P, such as fps0.5, for the video part information. The extracted video frames will be sent to the large model to describe the video in a language of less than 20 words to generate a visual description of the video:

[0041] C[c1,c2,c3,......c n ];

[0042] Among them, C is the description of the entire video, c n is the text description of the nth frame, C is Deposit it in C in the form of

[0043] At the same time, the audio information is segmented at the same frequency and converted into text through the ASR model to generate speech text:

[0044] T[t1,t2,t3,......t n ];

[0045] Among them, T is the description of the entire audio part, t n is the audio description of the nth segment, t is By setting the above steps, the preprocessing of long videos can be effectively realized. Figure 3 As shown, as an improved specific implementation method, the question classification module classifies the input questions in a specific way by using large model prompt words for classification. First, input question Q and option A [a1, a2, a3, a4], and then use the following large model prompt words "Please determine which category the question belongs to based on the questions and options in the following video: ["Video vision related questions", "Video audio related questions", "Video time related questions", "Ordinary video questions"], \nQuestion: Q\nOptions: (A) a1; (B) a2; (C) a3; (D) a4\nPlease read the question and options carefully and determine the type of question. Your answer only includes the type of question", and classify the input question Q according to the user's answer. By setting the above steps, the classification of questions can be simply and effectively implemented through the question-answering method. Refer to Figure 4As shown, as a specific implementation method of the improvement, the visual description C and audio description T of the entire long video, as well as the question Q and the option A [a1, a2, a3, a4] are input into the prompt words together to answer the question, wherein the prompt words are "Please provide a single letter answer (A, B, C, D) for the following multiple choice questions, and the answer must be one of the letters (A, B, C or D). You may not provide any other answers or explanations. If you are not sure, please answer the most likely answer. Give you some language descriptions of the first-person perspective video content. The video is 1fps and is described as letters for every two frames. The following is the description: C+T\nQuestion: Q\nOption:\n(A): a1\n(B): a2\n(C): a3\n(D): a4\n", and then obtain the answer to the above prompt words to complete the processing of ordinary video questions;

[0046] The visual description C is the frame image description C[c1, c2, c3, ... c n ], the audio description T is the speech text T[t1, t2, t3, ... t n ], through the settings of the above steps, it is possible to effectively handle common video problems through question-and-answer methods.

[0047] Reference Figure 5 As shown, as a specific implementation method of the improvement, the question processing flow module uses the visual description C to first obtain the summary SC based on the question Q, and then conducts question and answer based on the summary SC. The specific method is: first input the visual description C, question Q and option A[a1, a2, a3, a4] as well as the basic information of the video, the video duration D and the video frame rate P into the prompt word for question answering, wherein the prompt word is "give you some language descriptions of the visual content of the video. The video length is D seconds. Each sentence describes a {1 / P}s segment. These descriptions are continuous, without overlap, and accurately cover the entire video. The following is the description: C. When summarizing, please remember that your summary will be used to answer this multiple-choice question: Q". Then based on the above Get the summary SC by answering the above prompts, and then input the summary SC into the following prompts "Please provide a single-letter answer (A, B, C, D) to the following multiple-choice questions, and the answer must be one of the letters (A, B, C or D). You may not provide any other answers or explanations. I will give you some language descriptions of the visual content of a first-person perspective video. The video length is D seconds. Here is the description: SC You will answer a choice based on these descriptions, and your answer should be a letter selected from the options: Q\nThese are the option choices:\n(A): a1\n(B): a2\n(C): a3\n(D): a4\n", and then get the answers to the above prompts to complete the processing of vision-related questions.

[0048] Reference Figure 6 As shown, as a specific implementation method of the improvement, the question processing flow module uses the audio text description T to first obtain a summary ST based on the question Q, and then conducts question and answer based on the summary ST in the specific manner: first input the audio description T, the question Q and the options A[a1, a2, a3, a4] as well as the basic information of the video, the video duration D and the video frame rate P into the prompt words for question answering, wherein the prompt words are "give you some language descriptions of the visual content of the video. The video length is D seconds. Each sentence describes a {1 / P}s segment. These descriptions are continuous, without overlap, and accurately cover the entire video. The following is the description: T. When summarizing, please remember that your summary will be used to answer this multiple-choice question: Q". After that, a summary SC is obtained based on the answer to the above prompt words, and then the summary SC is used. Enter the following prompt words "Please provide a single letter answer (A, B, C, D) to the following multiple-choice questions, and the answer must be one of the letters (A, B, C or D). You may not provide any other answers or explanations. Give you some language descriptions of the visual content of the first-person perspective video. The video length is D seconds. Here is the description: ST You will answer a choice based on these descriptions, and your answer should be a letter selected from the options: Q\nThese are the option choices:\n(A): a1\n(B): a2\n(C): a3\n(D): a4\n", and then get the answer to the above prompt words to complete the processing of audio-related problems. By setting the above steps, you can effectively implement the question-and-answer method to simply and effectively implement the processing of audio-related problems.

[0049] Reference Figure 7 As shown, as an improved specific implementation method, the question processing flow module uses the visual description C and audio description T of the entire long video for question answering in the following specific manner: first input the visual description C, audio description T, question Q and option A[a1, a2, a3, a4] as well as basic video information, video duration D and video frame rate P into the prompt word for question answering, wherein the prompt word is “I will provide a video description and a question about the video. The video is 1FPS and the description is a video description of every {1 / P} frames. To answer this question, what is the minimum frame interval that needs to be confirmed first?

[0050] {frame_start_index,frame_end_index}, here is the description: \nC+T\ here is the question: Q\n please follow the output format: \n# Example 2: [30,60]\n# Example 3: [1,10]and[50,60]", then get the answer to the above prompts, and then convert the time through the frame label to complete the processing of video time related issues. By setting the above steps, you can effectively implement the question-answering method to simply and effectively process video time related issues. The visual and audio descriptions are with frame labels. Therefore, using specific prompt words can enable the large language model to have a concept of time for the video.

[0051] After applying the system of this embodiment, significant score improvements are obtained on traditional shorter video question-answering datasets such as ego4d, nextqa, intentqa, and nextgqa, and good results are obtained on Omagent's one-hour video question-answering benchmark.

[0052] To summarize, the long video question and answer processing system based on a large language model in this embodiment is based on the preprocessing of long videos, the classification of questions, the design of specific prompt words and the processing flow for different questions. On the one hand, it reduces the use of computing resources, and on the other hand, it strengthens the large language model's understanding of questions and long video content, significantly improving the performance of the large language model in long video question and answering.

[0053] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A long video question answering processing system based on a large language model, characterized by: include: A preprocessing module is used to separate long videos into visual part information and audio part information; The question classification module is used to classify the input questions into common video questions, video time-related questions, video vision-related questions, and video audio-related questions; The question processing flow module is connected to the preprocessing module and the question classification module, receives four different questions classified by the question classification module, and then enters different processing flows in combination with the video part information and the audio part information of the preprocessing module to complete the question-answering processing of the long video. The processing flow is as follows: For common video questions, the visual description C and audio description T of the entire long video, as well as the question Q and option A, will be input into the prompt words for answering the question; For vision-related questions, the visual description C is used to first obtain a summary SC based on the question Q, and then the question is answered based on the summary SC; For audio-related questions, the audio text description T is used to first obtain a summary ST based on the question Q, and then the question and answer are performed based on the summary ST; For questions related to video time, the visual description C and audio description T of the entire long video are directly used for question answering.

2. The long video question-answering processing system based on a large language model according to claim 1, characterized in that: After the preprocessing module separates the long video into visual information and audio information, it extracts frames from the video at a certain frequency P, such as fps0.5, and the extracted video frames are sent to the large model to generate a visual description of the video in a language of less than 20 words: C[c1,c2,c3,......c n ]; Among them, c is the description of the entire video, c n is the text description of the nth frame, c is Deposit it in c in the form of At the same time, the audio information is segmented at the same frequency and converted into text through the ASR model to generate speech text: T[t1,t2,t3,......t n ]; Among them, T is the description of the entire audio part, t n is the audio description of the nth segment, t is In addition, the video duration D and the video frame rate P will also be saved for subsequent processes.

3. The long video question-answering processing system based on a large language model according to claim 2, characterized in that: The specific method of the question classification module to classify the input questions is to use the large model prompt words for classification. First, the question Q and the option A [a1, a2, a3, a4] are input, and then the following large model prompt words are used: "Please judge which category the question belongs to based on the questions and options of the following video: ["Video vision related questions", "Video audio related questions", "Video time related questions", "Ordinary video questions"], \n Question: Q \n Option: (A) a1; (B) a2; (C) a3; (D) a4\nPlease read the question and options carefully and judge the type of question. Your answer only includes the type of question. The input question Q is classified according to the user's answer result.

4. The long video question-answering processing system based on a large language model according to claim 3, characterized in that: In the question processing flow module, the visual description C and audio description T of the entire long video, as well as the question Q and the option A [a1, a2, a3, a4] are input into the prompt words to answer the question, wherein the prompt words are "Please provide a single letter answer (A, B, C, D) for the following multiple choice questions, and the answer must be one of the letters (A, B, C or D). You may not provide any other answers or explanations. If you are not sure, please answer the most likely answer. Give you some language descriptions of the first-person perspective video content. The video is 1fps and the description is a letter for every two frames. The following is the description: C+T\nQuestion: Q\nOption:\n(A): a1\n(B): a2\n(C): a3\n(D): a4\n", and then obtain the answer to the above prompt words to complete the processing of ordinary video questions; The visual description C is the frame image description C[c1, c2, c3, ... c n ], the audio description T is the speech text T[t1, t2, t3, ... t n ].

5. The long video question-answering processing system based on a large language model according to claim 4, characterized in that: The question processing flow module uses the visual description C to first obtain a summary SC based on the question Q, and then conducts question answering based on the summary S·C in the following specific manner: first input the visual description C, the question Q and the options A[a1, a2, a3, a4] as well as the basic information of the video, the video duration D and the video frame rate P into the prompt words for question answering, wherein the prompt words are "give you some language descriptions of the visual content of the video. The video length is D seconds. Each sentence describes a {1 / P}s segment. These descriptions are continuous, without overlap, and accurately cover the entire video. The following is the description: C. When summarizing, please remember that your summary will be used to answer this multiple-choice question: Q". After that, the answer based on the above prompt words is obtained. Get the summary SC, and then enter the summary SC into the following prompt words "Please provide a single letter answer (A, B, C, D) to the following multiple-choice questions, and the answer must be one of the letters (A, B, C or D). You may not provide any other answers or explanations. Give you some language descriptions of the visual content of the first-person perspective video. The video length is D seconds. Here is the description: SC You will answer a choice based on these descriptions. Your answer should be a letter selected from the options: Q\nThese are the option choices:\n(A): a1\n(B): a2\n(C): a3\n(D): a4\n", then get the answer to the above prompt words to complete the processing of vision-related questions.

6. The long video question-answering processing system based on a large language model according to claim 5, characterized in that: The question processing flow module uses the audio text description T to first obtain a summary ST based on the question Q, and then conducts question-answering based on the summary ST in the following specific manner: first input the audio description T, the question Q and the options A[a1, a2, a3, a4] as well as the basic information of the video, the video duration D and the video frame rate P into the prompt words for question answering, wherein the prompt words are "give you some language descriptions of the visual content of the video. The video length is D seconds. Each sentence describes a {1 / P}s segment. These descriptions are continuous, without overlap, and accurately cover the entire video. The following is the description: T. When summarizing, please remember that your summary will be used to answer this multiple-choice question: Q". After that, the answer based on the above prompt words is obtained. Get the summary SC, and then enter the summary SC into the following prompt words "Please provide a single letter answer (A, B, C, D) to the following multiple-choice questions, and the answer must be one of the letters (A, B, C or D). You may not provide any other answers or explanations. Give you some language descriptions of the visual content of the first-person perspective video. The video length is D seconds. Here are the descriptions: ST You will answer a choice based on these descriptions. Your answer should be a letter selected from the options: Q\nThese are the option choices:\n(A): a1\n(B): a2\n(C): a3\n(D): a4\n", then get the answer to the above prompt words to complete the processing of audio-related questions.

7. The long video question-answering processing system based on a large language model according to claim 6, characterized in that: The question processing flow module uses the visual description C and audio description T of the entire long video to conduct question and answer in a specific manner: first input the visual description C, audio description T, question Q and option A[a1, a2, a3, a4] as well as the basic information of the video, the video duration D and the video frame extraction frequency P into the prompt word for question answering, wherein the prompt word is "I will provide a video description and a question about the video. The video is 1FPS, and the description is a video description of every {1 / P} frames. To answer this question, what is the minimum frame interval that needs to be confirmed first? {frame_start_index, frame_end_index}, here is the description: \nC+T\here is the question: Q\nPlease follow the output format: \n#Example 2: [30,60]\n#Example 3: [1,10]and[50,60]", then obtain the answer to the above prompt word, and then convert the time through the frame label to complete the processing of video time related problems.

Citation Information

Cited By

  • Voice question answering method and system based on retrieval enhancement generation, medium and product

    CN120561251A

  • Voice question answering method, system, medium and product based on retrieval-enhanced generation

    CN120561251B