Long video question and answer method based on agent reasoning and adaptive fragment extension mechanism

Abstract

The long video question and answer method based on the agent reasoning and self-adaptive fragment extension mechanism comprises the following steps: acquiring a video and a target question; video frames are extracted and coded into video frame features; extracting text phrases of the target question and encoding the text phrases into text features; generating a key frame set according to the similarity of the video frame features and the text features; performing fixed-time-length expansion on each key frame, and then performing merging processing on the segments with overlapped time intervals to obtain a video segment set; using a multi-modal large model to obtain semantic content description of the video clip and a relevancy score of the video clip and the target problem; storing the information of each video clip including the relevancy score into a storage space; and judging whether the information of the video clip in the storage space supports question answering or not by using the intelligent agent, and performing reasoning judgment. According to the method, on the basis of introducing an intelligent agent reasoning framework, a problem-oriented storage space enhancement mechanism is constructed, so that the robustness and question and answer accuracy in a long video scene are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of multimodal large models, intelligent agents, and long video understanding, and particularly to a long video question answering method based on intelligent agent reasoning and adaptive segment expansion mechanism. Background Technology

[0002] Compared to static images, video can more realistically and comprehensively reflect the dynamic changes of the real world and is more closely related to human daily life and cognitive methods. When perceiving and understanding the external environment, humans primarily rely on continuously changing visual input, rather than isolated static moments. With the rapid development of the internet and multimedia technologies, various video platforms have emerged, and the daily upload volume of video content worldwide has reached millions of hours. This phenomenon has not only greatly enriched information carriers but also placed higher demands on automated video analysis and understanding technologies, making video understanding an important direction in the fields of computer vision and multimodal intelligence.

[0003] The goal of video understanding is to endow computers with the ability to analyze and understand video content in a human-like manner, encompassing a variety of tasks from low-level object recognition and target tracking to mid-to-high-level action detection, event understanding, and complex logical and causal reasoning. However, compared to images, video data has a significantly expanded temporal dimension, containing not only a large amount of redundant information but also complex temporal dependencies and cross-frame semantic associations. These characteristics present numerous challenges for models in modeling long-term temporal relationships, capturing key details, and balancing computational efficiency with understanding capabilities. In recent years, multimodal large models have made significant progress in video understanding tasks, demonstrating superior performance on multiple benchmark datasets by jointly modeling visual, linguistic, and other modalities. However, limited by the model's context window size and computational resource constraints, these models still face significant bottlenecks when processing long videos. To alleviate this problem, existing research typically employs a top-down, stepwise refinement strategy: first, an initial answer is generated based on the complete video and scored; then, the video is divided into several shorter segments, and each segment attempts to answer the same question and is evaluated; based on the scoring results, the most relevant segment is selected for further refinement, and this process is iterated until the highest-scoring answer is finally used as the model's output.

[0004] While the methods described above alleviate the computational burden of long video processing to some extent, they still have significant limitations. Because the initial stage can only perform coarse-grained analysis of the video, the model often struggles to fully consider key local details. If an incorrect path is chosen early on, the subsequent search process may deviate from the segments containing the truly crucial information, leading to the omission of important content and ultimately affecting the accuracy and robustness of video understanding. This error propagation problem is particularly prominent in tasks involving fine-grained action recognition or complex event reasoning. Summary of the Invention

[0005] To address the common technical problems in existing long-video question-answering technologies, such as low efficiency in retrieving key information, redundant reasoning processes, insufficient adaptability to changes in video timescale, and limited question-answering accuracy, this invention proposes a long-video question-answering method based on agent-based reasoning and an adaptive segment expansion mechanism. This method, by introducing an agent-based reasoning framework, constructs a question-oriented storage space enhancement mechanism to achieve efficient organization and reuse of key information in multi-round reasoning. Furthermore, it combines an adaptive video segment expansion strategy to dynamically adjust the video analysis scope, thereby significantly improving the system's robustness and question-answering accuracy in complex long-video scenarios while maintaining reasoning efficiency.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] The first aspect is a long video question answering method based on agent reasoning and adaptive segment expansion mechanisms, which includes the following steps:

[0008] S1. Acquire video and target question; extract video frames and encode them into video frame features; extract text phrases from the target question and encode them into text features;

[0009] S2. Generate a set of keyframes based on the similarity between video frame features and text features;

[0010] S3. After extending each keyframe by a fixed duration, merge segments with overlapping time intervals to obtain a set of video segments;

[0011] S4. Use a multimodal large model to obtain semantic content descriptions of video segments and relevance scores between video segments and the target question; store the relevance score information for each video segment in the storage space;

[0012] S5. Use an intelligent agent to determine whether the information in the video clips in the storage space supports the answer to the question and make inferences.

[0013] Preferably, S2 includes selecting several video frames with the highest similarity to form a candidate keyframe set; setting the final keyframe set to initially contain only the frame with the highest similarity in the candidate keyframe set; then comparing the similarity of the remaining frames in the candidate keyframe set with the existing frames in the keyframe set in turn; if the similarity is higher than a preset threshold, it is determined to be a redundant frame and discarded, and finally the deduplicated keyframe set is obtained.

[0014] Preferably, in S5, the following process is executed repeatedly under the control of the intelligent agent until the preset maximum number of cycles is reached, and then the final answer to the question is output: if the information of the video clips already in the storage space can support the answer to the question, the final answer of the reasoning judgment is directly output; otherwise, the video clip is selected to perform an expansion operation and the storage space is cleared; the expanded video clip is re-input into the multimodal large model to obtain semantic content description, generate the relevance score between the expanded video clip and the target question and update it to the storage space.

[0015] Preferably, the information of the video segments in the storage space includes: a unique number used to identify the video segment, the time interval corresponding to the video segment, a semantic content description generated based on the target question, and a relevance score between the video segment and the target question.

[0016] As a preferred method, the video segment expansion method in S5 includes:

[0017] The video clips are processed by time slicing to generate a start time slice and an end time slice. The start time slice and the end time slice are input into a multimodal large model to determine their semantic relevance to the text phrases and obtain the corresponding relevance labels. The expansion direction of the video clips is adaptively determined based on the relevance labels.

[0018] As a preferred embodiment, S5 includes:

[0019] When the starting time slice is related to a text phrase, it expands towards the front of the video; when the ending time slice is related to a text phrase, it expands towards the back of the video; when both are related or unrelated, it expands in both directions simultaneously, thereby achieving adaptive adjustment of the video semantic boundaries.

[0020] Secondly, long-video question-answering systems based on agent reasoning and adaptive segment expansion mechanisms include:

[0021] The acquisition and feature extraction module is used to acquire video and target questions, extract video frames and encode them into video frame features; and extract text phrases from the target questions and encode them into text features.

[0022] The keyframe extraction module is used to generate a set of keyframes based on the similarity between video frame features and text features;

[0023] The extended merging module is used to extend each keyframe by a fixed duration and then merge segments with overlapping time intervals to obtain a set of video segments.

[0024] The information storage module uses a multimodal large model to obtain semantic content descriptions of video segments and relevance scores between video segments and the target question; it stores information including the relevance score for each video segment into the storage space.

[0025] The loop judgment module uses an intelligent agent to determine whether the information in the video clips in the storage space supports the answer to the question and performs reasoning to generate the final answer to the question.

[0026] The long video question answering system based on agent reasoning and adaptive segment expansion mechanism is used to implement the long video question answering method based on agent reasoning and adaptive segment expansion mechanism as described in the first aspect.

[0027] Thirdly, an electronic device including a processor and a memory;

[0028] The processor is connected to the memory;

[0029] The memory is used to store executable program code;

[0030] The processor runs a program corresponding to the executable program code stored in the memory, which is used to execute the long video question answering method and its steps based on agent reasoning and adaptive fragment expansion mechanism as described in the first aspect.

[0031] Fourthly, a computer-readable storage medium having a computer program stored thereon, the computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the long video question-answering method and steps based on agent reasoning and adaptive fragment expansion mechanism as described in the first aspect.

[0032] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0033] 1. Unlike the traditional top-down, layer-by-layer video segment refinement method, this method adopts a bottom-up video segment expansion mechanism. Starting from the key frames related to the question, it gradually expands the analysis scope, thereby more accurately covering the key semantic content corresponding to the question and significantly improving the accuracy of the question-and-answer results.

[0034] 2. Unlike agent-based reasoning schemes that rely on video subtitles or single text information, this approach introduces a storage-based agent reasoning mechanism to structure and dynamically update multimodal semantic information. This not only improves the relevance and information density between text information and the target question, but also effectively reduces the length of text sequences that the agent needs to process during reasoning. As a result, it significantly improves overall reasoning efficiency while ensuring reasoning accuracy. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the method flow of Embodiment 1 of the present invention;

[0036] Figure 2 This is a visual schematic diagram of Embodiment 2 of the present invention;

[0037] Figure 3 This is a diagram illustrating the effect of Embodiment 2 of the present invention;

[0038] Figure 4 This is a diagram illustrating the effect of Embodiment 2 of the present invention. Detailed Implementation

[0039] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0040] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0041] This invention proposes a long video question-answering method based on bottom-up video segment expansion. Addressing the issues of large event spans and sparse distribution of key content in long videos, this invention does not employ the traditional top-down uniform video segmentation strategy. Instead, it starts with local information related to the question and gradually expands adaptively forward and backward along the timeline until it covers a complete video segment capable of supporting the question's answer.

[0042] Specifically, this invention first acquires the video to be analyzed and the target question, and extracts text phrases that can represent key information from the target question; then, it performs similarity matching and filtering on video frames based on the text phrases to obtain several key frames; next, it generates video segments centered on the key frames and performs merging processing; the obtained video segments are input into a multimodal large model to generate question-oriented segment semantic content and its relevance score to the question, and the above information is uniformly stored in the storage space; the agent performs reasoning and judgment based on the multimodal information in the storage space, deciding whether to output the final answer or select some video segments for further expansion; if expansion is selected, the time range of the video segments is adaptively adjusted by analyzing the correlation between the time slices before and after the video segments and the text phrases, and the storage space is updated; the above process is executed cyclically under the control of the agent until the answer conditions are met or the preset maximum number of reasoning rounds is reached.

[0043] Example 1:

[0044] like Figure 1 As shown in the figure, this embodiment provides a long video question answering method based on agent reasoning and adaptive segment expansion mechanism, employing a bottom-up video content search and analysis strategy. First, after determining the video to be analyzed and the target question, the video is frame-sampled, and the video frames are encoded using a visual coding model to obtain and store video frame features. Then, a large language model performs semantic analysis on the target question, generating text phrases containing key question information, which are then encoded as text features. By calculating the similarity between text features and video frame features, several keyframes are indexed, and highly similar keyframes are deduplicated. The retained keyframes are then expanded into video segments, and video segments with overlapping time intervals are merged. Subsequently, the video segments and the target question are input into a multimodal large model to generate a question-oriented semantic description of the video segments and corresponding relevance scores, and this information is stored in the storage space. The agent performs reasoning and judgment based on the information in the storage space. If it believes that the current information is sufficient to support the question answer, it directly outputs the answer; if it believes that the information is insufficient, it selects some video segments for expansion. During the expansion process, semantic relevance is judged for the time slices before and after the video segments, and the time range of the video segments is adaptively adjusted accordingly. The updated fragment is then input into the multimodal large model for analysis, and the storage space is updated. The next round of inference then begins until the termination condition is met.

[0045] A long video question answering method based on agent reasoning and adaptive segment expansion mechanism includes the following steps:

[0046] S1. Acquire video and target question; extract video frames and encode them into video frame features; extract text phrases from the target question and encode them into text features;

[0047] S11. Determine the video to be analyzed. and the target problem The video is a long video containing the information needed to answer the question, and the target question is a question posed by the user in natural language. For example, the video could be a movie or animated feature film, and the target question could be "What did the protagonist do after going out?"

[0048] S12. After determining the video and the target problem, perform frame-level sampling on the video, for example, sampling video frames at a frequency of one frame per second, and then using an encoder. Encode the video frames to obtain video frame features:

[0049]

[0050] in, Represents video frame features, Indicates the first Frame characteristics Indicates the number of video frames. Indicates the first frame.

[0051] S13. At the same time, the large language model addresses the target problem. Perform semantic analysis to generate text phrases containing key information about the problem. and through the encoder Encode it:

[0052]

[0053]

[0054] in, Representing a large language model, This represents the text features corresponding to a text phrase.

[0055] S2. Generate a set of keyframes based on the similarity between video frame features and text features;

[0056] Calculate the similarity between text features and video frame features:

[0057]

[0058] in, The similarity vector has a dimension equal to the number of video frames. The set of candidate keyframes is formed by selecting the video frames with the highest similarity. .

[0059] To avoid redundancy in the indexing results, the final keyframe set is set as follows: Initially, it only contained The frame with the highest similarity; then... The remaining frames are sequentially connected with The existing frames are compared for similarity. If the similarity is higher than a preset threshold, the frame is identified as redundant and discarded, resulting in a deduplicated set of keyframes. .

[0060] S3. After extending each keyframe by a fixed duration, merge segments with overlapping time intervals to obtain a set of video segments:

[0061] Centered on a keyframe, each keyframe is expanded into an extended video clip of fixed duration. For example, expanding 8 seconds forward and backward from the keyframe generates a 16-second video clip, ensuring that the clip's time range does not exceed the video boundaries.

[0062] Subsequently, the extended video segments with overlapping time intervals were merged to obtain a set of video segments:

[0063]

[0064]

[0065]

[0066] in, Indicates by the first Video clips obtained through frame expansion Indicates the number of video clips. Indicates an extended set of video clips. This represents the merged set of video clips. It is a function that expands video frames into video segments. It is a function for merging video clips.

[0067] S4. Obtain the semantic content description of the video segments and the relevance score between the video segments and the target question; store the information of each video segment in the storage space;

[0068] Collection of video clips With the target problem Input a large multimodal model and generate semantic content descriptions for video clips. and the relevance score to the question :

[0069]

[0070]

[0071] in, This indicates the content of a video clip. This represents a relevance score, ranging from 1.0 to 10.0 with one decimal place. A higher score indicates a higher relevance to the question. It is a multimodal large model.

[0072] Store the information of each video segment in the storage space:

[0073]

[0074]

[0075] in, Indicates the first Information for each video segment, where id represents a unique identifier for the video segment. This represents the interval from the start to the end of a video segment. It refers to storage space.

[0076] S5. Use an intelligent agent to make inferences and judgments based on information from video clips in the storage space;

[0077] If the current information supports the answer to the question, output the final answer and terminate the process;

[0078] If the information is insufficient, clear the storage space and select one or more video clips for expansion. The expansion method is as follows: first, perform time slicing on the video clips, extracting the first 8 seconds of each video clip's initial phase. And the last 8 seconds of the final phase The processing procedure is expressed as follows:

[0079]

[0080] in, This represents a function for extracting video clips.

[0081] Then, slice the time. and Each input is a multimodal large model, and the semantic relevance between it and the text phrase is determined to obtain the corresponding relevance label:

[0082]

[0083]

[0084] in, and The result of the binary judgment is defined as follows:

[0085]

[0086]

[0087] According to correlation markers and The value of the result is used to adaptively determine the expansion direction of the video segment:

[0088] when and When it is determined that the target event may have occurred within a time range prior to the current video segment, the video segment is expanded forward along the time axis.

[0089] when and When it is determined that the target event may occur within a time range after the current video segment, the video segment is extended backward along the time axis;

[0090] when and If the target event is determined to be located within the current video segment but not yet fully covered, the video segment is simultaneously expanded in both forward and backward directions along the timeline.

[0091] when and If it is determined that no video event related to the target problem has been located, the video clip is expanded simultaneously in both directions along the timeline to broaden the search scope.

[0092] In any of the above-mentioned extension scenarios, the duration of the extended video segment is set to twice the duration of the original video segment. After the extension is completed, the segment semantics are generated again, the storage space is updated, and the agent reasoning is performed repeatedly until the maximum number of reasoning rounds is reached or the agent determines that it can output an answer.

[0093] Example 2:

[0094] like Figure 2 As shown, to illustrate the long video question-answering method based on sophisticated reasoning between an agent and video segments provided by this invention, a specific example is given below. In this example, the video to be analyzed is an animated feature film with a total duration of 1820 seconds, and the target question is "How many mice ran across the rooftops on the street?". The actual video time interval containing relevant information is from 226 to 227 seconds. The system first generates the text phrase "mice ran across the rooftops on the street" based on the target question and indexes multiple keyframes; after expansion and merging, multiple initial video segments are generated, and their relevance scores to the question are calculated. After multiple rounds of reasoning and segment expansion, the agent gradually expands the analysis scope from the initial local segments to cover the time interval of 226 to 227 seconds, and finally successfully locates the video content containing key information and outputs the correct answer.

[0095] To test the performance of the method of this invention, qualitative and quantitative experiments were conducted on multiple publicly available test sets. The qualitative experimental results are as follows: Figure 3 and Figure 4 As shown. In Figure 3 In the experimental example shown, the video content required to answer the target question is located between the 15th and 19th seconds of the original video. In the initial stage, the system indexes three keyframes based on the similarity between text phrases and video frame features, and generates corresponding video clips accordingly. In the first round of video clips, the clip with sequence number 0 has a time range of 47 to 63 seconds. Its content is described as showing a person walking in a snowy village, accompanied by textual information such as "poor" and "huge," and its relevance score to the target question is 2.5. The other two video clips also contain some textual content, but neither contains time or year information directly related to the target question.

[0096] In the first round of inference, the agent comprehensively analyzes the semantic content and relevance scores of each video segment in the storage space. It determines that none of the current video segments are sufficient to support the answer to the question. Considering the temporal position, it selects the video segment with sequence number 0, which is closer to the beginning of the video, for expansion. After multiple rounds of adaptive expansion, in the fourth round of inference, the time range of the video segment is expanded to 0 to 119 seconds, successfully covering the 15 to 19 seconds video time range required by the target question. Furthermore, the agent detects the subtitle information "In 1633, Japan closed its borders" within this segment, thus enabling it to correctly answer the target question.

[0097] As can be seen from the above process, the method of the present invention effectively reduces the original 3296-second long video that needed to be analyzed as a whole to a key video segment of 119 seconds. While significantly reducing the computational overhead, it can still accurately locate and extract key information related to the problem.

[0098] exist Figure 4In another experimental example, the video content corresponding to the target question is located in the time interval from 1181 seconds to 1190 seconds of the original video. During the initial indexing phase, the system obtains two candidate video segments with time intervals of 624 seconds to 640 seconds and 1190 seconds to 1206 seconds, respectively. Since the semantic content of neither of these video segments contains the relevant information about "the long-nosed man receiving payment" in the target question, the agent chooses to expand both video segments simultaneously during the inference phase. The expanded video segments have time intervals of 616 seconds to 648 seconds and 1182 seconds to 1214 seconds, with the latter successfully covering the video time interval required by the target question. In this video segment, the semantic content describes a character wearing a hat taking a gold coin from a bag and handing it to a long-nosed character. Based on this, the agent correctly answers the target question: the long-nosed man received a gold coin.

[0099] In terms of quantitative analysis, to comprehensively evaluate the performance of the proposed method in long video question answering tasks, comparative experiments were conducted on several publicly available long video understanding benchmark sets, comparing the proposed method with recent multimodal large model methods, other video question answering methods, and existing video reasoning methods. The corresponding quantitative comparative experimental results are shown in Tables 2, 3, and 4.

[0100] Table 2 presents the performance comparison results of the proposed method and the multimodal large model method on long video understanding benchmark sets. As shown in Table 2, the proposed method achieves the best experimental results on several typical long video understanding benchmark sets such as LVBench and EgoSchema. Furthermore, compared with the multimodal large model Qwen3-VL-235B used in this invention, the proposed method outperforms the baseline results of directly using the multimodal large model on all test sets. The above experimental results demonstrate that by introducing an agent reasoning mechanism and an adaptive video segment expansion strategy, this invention can more fully mine question-related information in long videos without increasing the scale of video frame analysis, thereby significantly improving overall question-answering performance and verifying the effectiveness of the overall framework design of this invention.

[0101] Table 3 shows the performance comparison results between the method of this invention and other existing video question answering methods on the same long video understanding benchmark set. The experimental results show that the method of this invention achieves the best performance on all test sets, indicating that by constructing a storage-based agent reasoning mechanism and combining it with a question-oriented video segment expansion strategy, this invention can effectively improve the reasoning accuracy and stability of the agent in long video scenarios.

[0102] Table 4 shows the performance comparison results of different video segment inference strategies on the long video understanding benchmark set. The experiment for the top-down video inference strategy was set as follows: A complete video and a target question were used as input. The evaluation model performed an overall analysis of the complete video and provided a relevance score, storing the results in storage. Subsequently, the agent determined whether further video segmentation was needed based on the information in storage. If so, the complete video was evenly divided into multiple video segments, and the above analysis and inference process was repeated until the agent output the final answer or reached the maximum inference steps.

[0103] Compared to the aforementioned top-down video segment reasoning strategy, the bottom-up video segment expansion strategy adopted in this invention starts with question-related keyframes and progressively and adaptively expands the video analysis scope. Experimental results show that, under the same test conditions, the method of this invention outperforms the top-down video reasoning strategy on all test sets, further demonstrating the effectiveness and superiority of the proposed bottom-up video segment expansion mechanism in long video question answering tasks.

[0104] Table 2: Performance comparison with multimodal large models on long video benchmark sets (metric: accuracy)

[0105]

[0106] Table 3: Performance comparison with other video question answering methods on long video benchmark sets (indicator is accuracy).

[0107]

[0108] Table 4: Performance Comparison of Two Video Segment Inference Strategies on Long Video Benchmark Sets (Indicator: Accuracy)

[0109]

[0110] Example 3:

[0111] A long-video question-answering system based on agent reasoning and adaptive segment expansion mechanisms includes:

[0112] The acquisition and feature extraction module is used to acquire video and target questions, extract video frames and encode them into video frame features; and extract text phrases from the target questions and encode them into text features.

[0113] The keyframe extraction module is used to generate a set of keyframes based on the similarity between video frame features and text features;

[0114] The extended merging module is used to extend each keyframe by a fixed duration and then merge segments with overlapping time intervals to obtain a set of video segments.

[0115] The information storage module uses a multimodal large model to obtain semantic content descriptions of video segments and relevance scores between video segments and the target question; it stores information including the relevance score for each video segment into the storage space.

[0116] The loop judgment module uses an intelligent agent to determine whether the information in the video clips in the storage space supports the answer to the question and makes inferences to generate the final answer to the question.

Claims

1. A long video question answering method based on agent reasoning and adaptive segment expansion mechanism, characterized in that, Includes the following steps: S1. Acquire video and target question; extract video frames and encode them into video frame features; extract text phrases from the target question and encode them into text features; S2. Generate a set of keyframes based on the similarity between video frame features and text features; S3. After extending each keyframe by a fixed duration, merge segments with overlapping time intervals to obtain a set of video segments; S4. Use a multimodal large model to obtain semantic content descriptions of video segments and relevance scores between video segments and the target question; store the relevance score information for each video segment in the storage space; S5. Use an intelligent agent to determine whether the information in the video clips in the storage space supports the answer to the question and make inferences.

2. The long video question answering method based on agent reasoning and adaptive segment expansion mechanism according to claim 1, characterized in that, S2 includes selecting several video frames with the highest similarity to form a candidate keyframe set; setting the final keyframe set to initially contain only the frame with the highest similarity in the candidate keyframe set; then comparing the similarity of the remaining frames in the candidate keyframe set with the existing frames in the keyframe set in turn, and if the similarity is higher than a preset threshold, it is determined to be a redundant frame and discarded, and finally the deduplicated keyframe set is obtained.

3. The long video question answering method based on agent reasoning and adaptive segment expansion mechanism according to claim 1, characterized in that, In S5, the following process is executed repeatedly under the control of the intelligent agent until the preset maximum number of cycles is reached, and then the final answer to the question is output: if the information of the video clips already in the storage space can support the answer to the question, the final answer of the reasoning judgment is directly output; otherwise, the video clip is selected to perform the expansion operation and the storage space is cleared. The expanded video clips are re-input into the multimodal large model to obtain semantic content descriptions, generate relevance scores between the expanded video clips and the target question, and update the storage space.

4. The long video question answering method based on agent reasoning and adaptive segment expansion mechanism according to claim 3, characterized in that, The information of video segments in the storage space includes: a unique number used to identify the video segment, the time interval corresponding to the video segment, a semantic content description generated based on the target question, and a relevance score between the video segment and the target question.

5. The long video question answering method based on agent reasoning and adaptive segment expansion mechanism according to claim 3, characterized in that, The methods for expanding video clips in S5 include: The video clips are processed by time slicing to generate a start time slice and an end time slice. The start time slice and the end time slice are input into a multimodal large model to determine their semantic relevance to the text phrases and obtain the corresponding relevance labels. The expansion direction of the video clips is adaptively determined based on the relevance labels.

6. The long video question answering method based on agent reasoning and adaptive segment expansion mechanism according to claim 5, characterized in that, S5 include: When the starting time slice is related to a text phrase, it expands towards the front of the video; when the ending time slice is related to a text phrase, it expands towards the back of the video; when both are related or unrelated, it expands in both directions simultaneously, thereby achieving adaptive adjustment of the video semantic boundaries.

7. A long video question-answering system based on agent reasoning and adaptive segment expansion mechanism, characterized in that, include: The acquisition and feature extraction module is used to acquire video and target questions, extract video frames and encode them into video frame features; and extract text phrases from the target questions and encode them into text features. The keyframe extraction module is used to generate a set of keyframes based on the similarity between video frame features and text features; The extended merging module is used to extend each keyframe by a fixed duration and then merge segments with overlapping time intervals to obtain a set of video segments. The information storage module uses a multimodal large model to obtain semantic content descriptions of video segments and relevance scores between video segments and the target question; it stores information including the relevance score for each video segment into the storage space. The loop judgment module uses an intelligent agent to determine whether the information in the video clips in the storage space supports the answer to the question and performs reasoning to generate the final answer to the question. The long video question answering system based on agent reasoning and adaptive segment expansion mechanism is used to implement the long video question answering method based on agent reasoning and adaptive segment expansion mechanism as described in claim 1.

8. An electronic device, characterized in that, Including the processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory, which is used to execute the long video question answering method and its steps based on agent reasoning and adaptive fragment expansion mechanism as described in claim 1.

9. A computer-readable storage medium, characterized in that, It stores a computer program thereon, and the computer-readable storage medium stores instructions that, when the instructions are executed on a computer or processor, cause the computer or processor to perform the long video question answering method and its steps based on agent reasoning and adaptive fragment expansion mechanism as described in claim 1.

Images