Video interleaving reasoning method and system based on trainable end-to-end agent system

Abstract

The invention provides a video interleaving reasoning method and system based on a trainable end-to-end agent system. The method comprises the following steps: S1, constructing a reasoning agent system; the reasoning agent system comprises a core reasoning model and a tool library; wherein the core reasoning model is used for realizing logical reasoning according to visual information and text information based on Qwen2.5-VL-7B to obtain a text reasoning process; the tool library is used for acquiring video spatio-temporal information by using multiple visual tools in the tool library for a target video; s2, training a core reasoning model in the reasoning agent system to obtain a trained reasoning agent system; and S3, obtaining a target video and a target question, and obtaining reasoning process information and an answer corresponding to the final question through video spatio-temporal information-text reasoning process interlaced reasoning by using the trained reasoning agent system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video reasoning, in particular to a video interleaved reasoning method based on a trainable end-to-end agent system. BACKGROUND

[0002] With the development of multi-modal technology, video understanding and question answering tasks have become a research hotspot in the field of artificial intelligence. The core requirement is to enable the model to accurately associate video visual information with text questions, implement efficient reasoning, and output reliable answers. However, existing video understanding technologies still have many key bottlenecks: on the one hand, conventional video large models directly output answers, lack intermediate reasoning processes, making it difficult to trace errors, and are difficult to handle complex problems that require fine-grained spatio-temporal information; on the other hand, traditional text thought chain large models can output reasoning processes, but cannot effectively focus on and mine key visual information in videos, often leading to reasoning errors due to missing core visual details.

[0003] At the same time, existing technologies have obvious shortcomings in processing video spatio-temporal information. Most multi-modal models rely on text prior knowledge rather than real video visual perception, and have insufficient sustained cognitive ability for long videos, accuracy of target positioning and tracking, and difficulty in flexibly combining different visual analysis tools to adapt to diverse problem scenarios. In addition, the training methods of existing models are difficult to balance the normativity of reasoning mode and the generalization ability of open world problems, further limiting their application effect in actual complex video reasoning tasks. Therefore, there is an urgent need for a reasoning scheme that can realize deep interlaced fusion of visual information and text reasoning, has flexible tool calling ability, and can accurately mine video spatio-temporal information, to break through the limitations of existing technologies.

[0004] Patent document CN119202149A (application number: 202410943597.7) discloses a long video question answering method based on large model agent multi-round reasoning. The initial key frame sequence is determined according to the question and the options, and the confidence score of the options is generated. The confidence score is used to prompt the large language model to still get a relatively reliable answer under insufficient input information. The positioned key frames are in the form of grid splicing, and are input into the large model agent with the text context to predict the answer. The large model agent evaluates based on the current provided information and the answer prediction process to decide whether to perform the next iteration to supplement the missing visual information. The adaptive multi-round reasoning question answering considers the different video frame numbers required for different categories of questions, effectively improving the accuracy and quality of video question answering. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a video interleaved reasoning method and system based on a trainable end-to-end agent system.

[0006] According to the application, a video interleaved reasoning method based on a trainable end-to-end intelligent agent system is provided, comprising: Step S1: constructing a reasoning intelligent agent system; the reasoning intelligent agent system comprises a core reasoning model and a tool library; wherein the core reasoning model is a text reasoning process obtained by implementing logical reasoning based on Qwen2.5-VL-7B according to visual information and text information; the tool library is used to obtain video space-time information by using a plurality of visual tools in the tool library for a target video; Step S2: training the core reasoning model in the reasoning intelligent agent system to obtain a trained reasoning intelligent agent system; Step S3: obtaining a target video and a target question, and using the trained reasoning intelligent agent system to obtain reasoning process information and an answer corresponding to the final question through video space-time information-text reasoning process interleaved reasoning.

[0007] Preferably, the plurality of visual tools comprises a time positioning tool, a time counting tool, a time cutting tool, a key frame selection tool, a space positioning tool, and a space tracking tool; The time positioning tool is used to position the time start point and end point of an event in a video according to the input event description; The time counting tool is used to count the number of times an event occurs in a video according to the input event description; The time cutting tool is used to cut out the corresponding segment from the video according to the input time start point and end point; The key frame selection tool is used to select a key frame that best matches the description from the video according to the input description when a static scene needs to be observed; The space positioning tool is used to position a target object in a key frame, which will give a positioning box of the corresponding object in the frame and visualize it on the image; The space tracking tool is used to track a target object in a video in space, and give a corresponding coordinate positioning box of the target object in each frame, and always maintain the consistency of the target.

[0008] Preferably, the step S2 comprises: the training of the core reasoning model comprises a supervised fine-tuning training phase and a reinforcement learning training phase; The supervised fine-tuning training phase comprises training the core reasoning model by using historical tool-enhanced visual-text interleaved thinking chain data, so that the trained core reasoning model can obtain video space-time information through multiple rounds of interleaved reasoning based on the called tools; The reinforcement learning training stage comprises: training the core reasoning model by using historical question and answer data after the supervised fine-tuning training stage, so that the trained core reasoning model can combine different tools to obtain answers corresponding to different questions.

[0009] Preferably, the tool-enhanced visual-text interleaved thinking chain data comprises: pre-constructed training data comprising a video, a question, a visual-text interleaved thinking chain, and an answer, wherein the visual-text interleaved thinking chain comprises a complete thinking and tool calling process.

[0010] Preferably, the step S3 comprises: Step S3.1: inputting the target video and the target question into the trained reasoning agent system, and generating a first round of text reasoning process according to the target video and the target question by using the core reasoning model in the trained reasoning agent system; Step S3.2: obtaining a tool calling request in the text reasoning process based on the currently generated text reasoning process; selecting a corresponding tool in the tool library based on the tool calling request, and calling the tool to process the target video to obtain corresponding video spatiotemporal information; Step S3.3: splicing the obtained video spatiotemporal information and the current round of text reasoning process to form a visual-text interleaved thinking process; Step S3.4: inputting the current visual-text interleaved thinking process, the target video, and the target question into the trained reasoning agent system, and generating a second round of text reasoning process by using the core reasoning model in the trained reasoning agent system; repeating steps S3.2 to S3.4 until the core reasoning model determines that the current text reasoning process meets a preset requirement, outputting each round of text reasoning process, and taking the final round of text reasoning process as an answer corresponding to the final question; Wherein, the video reasoning process is divided into corresponding rounds of text reasoning processes according to the number of tools to be called in the target question.

[0011] According to the video interleaved reasoning system based on the trainable end-to-end agent system provided by the application, the reasoning agent system is constructed, the core reasoning model in the reasoning agent system is trained, and the tool library is used to obtain video spatiotemporal information by using a plurality of visual tools in the tool library. Module M1: constructing a reasoning agent system; the reasoning agent system comprises: a core reasoning model and a tool library; wherein the core reasoning model is based on Qwen2.5-VL-7B to realize logical reasoning to obtain a text reasoning process according to visual information and text information; and the tool library is used to obtain video spatiotemporal information by using a plurality of visual tools in the tool library for a target video; Module M2: training the core reasoning model in the reasoning agent system to obtain a trained reasoning agent system; Module M3: Obtain the target video and the target question, and use the trained reasoning intelligent agent system to obtain the reasoning process information and the answer corresponding to the final question through the video spatio-temporal information-text reasoning process interleaved reasoning.

[0012] Preferably, the plurality of visual tools includes a time positioning tool, a time counting tool, a time cutting tool, a key frame selection tool, a space positioning tool, and a space tracking tool. The time positioning tool is configured to position a time start point and a time end point of an event in a video according to an input event description. The time counting tool is configured to count the number of times an event occurs in a video according to an input event description. The time cutting tool is configured to cut a corresponding segment from a video according to an input time start point and a time end point. The key frame selection tool is configured to select a key frame that best matches an input description from a video when a static scene needs to be observed. The space positioning tool is configured to position a target object in a key frame, and provide a positioning box of the corresponding object in the frame and visualize the positioning box on the image. The space tracking tool is configured to track a target object in a video, and provide a corresponding coordinate positioning box of the target object in each frame while maintaining the consistency of the target.

[0013] Preferably, the module M2 includes a supervised fine-tuning training phase and a reinforcement learning training phase for training the core reasoning model. The supervised fine-tuning training phase includes training the core reasoning model using historical tool-enhanced visual-text interleaved thinking chain data, so that the trained core reasoning model can obtain video spatio-temporal information based on the called tools through multiple rounds of interleaved reasoning. The reinforcement learning training phase includes training the core reasoning model using historical question and answer data after the supervised fine-tuning training phase, so that the trained core reasoning model can combine different tools to obtain answers corresponding to different questions.

[0014] Preferably, the tool-enhanced visual-text interleaved thinking chain data includes pre-constructed training data containing videos, questions, visual-text interleaved thinking chains, and answers, and the visual-text interleaved thinking chain contains complete thinking and tool calling process.

[0015] Preferably, the module M3 includes: Module M3.1: input the target video and the target question into the trained reasoning agent system, and generate a first round of text reasoning process according to the target video and the target question by using a core reasoning model in the trained reasoning agent system; Module M3.2: based on the currently generated text reasoning process, obtain a tool calling request in the text reasoning process; based on the tool calling request, select a corresponding tool in a tool library, and call the tool to process the target video to obtain corresponding video spatio-temporal information; Module M3.3: splice the obtained video spatio-temporal information and the current round of text reasoning process to form a visual-text interleaved thinking process; Module M3.4: input the current visual-text interleaved thinking process, the target video and the target question into the trained reasoning agent system, and generate a second round of text reasoning process by using the core reasoning model in the trained reasoning agent system, repeatedly trigger the module M3.2 to the module M3.4, until the core reasoning model judges that the current text reasoning process meets the preset requirement, output each round of text reasoning process, and take the final round of text reasoning process as an answer corresponding to the final question; Wherein, the video reasoning process is divided into corresponding rounds of text reasoning process according to the number of tools to be called in the target question.

[0016] Compared with the prior art, the present application has the following beneficial effects: 1、The present application solves the problem that a single reasoning model cannot well perceive spatio-temporal information in a video by using a reasoning agent system to call a tool library to complete a perception task, and achieves accurate perception to ensure understanding. 2、The present application solves the problem that a pure text thinking chain cannot perfectly convert other modal information and cannot gradually obtain visual information by using a visual-text interleaved reasoning method, and achieves the purpose of improving reasoning ability. 3、The present application solves the problem that a multi-agent system cannot solve the open world problem and needs sample examples as a reference by using an end-to-end training system architecture, and achieves the effect of stronger general ability. BRIEF DESCRIPTION OF DRAWINGS

[0017] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 The flow chart of the video interleaved reasoning method based on the trainable end-to-end agent system. DETAILED DESCRIPTION

[0018] The application will be described in detail below with specific examples. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the application. These are within the scope of the present application.

[0019] Example 1 According to the video interleaving reasoning method based on the trainable end-to-end agent system provided by the application, as shown in the formula: Figure 1 The formula shows that the method comprises the following steps: Step S1: constructing a reasoning agent system; the reasoning agent system comprises a core reasoning model and a tool library; wherein the core reasoning model is a text reasoning process obtained by implementing logical reasoning based on Qwen2.5-VL-7B according to visual information and text information; the tool library is used to obtain video space-time information by using a plurality of visual tools in the tool library for a target video; In this embodiment, the Qwen2.5-VL-7B is an open source multi-modal large model already released in the prior art.

[0020] The plurality of visual tools comprise a time positioning tool, a time counting tool, a time cutting tool, a key frame selection tool, a space positioning tool and a space tracking tool. The time positioning tool is used to position the time start point and end point of an event in a video according to the input event description; for example, UniVTG, Temp-R1 and UniTime. The time counting tool is used to count the number of times an event occurs in a video according to the input event description; for example, Qwen-2.5-VL and Intern-VL-3.5. The time cutting tool is used to cut out the corresponding segment from the video according to the input time start point and end point, and is mainly used in the case where the corresponding time start point and end point are directly provided in the question; no additional model is needed as a tool, and it can be realized directly through Python code. The key frame selection tool is used to select a key frame that best matches the description from a video according to the input description when a static scene needs to be observed; for example, Qwen2.5-VL and Intern-VL-3.5. The space positioning tool is used to position a target object in a key frame, which will give a positioning box of the corresponding object in the frame and visualize it on the image; for example, Grounding-Dino and YOLO. Spatial tracking tool, for spatial tracking of target objects in a video, giving a corresponding coordinate positioning frame for the target object in each frame, and always maintaining target consistency; for example: Grounded-SAM2.

[0021] Step S2: training the core reasoning model in the reasoning agent system to obtain a trained reasoning agent system; In this embodiment, the step S2 includes that the training of the core reasoning model includes a supervised fine-tuning training phase and a reinforcement learning training phase; The supervised fine-tuning training phase includes training the core reasoning model using historical tool-enhanced visual-text interleaved thinking chain data, so that the trained core reasoning model can obtain video spatio-temporal information through multiple rounds of interleaved reasoning based on the called tool; The historical tool-enhanced visual-text interleaved thinking chain data refers to pre-constructed training data containing video, question, visual-text interleaved thinking chain and answer. The visual-text interleaved thinking chain contains complete thinking and tool calling process, so it is called "tool enhancement". In this training phase, the visual-text interleaved thinking chain is not generated by the core reasoning model itself, but is equivalent to making the core reasoning model "imitate" the correct visual-text interleaved thinking chain defined by human beings, and learn the corresponding format and thinking mode.

[0022] The reinforcement learning training phase includes training the core reasoning model using historical question and answer data after the supervised fine-tuning training phase, so that the trained core reasoning model can combine different tools to obtain answers corresponding to different questions.

[0023] Step S3: obtaining a target video and a target question, and using the trained reasoning agent system to obtain reasoning process information and an answer corresponding to the final question through video spatio-temporal information-text reasoning process interleaved reasoning.

[0024] The application also provides a video interleaved reasoning system based on a trainable end-to-end agent system, which can be realized by executing the flow steps of the video interleaved reasoning method based on the trainable end-to-end agent system, that is, those skilled in the art can understand that the video interleaved reasoning method based on the trainable end-to-end agent system is a preferred embodiment of the video interleaved reasoning system based on the trainable end-to-end agent system.

[0025] Embodiment 2 Embodiment 2 is a preferred example of embodiment 1 The application provides a video interlaced reasoning method based on a trainable end-to-end intelligent agent system, which comprises the following steps: constructing an end-to-end trainable reasoning intelligent agent system, calling a plurality of video tools in a tool library through a core reasoning model in the trained reasoning intelligent agent system to obtain visual information, and completing visual-text interlaced reasoning.

[0026] In the reasoning process of each round, the core reasoning model first generates a reasoning step of text, and obtains a tool name and input content by matching a tool calling request in the reasoning step of text, and then selects a corresponding tool from a tool library to be used.

[0027] The reasoning intelligent agent system comprises a core reasoning model and a tool library, and the tool library comprises expert models of computer vision subfields with the best performance. The core reasoning model can output a reasoning process of text mode, and can obtain additional visual information by calling tools in the tool library to help reasoning. In the training process, the training is divided into two stages: in the first stage, a supervised fine-tuning is performed on the core reasoning model by using visual-text interlaced thinking chain data enhanced by tools, so that the core reasoning model learns an interlaced reasoning mode and acquires the ability to call tools to obtain visual information; and in the second stage, a reinforcement learning training is performed on the core reasoning model by using question-answer annotated general question and answer data, and only the final answer of the model is supervised in the training process, and the model is allowed to freely explore the multi-round thinking process, so that the core reasoning model freely explores how to use tools to solve different types of problems and enhances the generalization ability of the core reasoning model to open world problems. In the two-stage training process, all tool models do not participate in the training, but are directly used to ensure that the abilities of the tool models are stable.

[0028] The core reasoning model in the trained reasoning intelligent agent system calls a plurality of video tools in the tool library to obtain visual information, and completes visual-text interlaced reasoning, which comprises the following steps: The core reasoning model first outputs a textual thought process in each round, which includes specially defined character pairs (<tool_call> …< / tool_call> The content within this special character pair represents the information the core inference model needs to invoke, including the tool's name and the specific content to be input. The system then matches the appropriate tool from its tool library based on this information and uses it to generate corresponding visual information output. This visual information is then combined with the previously generated text-based thought process from the core inference model to form a visual-text interleaved thought process. Each generated interleaved thought process is reassembled to form a historical thought process, serving as a reference for the next round of text-based thought process output by the core inference model.

[0029] The core reasoning model automatically determines whether the current information is sufficient. If it deems the information sufficient, it will use special character pairs ( <answer> …< / answer> The system outputs the final answer, therefore it will also consider the text modality during each round of thinking. <answer> …< / answer> The system matches special character pairs. If a match is found, the reasoning process is considered complete, and the answer is extracted and used as the final output.

[0030] Example 3 Example 3 is a preferred example of Example 1 and / or Example 2. A video interleaved inference method based on a trainable end-to-end agent system, provided by the present invention, includes: Question: According to the video, how many people were on the bus when it stopped at a certain stop? Options: (A) 3; (A) 2; (A) 1; (A) 4 Standard video model (directly outputs answer options): outputs incorrect answers and cannot trace the source of the error at any stage.

[0031] The conventional text-based thought process model (first outputting the thought process in pure text mode, then arriving at the answer): Although the thought process is output, it fails to focus on effective visual information, fails to identify the driver inside the car, and ultimately gives the wrong answer.

[0032] The multi-agent system of the application: first, the first step is to select a time positioning tool, position the video segment when the bus stops at a certain station, and observe the overall events in the scene; the second step is to consider using a key frame selection tool to select the most relevant frame to observe the specific situation inside the vehicle; the third step is to consider using a spatial positioning tool to position all the people appearing in the vehicle in the key frame, and to circle the corresponding positioning box to help the core reasoning model understand the number of people in the vehicle; After the above three steps, the core reasoning model obtains a complete visual-textual thinking chain from coarse to fine, from the complete observation of the bus stopping at a certain station to the final key frame of the people appearing in the vehicle, so that the model obtains sufficient information and finally obtains the correct answer option.

[0033] Those skilled in the art know that, in addition to implementing the system, device and each module thereof provided by the application in the form of pure computer readable program code, the same program can be realized by logically programming the method steps to make the system, device and each module thereof provided by the application in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, the system, device and each module thereof provided by the application can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing methods and structures within hardware components.

[0034] The specific embodiments of the application are described above. It should be understood that the application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the application. In the case of no conflict, the embodiments of the application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A video interleaved inference method based on a trainable end-to-end intelligent agent system, characterized in that, include: Step S1: Construct the reasoning intelligent agent system; The reasoning agent system includes: a core reasoning model and a tool library; wherein, the core reasoning model is based on Qwen2.5-VL-7B to obtain the text reasoning process by performing logical reasoning based on visual and textual information; the tool library is used to obtain the spatiotemporal information of the target video using various visual tools in the tool library; Step S2: Train the core reasoning model in the reasoning agent system to obtain the trained reasoning agent system; Step S3: Obtain the target video and target question, and use the trained reasoning agent system to obtain the reasoning process information and the answer to the final question through interleaved reasoning of video spatiotemporal information and text reasoning process.

2. The video interleaved inference method based on a trainable end-to-end intelligent agent system according to claim 1, characterized in that, The various visual tools include: time positioning tools, time counting tools, time cropping tools, keyframe selection tools, spatial positioning tools, and spatial tracking tools; The time positioning tool is used to locate the start and end points of the event in the video based on the input event description. The time counting tool is used to count the number of times an event occurs in the video based on the input event description; The time-cutting tool is used to cut out corresponding segments from the video based on the input time start and end points; The keyframe selection tool is used to select a keyframe from the video that best matches the description when observing a static scene, based on the input description. The spatial positioning tool is used to locate target objects in keyframes, and will provide the corresponding object's bounding box in the frame and visualize it on the image. The spatial tracking tool is used to perform spatial tracking of target objects in the video, providing a corresponding coordinate bounding box for the target object in each frame, and always maintaining target consistency.

3. The video interleaved inference method based on a trainable end-to-end intelligent agent system according to claim 1, characterized in that, Step S2 includes: training the core reasoning model includes a supervised fine-tuning training phase and a reinforcement learning training phase; The supervised fine-tuning training phase includes: training the core reasoning model using visual-text interleaved thought chain data enhanced by historical tools, so that the trained core reasoning model can obtain video spatiotemporal information through multiple rounds of interleaved reasoning based on the invoked tools. The reinforcement learning training phase includes: after the supervised fine-tuning training phase, training the core reasoning model using historical question-answering data so that the trained core reasoning model can combine different tools to obtain answers to different questions.

4. The video interleaved inference method based on a trainable end-to-end intelligent agent system according to claim 3, characterized in that, The tool-enhanced visual-text interleaved thought chain data includes: pre-constructed training data containing videos, questions, visual-text interleaved thought chains, and answers, with the visual-text interleaved thought chain containing a complete thinking and tool invocation process.

5. The video interleaved inference method based on a trainable end-to-end intelligent agent system according to claim 1, characterized in that, Step S3 includes: Step S3.1: Input the target video and target question into the trained reasoning agent system, and use the core reasoning model in the trained reasoning agent system to generate the first round of text reasoning process based on the target video and target question; Step S3.2: Obtain the tool call request in the text reasoning process based on the currently generated text reasoning process; select the corresponding tool in the tool library based on the tool call request, call the tool to process the target video accordingly, and obtain the corresponding video spatiotemporal information; Step S3.3: Combine the acquired video spatiotemporal information with the current round of textual reasoning to form a visual-textual interwoven thinking process; Step S3.4: Input the current visual-text interleaved thought process, target video, and target question into the trained reasoning agent system. Use the core reasoning model in the trained reasoning agent system to generate the second round of text reasoning process. Repeat steps S3.2 to S3.4 until the core reasoning model determines that the current text reasoning process meets the preset requirements, output the text reasoning process of each round, and take the text reasoning process of the final round as the answer to the final question. Specifically, the video reasoning process is broken down into a corresponding number of rounds of text reasoning process based on the number of tools required in the target problem.

6. A video interleaved inference system based on a trainable end-to-end intelligent agent system, characterized in that, include: Module M1: Constructing an inference agent system; The reasoning agent system includes: a core reasoning model and a tool library; wherein, the core reasoning model is based on Qwen2.5-VL-7B to obtain the text reasoning process by performing logical reasoning based on visual and textual information; the tool library is used to obtain the spatiotemporal information of the target video using various visual tools in the tool library; Module M2: Trains the core reasoning model in the reasoning agent system to obtain the trained reasoning agent system; Module M3: Acquires the target video and target question, and uses the trained reasoning agent system to obtain reasoning process information and the answer to the final question through interleaved reasoning of video spatiotemporal information and text reasoning process.

7. The video interleaved inference system based on a trainable end-to-end intelligent agent system according to claim 6, characterized in that, The various visual tools include: time positioning tools, time counting tools, time cropping tools, keyframe selection tools, spatial positioning tools, and spatial tracking tools; The time positioning tool is used to locate the start and end points of the event in the video based on the input event description. The time counting tool is used to count the number of times an event occurs in the video based on the input event description; The time-cutting tool is used to cut out corresponding segments from the video based on the input time start and end points; The keyframe selection tool is used to select a keyframe from the video that best matches the description when observing a static scene, based on the input description. The spatial positioning tool is used to locate target objects in keyframes, and will provide the corresponding object's bounding box in the frame and visualize it on the image. The spatial tracking tool is used to perform spatial tracking of target objects in the video, providing a corresponding coordinate bounding box for the target object in each frame, and always maintaining target consistency.

8. The video interleaved inference system based on a trainable end-to-end intelligent agent system according to claim 6, characterized in that, The module M2 includes: training of the core inference model, including a supervised fine-tuning training phase and a reinforcement learning training phase; The supervised fine-tuning training phase includes: training the core reasoning model using visual-text interleaved thought chain data enhanced by historical tools, so that the trained core reasoning model can obtain video spatiotemporal information through multiple rounds of interleaved reasoning based on the invoked tools. The reinforcement learning training phase includes: after the supervised fine-tuning training phase, training the core reasoning model using historical question-answering data so that the trained core reasoning model can combine different tools to obtain answers to different questions.

9. The video interleaved inference system based on a trainable end-to-end intelligent agent system according to claim 8, characterized in that, The tool-enhanced visual-text interleaved thought chain data includes: pre-constructed training data containing videos, questions, visual-text interleaved thought chains, and answers, with the visual-text interleaved thought chain containing a complete thinking and tool invocation process.

10. The video interleaved inference system based on a trainable end-to-end intelligent agent system according to claim 6, characterized in that, The module M3 includes: Module M3.1: Input the target video and target question into the trained reasoning agent system, and use the core reasoning model in the trained reasoning agent system to generate the first round of text reasoning process based on the target video and target question; Module M3.2: Based on the currently generated text reasoning process, obtain the tool call request in the text reasoning process; based on the tool call request, select the corresponding tool in the tool library, call the tool to process the target video accordingly, and thus obtain the corresponding video spatiotemporal information; Module M3.3: Combines the acquired video spatiotemporal information with the current round of textual reasoning to form a visual-text interwoven thinking process; Module M3.4: Input the current visual-text interleaved thought process, target video, and target question into the trained reasoning agent system. Use the core reasoning model in the trained reasoning agent system to generate the second round of text reasoning process. Repeatedly trigger modules M3.2 to M3.4 until the core reasoning model determines that the current text reasoning process meets the preset requirements, output the text reasoning process of each round, and take the text reasoning process of the final round as the answer to the final question. Specifically, the video reasoning process is broken down into a corresponding number of rounds of text reasoning process based on the number of tools required in the target problem.

Images