Multi-agent collaboration method, system and equipment for multi-modal perception and medium

Abstract

The invention provides a multi-agent cooperation method, system and device for multi-modal perception and a medium, and relates to the technical field of computer vision and video information processing. The method comprises the following steps: determining a plurality of video clips of a target video; for each video clip, determining a prediction event and an event verification problem corresponding to the prediction event based on the historical clip description and a preset problem library; according to the event verification question, guiding the intelligent agent to carry out visual information analysis on the video clip, and determining a question answer, a key target and a video clip description of the video clip; determining short-term analysis information and long-term memory information based on the question answer, the key target, the video clip description and the historical clip description; and according to the video clip description, the short-term analysis information and the long-term memory information, analyzing the video clip to obtain an analysis result. The method and the device are used for improving interpretability and accuracy of video understanding.

Description

Technical Field

[0001] This application relates to the fields of computer vision and video information processing technology, and in particular to a multi-agent collaborative method, system, device and medium for multimodal perception. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, video information processing technology is playing an increasingly important role in fields such as smart security, autonomous driving, human-computer interaction, and multimedia content analysis. It can extract valuable information from massive amounts of video data, supporting decision-making in various industries and driving their intelligent development. For example, in the field of smart security, video information processing technology can monitor abnormal events in real time to ensure public safety; in the field of autonomous driving, it can help vehicles accurately understand their surroundings and make reasonable driving decisions.

[0003] Traditional video understanding methods typically rely on deep convolutional neural networks or Transformer architectures to extract spatiotemporal features from videos, thereby achieving action recognition or event detection. However, these methods are often limited to predefined category labels, lacking deep semantic understanding and logical reasoning capabilities for video content, making them ill-suited for complex and ever-changing real-world scenarios. In recent years, the emergence of Multimodal Large Language Models (MLLMs) has brought new breakthroughs to video understanding. MLLMs, through learning from massive amounts of text and image data, possess powerful generalization and text generation capabilities, enabling them to better understand the semantic information in videos. Multi-agent collaborative systems based on large language models are also gradually emerging. This technology decomposes complex video analysis tasks into multiple sub-tasks, each undertaken by a different agent role, simulating human perception and cognitive processes. Different agents interact and collaborate to jointly complete the video understanding task.

[0004] However, existing methods suffer from poor interpretability and accuracy in processing complex and ever-changing video data to achieve accurate understanding and decision-making. Summary of the Invention

[0005] The multi-agent collaborative method, system, device, and medium for multimodal perception provided in this application are intended to address the problem that existing methods suffer from poor interpretability and accuracy in processing complex and ever-changing video data to achieve accurate understanding and decision-making.

[0006] In a first aspect, embodiments of this application provide a multi-agent cooperative method for multimodal perception, including:

[0007] Identify multiple video segments of the target video;

[0008] For each video segment, based on historical segment descriptions and a pre-set question library, a predicted event and the corresponding event verification question are determined. The historical segment description is a description of the segments preceding the video segment in the target video.

[0009] Based on the event verification question, guide the agent to analyze the visual information of the video clip and determine the question answer, key objectives and video clip description of the video clip;

[0010] Based on question answers, key objectives, video clip descriptions, and historical clip descriptions, short-term analytical information and long-term memory information are identified.

[0011] Based on video segment descriptions, short-term analysis information, and long-term memory information, video segments are analyzed to obtain analysis results. These results are used to indicate the event category of the video segment and the basis for decision-making.

[0012] In one possible implementation, for each video segment, a predicted event and a corresponding event verification question are determined based on historical segment descriptions and a preset question library. This includes: for each video segment, determining a predicted event based on historical segment descriptions; and selecting an event verification question corresponding to the predicted event from the preset question library, wherein the question library includes a set of event verification questions for multiple event categories, and the set of event verification questions for each event category is determined based on the text descriptions of multiple historical event videos under that event category.

[0013] In one possible implementation, before determining the predicted event and the corresponding event verification question for each video segment based on historical segment descriptions and a pre-set question base, the method further includes: determining multiple event categories; for each event category, generating a set of event verification questions corresponding to the event category based on a large language model and a pre-set knowledge base, wherein the pre-set knowledge base includes text descriptions of multiple historical event videos corresponding to multiple event categories; and constructing a question base based on the set of event verification questions corresponding to all event categories.

[0014] In one possible implementation, short-term analysis information and long-term memory information are determined based on question answers, key objectives, video clip descriptions, and historical clip descriptions. This includes: integrating the video clip descriptions and historical clip descriptions to obtain a spatiotemporal scene map, which is used to indicate the short-term spatiotemporal context of the video clip; determining short-term analysis information based on the spatiotemporal scene map; determining key information of the video clip based on question answers, key objectives, and video clip descriptions; and storing the key information in a cue database to form long-term memory information. The cue database is used to integrate and store key information from the video clip and historical video clips, where historical video clips are segments preceding the target video clip.

[0015] In one possible implementation, before determining short-term analysis information based on the spatiotemporal scene graph, the method further includes: performing a preliminary analysis of the video segment based on question answers, key objectives, and video segment descriptions to obtain short-term decisions, which indicate whether a target event exists in the video segment and a list of event categories when a target event exists in the video segment; correspondingly, determining short-term analysis information based on the spatiotemporal scene graph includes: determining short-term analysis information based on the spatiotemporal scene graph and the short-term decisions.

[0016] In one possible implementation, after analyzing the video segments based on video segment descriptions, short-term analysis information, and long-term memory information to obtain analysis results, the method further includes: determining an initial event evaluation score for each video segment based on the analysis results corresponding to each video segment in the target video; determining the similarity and distance relationship between each video segment and all other video segments in the target video; determining the association weight between the video segment and all other video segments in the target video based on the similarity and distance relationship between the video segment and all other video segments in the target video; taking other video segments whose association weights satisfy preset weight conditions as similar segments of the video segment to obtain a set of similar segments; and determining the target event evaluation score of the video segment based on the initial event evaluation score of each similar segment in the set of similar segments and the association weight between the video segment and each similar segment.

[0017] In one possible implementation, the method further includes: performing temporal smoothing on the evaluation score sequence to obtain a segment-level evaluation score sequence, the evaluation score sequence including the event target evaluation scores of all video segments in the target video; mapping the segment-level evaluation score sequence to each frame of the target video to generate a frame-level evaluation score curve for the target video.

[0018] Secondly, embodiments of this application provide a multi-agent cooperative system for multimodal perception, including:

[0019] The video sampling module is used to determine multiple video segments of the target video;

[0020] The first intelligent agent is used to determine the predicted event and the corresponding event verification question for each video segment based on the historical segment description and the preset question library. The historical segment description is a description of the segments in the target video that precede the video segment.

[0021] The second intelligent agent is used to guide the intelligent agent to analyze the visual information of the video clip based on the event verification question of the first intelligent agent, and to determine the question answer, key target and video clip description of the video clip;

[0022] The third intelligent agent is used to determine short-term analysis information and long-term memory information based on the question answers, key objectives and video clip descriptions of the second intelligent agent, as well as historical clip descriptions.

[0023] The fourth agent is used to analyze video segments based on video segment descriptions, short-term analysis information, and long-term memory information, and obtain analysis results. The analysis results are used to indicate the event category of the video segment and the basis for decision-making.

[0024] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0025] The memory stores instructions that the computer executes;

[0026] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0027] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0028] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed, implements the first aspect and / or various possible implementations of the first aspect.

[0029] The multi-agent collaborative method, system, device, and medium for multimodal perception provided in this application determine multiple video segments of a target video. For each video segment, based on historical segment descriptions and a pre-set question library, a predicted event and a corresponding event verification question are determined. The historical segment description describes segments preceding the target video segment. Based on the event verification question, the agent is guided to perform visual information analysis on the video segment to determine the question answer, key objectives, and video segment description. Based on the question answer, key objectives, video segment description, and historical segment description, short-term analysis information and long-term memory information are determined. The video segment is analyzed based on the video segment description, short-term analysis information, and long-term memory information to obtain analysis results. The analysis results are used to indicate the event category of the video segment and the means of decision-making. By using fragmentation processing and a question-oriented visual information analysis strategy, future events are actively predicted and corresponding event verification questions are generated, thereby realizing a question-oriented visual information analysis strategy to guide the agent to accurately perceive the video segment. By capturing instantaneous scene states and accumulated historical clues through short-term analysis information and long-term memory information, the problems of event integrity destruction and long-distance dependency loss can be solved. This not only improves the accuracy of video understanding, enabling precise identification of various key information and event categories and avoiding misjudgments and omissions, but also clearly demonstrates the decision-making process and basis by presenting predicted events, question answers, and multi-information analysis processes, enhancing the interpretability of video understanding results and providing reliable support for related video analysis applications, thereby improving the quality and reliability of video understanding. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0031] Figure 1 A schematic diagram of the structure of the multi-agent cooperative system for multimodal perception provided in this application;

[0032] Figure 2 A flowchart illustrating the multi-agent cooperative method for multimodal perception provided in this application. Figure 1 ;

[0033] Figure 3 A flowchart illustrating the multi-agent cooperative method for multimodal perception provided in this application. Figure 2 ;

[0034] Figure 4 A schematic diagram illustrating the specific process of the multi-agent cooperative method for multimodal perception provided in this application;

[0035] Figure 5A schematic diagram of the structure of the multi-agent cooperative device for multimodal perception provided in this application;

[0036] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0037] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0039] First, let me explain the terms used in this application:

[0040] Large Language Model (LLM): refers to a deep learning model that uses a large-scale corpus for training to learn the patterns and structures of natural language and can simulate human language generation.

[0041] Multimodal Large Language Model (MLLM) refers to a deep learning model that can integrate information from multiple modalities (text, images, audio, etc.) for comprehensive understanding and reasoning, and ultimately generate text output.

[0042] Vertical Multimodal Large Language Model: This refers to a large multimodal model that can be deeply optimized for a specific industry or field. It is characterized by professionalism, compliance and scenario adaptability, and has more relevant professional knowledge for specific industries or fields, such as social security governance, medical care, and finance.

[0043] Vertical Multimodal LLM-based Multi-Agent Systems (MLLM-MAS) are systems composed of multiple agents with different multimodal vertical domain models. These agents combine real-world problems with knowledge from vertical domains and collaborate to complete complex tasks.

[0044] In this embodiment, multi-agent collaboration refers to the cooperation and collaboration of multiple agents with autonomous perception, analysis, and information processing capabilities to complete the analysis of a target video. These agents each possess different functions and capabilities; for example, some agents excel in image recognition, accurately detecting and identifying objects and people in video clips; others are proficient in natural language processing, understanding questions in a pre-defined question bank and generating reasonable answers and descriptions. When analyzing the target video, each agent does not work independently but, according to the overall process and requirements, leverages its own strengths to process the video clips based on acquired event verification questions and other information. Afterward, they share and exchange their processing results, such as question answers and key objectives. Through mutual collaboration and comprehensive analysis, combined with historical clip descriptions and other data, they jointly determine short-term analysis information and long-term memory information, ultimately achieving accurate judgment of the event category of the video clips and determining the basis for decision-making, thereby completing a comprehensive and in-depth analysis of the target video.

[0045] In existing technologies, traditional methods are limited to predefined category labels and lack the ability to deeply understand the semantics and make logical reasoning about video content, making it difficult to cope with complex situations in real-world scenarios. Multimodal large language models face challenges such as difficulty in capturing long-distance causal dependencies and the potential for content illusion when processing long videos. Furthermore, most existing multi-agent video analysis frameworks rely on passive perception, performing general feature extraction and global description of input video segments. They lack the ability to proactively predict future events based on historical context, and when faced with complex scenarios, they struggle to focus on fine-grained clues crucial to the qualitative nature of events, often resulting in the omission of key information or a waste of computational resources.

[0046] Furthermore, when processing long video streams, existing methods generally face the problem of long-distance dependency gaps in spatiotemporal information. Traditional sliding window or fixed-length sampling mechanisms are difficult to effectively separate and take into account both "short-term scene states" and "long-term historical clues." As the video length increases, relevant information from earlier periods is easily forgotten or obscured by subsequent redundant information, causing the model to be unable to establish logical connections across long time spans, thus compromising the integrity of event analysis.

[0047] Furthermore, decision-making mechanisms relying on a single model or perspective suffer from poor robustness and a lack of interpretability. When faced with ambiguous or complex video content, a single agent is prone to content illusions or making arbitrary misjudgments. The lack of cross-validation mechanisms among multiple roles to correct logical biases makes it difficult to meet the requirements of high-reliability scenarios.

[0048] Therefore, how to construct a video understanding framework that has proactive prediction capabilities, can effectively integrate long short-term memory, and improves the accuracy and interpretability of analysis through multi-agent collaboration and debate has become an urgent technical problem to be solved in the field of computer vision and video information processing.

[0049] To address the aforementioned issues, embodiments of this application provide a multi-agent collaborative method, system, device, and medium for multimodal perception. By dividing a target video into multiple video segments, for each segment, historical segment descriptions (including content features from previous segments) are used in conjunction with a pre-set question library. Based on the temporal continuity and correlation of video events, predicted events and event verification questions are determined. This provides forward-looking guidance for the analysis of the current video segment through historical information, giving the agent clear direction when processing video segments and enabling more targeted visual information perception and analysis. Then, based on the event verification questions, the agent is guided to perform visual information parsing of the video segments. The agent comprehensively utilizes multiple... Modal perception technology fully extracts key information from videos, avoiding a comprehensive but unfocused analysis of video content. This approach more accurately obtains question answers, key objectives, and video segment descriptions. Based on question answers, key objectives, video segment descriptions, and historical segment descriptions, short-term analysis information and long-term memory information are determined. Short-term analysis information captures short-term changes and features, while long-term memory information retains key information over longer periods in the video, thus addressing the issues of completeness of event information and long-distance dependencies, further improving the accuracy of the analysis. Finally, a comprehensive analysis of the video segment, combining video segment descriptions, short-term, and long-term information, yields analytical results indicating event categories and decision-making criteria. Therefore, through multi-agent collaboration, multimodal perception, and multi-information fusion, a deep understanding of video content is achieved, enabling users to clearly understand the process and reasons behind video understanding, enhancing the interpretability of the results, and improving the overall quality and reliability of video understanding.

[0050] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0051] The execution entity of the multi-agent collaborative method for multimodal perception provided in this application can be a computing device such as a server or server cluster. The server can be a mobile phone, computer, tablet, or other device. This application does not impose any particular restrictions on the implementation method of the execution entity, as long as the execution entity can determine multiple video segments of the target video; for each video segment, based on historical segment descriptions and a preset question library, determine the predicted event and the corresponding event verification question; the historical segment description is a description of segments preceding the video segment in the target video; based on the event verification question, guide the agent to perform visual information analysis on the video segment to determine the question answer, key objectives, and video segment description; based on the question answer, key objectives, video segment description, and historical segment description, determine short-term analysis information and long-term memory information; based on the video segment description, short-term analysis information, and long-term memory information, analyze the video segment to obtain analysis results, which are used to indicate the event category and decision basis of the video segment.

[0052] For example, the entity executing this method can be a system server that stores multi-agent cooperative methods for multimodal perception. Figure 1 This is a schematic diagram of the structure of the multi-agent cooperative system for multimodal perception provided in this application, such as... Figure 1 As shown, the multi-agent collaborative system 10 for multimodal perception includes: a video sampling module 101, used to determine multiple video segments of a target video; a first agent 102, used to determine a predicted event and a corresponding event verification question for each video segment based on historical segment descriptions and a preset question library, wherein the historical segment descriptions are descriptions of segments preceding the video segment in the target video; a second agent 103, used to guide the agent to perform visual information analysis on the video segment based on the event verification questions of the first agent 102, and determine the question answers, key objectives, and video segment descriptions of the video segment; a third agent 104, used to determine short-term analysis information and long-term memory information based on the question answers, key objectives, and video segment descriptions of the second agent 103, as well as the historical segment descriptions; and a fourth agent 105, used to analyze the video segment based on the video segment descriptions, short-term analysis information, and long-term memory information, and obtain analysis results, which are used to indicate the event category and decision basis of the video segment.

[0053] Furthermore, the first intelligent agent 102 (such as the future event prediction and question-asking agent QAgent) can accurately determine the predicted events that may occur in the current video segment by leveraging historical segment descriptions (i.e., relevant descriptions of segments preceding the current video segment in the target video) and a pre-set question library, through its own algorithms and logical reasoning capabilities, and generate corresponding event verification questions to guide subsequent analysis. The second intelligent agent 103 (such as the visual perception agent VAgent) uses the event verification questions generated by the first intelligent agent 102 as a basis to conduct in-depth analysis of the video segment using technologies such as visual recognition. It extracts key information from the video's visual information to determine the video segment's answers to the event verification questions, key targets in the scene, and a detailed description of the video segment.

[0054] The third agent 104 can generate short-term analysis information and long-term memory information based on the question answers, key objectives, video segment descriptions, and historical segment descriptions provided by the second agent 103. The short-term analysis information focuses on the local features and short-term changes of the current video segment, while the long-term memory information is used to retain key information over long periods of the video and uncover long-distance causal relationships. It should be noted that, when necessary, the third agent 104 can be split into two parts: generating short-term analysis information and long-term memory information. That is, the third agent 104 can consist of two agents (such as the short-term context information summarization agent SAgent and the clue extraction and analysis agent CAgent). These two agents can respectively analyze the video in depth from local short-term changes and long-term causal relationships, accurately capturing both instantaneous dynamics and grasping historical context. The combination of these two agents provides comprehensive information for event decision-making, improving the accuracy of video understanding, while clearly demonstrating the decision-making basis to enhance its interpretability, better adapting to complex scenarios, and ensuring system stability and reliability.

[0055] The fourth agent 105 (such as the event analysis and tracing agent TAgent) can integrate video segment descriptions, short-term analysis information, and long-term memory information, and perform comprehensive analysis of video segments based on a multi-agent adversarial debate mechanism. Specifically, the fourth agent 105 internally constructs two debate roles with opposing positions (such as a pro-agent agent and a con-agent agent) and a judge role (referee agent): the pro-agent agent, based on the input visual and memory information, asserts that a specific event category has occurred in the current video segment and lists evidence; the con-agent agent questions the pro-agent agent's viewpoint, corrects logical errors, or proposes non-event-related alternative explanations to stimulate deep divergent thinking about the video content; the referee agent monitors the debate process and, after multiple rounds of interactive debate, makes a final ruling by integrating the logic and factual basis of both sides' arguments, thereby eliminating the risk of illusion from a single perspective and obtaining the final analysis result. The analysis result is used to indicate the event category of the video segment and the decision-making basis including the debate reasoning process.

[0056] It should be noted that there are no particular restrictions on the specific implementation of the multiple agents involved in this system, as long as the functions of these agents can be realized to meet the functional requirements of each agent in a multi-agent collaborative system for multimodal perception. For example, multiple independent AI models can be used to build and execute corresponding functions respectively, i.e., a multi-agent mode, where each agent completes a specific task based on its own independent model architecture and algorithm logic; alternatively, a general model can simulate multiple roles by inputting different prompts, enabling the general model to possess the functions of each agent under the guidance of different prompts. Optionally, the agents can be determined based on a large multimodal model or a multimodal vertical domain model.

[0057] This multi-agent collaborative system for multimodal perception, through the division of labor and cooperation among multiple agents, makes full use of historical information and pre-set questions to accurately analyze visual information and make decisions based on multiple factors. It effectively improves the accuracy and interpretability of video understanding, enhances the model's generalization ability, maintains the integrity of events and long-distance causal relationships, and can better adapt to the needs of video understanding in complex scenarios, providing reliable support for related video analysis applications.

[0058] Figure 2 A flowchart illustrating the multi-agent cooperative method for multimodal perception provided in this application. Figure 1 ,like Figure 2 As shown, the method may include:

[0059] S201. Identify multiple video segments of the target video.

[0060] The target video refers to the video content that needs to be analyzed, and it can be videos from various sources and of various types, such as surveillance videos, movies, and short online videos. Video clips are relatively independent parts of the target video that are divided according to certain time intervals or logic.

[0061] In some examples, video processing techniques are used, such as segmenting the target video into multiple video segments according to a fixed time length (e.g., each segment is 8 seconds long) or based on features such as scene transitions and action changes in the video content. For example, for a target video, the video can be evenly divided into N consecutive non-overlapping segments of 8-second length: For each video segment, a video frame is randomly selected within 1 second at a sampling rate of 1-fps. As a video clip .

[0062] S202. For each video segment, based on the historical segment description and the preset question library, determine the predicted event and the event verification question corresponding to the predicted event. The historical segment description is a description of the segments in the target video that precede the video segment.

[0063] In this step, the historical segment description can be a textual description of the content, features, events, etc., of all segments preceding the current video segment in the target video. The preset question library can be a pre-built set of questions related to various possible events or scenarios, used to guide the analysis of the video segment.

[0064] Predicting events involves inferring possible events in the current video segment based on historical segment descriptions and an understanding of a pre-defined question library. Event verification questions, closely related to the characteristics of events predicted, are used to verify whether the predicted event actually occurs in the current video segment; these questions can include multiple instances.

[0065] This involves using natural language processing (NLP) technology to analyze historical video clip descriptions, combining them with question templates from a pre-set question database, and using algorithmic matching and reasoning to determine the predicted events that might occur in the current video clip, and generating corresponding event verification questions. For example, using... As the video segment that needs to be analyzed at the current moment, an intelligent agent (such as QAgent) is used to input continuous video segments from the past. The description of the event is used to make reasonable predictions about the subsequent development of the event by using prompt words (such as predicting what is most likely to happen next based on the above description and listing 3 issues that need attention). Then, based on the prediction of the future event, several possible key issues are selected from the preset question library Q to form a question set (corresponding to event verification issues).

[0066] S203. Based on the event verification question, guide the agent to analyze the visual information of the video clip and determine the question answer, key objectives and video clip description of the video clip.

[0067] In this step, visual information parsing refers to the process of analyzing and understanding visual elements such as images, actions, and objects in a video clip using pre-set computer vision technology within the intelligent agent. Question answering can be an answer given based on the visual information of the video clip to an event verification question; for example, a yes or no answer to the question of whether a gathering of people occurred. Key targets can refer to entities (such as objects, people, or actions) identified from the video clip that are crucial to the event analysis, such as suspicious persons or landmarks like train stations. Video clip description refers to a detailed textual description of the current video clip content, which may include information such as the main events, key targets, and the relationships between targets.

[0068] Furthermore, computer vision algorithms such as object detection and action recognition can be used to process video clips, extract relevant information based on the event verification question, generate corresponding question answers, identify key targets, and form a video clip description. In one example, the video clip and the event verification question are input together into an agent (such as VAgent), which outputs all the required information at once using pre-designed prompts. In another example, a visual model (such as YOLO) is first used to detect key targets, then an image description model is used to generate a video clip description, and finally the target, description, and question are input together into VAgent to answer the event verification question and generate a question answer.

[0069] For example, with As the video segment that needs to be analyzed at this moment, the video segment Encoded into base64 format, Preset prompts and QAgent for fragments Issues with generated event verification As input to VAgent, VAgent then processes the video clips. Output Including questions and answers Key targets in the video clip and video clip description :

[0070] .

[0071] S204. Based on question answers, key objectives, video clip descriptions, and historical clip descriptions, determine short-term analytical information and long-term memory information.

[0072] Short-term analysis information refers to the rapid analysis results of local features and short-term changes in the current video segment. This can include structured information about the spatiotemporal relationships between targets within the current and neighboring segments, such as scene diagrams; it can also include preliminary decisions made based on the event characteristics of the current video segment, such as the possible event categories. Long-term memory information, on the other hand, retains key information from long-term video history to demonstrate long-distance causal relationships. For example, long-term memory information can be determined through a database of key factual clues accumulated and extracted from the entire video history.

[0073] In this step, data fusion and analysis algorithms can be used to integrate question answers, key objectives, video clip descriptions, and historical clip descriptions. Short-term analysis models are used to obtain short-term analysis information, while long-term memory mechanisms are utilized to generate long-term memory information. In one example, the descriptions of the current and previous video clips and key objectives can be input into an agent (such as SAgent), which is then instructed to generate a scene graph describing the relationships between objectives (such as chasing, approaching, etc.) as short-term analysis information. An agent (such as CAgent) is then used to extract the most crucial event information (such as 14:05:02, Person B begins chasing Person A) from the parsed information of the current video clip and add it as a timestamp to a continuously growing list or knowledge graph, thereby obtaining long-term memory information.

[0074] S205. Based on the video segment description, short-term analysis information, and long-term memory information, analyze the video segment to obtain analysis results. The analysis results are used to indicate the event category of the video segment and the basis for decision-making.

[0075] Analyzing video clips can refer to using a multi-agent debate framework to introduce opposing viewpoints to verify whether the video clips truly reflect the characteristics of the relevant events, thereby determining the root cause of the events. Event categorization refers to classifying the events occurring in the video clips, such as combat, communication, or movement. Decision-making criteria refer to the reasons and evidence used to determine the event category.

[0076] Optionally, a decision-making algorithm or model (such as a collaborative architecture including a pro-party agent, a con-party agent, and a referee agent) can be used to comprehensively judge the event category and generate corresponding decision-making basis by combining video segment description, short-term analysis information, and long-term memory information. For example, the current segment description, short-term scene graph, and long-term clue database summary are input into a pre-set agent (such as TAgent). Through prompts (such as constructing a multi-role debate environment including a party, a con-party agent, and a referee, instructing the party to assert the occurrence of a specific event based on the current visual clues, instructing the con-party to adopt a "tit-for-tat" strategy to raise logical challenges or alternative explanations based on long-term memory information, and instructing the referee to generate a final ruling based on the logical consistency of the arguments after multiple rounds of interaction), the agent is guided to conduct adversarial reasoning, thereby generating a final analysis result containing a clear event category and decision-making basis verified by the debate.

[0077] It should be noted that each step of this multi-agent collaborative method for multimodal perception can be implemented through multiple independently deployed multimodal vertical domain models in a multi-agent collaborative manner (e.g., Figure 1The example of a multi-agent collaborative system oriented towards multimodal perception can also utilize a unified multimodal large language model, which can be invoked multiple times via role-based prompt templates pre-set for different steps, enabling it to perform prediction, visual analysis, information integration, and final decision-making functions respectively. In multi-agent collaborative systems oriented towards multimodal perception, the multimodal vertical domain model can leverage its multimodal data processing capabilities to comprehensively analyze the visual and audio information of videos. Simultaneously, with its deep understanding of knowledge, it can accurately identify various types of events and provide logical explanations consistent with relevant scenarios, industries, or domains, providing reliable support for related video analysis applications.

[0078] The multi-agent collaborative method for multimodal perception provided in this application first fragments the target video. Based on historical segment descriptions and a pre-set question database, it actively predicts relevant events for each video segment and generates corresponding verification questions, thereby constructing a question-oriented visual information parsing mechanism to guide the agent to accurately perceive video segment information. Then, based on the parsed question answers, key targets, video segment descriptions, and historical segment descriptions, it determines short-term analysis information and long-term memory information. Short-term analysis captures instantaneous scene features, while long-term memory accumulates historical clues, effectively solving the problems of event information integrity and long-distance dependency. This not only greatly improves the accuracy of video understanding, accurately identifying various key information and event categories and avoiding misjudgments, but also clearly demonstrates the decision-making process and basis by presenting the predicted events, question answers, and multi-information analysis process, enhancing the interpretability of the video understanding results and providing reliable support for related video analysis applications, significantly improving the quality and reliability of video understanding.

[0079] Based on the above embodiments, the method described in S202 for determining a predicted event and a corresponding event verification question for each video segment based on historical segment descriptions and a preset question library may include: determining a predicted event for each video segment based on historical segment descriptions; selecting an event verification question corresponding to the predicted event from a preset question library, wherein the question library includes a set of event verification questions for multiple event categories, and the set of event verification questions for each event category is determined based on the text descriptions of multiple historical event videos under that event category.

[0080] In this embodiment, further, for each video segment, based on the information provided by the historical segment description, the most likely predicted event in the future is determined through analysis and reasoning; then, from the preset question library, the corresponding event verification question is selected according to the event category associated with the predicted event. The set of event verification questions for each event category in the question library is pre-organized and summarized based on the text descriptions of multiple historical event videos under that event category.

[0081] Event categories can refer to the classification of events with different natures and characteristics. For example, events can be divided into different categories such as abnormal events, communication events, and sports events. Each category has similar event characteristics and behavioral patterns.

[0082] Text descriptions of historical event videos can be detailed descriptions of video content that occurred in the past and is related to a specific event category. This includes information such as the main events, people's actions, and scenes in the video, such as the time and place of the event, the people involved, and the specific manifestations of the event.

[0083] In some examples, a rule system can be pre-established to match the features of the predicted event with the features of different event categories in a question bank, thereby selecting the corresponding event verification question. Alternatively, an intelligent agent (such as QAgent) can be used, taking historical fragment descriptions and a pre-set question bank as input. Based on pre-set prompts, QAgent can make reasonable predictions about the development of subsequent events, and then select the appropriate event verification question from the question bank based on the prediction of future events.

[0084] For example, with As the video segment to be analyzed at the current moment, QAgent's task is to make reasonable predictions about the development of subsequent events based on the descriptions of consecutive past video segments as output. Then, based on the predictions of future events (i.e., predicting events), it selects several possible key questions (i.e., event verification questions) from the question base Q. Its execution method can be as follows:

[0085] ;

[0086] in, The output of QAgent includes and ; This is an issue related to event verification. To predict events; Description of historical fragments; These are prompt words for QAgent.

[0087] By selecting event verification questions from a pre-built question library based on predicted events, the determination of event verification questions becomes more accurate and targeted, avoiding the randomness and incompleteness of freely generated questions. This improves the efficiency and accuracy of question selection, and can more effectively guide the subsequent intelligent agents in the parsing and analysis process of videos. It further enhances the recognition ability and adaptability of the entire multi-agent collaborative method for multimodal perception to different video events.

[0088] Based on the above embodiments, before determining the predicted event and the event verification question corresponding to the predicted event for each video segment based on historical segment descriptions and a preset question base as described in S202, the method may further include: determining multiple event categories; for each event category, generating a set of event verification questions corresponding to the event category based on a large language model and a preset knowledge base, wherein the preset knowledge base includes text descriptions of multiple historical event videos corresponding to multiple event categories; and constructing a question base based on the set of event verification questions corresponding to all event categories.

[0089] In this embodiment, the preset knowledge base can refer to a pre-built knowledge storage set, which contains text descriptions of multiple historical event videos corresponding to multiple event categories, providing basic knowledge and data support for generating event verification questions.

[0090] Furthermore, in the process of building the question library, the multiple event categories that need to be detected are first identified. For example, in a video surveillance scenario, different event categories such as "fighting," "theft," and "fire accidents" are determined. Then, for each determined event category, a large language model is used, combined with historical event video text descriptions related to the event or scene in a pre-set knowledge base, to learn and generate a set of event verification questions for that event category. The event verification question sets corresponding to all event categories are integrated together to build a question library for subsequent use in multi-agent collaboration processes for multimodal perception.

[0091] In some examples, given a set of event categories For each event category From the pre-set knowledge base (This includes text descriptions corresponding to various event categories) to generate a set of questions related to the event, i.e., the event verification question set. Where M represents the number of event verification questions in the event verification question set. This represents a specific event verification problem within the set. The process can be represented as:

[0092] ;

[0093] in, For each event category i, a set of video descriptions will be input into the LLM during a forward propagation. To guide the model, we summarize the meta-information for each event category and generate prompts for related questions.

[0094] Finally, a corresponding question database is built. , where N represents the number of categories in the event category set N, and each event category contains multiple related questions.

[0095] By utilizing large language models and pre-set knowledge bases to construct a question database, the construction of the question database becomes more scientific and standardized, covering a wider range of event types and richer questions. This improves the quality and efficiency of question generation, as well as the quality and applicability of the question database, providing a more reliable foundation for subsequent accurate event analysis based on the question database. It also enhances the feasibility and adaptability of the entire multi-agent collaborative method for multimodal perception.

[0096] Based on the above embodiments, the method described in S204 for determining short-term analysis information and long-term memory information based on question answers, key objectives, video segment descriptions, and historical segment descriptions may include: integrating video segment descriptions and historical segment descriptions to obtain a spatiotemporal scene diagram, which is used to indicate the short-term spatiotemporal context of the video segment; determining short-term analysis information based on the spatiotemporal scene diagram; determining key information of the video segment based on question answers, key objectives, and video segment descriptions; and storing the key information in a cue database to form long-term memory information, where the cue database is used to integrate and store key information of the video segment and historical video segments, and historical video segments are segments preceding the target video segment.

[0097] In this embodiment, a spatiotemporal scene diagram refers to a structure that graphically presents the spatiotemporal information of a video segment. It integrates spatiotemporal related content from the video segment description and historical segment description, and can intuitively display the contextual information of the video segment in the temporal and spatial dimensions, such as changes in the position of objects at different times and scene transitions. Furthermore, the spatiotemporal scene diagram can be directly used as short-term analysis information, or the spatiotemporal relationships and changes within it can be analyzed based on the constructed spatiotemporal scene diagram to determine short-term analysis information that reflects the short-term characteristics and changes of the video segment.

[0098] For example, describe the current video segment and historical fragment descriptions After integration processing, the following results were obtained: Through preset prompt words SAgent is used to extract spatiotemporal elements, such as time sequence and spatial relationships, to construct a spatiotemporal scene graph. This demonstrates the short-term spatiotemporal information of the video clip:

[0099] .

[0100] A clue repository can refer to a database or knowledge base specifically designed to integrate and store key information from video clips and historical video clips, which can be queried by subsequent clips. As a carrier of long-term memory, it preserves important information from long periods of video footage for later analysis and decision-making.

[0101] In one example, key information crucial to the analysis of the current video segment is extracted from the answers to questions, key objectives, and video segment descriptions. The identified key information is stored in a cue database. As video segments are continuously processed, the cue database continuously accumulates and integrates key information to form long-term memory information.

[0102] In addition, optionally, the clue database can employ a dynamically updated management mechanism to update and optimize existing information based on key information from new video clips. For example, when new key information is related to or conflicts with existing information in the clue database, specific algorithms can be used to merge or correct the information, ensuring the accuracy and timeliness of long-term memory information.

[0103] It should be noted that there is no particular restriction on the order of determining short-term analysis information and long-term memory information, and they can be determined simultaneously. The execution steps in this embodiment are only for the purpose of clearly illustrating the determination process, but the determination process of this information is not limited to the execution steps described in this embodiment.

[0104] By constructing a spatiotemporal scene graph and utilizing a clue database, information from video clips and historical clips is effectively integrated. This allows short-term analysis information to more accurately reflect the spatiotemporal characteristics of video clips, while long-term memory information can better preserve key information from the video. This improves the comprehensive processing capability and analytical accuracy of multi-agent collaborative methods for multimodal perception.

[0105] Based on the above embodiments, before determining short-term analysis information according to the spatiotemporal scene diagram, the method may further include: performing a preliminary analysis of the video segment based on the question answer, key objectives, and video segment description to obtain a short-term decision, which is used to indicate whether the video segment has a target event and a list of event categories when the video segment has a target event; correspondingly, determining short-term analysis information according to the spatiotemporal scene diagram includes: determining short-term analysis information based on the spatiotemporal scene diagram and the short-term decision.

[0106] Short-term decision-making refers to the results obtained after preliminary analysis of video clips. It indicates whether a target event exists in the clip, and if so, provides a list of possible event categories, offering preliminary judgment for further analysis. A target event, on the other hand, refers to a specific event type defined based on the purpose of analyzing the video clip. It may be an event closely related to key objectives that requires focused attention and identification, and its specific meaning varies depending on the application context and analysis purpose. For example, in security monitoring scenarios, target events might be security threats such as illegal intrusion, fire, or explosion; in sports event analysis scenarios, target events might be athlete fouls, spectacular goals, or crucial defensive plays.

[0107] In this embodiment, before determining short-term analysis information based on the spatiotemporal scene map, a preliminary analysis of the video clip is conducted using question answers, key objectives, and video clip descriptions. By analyzing the correlations and characteristics between these information, it is determined whether the video clip contains a target event. If the video clip is determined to exist, its possible event category is further determined, forming a list of event categories, ultimately leading to a short-term decision. When determining short-term analysis information based on the spatiotemporal scene map, the short-term decision is combined with the spatiotemporal scene map, comprehensively considering the spatiotemporal context information in the spatiotemporal scene map and the event judgment results in the short-term decision, thereby determining the short-term analysis information.

[0108] For example, the question answer, key objective, and video clip description of the current segment are concatenated and input into an agent. By designing specific prompts, the agent is instructed to perform two tasks: 1) determine whether the target event exists; 2) if it exists, list the most likely event types and their confidence levels.

[0109] Optionally, the process of identifying key information from short-term decisions and video clips can be integrated into a single agent, such as the cue extraction and analysis agent CAgent. CAgent's primary task is to answer questions. Key objectives and video clip description (correspond Extract the key information presented in the current video clip. And store key information in the clue database. In addition, CAgent makes a preliminary judgment based solely on the current video segment description. This judgment requires that CAgent's assessment include whether the target event exists in the current segment and what the top 5 most likely event categories are for the event types displayed in the video segment. Therefore, CAgent's output... as follows:

[0110] .

[0111] in, Key information for video clips; For short-term decisions; These are the default prompt words for CAgent.

[0112] By conducting preliminary event analysis to obtain short-term decisions before determining short-term analysis information based on spatiotemporal scene graphs, and integrating these short-term decisions into subsequent analyses, the short-term analysis information not only considers the spatiotemporal context of video clips but also incorporates preliminary event judgment results. This improves the accuracy and relevance of short-term analysis information in analyzing event situations in video clips, and enhances the effectiveness of the entire multi-agent collaborative method for multimodal perception in the short-term analysis stage.

[0113] Based on the above embodiments, in some embodiments, the method described in S205 for analyzing video segments based on video segment descriptions, short-term analysis information, and long-term memory information to obtain analysis results may specifically include: constructing a multi-agent adversarial debate architecture comprising a positive agent, a negative agent, and a referee agent; providing video segment descriptions, short-term analysis information, and long-term memory information simultaneously as contextual inputs to both the positive and negative agents; guiding the positive agent to construct an initial argument regarding the specific event category presented in the current video segment based on the above information; guiding the negative agent to adopt a "head-to-head" adversarial strategy based on the same above information, by mining details or long-term contextual clues in the above information that contradict the initial argument, raising questions about the initial argument and providing non-event-related alternative explanations to stimulate divergent thinking in understanding the video content; guiding the referee agent to logically evaluate the multi-round interactive arguments of the positive and negative agents, synthesizing the different interpretation paths of the two sides regarding the above information, determining the final event category, and generating a decision basis containing the debate reasoning process as the final analysis result.

[0114] Based on the above embodiments, after analyzing the video segments according to the video segment description, short-term analysis information, and long-term memory information as described in S205 to obtain the analysis results, the method may further include: determining the initial event evaluation score of each video segment based on the analysis results corresponding to each video segment in the target video; determining the similarity and distance relationship between each video segment and all other video segments in the target video; determining the association weight between the video segment and all other video segments in the target video based on the similarity and distance relationship between the video segment and all other video segments in the target video; taking other video segments whose association weights satisfy preset weight conditions as similar segments of the video segment to obtain a set of similar segments; and determining the target event evaluation score of the video segment based on the initial event evaluation score of each similar segment in the set of similar segments and the association weight between the video segment and each similar segment.

[0115] In this embodiment, the initial event assessment score is a numerical score determined based on the analysis results of a single video segment, used to measure the initial performance or importance of the video segment in relation to a specific event.

[0116] Similarity refers to the degree of similarity between video clips in terms of content, features, etc.; distance relationship describes the proximity between video clips from a certain measurement perspective (such as abstract or concrete measurement methods such as time sequence, spatial location, etc.).

[0117] Association weight is a numerical indicator representing the degree of association between the current video segment and other video segments. It is determined based on similarity and distance relationships, reflecting the degree of mutual influence between different video segments. The similar segment set consists of other video segments whose association weights with the current video segment meet preset weight conditions; these segments share a certain degree of similarity with the current segment in content and features. The event target evaluation score is the final evaluation score determined for the video segment after comprehensively considering the initial event evaluation scores of similar segments in the similar segment set and their association weights with the current segment, thus more comprehensively and accurately reflecting the video segment's performance in event evaluation.

[0118] In one example, the visual features of each video segment can be represented as feature vectors. Similarity is determined by calculating the cosine similarity between vectors, and distance relationships and association weights are further determined based on the similarity. For instance, for the feature vectors of two video segments, their cosine similarity value is calculated. If this value is greater than a certain threshold, the similarity is considered high, and a higher association weight is determined.

[0119] In another example, a deep learning model can be used to extract features from video clips. After being trained on a large amount of data, the model can extract more accurate visual features. Then, another machine learning model is used to learn and determine the similarity, distance relationships, and association weights between video clips based on the extracted features. For example, a convolutional neural network can be used to extract features, and then a graph neural network can be used to analyze the associations between clips.

[0120] Furthermore, some rules can be pre-defined to determine the association weights. For example, for video clips... To calculate the similarity between each video segment and all other video segments, the visual features of each segment are first extracted using the Imagebind encoder. Then, cosine similarity is used to calculate the visual similarity between video clips, and the distance relationship between video clips is further combined to obtain the final association weight between clips. :

[0121] ;

[0122] ;

[0123] in, For the current video clip Visual features; For the j-th video segment among other video segments Visual features; The distance relationship between video clips, .

[0124] Then, the initial evaluation scores for the events of the top-k most similar video clips are retrieved: ,in, This is used to score the current video segment. Optimize:

[0125] ;

[0126] in, This is the optimized evaluation score, i.e., the event target evaluation score for this video clip.

[0127] Ultimately, the segment-level evaluation score for a target video is: .

[0128] By comprehensively considering factors such as the analysis results of the video segment itself, its similarity to other segments, and the weight of its association, the event target evaluation score can be determined. This allows for a more comprehensive and accurate assessment of the event relevance of each video segment in the target video, providing more refined evaluation indicators for the understanding, analysis, and management of video content, thereby improving the quality and reliability of video understanding.

[0129] Based on the above embodiments, the method may further include: performing temporal smoothing on the evaluation score sequence to obtain a segment-level evaluation score sequence, wherein the evaluation score sequence includes the event target evaluation scores of all video segments in the target video; mapping the segment-level evaluation score sequence to each frame of the target video to generate a frame-level evaluation score curve of the target video.

[0130] In this embodiment, the evaluation score sequence is a sequence composed of the event target evaluation scores of all video segments in the target video, arranged in the order of the video segments. Temporal smoothing is used to eliminate random fluctuations and noise in the evaluation score sequence, making the data changes in the sequence smoother and more continuous, so as to better reflect the changing trend of evaluation scores over time. The segment-level evaluation score sequence is a new sequence obtained after temporal smoothing. It is smoother than the evaluation score sequence and can more clearly show the overall change of video segment evaluation scores over time.

[0131] A frame-level evaluation score curve is a curve formed by mapping the segment-level evaluation score sequence to each frame of the target video. It can intuitively show the changes in the evaluation score corresponding to each frame of the target video.

[0132] For example, the event target evaluation scores of all video segments in the target video are combined in chronological order to form an evaluation score sequence; temporal smoothing technology is used to process the evaluation score sequence to remove irregular fluctuations and obtain a segment-level evaluation score sequence; according to the correspondence between video segments and frames, the scores in the segment-level evaluation score sequence are mapped to each frame of the target video, thereby generating the frame-level evaluation score curve of the target video.

[0133] For example, for the evaluation score sequence First, a one-dimensional Gaussian filter is applied to smooth the surface within the local time context:

[0134] ;

[0135] in, For segment-level evaluation score sequences; The distance to the center of the Gaussian kernel; Let Variance be the variance.

[0136] Ultimately, The segment-level evaluation scores are mapped to the corresponding sampling intervals to obtain the video frame-level evaluation scores. Similarly, the same Gaussian filter is used for the video frame-level evaluation scores: Smoothing is performed to obtain the frame-level evaluation score curve of the target video.

[0137] By performing temporal smoothing on the evaluation score sequence to generate a segment-level evaluation score sequence, and mapping it to each frame to generate a frame-level evaluation score curve, the representation of evaluation scores becomes more refined and continuous. This provides reliable support for related video analysis applications (such as fine-grained video analysis, keyframe extraction, and video summarization), thereby improving the quality and reliability of video understanding.

[0138] Figure 3 A flowchart illustrating the multi-agent cooperative method for multimodal perception provided in this application. Figure 2 , Figure 4 A schematic diagram illustrating the specific process of the multi-agent cooperative method for multimodal perception provided in this application is shown below. Figure 3 and Figure 4 As shown, in this embodiment... Figure 2 Based on the examples, a multi-agent cooperative method for multimodal perception is described in detail. This method includes:

[0139] S301, Video Sampling and Encoding: Bin-wise Random Sampling.

[0140] For a given target video, the video is evenly divided into N consecutive, non-overlapping segments with a length of 8 seconds: For each video segment, a video frame is randomly selected within 1 second at a sampling rate of 1-fps. As a video clip .

[0141] S302, Composition and Task Definition of Multi-Agent Systems.

[0142] (1) The multi-agent design in this embodiment mainly includes QAgent for future event prediction and questioning, VAgent for visual perception, SAgent for short-term context information summarization, CAgent for clue extraction and analysis, and TAgent for analysis and tracing.

[0143] (2) First, QAgent predicts possible future events based on short-term contextual information. Based on these predictions, it formulates relevant questions about factors related to specific future events and asks VAgent questions, prompting VAgent to further explore potential factors for subsequent verification and analysis. VAgent is mainly responsible for extracting visual information from input video clips, tracking the visual features, actions, and interacting targets and environmental factors of the main targets in the video clips, and receiving relevant questions from QAgent to specifically explore event-related factors and generate a description of the video clips. SAgent integrates video clip descriptions from the past period to form a short-term spatiotemporal scene diagram. Then, CAgent receives the text description generated by VAgent for the current video clip and performs short-term and independent analysis, extracting key information related to the current video clip to maintain a clue library. The clue library contains key information extracted from the current clip and all previous video clips as long-term information. TAgent analyzes and verifies the current video clip based on the output of CAgent and the information in the clue library as the basis for reasoning, and then makes a final decision on the video clip.

[0144] S303. Construct a question library based on text descriptions of videos of different event categories.

[0145] Given a set of event categories For each event category From the pre-set knowledge base This generates a set of questions related to the event, namely the event verification question set. Then build a problem library. .

[0146] S304. Visual perception for future event prediction and question guidance based on the information above.

[0147] by As the video segment to be analyzed at the current moment, QAgent's task is to make reasonable predictions about the development of subsequent events based on descriptions of consecutive past video segments as output. Then, based on these predictions, it selects several possible key questions from the question library Q, aiming to enable VAgent to analyze the video segment based on the corresponding questions. Further investigation revealed potential factors related to the event's occurrence, which were then used for subsequent verification.

[0148] ;

[0149] in, This represents the video segment that needs to be analyzed at the moment. The corresponding visual perception agent (VAgent) needs to answer the question of event verification. This represents QAgent's prediction of future events, i.e., event prediction; It represents The previous series of video clips are described, i.e., the historical clips are described.

[0150] Then will be based on This guides VAgent to uncover more relevant event information. The video clips sampled from S301... Encoded into base64 format, , and QAgent generated As input to VAgent, VAgent then processes the video clips. Output Including questions and answers Key targets in the video clip and video clip description .

[0151] S305, Construction of key information in the context of both short-term and long-term information.

[0152] First, SAgent describes M consecutive video segments. As input, descriptions from multiple video clips are integrated into a spatiotemporal scene graph. This provides structured information about the short-term spatiotemporal context for subsequent analysis;

[0153] Next, regarding the output of VAgent The primary task of CAgent is to answer questions. and Extract the key information presented in the current video clip. (Among them, here) This could include key objectives. (Video clip descriptions of relevant information), and store key information in the clue database. In addition, CAgent makes a preliminary judgment based solely on the current video segment description. The judgment must include the top 5 most likely event categories for the current segment to determine if any possible event types exist. Therefore, CAgent's output... .

[0154] Among them, the results obtained in this process as well as They are used as short-term and long-term information, respectively, for the final comprehensive analysis.

[0155] S306. Event Analysis and Decision Making Based on Multi-Agent Adversarial Debate

[0156] TAgent's task is to construct a multi-role debate environment that includes the affirmative, negative, and referee sides, incorporating short-term contextual information. and the clue database maintained by CAgent The final decision is made on the video clip. During this process, the agent uses the initial output of CAgent as a basis. The opposing intelligent agent asserts the occurrence of a specific event; while the opposing intelligent agent utilizes... Historical clues and The AI ​​agent monitors the debate process, identifying logical flaws in the affirmative side's arguments or proposing alternative explanations (such as normal background activities) to stimulate divergent thinking. The adjudicating agent not only determines the final event category but also, by synthesizing evidence cited by both sides in multiple rounds of interaction, indicates which evidence was referenced during the analysis. Which specific clues are used to ensure the credibility and transparency of the explanation? Therefore, TAgent's input and output are:

[0157] ;

[0158] in, The output of TAgent is the event analysis results that have been validated through debate. The affirmative and negative sides are represented by their respective prompts. and A guided, multi-round adversarial reasoning process; The referee agent is based on preset prompts. A summary and ruling on the history of debate; The video clip for final judgment Event categories; This represents the basis for the judges' decisions based on the logic of victory or defeat generated by both sides during the debate.

[0159] In this embodiment, based on the above steps, after event analysis and decision-making based on multi-agent adversarial debate, the method may further include: event evaluation score optimization based on complete context.

[0160] Since the multi-agent system ultimately outputs event categories for video clips without considering the gradual changes in event development, this paper addresses this issue by optimizing the event evaluation labels through calculating the similarity and distance dependencies between video clips. The specific method is as follows:

[0161] (1) For video clips To calculate the similarity between each video segment and all other video segments, the visual features of each segment are first extracted using the Imagebind encoder. Then, cosine similarity is used to calculate the visual similarity between video clips, and the distance relationship between video clips is further combined to obtain the final association weight between clips. ;

[0162] (2) Event evaluation scores of the top-k most similar video segments retrieved: ,in, This is used to evaluate the event score of the current video segment. Optimization is performed to obtain the optimized event evaluation score. Ultimately, the segment-level evaluation score for the target video is: .

[0163] (3) For the evaluation score sequence First, a one-dimensional Gaussian filter is applied to smooth the sequence within the local time context, resulting in a fragment-level evaluation score sequence. Ultimately The segment-level evaluation scores are mapped to the corresponding sampling intervals to obtain the video frame-level evaluation scores. Similarly, the same Gaussian filter is used to smooth the video frame-level evaluation scores.

[0164] The multi-agent collaborative method for multimodal perception provided in this application predicts the direction of event development through question-guided visual perception, focusing on more relevant visual information; it analyzes and organizes contextual information to provide key clues needed for event analysis; and it performs comprehensive analysis based on these clues, combined with a long-distance clue database. This method selectively poses relevant questions based on predictions of future event development to guide agents to focus on more relevant visual factors, containing more key information than visual descriptions obtained through generalized prompts. Simultaneously, by analyzing and organizing contextual information, it fully and rationally utilizes the contextual information of the video, and through event analysis based on contextual clues, it places independent video segments within a more complete event for analysis and discussion, achieving more rational event analysis and long-distance causal tracing. Furthermore, this method applies a multi-agent collaborative system to video understanding. By utilizing the different perspectives of different agents, it can conduct comprehensive video event analysis from multiple angles, and extract and organize the information above. It can also use the persistence and dependency of events to uncover the essence of the event from some early behaviors. Compared with analyzing video segments individually, it can provide a more credible and accurate explanation of video events.

[0165] Figure 5 This is a schematic diagram of the structure of the multi-agent cooperative device for multimodal perception provided in this application, as shown below. Figure 5 As shown, the multi-agent cooperative device 50 for multimodal perception provided in this embodiment includes:

[0166] The video segment determination module 501 is used to determine multiple video segments of the target video;

[0167] The event problem determination module 502 is used to determine the predicted event and the event verification question corresponding to the predicted event for each video segment based on the historical segment description and the preset question library. The historical segment description is a description of the segments in the target video that precede the video segment.

[0168] The visual analysis module 503 is used to analyze the visual information of video clips based on the event verification questions, and to determine the question answers, key targets and video clip descriptions of the video clips.

[0169] The short-term and long-term information analysis module 504 is used to determine short-term analysis information and long-term memory information based on question answers, key objectives, video clip descriptions, and historical clip descriptions.

[0170] The event analysis and causation module 505 is used to analyze video segments based on video segment descriptions, short-term analysis information, and long-term memory information to obtain analysis results. The analysis results are used to indicate the event category of the video segment and the basis for decision-making.

[0171] In one possible implementation, the event question determination module 502 can also be used to: for each video segment, determine a predicted event based on historical segment descriptions; select an event verification question corresponding to the predicted event from a preset question library, wherein the question library includes a set of event verification questions for multiple event categories, and the set of event verification questions for each event category is determined based on the text descriptions of multiple historical event videos under that event category.

[0172] In one possible implementation, the event question determination module 502 can also be used to: determine multiple event categories; for each event category, generate a set of event verification questions corresponding to the event category based on a large language model and a preset knowledge base, wherein the preset knowledge base includes text descriptions of multiple historical event videos corresponding to multiple event categories; and construct a question library based on the set of event verification questions corresponding to all event categories.

[0173] In one possible implementation, the short-term and long-term information analysis module 504 can also be used to: integrate the video segment description and the historical segment description to obtain a spatiotemporal scene diagram, which is used to indicate the short-term spatiotemporal context of the video segment; determine short-term analysis information based on the spatiotemporal scene diagram; determine the key information of the video segment based on the question answer, key objectives, and video segment description; and store the key information in a clue library to form long-term memory information. The clue library is used to integrate and store the key information of the video segment and historical video segments, where historical video segments are segments preceding the target video segment.

[0174] In one possible implementation, the short-term and long-term information analysis module 504 can also be used to: perform a preliminary analysis of the video segment based on the question answer, key objectives, and video segment description to obtain a short-term decision. The short-term decision is used to indicate whether the video segment contains a target event and a list of event categories when the video segment contains a target event. Correspondingly, short-term analysis information is determined based on the spatiotemporal scene diagram, including: determining short-term analysis information based on the spatiotemporal scene diagram and the short-term decision.

[0175] In one possible implementation, the event analysis and cause determination module 505 can also be used to: determine the initial event evaluation score of each video segment based on the analysis results corresponding to each video segment in the target video; for each video segment, determine the similarity and distance relationship between the video segment and all other video segments in the target video; determine the association weight between the video segment and all other video segments in the target video based on the similarity and distance relationship between the video segment and all other video segments in the target video; take other video segments whose association weights satisfy the preset weight conditions as similar segments of the video segment to obtain a set of similar segments; and determine the target event evaluation score of the video segment based on the initial event evaluation score of each similar segment in the set of similar segments and the association weight between the video segment and each similar segment.

[0176] In one possible implementation, the event analysis and cause determination module 505 can also be used to: perform temporal smoothing on the evaluation score sequence to obtain a segment-level evaluation score sequence, the evaluation score sequence including the event target evaluation scores of all video segments in the target video; map the segment-level evaluation score sequence to each frame of the target video to generate a frame-level evaluation score curve of the target video.

[0177] The multi-agent collaborative device for multimodal perception provided in this embodiment can execute the method provided in the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0178] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0179] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0180] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0181] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0182] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0183] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0184] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0185] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0186] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0187] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0188] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0191] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0193] It should be understood that the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover but not exclude inclusion. For example, a product or device that includes a series of components is not necessarily limited to those components that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0194] As used in this application, the term "module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0195] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A multi-agent cooperative method for multimodal perception, characterized in that, include: Identify multiple video segments of the target video; For each video segment, based on historical segment descriptions and a pre-set question library, a predicted event and an event verification question corresponding to the predicted event are determined. The historical segment description is a description of the segments preceding the video segment in the target video. Based on the event verification question, guide the agent to perform visual information analysis on the video segment to determine the question answer, key objectives, and video segment description of the video segment; Based on the answers to the questions, the key objectives, the video clip descriptions, and the historical clip descriptions, short-term analysis information and long-term memory information are determined. The video segment is analyzed based on the video segment description, short-term analysis information, and long-term memory information to obtain analysis results. The analysis results are used to indicate the event category of the video segment and the basis for decision-making.

2. The method according to claim 1, characterized in that, For each video segment, based on historical segment descriptions and a pre-set question library, the process of determining a predicted event and the corresponding event verification question includes: For each video segment, the predicted event is determined based on historical segment descriptions; Select an event verification question corresponding to the predicted event from a preset question library. The question library includes a set of event verification questions for multiple event categories. The set of event verification questions for each event category is determined based on the text descriptions of multiple historical event videos under that event category.

3. The method according to claim 1, characterized in that, Before determining the predicted event and the event verification question corresponding to the predicted event for each video segment based on historical segment descriptions and a preset question library, the method further includes: Identify multiple event categories; For each event category, a set of event verification questions corresponding to the event category is generated based on a large language model and a preset knowledge base. The preset knowledge base includes text descriptions of multiple historical event videos corresponding to multiple event categories. A question library is built based on the set of event verification questions corresponding to all event categories.

4. The method according to claim 1, characterized in that, The determination of short-term analysis information and long-term memory information based on the question answers, the key objectives, the video segment descriptions, and the historical segment descriptions includes: The video segment description and the historical segment description are combined to obtain a spatiotemporal scene diagram, which is used to indicate the short-term spatiotemporal context of the video segment. Based on the spatiotemporal scene diagram, short-term analysis information is determined; Based on the answers to the questions, the key objectives, and the descriptions of the video segments, the key information of the video segments is determined. The key information is stored in a clue database to form long-term memory information. The clue database is used to integrate and store key information of the video clip and historical video clips. The historical video clips are segments in the target video that precede the video clip.

5. The method according to claim 4, characterized in that, Before determining short-term analysis information based on the spatiotemporal scene diagram, the method further includes: Based on the answers to the questions, the key objectives, and the video clip descriptions, a preliminary analysis of the video clips is performed to obtain short-term decisions. These short-term decisions are used to indicate whether a target event exists in the video clip and to provide a list of event categories when a target event exists in the video clip. Correspondingly, The step of determining short-term analysis information based on the spatiotemporal scene diagram includes: Based on the spatiotemporal scene diagram and the short-term decision, short-term analysis information is determined.

6. The method according to any one of claims 1-5, characterized in that, After analyzing the video segment based on the video segment description, short-term analysis information, and long-term memory information to obtain the analysis result, the method further includes: Based on the analysis results corresponding to each video segment in the target video, determine the initial event evaluation score for each video segment; For each video segment, determine the similarity and distance relationships between the video segment and all other video segments in the target video; Based on the similarity and distance relationships between the video segment and all other video segments in the target video, the association weight between the video segment and all other video segments in the target video is determined. Other video segments whose association weights satisfy the preset weight conditions are taken as similar segments of the video segment, and a set of similar segments is obtained; The event target evaluation score of the video segment is determined based on the initial event evaluation score of each similar segment in the set of similar segments and the association weight between the video segment and each similar segment.

7. The method according to claim 6, characterized in that, The method further includes: The evaluation score sequence is subjected to temporal smoothing to obtain a segment-level evaluation score sequence, which includes the event target evaluation scores of all video segments in the target video; The segment-level evaluation score sequence is mapped to each frame of the target video to generate the frame-level evaluation score curve of the target video.

8. A multi-agent cooperative system for multimodal perception, characterized in that, include: The video sampling module is used to determine multiple video segments of the target video; The first intelligent agent is used to determine, for each video segment, a predicted event and an event verification question corresponding to the predicted event, based on historical segment descriptions and a preset question library. The historical segment descriptions are descriptions of segments preceding the video segment in the target video. The second intelligent agent is used to guide the intelligent agent to perform visual information analysis on the video segment based on the event verification question of the first intelligent agent, and to determine the question answer, key target and video segment description of the video segment; A third intelligent agent is used to determine short-term analysis information and long-term memory information based on the question answers, key objectives and video clip descriptions of the second intelligent agent, as well as the historical clip descriptions. A fourth intelligent agent is used to analyze the video segment based on the video segment description, the short-term analysis information, and the long-term memory information to obtain analysis results. The analysis results are used to indicate the event category and decision basis of the video segment.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, are used to implement the method as described in any one of claims 1-7.

Images