Multi-agent collaborative situation awareness method based on large language model and multi-modal data

Through the large language model and multi-agent collaborative situation awareness method, the efficiency and scalability problems of the traditional multi-modal situation awareness method when processing multi-source multi-modal data are solved, and efficient and accurate situation awareness and resource optimization are achieved. Enhance the transparency of decision-making.

CN119066608APending Publication Date: 2024-12-03NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411076151.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

When traditional multimodal situational awareness methods process large amounts of real-time multimodal data, it is difficult to balance efficient processing and accurate perception. In addition, deep learning methods require a large amount of labeled data and high computing costs, resulting in poor scalability and large resource consumption. It is difficult to achieve real-time processing.

Method used

Using a multi-agent collaborative situational awareness method based on large language model and multi-modal data, through the multi-agent system and large-modal prompt word template, the design adapter converts multi-modal data into text frame input, and uses the summary chain system to generate comprehensive Situational awareness realizes data optimization and structured processing.

Benefits of technology

It improves the generality of large models and long-term strategic planning, reduces dependence on additional data and training, enhances the transparency and interpretability of decisions, and achieves efficient and accurate situational awareness and resource utilization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066608A_ABST
    Figure CN119066608A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-agent cooperative situation awareness method based on a large language model and multi-modal data, and the method comprises the steps: defining a multi-agent part observable Markov decision process according to a situation awareness task, and the process comprises an action space and a state space; designing a text situation awareness system, namely an adapter for observing texts in a multi-modal manner; performing data optimization, information compression and structuring on the text frame data; compiling a situation awareness large model cue word template; designing an agent and text frame observation interaction method; a multi-agent collaborative environment perception method is designed, and the understanding of a large language model on the whole environment is realized by summarizing different observations. According to the method, the problems of processing of multi-source and multi-modal data and fineness of microscopic perception of the large language model in a complex environment situation perception task are effectively solved, the long-term strategic perception capability and interpretability of the large language model are improved, and the dependency of the large language model on additional data and training is relieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and multi-agent collaborative control systems, and in particular to a multi-agent collaborative situation awareness method based on a large language model and multimodal data. Background Art

[0002] Situation awareness is the ability to perceive and understand key factors in the environment. It refers to the recognition of targets in the environment, the understanding of their meaning, and the prediction of their recent status. It is the ability to gain insight into security risks from the overall perspective based on the dynamics of the environment. Single-modal perception usually relies on a single data source, such as visual, auditory or temperature sensors, which are easily affected by environmental factors, resulting in inaccurate or incomplete perception results. In today's era of information explosion, a large amount of data from different sources, such as text, images, videos and other sensor data, is being generated at an unprecedented speed and scale. These data have also further promoted the transformation of traditional single-modal situation awareness methods to multi-modal situation awareness methods. Multi-modal situation awareness is based on richer and more comprehensive information, which greatly improves the accuracy and reliability of the perception system. The design of an adapter from multi-modal observation to text observation solves the problem that the original observation information of multi-source and multi-modality is difficult to be uniformly processed. Finally, an innovative solution that is more suitable for multi-modal multi-agent collaborative situation awareness tasks based on large language models is obtained.

[0003] Multimodal situational awareness plays an important role in modern technology and applications. Its application range is wide, covering multiple fields such as military, intelligent transportation and industrial automation. In the military field, multimodal situational awareness systems can achieve comprehensive monitoring and analysis of battlefield situations by integrating multimodal data such as video, radar, and infrared, helping commanders obtain more comprehensive battlefield information in real time and improve the accuracy and response speed of battlefield decision-making. In intelligent transportation systems, through data such as video surveillance, vehicle sensors and GPS, multimodal situational awareness can monitor traffic and predict traffic flow in real time, optimize the control of traffic signals, reduce the occurrence of traffic accidents, and enable the construction of smart and civilized cities. In industrial automation, multimodal situational awareness systems can monitor the production and operation status of equipment in real time by integrating data from multiple sensors such as vibration, temperature, and current, give equipment production suggestions, and infer the probability of equipment failure, thereby improving equipment production efficiency and reducing losses caused by equipment failure.

[0004] However, multimodal situational awareness also faces many challenges. Multimodal data sources provide rich information, but the heterogeneity and high dimensionality of these data also bring huge challenges. How to extract meaningful information from these multi-source data and form a comprehensive understanding of the environment has become a key issue in the field of situational awareness. Traditional multimodal data uses data fusion methods to understand multimodal data. Data fusion methods usually include low-level sensor data fusion, mid-level feature fusion, and high-level decision fusion. Low-level fusion directly processes raw sensor data and integrates multi-source data through weighted average, Kalman filtering and other methods. It is suitable for real-time processing in simple environments. However, low-level fusion is highly sensitive to sensor noise and uncertainty, which can easily lead to unstable fusion results. Mid-level feature fusion is fused after the feature extraction of each sensor, and statistical methods are used to integrate feature information. This method can effectively reduce data redundancy while maintaining data diversity, but it faces the challenges of high-dimensional data processing and feature selection. High-level decision fusion comprehensively judges the final situation by fusing the independent decision results of each sensor, such as voting method, Bayesian network, etc. Although this method simplifies the data processing process, the quality of decision-making depends on the performance of a single sensor. Once an error occurs in a sensor, the overall decision may be seriously affected.

[0005] Traditional data fusion methods have difficulty in achieving a balance between efficient processing and accurate perception when faced with a large amount of real-time multimodal data, resulting in insufficient real-time and robustness in practical applications. In response to the above problems, artificial intelligence-related technologies have been introduced into multimodal situational awareness methods. Learning is a very important concept in artificial intelligence technology. With the help of artificial intelligence technology, multimodal situational awareness methods can obtain the ability to efficiently and accurately process a large amount of real-time multimodal data by learning to fully understand the large amount of complex data prepared in advance.

[0006] Therefore, deep learning technology, especially convolutional neural networks and recurrent neural networks, has begun to be applied in multimodal situational awareness systems. These technologies have achieved remarkable results in single-modal tasks such as computer vision and natural language processing, demonstrating their strong capabilities in feature extraction and pattern recognition.

[0007] However, these deep learning methods usually require a large amount of labeled data for training, and acquiring and labeling multimodal data is both expensive and time-consuming, which limits the widespread promotion of these methods in practical applications. At the same time, deep learning methods need a large amount of data of different types for training when facing different types of input data. When new data types are added, the model must be retrained, resulting in poor scalability of the system. The high computational cost of deep learning models also makes it difficult to achieve real-time processing in resource-constrained environments, which is an important problem in situational awareness applications that require rapid response.

[0008] In recent years, large language model technology has developed rapidly and has become a hot topic in artificial intelligence technology. These large models have learned to extract rich contextual information and deep features from data of different modalities through pre-training on massive data, and then adapted to specific tasks through fine-tuning. This process not only reduces the need for a large amount of labeled data, but also significantly improves the versatility and adaptability of the model. Large models can process multiple types of data such as images, text, and audio at the same time, and use their powerful representation capabilities and complex pattern recognition capabilities to automatically fuse and interpret these data, thereby providing more accurate and comprehensive situational awareness. However, the multimodal situational awareness method based on fine-tuning of large models still requires a certain amount of data and training costs, and still faces defects such as high performance overhead and poor scalability when facing flexible and changeable application scenarios. Summary of the invention

[0009] The purpose of the present invention is to provide a multi-agent collaborative situation awareness method based on a large language model and multimodal data to address the problems existing in the above-mentioned prior art. The multi-agent system is innovatively introduced into the situation awareness process of the large language model, avoiding the problems of the large model's low processing ability for multi-source multimodal data, poor long-term strategic planning, lack of interpretability, etc. The powerful natural language processing and understanding capabilities of the large language model also bring significant benefits to multi-agent collaborative decision-making, including higher decision-making efficiency and stronger environmental adaptability.

[0010] The technical solution to achieve the purpose of the present invention is a multi-agent collaborative situation awareness method based on a large language model and multimodal data, the method comprising the following steps:

[0011] Step 1: Define a multi-agent partially observable Markov decision model based on the situation awareness task;

[0012] Step 2: Design and initialize the adapter from multimodal observation to text observation to convert different inputs into text frame inputs;

[0013] Step 3: Summarize and abstract the text frame data, extract key information, and optimize the data;

[0014] Step 4, design a large model prompt word template for multi-agent collaborative situation awareness;

[0015] Step 5: Initialize the summary chain system, and the interaction between the single agent and the environment generates a local situation overview;

[0016] Step 6: Multiple agents collaborate through the summary chain system to generate comprehensive situational awareness.

[0017] Furthermore, step 1 specifically includes:

[0018] Step 1-1, formulate the situation awareness task based on the real environment into a multi-agent partially observable Markov decision process, which is represented as a six-tuple<S,A,R,Ω,T,O> , where S represents the state space, A represents the action space, R represents the reward function, Ω represents the observation space, T represents the state transfer function, and O represents the text frame observation;

[0019] Step 1-2, determine the state space and observation space of the agent's situation awareness task;

[0020] The state space S is the set of all possible states that the system can be in;

[0021] The observation space Ω includes all observation information I obtained by the system from the environment. In the text situation awareness system, the observation is finally expressed in the form of text frame o, and forms a text frame observation O = {o1, o2, ..., o n}, n ≥ 1, reflecting the key information and dynamic changes in the environment, n represents the number of text frames; the text frame observation O∈Ω contains the target point, building, and mobile obstacle information; the text frame observation O∈V K Represented by natural language sentences, where V is a natural language vocabulary containing a finite number of words;

[0022] Step 1-3, determine the action space that can be taken in the situation awareness task A = {continue monitoring, key information extraction, information aggregation, situation analysis}, where continued monitoring means continuing to monitor and observe the environment, the system continues to collect data from the environment, and updates the current situation cognition; key information extraction means screening and extracting key data points from observations; information aggregation means aggregating and integrating information collected from multiple observations to form a comprehensive understanding of the current environmental situation; situation analysis means situation analysis of the current state, long-term and strategic analysis of future trends, identification of potential threats, opportunities or changes, and serving the overall situation analysis;

[0023] Determine the state transfer function T, when the text frame is observed o t After taking any action, the current observation will be transferred to the next text frame observation o t+1 , until all text frame observations are traversed and the terminal state is entered;

[0024] Step 1-4, determine the reward function R(s,a) of the situation awareness task, which is the reward obtained by the system after completing the situation awareness task; the reward function is set to {-1,0,1}, which respectively represent inaccurate, partially accurate and completely accurate states.

[0025] Furthermore, step 2 specifically includes:

[0026] Step 2-1, define an adapter F for converting multimodal observation I into text frame observation O, expressed as:

[0027] F:I→O

[0028] Among them, the multimodal observation I is the original environmental observation data, and the text frame observation O is the text observation data converted by the adapter F from the multimodal observation I, which consists of natural language descriptions. Each text frame has a time tag t, indicating its spatiotemporal relationship with other text frames;

[0029] Step 2-2, modeling the spatiotemporal relationship of multimodal observations I to generate a spatiotemporal relationship graph G;

[0030] Step 2-3, use adapter F to convert the spatiotemporal relationship graph G into a natural language spatiotemporal text frame observation O to achieve text processing and unified format output.

[0031] Furthermore, the input form of the multimodal observation I is the real-time video I of multiple source cameras. cam 、Video File I video , multiple pictures I multi_image , Single Image I image 、Text I text And composite input of pictures and text image_text One of:

[0032] I∈{I cam ,I video ,I milti_image ,I image ,I text, I image_text}.

[0033] Furthermore, step 2-2 models the spatiotemporal relationship of the multimodal observation I to generate a spatiotemporal relationship graph G, which specifically includes:

[0034] (1) Real-time video from multiple cameras cam , split it into multiple video frames F i,t , where i and t represent the video frames of the i-th camera at time t; for each video frame F i,t , extract its timestamp as the timestamp t, and record the specific time position of the frame in the video stream Timestamp(F i,t ) = t, and then integrate the video frames of all cameras at time t to form the total video frame F at time t. t =∪ i F i,t ; Then for each total video frame F t Analyze the scene in the scene and mark the overall situation of important objects SituationInfo (Ft )={(Object j ,Position j ,State j )}, where Object j Represents important object j, Position j With State j Respectively represent the position and posture information of important object j; summarize the data of each video frame, and construct a spatiotemporal relationship diagram based on the spatiotemporal relationship of the video frames:

[0035] G={(t,SituationInfo(F t ))}

[0036] (2) For video file I video , split it into multiple video frames F t , where t represents the relative time position of the video frame in the video, and this is used as its time mark t; then for each video frame F t Analyze the scene in the scene and mark the overall situation of important objects SituationInfo (F t )={(Object j ,Position j ,State j )}, where Object j Represents important object j, Position j With State j Respectively represent the position and posture information of important object j; summarize the data of each video frame, and construct a spatiotemporal relationship diagram based on the spatiotemporal relationship of the video frames:

[0037] G={(t,SituationInfo(F t ))}

[0038] (3) For multiple images I milti_image , which is considered as a set of pictures {P1,P2,...,P n}, P n Represents the nth picture. For a single picture I image , regard it as a picture P; analyze the spatiotemporal relationship between pictures, mark each picture with a relative time tag t, and thus regard each picture as P t , record the time series Timestamp (P t )=t; then for P t , extract the key information KeyInfo(P t )={(Object j ,Position j)}; Combine the parsed spatiotemporal information to construct a spatiotemporal relationship diagram:

[0039] G={(t,KeyInfo(P t ))}

[0040] If it is a single image P, just give the information pair (t, KeyInfo(P));

[0041] (4) For text input I text , split it into text descriptions of different time periods {T1,T2,...,T n}, and mark it with a relative time tag t, so that the text description of each time period is regarded as T t , then parse the text T t Key event information EventInfo(T t )={(Event k ,Object j )}, where Event k Represents the kth event in the text description; from this, a spatiotemporal relationship graph is constructed:

[0042] G={(t,EventInfo(T t ))}

[0043] (5) Composite input of image and text I image_text , consider the picture as a picture set {P1,P2,...,P n} or a single image P, split the text into text descriptions of different time periods {T1,T2,...,T n}, and then combine the two through spatiotemporal relationship analysis to form a picture-text pair U describing the environmental information at a certain time t t ={(P t ,T t )}, and use t as U t time tag; integrate visual information and text description to generate comprehensive information MixInfo (U t )={(Event k ,Object j ,Position j )}, construct a spatiotemporal relationship diagram:

[0044] G={(t,MixInfo(U t ))}.

[0045] Furthermore, steps 2-3 specifically include:

[0046] For each node (t,Info t), and convert the node into a standard description information Description(V,t,Info t ) and use it as a text frame o t , and then form a text frame observation O = {o1,o2,...,o n}, n represents the number of text frames; where Info t Represents SituationInfo(F t )、KeyInfo(P t )、EventInfo(T t ) and MixInfo(U t ).

[0047] Furthermore, step 3 specifically includes:

[0048] Step 3-1, preliminarily filter the text data generated in step 2 to remove redundant information and irrelevant noise data;

[0049] Step 3-2, further extracting key information from the filtered text data; specifically including:

[0050] Extract key game information and events from the filtered text data, including main objects, events, time nodes and dynamic changes. By analyzing the information in each text frame, extract the content that is crucial to the situation awareness task, and thus obtain the compressed text frame observation O k ;

[0051] Based on the compressed text frame observation k , identify and mark the main objects, events, time nodes and dynamic changes related to the situation awareness task, and obtain the marked text frame observation O label ;

[0052] Step 3-3, observe the marked text frame label Perform structural processing to convert the marked key information into a unified, easy-to-understand and easy-to-analyze structured text format to obtain a structured text frame observation. structure , as a text frame observation O for summarizing the chain system.

[0053] Furthermore, step 3-1 specifically includes:

[0054] Treat text frame observation O as the initial text frame observation O inital , for O inital Perform noise filtering, identify and delete repeated information, irrelevant content and noise data, and obtain the filtered text frame observation O filter ;

[0055] Observe the filtered text frame O filter Merge text frames with similar or repeated contents to summarize similar or closely related information and generate a preliminary summary text frame containing complete information. summarized ;

[0056] The preliminary summary text frame observation O summarized Observe O with the initial text frame inital Integrate to obtain the integrated text frame observation O integrate .

[0057] Furthermore, step 5 specifically includes:

[0058] Step 5-1, initialize the summary chain system, create a frame queue FQ for multi-text frame summarization, and set the hyperparameter K, that is, the length of the frame queue FQ;

[0059] Step 5-2, the agent observes and summarizes a single text frame, including:

[0060] Based on the multi-agent partially observable Markov decision model, the agent obtains text frame observations O from the environment through the adapter F of multimodal observation to text frame observations, and these observations are obtained through the environment state S; the agent realizes its interaction with the environment S that cannot be directly observed through the interaction with the text frame observations O;

[0061] The agent selects the text frame o according to the action space A t Then, the single-frame processing function D(.) is used to process the text frame o. t The process is expressed as:

[0062]

[0063] Among them, the text frame o t is the agent's observation at time t, It is the processed text frame data;

[0064] The processed text frame data Put it into the frame queue FQ;

[0065] Step 5-3, determine whether the length of the current frame queue FQ reaches the preset hyperparameter K, if so, execute step 5-4, otherwise return to execute step 5-2;

[0066] Step 5-4: The agent summarizes the multiple text frames and generates a local situation overview of the environment; specifically, it includes:

[0067] Use the aggregation function LA to summarize the text frames in FQ in multiple frames:

[0068] LSt =LA(FQ,promptsL)

[0069] Among them, LS t is the situation summary of the agent at time t, promptsL is the prompt word information for the local situation summary of the single agent, which is obtained from the large model prompt word template described in step 4;

[0070] Step 5-5, clear some old data in FQ.

[0071] Furthermore, step 6 specifically includes:

[0072] Step 6-1, create a comprehensive situation overview queue GQ, and set the length of the situation overview queue GQ through the hyperparameter GK;

[0073] Step 6-2: In the multi-agent collaboration stage, each agent i Initialize the adapter in step 1 and get its text frame observation O i ; Each agent i Observe it with the text frame i The processed text frame observations obtained after the interaction are stored in their respective frame queues FQ i In, and when FQ i Generate situational awareness summary LS when reaching a certain length t,i ;

[0074] The system will each agent i Overview of the local situation LS t,i Synthesize into a global situation overview information:

[0075]

[0076] Among them, GS t is the global situation overview information, N is the number of agents;

[0077] GS t Put it into the comprehensive situation overview queue GQ;

[0078] Step 6-3, when the comprehensive situation overview queue GQ reaches the hyperparameter GK, the overall aggregation function GA integrates the comprehensive situation overview queue to generate the final holistic comprehensive situation awareness information:

[0079] GSW t =GA{GQ,promptsG}

[0080] Among them, GSW tis the comprehensive situation awareness information generated by the large language model, promptsG is the prompt word information for multi-agent comprehensive situation awareness, obtained from the large model prompt word template described in step 4;

[0081] Step 6-4, clear some old data in GQ;

[0082] Step 6-5: Refine the overall GSW information t , divide it into the following three parts and output:

[0083] Environmental analysis: The system integrates the observation data of each agent to generate an overall analysis of the current state of the environment, including key changes, potential threats and opportunities;

[0084] Enemy analysis: Based on the collaborative observation of multiple agents, the system evaluates the enemy's strategy, intentions and behavior;

[0085] Recommendations: The big language model gives comprehensive strategic recommendations that will support decision making, optimize resource allocation and action plans.

[0086] Compared with the prior art, the present invention has the following significant advantages:

[0087] (1) The present invention introduces a multi-agent system to process multi-source and multi-modal data, thereby enhancing the versatility of large models.

[0088] (2) The present invention does not rely on additional data and training for large models, thereby improving efficiency for different environments and tasks.

[0089] (3) The present invention enhances the long-term strategic planning of large models when analyzing data, and no longer focuses only on the current moment.

[0090] (4) The present invention can help explain the decision-making process of intelligent agents and large models, making the decisions more transparent and explainable.

[0091] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 The present invention is a flowchart of a multi-agent collaborative situation awareness method based on a large language model and multimodal data.

[0093] Figure 2 A flowchart of the present invention using a large language model and a multi-agent system to make decisions for multimodal situational awareness.

[0094] Figure 3 It is a schematic diagram of the system framework icon of the multi-agent collaborative situation awareness method based on a large language model and multimodal data of the present invention. DETAILED DESCRIPTION

[0095] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0096] It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0097] In response to the problems of low multi-source and multi-modal data processing capabilities, poor long-term strategic planning, and reliance on additional data and training in traditional large-model situation awareness, the present invention proposes a multi-agent collaborative situation awareness method based on a large language model and multi-modal data, providing a more transparent and explainable approach.

[0098] In one embodiment, in combination Figure 1 The multi-agent collaborative situation awareness method based on a large language model and multimodal data comprises the following steps:

[0099] Step 1: Define a multi-agent partially observable Markov decision model based on the situation awareness task;

[0100] Step 2, design and initialize the adapter from multimodal observation to text observation, i.e., the text situation awareness system, to convert different inputs into text frame inputs;

[0101] Step 3: Summarize and abstract the text frame data, extract key information, and optimize the data;

[0102] Step 4, design a large model prompt word template for multi-agent collaborative situation awareness;

[0103] Step 5: Initialize the summary chain system, and the interaction between the single agent and the environment generates a local situation overview;

[0104] Step 6: Multiple agents collaborate through the summary chain system to generate comprehensive situational awareness.

[0105] Furthermore, in one embodiment, step 1 specifically includes:

[0106] Step 1-1, formulate the situation awareness task based on the real environment into a multi-agent partially observable Markov decision process, which is represented as a six-tuple<S,A,R,Ω,T,O> , where S represents the state space, A represents the action space, R represents the reward function, Ω represents the observation space, T represents the state transfer function, and O represents the text frame observation;

[0107] Step 1-2, determine the state space and observation space of the agent's situation awareness task;

[0108] The state space S is the set of all possible states that the system can be in; these states are usually invisible to the agent but are inferred and updated through observation data and historical information;

[0109] The observation space Ω includes all observation information I obtained by the system from the environment. In the text situation awareness system, the observation is finally expressed in the form of text frame o, and forms a text frame observation O = {o1, o2, ..., o n}, n ≥ 1, reflecting the key information and dynamic changes in the environment, n represents the number of text frames; the text frame observation O∈Ω includes target points, buildings, mobile obstacles, etc.; the text frame observation O∈V K Represented by natural language sentences, where V is a natural language vocabulary containing a finite number of words;

[0110] Step 1-3, determine the action space that can be taken in the situation awareness task A = {continue monitoring, key information extraction, information aggregation, situation analysis}, where continued monitoring means continuing to monitor and observe the environment, the system continues to collect data from the environment, updates the current situation cognition, and ensures real-time and accuracy; key information extraction means screening and extracting key data points from the observation for subsequent analysis; information aggregation means aggregating and integrating information collected from multiple observations to form a comprehensive understanding of the current environmental situation, reduce information redundancy and enhance the comprehensiveness of perception; situation analysis means situation analysis of the current state, long-term and strategic analysis of future trends, identification of potential threats, opportunities or changes, and serving the overall situation analysis;

[0111] Determine the state transfer function T, when the text frame is observed o t After taking any action, the current observation will be transferred to the next text frame observation o t+1 , until all text frame observations are traversed and the terminal state is entered;

[0112] Step 1-4, determine the reward function R(s,a) of the situation awareness task, which is the reward obtained by the system after completing the situation awareness task; the reward function is set to {-1,0,1}, which respectively represent inaccurate, partially accurate and completely accurate states.

[0113] Further, in one embodiment, in combination Figure 3 , step 2 specifically includes:

[0114] Step 2-1, define an adapter F for converting multimodal observation I into text frame observation O, expressed as:

[0115] F:I→O

[0116] Among them, the multimodal observation I is the original environmental observation data, and the text frame observation O is the text observation data converted by the adapter F from the multimodal observation I, which consists of natural language descriptions. Each text frame has a time tag t, indicating its spatiotemporal relationship with other text frames;

[0117] Step 2-2, modeling the spatiotemporal relationship of multimodal observations I to generate a spatiotemporal relationship graph G;

[0118] Step 2-3, use adapter F to convert the spatiotemporal relationship graph G into a natural language spatiotemporal text frame observation O to achieve text processing and unified format output.

[0119] Here, in order to achieve multimodal situational awareness, the present invention designs an adapter to convert different types of input data into a unified text frame input. The design of the adapter needs to consider the spatiotemporal relationship of the data to ensure that the generated text frame can accurately reflect the key information in the environment. For example, key frames extracted from videos can be converted into descriptive text, and sensor data can be converted into structured text descriptions. Such conversion simplifies the processing of multimodal data and improves the interpretability of the data.

[0120] Here, in some embodiments, the input form of the multimodal observation I is real-time video I from multiple source cameras. cam 、Video File I video , multiple pictures I multi_image , Single Image I image 、Text I text And composite input of pictures and text image_text One of:

[0121] I∈{I cam ,I video ,I milti_image ,I image ,I text, I image_text}.

[0122] Here, in some embodiments, the step 2-2 of modeling the spatiotemporal relationship of the multimodal observation I to generate the spatiotemporal relationship graph G specifically includes:

[0123] (1) Real-time video from multiple cameras cam , split it into multiple video frames F i,t , where i and t represent the video frames of the i-th camera at time t; for each video frame F i,t , extract its timestamp as the timestamp t, and record the specific time position of the frame in the video stream Timestamp(F i,t ) = t, and then integrate the video frames of all cameras at time t to form the total video frame F at time t. t =∪ i F i,t ; Then for each total video frame F t Analyze the scene in the scene and mark the overall situation of important objects SituationInfo (F t )={(Object j ,Position j ,State j )}, where Object j Represents important object j, Position j With State j Respectively represent the position and posture information of important object j; summarize the data of each video frame, and construct a spatiotemporal relationship diagram based on the spatiotemporal relationship of the video frames:

[0124] G={(t,SituationInfo(F t ))}

[0125] (2) For video file I video , split it into multiple video frames F t , where t represents the relative time position of the video frame in the video, and this is used as its time mark t; then for each video frame F t Analyze the scene in the scene and mark the overall situation of important objects SituationInfo (F t )={(Object j ,Position j ,State j )}, where Object j Represents important object j, Position j With State jRespectively represent the position and posture information of important object j; summarize the data of each video frame, and construct a spatiotemporal relationship diagram based on the spatiotemporal relationship of the video frames:

[0126] G={(t,SituationInfo(F t ))}

[0127] (3) For multiple images I milti_image , which is considered as a set of pictures {P1,P2,...,P n}, P n Represents the nth picture. For a single picture I image , regard it as a picture P; analyze the spatiotemporal relationship between pictures, mark each picture with a relative time tag t, and thus regard each picture as P t , record the time series Timestamp (P t )=t; then for P t , extract the key information KeyInfo(P t )={(Object j ,Position j )}; Combine the parsed spatiotemporal information to construct a spatiotemporal relationship diagram:

[0128] G={(t,KeyInfo(P t ))}

[0129] If it is a single image P, just give the information pair (t, KeyInfo(P));

[0130] (4) For text input I text , split it into text descriptions of different time periods {T1,T2,...,T n}, and mark it with a relative time tag t, so that the text description of each time period is regarded as T t , then parse the text T t Key event information EventInfo(T t )={(Event k ,Object j )}, where Event k Represents the kth event in the text description; from this, a spatiotemporal relationship graph is constructed:

[0131] G={(t,EventInfo(T t ))}

[0132] (5) Composite input of image and text I image_text , consider the picture as a picture set {P1,P2,...,P n} or a single image P, split the text into text descriptions of different time periods {T1,T2,...,T n}, and then combine the two through spatiotemporal relationship analysis to form a picture-text pair U describing the environmental information at a certain time t t ={(P t ,T t )}, and use t as U t time tag; integrate visual information and text description to generate comprehensive information MixInfo (U t )={(Event k ,Object j ,Position j )}, construct a spatiotemporal relationship diagram:

[0133] G={(t,MixInfo(U t ))}.

[0134] Here, in some embodiments, steps 2-3 specifically include:

[0135] For each node (t,Info t ), and convert the node into a standard description information Description(V,t,Info t ) and use it as a text frame o t , and then form a text frame observation O = {o1,o2,...,o n}, for subsequent summary chain operations and system-level information processing, n represents the number of text frames; where Info t Represents SituationInfo(F t )、KeyInfo(P t )、EventInfo(T t ) and MixInfo(U t ).

[0136] Furthermore, combined with Figure 2 , in step 3 specifically including:

[0137] Step 3-1, preliminarily filter the text data generated in step 2 to remove redundant information and irrelevant noise data; specifically including:

[0138] Treat text frame observation O as the initial text frame observation O inital , for O inital Perform noise filtering, identify and delete repeated information, irrelevant content and noise data, ensure that the remaining text frame information is concise and relevant, and obtain the filtered text frame observation O filter =Ffilter (O inital );

[0139] Observe the filtered text frame O filter Merge text frames with similar or repeated contents to summarize similar or closely related information and generate a preliminary summary text frame containing complete information. summarized =F summarized (O filter );

[0140] The preliminary summary text frame observation O summarized Observe O with the initial text frame inital Integrate to obtain the integrated text frame observation O integrate =F integrate (O inital ,O summarized ) to ensure the completeness and accuracy of information, to ensure that the information of each key point is complete and without omission, and to provide a basis for subsequent processing;

[0141] Step 3-2, further extracting key information from the filtered text data; specifically including:

[0142] Observe O from the integrated text frame integrate The key game information and events, including main objects, events, time nodes and dynamic changes, are extracted from the text frame, and the content that is crucial to the situation awareness task is extracted by analyzing the information in each text frame, thereby obtaining the compressed text frame observation O k ;

[0143] Based on the compressed text frame observation k , identify and mark the main objects, events, time nodes and dynamic changes related to the situation awareness task, ensure that each key information can be accurately identified and used, facilitate subsequent structured processing, and obtain the marked text frame observation O label ;

[0144] Step 3-3, observe the marked text frame label Perform structured processing to convert the marked key information into a unified, easy-to-understand and easy-to-analyze structured text format, ensuring that the final output text data is concise and efficient, and providing preparation for subsequent comprehensive analysis and situation understanding:

[0145] O structure =F structuret (O label )

[0146] After the above processing, we get the structured text frame observation O structure , as a text frame observation O for summarizing the chain system.

[0147] Further, in one embodiment, in combination Figure 3 , step 5 specifically includes:

[0148] Step 5-1, initialize the summary chain system, create a frame queue FQ for multi-text frame summarization, and set the hyperparameter K, that is, the length of the frame queue FQ, which determines how many text frames are relied on when generating local situation awareness;

[0149] Step 5-2, the agent observes and summarizes a single text frame, including:

[0150] Based on the multi-agent partially observable Markov decision model, the agent obtains text frame observations O from the environment through the adapter F of multimodal observation to text frame observations, and these observations are obtained through the environment state S; the agent realizes its interaction with the environment S that cannot be directly observed through the interaction with the text frame observations O;

[0151] The agent selects the text frame o according to the action space A t Then, the single-frame processing function D(.) is used to process the text frame o. t The process is expressed as:

[0152]

[0153] Among them, the text frame o t is the agent's observation at time t, It is the processed text frame data;

[0154] The processed text frame data Put it into the frame queue FQ;

[0155] Step 5-3, determine whether the length of the current frame queue FQ reaches the preset hyperparameter K, if so, execute step 5-4, otherwise return to execute step 5-2;

[0156] Step 5-4: The agent summarizes the multiple text frames and generates a local situation overview of the environment; specifically, it includes:

[0157] Using the aggregation function LA Multi-frame summary of text frames in:

[0158] LS t =LA(FQ,promptsL)

[0159] Among them, LS t is the situation summary of the agent at time t for its observation, promptsL is the prompt word information used for the local situation summary of the single agent, that is, the large model prompt word template described in step 4, which helps guide the aggregation and subsequent reasoning process;

[0160] Step 5-5, clearing a portion of old data in the FQ. Preferably, clearing two-thirds of the old data in the frame queue FQ and retaining one-third of the old data to ensure the consistency and longevity of the situation summary.

[0161] Further, in one embodiment, in combination Figure 3 , step 6 specifically includes:

[0162] Step 6-1, create a comprehensive situation overview queue GQ, and set the length of the situation overview queue GQ through the hyperparameter GK, which is used for the large language model to generate situation awareness of the overall environment;

[0163] Step 6-2: In the multi-agent collaboration stage, each agent i Initialize the adapter in step 1 and get its text frame observation O i ; Each agent i Observe it with the text frame i The processed text frame observations obtained after the interaction are stored in their respective frame queues FQ i In, and when FQ i Generate situational awareness summary LS when reaching a certain length t,i ;

[0164] The system will each agent i Overview of the local situation LS t,i Synthesize into a global situation overview information:

[0165]

[0166] Among them, GS t is the global situation overview information, N is the number of agents;

[0167] GS t Put it into the comprehensive situation overview queue GQ;

[0168] Step 6-3, when the comprehensive situation overview queue GQ reaches the hyperparameter GK, the overall aggregation function GA integrates the comprehensive situation overview queue to generate the final holistic comprehensive situation awareness information:

[0169] GSW t =GA{GQ,promptsG}

[0170] Among them, GSW t is the comprehensive situation awareness information generated by the large language model, and promptsG is the prompt word information used for multi-agent comprehensive situation awareness;

[0171] Step 6-4, clearing a portion of the old data in the GQ; preferably, clearing two-thirds of the old data in the frame queue GQ and retaining one-third of the old data to ensure the consistency and long-term nature of the situation overview.

[0172] Step 6-5: Refine the overall GSW information t , divide it into the following three parts and output:

[0173] Environmental analysis: The system integrates the observation data of each agent to generate an overall analysis of the current state of the environment, including key changes, potential threats and opportunities;

[0174] Enemy Analysis: Based on the collaborative observation of multiple agents, the system evaluates the enemy's strategy, intentions and behavior. This step deeply analyzes the enemy's action patterns and possible future actions.

[0175] Recommendations: The big language model gives comprehensive strategic recommendations that will support decision making, optimize resource allocation and action plans.

[0176] The present invention introduces a multi-agent collaborative auxiliary large model, and guides the large model through prompt, which is an innovative method. Specifically, multi-source multi-modal data is processed by an adapter and handed over to multi-agents for collaborative processing, and then the environmental information and self-state observed by the agent are input into the large language model through the prompt written by the large model prompt word template, and the model analyzes and guides the agent's actions. This method has the following advantages: 1. Multi-source data integration: The multi-modal situation awareness method can effectively integrate data from different sources, such as text, images, videos and sensor data. It improves the ability to fully understand the environment, and improves the accuracy and reliability of perception through the mutual complementation of multi-modal information. 2. No additional data and training are required: Through prompt engineering, the model can be effectively guided to process multi-modal data and perform situation awareness. This method is very low in cost, does not require any training data and performance cost, and can be input into the large model by processing the real-time input to a certain extent and converting it into prompt. At the same time, it is extremely flexible. For different types of input data, only the processing program of the input data needs to be changed, and the large model does not need to be trained again. It has strong scalability and low performance overhead. 3. Dynamic adaptability in complex environments: By combining the Markov decision process and the partially observable Markov decision process, effective decisions can be made under incomplete information and uncertainty. This makes the system highly adaptable and flexible in dynamic and complex environments, and can adjust strategies in real time according to environmental changes to ensure the timeliness and accuracy of decisions. 4. Multi-agent collaboration: The application of multi-agent systems enables multiple autonomous agents to work together in the same environment to complete complex tasks. Each agent can not only improve the efficiency of task completion by communicating and collaborating with each other, but also share information among multiple agents to form a comprehensive understanding of the overall situation. 5. Adapter design and data optimization: The adapter design converts different types of input data into a unified text frame input, and ensures that the generated text frame can accurately reflect the key information in the environment by considering the spatiotemporal relationship of the data. This design simplifies the processing flow of multimodal data and improves the interpretability of the data. In addition, the data optimization and structured processing steps remove redundant information and noise, extract key information, and convert it into a unified format for subsequent comprehensive analysis and decision support, thereby improving the overall performance of the system. 6. Summary chain system and long-term strategic perception: The summary chain system can summarize multiple text frame observations to generate local and global situation overviews. By setting the frame queue length and aggregation function, the system can generate a comprehensive understanding of the environment in multi-agent collaboration to support long-term strategic analysis and decision-making. The long-term strategic perception capability enables the system to not only accurately perceive the current situation, but also identify potential threats and opportunities, provide strategic advice, and optimize resource allocation and action plans.7. High interpretability: Through structured data processing and detailed situation overview, the system can generate output results that are easy to understand and interpret, enhancing users’ trust in system behavior and decisions. This is of great significance for user experience and system promotion in practical applications.

[0177] In one embodiment, a multi-agent collaborative situation awareness system based on a large language model and multimodal data is provided, the system comprising:

[0178] The first module is used to implement: defining a multi-agent partially observable Markov decision model based on the situation awareness task;

[0179] The second module is used to implement: design and initialize the adapter from multimodal observation to text observation, converting different inputs into text frame inputs;

[0180] The third module is used to achieve: summarizing and abstracting text frame data, extracting key information, and optimizing data;

[0181] The fourth module is used to implement: designing a large model prompt word template for multi-agent collaborative situation awareness;

[0182] The fifth module is used to implement: initialization of the summary chain system, the interaction between the single agent and the environment to generate a local situation overview;

[0183] The sixth module is used to achieve: multiple agents collaborate through the summary chain system to generate comprehensive situational awareness.

[0184] For the specific limitations of the multi-agent collaborative situational awareness system based on a large language model and multimodal data, please refer to the limitations of the multi-agent collaborative situational awareness method based on a large language model and multimodal data in the above text, which will not be repeated here. Each module in the above-mentioned multi-agent collaborative situational awareness system based on a large language model and multimodal data can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0185] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0186] Step 1: Define a multi-agent partially observable Markov decision model based on the situation awareness task;

[0187] Step 2: Design and initialize the adapter from multimodal observation to text observation to convert different inputs into text frame inputs;

[0188] Step 3: Summarize and abstract the text frame data, extract key information, and optimize the data;

[0189] Step 4, design a large model prompt word template for multi-agent collaborative situation awareness;

[0190] Step 5: Initialize the summary chain system, and the interaction between the single agent and the environment generates a local situation overview;

[0191] Step 6: Multiple agents collaborate through the summary chain system to generate comprehensive situational awareness.

[0192] For the specific limitations of each step, please refer to the above limitations on the multi-agent collaborative situational awareness method based on a large language model and multimodal data, which will not be repeated here.

[0193] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements:

[0194] Step 1: Define a multi-agent partially observable Markov decision model based on the situation awareness task;

[0195] Step 2: Design and initialize the adapter from multimodal observation to text observation to convert different inputs into text frame inputs;

[0196] Step 3: Summarize and abstract the text frame data, extract key information, and optimize the data;

[0197] Step 4, design a large model prompt word template for multi-agent collaborative situation awareness;

[0198] Step 5: Initialize the summary chain system, and the interaction between the single agent and the environment generates a local situation overview;

[0199] Step 6: Multiple agents collaborate through the summary chain system to generate comprehensive situational awareness.

[0200] For the specific limitations of each step, please refer to the above limitations on the multi-agent collaborative situational awareness method based on a large language model and multimodal data, which will not be repeated here.

[0201] The multi-agent collaborative situation awareness method based on large language models and multimodal data in the present invention not only solves the problem of large models processing multi-source multimodal data, enhances the long-term strategic planning of large model situation awareness tasks, but also greatly enhances the interpretability of processing results, effectively reduces the dependence of large language models on additional data and training, improves resource utilization efficiency, and provides new ideas and methods for the development of the field of situation awareness.

[0202] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-agent collaborative situation awareness method based on a large language model and multimodal data, characterized in that: The method comprises the following steps: Step 1: Define a multi-agent partially observable Markov decision model based on the situation awareness task; Step 2: Design and initialize the adapter from multimodal observation to text observation to convert different inputs into text frame inputs; Step 3: Summarize and abstract the text frame data, extract key information, and optimize the data; Step 4, design a large model prompt word template for multi-agent collaborative situation awareness; Step 5: Initialize the summary chain system, and the interaction between the single agent and the environment generates a local situation overview; Step 6: Multiple agents collaborate through the summary chain system to generate comprehensive situational awareness.

2. The multi-agent collaborative situation awareness method based on a large language model and multimodal data according to claim 1 is characterized in that: Step 1 specifically includes: Step 1-1, formulate the situation awareness task based on the real environment into a multi-agent partially observable Markov decision process, which is represented as a six-tuple<S,A,R,Ω,T,O> , where S represents the state space, A represents the action space, R represents the reward function, Ω represents the observation space, T represents the state transfer function, and O represents the text frame observation; Step 1-2, determine the state space and observation space of the agent's situation awareness task; The state space S is the set of all possible states that the system can be in; The observation space Ω includes all observation information I obtained by the system from the environment. In the text situation awareness system, the observation is finally expressed in the form of text frame o, and forms a text frame observation O = {o1, o2, ..., o n }, n ≥ 1, reflecting the key information and dynamic changes in the environment, n represents the number of text frames; the text frame observation O∈Ω contains the target point, building, and mobile obstacle information; the text frame observation O∈V K Represented by natural language sentences, where V is a natural language vocabulary containing a finite number of words; Step 1-3, determine the action space that can be taken in the situation awareness task A = {continue monitoring, key information extraction, information aggregation, situation analysis}, where continued monitoring means continuing to monitor and observe the environment, the system continues to collect data from the environment, and updates the current situation cognition; key information extraction means screening and extracting key data points from observations; information aggregation means aggregating and integrating information collected from multiple observations to form a comprehensive understanding of the current environmental situation; situation analysis means situation analysis of the current state, long-term and strategic analysis of future trends, identification of potential threats, opportunities or changes, and serving the overall situation analysis; Determine the state transfer function T, when the text frame is observed o t After taking any action, the current observation will be transferred to the next text frame observation o t+1 , until all text frame observations are traversed and the terminal state is entered; Step 1-4, determine the reward function R(s,a) of the situation awareness task, which is the reward obtained by the system after completing the situation awareness task; the reward function is set to {-1,0,1}, which respectively represent inaccurate, partially accurate and completely accurate states.

3. The multi-agent collaborative situation awareness method based on a large language model and multimodal data according to claim 1 is characterized in that: Step 2 specifically includes: Step 2-1, define an adapter F for converting multimodal observation I into text frame observation O, expressed as: F:I→O Among them, the multimodal observation I is the original environmental observation data, and the text frame observation O is the text observation data converted by the adapter F from the multimodal observation I, which consists of natural language descriptions. Each text frame has a time tag t, indicating its spatiotemporal relationship with other text frames; Step 2-2, modeling the spatiotemporal relationship of multimodal observations I to generate a spatiotemporal relationship graph G; Step 2-3, use adapter F to convert the spatiotemporal relationship graph G into a natural language spatiotemporal text frame observation O to achieve text processing and unified format output.

4. The multi-agent collaborative situation awareness method based on a large language model and multimodal data according to claim 3 is characterized in that: The input form of the multimodal observation I is the real-time video I of multiple source cameras. cam 、Video File I video , multiple pictures I multi_image , Single Image I image 、Text I text And composite input of pictures and text image_text One of: I∈{I cam ,I video ,I milti_image ,I image ,I text, I image_text }。 5. The multi-agent collaborative situation awareness method based on a large language model and multimodal data according to claim 4 is characterized in that: Step 2-2 models the spatiotemporal relationship of the multimodal observation I to generate a spatiotemporal relationship graph G, which specifically includes: (1) Real-time video from multiple cameras cam , split it into multiple video frames F i,t , where i and t represent the video frames of the i-th camera at time t; for each video frame F i,t , extract its timestamp as the timestamp t, and record the specific time position of the frame in the video stream Timestamp(F i,t ) = t, and then integrate the video frames of all cameras at time t to form the total video frame F at time t. t =∪ i F i,t ; Then for each total video frame F t Analyze the scene in the scene and mark the overall situation of important objects SituationInfo (F t )={(Object j ,Position j ,State j )}, where Object j Represents important object j, Position j With State j Respectively represent the position and posture information of important object j; summarize the data of each video frame, and construct a spatiotemporal relationship diagram based on the spatiotemporal relationship of the video frames: G={(t,SituationInfo(F t ))} (2) For video file I video , split it into multiple video frames F t , where t represents the relative time position of the video frame in the video, and this is used as its time mark t; then for each video frame F t Analyze the scene in the scene and mark the overall situation of important objects SituationInfo (F t )={(Object j ,Position j ,State j )}, where Object j Represents important object j, Position j With State j Respectively represent the position and posture information of important object j; summarize the data of each video frame, and construct a spatiotemporal relationship diagram based on the spatiotemporal relationship of the video frames: G={(t,SituationInfo(F t ))} (3) For multiple images I milti_image , which is considered as a set of pictures {P1,P2,...,P n }, P n Represents the nth picture. For a single picture I image , regard it as a picture P; analyze the spatiotemporal relationship between pictures, mark each picture with a relative time tag t, and thus regard each picture as P t , record the time series Timestamp (P t ) = t; then for P t , extract the key information KeyInfo(P t )={(Object j ,Position j )}; Combine the parsed spatiotemporal information to construct a spatiotemporal relationship diagram: G={(t,KeyInfo(P t ))} If it is a single image P, just give the information pair (t, KeyInfo(P)); (4) For text input I text , split it into text descriptions of different time periods {T1,T2,...,T n }, and mark it with a relative time tag t, so that the text description of each time period is regarded as T t , then parse the text T t Key event information EventInfo(T t )={(Event k ,Object j )}, where Event k Represents the kth event in the text description; from this, a spatiotemporal relationship graph is constructed: G={(t,EventInfo(T t ))} (5) Composite input of image and text I image_text , consider the picture as a picture set {P1,P2,...,P n } or a single image P, split the text into text descriptions of different time periods {T1, T2, ..., T n }, and then combine the two through spatiotemporal relationship analysis to form a picture-text pair U describing the environmental information at a certain time t t ={(P t ,T t )}, and use t as U t time tag; integrate visual information and text description to generate comprehensive information MixInfo (U t )={(Event k ,Object j ,Position j )}, construct a spatiotemporal relationship diagram: G={(t,MixInfo(U t ))}。 6. The multi-agent collaborative situation awareness method based on a large language model and multimodal data according to claim 3 is characterized in that: Steps 2-3 specifically include: For each node (t,Info t ), and convert the node into a standard description information Description(V,t,Info t ) and use it as a text frame o t , and then form a text frame observation O = {o1,o2,...,o n }, n represents the number of text frames; where Info t Represents SituationInfo(F t )、KeyInfo(P t )、EventInfo(T t ) and MixInfo(U t ).

7. The multi-agent collaborative situation awareness method based on a large language model and multimodal data according to claim 3 is characterized in that: Step 3 specifically includes: Step 3-1, preliminarily filter the text data generated in step 2 to remove redundant information and irrelevant noise data; Step 3-2, further extracting key information from the filtered text data; specifically including: Extract key game information and events from the filtered text data, including main objects, events, time nodes and dynamic changes. By analyzing the information in each text frame, extract the content that is crucial to the situation awareness task, and thus obtain the compressed text frame observation O k ; Based on the compressed text frame observation k , identify and mark the main objects, events, time nodes and dynamic changes related to the situation awareness task, and obtain the marked text frame observation O label ; Step 3-3, observe the marked text frame label Perform structural processing to convert the marked key information into a unified, easy-to-understand and easy-to-analyze structured text format to obtain a structured text frame observation. structure , as a text frame observation O for summarizing the chain system.

8. The multi-agent collaborative situation awareness method based on a large language model and multimodal data according to claim 7 is characterized in that: Step 3-1 specifically includes: Treat text frame observation O as the initial text frame observation O inital , for O inital Perform noise filtering, identify and delete repeated information, irrelevant content and noise data, and obtain the filtered text frame observation O filter ; Observe the filtered text frame O filter Merge text frames with similar or repeated contents to summarize similar or closely related information and generate a preliminary summary text frame containing complete information. summarized ; The preliminary summary text frame observation O summarized Observe O with the initial text frame inital Integrate to obtain the integrated text frame observation O integrate .

9. The multi-agent collaborative situation awareness method based on a large language model and multimodal data according to claim 7, characterized in that: Step 5 specifically includes: Step 5-1, initialize the summary chain system, create a frame queue FQ for multi-text frame summarization, and set the hyperparameter K, that is, the length of the frame queue FQ; Step 5-2, the agent observes and summarizes a single text frame, including: Based on the multi-agent partially observable Markov decision model, the agent obtains text frame observations O from the environment through the adapter F of multimodal observation to text frame observations, and these observations are obtained through the environment state S; the agent realizes its interaction with the environment S that cannot be directly observed through the interaction with the text frame observations O; The agent selects the text frame o according to the action space A t Then, the single-frame processing function D(.) is used to process the text frame o. t The process is expressed as: Among them, the text frame o t is the observation of the agent at time t, It is the processed text frame data; The processed text frame data Put it into the frame queue FQ; Step 5-3, determine whether the length of the current frame queue FQ reaches the preset hyperparameter K, if so, execute step 5-4, otherwise return to execute step 5-2; Step 5-4: The agent summarizes the multiple text frames and generates a local situation overview of the environment; specifically, it includes: Use the aggregation function LA to summarize the text frames in FQ in multiple frames: LS t =LA(FQ,promptsL) Among them, LS t is the situation summary of the agent at time t, promptsL is the prompt word information for the local situation summary of the single agent, which is obtained from the large model prompt word template described in step 4; Step 5-5, clear some old data in FQ.

10. The multi-agent collaborative situation awareness method based on a large language model and multimodal data according to claim 8, characterized in that: Step 6 specifically includes: Step 6-1, create a comprehensive situation overview queue GQ, and set the length of the situation overview queue GQ through the hyperparameter GK; Step 6-2: In the multi-agent collaboration stage, each agent i Initialize the adapter in step 1 and get its text frame observation O i ; Each agent i Observe it with the text frame i The processed text frame observations obtained after the interaction are stored in their respective frame queues FQ i In, and when FQ i Generate situational awareness summary LS when reaching a certain length t,i ; The system will each agent i Overview of the local situation LS t,i Synthesize into a global situation overview information: Among them, CS t is the global situation overview information, N is the number of agents; GS t Put it into the comprehensive situation overview queue GQ; Step 6-3, when the comprehensive situation overview queue GQ reaches the hyperparameter GK, the overall aggregation function GA integrates the comprehensive situation overview queue to generate the final holistic comprehensive situation awareness information: GSW t =GA{GQ,promptsG} Among them, GSW t is the comprehensive situation awareness information generated by the large language model, promptsG is the prompt word information for multi-agent comprehensive situation awareness, obtained from the large model prompt word template described in step 4; Step 6-4, clear some old data in GQ; Step 6-5: Refine the overall GSW information t , divide it into the following three parts and output: Environmental analysis: The system integrates the observation data of each agent to generate an overall analysis of the current state of the environment, including key changes, potential threats and opportunities; Enemy analysis: Based on the collaborative observation of multiple agents, the system evaluates the enemy's strategy, intentions and behavior; Recommendations: The big language model gives comprehensive strategic recommendations that will support decision making, optimize resource allocation and action plans.

Citation Information

Cited By

  • Situation simulation method and device based on large language model

    CN120409233A

  • Detection method and system capable of explaining video hatred content based on multi-agent cooperation

    CN120472370A

  • A method and system for detecting hate content in videos based on multi-agent collaboration

    CN120472370B

  • Mechanical arm motion control method based on multi-agent cooperation

    CN120620234A

  • A robotic arm motion control method based on multi-agent cooperation

    CN120620234B