System for optimizing communication between ai agents

Abstract

The invention provides a system for optimizing communication between AI agents, which can respond to requests related to tasks and process and explain real-time scene data. The system comprises one or more sensing devices used for collecting scene data in real time according to a request of a second AI agent; the large language model module is used for generating a task-specific event dictionary when the request is related to the task; the sensing engine is used for processing the collected data; the cognitive engine is used for extracting semantic information; and the decision engine is used for identifying events related to the task according to the event dictionary. An operation platform of the system comprises a communication module, a processor and a memory, and when stored instructions are executed, the system can run among a perception layer, a cognitive layer and a decision layer. According to the invention, context-based hierarchical information exchange is ensured, so that the task requirements of the AI intelligent agent are met.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202510975896.3 (22) Application Date 2025.07.15 (30) Priority Data 63 / 671,282 2024.07.15 US (71) Applicant City University of Hong Kong Address Tat Chee Road, Kowloon, Hong Kong, China (72) Inventors Wu Dapeng Wang Shiqi Chen Peilin (74) Patent Agency Shenzhen Yibao Intellectual Property Agency (General Partnership) 44588 Patent Attorney Wang Qin Cao Yucun (51) Int.Cl. G06F 9 / 54 (2006.01) G06N 5 / 04 (2023.01) G06N 3 / 0455 (2023.01) (54) Invention Title: System for Optimizing Communication Between AI Agents (57) Abstract: This invention provides a system for optimizing communication between AI agents, capable of responding to task-related requests and processing and interpreting real-time scene data. The system includes: one or more sensing devices for real-time acquisition of scene data according to requests from a second AI agent; a large language model module for generating a task-specific event dictionary when the request is task-related; a perception engine for processing the acquired data; a cognitive engine for extracting semantic information; and a decision engine for identifying task-related events based on the event dictionary. The system's operating platform includes a communication module, a processor, and a memory, where stored instructions, when executed, enable the system to operate between the perception layer, the cognitive layer, and the decision layer. This invention ensures context-based, hierarchical information exchange to meet the task requirements of AI agents. Claims (2 pages), Description (8 pages), Drawings (4 pages), CN 121349723 A, 2026.01.16, CN 1 21 34 97 23 A. 1. A system for optimizing communication between AI agents, comprising: one or more sensing devices configured to collect data from a scene in real time, wherein the scene is related to a request from a second AI agent; a large language model module configured to construct a task-specific event dictionary when the request is related to a task assigned to the second AI agent; a perception engine configured to process the collected data; a cognitive engine configured to extract semantic information from the processed data; a decision engine configured to detect one or more task-related events from the extracted semantic information based on the task-specific event dictionary; and an operating platform including a communication module, one or more processors, and a memory, the processors communicating with the one or more processors and storing instructions, which, when executed by the one or more processors, cause the system to:The system operates at the perception layer, wherein the perception engine is further configured to encode the processed data, and the communication module is configured to transmit the encoded data to the second AI agent; or it operates at the cognitive layer, wherein the cognitive engine is further configured to select one or more scene descriptions from the extracted semantic information, and the communication module is configured to transmit the one or more selected scene descriptions to the second AI agent; or it operates at the decision layer, wherein the decision engine is further configured to generate task-related event descriptors based on the one or more detected task-related events, and the communication module is configured to transmit the generated task-related event descriptors to the second AI agent. 2. The system according to claim 1, wherein the perception engine comprises: a data processing module configured to process the collected data; and a data encoding module configured to encode the processed data. 3. The system according to claim 2, wherein the data encoding module is further configured to encode the processed data using a rate-distortion-power-latency optimization algorithm, wherein bit rate, visual distortion, power consumption, and end-to-end latency are balanced according to a specific perception rate-distortion relationship. 4. The system according to claim 1, wherein the cognitive engine comprises: one or more natural language processors configured to extract the semantic information from the processed data; and a scene description selection module configured to select one or more scene descriptions from the extracted semantic information. 5. The system according to claim 4, wherein the scene description selection module is further configured to select the one or more scene descriptions from the extracted semantic information based on the accuracy reflecting common sense under a bitrate budget using a rate precision optimization algorithm. 6. The system according to claim 4, wherein the scene description selection module is further configured to select the one or more scene descriptions from the extracted semantic information based on the accuracy reflecting task-related information under a bitrate budget using a rate precision optimization algorithm. 7. The system according to claim 1, wherein the decision engine comprises: an event detection module configured to detect one or more task-related events from the extracted semantic information based on the task-specific event dictionary; and an event analysis module configured to: construct a scene graph, wherein the scene graph includes one or more triples, each representing one or more detected events occurring in the scene; assign a utility value to each triple in the scene graph; select one or more triples with the highest utility value; and a deep learning-based converter configured to convert the one or more selected triples into the one...Or multiple task-related event descriptors. 8. The system of claim 7, wherein the one or more triples having the highest utility value are selected by a rate value optimization algorithm under a specified bit rate budget. 9. The system of claim 7, wherein the task-related events are detected by multimodal sensing interpretation. 10. The system of claim 7, wherein the utility value is obtained by computation through a trained deep learning model; and the trained deep learning model is configured to evaluate the contribution of each triple to the task objective based on historical data and task context. Claims 2 / 2 Page 3 CN 121349723 A System for Optimizing Communication Between AI Agents Technical Field

[0001] The present invention relates to the field of artificial intelligence (AI), and more particularly to an efficient framework for optimizing communication between artificial intelligence agents (also known as AI agents). Background Art

[0002] Next-generation industries (such as Industry 4.0) rely heavily on communication between AI agents at the perception, cognition and decision-making levels. For example, when an AI agent (or receiver) wants to see events happening in the distance "with its own eyes," it can choose perception layer communication. In this layer, non-textual sensory signals such as video are directly acquired and perceived by the receiver, rather than being relayed through other agents. When the receiver wants to save time and bandwidth resources, it can choose cognitive layer communication, that is, to summarize the video information captured by the perception layer by generating text messages. At this time, the video is perceived and preliminarily processed by other agents. When the receiver only needs to receive text information related to its task (such as safe driving), it can choose decision layer communication. In this layer, video summary text oriented towards a specific task is generated to support task execution.

[0003] In the existing communication infrastructure, non-textual signal communication of the perception layer and semantic communication of the cognitive layer are usually studied as independent topics, lacking synergistic effect. Research on goal-oriented communication at the decision layer is even more limited. Existing signal quality evaluation indicators are difficult to effectively measure the actual value of information for a specific task. Therefore, it is not feasible to simply splice together the existing communication solutions of the perception, cognition and decision layers to build a multi-layer communication mechanism between AI agents.

[0004] Previous work on the perception layer has been reduced to optimizing signal representations based on bit-rate-constrained (e.g., video coding) signal fidelity. However, the receiving characteristics of AI agents or human perception have not been adequately considered. For the cognitive layer, existing strategies transmit all scene descriptions without considering their relevance to the receiver, resulting in bandwidth waste due to the transmission of irrelevant messages. For the decision-making layer, existing task-oriented semantic communication techniques focus on signal / information processing tasks (e.g., image segmentation), which are not inherently high-level tasks in everyday human social activities.

[0005] Previous research on the perception layer has mainly focused on optimizing signal fidelity (such as video coding) under bit-rate constraints, but has not fully considered the perception characteristics of the information receiver (whether it is an AI agent or a human user). For the cognition layer, existing methods usually transmit complete scene descriptions without filtering the relevance of the information to the receiver, resulting in unnecessary information transmission and bandwidth waste. At the decision layer, current task-oriented semantic communication technologies are mostly focused on low-level signal processing tasks such as image segmentation, which are not high-level tasks in human society. Summary of the Invention

[0006] The purpose of this invention is to provide a system and related network infrastructure configured to facilitate structured, context-aware communication between autonomous agents operating at the perception, cognition and decision layers. The system can be widely deployed in various technical fields, including but not limited to autonomous vehicle control systems and intelligent security monitoring platforms.

[0007] According to one aspect of the invention, a system for optimizing communication between AI agents is provided. The system includes: one or more sensing devices configured to collect data from a scene in real time, wherein the scene is related to a request from a second AI agent; a large language model module configured to construct a task-specific event dictionary when the request is related to a task assigned to the second AI agent; a perception engine configured to process the collected data; a cognition engine configured to extract semantic information from the processed data; and a decision engine configured to detect one or more task-related events and operating platforms from the extracted semantic information based on the task-specific event dictionary. The system includes a communication module, one or more processors, and a memory. The processors communicate with the one or more processors and store instructions. When the instructions are executed by the one or more processors, the instructions cause the system to: operate at the perception layer, such that the perception engine is further configured to encode the processed data, and the communication module is configured to transmit the encoded data to the second AI agent; operate at the cognition layer, such that the cognition engine is further configured to select one or more scene descriptions from the extracted semantic information, and the communication module is configured to transmit the one or more selected scene descriptions to the second AI agent; or operate at the decision layer, such that the decision engine is further configured to generate task-related event descriptors based on the one or more detected task-related events, and the communication module is configured to transmit the generated task-related event descriptors to the second AI agent.

[0008] Preferably, the perception engine includes: a data processing module configured to process the collected data; and a data encoding module configured to encode the processed data.

[0009] Preferably, the data encoding module is further configured to encode the processed data using a rate-distortion-power-latency optimization algorithm, wherein bit rate, visual distortion, power consumption, and end-to-end latency are balanced according to a specific perceptual rate-distortion relationship.

[0010] Preferably, the cognitive engine includes: one or more natural language processors configured to extract the semantic information from the processed data; and a scene description selection module configured to select one or more scene descriptions from the extracted semantic information.

[0011] Preferably, the scene description selection module is further configured to select the one or more scene descriptions from the extracted semantic information based on the accuracy reflecting common sense under a bit rate budget using a rate precision optimization algorithm.

[0012] Preferably, the scene description selection module is further configured to select the one or more scene descriptions from the extracted semantic information based on the accuracy reflecting task-related information under a bit rate budget using a rate precision optimization algorithm.

[0013] Preferably, the decision engine includes: an event detection module configured to detect one or more task-related events from the extracted semantic information based on the task-specific event dictionary; an event analysis module configured to: construct a scene graph, wherein the scene graph includes one or more triples, each representing one or more detected events occurring in the scene; assign a utility value to each triple in the scene graph; select one or more triples with the highest utility value; and a deep learning-based converter configured to convert the one or more selected triples into the one or more task-related event descriptors respectively.

[0014] Preferably, the one or more triples with the highest utility value are selected using a rate value optimization algorithm under a specified bit rate budget.

[0015] Preferably, the task-related events are detected through multimodal sensing interpretation.

[0016] Preferably, the utility value is obtained by calculation through a trained deep learning model; and the trained deep learning model is configured to evaluate the contribution of each triple to the task objective based on historical data and task context.

[0017] This invention introduces a task-oriented communication scheme that can transmit and interpret data according to the specific functional roles and work objectives assigned to AI agents. The provided system can automatically extract task-related information and determine its priority, thereby enabling goal-oriented interaction among distributed AI agents. In addition, this invention provides a general modular functional programming framework that can operate collaboratively between the perception, cognition, and decision-making layers. The framework supports adaptive expansion and can add or remove modules (such as third-party algorithm libraries, edge inference nodes, etc.) according to actual application needs, facilitating flexible deployment in various scenarios.The framework also allows for the implementation of communication logic at specific levels while maintaining compatibility through standardized interfaces. This approach offers significant improvements in communication efficiency, scalability, and responsiveness, especially in distributed, autonomous operating environments. Brief Description of the Drawings

[0018] Embodiments of the invention will now be described in more detail with reference to the accompanying drawings, in which:

[0019] FIG1A illustrates a system according to one embodiment of the invention; FIG1B illustrates a system according to another embodiment of the invention;

[0020] FIG2 illustrates an exemplary implementation of the system in an autonomous driving scenario;

[0021] FIG3 illustrates the workflow of the system operating at the decision-making level. Detailed Description

[0022] In the following description, details of the invention will be described by way of preferred embodiments. Modifications, additions, or substitutions will be apparent to those skilled in the art without departing from the spirit and scope of the invention. Certain specific details omitted in this description are intended to avoid redundant descriptions that obscure the core content of the invention. However, the purpose of this disclosure is to enable those skilled in the art to understand and practice the technical solutions disclosed in the invention without excessive experimentation.

[0023] FIG1A illustrates a system 100A for optimizing communication between AI agents in a scenario according to an embodiment of the invention. The AI ​​agent may include a service provider AI agent (also known as a service sender AI agent) 10 and a service requester AI agent (also known as a service receiver AI agent) 20, wherein the service provider AI agent 10 is configured to respond to a request from the service requester AI agent 20 in the scenario.

[0024] As shown in FIG1A, the system 100A is set in or network-connected to the service sender AI agent 10, and includes: one or more sensing devices 110 for collecting data from the scenario in real time; and a large language model module 111 for constructing a task-specific event dictionary (containing a structured list of relevant event types) when the request is associated with a task assigned to the service receiver AI agent 20.

[0025] System 100A further includes a perception engine 120, configured to process the collected data (including but not limited to data filtering, feature extraction, preprocessing, etc.); a cognitive engine 130, configured to extract semantic information (including but not limited to AI algorithms such as natural language understanding, semantic segmentation, entity recognition, etc.) from the processed data; and a decision engine 140, configured to detect one or more task-related events from the extracted semantic information based on the task-specific event dictionary.

[0026] System 100A further includes an operating platform 160, including one or more processors and communication with one or more processors.The instructions are configured to, when executed by the one or more processors, cause the system to: operate at a perception layer, such that the perception engine is further configured to encode the processed data; operate at a cognitive layer, such that the cognitive engine is further configured to select one or more scene descriptions from the extracted semantic information; or operate at a decision layer, such that the decision engine is further configured to generate task-related event descriptors based on one or more detected task-related events.

[0027] System 100A also includes a communication module 150, which is configured to: transmit encoded data to a service recipient AI agent at the perception layer; transmit one or more selected scene descriptions to a service recipient AI agent at the perception layer; or transmit generated task-related responses to a service recipient AI agent at the decision layer. Specification 3 / 8 pages 6 CN 121349723 A

[0028] The selection of the communication module depends on the specific requirements of the AI ​​agent application for communication range, power consumption, bandwidth, and network topology. For example, the communication module may include, but is not limited to, a Wi-Fi module for local network or cloud connection; a Bluetooth or BLE module for short-range, low-power communication, particularly suitable for wearable devices or mobile internet systems; a LoRa module for long-distance or outdoor deployment; a Zigbee or Thread module for reliable low-bandwidth communication; and a cellular communication module for providing wide-area connectivity for mobile or remote autonomous units.

[0029] The operating platform 160 may serve as a communication hub between the sensing device 110, the LLM module 111, the perception engine 120, the cognitive engine 130, the decision engine 140, and the communication module 150.

[0030] Specifically, when the system 100A is running at the perception layer, the operating platform 160 may be configured to receive requests for data collected from the sensing device 110 and transmit the collected data to the perception engine 120; receive encoded data from the perception engine 120 and transmit the encoded data to the communication module 150. When system 100A runs at the cognitive layer, operation platform 160 can be configured to receive requests from the communication module and send the requests to the cognitive engine 130; receive a selected scene description from the cognitive engine 130 and transmit the selected scene description to the communication module 150. When system 100A runs at the decision layer, operation platform 160 can be configured to receive requests from the communication module and send the requests to the LLM module; receive a task-specific event dictionary generated from the LLM module and transmit the task-specific event dictionary to the decision engine 140; receive task-related event descriptors generated by the decision engine 140 and transmit the generated task-related event descriptors to the communication module 150.

[0031] Figure 1B illustrates a system 100B for optimizing communication between AI agents 10 and 20 in a scene according to another embodiment of the present invention. System 100B is similar to system 100A, except that sensing device 110, perception engine 120, cognitive engine 130, decision engine 140, LLM module 111, and communication module 150 can be interconnected to directly exchange data, commands, and processing results.

[0032] Specifically, communication module 150 can be configured to transmit requests directly to the LLM module. Sensing device can be configured to transmit collected data directly to perception engine 120. The communication module 150 can also be configured to: when the system 100B is running in the perception layer, receive encoded data from the perception engine 120 and send the encoded data to the service recipient AI agent; when the system 100B is running in the cognition layer, receive a selected scene description from the cognition engine 130 and send the selected scene description to the service recipient AI agent; or when the system 100B is running in the decision layer, receive a task-related event descriptor generated by the decision engine 140 and transmit the generated task-related event descriptor to the service recipient AI agent.

[0033] Figure 2 shows an exemplary implementation of the system in an autonomous driving scenario. For example, when an autonomous vehicle with an AI agent is stuck in traffic due to a traffic accident ahead and wants to understand and view the cause of the traffic jam, it can act as a service recipient and select to request another autonomous vehicle with an AI agent that is closer to the accident scene as a service sender to provide relevant services through the perception layer. In response to the request of the service recipient vehicle, the service sender vehicle can directly transmit non-textual sensory signals such as video collected when approaching the accident scene to the service recipient vehicle. In other words, the perception layer communication provides the receiving vehicle with a viewing mode similar to "movie playback," enabling it to autonomously perceive and understand the accident scene.

[0034] If the receiving vehicle wants to save time and bandwidth, it can choose to communicate with the sending vehicle through the cognition layer. In the cognition layer, the sending vehicle can summarize the collected video, generate a text message, and then transmit the text message to the receiving vehicle. In this case, the video is perceived by the sending vehicle, not the receiving vehicle. In other words, the cognition layer provides a storytelling mode that allows the receiving vehicle to gain cognition of the event.

[0035] The receiving vehicle can also choose to communicate with the sending vehicle through the decision layer to inform the sending vehicle of the assigned task (e.g., safe driving) as specified in the specification (page 4 / 8, 7 CN 121349723 A), allowing the sending vehicle to generate only task-related text messages to summarize the acquired video. In other words, the decision layer provides a "storytelling" communication mode to guide the receiving vehicle's response.Meaningful text information with specific task objectives is automatically generated and transmitted. This information is similar to jokes that the receiving vehicle is "interested" in, which can attract its "attention" or even make it "laugh". For example, when receiving a message that "there is a traffic accident ahead", the receiving vehicle may choose to detour to avoid getting stuck in congestion.

[0036] In some embodiments, the perception engine 120 may include a data processing module 121 for processing the collected data; and a data encoding module 122 for encoding the processed data.

[0037] At the perception layer, the perception engine 120 is configured to promote efficient distributed perception with optimized resource utilization and perception clarity. Unlike traditional video encoding modules that often simply equate visual quality with encoding bit rate (e.g., without considering the specific characteristics of the video content, assuming that the higher the bit rate, the better the visual quality), the encoding module 122 in this invention relies on an interpretable and efficient perception quality measurement method to achieve intelligent control of the encoder. This quality metric adopts a data-driven design approach, which can not only accurately predict video quality from the perspective of human or artificial intelligence perception, but also explain how global or local quality degradation affects the predicted quality score.

[0038] For example, in autonomous driving scenarios, video clips captured by vehicle cameras may experience local quality degradation due to severe weather (such as raindrops partially obscuring the lens). Traditional encoders may indiscriminately encode the entire frame at a higher bit rate to improve overall perception quality, thereby causing unnecessary waste of resources. In contrast, the encoding module 122 is configured to identify and locate the specific factors causing local degradation, and selectively allocate the bit rate to these identified degradation areas, thereby improving the perception clarity of key areas while achieving more efficient resource utilization.

[0039] The encoding module 122 is also configured to perform rate-distortion-power-delay optimization (RDO) to improve overall communication performance when transmitting signals used for perception. This module implements a multi-objective optimization strategy by considering end-to-end delay and power consumption, combined with a specific perception rate-distortion relationship, thereby achieving video encoding optimization for communication scenarios.

[0040] The Rate-Distortion-Power-Latency Optimization (RDO) algorithm is a multi-objective optimization algorithm that aims to achieve a balance among four key dimensions: bit rate, visual distortion, power consumption, and end-to-end latency. This integrated optimization method not only ensures the visual quality of video transmission but also takes into account practical constraints in terms of latency and energy efficiency, making it particularly crucial for applications with extremely high requirements for real-time performance and resource efficiency, such as autonomous driving and security monitoring.

[0041] For example, in autonomous driving scenarios, vehicles may need to transmit real-time video streams to a central processing unit or nearby vehicles to achieve collaborative perception. Traditional methods often focus on compressing the bit rate to reduce bandwidth consumption, but this may lead to...Increased latency or limited processing power are unacceptable for safety-critical tasks with extremely high real-time requirements. Within the RDO algorithm framework, end-to-end latency, power budget, and perceived quality are explicitly incorporated into the optimization constraints. Video coding parameters are strategically selected to maximize perceived video quality while ensuring that latency and power limits are not exceeded, thereby meeting the dual requirements of real-time perception tasks for communication efficiency and reliability.

[0042] Another example is battery-powered drones used for remote monitoring. In such scenarios, power efficiency is crucial, and transmission latency must be kept to a minimum to ensure the system's real-time responsiveness. With the support of the RDO algorithm, coding strategies (such as selective frame dropping, adaptive compression levels, or region-first coding) can be dynamically adjusted to achieve the optimal trade-off between visual quality, energy consumption, and latency. By introducing the perceived rate-distortion relationship, the system can accurately evaluate and prioritize the transmission of visual information that is most relevant from a human or AI perception perspective, thereby further improving transmission efficiency and task execution performance.

[0043] In some embodiments, the cognitive engine 130 may include a natural language processor 131 for extracting and deriving semantic information from the perception results; and a scene description selection module (or edge server) 132 for performing cognitive processing on the extracted semantic information.

[0044] At the cognitive layer, the cognitive engine 130 is configured to facilitate efficient distributed perception and cognition (DPC) through semantic communication technology. Distributed perception is achieved collaboratively by multiple sensors connected by a mobile network. The scene description selection module 132 selects and transmits only the features and channel information that meet the system requirements (including accuracy, bandwidth, energy consumption, and latency) based on a joint evaluation of feature importance and channel state, and according to the overall system performance selection criteria dominated by the rate-accuracy optimization (RAO) algorithm. After the server coordinates the selective transmission at the source and channel ends, the scene description selection module 132 will converge to the globally optimal performance through iterative optimization under a given bit rate constraint (or budget).

[0045] In some embodiments, the scene description selection module 132 may be configured to use the RAO algorithm to select the most relevant scene description from the extracted semantic information based on the accuracy of reflecting common sense, within the bit rate budget. Thus, messages that the receiver would be interested in from a common-sense perspective will be prioritized for transmission, achieving more semantically relevant and efficient communication.

[0046] In some embodiments, the scene description selection module 132 may be configured to use the RAO algorithm to strategically select candidate scene descriptions based on their semantic importance relative to the receiver's task. Each candidate description will be selected based on its semantic relevance (i.e., whether it accurately reflects key, executable, or task-related information) and the required transmission resources (e.g., bits).The evaluation is performed using the bit rate. The optimization objective is to maximize semantic value while minimizing bandwidth consumption under bit rate constraints.

[0047] For example, consider the scenario of autonomous driving. The description "a car is reversing" is highly relevant to the safe driving task because it directly affects navigation decisions and is related to driving safety. Based on the RAO algorithm, this description is given high semantic importance and is therefore transmitted first. In contrast, the description "the leaves are green" may occupy similar transmission resources, but it has very low relevance to vehicle navigation or immediate safety decisions. Therefore, according to the RAO algorithm, the semantic priority of this description is rated as very low and it is not selected for transmission. Essentially, the RAO algorithm prioritizes descriptions that are highly relevant to the task, such as executable or safety-critical events, and achieves efficient bandwidth utilization by excluding unnecessary semantic information.

[0048] The decision engine 140 may include: an event detection module 141, used to detect one or more task-related events from extracted semantic information based on a task-specific event dictionary; and an event analysis module 142, used to: construct a scene graph, wherein the scene graph includes one or more triples, each representing one or more detected events occurring in the scene; assign a utility value to each triple in the scene graph; select one or more triples with the highest utility value; and convert the one or more selected triples into one or more task-related event descriptors.

[0049] As shown in FIG3, at the decision layer, task-driven communication is implemented through mechanisms such as task-related event dictionary construction, event detection, and event description. Specifically, the task-related event dictionary, as a structured repository (or list), contains event entries closely related to a specific predefined task, thereby supporting targeted event detection and semantic description. Each task corresponds to a dedicated dictionary, which is created using domain knowledge and a large language model tailored to task requirements.

[0050] For example, suppose the assigned task is autonomous driving, focusing on traffic safety. In this context, a task-related event dictionary might include entries such as: "traffic accident," "sudden vehicle stop," "pedestrian crossing the road," "sudden vehicle lane change," "traffic congestion," "road blockage," "approaching emergency vehicle," "object falling into the lane," and "vehicle reversing on a main road." Such dictionaries explicitly exclude events unrelated to driving safety, such as "bird flight" or "rustling leaves," thereby ensuring that computing and communication resources are used only for critical events that have a direct impact on the task outcome.

[0051] It should be understood that the exemplary task-related event dictionaries listed are representative but not exhaustive. A complete dictionary would more comprehensively list all task-specific events relevant to the actual decision-making process, especially in terms of bandwidth or computing power.Under practical application constraints such as limited computing resources, ensure that the system can still respond efficiently to key events in resource-constrained environments. (Page 6 / 8, CN 121349723 A)

[0052] In some embodiments, the event dictionary can be autonomously generated under the guidance of an agent based on a large language model (LLM) (such as GPT), thereby ensuring its accuracy and completeness. For example, if the task is to detect criminal activity, the autonomous vehicle can send a query to an LLM specifically customized for public safety scenarios and construct a dictionary containing various criminal behavior events based on the generated results.

[0053] Each entry in the event dictionary typically represents a specific event category or type (e.g., "traffic accident", "pedestrian crossing"). In actual operation, if the sensor data stream simultaneously contains feature information of multiple dictionary events, the system can detect multiple events in parallel in the same scenario. Therefore, depending on the real-time scenario, the system can simultaneously or continuously identify multiple related events, supporting comprehensive perception and response to complex dynamic environments.

[0054] When selecting an event from the task-related event dictionary, the event detection module 141 analyzes the captured sensor data through multimodal sensing interpretation to detect and confirm the occurrence of the event. Specifically, this detection process integrates information from multiple sensor modalities, such as cameras, LiDAR, radar, and acoustic sensors, and uses advanced deep learning models (such as Transformer fusion networks, cross-modal alignment networks, etc.) to achieve integrated analysis of cross-modal data. These models can jointly model and semantically associate different types of sensor inputs, and can reliably identify features related to the target event even in complex or highly interfering environmental conditions.

[0055] For example, for the event of "the vehicle suddenly stops," the event detection module 141 combines multiple sensor data for comprehensive analysis: detecting sudden speed changes through optical flow in visual data, confirming the sudden stillness of the object's position with LiDAR data, and identifying the sound of emergency braking through acoustic signals. Compared with traditional single-modal detection systems, this multimodal fusion method significantly improves the accuracy and robustness of event recognition.

[0056] Once an event is detected, the event analysis module 142 will construct a corresponding scene graph. A scene graph is a structured representation of detected events, explicitly capturing objects / entities in the scene and their semantic relationships. This process includes identifying relevant objects, extracting their attributes, and deriving relationships and contextual information between entities based on the detected events, thereby supporting subsequent semantic understanding and task decision-making.

[0057] For example, for the event "pedestrian crossing the road," the scene graph may contain nodes representing entities such as "pedestrian," "road," and "pedestrian crossing," and connect these nodes through relational edges, such as "pedestrian is crossing the road" or "located on the pedestrian crossing."Thus, a structured graph that can clearly express the semantics of events is constructed, intuitively presenting the key objects and their interaction relationships in the event scene.

[0058] Therefore, the scene graph provides an intuitive and structured representation of detected events, supporting efficient downstream processing (e.g., utility value assessment and task decision-making) by explicitly encoding the spatial, semantic, and relational dynamics of events in the scene. Specifically, each triple in the scene graph is assigned a utility value to reflect its importance or relevance relative to the specific task of the receiver. This utility value is calculated by a trained deep learning model (e.g., a sparse isotonic Shapley regression model), which evaluates the contribution of each triple to the task objective based on historical data and task context.

[0059] Subsequently, the event analysis module 142 uses graph operations and the Rate-Value Optimization (RVO) algorithm to identify the event to be transmitted based on the valuation in the scene graph. The RVO algorithm uses a greedy or iterative strategy to compress the scene graph by selecting the triple with the highest cumulative utility value under specified resource constraints (e.g., bit rate or computational budget). For example, when the bit rate budget only allows for the transmission of a limited number of triples or bytes, the algorithm will prioritize the triples that bring the greatest practical value per unit of resource consumption. This process is called scene graph compression, which ensures that the transmitted information is both concise and highly relevant to the receiver's task requirements by strategically selecting a subset of triples (subject-verb-object) from the complete scene graph.

[0060] For example, in autonomous driving safety tasks, if the original scene graph contains multiple triples, such as (vehicle, stopped, intersection), (pedestrian, crossing, road), and (sky, present, cloud), under tight resource budget conditions, the RVO algorithm will prioritize the transmission of the first two triples because they have high practical value and are closely related to real-time safety decisions. The third triple, although occupying similar transmission resources, is omitted because it has lower relevance to the current task.

[0061] The "maximum value" can be defined as the sum of the practical values ​​of the selected triples in the compressed scene graph. This sum serves as an objective metric to evaluate the efficiency of the scene graph representation in retaining relevant information for critical tasks under resource-constrained conditions. The goal is to maximize the total utility value while strictly adhering to specified resource constraints (such as bit rate or computational budget). Therefore, the final compressed scene graph retains only those triples with the highest cumulative utility value, ensuring efficient resource utilization and providing strong support for critical task decisions.

[0062] Finally, a specific deep learning-based converter module is used to convert the triples in the compressed scene graph into task-related semantic descriptors. By maintaining the simplicity of event descriptions, the detected objects and their associated information can be accurately conveyed.The following relationships are discussed, and redundant information transmission and representation are avoided.

[0063] According to embodiments of this disclosure, the various functional units and modules of the system can be implemented by computing devices, computer processors, or electronic circuits, including but not limited to application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, and other programmable logic devices that can be configured or programmed according to the contents of this disclosure. Based on the teachings of this disclosure, those skilled in software or electronics can easily write computer instructions or software code that run on the above-mentioned devices.

[0064] All or part of the methods of this invention can be implemented in one or more computing devices, such as server computers, personal computers, laptops, smartphones, tablets, and other mobile computing devices.

[0065] Embodiments of this invention may also include computer-readable storage media, including transient or non-transient storage devices storing computer instructions or software code, for programming or configuring computing devices, processors, or electronic circuits to perform the methods of this invention. Such storage media include, but are not limited to, floppy disks, optical disks, Blu-ray discs, DVDs, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), flash memory devices, or any other medium or device suitable for storing instructions, code, and / or data.

[0066] The functional units and modules in the embodiments can also be implemented in a distributed computing environment or cloud computing environment, in which one or more processing devices interconnected by a communication network can execute all or part of the machine instructions in a distributed manner, for example through an intranet, local area network (LAN), wide area network (WAN), Internet or other forms of data communication media.

[0067] Although this disclosure has been described and illustrated in conjunction with specific embodiments, these contents should not be regarded as limitations on the invention. The illustrations are not necessarily drawn to scale, and the actual devices may differ from the illustrations due to manufacturing processes and tolerances. Other embodiments of the invention are not specifically described herein, but those skilled in the art can make modifications based on the content of this disclosure to meet specific application scenarios, materials, components, methods or processes, and all such modifications should be considered to be included within the scope of protection of the appended claims. Although some methods are described as being performed in a specific order, unless explicitly stated otherwise, these steps can be combined, split or reordered to achieve functionally equivalent implementations without departing from the basic principles of this disclosure. Therefore, the arrangement and grouping of steps are not restrictive unless otherwise specified. Instruction manual, page 8 / 8, 11 CN 121349723 A, Figure 1A; Instruction manual drawing, page 1 / 4, 12 CN 121349723 A, Figure 1B; Instruction manual drawing, page 2 / 4, 13 CN 121349723 A, Figure 2; Instruction manual drawing, page 3 / 4, 14 CN 121349723 AFigure 3说明书附图4 / 4页15 CN 121349723 A Abstract The present invention provides a system for optimizing communication between AI agents, capable of processing and interpreting real-time scene data in response to mission-related requests. The system includes one or more sensory devices for collecting the scene data in real-time according to a request from a second AI agent; a large language model module for constructing a mission- specific event dictionary when the request is relevant to a mission; a perception engine for processing the collected data; a cognition engine for extracting semantic information; and a decision engine for identifying mission-relevant events according to the event dictionary. Operating platform of the system includes a communication module, a processor, and a memory storing instructions, when executed, cause the system to operate among a perception tier, a cognition tier, and a decision tier. The present invention ensures context-aware, tier-specific informationexchange fulfilling AI agent’s mission needs.

Claims

1. A system for optimizing communication between AI agents, comprising: one or more sensing devices configured to collect data in real-time from a scene, wherein the scene is related to a request from a second AI agent; a large language model module configured to construct a task-specific event dictionary when the request is related to a task assigned to the second AI agent; a perception engine configured to process the collected data; a cognition engine configured to extract semantic information from the processed data; a decision engine configured to detect one or more task-related events from the extracted semantic information based on the task-specific event dictionary; and an operating platform comprising a communication module, one or more processors, and a memory, the processor in communication with the one or more processors and storing instructions that, when executed by the one or more processors, cause the system to: operate at a perception layer such that the perception engine is further configured to encode the processed data and such that the communication module is configured to transmit the encoded data to the second AI agent; operate at a cognition layer such that the cognition engine is further configured to select one or more scene descriptions from the extracted semantic information and such that the communication module is configured to transmit the one or more selected scene descriptions to the second AI agent; or operate at a decision layer such that the decision engine is further configured to generate a task-related event descriptor based on the one or more detected task-related events and such that the communication module is configured to transmit the generated task-related event descriptor to the second AI agent. the perception engine comprises:

2. The system of claim 1, wherein, a data processing module configured to process the collected data; and a data encoding module configured to encode the processed data. the data encoding module is further configured to encode the processed data through a rate-distortion-power-delay optimization algorithm, wherein a bit rate, a visual distortion, a power consumption, and an end-to-end delay are balanced according to a specific perception rate-distortion relationship.

3. The system of claim 2, wherein, the cognition engine comprises:

4. The system of claim 1, wherein, one or more natural language processors configured to extract the semantic information from the processed data; and a scene description selection module configured to select one or more scene descriptions from the extracted semantic information. the scene description selection module is further configured to select the one or more scene descriptions from the extracted semantic information through a rate-accuracy optimization algorithm based on an accuracy reflecting common sense under a bit rate budget.

5. The system of claim 4, wherein, the scene description selection module is further configured to select the one or more scene descriptions from the extracted semantic information through a rate-accuracy optimization algorithm based on an accuracy reflecting task-related information under a bit rate budget.

6. The system of claim 4, wherein, the decision engine comprises:

7. The system of claim 1, wherein, an event detection module configured to detect one or more task-related events from the extracted semantic information based on the task-specific event dictionary; an event analysis module configured to: ​ constructing a scene graph, wherein the scene graph comprises one or more triples respectively representing one or more detected events occurring in the scene; assigning a utility value to each triple in the scene graph; and selecting one or more triples with the highest utility value; and a deep learning based converter configured to convert the one or more selected triples into one or more task related event descriptors respectively.

8. The system of claim 7, wherein, the one or more triples with the highest utility value are selected by a rate value optimization algorithm under a specified bitrate budget.

9. The system of claim 7, wherein, the task related events are detected by multi-modal sensor interpretation.

10. The system of claim 7, wherein the utility value is computed by a trained deep learning model; and the trained deep learning model is configured to evaluate the contribution of each triple to the task objective based on historical data and task context.

Images