Multi-agent home task tool cooperation method and related product

Abstract

The invention discloses a multi-agent home task tool cooperation method and related products, and belongs to the technical field of multi-agent cooperative control. According to the multi-agent home task cooperation method provided by the invention, a unified structured semantic packet is generated through environment perception, a multi-branch gating communication algorithm is adopted to carry out communication value evaluation, and task cooperation is triggered based on an evaluation result; the problem of low-efficiency cooperative communication caused by the fact that the unified communication cost is not learned into the strategy is effectively solved; specifically, a first semantic packet, a second semantic packet and a third semantic packet are scored through a multi-branch gating communication algorithm, a multi-branch gating decision of the current agent is generated, information contents and values of different semantic packets can be comprehensively considered, and the necessity of communication is quantitatively evaluated, so that the communication efficiency is improved. The single agent can accurately judge the necessity and effectiveness of communication, and blind sending of redundant information is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-agent collaborative control, in particular to a multi-agent home task embodiment cooperation method and related products. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, multi-agent collaborative systems have shown great application potential and broad development prospects in many fields, and have become one of the current hotspots of scientific research. The core advantage of multi-agent collaborative systems is that through information sharing and coordination between multiple agents, complex tasks that a single agent cannot handle can be completed. This collaborative working mode breaks the limitations of single agents in terms of ability, resources, etc., fully utilizes the advantages of group intelligence, and provides new ideas and methods for solving complex problems.

[0003] In terms of communication technology in multi-agent collaborative systems, there are currently various frameworks and models that have been proposed and applied in actual systems. Systems represented by CoELA, etc. explicitly model "sending messages" as environmental actions. This design makes communication an important part of the system's running process.

[0004] However, due to the lack of a unified communication value evaluation mechanism in existing systems, agents cannot accurately judge the necessity and effectiveness of communication when making communication decisions. In this case, agents may blindly send a large amount of redundant information, wasting valuable bandwidth resources. SUMMARY

[0005] The purpose of the present application is to provide a multi-agent home task embodiment cooperation method and related products to overcome the problem of bandwidth resource waste caused by redundant communication in existing multi-agent systems.

[0006] The present application solves the above technical problems through the following technical solutions: The present application provides a multi-agent home task embodiment cooperation method, comprising the following steps: Each single agent in the multi-agent system processes the acquired indoor home environment information through object-centered perception and ROI codebook semantic compression to generate a unified structured semantic packet; The first semantic packet, the second semantic packet and the third semantic packet are scored respectively through a multi-branch gating communication algorithm to generate a multi-branch gating decision of the current agent; Based on the multi-branch gating decisions of all agents, the task embodiment cooperation of the multi-agent system is triggered; The multi-branch gating decision includes whether the current single agent receives and fuses the semantic packet of the single agent A, whether the current single agent point-to-point sends the semantic packet of the current single agent to the single agent B, and whether the current single agent calls a large model to update an action plan.

[0007] The multi-branch gating communication algorithm is specifically as follows:

[0008] The score is a score of the first semantic packet. The score is a score of the second semantic packet. The score is a score of the third semantic packet. The score is a score of the third semantic packet. The score is a score of the third semantic packet. The score is a score of the third semantic packet. The score is a score of the third semantic packet. The score is a score of the third semantic packet. The score is a score of the third semantic packet.

[0009] The multi-branch gating decision of the current agent is specifically as follows: When the score of the first semantic packet is greater than zero, the decision is that the current single agent receives and fuses the semantic packet of the single agent A. When the score of the second semantic packet is greater than zero, the decision is that the current single agent point-to-point sends the semantic packet of the current single agent to the single agent B. When the score of the third semantic packet is greater than zero, the decision is that the current single agent calls a large model to update an action plan.

[0010] When the current single agent calls a large model to update an action plan, the following steps are further included: Based on the third semantic packet, a natural language one-sentence summary for environment interaction is generated, and the natural language one-sentence summary is used to call a large model to update an action plan.

[0011] The single agent processes indoor home environment information through object center perception and ROI codebook semantic compression, generates a unified structured semantic packet, and specifically includes the following steps: An RGB image of indoor home environment information is acquired. Instance segmentation and target detection are performed on the RGB image by using a pre-trained Mask-RCNN, sparse 3D point clouds and bird's eye view representations are constructed, and high-dimensional semantic features are formed; The high-dimensional semantic features are discretized and compressed by using a VQ codebook, an ROI region is selected, and a unified structured semantic packet is generated.

[0012] Further improvements of the application are that the generation of the unified structured semantic packet further comprises the following steps: preprocessing the semantic packet, and the preprocessing is filtering and alignment.

[0013] The application also provides a multi-agent home task embodiment cooperation system, which comprises: The first module is configured to generate a unified structured semantic packet by subjecting the acquired indoor home environment information to object center perception and ROI codebook semantic compression processing in each single agent in the multi-agent. The second module is configured to score the first semantic packet, the second semantic packet and the third semantic packet respectively by using a multi-branch gating communication algorithm, and generate a multi-branch gating decision of the current agent; the multi-branch gating decision comprises whether the current single agent receives and fuses the semantic packet of the single agent A, whether the current single agent sends the semantic packet of the current single agent to the single agent B, and whether the current single agent calls a large model to update an action plan; the first semantic packet is the semantic packet of the single agent A; the second semantic packet is the semantic packet of the current agent; the third semantic packet is the semantic packet of the current agent and the single agent A; the single agent A is a certain single agent in the multi-agent which sends a message to the current single agent; and the single agent B is all single agents in the multi-agent except the current single agent The third module is configured to trigger the task embodiment cooperation of the multi-agent based on the multi-branch gating decisions of all agents.

[0014] The application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the multi-agent home task embodiment cooperation method when executing the computer program.

[0015] The application also provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the multi-agent home task embodiment cooperation method when executed by a processor.

[0016] The application also provides a computer program product comprising a computer program, and the computer program implements the steps of the multi-agent home task embodiment cooperation method when executed by a processor.

[0017] Compared with the prior art, the application has the following positive progress effects: The multi-agent home task embodiment cooperation method provided by the application generates a unified structured semantic packet through environment perception, adopts a multi-branch gated communication algorithm to perform communication value evaluation, and triggers task cooperation based on the evaluation result, effectively overcoming the problem of inefficient cooperative communication caused by not using a unified communication cost strategy; specifically, the multi-branch gated communication algorithm is used to score the first semantic packet, the second semantic packet and the third semantic packet respectively, to generate a multi-branch gated decision of the current agent, which can comprehensively consider the information content and value of different semantic packets, quantitatively evaluate the necessity of communication, enable a single agent to accurately judge the necessity and effectiveness of communication, and avoid blind sending of redundant information. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of this application. The schematic embodiments of the application and their descriptions are used to explain the application, and do not constitute an improper limitation on the application.

[0019] Figure 1 A flowchart of a multi-agent home task embodiment cooperation method in an embodiment of the application.

[0020] Figure 2 A schematic diagram of an object-centered perception and ROI codebook semantic compression algorithm constructed in an embodiment of the application.

[0021] Figure 3 A schematic diagram of a value-cost integrated multi-branch gated communication algorithm constructed in an embodiment of the application.

[0022] Figure 4 A schematic diagram of a multi-agent indoor embodiment cooperation system for efficient communication constructed in an embodiment of the application.

[0023] Figure 5 A schematic diagram of a multi-agent home task embodiment cooperation system in an embodiment of the application. DETAILED DESCRIPTION

[0024] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings of the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0025] In the description of the present application, it should be understood that the terms "include" and "contain" indicate the presence of described features, whole, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0026] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0027] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe the preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range without departing from the scope of the embodiments of the present application.

[0028] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detecting (a stated condition or event)" can be interpreted to mean "when determined" or "in response to determining" or "when detecting (a stated condition or event)" or "in response to detecting (a stated condition or event)".

[0029] The present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments, which are an explanation of the present application rather than a limitation.

[0030] Referring to Figure 3 The present application provides a multi-agent home task embodiment cooperation method, comprising the following steps: Each single agent in the multi-agent processes the acquired indoor home environment information through object-centered perception and ROI codebook semantic compression, to generate a unified structured semantic packet; The first semantic packet, the second semantic packet and the third semantic packet are scored respectively through a multi-branch gating communication algorithm, to generate a multi-branch gating decision of the current agent; Based on the multi-branch gating decisions of all agents, the task embodiment cooperation of the multi-agent is triggered; The multi-branch gating decision includes whether the current single agent receives and fuses the semantic packet of the single agent A, whether the current single agent point-to-point sends the semantic packet of the current single agent to the single agent B, and whether the current single agent calls a large model to update an action plan.

[0031] The multi-agent home task embodiment cooperation method provided by the application generates a unified structured semantic packet through environment perception, adopts a multi-branch gating communication algorithm to evaluate communication value, and triggers task cooperation based on the evaluation result, effectively overcoming the problem of inefficient cooperative communication caused by not using a unified communication cost learning strategy.

[0032] Specifically, the multi-branch gating communication algorithm is specifically:

[0033] Among them, is a score; is a reward gain; is a weight coefficient of information gain; is information gain; is a weight coefficient of bit value; is a bit value; is a weight coefficient of time delay; is a time delay.

[0034] Specifically, the multi-branch gating decision of the current agent is specifically: When the score of the first semantic packet obtained by the multi-branch gating communication algorithm is greater than zero, the decision is that the current single agent receives and fuses the semantic packet of the single agent A; When the score of the second semantic packet obtained by the multi-branch gating communication algorithm is greater than zero, the decision is that the current single agent point-to-point sends the semantic packet of the current single agent to the single agent B; When the score of the third semantic packet obtained by the multi-branch gating communication algorithm is greater than zero, the decision is that the current single agent calls a large model to update an action plan.

[0035] Specifically, when the current single agent calls the large model to update the action plan, the following steps are further included. Based on the third semantic packet, a natural language one-sentence summary for environment interaction is generated, and the large model is called to update the action plan through the natural language one-sentence summary.

[0036] Referring to Figure 2 Specifically, the single agent processes the indoor home environment information through object-centric perception and ROI codebook semantic compression to generate a unified structured semantic packet, which includes the following steps: Obtain the RGB image of the indoor home environment information; Use the pre-trained Mask-RCNN to perform instance segmentation and target detection on the RGB image, construct a sparse 3D point cloud and a bird's eye view representation, and form a high-dimensional semantic feature; Discretize and compress the high-dimensional semantic feature using a VQ codebook, select an ROI region, and generate a unified structured semantic packet.

[0037] Preferably, generating a unified structured semantic packet further includes the following steps: preprocessing the semantic packet, which is filtering and alignment.

[0038] Embodiment one Referring to Figure 1 and Figure 5 A multi-agent home task embodiment cooperation method includes the following steps: Step S1, construct an object-centric perception and ROI codebook semantic compression framework: use the pre-trained Mask-RCNN to perform instance segmentation and target detection on the RGB image of the single agent, construct a sparse 3D point cloud and a bird's eye view (BEV) representation based on RGB-D depth estimation, and form a structured high-dimensional semantic feature of the object slot. Further design an ROI adapter, discretize and compress the high-dimensional semantic feature using a VQ codebook, and select the most information valuable object and ROI region to generate a unified structured semantic packet and a natural language one-sentence summary for environment interaction, thereby realizing semantic sparsification and bit number reduction of information communication. The framework is mainly composed of a feature extraction perception module and an ROI codebook semantic compression module.

[0039] Step S2: Constructing a multi-branch gating communication framework integrating value and cost: Based on a lightweight MLP network, a delayed asynchronous adapter is designed to perform time alignment, validity filtering, and extrapolation processing on asynchronous communication messages from other agents. Then, based on a unified value-cost function, a multi-output gating mechanism is designed to comprehensively estimate the reward gain, information value, communication cost, and latency cost of actions, performing multi-branch gating decisions. Finally, the information from the gating decisions is integrated into the single agent's world model by the fusion module. This establishes a multi-branch gating communication algorithm integrating value and cost. This framework mainly consists of three parts: an information filtering and alignment module, multi-branch gating, and a world knowledge fusion module.

[0040] See Figure 4 Step S3: Based on decentralized multi-agent collaboration, construct an efficient multi-agent indoor embodied collaboration system: For each agent, based on the decision result of the gating system, communication, plan update, and executor response are determined by event triggering: If communication message reception is triggered, the filtered and aligned messages from other agents are integrated into the local world knowledge; if communication is triggered, local small packets are sent for information exchange; if plan modification is triggered, the LLM (Large Language Model) is called to update the plan and execute the corresponding actions; if no trigger is triggered, the original plan is maintained, realizing decentralized closed-loop efficient collaboration of "perception-communication-planning-execution". This closed-loop strategy reduces communication bandwidth and latency overhead, and improves the robustness of asynchronous communication and the stability of task execution.

[0041] This embodiment discloses a multi-agent collaborative communication optimization method based on perception priority and value-driven principles. The method includes: modeling unified payment actions to incorporate communication, replanning, and querying operations into a value-cost trade-off framework, constructing a closed-loop decision-making mechanism between information value and transmission cost, achieving intelligent judgment of communication necessity and optimal resource allocation; adopting the See-Then-Say perception priority principle and structured semantic representation, and achieving the best balance between semantic fidelity and bandwidth saving through vector quantization codebooks and information filling strategies; designing three types of plug-and-play adapters—delay-asynchronous, collaborative fusion, and region-of-interest codebooks—to address key challenges such as time asynchrony, information heterogeneity, and semantic redundancy, ensuring stable system operation under non-ideal conditions; establishing a unified gating mechanism based on advantage regression and information gain, and achieving adapter capability cultivation and system-level joint optimization through a two-stage training strategy; reducing communication overhead in multi-agent systems, improving latency robustness and collaborative efficiency, and significantly reducing bandwidth consumption and energy consumption while ensuring task performance, providing a feasible technical path for the practical deployment of multi-agent systems in resource-constrained environments.

[0042] The objective of step S1 is: given a single agent at time... tRGB image and depth images Constructing a high-dimensional structured representation of the object center Bird's-eye view (BEV) representation Then, through semantic compression of the ROI codebook, a unified structured semantic package is generated. and its natural language one-sentence summary The packet meets strict bit budget and robustness fields to support subsequent low-bit communication and redundancy-free fusion of multiple agents.

[0043] Step S1 is mainly implemented by the following two modules: (1) Feature extraction and perception module Using a pre-trained Mask R-CNN, the class of each instance is output. Bounding box Pixel-level mask With confidence level Mask R-CNN adds parallel branches to the Faster R-CNN backbone to predict instance masks, combining detection and instance segmentation capabilities and can be reused for downstream ROI alignment. This enables the backbone feature pyramid... The appearance representation is obtained by pooling the mask region using RoIAlign. .

[0044] Pixels With depth Back projection onto camera coordinates:

[0045] Then, transform the coordinates to the body coordinates using the extrinsic parameter (R,t):

[0046] The position of the object is obtained by taking the mean and covariance of the set of pixels within the mask. ,variance 3D geometric properties, etc.

[0047] Will Projected onto the BEV plane, using a regular grid resolution. With boundary BEV represents the adoption of the BEVFormer paradigm: spatial information is extracted from BEV queries through spatial cross-attention and multi-view features, and historical BEVs are fused using temporal self-attention to obtain the BEV semantic graph. This enables the construction of BEV (bird's-eye view) representation.

[0048] Build an object slot for each instance ,in, The slot set can be estimated from the optical flow of adjacent frames. Long-range tracking via GRU unit:

[0049] in, This "object-centric" paradigm is naturally adapted to combination generalization and downstream decision-making.

[0050] (2) ROI codebook semantic compression module This module will and Discretize to a finite codebook and select the most critical Regions of Interest (ROIs) to generate unified structured semantic packets. With Natural Language Summarization The core objective is to minimize the rate-distortion Lagrange quantity:

[0051] in, For the number of message bits, The distortion function is the representation after fusion. Control rate-distortion tradeoff Count bits by field.

[0052] For each object slot Its BEV region Define information value score:

[0053] in, Used to approximate “observation-fusion entropy reduction” Measuring the difference from historical data (e.g.) ), The entropy represents the uncertainty of the detection. The ROI selection is based on the task requirements (such as occupying critical channels, approaching the target, filling blind spots, etc.).

[0054] VQ-VAE-style vector quantization is used for semantic embedding of the selected object slots. Perform nearest neighbor quantization:

[0055]

[0056] During training, the standard VQ-VAE loss, including commitment loss and codebook update term, is used to learn stable discrete representations.

[0057] in, This indicates that the gradient has stopped. To commit to loss weight, This represents the weight of the codebook update term. This discretization can map high-dimensional continuous features to integer codeword indices, which is bandwidth-friendly and is the key to the compression design of this module, thereby realizing VQ codebook discretization.

[0058] Based on this, a unified structured semantic package is constructed to generate structured message entries with unified fields for each ROI, and these are aggregated in the current step. The field pattern is as follows: <obj_id,cls,ROI / BEV_idx,code_idx[],conf,ETA,reason_code,t_send,TTL,FEC_bits,receiver_ids> In this context, code_idx[] comes from the VQ codebook, ROI / BEV_idx provides the grid coordinates, ETA is the estimated time to reach the key pose / waypoint, reason_code indicates the reason for sending the communication packet (such as occlusion patching / planning conflict), TTL and light FEC_bits are used to enhance out-of-order and packet loss robustness, and the entire communication packet satisfies the bit budget. .

[0059] Bit estimation is the sum of the code lengths of each field:

[0060]

[0061]

[0062] Where M is the number of codewords, and the remaining short fields are each 4–16 bits.

[0063] To ensure compatibility with LLM and specific environment actions, each structured record is templated and mapped to a very short text summary. For example: "The transport box (confidence 0.91) is located 6.2m ahead of the occlusion filler, in channel G(12,27), ETA 4.3s." The text is only used as a natural language summary and compatibility layer, and is uniformly charged on a bit-delay basis to avoid token fluctuations polluting costs.

[0064] In summary, step S1 constructs object center 3D / BEV semantics using Mask R-CNN, and then compresses the high-dimensional semantics into structured semantic packets with controlled bits using ROI selection and VQ codebook. This establishes an object center perception and ROI codebook semantic compression algorithm, providing sparse and critical candidate payloads for subsequent value-cost gated communication.

[0065] Step S2 mainly includes three modules: delayed asynchronous adapter → three-branch unified value cost gating → world knowledge fusion, which unifies the adjudication of "receiving / sending / modifying / querying" under a single gating: using To indicate "the benefit of triggering", use This represents the "cost of triggering". Under the basic conditions of message step-by-step and length-limited settings, this gating uses budget-controllable structured small packets as carriers, coordinates the ROI in step S101 with the sparse representation of the codebook, and finally achieves value-cost closed-loop communication with "few bits, less redundancy, and stable latency".

[0066] (1) Formal definition At any moment , No. The action set of an agent is decomposed into low-level environmental actions. With paid actions , For any paid action during the deployment period Calculate the gating score:

[0067] in, For the corresponding weighting coefficients, when If the condition is met, the action is triggered; otherwise, it remains silent. Advantage regression is employed. To approximate the difference in reward advantage between whether or not the trigger is used To approximate the reduction in belief entropy and the gain in correlation brought about by triggering. Learnable estimates from message protocols and delay adapters.

[0068] (2) Information filtering and alignment module Design a delayed asynchronous adapter to perform validity filtering, time alignment, and short-term extrapolation on inbound asynchronous messages, providing reliable support for gating and fusion while preserving valuable information. Estimation, validity weights, and aligned spatiotemporal references.

[0069] a) Message format and rule-level pre-filtering Inbound messages are uniformly structured communication packets (CommPacket), with the same format as defined above: <obj_id,cls,ROI / BEV_idx,code_idx[],conf,ETA,reason_code,t_send,TTL,FEC_bits,receiver_ids> Design pre-filtering rules for each situation: TTL expired: If →Discard; Lightweight FEC check failure: such as BCH / CRC small code → discard; Duplicate packets: identical<obj_id, t_send> →Remove duplicates; Self-conflict: The same object contradicts each other but originates from the same entity and the time difference is very small → the latest shadow entry is left and does not participate in the fusion.

[0070] b) Out-of-order buffering and time alignment Maintain sliding window buffer Generate alignment features for each candidate message:

[0071]

[0072]

[0073] in, Extrapolation for BEV-flow References to aligned local BEV features; As a time decay constant, the adapter only outputs aligned references and delay distribution estimates for gating use.

[0074] c) Delay estimation and efficiency weights Perform TTL / CRC checks, deduplication, and out-of-order buffering on inbound messages. Address latency. The modeling employs a log-normal non-negative heavy-tailed distribution, and online quantile regression calibration is performed; output With calibrated validity weights This is for subsequent gating, weighted fusion, and "budget selection" applications.

[0075] (3) Multi-branch gating In order to score under a unified value proposition, the three branches are defined as follows: RX-branch: Whether to receive and merge a specific inbound message m; TX-branch: Whether to send local candidate CommPackets (from the ROI codebook packet in step S1) at a fixed point; REPLAN-branch: Whether to trigger plan modification, and then call LLM for query and evaluation (token is recorded in Bits, round trip is recorded in Delay).

[0076] In this system, all branches share the trunk, with only the branch heads being independent. (a) Gated input splicing Constructing the gated input vector :

[0077] in, For local ROI summary, Indicates link status such as delay, packet loss, and out-of-order delivery. This indicates the project's progress and conflict level, used to mitigate fluctuations caused by plan modifications. This represents the candidate packet cost estimate, derived from the codebook in step S1 and the delay estimate in this step.

[0078] (b) Learnable approximation of information value and return gain Return Gain The reward difference between "triggered / not triggered" is approximated by advantage regression, and pairs are constructed using counterfactual replay, minimizing the difference during training. This approach can significantly reduce invalid query / dialogue rounds.

[0079] Information gain Using a weakly supervised approximation of belief entropy reduction, correlation estimation, and trajectory verification, a correlation discriminator can be trained in practice. Output the probability of "strongly correlated with the current plan", and give additional rewards when the occlusion is broken and the short-term prediction is successful.

[0080] (c) Multi-branch scoring and triggering criteria The three branches share a common MLP trunk, and the branch header outputs are as follows: , ,

[0081] All follow a uniform format:

[0082] in, The corresponding weighting coefficients; the rx branch scores each inbound message independently; the tx branch evaluates the local candidate packet set. Scoring and making selections within the budget; the replan branch is triggered by a global event. When The corresponding action is triggered at that time.

[0083] (d) Directional communication If the environment allows for targeted transmission and reception, a targeted selection head is added to the TX-branch to adaptively select which agent(s) to send to. This inherits the ideas of IC3Net and TarMAC from early multi-agent reinforcement learning (MARL) methods for communication, but incorporates the paid action modeling of GateScore.

[0084] (3) World Knowledge Integration Module By performing object-level matching, confidence-weighted fusion, and time consistency constraints on gated inbound messages and the local world model, a shared belief summary for the team will be generated. This is for the next step of the plan's execution and subsequent gate control use.

[0085] (a) Object-level association and conflict resolution Given a set of local objects With arrival at the station Define association cost:

[0086] in For the sender to be confident, This represents the validity weights output by the information filtering and alignment module. Solving the Hungarian matching yields... Unmatched objects enter the temporary pool and participate in map-level fusion with a lower weight.

[0087] (b) Confidence-weighted fusion and time consistency For matching pairs The pose and box parameters are defined as follows:

[0088] Aligned features:

[0089] in, , For time alignment operators, For time consistency weighting function:

[0090] Final output: team-level summary Its entropy measurement This serves as the input for the next step planner and gating.

[0091] During the training process, a two-stage training method is adopted: Stage I trains the adapter, and Stage II trains the gating system. Phase I (Freezing LLM): Delay adapter: Alignment loss During training, a delay of {0,100,250,500}ms is injected, along with out-of-order and packet loss scenarios.

[0092] ROI Codebook Adapter: VQ quantization + sparse regularization makes bits learnable and packets shorter.

[0093] Coop Fusion Adapter: Matching supervision + consistency regularization, outputting stable values. .

[0094] Phase II (Gating / Decision Making): Constructing a "counterfactual replay": In the same Below, training advantage regression head minimization ; and through correlation and weak supervision optimization. Estimation (e.g., successful occlusion removal, improved recall of key targets, etc. as positive samples).

[0095] Total loss:

[0096] in, The cross-entropy is used for relevance, FPR suppresses false triggers, and ECE is used for calibration. Empirically, advantage regression helps reduce invalid queries and rounds.

[0097] Step S3 builds upon the aforementioned object-centric perception and value-cost integrated communication framework to construct an event-triggered decentralized multi-agent collaborative system, achieving closed-loop control of "perception-communication-planning-execution".

[0098] (1) Decentralized collaborative architecture design For each agent in the system Establish local trajectory representations, where the action space is decomposed into a set of environmental interaction actions. With paid communication action set Design a unified optimization objective function that considers both task reward and resource cost:

[0099] in, These are the weighting coefficients for bit overhead, latency cost, query cost, and replanning cost, respectively. The system follows the constraint that message sending occupies a single environment step and has a length limit (≤500 characters / frame).

[0100] For each agent in the multi-agent team, based on the gating mechanism in step S1, for multiple event branches, when At that time, the corresponding gating is triggered, and multi-source information fusion is achieved through the CoopAdapter: the inbound messages passed through the gating are matched with the local world model at the object level, fused with confidence weights, and constrained by time consistency to generate a belief summary shared by the team. .

[0101] (2) Actuator design under safety constraints Actuator Map the updated plan to low-level control actions. A control barrier function (CBF) is introduced to ensure execution safety:

[0102] in For unconstrained planned actions, the system continues to execute the original plan when no payment action is triggered, effectively reducing control jitter and additional latency overhead.

[0103] (3) System operation closed-loop process At each time step, each agent executes the following closed-loop process: 1) A world-centric model based on local observation, inbound message queues, and historical plan updates; 2) Perform TTL checks, forward error correction, and deduplication on inbound messages using a delay adapter to achieve time alignment and status extrapolation; 3) The collaboration and integration module generates an assessment of differences in team shared beliefs and intentions; 4) The ROI codebook adapter generates candidate structured semantic packets and estimates the transmission cost; 5) The unified gate controller calculates the gate score for various payment actions and executes event triggering decisions; 6) The actuator outputs low-level control commands under safety constraints; 7) Record the resource consumption metrics (Bits, Latency, Query, Replan) and value returns for the current step, for online adaptive adjustment and offline performance evaluation.

[0104] The decentralized collaboration framework provided by this invention reduces communication bandwidth requirements and system latency through event triggering mechanism and structured semantic compression, improves robustness and task execution stability under asynchronous communication conditions, and realizes efficient multi-agent collaboration in resource-constrained environments.

[0105] In the TWD-MAT indoor home simulation environment, the average number of frames required to successfully deliver the first item in the two-agent collaborative transportation task under the CoELA framework is 341 frames, the average total number of frames consumed to complete the task is 2835 frames, and the average number of tokens (GPT4) required to query the LLM to complete the task is 3.5 million, which cannot meet the collaborative efficiency required for actual application deployment.

[0106] The collaborative method of this invention can reduce the number of frames consumed by the CoELA method to complete the task in a two-agent collaborative transportation task in a TWD-MAT environment by 47%, reduce the average token consumption to 600,000, and still maintain the task success rate.

[0107] Based on the same inventive concept, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a multi-agent home task embodied collaboration method. The memory may include main memory, such as high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device. The processor, network interface, and memory are interconnected via an internal bus, which may be an industry-standard architecture bus, a peripheral component interconnection standard bus, an extended industry-standard architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory is used to store the program; specifically, the program may include program code, which includes computer operation instructions. The memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0108] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the multi-agent home task embodied collaboration method. Specifically, the computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may include RAM (Random Access Memory) and / or cache memory, etc. The non-volatile memory may include ROM (Read-Only Memory), hard disk, flash memory, optical disk, magnetic disk, etc.

[0109] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described multi-agent home task embodied collaboration method.

[0110] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) containing computer-usable program code.

[0111] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0115] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multi-agent home task embodied collaboration method, characterized in that, Includes the following steps: Each agent in the multi-agent system uses object-centric perception and ROI codebook semantic compression to process the acquired indoor home environment information and generate a unified, structured semantic package. The first semantic packet, the second semantic packet, and the third semantic packet are scored by a multi-branch gating communication algorithm to generate the multi-branch gating decision of the current agent. Based on multi-branch gating decisions of all agents, multi-agent task embodiment collaboration is triggered. The multi-branch gating decision includes whether the current single agent receives and merges the semantic packets of single agent A, whether the current single agent sends its own semantic packets to single agent B at a specific point, and whether the current single agent calls the large model to update the action plan; the first semantic packet is the semantic packet of single agent A; the second semantic packet is the semantic packet of the current agent; the third semantic packet is the semantic packets of the current agent and single agent A; single agent A is a single agent in the multi-agent group that sends a message to the current single agent; single agent B is all single agents in the multi-agent group except the current single agent.

2. The multi-agent home task embodied collaboration method according to claim 1, characterized in that, The multi-branch gated communication algorithm is as follows: in, For fractions; For reward gains; These are the weighting coefficients for information gain; For information gain; The weighting coefficients for the bit values; Bit value; This is the weighting coefficient for time delay; For time delay.

3. The multi-agent home task embodied collaboration method according to claim 1, characterized in that, The multi-branch gating decision of the current intelligent agent is specifically as follows: When the score of the first semantic packet is greater than zero by the multi-branch gated communication algorithm, the decision is: the current single agent receives and merges the semantic packet of single agent A; When the score of the second semantic packet is greater than zero by the multi-branch gating communication algorithm, the decision is: the current single agent sends the semantic packet of the current single agent to the single agent B at a fixed point; When the score of the third semantic packet obtained through the multi-branch gating communication algorithm is greater than zero, the decision is: the current single agent calls the large model to update the action plan.

4. The multi-agent home task embodied collaboration method according to claim 1, characterized in that, When a single agent calls a large model to update its action plan, the following steps are also included: Based on the third semantic packet, a one-sentence natural language summary for environmental interaction is generated, and the action plan is updated by calling the large model through the one-sentence natural language summary.

5. The multi-agent home task embodied collaboration method according to claim 1, characterized in that, A single agent processes indoor home environment information through object-centric perception and ROI codebook semantic compression to generate a unified structured semantic package, specifically including the following steps: Acquire RGB images containing information about the indoor home environment; Using pre-trained Mask-RCNN, we perform instance segmentation and object detection on RGB images, construct sparse 3D point clouds and bird's-eye view representations, and form high-dimensional semantic features. The VQ codebook is used to discretize and compress high-dimensional semantic features, select ROI regions, and generate unified structured semantic packets.

6. The multi-agent home task embodied collaboration method according to claim 1, characterized in that, Generating a unified, structured semantic package further includes the following steps: preprocessing the semantic package, wherein the preprocessing consists of filtering and alignment.

7. A multi-agent home task embodied collaboration system, characterized in that, include: The first module is used by each single agent in the multi-agent system to generate a unified structured semantic package by perceiving the indoor home environment information obtained through object center perception and ROI codebook semantic compression. The second module is used to score the first semantic packet, the second semantic packet, and the third semantic packet respectively using a multi-branch gating communication algorithm to generate the multi-branch gating decision of the current agent. The multi-branch gating decision includes whether the current single agent receives and merges the semantic packet of single agent A, whether the current single agent sends the semantic packet of the current single agent to single agent B at a fixed point, and whether the current single agent calls the large model to update the action plan. The first semantic packet is the semantic packet of single agent A; the second semantic packet is the semantic packet of the current agent; the third semantic packet is the semantic packet of the current agent and single agent A; single agent A is a single agent in the multi-agent group that sends a message to the current single agent; single agent B is all single agents in the multi-agent group except the current single agent. The third module is used to trigger multi-agent task embodied collaboration based on multi-branch gating decisions of all agents.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multi-agent home task embodied collaboration method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-agent home task embodied collaboration method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multi-agent home task embodied collaboration method according to any one of claims 1 to 6.

Images