A multi-agent endogenous model training method and system

Abstract

The application belongs to the technical field of artificial intelligence, and provides a multi-agent endogenous model training method and system, which samples tasks, extracts embedding representations of each Agent and meta-feature vectors for task execution, combines the embedding representations of each Agent, the meta-feature vectors for task execution and a self-supervised graph neural network to generate a cooperation probability of each pair of Agents to form an Agent cooperation plan matrix; corresponding communication resources are allocated according to the Agent cooperation plan matrix, and each Agent performs asynchronous message interaction on an interaction bus based on the allocated communication resources to obtain an Agent output distribution, and the Agent output distribution is mapped into a global reward vector and a target of a final task. End-to-end differentiable multi-agent cooperation optimization is realized, and the generalization ability and resource utilization of the model are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a multi-agent endogenous model training method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Existing multi-agent frameworks (such as AutoGen, LangGraph, etc.) usually call several role agents to complete complex tasks through API in the reasoning stage, and the collaboration logic is explicitly controlled by external orchestration scripts. Training and reasoning are separated, and it is difficult for the model to truly "learn" the collaboration strategy. This external dependence leads to slow response of the system when facing dynamic tasks, and the collaboration knowledge cannot be internalized through end-to-end optimization, limiting the generalization ability of the model.

[0004] In 2024, OpenAI o1 first wrote the Chain-of-Thought (CoT) capability into the pre-training process; in 2025, Grok 4 Heavy further allowed multiple agents to participate in debates and self-checks during training, which is referred to as "multi-agent endogenization" in the industry. For example, Grok 4 Heavy significantly improves the model's performance on abstract reasoning tasks (such as the ARC-AGI benchmark) by conducting multi-agent debates through reinforcement learning on a pre-training scale. This method proves that embedding collaboration capabilities within the model can significantly improve complex reasoning and cross-domain generalization performance. However, existing multi-agent endogenization schemes (such as MAPoRL and MAS-GPT) still mainly use fixed Agent numbers and static roles, lack task-oriented autonomous orchestration and dynamic evolution mechanisms, and do not address key issues such as cross-agent gradient alignment and flexible resource utilization. Although these schemes introduce multi-agent interaction during training, the collaboration topology is often predefined and cannot adapt to task complexity, leading to performance bottlenecks in long-context or multi-modal tasks. SUMMARY

[0005] To solve at least one of the technical problems in the background art, the present application provides a multi-agent endogenous model training method and system, which realizes end-to-end differentiable multi-agent collaboration optimization, significantly improves the generalization ability and resource utilization of the model compared to existing external frameworks, and supports cross-modal expansion and bootstrap evolution.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] The first aspect of the present application provides a multi-agent endogenous model training method, comprising the following steps:

[0008] training preparation and training initialization are performed;

[0009] sampling tasks, extracting embedding representations of each Agent and meta feature vectors of performing tasks, combining the embedding representations of each Agent, the meta feature vectors of performing tasks and the self-supervised graph neural network to generate a cooperation probability of each pair of Agents, and forming an Agent cooperation plan matrix;

[0010] allocating corresponding communication resources according to the Agent cooperation plan matrix, each Agent performing asynchronous message interaction on the interaction bus based on the allocated communication resources to obtain an Agent output distribution, and mapping the Agent output distribution to a global reward vector and a target of the final task.

[0011] Further, training preparation and training initialization are performed, including initialization of the main model and each Agent in the hardware environment, initialization of the meta scheduler, training configuration and determination of a safety policy.

[0012] Further, the cooperation probability calculation formula of each pair of Agents is:

[0013] ,

[0014] ,

[0015] wherein, represents the cooperation probability of to , is a normalization function to ensure that the probability sum is 1, Attention is an attention mechanism to calculate the similarity between the task feature and the role embedding, is the meta feature vector of the task T, is the embedding representation of the role, is the embedding representation of the role.

[0016] Further, after generating the cooperation probability of each pair of Agents, a differentiable function is used to map the cooperation probability of each pair of Agents to [0, 1].

[0017] Further, the allocation of corresponding communication resources according to the Agent cooperation plan matrix includes determining the total available communication resources, including the total bandwidth or the number of channels, and allocating resources according to the cooperation probability, including that the resources allocated to each pair of Agents are proportional to their cooperation probability, and finally ensuring that the total sum of the allocated resources does not exceed the total amount.

[0018] ​​Further, each Agent performs asynchronous message interaction on the interaction bus based on the allocated communication resources to obtain the output distribution of each Agent, and exchanges through a high-throughput message queue and a bidirectional memory cache mechanism.

[0019] Further, at each interaction step, a Transformer-based encoder is used to realize differentiable propagation, and the message representation of interaction is:

[0020] ,

[0021] wherein, represents the message vector sent to ; is the hidden state of , is the hidden state of , is a multi-layer perceptron for nonlinear transformation; is a vector splicing operation.

[0022] Further, contrastive learning is introduced, and the global reward vector is obtained by minimizing the KL divergence of the multi-agent output distribution, represented as:

[0023] ,

[0024] wherein, represents the global reward vector; represents the maximization of the reward candidate r ; represents the logarithmic sum of conditional probabilities, measuring the likelihood of the final answer under the reward r ; β is a balance hyperparameter; represents the joint output distribution of multi-agents, which is the product of the output distribution of each agent, wherein, is the Boltzmann distribution based on the reward vector r , wherein, is the feature mapping vector of the output of the agent , is a normalization constant, Q represents a preset reference distribution (such as a uniform distribution), represents the KL divergence of the output distribution P and the reference distribution Q , used for regularization.

[0025] Further, the total loss function of training is:

[0026] ,

[0027] ,

[0028] wherein, denotes the total loss; denotes the task-specific loss, denotes the weight hyper-parameter of the alignment loss, denotes the weight hyper-parameter of the scheduling loss; denotes the scheduler optimization loss, denotes the alignment loss; denotes the matrix trace operation; denotes the covariance matrix, calculating the covariance between the gradient vectors; to denotes the gradient vector of each Agent parameter , which is obtained by taking the partial derivative of the global objective function of the multi-agent system with respect to the parameter of each agent .

[0029] The second aspect of the present application provides a multi-agent endogenous model training system, comprising:

[0030] a training initialization module for training preparation and training initialization;

[0031] a multi-agent endogenous module for analyzing sampling tasks in collaboration with agents and extracting embedding representations of each Agent and meta-feature vectors for performing tasks, combining the embedding representations of each Agent, the meta-feature vectors for performing tasks, and a self-supervised graph neural network to generate a collaboration probability for each pair of Agents, and composing an Agent collaboration plan matrix;

[0032] allocating corresponding communication resources according to the Agent collaboration plan matrix, each Agent performing asynchronous message interaction on the interaction bus based on the allocated communication resources to obtain an Agent output distribution, and mapping the Agent output distribution to a global reward vector and a target of the final task.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] The application embeds the complex agent cooperation behavior into the pre-training and fine-tuning process of the model, and the model can autonomously learn and optimize its internal decision logic, dynamic role allocation strategy and gradient consistency optimization between agents, thereby significantly improving its processing of complex reasoning tasks, implementation of cross-domain knowledge generalization and efficient use of computing resources, solving the problems of training and reasoning being separated and being unable to internalize cooperation strategies; at the same time, the static role design and non-differentiable interaction limit the self-organizing ability and gradient optimization efficiency of the model.

[0035] Advantages of the additional aspects of the application will be partially given in the following description, partially will become apparent from the following description, or will be understood by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification. The embodiments of these drawings are set to explain the application, and do not constitute an improper limitation to the application.

[0037] Figure 1 is a multi-agent endogenous model training method flowchart provided by an embodiment of the application;

[0038] Figure 2 is a multi-agent endogenous model training system block diagram provided by an embodiment of the application;

[0039] Figure 3 is a structural schematic diagram of a computer device provided by an embodiment of the application. DETAILED DESCRIPTION

[0040] The application will be further described below in conjunction with the drawings and embodiments.

[0041] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0042] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or combination thereof.

[0043] With the research and progress of artificial intelligence technology, artificial intelligence technology is researched and applied in multiple fields, such as common smart home, smart wearable device, virtual assistant, automatic driving, unmanned aerial vehicle, robot, intelligent medical treatment and the like. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important value.

[0044] The scheme provided by the embodiments of the present application can involve computer vision, machine learning and other related technologies of artificial intelligence. The specific implementation is described as follows.

[0045] In the embodiments provided by the present application, the plurality of agents included in the agent system can be hardware devices or software programs in form.

[0046] When the agent is a hardware device, the agent can include a processor, a memory and the like inside. The memory is used to store data related to the processing of the agent, for example, it can be program code related to the processing of the agent. The processor can load and process the data stored in the memory, and then implement the method for determining the processing of the agent provided by the present application and the like. In addition, the agent can also include a communication component for communicating with other agents in the multi-agent system to realize data transmission between the agents. When the agent is a hardware device, the agent can be an unmanned aerial vehicle, a robot, a vehicle and the like.

[0047] When the agent is a software program, the computer device running the agent can include a processor, a memory and the like. The memory is used to store data related to the processing of the agent, for example, it can be program code related to the agent. The processor can load and process the data stored in the memory, and then implement the method for determining the processing of the agent provided by the present application and the like. The plurality of agents in the multi-agent system can run on the same computer device. The computer device can start a corresponding processing process for each agent for running the corresponding agent, and can realize data transmission between the agents based on the communication between the processes. When the agent is a software program, the agent can be an AI game character and the like.

[0048] In the multi-agent system, each agent is in the same environment, and each agent can execute the set task in the environment by mutual cooperation. In the process of executing the task, the agent needs to determine the corresponding environmental features of the surrounding environment according to the preset decision period, and then calculates the environmental features through the strategy network to obtain the processing that the agent needs to execute in each decision period. For example, when the agent is a soccer robot, the game field where the soccer robot is located is the environment where each agent is located, and winning the game is the task set for the soccer robot. The processing executed by the soccer robot in each decision period can include executing different specified actions and the like.

[0049] The technical problems to be solved by the endogenized multi-agent model training method at the present stage include:

[0050] 1. The existing method relies on external scripts, which limits the adaptability of the model to unknown tasks. How to realize the "self-organization" of the agent in the training stage instead of artificial script hard coding;

[0051] 2. The interaction signal in the traditional multi-agent framework is often non-differentiable, which leads to the interruption of gradient propagation and cannot be optimized end-to-end. How to effectively convert the high-dimensional signal generated by multi-agent interaction into a differentiable gradient of the main model and each sub-model;

[0052] 3. The multi-agent system is prone to resource waste and gradient conflict, affecting the stability of training. How to maintain parallel efficiency and stable convergence on a large-scale cluster with limited computing power;

[0053] The present application proposes an "endogenized multi-agent training (EMAT)" framework, which innovatively embeds multi-agent collaboration into the training loop, realizes end-to-end optimization through self-supervised learning and differentiable operators, and solves the limitations of static roles and external arrangement. End-to-end differentiable multi-agent collaboration optimization is realized.

[0054] Figure 1 A flowchart of a multi-agent endogenized model training method is shown, see Figure 1 The embodiment includes the following steps:

[0055] Step 1: training preparation and training initialization;

[0056] In this embodiment, the training preparation and initialization specifically include the initialization of the main model and each Agent in the hardware environment, the initialization of the meta-scheduler, and the training configuration and determination of the safety policy;

[0057] Specifically, the hardware environment of the embodiment uses a starting scale of 512xH200 GPU, and uses a mixed strategy of tensor parallelism + pipeline parallelism;

[0058] The initialization of the main model in the hardware environment includes loading 70B level base weights, such as Llama 3;

[0059] The initialization of each Role-Specialized Agent includes sharing a word table, adjusting 2% to 3% parameters using LoRA-Adapter, and supporting hot plugging;

[0060] Role-Specialized Agents are agents designed or trained to specialize in specific roles or perform specific types of tasks in artificial intelligence, particularly in agent systems based on large language models. For convenience of representation, Role-Specialized Agents are simplified as Agents.

[0061] Initialize the initialization meta-scheduler Meta-Scheduler as a self-supervised graph neural network (GNN) to generate a differentiable collaboration plan matrix P;

[0062] Meta-Scheduler is a framework for dynamically adjusting resource allocation strategies, mainly used to balance system efficiency and fairness, especially in handling data-intensive computing tasks to achieve efficient scheduling of resources. Meta-Scheduler adjusts the scheduling priority and resource allocation ratio of different components dynamically to achieve the trade-off between efficiency and fairness. For example, in a multi-task environment, it can automatically adjust the execution order according to the task urgency and resource occupation, avoiding the decline of overall efficiency caused by long-term occupation of resources by a single task.

[0063] The training configuration includes Optimizer (Lion + Gradient Centralization), learning rate (2e-5 main model, 5e-5 Scheduler&Agents), batch size (256 tasks / batch), and debate rounds k=3.

[0064] Security mechanism: inject content security filter and bias detector in Consensus Reducer; implement task-level security policy selection in Scheduler.

[0065] Consensus Reducer is a mechanism or component used in multi-agent systems or distributed systems, mainly used to gather and integrate the opinions, results or inferences of multiple agents or nodes to form a unified decision or consensus. It is commonly used when multiple subjects need to collaborate, have different opinions or conflicts, helping the system to automatically and effectively reach a consensus.

[0066] Step 2: Sample tasks and extract the embedding representation of each Agent and the meta-feature vector of the task, combine the embedding representation of each Agent, the meta-feature vector of the task, and the self-supervised graph neural network to generate the collaboration probability of each pair of Agents, and form an Agent collaboration plan matrix;

[0067] Specifically, the following steps are included:

[0068] Step 201: Sample a task T from a training corpus or a synthetic task pool;

[0069] The training corpus or the synthetic task pool is based on historical collaborative data or public datasets; assuming there is a task distribution from which tasks can be sampled, each task can be represented as a dataset or a set of observations; for example, in multi-agent reinforcement learning (MARL), the task can be an environment instance that requires multiple agents to cooperate to solve.

[0070] Step 202: Based on the sampled task, extract the meta-feature vector of each Agent executing task T ;

[0071] In this embodiment, an embedding model is used to extract the meta-feature vector of task T , including the structural features, dynamic features, and collaboration features of the task, and the meta-feature vector is obtained by concatenating the standardized structural features, dynamic features, and collaboration features of the task ;

[0072] Among them, the structural features of the task include task dependency graph diameter, critical path length, parallelism factor, coupling degree of subtasks, etc.; the dynamic features include state change entropy, resource demand volatility, environmental disturbance index, etc.; the collaboration features include historical collaboration success rate, communication delay distribution, conflict probability, etc.

[0073] It should be noted that the structural features, dynamic features, and collaboration features of the task are extracted using existing feature extraction networks, for example, the structural features can be analyzed using a graph neural network, the dynamic features can be analyzed using an LSTM time series, and the collaboration features can be obtained by statistical analysis of collaboration history; the features extracted here can be determined according to actual needs. Since the dimensions of the obtained features differ, the structural features, dynamic features, and collaboration features of the task are standardized before being concatenated to obtain the meta-feature vector .

[0074] For example, there is a set of text classification tasks, the purpose of which is to analyze the difficulty of each task, and for each task T, "task description text" or "task examples" are obtained; an embedding model such as BERT is used to extract these vectors from the text and convert them into task difficulty representation vectors, denoted as ; After a certain training / tuning, the vector distribution of "complex tasks" in the vector space is close, which facilitates subsequent clustering, difficulty assessment, and transfer learning operations.

[0075] Step 203: Combine the embedding representation of each Agent, the meta-feature vector of executing the task, and the self-supervised graph neural network to generate the collaboration probability of each pair of Agents, forming an Agent collaboration plan matrix;

[0076] In this embodiment, the Meta-Scheduler is based on meta-features Real-time generation of Agent topology and process, calculation of collaboration plan matrix n The number of Agents), wherein, represents the probability of sending messages, and the embedding representation of each Agent and the meta-feature vector of the executed task are taken as the input of the self-supervised GNN, and the collaboration probability between Agents is generated through self-supervised task training combined with an attention mechanism; The calling / collaboration probability of each pair of Agents is represented as:

[0077]

[0078] ,

[0079] ,

[0080] wherein, represents the collaboration probability from to is a standardization function, ensuring that the probability sum is 1, is an attention mechanism, calculating the similarity between the task feature and the role embedding; is the meta-feature vector of the task T, is the embedding representation of the role, is the embedding representation of the role, ensuring that the probability distribution satisfies , and realizing adaptive resource allocation. In order to ensure the differentiability of the collaboration plan, this embodiment uses a differentiable function such as sigmoid to map to [0, 1], and the entire generation process from input, to GNN embedding, to collaboration matrix

[0081] is differentiable at each step, so it can be trained through backpropagation. This step supports dynamic adjustment of the number of Agents , and according to the GPU / TPU queue congestion, the resource elasticity allocation is realized.

[0082] Step 3: According to the Agent collaboration plan matrix, the corresponding communication resources are allocated, and each Agent performs asynchronous message interaction on the interaction bus based on the allocated communication resources to obtain an Agent output distribution, and the Agent output distribution is mapped to a global reward vector and a final task target;

[0083]

[0084] ​​​​Specifically comprising the following steps:

[0085] Step 301, distributing corresponding communication resources according to the Agent cooperation plan matrix;

[0086] In this embodiment, when distributing corresponding communication resources according to the Agent cooperation plan matrix, each element represents the cooperation probability between Agent pairs. First, the total available communication resources, such as the total bandwidth or the number of channels, are determined. Then, the resources are distributed according to the cooperation probability, for example, using proportional distribution, that is, the resources allocated to each pair of Agents are proportional to their cooperation probability, and finally ensuring that the sum of the allocated resources does not exceed the total amount, generating a resource allocation matrix, which shows the specific amount of resources allocated to each pair of Agents.

[0087] Step 302, each Agent Based on the allocated communication resources, asynchronous message interaction is carried out on the interaction bus to obtain each Agent Agent output distribution;

[0088] In this embodiment, each Agent According to the allocated communication resources, asynchronous communication is carried out on the interaction bus Interaction Bus, and k-step loop iteration of asynchronous debate (Debate), criticism (Critique), and refinement (Refine) is carried out to obtain the results of multiple rounds of debate;

[0089] It should be noted that the above each Agent Asynchronous debate (Debate), criticism (Critique), and refinement (Refine) are carried out in the k Step loop iteration process can be realized by using existing mechanisms;

[0090] Further, when each Role-Specialized Agent asynchronously communicates on the interaction bus Interaction Bus, it communicates through a high-throughput message queue and a bidirectional memory cache mechanism. The high-throughput message queue supports asynchronous concurrent communication (such as RabbitMQ / Kafka) and ensures low-latency transmission. The bidirectional memory cache mechanism includes short-term cache and long-term memory. The short-term cache stores the intermediate state of the current debate (such as argument tree and evidence pool), and the long-term memory archives historical decision logic for subsequent task retrieval and reference.

[0091] Further, at each step of interaction, a Transformer-based encoder is used to realize differentiable propagation, reduce information loss, and the message representation of interaction is:

[0092] ,

[0093] Among them, represents message vector sent to ; for hidden state, for hidden state, MLP is a multi-layer perceptron for nonlinear transformation; is a vector concatenation operation;

[0094] Step 303, mapping each agent output distribution to a global reward vector R and the target of the final task, and performing cross-agent gradient alignment ground-truth or self-generated teacher signal;

[0095] In this embodiment, contrastive learning is introduced, and the multi-agent output distribution is minimized using KL divergence to obtain the global reward vector, denoted as:

[0096] ,

[0097] wherein, represents the global reward vector; represents the reward candidate r maximization; represents the conditional probability logarithmic sum, measuring the likelihood of the final answer under the reward r ; β is a balance hyperparameter; represents the joint output distribution of multiple agents, which is the product of the output distribution of each agent, wherein, is the Boltzmann distribution based on the reward vector r , wherein, the agent characteristic mapping vector of the output , is a normalization constant, Q represents a preset reference distribution (such as a uniform distribution), represents the KL divergence of the output distribution P and the reference distribution Q , used for regularization;

[0098] This formula ensures unbiased estimation of consensus. This step provides fine-grained error signals to improve multi-role self-checking and adversarial debate effect.

[0099] In this embodiment, during training, cross-agent gradient alignment and conflict elimination are performed, and the alignment loss based on cooperative covariance minimization is:

[0100]

[0101] where, represents the alignment loss; represents the matrix trace operation; represents the covariance matrix, the covariance between the gradient vectors is calculated; to represents the gradient vector of each Agent parameter , is obtained by taking the partial derivative of the global objective function of the multi-agent system with respect to the parameters of each agent , where the output distribution of each agent is parameterized by the parameters , which can be network weights if the agent is a neural network model, for each agent , the gradient vector is specifically calculated as:

[0102] ,

[0103] The first term is the partial derivative of the Boltzmann distribution, and the second term is the contribution of the KL regularization.

[0104] Finally, the parameters of the master model and each Agent are trained by backpropagation through the joint loss, and the total loss of the training is represented as:

[0105] ,

[0106] where, represents the total loss; represents the task-specific loss (such as cross-entropy), represents the weight hyperparameter of the alignment loss, represents the weight hyperparameter of the scheduling loss; represents the scheduler optimization loss.

[0107] The whole process of the present application adopts continuous relaxation (such as Gumbel-Softmax) to ensure differentiability and support end-to-end optimization.

[0108] In the inference phase, the Agent whose cooperation probability is greater than the set threshold is enabled according to the actual task requirement;

[0109] In this embodiment, the Agent with an activation probability threshold greater than 0.4 is activated by default to realize lightweight inference with a time complexity of O( / m) ) ; if extreme performance is required, all Agents can be enabled;

[0110] ​The framework supports scalable applications, including: code auto-generation and review (an endogenous closed loop formed by three types of Agents, "generation-review-unit test", improving code quality by 40% based on SWE-Bench); cross-modal reasoning (visual and voice Agents mounted with LoRA-Vision / Audio Adapter, supporting video understanding and improving ARC-AGI performance); bootstrap model evolution (meta-learning ability of Meta-Scheduler periodically outputs next-generation topology, realizing a "co-evolution" double cycle, drawing on MAS-GPT, and improving long-term generalization).

[0111] The present application proposes an "endogenized" training mode that deeply integrates Multi-Agent autonomous scheduling and collaboration mechanisms during the model training phase. This mode aims to embed the complex Agent collaboration behaviors traditionally achieved through external scripts or artificial rules during the reasoning phase into the pre-training and fine-tuning processes of the model. Through this endogenized design, the model can autonomously learn and optimize its internal decision logic, dynamic role allocation strategies, and gradient consistency optimization across Agents, thereby significantly improving its comprehensive performance in handling complex reasoning tasks, achieving cross-domain knowledge generalization, and efficiently utilizing computing resources. The present application not only represents a major breakthrough in existing LLM training methods, but also lays a solid foundation for the development of future Artificial General Intelligence (AGI), enabling it to more autonomously and efficiently adapt to and solve unknown complex problems.

[0112] Figure 2 A block diagram of a multi-agent endogenized model training system is shown, referring to Figure 2 The embodiment includes:

[0113] The training initialization module 201 is used for training preparation and training initialization;

[0114] The multi-agent endogenized training module 202 is used for Agent collaboration analysis and sampling tasks, and extracts the embedding representation of each Agent and the meta-feature vector for task execution. The embedding representation of each Agent, the meta-feature vector for task execution, and the self-supervised graph neural network are combined to generate the collaboration probability of each pair of Agents, forming an Agent collaboration plan matrix.

[0115] According to the Agent collaboration plan matrix, the corresponding communication resources are allocated, and each Agent performs asynchronous message interaction on the interaction bus based on the allocated communication resources to obtain the output distribution of each Agent. The output distribution of each Agent is mapped to a global reward vector and a target of the final task.

[0116] It should be noted that the specific implementation of the embodiment of the application is similar to the specific implementation of the embodiment of the application. For brevity, the description of the method part is not repeated here.

[0117] Figure 3 A computer device structure diagram provided by the embodiment of the application. When the agent is a hardware device, the computer device 300 can be the agent in the above embodiment, and when the agent is a software program, the computer device 300 can be used to run the agent. The computer device 300 can be quite different due to different configurations or performances, and can include one or more processors (central processing units, CPUs) 301 and one or more memories 302. The memory 302 stores at least one instruction, which is loaded and executed by the processor 301 to implement the method provided by the above-mentioned various method embodiments. Of course, the computer device 302 can also have a wired or wireless network interface, various input and output interfaces, and other components for realizing the functions of the device, so as to perform input and output. The computer device 300 can also include other components for realizing the functions of the device, which are not described here.

[0118] In an exemplary embodiment, a computer readable storage medium, such as a memory including instructions, is also provided, which can be executed by a processor in a terminal to complete the multi-agent endogenous model training method in the above embodiment. The computer readable storage medium can be non-transitory. For example, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0119] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing related hardware, and the program can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk.

[0120] In an exemplary embodiment, a computer program product is also provided, which includes at least one instruction, which can be loaded and executed by a processor to implement the multi-agent endogenous model training method provided by the application.

[0121] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the environmental characteristics involved in the present application are obtained under full authorization.

[0122] The above only describes optional embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a multi-agent endogenous model, characterized in that, The method comprises the following steps: training preparation and training initialization are performed; a sampling task is performed, and an embedding representation of each agent and a meta feature vector for performing the task are extracted, a cooperation probability of each pair of agents is generated by combining the embedding representation of each agent, the meta feature vector for performing the task and a self-supervised graph neural network, and an agent cooperation plan matrix is formed; communication resources are allocated according to the agent cooperation plan matrix, and each agent performs asynchronous message interaction on an interaction bus based on the allocated communication resources to obtain an agent output distribution, and the agent output distribution is mapped into a global reward vector and a target of a final task; wherein the cooperation probability of each pair of agents is calculated according to the following formula: ， ， wherein, represents to the probability of collaboration, is a normalization function, ensuring the probabilities sum to 1, Attention is an attention mechanism, computing the similarity of task features and role embeddings, is a meta-feature vector for task T, is an embedding representation of a role, is an embedding representation of a role; When each agent performs asynchronous message interaction on the interaction bus based on the allocated communication resources to obtain the agent output distribution, communication is performed through a high-throughput message queue and a bidirectional memory cache mechanism; At each interaction step, a Transformer-based encoder is used to realize differentiable propagation, and the message representation of the interaction is as follows: ， in, express Send to message vector; for The hidden state of for The hidden state of It is a multi-layer perceptron, used for nonlinear transformation; It is a vector concatenation operation. 2.The multi-agent endogenous model training method of claim 1, wherein, Training preparation and training initialization are performed, including initialization of a main model and each agent in a hardware environment, initialization of a meta scheduler and training configuration and determination of a safety policy. 3.The multi-agent endogenous model training method of claim 1, wherein, After the cooperation probability of each pair of agents is generated, a differentiable function is used to map the cooperation probability of each pair of agents to [0, 1]. 4.The method of claim 1, wherein, The allocation of the corresponding communication resources according to the agent cooperation plan matrix comprises the following steps: determining the total available communication resources, including the total bandwidth or the number of channels; and allocating resources according to the cooperation probability, including that the resources allocated to each pair of agents are proportional to the cooperation probability thereof, and finally ensuring that the sum of the allocated resources does not exceed the total amount.

5. The method of claim 1, wherein, Contrastive learning is introduced, and a global reward vector is obtained by minimizing the KL divergence of the multi-agent output distribution, and is expressed as: ， where, represents the global reward vector; represents the reward candidate r maximization; represents the conditional probability log-sum-exp, measuring the final answer likelihood under the reward r ; β is the balancing hyperparameter; represents the multi-agent joint output distribution, which is the product of individual agent output distributions, where, is the Boltzmann distribution based on the reward vector r , where, is the agent feature mapping vector for the output , is the normalization constant, , Q represents a pre-set reference distribution (e.g., uniform distribution), represents the KL divergence of the output distribution P from the reference distribution Q , used for regularization.

6. The method of claim 1, wherein, The total loss function of the training is as follows: ， ， in, Indicates the total loss; represents the task-specific loss, represents the weight hyperparameter of the alignment loss, The weight hyperparameter representing the scheduling loss; represents the scheduler optimization loss, represents the alignment loss; represents the matrix trace operation; Represents the covariance matrix, which calculates the covariance between gradient vectors; arrive Indicates the parameters of each Agent The gradient vector of the global objective function of the multi-agent system Relative to each agent Parameters Find the partial derivative to get .

7. A multi-agent endogenous model training system, characterized in that, The method comprises the following steps: training preparation and training initialization are performed; a sampling task is performed, and an embedding representation of each agent and a meta feature vector for performing the task are extracted, a cooperation probability of each pair of agents is generated by combining the embedding representation of each agent, the meta feature vector for performing the task and a self-supervised graph neural network, and an agent cooperation plan matrix is formed; communication resources are allocated according to the agent cooperation plan matrix, and each agent performs asynchronous message interaction on an interaction bus based on the allocated communication resources to obtain an agent output distribution, and the agent output distribution is mapped into a global reward vector and a target of a final task.

Images