Multi-agent cooperation mechanism generation method and device based on PPO, equipment and medium

Abstract

The invention relates to the technical field of intelligent decision, and discloses a PPO-based multi-agent cooperation mechanism generation method, device, equipment and medium, and the method comprises the steps: generating a single-agent output action through PPO, an independent agent and a current state vector, and calculating a single-agent advantage value according to GAE; and generating an agent cooperation mechanism according to the negative extrinsic punishment mechanism, the priority rule, the single agent output action and the single agent advantage value. Through the above mode, the heterogeneous data is converted into the unified state vector, the single-agent action is generated through the PPO algorithm, and the single-agent advantage value is calculated through the GAE algorithm, so that the problems of training oscillation and convergence difficulty of multiple agents are inhibited, and through a penalty mechanism and a priority rule, the multi-agent training efficiency is improved. The problem of deviation between local optimum and global target of multiple agents is solved. The method can be applied to the business fields such as financial science and technology and medical health care, and the reliability of parallel processing of multiple businesses by the intelligent customer service system is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent decision-making technology, and in particular to a method, apparatus, device and medium for generating a multi-agent cooperation mechanism based on PPO. Background Technology

[0002] Driven by the wave of digital transformation, fintech and healthcare / elderly care are undergoing profound business model integration. In the fintech sector, taking the insurance industry as an example, the user journey has evolved from a linear process to a dynamic network covering the entire lifecycle and multiple touchpoints, encompassing "risk identification, product adaptation, service response, and relationship maintenance." In the healthcare / elderly care sector, users expect a one-stop, continuous service experience, ranging from health consultations, disease prevention, and treatment support to long-term care and retirement planning.

[0003] However, the current technical architecture supporting such integrated businesses generally adopts single-agent systems or rule-engine-driven automation solutions, which expose the following structural defects when dealing with complex and dynamic user needs across multiple domains. Existing systems typically build independent agents for single business scenarios (such as insurance product recommendations or online health Q&A). These agents lack effective communication and collaboration mechanisms, forming "decision silos." For example, when health management software detects a user's blood pressure continuously rising, this critical risk signal cannot be effectively and in real time transmitted to the insurance service agent, preventing the latter from dynamically adjusting its health insurance product recommendation strategy or issuing underwriting reminders. Conversely, when the insurance claims agent is processing a critical illness claim, the customer service agent may still be pushing irrelevant health and wellness content to the user. This lack of collaboration leads to a broken service chain and a fragmented user experience. Therefore, in the fields of fintech and healthcare / elderly care, improving the reliability of intelligent customer service systems in handling multiple businesses in parallel has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a method, apparatus, device, and medium for generating a multi-agent collaboration mechanism based on PPO, so as to improve the reliability of intelligent customer service systems in parallel processing of multiple services.

[0005] Firstly, this application provides a method for generating a multi-agent cooperative mechanism based on PPO, the method comprising: The multimodal input data of each independent agent is converted into a current state vector through a feature encoding model; The single agent output action is generated by using the proximal policy optimization algorithm PPO, each independent agent and each current state vector, and the single agent advantage value of each single agent output action is calculated by the generalized advantage estimation algorithm GAE. An agent cooperation mechanism is generated based on a negative externality penalty mechanism, priority rules, the output actions of each agent, and the advantage value of each agent.

[0006] Secondly, this application also provides a multi-agent cooperation mechanism generation device based on PPO, the device comprising: The current state vector transformation module is used to convert the multimodal input data of each independent agent into a current state vector through a feature encoding model; The single agent dominance value calculation module is used to generate single agent output actions through the proximal policy optimization algorithm PPO, each independent agent and each current state vector, and calculate the single agent dominance value of each single agent output action according to the generalized dominance estimation algorithm GAE. The agent cooperation mechanism generation module is used to generate an agent cooperation mechanism based on the negative externality penalty mechanism, priority rules, the output actions of each agent, and the advantage value of each agent.

[0007] Thirdly, this application also provides a computer device, the computer device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the multi-agent cooperative mechanism generation method based on PPO as described above.

[0008] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the PPO-based multi-agent cooperation mechanism generation method described above.

[0009] This application discloses a method, apparatus, device, and medium for generating a multi-agent cooperative mechanism based on Policy Proximity Optimization (PPO). The method includes converting the multimodal input data of each independent agent into a current state vector using a feature encoding model; generating single-agent output actions using the PPO algorithm, each independent agent, and each current state vector; calculating the single-agent dominance value of each single-agent output action using the Generalized Advantage Estimation (GAE) algorithm; and generating an agent cooperative mechanism based on a negative externality penalty mechanism, priority rules, each single-agent output action, and each single-agent dominance value. Through this method, this application enhances the generalization capability for multimodal data by converting heterogeneous data into a unified state vector. By generating single-agent actions using the PPO algorithm and calculating single-agent advantage values ​​using the GAE algorithm, the training oscillations and convergence difficulties caused by policy non-stationarity in multi-agent reinforcement learning are suppressed. Through a coordination framework combining negative externality penalty mechanism and priority rules, the technical problem of the deviation between local optima and global objectives in multi-agent systems is solved. In business areas such as fintech and healthcare and elderly care, the reliability of intelligent customer service systems in parallel processing of multiple business is improved. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic flowchart of a PPO-based multi-agent cooperation mechanism generation method provided in an embodiment of this application; Figure 2 A schematic block diagram of a PPO-based multi-agent cooperation mechanism generation device provided for embodiments of this application; Figure 3 A schematic block diagram of the structure of a computer device provided for an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0014] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] This application provides a method, apparatus, device, and medium for generating a multi-agent cooperation mechanism based on PPO. The PPO-based multi-agent cooperation mechanism generation method can be applied to intelligent customer service systems. By converting heterogeneous data into a unified state vector, it enhances the generalization capability for multimodal data. By generating single-agent actions through the PPO algorithm and calculating single-agent advantage values ​​through the GAE algorithm, it suppresses training oscillations and convergence difficulties caused by policy non-stationarity in multi-agent reinforcement learning. Through a coordination framework combining a negative externality penalty mechanism and priority rules, it solves the technical problem of the deviation between local optima and global objectives in multi-agent systems. In business areas such as fintech and healthcare / elderly care, it improves the reliability of intelligent customer service systems in parallel processing of multiple services.

[0017] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0018] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a PPO-based multi-agent collaboration mechanism generation method provided in an embodiment of this application. This PPO-based multi-agent collaboration mechanism generation method can be applied to intelligent customer service systems to improve the reliability of parallel processing of multiple services in business areas such as fintech and healthcare / elderly care.

[0019] like Figure 1 As shown, the PPO-based multi-agent cooperation mechanism generation method specifically includes steps S10 to S30.

[0020] Step S10: Convert the multimodal input data of each independent agent into the current state vector through the feature encoding model; Specifically, the functional modules within the current business scenario are broken down into multiple independent intelligent agents, each responsible for a specific sub-task. For example, the independent intelligent agents can be: Product recommendation AI agent: responsible for recommending products / content (such as insurance packages, value-added services); Customer service AI agent: handles online inquiries and answers questions; Claims processing AI: Optimizes the claims process and provides real-time progress updates and support services; Complaint resolution agent: Quickly respond to and resolve customer complaints to improve customer satisfaction.

[0021] All agents are built on the following unified structure, with only the input and output layers adjusted. The policy network of each agent is based on the PPO (Proximal Policy Optimization) algorithm, but the state space, action space, and reward function are customized according to the task characteristics: Intelligent agent structure: Input layer → Feature encoder → Policy network → Action output.

[0022] Taking the product recommendation agent as an example, the structure of an independent agent will be explained.

[0023] 1. Input layer: ① Structured data: Customer profile (age, risk preference, life stage), policy status (remaining term, claims history); ② Unstructured data: real-time interactive content (such as conversation text), user app behavior data (such as clicks, browsing, and dwell time); ③ Cross-agent input: Dialogue intent labels output by customer service agents (e.g., "Focus on claims process"), and complaint type labels output by complaint resolution agents (e.g., "Dissatisfied with premium"). 2. Feature Encoder: ①Structured data encoding: For numerical fields (such as age, insurance amount), normalization is performed; for categorical fields (such as occupation, health level), one-hot encoding algorithm is used to map them into low-dimensional vectors; ② Unstructured data encoding: For text data, the BERT model is used to extract semantic features and form text feature vectors; 3. Policy Network: ①State space: Concatenate structured and unstructured features to form a unified state vector; ②Action space: the continuous and discrete actions output by other agents; ③ Reward function: R = 0.6 * click rate + 0.3 * LTV growth rate - 0.1 * complaint rate, where LTV represents the total lifetime value of the user (average daily premium income * user lifetime); 4. Output layer: ① Discrete actions: Recommend insurance plans and prioritize recommended content; ② Continuous action: Dynamically adjust recommendation weights.

[0024] Independent intelligent agents also include customer service intelligent agents. For customer service intelligent agents, the structured data of the input layer can be customer type labels (such as "high-value customer", "complaint sensitive type"), policy status (such as "expiring soon", "claims being processed"); unstructured data: real-time dialogue text, historical dialogue records; cross-agent input: progress status output by the claims processing intelligent agent (such as "approved").

[0025] The feature encoder of the customer service agent is the same as that of the product recommendation agent. The policy network is changed to a state space: concatenating structured and unstructured features to form a unified state vector; action space: continuous and discrete actions output by other agents; reward function: R=0.5*customer satisfaction +0.3*problem resolution rate -0.2*number of human interventions; The output layer of the customer service agent includes the following content: ① Discrete actions: Intent-based classification tags (such as "claims inquiry", "policy renewal"), fixed template replies (such as "your claim is under review"); ② Continuous actions: Dynamic script generation (e.g., "We suggest you provide supplementary medical certificates to expedite the review process").

[0026] For example, the input layer of the claims processing agent includes: ① Structured data: claims application fields (amount, date, accident type), policy terms (payout ratio, deductible); ② Unstructured data: medical documents recognized by OCR, supplementary materials from the customer; ③ Cross-agent input: dialogue text output by the customer service agent (such as "customer urging claims progress"). The feature encoder of the customer service agent is the same as that of other independent agents.

[0027] The policy network of the customer service agent includes: ①State space: Concatenate structured and unstructured features to form a unified state vector; ②Action space: the continuous and discrete actions output by other agents; ③ Reward function: R = 0.4 * automatic approval rate + 0.3 * customer satisfaction - 0.3 * manual review cost; The output of the customer service agent includes: ① Discrete actions: Triggered by automated processes (such as "manual review"); ② Continuous actions: Dynamically adjust the review priority (e.g., increase the priority coefficient for high-value customers).

[0028] The output of the complaint resolution agent includes: (1) Input layer: ① Structured data: historical complaint type labels (such as "claim delay", "premium dispute"), number of customer historical complaints; ② Unstructured data: historical complaint texts; ③ Cross-agent input: The progress status output by the claims processing agent (e.g., "Reason for review delay: incomplete materials").

[0029] The policy network of the complaint resolution agent includes: ①State space: Concatenate structured and unstructured features to form a unified state vector; ②Action space: the continuous and discrete actions output by other agents; ③ Reward function: R = 0.5 * complaint resolution rate + 0.3 * customer satisfaction - 0.2 * compensation cost; The output of the complaint resolution agent includes: ① Discrete actions: Complaint classification tags (such as "service process issues", "system failure"); ② Continuous actions: Matching complaint solutions (such as issuing coupons).

[0030] The multimodal input data of each independent agent is processed through a shared feature encoding model. Specifically, structured data (such as user profiles, policy status, and health indicators) from fintech and healthcare / elderly care businesses are normalized and one-hot encoded, while unstructured data (such as customer service dialogue text and medical document recognition) are extracted using pre-trained models such as BERT. All encoded features are then concatenated into a unified current state vector.

[0031] In the field of medical and health care and elderly care business, multi-source heterogeneous data is collected from various independent intelligent agents (such as health monitoring intelligent agents, medication management intelligent agents, rehabilitation guidance intelligent agents, and emergency rescue intelligent agents). This includes numerical vital sign data (blood pressure, blood sugar, etc.), categorical medical data (disease classification, nursing level, etc. are converted into sparse vectors through one-hot encoding), and unstructured text data (electronic medical records, elderly voice requests, etc., semantic features are extracted through the BERT model). The standardized numerical features, sparsely encoded categorical features, and deep semantic text features are multimodally fused and spliced ​​to form a unified current state vector.

[0032] In the fintech business, multi-source heterogeneous data is collected from various independent intelligent agents (such as risk assessment agents, investment advisor agents, anti-fraud agents, and customer service agents). This includes numerical transaction data (transaction amount, account balance, etc.), categorical customer data (occupation type, risk level, etc. are converted into sparse vectors through one-hot encoding), and unstructured text data (semantic features such as customer consultation records and contract texts). The standardized numerical features, sparsely encoded categorical features, and financial-specific text semantic features are then fused in a multimodal manner and spliced ​​together to form a unified current state vector.

[0033] Step S20: Generate single agent output actions using the Proximal Policy Optimization (PPO) algorithm, each independent agent, and each current state vector; and calculate the single agent advantage value of each single agent output action using the Generalized Advantage Estimation (GAE) algorithm. Specifically, each independent agent takes its current state vector as input and generates single-agent output actions via a policy network driven by the PPO algorithm—including discrete actions (such as recommending insurance plans or classifying complaint types) and continuous actions (such as dynamically adjusting review priorities). Subsequently, the agents interact with the environment to collect trajectory data, the value network estimates the state, and uses the generalized advantage estimation (GAE) algorithm to calculate the single-agent advantage value for each action based on the reward sequence and state value sequence, thereby providing guiding signals for policy updates that take into account both immediate gains and long-term value.

[0034] Step S30: Generate an agent cooperation mechanism based on the negative externality penalty mechanism, priority rules, the output actions of each agent, and the advantage value of each agent.

[0035] Specifically, when an action of a certain agent is detected to interfere with the tasks of other agents, its advantage value is reduced according to the negative externality penalty mechanism. At the same time, the execution of actions of multiple agents is coordinated according to the preset priority rules (such as complaint handling > claims handling > product recommendation). By filtering conflicting actions and allocating execution rights, a coordinated and consistent multi-agent decision output is finally formed.

[0036] This embodiment discloses a method, apparatus, device, and medium for generating a multi-agent cooperative mechanism based on Policy Proximity Optimization (PPO). The method includes converting the multimodal input data of each independent agent into a current state vector using a feature encoding model; generating single-agent output actions using the Proximity Policy Optimization (PPO) algorithm, each independent agent, and each current state vector; calculating the single-agent dominance value of each single-agent output action using the Generalized Advantage Estimation (GAE) algorithm; and generating an agent cooperative mechanism based on a negative externality penalty mechanism, priority rules, each single-agent output action, and each single-agent dominance value. Through this method, this application enhances the generalization capability for multimodal data by converting heterogeneous data into a unified state vector. By generating single-agent actions using the PPO algorithm and calculating single-agent advantage values ​​using the GAE algorithm, the training oscillations and convergence difficulties caused by policy non-stationarity in multi-agent reinforcement learning are suppressed. Through a coordination framework combining negative externality penalty mechanism and priority rules, the technical problem of the deviation between local optima and global objectives in multi-agent systems is solved. In business areas such as fintech and healthcare and elderly care, the reliability of intelligent customer service systems in parallel processing of multiple business is improved.

[0037] based on Figure 1 In the illustrated embodiment, step S10 includes: When the multimodal input data is a numerical field, the multimodal input data is normalized using the feature encoding model to generate the current state vector; When the multimodal input data is a categorical field, the multimodal input data is converted into a low-dimensional sparse vector through one-hot encoding to generate the current state vector; When the multimodal input data is unstructured, the semantic feature vector of the multimodal input data is extracted by a language model, and the semantic feature vector is determined as the current state vector.

[0038] Specifically, the steps for converting multimodal input data into a current state vector using a feature encoding model include: when the multimodal input data is a numerical field, the max-min normalization method is used to map the original numerical values ​​to the [0,1] interval to eliminate the influence of dimensions; when the multimodal input data is a categorical field, one-hot encoding is used to convert each categorical value into a sparse vector containing only a single 1 value, realizing the numerical representation of discrete features; when the multimodal input data is unstructured text data, a pre-trained BERT language model is used to extract deep semantic features of the text, outputting a fixed-dimensional semantic feature vector. Finally, the processed numerical feature vector, categorical sparse vector, and semantic feature vector are concatenated and fused to form a unified current state vector.

[0039] based on Figure 1 In the illustrated embodiment, step S20 includes: Based on the PPO and each of the independent agents, establish a PPO policy network; The action probability distribution is generated through the PPO policy network and the current state vector, and the single agent output action is generated based on the action probability distribution.

[0040] Specifically, a PPO policy network is established for each independent agent, which includes an Actor network (policy network) and a Critic network (value network). The Actor network consists of fully connected layers and outputs the action probability distribution, while the Critic network is used to evaluate the state value.

[0041] The current state vector generated by the feature encoding model is input into the PPO policy network. The Actor network calculates the original logical values ​​of the output layer through forward propagation. For discrete actions, a Softmax function is used to convert them into a probability distribution, while for continuous actions, the mean and variance parameters of a Gaussian distribution are output. Random sampling is performed based on the action probability distribution to obtain the specific single-agent output action—specific actions are selected by sampling from the category distribution for discrete actions, and specific values ​​are obtained by sampling from the parameterized Gaussian distribution for continuous actions, thus completing the complete closed loop from state perception to action decision-making.

[0042] In a specific embodiment, step S20 further includes: Trajectory data is generated by outputting actions from each of the independent intelligent agents and the single intelligent agent; A state value estimation sequence is generated based on the preset value network and the trajectory data; The single agent's advantage value is calculated using the GAE based on the state value estimation sequence.

[0043] Specifically, each independent agent executes its single-agent output action in the environment, recording the resulting state transition sequence, reward signal, and termination flag to form complete trajectory data. Each state in the trajectory is input into a pre-defined value network, which, based on a deep neural network structure, outputs a state value estimate for each state through forward propagation, forming a state value estimate sequence. A generalized advantage estimation algorithm is employed, using the state value estimate sequence and the actual obtained reward sequence as input, and calculating the advantage value estimate for each time step through an exponentially weighted summation of temporal difference errors.

[0044] In one embodiment, each independent agent executes the single-agent output action to interact with the environment, collecting trajectory data containing "current state-action-reward-next state". The state at each time step in the trajectory data is forward-propagated through a pre-defined value network to generate a corresponding state value estimation sequence. Based on the state value estimation sequence, a generalized advantage estimation algorithm is used to quantify the advantage of each action relative to the state baseline by weighting it with temporal difference error and discount factor, obtaining the single-agent advantage value used for policy network updates.

[0045] based on Figure 1 In the illustrated embodiment, step S30 includes: Based on the negative externality penalty mechanism and the output actions of each individual agent, negative externality interference monitoring results are generated among the independent agents. Obtain the current priority relationship of each independent intelligent agent, and generate priority monitoring results based on the current priority relationship and the priority rules; The agent cooperation mechanism is generated based on the priority monitoring results, the negative externality interference monitoring results, and the dominance values ​​of each agent.

[0046] Specifically, based on the negative externality penalty mechanism, the mutual influence between the output actions of each independent agent is monitored in real time. When it is identified that the action of a certain agent interferes with the task execution of other agents, a negative externality interference monitoring result containing the type and degree of interference is generated.

[0047] Simultaneously, the current priority relationships among the agents are obtained, and dynamic priority monitoring results are established, prioritizing complaint handling over claims processing, and claims processing over product recommendation. Combining these two types of monitoring results with the advantage values ​​of each individual agent, an agent collaboration mechanism is constructed. Agents generating negative externalities are penalized with advantage value attenuation based on the degree of their impact. Furthermore, conflicting actions are arbitrated according to priority relationships, prioritizing the actions of high-priority agents and suppressing or delaying conflicting actions of low-priority agents, thereby forming a collaborative and efficient multi-agent decision-making mechanism.

[0048] In a specific embodiment, based on the negative externality penalty mechanism and the output actions of each individual agent, negative externality interference monitoring results between each independent agent are generated, including: Calculate whether the single agent output action of each of the independent agents causes negative externality interference to the other independent agents; When the output action of the single agent of the independent agent causes negative externality interference to other independent agents, the advantage value of the single agent is penalized and attenuated based on the negative externality penalty mechanism to generate a penalized advantage value. The negative externality interference monitoring results are generated based on each of the aforementioned penalty advantage values.

[0049] Specifically, by analyzing the output action combinations of each independent agent through preset interference judgment rules, it identifies whether there are action conflicts or resource competition. When it is detected that an agent's action causes a decrease in the task execution efficiency of other agents or obstruction of their goals, it is determined that a negative externality interference has occurred. For the agent that caused the interference, its original single agent advantage value is multiplied by a penalty coefficient between 0 and 1 according to the severity of the interference to generate a decayed penalty advantage value. The complete information, including the interference type, the responsible agent's identifier, and the corresponding penalty advantage value, is integrated and output to generate the negative externality interference monitoring result.

[0050] In one embodiment, for the auto insurance sector, a predefined interference judgment rule base is used to detect in real time whether there are interference situations such as the auto insurance product recommendation agent pushing marketing content during the auto insurance complaint resolution agent's processing, the delayed processing action of the auto insurance claims processing agent triggering repeated inquiries from the auto insurance customer service agent, or the wording guidance of the auto insurance customer service agent contradicting the recommendation strategy of the auto insurance product recommendation agent. Once it is determined that an agent's action generates negative externality interference, a negative externality penalty mechanism is used to penalize the agent's single agent advantage value. The advantage value is multiplied by a decay coefficient of 0.5 to 0.9 to generate a penalty advantage value, and the decay strength is positively correlated with the severity of the interference.

[0051] By summarizing the penalty advantage values ​​of each agent, the identifier of the interfered object, and the type of interference, a structured negative externality interference monitoring result is constructed, which includes the interference source, the affected agent, the penalty coefficient, and the corrected advantage value. In a specific embodiment, when the single agent's output action causes negative externality interference to other independent agents, the advantage value of the single agent is penalized and attenuated based on the negative externality penalty mechanism. Before generating the penalized advantage value, the following steps are included: Establish the relationship matrix between the independent intelligent agents; When the single agent output action is executed by the independent agent, the key task indicators of the other independent agents are collected. If the key performance indicator of the task is less than a preset threshold, it is determined that the single agent output action of the independent agent causes the negative externality interference to other independent agents.

[0052] Specifically, a matrix is ​​constructed to describe the interaction relationships between independent agents, with agents as the row and column dimensions and element values ​​representing the task association strength between agents. When an independent agent executes its single-agent output action, key task indicators of other associated agents are collected in real time, including quantitative parameters such as task completion rate, response time, and customer satisfaction. Through comparative analysis, when a significant decrease in the key task indicators of an associated agent is detected after the action is executed and falls below a preset threshold range, it is determined that the currently executed single-agent output action has generated negative externality interference to other agents.

[0053] Please see Figure 2 , Figure 2 This application provides a schematic block diagram of a PPO-based multi-agent cooperation mechanism generation device, which is used to execute the aforementioned PPO-based multi-agent cooperation mechanism generation method. The PPO-based multi-agent cooperation mechanism generation device can be configured on a server.

[0054] like Figure 2 As shown, the PPO-based multi-agent cooperation mechanism generation device 400 includes: The current state vector conversion module 410 is used to convert the multimodal input data of each independent agent into a current state vector through a feature encoding model; The single agent dominance value calculation module 420 is used to generate single agent output actions through the proximal policy optimization algorithm PPO, each of the independent agents and each of the current state vectors, and calculate the single agent dominance value of each single agent output action according to the generalized dominance estimation algorithm GAE. The agent cooperation mechanism generation module 430 is used to generate an agent cooperation mechanism based on the negative externality penalty mechanism, priority rules, the output actions of each single agent, and the advantage value of each single agent.

[0055] Furthermore, the current state vector transformation module 410 includes: The numerical field conversion unit is used to normalize the multimodal input data through the feature encoding model to generate the current state vector when the multimodal input data is a numerical field. The categorical field conversion unit is used to convert the multimodal input data into a low-dimensional sparse vector through one-hot encoding when the multimodal input data is a categorical field, thereby generating the current state vector; The unstructured data conversion unit is used to extract the semantic feature vector of the multimodal input data through a language model when the multimodal input data is unstructured, and to determine the semantic feature vector as the current state vector.

[0056] Furthermore, the single-agent dominance value calculation module 420 includes: The PPO policy network establishment unit is used to establish a PPO policy network based on the PPO and each of the independent intelligent agents. A single agent output action generation unit is used to generate an action probability distribution through the PPO policy network and the current state vector, and generate the single agent output action according to the action probability distribution.

[0057] Furthermore, the single-agent dominance value calculation module 420 includes: A trajectory data generation unit is used to generate trajectory data by outputting actions from each of the independent intelligent agents and the single intelligent agent; A state value estimation sequence generation unit is used to generate a state value estimation sequence based on a preset value network and the trajectory data; A single agent dominance value calculation unit is used to calculate the single agent dominance value based on the state value estimation sequence through the GAE.

[0058] Furthermore, the intelligent agent cooperation mechanism generation module 430 includes: The negative externality interference monitoring result generation unit is used to generate negative externality interference monitoring results between the independent agents based on the negative externality penalty mechanism and the output actions of each single agent. The priority monitoring result generation unit is used to obtain the current priority relationship of each independent intelligent agent and generate priority monitoring results according to the current priority relationship and the priority rules. The agent cooperation mechanism generation unit is used to generate the agent cooperation mechanism based on the priority monitoring results, the negative externality interference monitoring results, and the dominance values ​​of each individual agent.

[0059] Furthermore, the negative externality interference monitoring result generation unit includes: The negative externality interference judgment subunit is used to calculate whether the single agent output action of each independent agent causes negative externality interference to other independent agents; The penalty advantage value generation subunit is used to generate a penalty advantage value by penalizing the advantage value of the single agent based on the negative externality penalty mechanism when the single agent output action of the independent agent causes the negative externality interference to other independent agents. The negative externality interference monitoring result generation subunit is used to generate the negative externality interference monitoring result based on each of the aforementioned penalty advantage values.

[0060] Furthermore, the negative externality interference monitoring result generation unit includes: The relation matrix establishment subunit is used to establish the relation matrix between the independent intelligent agents. The task key indicator collection subunit is used to collect the task key indicators of other independent intelligent agents when the single intelligent agent outputs the action through the independent intelligent agent. The negative externality interference determination subunit is used to determine, when the key indicators of the task are less than a preset threshold, that the single agent output action of the independent agent causes the negative externality interference to other independent agents.

[0061] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0062] The aforementioned apparatus can be implemented as a computer program, which can be used in, for example... Figure 3 It runs on the computer device shown.

[0063] Please see Figure 3 , Figure 3 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0064] See Figure 3 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0065] The non-volatile storage medium can store the operating system and computer program. The computer program includes program instructions that, when executed, cause the processor to perform any PPO-based multi-agent cooperative mechanism generation method.

[0066] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0067] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to execute any multi-agent cooperative mechanism generation method based on PPO.

[0068] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0069] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0070] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: The multimodal input data of each independent agent is converted into a current state vector through a feature encoding model; The single agent output action is generated by using the proximal policy optimization algorithm PPO, each independent agent and each current state vector, and the single agent advantage value of each single agent output action is calculated by the generalized advantage estimation algorithm GAE. An agent cooperation mechanism is generated based on a negative externality penalty mechanism, priority rules, the output actions of each agent, and the advantage value of each agent.

[0071] In one embodiment, a feature encoding model is used to convert the multimodal input data of each independent agent into a current state vector, which is used to achieve: When the multimodal input data is a numerical field, the multimodal input data is normalized using the feature encoding model to generate the current state vector; When the multimodal input data is a categorical field, the multimodal input data is converted into a low-dimensional sparse vector through one-hot encoding to generate the current state vector; When the multimodal input data is unstructured, the semantic feature vector of the multimodal input data is extracted by a language model, and the semantic feature vector is determined as the current state vector.

[0072] In one embodiment, a single agent output action is generated using the Proximal Policy Optimization (PPO) algorithm, each independent agent, and each current state vector, to achieve: Based on the PPO and each of the independent agents, establish a PPO policy network; The action probability distribution is generated through the PPO policy network and the current state vector, and the single agent output action is generated based on the action probability distribution.

[0073] In one embodiment, the single-agent advantage value of each single agent's output action is calculated according to the Generalized Advantage Estimation Algorithm (GAE) to achieve: Trajectory data is generated by outputting actions from each of the independent intelligent agents and the single intelligent agent; A state value estimation sequence is generated based on the preset value network and the trajectory data; The single agent's advantage value is calculated using the GAE based on the state value estimation sequence.

[0074] In one embodiment, an agent cooperation mechanism is generated based on a negative externality penalty mechanism, priority rules, the output actions of each agent, and the dominance value of each agent, to achieve the following: Based on the negative externality penalty mechanism and the output actions of each individual agent, negative externality interference monitoring results are generated among the independent agents. Obtain the current priority relationship of each independent intelligent agent, and generate priority monitoring results based on the current priority relationship and the priority rules; The agent cooperation mechanism is generated based on the priority monitoring results, the negative externality interference monitoring results, and the dominance values ​​of each agent.

[0075] In one embodiment, based on the negative externality penalty mechanism and the output actions of each individual agent, negative externality interference monitoring results between the independent agents are generated to achieve: Calculate whether the single agent output action of each of the independent agents causes negative externality interference to the other independent agents; When the output action of the single agent of the independent agent causes negative externality interference to other independent agents, the advantage value of the single agent is penalized and attenuated based on the negative externality penalty mechanism to generate a penalized advantage value. The negative externality interference monitoring results are generated based on each of the aforementioned penalty advantage values.

[0076] In one embodiment, when the single agent's output action causes negative externality interference to other independent agents, the single agent's advantage value is penalized and attenuated based on the negative externality penalty mechanism. Before generating the penalized advantage value, the following is implemented: Establish the relationship matrix between the independent intelligent agents; When the single agent output action is executed by the independent agent, the key task indicators of the other independent agents are collected. If the key performance indicator of the task is less than a preset threshold, it is determined that the single agent output action of the independent agent causes the negative externality interference to other independent agents.

[0077] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the PPO-based multi-agent cooperation mechanism generation methods provided in the embodiments of this application.

[0078] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0079] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating a multi-agent cooperative mechanism based on PPO, characterized in that, include: The multimodal input data of each independent agent is converted into a current state vector through a feature encoding model; The single agent output action is generated by using the proximal policy optimization algorithm PPO, each independent agent and each current state vector, and the single agent advantage value of each single agent output action is calculated by the generalized advantage estimation algorithm GAE. An agent cooperation mechanism is generated based on a negative externality penalty mechanism, priority rules, the output actions of each agent, and the advantage value of each agent.

2. The method for generating a multi-agent cooperation mechanism based on PPO according to claim 1, characterized in that, The multimodal input data includes structured and unstructured data. The structured data includes numerical fields and categorical fields. The process of converting the multimodal input data of each independent agent into a current state vector through a feature encoding model includes: When the multimodal input data is a numerical field, the multimodal input data is normalized using the feature encoding model to generate the current state vector; When the multimodal input data is a categorical field, the multimodal input data is converted into a low-dimensional sparse vector through one-hot encoding to generate the current state vector; When the multimodal input data is unstructured, the semantic feature vector of the multimodal input data is extracted by a language model, and the semantic feature vector is determined as the current state vector.

3. The method for generating a multi-agent cooperation mechanism based on PPO according to claim 1, characterized in that, The process of generating a single agent output action using the Proximal Policy Optimization (PPO) algorithm, each independent agent, and each current state vector includes: Based on the PPO and each of the independent agents, establish a PPO policy network; The action probability distribution is generated through the PPO policy network and the current state vector, and the single agent output action is generated based on the action probability distribution.

4. The method for generating a multi-agent cooperation mechanism based on PPO according to claim 3, characterized in that, The calculation of the single agent dominance value of each single agent's output action based on the Generalized Dominance Estimation Algorithm (GAE) includes: Trajectory data is generated by outputting actions from each of the independent intelligent agents and the single intelligent agent; A state value estimation sequence is generated based on the preset value network and the trajectory data; The single agent's advantage value is calculated using the GAE based on the state value estimation sequence.

5. The method for generating a multi-agent cooperation mechanism based on PPO according to claim 1, characterized in that, The agent cooperation mechanism, generated based on the negative externality penalty mechanism, priority rules, the output actions of each agent, and the dominance value of each agent, includes: Based on the negative externality penalty mechanism and the output actions of each individual agent, negative externality interference monitoring results are generated among the independent agents. Obtain the current priority relationship of each independent intelligent agent, and generate priority monitoring results based on the current priority relationship and the priority rules; The agent cooperation mechanism is generated based on the priority monitoring results, the negative externality interference monitoring results, and the dominance values ​​of each agent.

6. The method for generating a multi-agent cooperation mechanism based on PPO according to claim 5, characterized in that, The step of generating negative externality interference monitoring results among the independent agents based on the negative externality penalty mechanism and the output actions of each individual agent includes: Calculate whether the single agent output action of each of the independent agents causes negative externality interference to the other independent agents; When the output action of the single agent of the independent agent causes negative externality interference to other independent agents, the advantage value of the single agent is penalized and attenuated based on the negative externality penalty mechanism to generate a penalized advantage value. The negative externality interference monitoring results are generated based on each of the aforementioned penalty advantage values.

7. The method for generating a multi-agent cooperative mechanism based on PPO according to claim 6, characterized in that, In the case where the single agent's output action causes negative externality interference to other independent agents, the advantage value of the single agent is penalized and attenuated based on the negative externality penalty mechanism. Before generating the penalized advantage value, the process includes: Establish the relationship matrix between the independent intelligent agents; When the single agent output action is executed by the independent agent, the key task indicators of the other independent agents are collected. If the key performance indicator of the task is less than a preset threshold, it is determined that the single agent output action of the independent agent causes the negative externality interference to other independent agents.

8. A device for generating a multi-agent cooperative mechanism based on PPO, characterized in that, include: The current state vector transformation module is used to convert the multimodal input data of each independent agent into a current state vector through a feature encoding model; The single agent dominance value calculation module is used to generate single agent output actions through the proximal policy optimization algorithm PPO, each independent agent and each current state vector, and calculate the single agent dominance value of each single agent output action according to the generalized dominance estimation algorithm GAE. The agent cooperation mechanism generation module is used to generate an agent cooperation mechanism based on the negative externality penalty mechanism, priority rules, the output actions of each agent, and the advantage value of each agent.

9. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the PPO-based multi-agent cooperative mechanism generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the PPO-based multi-agent cooperative mechanism generation method as described in any one of claims 1 to 7.

Images