Social balance inspired multi-agent reinforcement learning game simulation method

Abstract

The application discloses a social balance inspired multi-agent reinforcement learning game simulation method, and belongs to the technical field of artificial intelligence multi-agent evolutionary game. The method constructs two types of heterogeneous agents, namely, Hyde type and friend focus type, stores the trust evaluation of each subject independently by using a private reputation matrix, distinguishes the acceptance rules of the indirect evaluation of the two types of agents on the positive and negative relationship chains based on the social balance theory, compresses the perceived reputation into a binary reputation label, decides the scene type and the binary reputation label to form a lightweight unified state, integrates three types of decisions, namely, prisoner's dilemma game, connection establishment and connection removal, into the same reinforcement learning framework, synchronously updates the action value according to the discount cumulative return, and forms a co-evolution closed loop of three layers of coupling of reputation, strategy and network topology. The application solves the problems of global reputation distortion, static network topology, fragmented decision logic and state space expansion in the prior art, can be applied to the simulation of a trusted cooperation network of a supply chain, a computing power sharing platform and a service outsourcing platform, and supports heterogeneous rule comparison and breakpoint continuous training.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary fields of artificial intelligence multi-agent systems, evolutionary game theory, and reinforcement learning. Specifically, it relates to a social equilibrium-inspired multi-agent reinforcement learning game simulation method, which can be applied to networked trusted cooperative game simulation, platform collaboration mechanism pre-simulation, and dynamic trust network evolution deduction in scenarios such as service outsourcing networks, supply chain collaboration, computing resource sharing, distributed service invocation, and autonomous agent relationship governance. Background Technology

[0002] In scenarios such as service outsourcing, supply chain collaboration, computing resource sharing, and distributed autonomous scheduling, numerous intelligent agents with autonomous decision-making capabilities need to assess the trustworthiness of collaborators in real time, autonomously select partners, and dynamically establish or terminate cooperative connections during long-term, continuous interactions. This type of scenario can be abstracted as a networked collaboration dilemma characterized by incomplete local information, dynamic changes in network topology, and conflicting interests among multiple parties. To assess the impact of different governance mechanisms, reward / punishment rules, and admission strategies on the level of group cooperation before the system goes live, the industry commonly uses multi-agent game simulations to computer-generated models of the aforementioned collaborative processes, using data such as cooperation rates, network connectivity, and the number of isolated nodes obtained from these simulations as quantitative bases for mechanism design.

[0003] Reinforcement learning, exemplified by Q-learning, allows agents to accumulate experience and update their strategies through interactive trial and error and based on reward feedback. This differs from the fixed imitation update rules used in traditional evolutionary game theory, and has therefore been introduced into cooperative game simulations to improve the realism of agent decisions. However, the inventors have discovered in practice that existing networked cooperative game simulation models have several inherent flaws, making it difficult to accurately reproduce the operational rules of real-world business and resource collaboration networks. These flaws are as follows.

[0004] First, the network topology remains fixed. In existing schemes, agents can only choose to cooperate or betray from a pre-defined static set of neighbors. They do not support autonomously disconnecting existing connections or actively establishing new cooperative relationships. Therefore, they cannot simulate the dynamic interaction process of enterprise nodes autonomously selecting and eliminating partners in supply chains, outsourcing platforms, and computing power sharing networks. This results in a systematic deviation between the simulated network structure and the real collaborative network in terms of connectivity and evolution trajectory.

[0005] Secondly, a globally unified public reputation is adopted. In existing schemes, all agents share the same global reputation score, which fails to characterize the private cognitive characteristic that the reputation of the same partner varies in the eyes of different agents. This smooths out the differences in individual subjective judgments and makes it impossible to reproduce real network phenomena such as local trust, group differentiation, and mis-propagation of information. As a result, the simulation results are difficult to support the evaluation of differentiated trust governance strategies.

[0006] Third, the decision-making systems are fragmented. In existing solutions, cooperative game behavior is trained using reinforcement learning, while the adjustment of network connections relies on fixed heuristic rules. These two independent decision-making logics are not interconnected and cannot be optimized collaboratively. As a result, a complete evolutionary loop cannot be formed between reputation judgment, behavior selection, relationship adjustment, and long-term benefit feedback. The agent cannot autonomously decide whether to maintain or restructure its cooperative relationship based on long-term benefits.

[0007] Fourth, the state space expands rapidly. Existing solutions directly use the complete neighbor structure, global network topology, and long-term interaction history as input to the reinforcement learning model, causing the dimension of the action value table to expand rapidly with the network size. This leads to technical problems such as slow learning convergence, excessively long training time, and unstable numerical values ​​in long-term simulations, which limits the network size that the simulation can handle and the number of control experiments that can be performed.

[0008] Furthermore, the cognitive rules of all agents in existing schemes are highly homogeneous, making it impossible to quantify and compare the differentiated impact of different trust inference logics on the evolution of group cooperation in the same simulation environment. This makes it difficult to provide a comparable quantitative basis for platform governance and mechanism design. In summary, how to construct a networked cooperative game simulation method with reputation privatization, unified decision-making, and reconfigurable relationships without introducing complete global network information as state input, thereby suppressing the explosion of state space dimensions, has become an urgent problem to be solved in this technical field. Summary of the Invention

[0009] Purpose of the invention To address the technical bottlenecks in existing networked collaborative game simulations, which employ globally unified public reputation, fragmented cooperative strategy learning and network topology adjustment, and direct use of complete network information as reinforcement learning state input, leading to state space dimensionality explosion, slow learning convergence, and poor simulation stability, this invention provides a social equilibrium-inspired multi-agent reinforcement learning game simulation method. This method constructs an independent private reputation evaluation for each agent and compresses it into a binary reputation label as a lightweight state. Using the unified state composed of the current decision scenario type and the binary reputation label, it incorporates three types of decisions—Prisoner's Dilemma, connection establishment, and connection dissolution—into the same reinforcement learning framework. Without introducing the complete global network topology and all neighbor information as state input, it achieves low-dimensional stable co-evolution of reputation, strategy, and network topology through the principle of state representation dimensionality reduction and multi-scenario decision unification.

[0010] Technical solution The technical solution of this invention is as follows: A social equilibrium-inspired multi-agent reinforcement learning game simulation method, executed by a simulation computer, includes the following steps: Step 1: Initialize the heterogeneous agent interaction network, private reputation matrix, connection matrix, and reinforcement learning parameters. The heterogeneous agent interaction network contains N agents, divided into two categories: Heidegger agents and friend-focused agents. The connection matrix is ​​used to represent the cooperative connection state between agents at any given time, and the private reputation matrix is ​​used to store each agent's independent private reputation evaluation of the other agents. The reinforcement learning parameters include the total number of agents, the number of Heidegger agents, the total number of simulation rounds, the number of Prisoner's Dilemma interactions per round, the network structure update ratio, the reputation update step size, the learning rate, the discount factor, and the Boltzmann inverse temperature. An independent action value table, upper and lower limits of reputation values, and initial cooperative connection relationships are also initialized for each agent. Step 2: Perform simulation in rounds and iteratively. In each round, generate a mixture of Prisoner's Dilemma interaction events and network structure update events. The number of network structure update events is the product of the network structure update ratio and the number of Prisoner's Dilemma interactions in a single round, rounded down. The execution order of the two types of events is randomly shuffled. Step 3: For the target agent to be interacted with, the initiating agent calculates the perceived reputation of the target agent based on the private reputation evaluation of the target agent in the private reputation matrix, the common neighbor information of the two, and the indirect information acceptance rules corresponding to the type of the initiating agent. The perceived reputation is mapped to a binary reputation label of good reputation or bad reputation, and the current decision scenario type and the binary reputation label constitute a unified state for entering reinforcement learning. Step 4: For agents with cooperative connections, perform the Prisoner's Dilemma interaction event in pairs. Both parties obtain corresponding benefits based on their cooperative or betrayal actions selected based on their action value tables. After the interaction is completed, update the private reputation matrix. Cooperative actions improve the other party's private reputation evaluation from the perspective of the agent. The update range is controlled by the reputation update step size, and the result is constrained within the upper and lower limits of the reputation value range. Step 5: Execute the network structure update event and randomly select a pair of agents to make a dynamic relationship reconstruction decision: If a cooperative connection already exists, enter the connection termination scenario, and the connection is deleted if either party chooses to terminate; if no cooperative connection exists, enter the connection establishment scenario, and the connection is added only if both parties choose to establish; both connection termination and connection establishment are learnable actions for reinforcement learning. Step 6: After all events have been executed in each round, the action value tables for the three decision scenarios—Prisoner's Dilemma interaction scenario, connection dissolution scenario, and connection establishment scenario—are updated synchronously using the cumulative discount reward as the update target and a unified action value update rule. Step 7: Record the evolution indicators of cooperation ratio, network topology, private reputation distribution and action value for two types of agents: Heide-type agents and friend-focused agents. Save snapshots of the connection matrix and private reputation matrix at set intervals to support breakpoint resume training. Iterate in a loop until the total number of simulation rounds is reached.

[0011] This invention also provides a social equilibrium-inspired multi-agent reinforcement learning game simulation system, comprising: a heterogeneous agent and private reputation network initialization module, used to construct a heterogeneous agent interaction network and initialize the connection matrix, private reputation matrix, action value table, and reinforcement learning parameters; a social equilibrium-inspired private reputation inference module, used to calculate perceived reputation, accept indirect evaluations according to heterogeneous types, and generate a unified state composed of scenario type and binary reputation labels; a prisoner's dilemma interaction and private reputation feedback module, used to execute prisoner's dilemma interactions, allocate payoffs, and update the private reputation matrix; a dynamic relationship reconstruction and unified reinforcement learning module, used to incorporate three types of decision scenarios into the same reinforcement learning framework and synchronously update the action value table according to discounted cumulative rewards; and a snapshot and indicator recording module, used to classify and record evolution indicators and save matrix snapshots to support breakpoint continuation training.

[0012] Beneficial effects The beneficial effects of this invention are as follows: First, by constructing an independent private reputation evaluation for each agent and compressing continuous perceived reputation into binary reputation labels of good and bad reputation, and then using the current decision scenario type and the binary reputation label to form a unified state for entering reinforcement learning, effective compression of the reinforcement learning state space is achieved. The mechanism is that when an agent makes decisions about cooperation, establishing connections, or disconnecting, what truly determines the outcome is not the complete continuous reputation value of the interacting object or the precise structure of the global network, but rather a coarse-grained judgment of whether the object is trustworthy. Therefore, by continuously evolving the continuous reputation within a private reputation matrix and entering the state only with binary labels, the state representation required for decision-making is decoupled from the evolutionary information carried by the reputation, and the state size no longer expands with the network size and historical length. Compared to existing schemes that directly use the complete neighbor structure, global topology, and long-term interaction history as input, this scheme significantly reduces the dimensionality of the action value table, thereby significantly improving the convergence speed of Q-learning and enhancing the numerical stability of long-term simulations.

[0013] Second, by setting differentiated indirect information acceptance rules for Heidegger-type agents and friend-focused agents—Heidegger-type agents accept indirect evaluations transmitted by common neighbors through both positive and negative relationship chains, while friend-focused agents only accept indirect evaluations transmitted by their own positive relationship neighbors—this approach restores the differentiated private trust cognition mechanism. The mechanism lies in the fact that the two types of heuristic inference logics based on social equilibrium theory have different levels of trust in positive and negative relationship chains. This allows the same target object to be mapped to different binary reputation labels in the eyes of different agents, leading to phenomena such as local trust, information bias, and cooperative network differentiation at the group level. Compared to existing technologies that use a globally unified public reputation to smooth out subjective differences, this approach can quantitatively compare the impact of different trust inference logics on cooperative stability and network differentiation in the same simulation environment.

[0014] Third, by unifying the three types of decisions—Prisoner's Dilemma, connection discontinuation, and connection establishment—into a single reinforcement learning framework with scenario type and binary reputation labels as states, and synchronously updating the action values ​​of the three scenarios using the same action value update rule based on discounted cumulative rewards, the problem of the separation between cooperative strategy learning and network topology adjustment in existing technologies is eliminated. The mechanism is that by incorporating the decision scenario type into the state representation, a single action value table can respectively carry the semantics of the three types of decisions. This transforms the establishment and discontinuation of network connections from externally fixed heuristic rules into learnable reinforcement learning actions driven by long-term gains, thereby enabling the agent to autonomously reconstruct its network topology based on cooperative gains.

[0015] Fourth, the aforementioned technical features are coupled with each other, generating a synergistic effect that exceeds the sum of the individual effects of each feature. Specifically, the feedback update of private reputation drives the determination of binary reputation labels. The binary reputation labels, along with the scenario type, drive the agent's cooperation, connection establishment, and disconnection actions in three scenarios. Connection establishment and disconnection actions change the network topology, thereby changing the set of subsequent interaction objects. The round-level discount cumulative reward, in turn, updates the action value table, thus forming a co-evolutionary closed loop of reputation, policy, and network topology coupled at three layers. This closed loop simultaneously depicts the co-evolutionary process of micro-agent cognition, local interaction behavior, and macro-network structure in a low-dimensional state representation. This allows the scheme to simulate the entire process of enterprises or nodes autonomously selecting and eliminating partners. Furthermore, due to the use of matrix snapshots and breakpoint retraining mechanisms, it can support long-term, multi-random seed control experiments, making the simulation process interruptible, resumable, and the experimental results verifiable and reproducible. Detailed Implementation

[0016] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Equivalent substitutions and improvements made without departing from the concept of this invention should fall within the protection scope of this invention. The overall simulation architecture of this invention consists of four mutually coupled functional modules, which operate according to a unified closed-loop iterative process. The composition and working method of each functional module will be explained first, and then illustrated with examples from three application scenarios.

[0017] I. Initialization Module for Heterogeneous Intelligent Agents and Private Reputation Networks This module constructs a dynamic interactive network with N agents, categorizing them into two types: Heidegger agents and friend-focused agents. The difference between the two lies in their rules for accepting indirect information, a distinction inspired by social equilibrium theory. The module defines a connection matrix to represent the cooperative connection state between agents at any given time; and a private reputation matrix, where each component represents an agent's independent private reputation evaluation of another agent at the current moment. This allows the same evaluated object to have different reputation values ​​from different evaluators, unlike a globally uniform single reputation score. The module's initialization parameters include the total number of agents, the number of Heidegger agents, the total number of simulation rounds, the number of Prisoner's Dilemma interactions per round, the network structure update ratio, the reputation update step size, the learning rate, the discount factor, and the Boltzmann inverse temperature. It also independently initializes each agent's action value table, reputation value upper and lower limits, initial cooperative connection relationships, and agent type, providing a unified initial state for subsequent private reputation inference, Prisoner's Dilemma interactions, and dynamic relationship reconstruction.

[0018] The private reputation matrix defined in this module is the foundation that distinguishes this invention from existing globally unified reputation schemes. In this matrix, for the same evaluated object, the private reputation evaluations held by different evaluators are independent and not shared. Therefore, the same collaborator can have different levels of credibility in the eyes of different agents, which aligns with the locality and subjectivity of reputation in real collaborative networks. During the initialization phase, all private reputation evaluations can be uniformly set to initial values ​​representing neutral trust, ensuring that each agent is free from prior bias at the start of the simulation. As the prisoner's dilemma interaction progresses, the private reputation evaluations dynamically evolve by agent based on the actual cooperation or betrayal actions of the interacting objects, gradually forming local differences and differentiations in trust at the group level. The action value table independently maintained by this module for each agent carries its value estimation under three decision-making scenarios. The upper and lower limits of reputation values ​​constrain the evolution range of private reputation, while the initial cooperative connection relationships and their types jointly determine the network topology and heterogeneous agent composition at the start of the simulation.

[0019] II. Social Equilibrium Theory Heuristic Private Reputation Inference Module When an agent needs to determine the trustworthiness of a target agent, the initiating agent calculates its perceived reputation of the target agent. This perceived reputation consists of two parts: first, direct evaluations formed during the direct interaction history between the initiating and target agents; and second, indirect evaluations transmitted by their common neighbors. The sum of these two parts yields the perceived reputation. Two types of agents employ differentiated rules for accepting indirect information: Heidegger-type agents simultaneously accept reputation information transmitted by common neighbors through both positive and negative relationship chains, meaning they not only trust those approved by their friends but also lower their evaluation of the target agent due to negative information transmitted through negative relationship chains; friend-focused agents only accept indirect reputation information transmitted by their own positive relationship neighbors, ignoring information transmitted through negative relationship chains. These two types of acceptance rules are inspired by the concepts of relationship triangle balance and positive / negative relationship transmission in social equilibrium theory, corresponding to two typical trust inference logics: greater sensitivity to negative information and trust only positive connections. The calculated continuous perceived reputation is then mapped to a binary reputation label of good or bad reputation as a lightweight state input, thus avoiding the explosion of state dimensions caused by taking the complete global network structure as input.

[0020] Specifically, the Heidegger-type agent corresponds to a trust inference logic that is more sensitive to negative information: when its common neighbors transmit negative evaluations of a target object through negative relationship chains, the Heidegger-type agent will correspondingly lower its perceived reputation of the target object, thus making it more likely to identify the target object as having a bad reputation and thereby disconnect or refuse to establish a connection. The friend-focused agent, on the other hand, corresponds to a trust inference logic that only trusts positive connections. It only adopts positive information transmitted by its own positive relationship neighbors and ignores negative information transmitted through negative relationship chains, thus being relatively insensitive to the spread of negative information. Therefore, the same target object can be assigned different private reputation evaluations in the private reputation matrices of the Heidegger-type and friend-focused agents, and further mapped to different binary reputation labels. This allows the present invention to characterize two typical trust inference logics and their differentiated impact on group cooperative evolution in parallel within the same simulation environment, without needing to maintain separate global reputation views for different agents.

[0021] III. Prisoner's Dilemma Interaction and Private Reputation Feedback Module Agents with established cooperative relationships engage in a standard Prisoner's Dilemma game in pairs: When both parties cooperate, they each gain a mutually beneficial benefit; when one betrays and the other cooperates, the betrayer gains a tempting higher benefit while the cooperating party suffers a loss; when both parties betray, they each receive the minimum benefit. After each game, the module updates the private reputation matrix based on the actual actions of both parties: cooperative behavior increases the other party's private reputation rating from the agent's perspective, while betrayal behavior decreases the other party's private reputation rating from the agent's perspective. The update magnitude is controlled by the reputation update step size, and all reputation values ​​are forcibly constrained within preset upper and lower limits to prevent unbounded growth or decay. The private reputation iterates dynamically with each interaction and serves as a direct evaluation item for subsequent perceived reputation calculations, thus continuously influencing all subsequent decision-making behaviors of the agents, forming a feedback loop from interaction behavior to reputation evaluation.

[0022] The payoff structure employed in this module follows the order of the standard prisoner's dilemma: the tempting payoff of betraying the other party and failing to cooperate is higher than the mutually beneficial payoff of cooperation; the mutually beneficial payoff is higher than the minimum payoff of betrayal; and the minimum payoff is higher than the loss payoff of cooperating while the other party betrays. This creates a tension between the short-term temptation of betrayal and the long-term benefits of cooperation at the individual level. The private reputation feedback update introduces a long-term constraint on this tension: while betrayal may bring a high single payoff, it lowers the individual's evaluation in the other party's private reputation matrix and further influences other subjects' judgments of the individual through indirect evaluations from common neighbors. Ultimately, the disconnection reduces the individual's subsequent interactive objects and long-term cumulative benefits. Therefore, the private reputation feedback not only records historical interactions but also, through the coupling of binary reputation labels and a unified reinforcement learning framework, enables the agent to exhibit behavioral patterns similar to those in real collaborative networks when weighing short-term gains against long-term reputation.

[0023] IV. Dynamic Relationship Reconstruction and Unified Reinforcement Learning Module This module uniformly categorizes all agent decision-making behaviors into three decision scenarios, sharing the same action value table and action selection and value update mechanism. The three decision scenarios are: a prisoner's dilemma interaction scenario, with action space of cooperation and betrayal; a connection termination scenario, with action space of maintaining and breaking the connection; and a connection establishment scenario, with action space of establishing and not establishing a connection. This module uses the current decision scenario type and the target object's binary reputation label to constitute a unified state for entering reinforcement learning, allowing a single action value table to carry the semantics of the three decision categories based on the scenario type. The agent uses a Boltzmann strategy to select actions based on their action values, increasing the probability of selecting any action as its action value in the corresponding state increases, with the Boltzmann inverse temperature regulation balancing exploration and utilization. Network connection updates employ an asymmetric rule: if either party in an established connection chooses to terminate, the corresponding connection is directly deleted; both parties in an unestablished connection must simultaneously choose to establish a new cooperative relationship. This asymmetric rule makes the network more prone to sparsification during evolution, thus enabling the reproduction of local breaks and differentiations in cooperative networks. At the end of each round, the module updates the value of actions accessed in the three decision-making scenarios synchronously, using the cumulative discount reward as the update target and a unified action value update rule. Furthermore, the module supports matrix snapshot saving and breakpoint resume training, and records multi-dimensional evolutionary indicators such as cooperation rate, network topology, reputation distribution, and action value for two types of agents: Heidegger-type and friend-focused.

[0024] It should be noted that the asymmetric connection reconstruction rule adopted by this module has a clear technical meaning: the termination of a connection requires only a unilateral decision, while the establishment of a connection requires the mutual consent of both parties. This asymmetry makes the network naturally tend to become sparsity in long-term evolution, thus replicating the characteristic of real cooperative networks where cooperative relationships are easily terminated unilaterally but require mutual consensus to be established. This, together with the differentiated evolution of private reputation, leads to local breaks and group differentiation in cooperative networks. Simultaneously, the value of connection termination and connection establishment actions is autonomously adjusted through the reinforcement learning training of the agents, making the network topology driven by the cooperative gains of the agents, rather than determined by externally preset fixed heuristic rules. This module uses the cumulative reward of discounts as a unified update target, synchronously updating the action values ​​of the three types of decision scenarios accessed in a single round. The random shuffling of the execution order of two types of events is used to eliminate simulation bias introduced by fixed execution sequences. The matrix snapshot and breakpoint continuation training mechanism ensure that the simulation process can be interrupted and resumed under long-term, multi-random seed control experiments, and the experimental results can be verified and reproduced.

[0025] V. Overall Closed-Loop Simulation Process The overall simulation of this invention is executed iteratively in a closed-loop process under the coordination of the above four modules. First, the system's global parameters, agent set, initial connection matrix, initial private reputation matrix, and action value table for each agent are initialized. Then, iterative rounds are initiated: within each round, Prisoner's Dilemma interaction events and network structure update events are determined and randomly generated based on the network structure update ratio and the number of Prisoner's Dilemma interactions in a single round. The number of network structure update events is rounded down from the product of the network structure update ratio and the number of Prisoner's Dilemma interactions in a single round. The execution order of the two types of events is randomly shuffled to eliminate simulation bias caused by a fixed execution sequence. For each target agent to be interacted with, the perceived reputation is calculated based on the private reputation matrix, common neighbor information, and the type acceptance rules, and mapped to a binary reputation label to generate a lightweight unified state. When executing Prisoner's Dilemma interaction events, both parties engage in a game to obtain immediate gains and update the private reputation matrix according to the game actions. When executing network structure update events, a pair of agents is randomly selected to complete a bidirectional decision to establish or dissolve a connection, dynamically modifying the global network topology. After all events in a single round have been executed, the action value tables for the three decision scenarios are updated uniformly based on the cumulative reward of the discount. Finally, all-dimensional evolution indicators are recorded, and snapshots of the connection matrix and private reputation matrix are saved at preset intervals, supporting breakpoint resume training, and iterating in a loop until the total number of simulation rounds is reached.

[0026] As can be seen from the closed-loop process described above, this invention connects the feedback update of private reputation, the generation of binary reputation labels, the unified action selection for three types of decision scenarios, and the round-level action value update into a complete iterative unit in each round, enabling the three levels of reputation, policy, and network topology to evolve synchronously under the same low-dimensional state representation. This closed loop does not rely on complete global network information as state input, and it carries multiple types of decision semantics through the combination of scenario type and binary reputation labels, thereby suppressing the explosion of state space dimensions while ensuring that the simulation process is interruptible, recoverable, and the results are reproducible.

[0027] Example 1: Simulation of Trusted Cooperative Game Theory for Supply Chain Collaborative Networks This embodiment takes the supply chain collaboration network as the application object, abstracts each supply chain enterprise as an intelligent agent, and the cooperative supply relationship between enterprises is abstracted as the cooperative connection between intelligent agents. The dynamic process of enterprises autonomously selecting and eliminating partners in supply chain collaboration is simulated and deduced.

[0028] In terms of simulation environment and parameter configuration, the total number of agents was set to 50, forming a heterogeneous agent interaction network composed of 50 supply chain enterprises; among them, there were 25 Heidegger-type agents and 25 friend-focused agents. The total number of simulation rounds was set to 2000, the number of Prisoner's Dilemma interactions per round was 100, the network structure update ratio was 0.3, the reputation update step size was 0.05, the learning rate was 0.1, the discount factor was 0.9, the Boltzmann inverse temperature was 8, and the upper and lower limits of private reputation ranged from -1 to +1. Specifically, the network structure update ratio controls the activity level of network structure adjustment in each round; the reputation update step size controls the impact of a single interaction on the private reputation evaluation; the learning rate controls the speed at which the action value table converges to the cumulative discounted return; the discount factor controls the agent's emphasis on future gains; and the Boltzmann inverse temperature adjusts the balance between exploration and exploitation in action selection, with a higher inverse temperature indicating a greater tendency to exploit currently estimated high-value actions.

[0029] In terms of the initialization process, an initial supply chain cooperation connection matrix is ​​randomly generated, so that some enterprises have pre-existing cooperation connections; all private reputation ratings in the private reputation matrix are initialized to 0, indicating that each enterprise holds a neutral trust attitude at the start of the simulation; a blank action value table is initialized for each agent, and type labels are assigned to each agent according to the ratio of 25 Heide type and 25 friend-focused type.

[0030] In terms of single-round iterative execution, the event composition is first determined according to the network structure update ratio and the number of single-round Prisoner's Dilemma interactions within each round, i.e., 100 Prisoner's Dilemma interaction events are generated, and the number of network structure update events is 30, which is the product of the network structure update ratio of 0.3 and 100, rounded down. Then, the execution order of the above 130 events is randomly shuffled to eliminate the bias introduced by the fixed execution sequence to the simulation results. For each Prisoner's Dilemma interaction event to be executed, the initiating enterprise calculates the perceived reputation for its interaction object: the direct evaluation items formed by the initiating enterprise and the object in the direct interaction history are added to the indirect evaluation items transmitted by their common partners to obtain the perceived reputation; among them, the Heidegger-type enterprise accepts the indirect evaluations transmitted by its common partners through both positive and negative relationship chains, while the friend-focused enterprise only accepts the indirect evaluations transmitted by its own positive partners. Subsequently, the continuous perceived reputation is binarized into a binary reputation label, which is composed of good reputation and bad reputation. The current scenario type, which is the Prisoner's Dilemma interaction scenario, is then used to form a unified state with the binary reputation label. Based on this, the initiating company selects the action of cooperation or betrayal from the action value table according to the Boltzmann strategy.

[0031] Two companies that have established a cooperative connection gain benefits based on their chosen actions: If both choose to cooperate, they each gain mutually beneficial benefits; if one betrays and the other cooperates, the betraying party gains a tempting higher benefit, while the cooperating party suffers a loss; if both betray, they each gain the minimum benefit. After each interaction, the private reputation matrix is ​​updated based on the actions of both parties—cooperative behavior increases the other party's private reputation evaluation from one's own perspective by a step size of 0.05, while betrayal behavior decreases the corresponding evaluation by the same amount. All update results are truncated within the range of negative 1 to positive 1. Private reputation iterates dynamically with each interaction and serves as the direct evaluation item for subsequent calculations of perceived reputation, continuously influencing all subsequent decision-making behaviors of the companies. For each pending network structure update event, a pair of companies is randomly selected for dynamic relationship reconstruction decisions: if a cooperative connection already exists between them, the connection is terminated, and either party can delete the cooperative connection by choosing to terminate it according to the Boltzmann strategy; if no cooperative connection exists between them, the connection is established, and a new cooperative connection is added only if both parties choose to establish a connection. The value of connection termination and connection establishment actions shares the same action value table as the value of Prisoner's Dilemma actions, and is distinguished by scenario type.

[0032] After all 130 events in a single round have been executed, the cumulative discounted reward is calculated based on the state, selected actions, and immediate benefits of each agent cached in this round. Using a unified action value update rule, the action values ​​accessed in the three decision scenarios—Prisoner's Dilemma interaction scenario, connection dissolution scenario, and connection establishment scenario—are updated uniformly in the direction of the cumulative discounted reward, with a learning rate of 0.1. Every 100 rounds, a snapshot of the connection matrix and private reputation matrix is ​​saved so that it can be restored from the most recent snapshot and iterated again after the simulation is interrupted. The above process is repeated until the total number of simulation rounds reaches 2000. During the simulation, the following evolutionary indicators are recorded for two types of agents: Heidegger-type and friend-focused agents: average cooperation rate of the two types of agents, average number of network connections, number of isolated enterprise nodes, global private reputation distribution, and frequency of cooperative relationship breakage.

[0033] In terms of implementation effects and mechanisms, the experiment revealed significant differentiated evolutionary characteristics: Heidegger-type agents, due to simultaneously accepting indirect evaluations transmitted through both positive and negative relationship chains, are easily influenced by negative neighbor information and frequently sever cooperative relationships, resulting in a higher frequency of cooperative relationship breakages and more isolated enterprise nodes in their subnets; Friend-focused agents, on the other hand, due to only accepting information transmitted by positive partners, are relatively insensitive to the spread of negative information, resulting in stronger cooperative network stability and a higher average number of connections. Traditional simulation models using globally unified reputation and fixed network topology cannot reproduce the evolutionary differentiation patterns caused by the differences in heterogeneous acceptance rules. Furthermore, since this embodiment only uses binary reputation labels and scenario types to construct reinforcement learning states, rather than inputting the complete 50-node global topology and full history, the size of the action value table is effectively constrained, allowing Q-learning to converge stably within 2000 rounds, and long-term simulations did not exhibit numerical instability caused by state dimension expansion.

[0034] As a comparison, this embodiment can also, while keeping other configurations unchanged, configure the proportions of Heide-type and friend-focused agents to different values ​​and run repeatedly to examine the impact of heterogeneous proportions on the supply chain collaboration network in batches. Experiments show that: the higher the proportion of Heide-type agents, the more frequent the supply relationship disruptions caused by the spread of negative evaluations along the negative relationship chain in the supply chain network, the more isolated enterprise nodes there are, and the overall cooperation rate decreases accordingly; the higher the proportion of friend-focused agents, the higher the average number of connections and connectivity of the supply chain network, and the more stable the cooperation relationships. The quantitative indicators such as cooperation rate, number of isolated nodes, and frequency of relationship disruptions obtained from the above comparative experiments can provide simulation basis for supply chain platforms when formulating entry thresholds, penalties for breach of contract, and rules for cooperation rewards and punishments. Because this invention uses a matrix snapshot and breakpoint retraining mechanism, the above multiple sets of comparative experiments can be resumed from the most recent snapshot and run repeatedly with multiple random seeds after simulation interruption, thereby ensuring the verifiability and reproducibility of the experimental results.

[0035] Example 2: Simulation of Node Trust Differentiation for Computing Resource Sharing Platform This embodiment uses a computing power resource sharing platform as the application object, abstracting each computing power node as an intelligent agent, and the collaborative relationship of shared computing power between nodes as a cooperative connection. It simulates the process of selective trust between computing power nodes and dynamic reorganization of cooperative partners within the platform. The parameters adjusted in this embodiment are as follows: the total number of intelligent agents is set to 100, and the network structure update ratio is increased to 0.5 to simulate more frequent reorganization of cooperative relationships in a computing power sharing scenario; other parameters such as reputation update step size, learning rate, discount factor, Boltzmann inverse temperature, and reputation value range can be used as configured in Embodiment 1. To examine the impact of the proportion of heterogeneous intelligent agents on the platform's collaborative evolution, this embodiment sets up multiple control experiments, setting the proportion of Heidegger-type intelligent agents to be 20%, 50%, and 80%, respectively. Under the same initialization method and iteration process, the computing power sharing efficiency and node trust differentiation of the platform under different proportions are compared.

[0036] In terms of iterative execution, this embodiment follows the same single-round event hybrid scheduling, perceived reputation calculation and heterogeneous trust acceptance, prisoner's dilemma interaction and private reputation feedback, dynamic relationship reconstruction, and round-level action value unified update process as Embodiment 1; the only difference lies in the values ​​of agent size, network update ratio, and heterogeneity ratio. The simulation also saves snapshots of the connection matrix and private reputation matrix at a set period to support breakpoint resume training, and supports repeated running with multiple random seeds to verify and reproduce the results. Experimental results show that as the proportion of Heidegger-type agents increases from 20% to 80%, the number of cooperative relationship breakage events caused by the spread of negative information in the platform increases, node trust shows a more significant differentiation, and some high-load or previously betrayed nodes are gradually disconnected by most nodes and tend to be isolated, and the overall computing power sharing efficiency of the platform decreases accordingly; while with a higher proportion of friend-focused agents, the cooperative network maintains higher connectivity and stability, and the computing power sharing efficiency is relatively higher. This embodiment relies on a private reputation mechanism to recreate the selective trust behavior of computing nodes, which can provide quantitative simulation support for the design of platform access rules and reward and punishment mechanisms. For example, the platform can use this to assess what kind of access threshold and betrayal punishment should be adopted under different trust inference logic proportions, so as to achieve a balance between maintaining node trust and ensuring sharing efficiency.

[0037] Furthermore, this embodiment can record indicators such as the average cooperation rate, average number of network connections, number of isolated nodes, global private reputation distribution, and frequency of cooperative relationship breakage for the two types of agents in multiple sets of control experiments. It also compares the evolution trends of each indicator with different Heidegger-type proportions over simulation rounds, thereby quantitatively characterizing the impact of trust inference logic on the stability of the computing power sharing platform. Since this embodiment also uses only binary reputation labels and scenario types to constitute the reinforcement learning state, rather than using the global topology of 100 nodes and the full interaction history as input, the dimension of the action value table does not expand with the number of nodes and the length of the history. This allows Q-learning to still converge stably within a set number of rounds even when the node scale is expanded to 100 and the network structure is updated more frequently. Combined with matrix snapshots and breakpoint training mechanisms, the aforementioned multiple sets of control experiments can be resumed after simulation interruption and repeated with multiple random seeds, ensuring that the experimental results are verifiable and reproducible. This provides stable and reliable quantitative simulation support for the design of admission and reward / punishment mechanisms for the platform under different trust structures.

[0038] Example 3: Simulation of Collaborator Reorganization in Service Outsourcing Networks This embodiment uses a service outsourcing network as the application object, abstracting each service provider and demander as an intelligent agent, and the outsourcing collaboration relationship as a cooperative connection. It simulates the process by which parties in the outsourcing platform autonomously select and eliminate collaborative partners based on their performance. This embodiment can set the total number of intelligent agents to 80, with half being Heidegger-type and half being friend-focused. The total number of simulation rounds and the number of Prisoner's Dilemma interactions per round can follow the same order of magnitude as in the previous embodiment. The network structure update ratio is set to 0.4 to reflect a moderate reorganization frequency of outsourcing relationships. The simulation follows the same process as the previous embodiment in each stage: initialization, hybrid event scheduling, perceived reputation inference, Prisoner's Dilemma feedback, dynamic relationship reconstruction, and unified update of round-level action value. In this scenario, service providers with good performance continuously improve their private reputation from the perspective of most demanders, making it easier to establish or maintain connections with them. Conversely, service providers that default experience a decline in private reputation from the perspective of demanders with whom they directly interact. This reputation, in turn, influences the judgments of other entities through indirect evaluations from mutual neighbors. Heidegger-type demanders will more quickly disconnect from these service providers due to the negative evaluations transmitted through negative relationship chains, thus creating a collaborative elimination of low-performing parties at the network level. This embodiment demonstrates that the present invention can be used to simulate the dynamic process of collaborative partner reorganization under different outsourcing governance rules, providing a quantitative basis for platform rule design.

[0039] In terms of parameter configuration, this embodiment can set the reputation update step size to 0.05, the learning rate to 0.1, the discount factor to 0.9, the Boltzmann inverse temperature to 8, and the upper and lower limits of the private reputation value to be negative 1 to positive 1, consistent with the previous embodiment, so as to examine the topology reorganization pattern of the service outsourcing network separately under the same algorithm configuration. Within a single round, the number of network structure update events is determined by rounding down the product of the network structure update ratio of 0.4 and the number of Prisoner's Dilemma interactions in a single round. This number is then randomly mixed with the Prisoner's Dilemma interaction events and the execution order is shuffled. Fulfillment corresponds to the cooperative action in the Prisoner's Dilemma, and breach of contract corresponds to the betrayal action. The private reputation is increased or decreased by 0.05 accordingly and truncated within the value range.

[0040] Regarding the implementation effects, the experiment observed that as the proportion of Heidegger-type demanders increased, the phenomenon of defaulting service providers being quickly disconnected by multiple demanders through negative relationship chains and becoming isolated became more pronounced, and the collaborative elimination of low-performing parties in the network was more rapid. However, when the proportion of friend-focused demanders was high, due to the suppression of negative evaluation propagation, some service providers who had occasionally defaulted but generally performed well were able to retain more connections, resulting in higher connectivity of the cooperative network. Therefore, the platform can evaluate the strength of its performance evaluation mechanism and blacklist removal rules based on quantitative indicators such as the number of isolated nodes, the frequency of cooperative relationship breaks, and the average number of connections under different trust inference logic proportions in this embodiment, in order to achieve a balance between eliminating low-performing parties and maintaining network connectivity. This embodiment also uses only binary reputation labels and scenario types to construct reinforcement learning states, ensuring that the size of the action value table does not expand with the increase in the number of service providers and demanders, thus maintaining convergence stability and reproducibility in long-term, multi-random seed controlled experiments.

[0041] Scalable implementation The method framework of this invention has good scalability. First, the heterogeneous types of agents are not limited to Heidegger type and friend-focused type. New types of agents that accept rules based on other indirect information can be introduced in the same way, while maintaining the private reputation inference and binary label compression mechanisms, to compare more trust inference logics. Second, the underlying game is not limited to the Prisoner's Dilemma; it can be replaced by other symmetric two-player games such as snowplow game and public goods game, requiring only corresponding adjustments to the payout rules, while the reputation feedback, state compression, and unified reinforcement learning framework remain unchanged. Third, the initial network topology can adopt different structures such as random networks, scale-free networks, or small-world networks to examine the impact of the initial topology on cooperative evolution. Fourth, parameters such as reputation update step size, network structure update ratio, and Boltzmann inverse temperature can all be adjusted within a reasonable range to compare the group cooperation level, number of isolated nodes, and frequency of relationship breakage under different parameter configurations in batches. All of the above extensions do not change the core concept of the three-layer coupled co-evolution of reputation, policy, and network topology in this invention, and therefore all fall within the protection scope of this invention.

[0042] Furthermore, the implementation details such as the mapping threshold from private reputation to binary reputation labels, the relative weights of direct and indirect evaluation items in perceived reputation, and the snapshot saving period in this invention can all be set by those skilled in the art in combination with specific application scenarios without departing from the concept of this invention. Corresponding equivalent substitutions and improvements should also fall within the protection scope of this invention. Based on this, those skilled in the art can flexibly adapt this invention to a wider range of networked trusted collaborative game simulation scenarios.

[0043] This invention can be implemented using a general-purpose simulation computer. The method can be stored in the form of a computer program on a computer-readable storage medium and read and executed by a processor. The computer-readable storage medium includes, but is not limited to, read-only memory, random access memory, magnetic disks, or optical disks, which are media capable of storing program code. Those skilled in the art should understand that the above embodiments are merely preferred embodiments of the present invention, and the present invention is not limited to the above embodiments. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A social equilibrium-inspired multi-agent reinforcement learning game simulation method, applied to networked trusted cooperative game simulation in service outsourcing networks, supply chain collaboration, or computing resource sharing scenarios, characterized in that... Executed by a simulation computer, including the following steps: Step 1: Initialize the heterogeneous agent interaction network, private reputation matrix, connection matrix, and reinforcement learning parameters. The heterogeneous agent interaction network contains N agents, divided into two categories: Heidegger agents and friend-focused agents. The connection matrix represents the cooperative connection state between agents, and the private reputation matrix stores each agent's independent private reputation evaluation of the other agents. The reinforcement learning parameters include the total number of agents, the number of Heidegger agents, the total number of simulation rounds, the number of Prisoner's Dilemma interactions per round, the network structure update ratio, the reputation update step size, the learning rate, the discount factor, and the Boltzmann inverse temperature. For each agent, initialize an independent action value table, upper and lower limits of reputation values, and initial cooperative connection relationships. Step 2: Perform simulation in rounds and iteratively. In each round, generate a mixture of Prisoner's Dilemma interaction events and network structure update events. The number of network structure update events is the product of the network structure update ratio and the number of Prisoner's Dilemma interactions in a single round, rounded down. The execution order of the two types of events is randomly shuffled. Step 3: For the target agent to be interacted with, the initiating agent calculates the perceived reputation of the target agent based on the private reputation evaluation of the target agent in the private reputation matrix, the common neighbor information of the two, and the indirect information acceptance rules corresponding to the type of the initiating agent. The perceived reputation is mapped to a binary reputation label of good reputation or bad reputation, and the current decision scenario type and the binary reputation label constitute a unified state for entering reinforcement learning. Step 4: For agents with cooperative connections, execute the Prisoner's Dilemma interaction event in pairs. Both parties obtain corresponding benefits based on the cooperative action or betrayal action selected by each party according to the action value table. After the interaction is completed, update the private reputation matrix. Cooperative actions improve the private reputation evaluation of the other party from the perspective of the agent. The update range is controlled by the reputation update step size, and the result is constrained within the upper and lower limits of the reputation value. Step 5: Execute the network structure update event and randomly select a pair of agents to make a dynamic relationship reconstruction decision: If the two agents already have a cooperative connection, the connection is terminated in the scenario where either agent chooses to terminate the connection based on the action value table; if the two agents do not have a cooperative connection, the connection is established in the scenario where the connection is added only when both agents choose to establish the connection based on the action value table; both connection termination and connection establishment are learnable actions for reinforcement learning. Step 6: After all events have been executed in each round, the action value table is updated synchronously for the three decision scenarios of Prisoner's Dilemma interaction scenario, connection dissolution scenario and connection establishment scenario, with the cumulative discount reward as the update target and a unified action value update rule as the update target. Step 7: Record the evolution indicators of cooperation ratio, network topology, private reputation distribution and action value for two types of intelligent agents: Heide-type agents and friend-focused agents. Save snapshots of the connection matrix and the private reputation matrix at a set period to support breakpoint resume training. Iterate in a loop until the total number of simulation rounds is reached.

2. The social equilibrium-inspired multi-agent reinforcement learning game simulation method according to claim 1, characterized in that, The unified state for entering reinforcement learning, which consists of the current decision scenario type and the binary reputation label, includes: binarizing and compressing the continuously valued perceived reputation into two discrete values, good reputation and bad reputation, to obtain the binary reputation label; using the binary reputation label together with the current decision scenario type as the state for entering reinforcement learning, without using the complete global network topology, the full neighbor structure, and the historical interaction sequence as the state input for reinforcement learning.

3. The social equilibrium-inspired multi-agent reinforcement learning game simulation method according to claim 2, characterized in that, The calculation of the perceived reputation of the target agent includes: adding the direct evaluation items formed by the direct interaction history between the initiating agent and the target agent to the indirect evaluation items transmitted by their common neighbors to obtain the perceived reputation; wherein, when the initiating agent is a Heidegger-type agent, the indirect evaluations transmitted by the common neighbors through both positive and negative relationship chains are accepted; when the initiating agent is a friend-focused agent, only the indirect evaluations transmitted by the initiating agent's own positive relationship neighbors are accepted.

4. The social equilibrium-inspired multi-agent reinforcement learning game simulation method according to claim 3, characterized in that, The updating of the private reputation matrix includes: after each Prisoner's Dilemma interaction, according to the cooperative or betrayal actions of both parties, increasing or decreasing the corresponding private reputation evaluation in the private reputation matrix by the magnitude of the reputation update step, and truncating the increased or decreased result within the upper and lower limits of the reputation value; the updated private reputation evaluation dynamically iterates with each interaction and serves as the basis for calculating the direct evaluation item of the perceived reputation in step three, continuously influencing the initiating agent's binary reputation label determination of the target agent.

5. The social equilibrium-inspired multi-agent reinforcement learning game simulation method according to claim 1, characterized in that, The action spaces for the three decision-making scenarios are as follows: the action space for the Prisoner's Dilemma interaction scenario is cooperation versus betrayal; the action space for the connection termination scenario is maintaining the connection versus breaking the connection; and the action space for the connection establishment scenario is establishing the connection versus not establishing the connection. The three decision-making scenarios share the same action selection mechanism based on the Boltzmann strategy, which increases the probability of selecting any action as the action value of that action in the corresponding state increases, and the determinism of action selection is adjusted by the Boltzmann inverse temperature regulation.

6. The social equilibrium-inspired multi-agent reinforcement learning game simulation method according to claim 5, characterized in that, The network structure update event is executed using an asymmetric connection reconstruction rule: in the connection termination scenario, either party in a pair of agents with an existing cooperative connection chooses to terminate the connection, thus deleting the cooperative connection between them; in the connection establishment scenario, both parties in a pair of agents without a cooperative connection must choose to establish a new cooperative connection between them; the action value of the connection termination action and the connection establishment action is autonomously adjusted through the reinforcement learning training of the agents, so that the network topology is driven by the cooperative benefits of the agents rather than by external preset.

7. The social equilibrium-inspired multi-agent reinforcement learning game simulation method according to claim 1, characterized in that, The method of synchronously updating the action value table under the three decision scenarios with the cumulative discount reward as the update target includes: caching the state, selected action, and immediate reward of all decision scenarios in each round; calculating the cumulative discount reward for each agent after all Prisoner's Dilemma interaction events and network structure update events in that round have been executed; and updating the accessed action values ​​under the three decision scenarios of the Prisoner's Dilemma interaction scenario, the connection termination scenario, and the connection establishment scenario in the direction of the cumulative discount reward using the same action value update rule; wherein the random shuffling of the execution order of the two types of events is used to eliminate the simulation bias introduced by the fixed execution sequence.

8. The social equilibrium-inspired multi-agent reinforcement learning game simulation method according to claim 1, characterized in that, The method forms a co-evolutionary closed loop with three layers of coupling: reputation, policy, and network topology. The prisoner's dilemma interaction updates the private reputation matrix. The private reputation matrix drives the agent's cooperative actions, connection establishment actions, and connection dissolution actions through the perceived reputation and the binary reputation label. The connection establishment and connection dissolution actions change the network topology, thereby changing the set of subsequent interactive objects. The round-level discount cumulative reward updates the action value table in reverse. The method also saves snapshots of the connection matrix and the private reputation matrix at the set period. After the simulation is interrupted, the snapshots are restored and the iteration continues.

9. A social equilibrium-inspired multi-agent reinforcement learning game simulation system, characterized in that, The system is implemented by a simulation computer and includes: The heterogeneous agent and private reputation network initialization module is used to construct a heterogeneous agent interaction network containing N agents, divide the N agents into two categories: Heidegger agents and friend-focused agents, and initialize the connection matrix, private reputation matrix, action value table of each agent, upper and lower limits of reputation value, and reinforcement learning parameters including network structure update ratio, reputation update step size, learning rate, discount factor and Boltzmann inverse temperature. The social balance heuristic private reputation inference module is used to calculate the perceived reputation of the target agent to be interacted with based on the private reputation evaluation in the private reputation matrix, the information of common neighbors, and the indirect information acceptance rules corresponding to the type of the initiating agent. The perceived reputation is mapped to a binary reputation label of good reputation or bad reputation. The current decision scenario type and the binary reputation label constitute a unified state for entering reinforcement learning. Among them, the Heidegger type agent accepts the indirect evaluations transmitted by common neighbors through positive and negative relationship chains, while the friend-focused type agent only accepts the indirect evaluations transmitted by its own positive relationship neighbors. The Prisoner's Dilemma Interaction and Private Reputation Feedback Module is used to perform Prisoner's Dilemma Interactions on pairs of intelligent agents with cooperative connections, allocate benefits based on cooperative actions or betrayal actions, and increase or decrease the corresponding private reputation evaluation by the magnitude of the reputation update step size after the interaction is completed, and constrain it within the upper and lower limits of the reputation value range. The Dynamic Relationship Reconstruction and Unified Reinforcement Learning module is used to incorporate three decision scenarios—Prisoner's Dilemma Interaction, Connection Deconstruction, and Connection Establishment—into the same reinforcement learning framework. In the connection deconstruction scenario, either party in a pair of connected agents chooses to delete the connection. In the connection establishment scenario, both parties in a pair of unconnected agents jointly choose to add a connection. At the end of each round, the action value table for the three decision scenarios is updated synchronously with the cumulative reward of the discount as the update target. The snapshot and metric recording module is used to record the evolution metrics of cooperation ratio, network topology, private reputation distribution and action value for two types of agents: Heide-type agents and friend-focused agents. It also saves snapshots of the connection matrix and the private reputation matrix at a set period to support breakpoint resume training.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the social equilibrium-inspired multi-agent reinforcement learning game simulation method as described in any one of claims 1 to 8.