Opinion evolution analysis method based on multi-agent reinforcement learning

Abstract

The application aims to provide a public opinion evolution analysis method based on multi-agent reinforcement learning, comprising: after initializing an agent decision model, the agent obtains an implementation reward value by exploring an environment and combining a pre-set reward function, and updates an agent public opinion value through a dynamics model; when the agent completes convergence or reaches a maximum step limit, a round of exploration is ended, and information obtained in the round of exploration is used to update the agent decision model through a policy gradient method; through the method, the problem of the agent being unable to automatically adapt to learn a decision promoting consistency in the related art is solved, and the consistency performance of a system in public opinion dynamics is enhanced in a limited number of steps.

Description

Technical Field

[0001] This invention relates to a public opinion evolution analysis method based on multi-agent reinforcement learning, mainly involving the acquisition of public opinion perception by agents and the analysis of system public opinion evolution. Background Technology

[0002] With the rapid development of wireless communication and internet technologies in recent years, people are able to quickly exchange opinions on a large scale in real time. Among these, Social Network Group Decision Making (SNGDM) is becoming an effective method for explaining human behavior, offering potential value to academic and engineering fields, such as supplier selection and public opinion management. Public opinion management, for example, involves ensuring that everyone initially has an opinion value, and through public opinion management, that everyone's public opinion values ​​reach a consensus or converge into several different opinions.

[0003] In today's era of rapid internet technology development, the internet is rife with a mix of reliable and unreliable information, making people easily misled by false information and prone to forming erroneous opinions. Therefore, public opinion management is often used to effectively monitor the internet and curb the spread of misinformation. By analyzing the evolution of current public opinion, the final outcome can be obtained, allowing relevant departments or personnel to take timely action. However, common problems in public opinion evolution analysis include: the analysis requires excessively long steps, hindering the rapid acquisition of the final outcome; and the analysis focuses too much on the outcome, resulting in numerous clusters of convergence, making it difficult for relevant personnel to address the issue effectively.

[0004] SNGDM comprises a group of agents capable of expressing their own viewpoints. These agents gradually achieve their goals by communicating with their neighbors and selecting the optimal candidate solution. Viewpoints are one of the key factors influencing and defining the behavior of agents. Each agent in the system adjusts its own viewpoint value by considering the viewpoint values ​​of its neighbors, thereby achieving a stable structure of consistency, polarization, and classification within the system. This process of agent evolution is called viewpoint evolution, also known as public opinion dynamics.

[0005] While the SNGDM method is widely used, the complexity of social networks and the presence of sparse network connections make it difficult to achieve comprehensive consistency. Furthermore, the sparsity of the network and the limited perception range of agents result in variable-length neighbor state information perceived by different agents at different times. Therefore, designing an efficient communication method to enhance the consistency of SNGDM systems in large-scale scenarios, given the variable perception range of agents, remains a challenging issue.

[0006] Currently, consensus methods in SNGDM include consensus evolution methods considering pressure parameters and consensus evolution methods based on common neighbors. The consensus evolution method considering pressure parameters establishes a subject-based group pressure bounded information model by taking into account the influence of neighbors on agents within a confidence interval. This ensures that agents are only influenced by neighbors within a certain range; that is, agents only communicate with each other when the distance between them is less than a certain range. This method achieves consensus among individual agents through group pressure. However, if the confidence interval is set inappropriately, it can easily lead to the final evolutionary result splitting into multiple different clusters.

[0007] The consensus evolution method based on common neighbors further considers common neighbors among agents, using these common neighbors to help agents select remote neighbors. This method helps the system quickly achieve consensus by building remote connections between agents; however, the preference parameters for agents selecting remote neighbors need to be manually adjusted, which presents uncontrollable limitations in practical applications.

[0008] In summary, existing consensus evolution methods do not consider the agent's proactive choice of neighbors, but rather achieve system consensus by altering the influence of neighbors on the agent. Furthermore, in existing methods, agents struggle to proactively explore the environment and cannot adaptively learn decisions that promote consensus, leading to reduced discriminative ability and consequently lower system consensus performance. In other words, in large-scale social networks, there remains a lack of effective methods for reconciling conflicting opinions, reducing cluster size at system stability, and thus enhancing public opinion consensus. Summary of the Invention

[0009] The purpose of this invention is to provide a public opinion evolution analysis method based on multi-agent reinforcement learning, so as to solve at least one technical problem in the prior art.

[0010] The technical solution of this invention is:

[0011] A public opinion management method based on multi-agent reinforcement learning includes:

[0012] Initialize the agent decision-making model and experience pool based on the LSTM network, and initialize the agent system environment, including initializing the agent decision-making model policy parameters θ;

[0013] Each agent saves the set of neighbors it detects and the set of exploration actions at any given time into a temporary experience pool.

[0014] The communication topology matrix for any given moment is generated by the actions of all agents at any given moment; and the agent system environment for the next moment is updated based on the communication topology matrix and the agent system environment.

[0015] Using the pre-set reward method, obtain the reward of each agent at this moment; and put the reward into the corresponding agent exploration experience pool, and then save the data in the agent exploration experience pool into the temporary experience pool respectively;

[0016] According to the pre-set stopping condition, randomly select at least 1 agent trajectory information from the experience pool as the input of the policy gradient method, and obtain the policy gradient output to update the agent decision-making model;

[0017] Set the maximum number of training rounds, explore and train the agent decision-making model, and obtain the learned agent decision-making model;

[0018] Through the learned agent decision-making model, enable the agent to make consistent decisions adaptively according to the agent system environment.

[0019] The experience pool D includes:

[0020] D = {D_i|i ∈ [0, N×M]}, where,

[0021] D_i = {D_i_j|j ∈ [0, T]}, where,

[0022] D_i_j = {s i (k), a i (k), r i (k)};

[0023] In D_i_j, s i (k) represents the set of neighbor public opinion values detected by agent i at time k; a i (k) represents the exploration action set of agent i at time k; r i (k) represents the reward value obtained by agent i at time k.

[0024] The agent system environment x(k) includes:

[0025] k = 0, x(k) = {agent_i|i ∈ [0, N]}, where, agent_i represents the public opinion value of agent i.

[0026] Agent i obtains the neighbor set s i (k) according to the detection radius rc, s i (k) = {x j (k)||x i (k)-x j (k)| < rc|, j ∈ [0, N]}, where, x i(k) represents the sentiment value of agent i in the system environment x(k);

[0027] The intelligent agent i will select the neighbor set s i Combining (k) and its corresponding subscripts, we obtain the input to the agent's decision-making model. After passing through an LSTM bidirectional recurrent network and a Softmax activation function, we output the action of agent i towards its neighbors at the current time k, thus obtaining the set of exploration actions ai(k). i (k)={p j |j∈[0,N]},p i p represents the selection weight of agent i for agent j. i The sum is 1;

[0028] The neighbor set s i (k) and the set of exploration actions a i (k) Save to the temporary experience pool.

[0029] The phrase "generating a communication topology matrix at any given moment based on the actions of all agents; and updating the agent system environment for the next moment based on the communication topology matrix and the agent system environment" includes:

[0030] From temporary experience pool D i Take the neighbor set s from (k) i (k) and the set of exploration actions a i (k), construct the communication topology matrix L(k) at time k, L(k) = [l ij (k)]where l ij (k) represents the communication topology between agents i and j at time k; N i (k) represents the set of neighbor indices N of agent i at time k. i (k)={j||x i (k)-x j (k)| <rc|,j∈[0,N]}

[0031] Based on the communication topology matrix L(k) and the agent system environment X(k), update the agent system environment at time k+1 according to the following formula: X(k+1)=L(x)X(k);

[0032] The phrase "using a pre-set reward method to obtain the reward for each agent at that moment; and placing the reward into the corresponding agent's exploration experience pool, and then saving the data in the agent's exploration experience pool to the temporary experience pool" includes:

[0033] Let G1(·), G2(·), and G3 represent three different objective reward functions; r i(k) represents the reward value of agent i at time k, then:

[0034] r i (k)=G1(X(k+1))+G2(X(k+1))+G3

[0035] Wherein, G1(·): is used to help the agent learn and enhance the consistency of the system, and its formula is as follows:

[0036] G1=g1(X(k+1))-g1(X -i (k+1))

[0037] g1(X(k+1))=std(X(k+1)) / N

[0038] N represents the number of agents in the system, X -i (k+1) represents the system environment after removing agent i, and std(·) represents the standard deviation operation;

[0039] G2(·): Used to help the agent learn to improve the connection density of the system, and its formula is as follows:

[0040] G2=g2(X(k+1))–g2(X -i (k+1))

[0041]

[0042] Where, ∑ i∈V |N i (k+1)| represents the number of connections existing in the system at time k+1;

[0043] G3: Used to enable agents to learn to reduce the steps required for the system to achieve the following formula:

[0044] G3 = -0.01;

[0045] The value of G3 ranges from [0, -0.01×S], where S represents the number of steps required for the system to reach stability in this round.

[0046] The obtained r i(k) Save the results to the corresponding temporary experience pool D_i_k, and save all temporary experience pools D_i_k obtained in each round to the temporary experience pool D_i corresponding to the current round. Here, g1 ranges from [0,-1], and the closer it is to 0, the better the convergence of the system. G1 ranges from [0,1), and the closer it is to 1, the more agent i helps improve the global consistency of the system. g2 ranges from [0,1], and the closer it is to 1, the higher the connection density of the system. G2 ranges from [0,1), and the closer it is to 1, the more agent i helps improve the global density. The closer the value of G3 is to 0, the fewer steps are needed for the system to reach stability.

[0047] The "pre-set stop conditions" include:

[0048] The value of k+1 at the next moment is greater than the maximum number of training rounds T.

[0049] The phrase "randomly selecting at least one agent trajectory information from the experience pool as input to the policy gradient method, obtaining the policy gradient output, and updating the agent decision model" includes:

[0050] Randomly select batch_size agent trajectory information (D_i) from the experience pool D without repetition as input to the policy gradient method, and initialize loss_all = 0.

[0051] Extract the information from the agent's trajectory information D_i to obtain: the list of observed values ​​for the agent's trajectory obs = {s i (k)|k∈(0,len(D_i))}, the action list of agent i's trajectory acts={a i (k)|k∈(0,len(D_i))} and the reward list for the trajectory of agent i, reward = {r i (k)|k∈(0,len(D_i))};where len(D_i) is the length of the trajectory information D_i;

[0052] Calculate the decaying reward value reward_sum from the obtained reward list rewards; extract the reward values ​​reward from the reward list rewards in reverse order, initially reward_sum = 0, according to the formula:

[0053] reward_sum = reward × 0.9 + reward_sum', to obtain the decaying reward reward_sum. Replace the reward value reward at the corresponding position in the decaying reward_sum list with the decaying reward reward_sum obtained in each step to obtain the decaying reward list reward_rd. Perform a standardization operation on the obtained decaying reward list reward_rd; reward_sum' represents the decaying reward reward_sum obtained in the previous step.

[0054] Based on the action list obs and the decaying reward list reward_rd of agent i, obtain the current loss value:

[0055] Determine whether all selected agents have been traversed, and update the agent decision model accordingly.

[0056] The obtained decaying reward table reward_rd is standardized, including:

[0057] For each value in the decaying reward table, apply the following formula:

[0058] reward=[reward–avg(reward_rd)] / std(reward_rd);

[0059] Where avg(reward_rd) represents the average value of the elements in the decaying reward list reward_rd, and std(reward_rd) represents the standard deviation of the elements in the decaying reward list reward_rd.

[0060] "Determine whether all selected agents have been traversed and update the agent decision model," including:

[0061] The loss value is summed in each round by summing the current loss value: loss_all = loss_all' + loss; where loss_all' represents the loss_all obtained by iterating through the previous agent.

[0062] After iterating through all selected agents, proceed according to the formula. Get the average value of loss_all And according to the formula: The agent decision model is updated, where ζ is the learning rate; θ is the decision parameter of the agent decision model; and batch_size is the maximum number of trajectories selected in one learning process.

[0063] The beneficial effects of the present invention include at least the following:

[0064] The method described in this invention, after initializing the agent's decision-making model, involves the agent exploring the environment and obtaining a reward value by combining it with a pre-set reward function, and updating the agent's sentiment value through a dynamic model. When the agent achieves convergence or reaches the maximum number of steps, the exploration round ends, and the information obtained in this round is used to update the agent's decision-making model through a policy gradient method. This method solves the problem in related technologies where agents cannot automatically adapt to learn decisions that promote consistency, enhancing the consistency performance of the system in sentiment dynamics within a limited number of steps. Specifically, in large-scale social networks, it can effectively and quickly reconcile conflicting opinions among groups, reducing the cluster size at system stability and the number of steps required to achieve stability. Attached Figure Description

[0065] Figure 1 This is a flowchart of the method described in this invention;

[0066] Figure 2 This is a structural diagram of an agent decision-making model;

[0067] Figure 3 This is a graph showing the public opinion value of the intelligent agent according to the method described in this invention.

[0068] Figure 4 A graph showing the sentiment value curve of an agent using the CNR analysis method;

[0069] Figure 5 A graph showing the sentiment value of an agent using the GP analysis method;

[0070] Figure 6 This is a graph showing the sentiment value of the intelligent agent using the HK analysis method. Detailed Implementation

[0071] The present application will now be further described with reference to the accompanying drawings.

[0072] Specific Implementation Example I:

[0073] The purpose of the method described in this invention is to enable intelligent agents to adaptively explore according to the system environment and promote system consistency through active intervention by the intelligent agents. To solve the above technical problems, the technical solution adopted in this embodiment includes the following steps:

[0074] Step A: Initialize the agent decision-making model and the agent exploration experience pool Di_i_k based on the LSTM network, and then proceed to Step B.

[0075] Step B: Begin a round of exploration, initialize the agent system environment x(k), obtain the current training round number m, and proceed to step C.

[0076] Step C: All agents explore according to the exploration method. Each agent i will record the set of neighbors s detected by itself at time k. i (k), explore action set a i (k) are saved to the temporary experience pool D_i_k respectively. After all agents in k have completed their exploration at this moment, proceed to step D.

[0077] Step D: Analyze the actions a of all agents at this moment k. i (k) Generate the communication topology matrix L(k) at time k, update the agent system environment x(k+1) at time k+1 based on L(k) and the current agent system environment x(k), and proceed to step E.

[0078] Step E: Calculate the reward r for each agent i at time k according to the predefined reward method. i (k), and r i (k) is placed into the corresponding temporary experience pool D_i_k. Then D_i_k is saved into the temporary experience pool D_i respectively. Proceed to step F.

[0079] Step F: Determine whether x(k+1) meets the stopping condition. If yes, save all temporary experience pools D_i in this round to experience pool D and proceed to step G. If not, return to step C and proceed to the next moment k = k+1.

[0080] Step G: Randomly select batch_size agent trajectory information from the experience pool D as input to the policy gradient method, and update the agent decision model with the policy gradient output, and then proceed to step H.

[0081] Step H: Determine if the current round number is less than the maximum training round number M. If yes, return to step B. If not, the exploration training is complete, and the agent decision model is obtained. At this point, the agent decision model enables the agent to make consistent decisions adaptively according to the system environment, and the process ends.

[0082] As a preferred technical method of the present invention, the experience pool D described in step A is as follows:

[0083] D = {D_i|i∈[0,N×M]}, where...

[0084] D_i={D_i_j|j∈[0,T]}, where,

[0085] D_i_j={s i (k),a i (k),r i (k)};

[0086] In Di_i_j, si (k) represents the set of neighbor public opinion values detected by agent i at time k; a i (k) represents the exploration action set of agent i at time k; r i (k) represents the reward value obtained by agent i at time k.

[0087] As a preferred technical method of the present invention, the relevant parameters described in step A are as follows:

[0088] The maximum number of training rounds M, the maximum number of training steps T, the learning batch size, the agent detection radius rc, the number of agents N, the experience pool D, and the minimum number of clusters min_clu in each round.

[0089] As a preferred technical solution of the present invention, the agent system environment x(k) described in step B is as follows:

[0090] When k = 0, x(k) = {agent_i | i ∈ [0, N]}. Where agent_i represents the public opinion value of agent i.

[0091] As a preferred technical solution of the present invention, step C includes the following steps:

[0092] Step C1, agent i obtains the neighbor set s i (k), s i (k) = {x j (k) || x i (k) - x j (k) | < rc |, j ∈ [0, N]}, where x i (k) represents the public opinion value of agent i in the system environment x(k), and proceeds to step C2.

[0093] Step C2, the agent i combines the neighbor set s i (k) and its corresponding subscripts to obtain the input input for the agent decision-making model. After passing through the LSTM bidirectional recurrent network and the Softmax activation function, the actions of agent i for its neighbors at the current time k are output, and the exploration action set a i (k) is obtained, a i (k) = {p j | j ∈ [0, N]}, p i represents the selection weight of agent i for agent j, and the sum of p i is 1; and proceeds to step C3;

[0094] Step C3, combine s i (k) and a i(k) Save it to the temporary experience pool D_i_k. Then proceed to step D.

[0095] As a preferred embodiment of the present invention, step D includes the following steps:

[0096] Step D1: Take s from Di_i_k i (k) and a i (k), construct the communication topology matrix L(k) at this time k, and proceed to step D2.

[0097] Step D2: Based on L(k) and x(k), calculate the system environment x(k+1) at the next time step k+1 using the following formula: X(k+1)=L(x)X(k), and proceed to step E.

[0098] As a preferred embodiment of the present invention, step E includes the following steps:

[0099] Step E1: Select agent i, and based on the system environment x(k+1) at the next time step k+1, use the formula:

[0100] G1=g1(X(k+1))-g1(X -i (k+1))

[0101] g1(X(k+1))=std(X(k+1)) / N

[0102] Calculate the reward value G1 and proceed to step E2.

[0103] Step E2: Select agent i, and based on the system environment x(k+1) at the next time step k+1, use the formula:

[0104] G2=g2(X(k+1))–g2(X -i (k+1))

[0105]

[0106] Calculate the reward value G2 and proceed to step E3:

[0107] Step E3: Select agent i, according to the formula:

[0108] G3 = -0.01

[0109] Calculate the reward value G3 and proceed to step E4.

[0110] Step E4: According to the formula:

[0111] r i (k)=G1(X(k+1))+G2(X(k+1))+G3

[0112] Calculate the reward value of agent i at time k. Let r i (k) Save it to the corresponding temporary experience pool D_i_k. Then proceed to step E5.

[0113] Step E5: Save all temporary experience pools D_i_k obtained in the round to the temporary experience pool D_i in this round, and proceed to step F.

[0114] As a preferred embodiment of the present invention, step F includes the following steps:

[0115] Step F1: Determine if x(k+1) meets the stopping condition. If the absolute value of the difference between the sentiment values ​​of any two agents in x(k+1) is less than or equal to 0.01, or if k+1 is greater than the maximum training steps T, then the stopping condition is met, and proceed to F2. Otherwise, proceed to step C1, and let k = k+1.

[0116] Step F2: Save all temporary experience pools D_i from this round into experience pool D, and proceed to step G.

[0117] As a preferred embodiment of the present invention, step G includes the following steps:

[0118] Step G1: Randomly select batch_size agent trajectory information (D_i) from the experience pool without repetition as input to the policy gradient method, initialize loss_all = 0, and proceed to step G2.

[0119] Step G2: Extract the information from the selected agent trajectory information D_i to obtain: the observation list obs = {s} of agent i's trajectory. i (k)|k∈(0,len(D_i))}, the action list of the trajectory of agent i, acts={a i (k)|k∈(0,len(D_i))}, the reward list for the trajectory of agent i is rewards={r i (k)|k∈(0,len(D_i))} where len(D_i) is the length of the trajectory information D_i. Proceed to step G3.

[0120] Step G3: Calculate the decaying reward value reward_sum from the obtained rewards list. Take the reward values ​​from the rewards list in reverse order, initially reward_sum = 0, according to the formula:

[0121] The decaying reward `reward_sum` is calculated by multiplying `reward_sum` by `reward_0.9 + reward_sum'`. The corresponding reward value `reward` in the decaying reward list is then replaced with the obtained `reward_sum` value from each step, resulting in the decaying reward list `reward_rd`. The `reward_rd` list is then standardized using the formula: `reward = [reward – avg(reward_rd)] / std(reward_rd)`, where `avg(reward_rd)` represents the average value of the elements in the decaying reward list `reward_rd`, and `std(reward_rd)` represents the standard deviation of the elements in the decaying reward list `reward_rd`. The process then proceeds to step G4.

[0122] Step G4: Based on the obtained action list obs and decaying reward list reward_rd of agent i, according to the formula: Calculate the current loss value. Proceed to step G5.

[0123] Step G5: Sum the losses obtained in each round according to the formula: loss_all = loss_all' + loss. Determine if all selected agents have been traversed. If yes, proceed to step G6; otherwise, return to G2 and calculate the loss for the next agent. Here, loss_all' represents the loss_all obtained by traversing the previous agent.

[0124] Step G6: Calculate the average value of loss_all according to the formula. Calculate the average value According to the formula: Update the agent's decision-making model, where ζ is the learning rate; proceed to step H. `batch_size` is the maximum number of trajectories selected in one learning cycle.

[0125] The symbols appearing in this method are explained as follows:

[0126] Di_i_k: The agent's exploration experience pool, used to store the agent's trajectory information;

[0127] x(k): The environment of the agent system at time k, its value is the public opinion value corresponding to the agent;

[0128] L(k): The communication topology matrix of the agent at time k;

[0129] min_clu: The minimum number of clusters in the system environment across all rounds;

[0130] M: Maximum number of training rounds;

[0131] m: Current training round number;

[0132] T: The maximum number of steps (maximum moment) in the evolution of the agent's public opinion during each training round;

[0133] k: Current step number (time point);

[0134] S: The number of steps required for the system to reach stability in this round;

[0135] batch_size: The number of agent trajector trajector trajector trajector trajector trajector trajectories learned by the decision model from the experience pool after each round;

[0136] rc: Detection radius of the agent;

[0137] ζ: Learning rate of the agent's decision-making model;

[0138] agent_num: The upper limit of the agent's public opinion value;

[0139] s i (k): The sentiment value of neighboring agents detected by agent i at time k in the current training round;

[0140] a i (k): The action of agent i at time k in the current training round, derived by the decision model;

[0141] r i (k): The real-time reward value obtained by agent i at time k in the current training round;

[0142] θ: Policy parameters of the agent decision-making model;

[0143] batch_size: The maximum number of trajectories selected during a single learning process;

[0144] How the action list is obtained: In a training round, time k ranges from 0, 1, ... to k. At each time step, agent i's action a... i (k) together form the action list obs, rewad_dd; similarly, the corresponding action list is a. i The reward value of (k);

[0145] loss_all: The sum of the loss values ​​generated during the training of the agent's decision-making model. Since multiple trajectory information is used, the average of loss_all is used as the final loss for model training.

[0146] Verification process:

[0147] To verify the effectiveness of the method described in this invention in enhancing the consistency of public opinion in the system, the following verification process was conducted:

[0148] This invention constructs the public opinion evolution analysis method based on multi-agent reinforcement learning, as well as the CNR analysis method, GP analysis method, and HK analysis method, and generates agent public opinion value curves under the condition that the initial public opinion values ​​are uniformly distributed within a certain range; through... Figure 3-6 Analysis shows that the method described in this invention can promote consensus on public opinion values ​​among intelligent agents within the system more quickly, and when the system is stable, the number of clusters formed, that is, the number of opinions in the system, is the smallest, meaning the system has a better degree of consistency.

[0149] The above descriptions only cover a few specific embodiments of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention. The above-mentioned serial numbers are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

Claims

1. A public opinion evolution analysis method based on multi-agent reinforcement learning, characterized in that, include: Initialize the agent decision-making model and experience pool based on the LSTM network, and initialize the agent system environment; Each agent saves the set of neighbors it detects and the set of exploration actions at any given time into a temporary experience pool. The communication topology matrix for any given moment is generated by the actions of all agents at any given moment; and the agent system environment for the next moment is updated based on the communication topology matrix and the agent system environment. Using a pre-set reward method, obtain the reward for each agent at that moment; and put the reward into the corresponding agent's exploration experience pool, and then save the data in the agent's exploration experience pool into the temporary experience pool respectively; Based on a pre-set stopping condition, at least one agent trajectory information is randomly selected from the experience pool as input to the policy gradient method, and the policy gradient output is obtained to update the agent decision model. Set a maximum number of training rounds, explore and train the agent decision-making model to obtain the learned agent decision-making model; The learned agent decision-making model enables the agent to make consistent decisions adaptively based on the agent system environment. The step of randomly selecting at least one agent trajectory information from the experience pool as input to the policy gradient method and obtaining the policy gradient output to update the agent decision model includes: Randomly select batch_size agent trajectory information (D_i) from the experience pool D without repetition as input to the policy gradient method, and initialize loss_all = 0; Extract the information from the agent's trajectory information D_i to obtain: the list of observed values ​​for the agent's trajectory obs = {s i (k)|k∈(0,len(D_i))}, the action list of agent i's trajectory acts={a i (k)|k∈(0,len(D_i))} and the reward list for the trajectory of agent i, reward = {r i (k)|k∈(0,len(D_i))};where len(D_i) is the length of the trajectory information D_i; Calculate the decaying reward value reward_sum from the obtained reward list rewards; extract the reward values ​​reward from the reward list rewards in reverse order, initially reward_sum = 0, according to the formula: reward_sum=reward×0.9+reward_sum ， The decaying reward `reward_sum` is obtained, and the corresponding reward value `reward` in the decaying reward list `reward_sum` is replaced with the decaying reward `reward_sum` obtained in each step, resulting in a decaying reward list `reward_rd`. The obtained decaying reward list `reward_rd` is then standardized. ， This represents the decaying reward_sum obtained in the previous step; Based on the action list obs and the decaying reward list reward_rd of agent i, obtain the current loss value: Determine whether all selected agents have been traversed, and update the agent decision model accordingly; The obtained decaying reward table reward_rd is standardized, including: For each value in the decaying reward table, apply the following formula: reward=[reward–avg(reward_rd)] / std(reward_rd); Where avg(reward_rd) represents the average value of the elements in the decaying reward list reward_rd, and std(reward_rd) represents the standard deviation of the elements in the decaying reward list reward_rd.

2. The public opinion evolution analysis method based on multi-agent reinforcement learning according to claim 1, characterized in that, The experience pool D includes: D = {D_i|i∈[0,N×M]}, where... D_i={D_i_j|j∈[0,T]}, where, D_i_j={s i (k),a i (k),r i (k)}; In Di_i_j, s i (k) represents the set of neighbor sentiment values ​​detected by agent i at time k; a i (k) represents the set of exploratory actions of agent i at time k; r i (k) represents the reward value obtained by agent i at time k.

3. The public opinion evolution analysis method based on multi-agent reinforcement learning according to claim 1, characterized in that, The intelligent agent system environment x(k) includes: k = 0, x(k) = {agent_i|i∈[0,N]}, where agent_i represents the sentiment value of agent i.

4. The public opinion evolution analysis method based on multi-agent reinforcement learning according to claim 1, characterized in that, The phrase "each agent saves its detected set of neighbors and set of exploration actions at any given time to a temporary experience pool" includes: The agent i obtains the neighbor set s according to the detection radius rc i (k), s i (k) = {x j (k) || x i (k) - x j (k) | < rc |, j ∈ [0, N]}, where x i (k) represents the public opinion value of the agent i in the system environment x(k); x j (k) represents the public opinion value of the agent j in the system environment x(k); where, i ≠ j; The intelligent agent i will select the neighbor set s i Combining (k) and its corresponding index, we obtain the input to the agent's decision-making model. After passing through a bidirectional recurrent network and an activation function, we output the action of agent i towards its neighbors at the current time k, thus obtaining the set of exploration actions a. i (k), a i (k)={p j |j∈[0,N]},p i p represents the selection weight of agent i for agent j. i The sum is 1; The neighbor set s i (k) and the set of exploration actions a i (k) Save to the temporary experience pool.

5. The public opinion evolution analysis method based on multi-agent reinforcement learning according to claim 1, characterized in that, The phrase "generating a communication topology matrix at any given moment based on the actions of all agents; and updating the agent system environment for the next moment based on the communication topology matrix and the agent system environment" includes: From temporary experience pool D i Take the neighbor set s from (k) i (k) and the set of exploration actions a i (k), construct the communication topology matrix L(k) at time k, L(k) = [l ij (k)]where l ij (k) represents the communication topology between agents i and j at time k; N i (k) represents the set of neighbor indices N of agent i at time k. i (k)={j||x i (k)-x j (k)| <rc|,j∈[0,N]} Based on the communication topology matrix L(k) and the agent system environment X(k), update the agent system environment at time k+1 according to the following formula: X(k+1)=L(x)X(k).

6. The public opinion evolution analysis method based on multi-agent reinforcement learning according to claim 1, characterized in that, The phrase "using a pre-set reward method to obtain the reward for each agent at that moment; and placing the reward into the corresponding agent's exploration experience pool, and then saving the data in the agent's exploration experience pool to the temporary experience pool" includes: Let G1(·), G2(·), and G3 represent three different objective reward functions; r i (k) represents the reward value of agent i at time k, then: r i (k)=G1(X(k+1))+G2(X(k+1))+G3 Wherein, G1(·): is used to help the agent learn and enhance the consistency of the system, and its formula is as follows: G1=g1(X(k+1))-g1(X -i (k+1)) g1(X(k+1))=std(X(k+1)) / N N represents the number of agents in the system, X -i (k+1) represents the system environment after removing agent i, and std(·) represents the standard deviation operation; G2(·): Used to help the agent learn to improve the connection density of the system, and its formula is as follows: G2=g2(X(k+1))–g2(X -i (k+1)); Where, ∑ i∈V |N i (k+1)| represents the number of connections existing in the system at time k+1; G3: Used to enable agents to learn to reduce the steps required for the system to achieve the following formula: G3＝-0.01； The value of G3 ranges from [0, -0.01×S], where S represents the number of steps required for the system to reach stability in this round. The obtained r i (k) save to the corresponding temporary experience pool D_i_k, and save all the temporary experience pools D_i_k obtained in the round to the corresponding temporary experience pool D_i in this round of exploration.

7. The public opinion evolution analysis method based on multi-agent reinforcement learning according to claim 1, characterized in that, The "pre-set stop conditions" include: In the agent system environment x(k+1) at the next time step, the absolute value of the difference in sentiment values ​​between any two agents is less than or equal to 0.01; or, The value of k+1 at the next moment is greater than the maximum number of training rounds T.

8. The public opinion evolution analysis method based on multi-agent reinforcement learning according to claim 1, characterized in that, "Determine whether all selected agents have been traversed and update the agent decision model," including: The loss value is summed in each round by summing the current loss value: loss_all = loss_all' + loss; where loss_all' represents the loss_all obtained by iterating through the previous agent. After iterating through all selected agents, proceed according to the formula. Get the average value of loss_all And according to the formula: The agent decision model is updated, where ζ is the learning rate; θ is the decision parameter of the agent decision model; and batch_size is the maximum number of trajectories selected in one learning process.

Images