Adjacent avoidance event coping strategy generation method and device based on reinforcement learning, and medium

Abstract

The invention discloses a reinforcement learning-based neighborhood event coping strategy generation method and device and a medium, and the method comprises the steps: setting parameters according to a scene of a neighborhood event, and constructing a reinforcement learning model; and obtaining an initial state, carrying out reinforcement learning on the reinforcement learning model by adopting a Q-Learning algorithm, and iteratively determining an optimal strategy by taking reward maximization as a target. The device comprises a setting module and a simulation module. The medium comprises a computer program in which the reinforcement learning-based neighborhood event coping strategy generation method is stored. By using the method, the problem that a traditional risk crisis analysis method lacks universality and dynamic uncertainty is solved, and the unstable risk caused by the neighbor avoidance event can be effectively solved by outputting the optimal behavior strategy. The method can be widely applied to the field of strategy self-generation.

Description

Technical Field

[0001] The present invention relates to the field of strategy self-generation, and in particular to a method, device and medium for generating a NIMBY event response strategy based on reinforcement learning. Background Art

[0002] With the advancement of urbanization, NIMBY incidents emerge in an endless stream, and NIMBY incidents cause conflicts between public interests and economic interests. Strategic behaviors in the face of NIMBY incidents have greatly affected public perception and led to changes in public behavior. How to evaluate the impact of local-public interaction, changes in the external institutional environment and other conditions in the process of NIMBY incident governance, and adjust one's own behavior in a timely manner to balance the relationship between economic interests and social stability and control major social stability risks is an issue to be solved.

[0003] At present, most of the research on NIMBY risk adopts case study, mathematical modeling analysis, fuzzy comprehensive evaluation method and grey correlation analysis method, etc., which focuses on qualitative case analysis and summary. There are two difficulties in analysis and prediction: one is the multiple possibilities of system development and evolution, and the other is the occurrence of unpredictable events. These methods generally lack consideration of universality and dynamic uncertainty. Summary of the invention

[0004] In view of this, in order to solve the problems of lack of universality and dynamic uncertainty in existing risk crisis analysis methods, the present invention proposes a method for generating a NIMBY event response strategy based on reinforcement learning, and outputting an optimal behavior strategy can effectively resolve the unstable risks caused by NIMBY events. The method comprises the following steps:

[0005] According to the scenario of the NIMBY event, set parameters and build a reinforcement learning model;

[0006] The initial state is obtained, and the reinforcement learning model is reinforced by using the Q-Learning algorithm, with the maximum reward as the goal, and the optimal strategy is iteratively determined.

[0007] In some embodiments, the step of setting parameters and constructing a reinforcement learning model according to the scenario of the NIMBY event specifically includes:

[0008] The executor of the NIMBY event is set as the subject of the reinforcement learning model;

[0009] Setting the public as an environment variable of the reinforcement learning model;

[0010] Setting the public state as a state variable of the reinforcement learning model;

[0011] Setting the behavior strategy preset by the subject variable as the action variable of the reinforcement learning model;

[0012] The preset reward function is used as the reward variable of the reinforcement learning model.

[0013] In some embodiments, the step of obtaining the initial state, using the Q-Learning algorithm to perform reinforcement learning on the reinforcement learning model, taking the maximum reward as the goal, and iteratively determining the optimal strategy specifically includes:

[0014] Initialize each Q value in the Q-Learning algorithm;

[0015] In the initial state, the subject selects the action that maximizes the Q value from the Q table at a first preset exploration rate, and randomly explores other actions at a second preset exploration rate to obtain an action sample sequence, and updates the Q value according to a preset iterative update model;

[0016] Determine whether the subject has reached the preset target state;

[0017] If not, greedy exploration search is used again to obtain an updated action sample sequence, the Q value is updated according to the iterative update model, and whether the subject has reached the target state is re-judged until the subject reaches the preset target state;

[0018] If so, the Q-Learning algorithm returns to the initial state, re-explores, obtains an updated action sample sequence, and updates the Q value according to the iterative update model;

[0019] Determine whether the preset discount reward function converges;

[0020] If not, re-adopt greedy exploration search to obtain an updated action sample sequence, update the Q value according to the iterative update model, and re-judge whether the discounted reward function converges until the discounted reward function converges;

[0021] If so, the optimal strategy is output according to the preset optimal strategy output model.

[0022] In some embodiments, the action sample sequence is t ,a t ,s t+1 ,r t+1 >

[0023] The iterative update model is:

[0024]

[0025] Among them, s t is the state at time t, a t is the action taken at time t, s t+1 is the state at time t+1, r t+1 ​is the reward of the environment feedback at time t+1, α is the learning rate, γ is the discount factor, Q(s t ,a t ) is state s t Take action a t The discounted reward value obtained, Q(s t+1 ,a) is state s t+1 The discounted reward value obtained by taking action a.

[0026] In some embodiments, the discounted reward function is expressed as follows:

[0027]

[0028] Among them, Q*(s,a) refers to the sum of the optimal discounted reward values ​​obtained by taking action a in state s, r(s,a) is the reward obtained by taking action a in state s, and T(s,a,s t ) represents the sum of all states and actions in a period of time starting from time t, Q(s t ,a t ) is state s t Next, take action a t The sum of the discount reward values ​​obtained.

[0029] In some embodiments, the optimal strategy output model is expressed as follows:

[0030]

[0031]

[0032] Among them, π * is the optimal strategy for output, A is the action variable space, Q π (s t ,a t ) is the state st in which action a is taken t The state-action value, γ t is the discount factor at time t.

[0033] In some embodiments, the reward function model is expressed as follows:

[0034] R(s t ,a t )=b1×R Development +b2×R Stability

[0035] Among them, R(s t ,a t ) is the state st in which action a is taken t Rewards received, R Development represents the reward of economic development factors, RStability represents the reward of social stability factors, b1 and b2 represent the weights of economic development factors and social stability factors respectively.

[0036] The present invention also proposes a device for generating a NIMBY event response strategy based on reinforcement learning, the device comprising:

[0037] Setting module, setting parameters and building reinforcement learning model according to the scenario of NIMBY event;

[0038] The simulation module obtains the initial state, uses the Q-Learning algorithm to perform reinforcement learning on the reinforcement learning model, takes the maximum reward as the goal, and iteratively determines the optimal strategy.

[0039] The present invention also proposes a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the reinforcement learning-based NIMBY event response strategy generation method as described above.

[0040] Based on the above scheme, the present invention provides a method, device and medium for generating a NIMBY event response strategy based on reinforcement learning. According to the NIMBY event scenario, a reinforcement learning model suitable for the scenario is built. In the reinforcement learning process, the action variables and state variables are transformed according to the reward variables, and the optimal strategy with the maximum reward is iteratively determined. The problem of the lack of universality and dynamic uncertainty of the traditional risk crisis analysis method is solved, and the output of the optimal behavior strategy can effectively resolve the unstable risks caused by NIMBY events. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the steps of a method for generating a NIMBY event response strategy based on reinforcement learning of the present invention;

[0042] Figure 2 is a schematic diagram of a dynamic process of reinforcement learning provided by an embodiment of the present invention;

[0043] Figure 3 It is a structural block diagram of a device for generating a NIMBY event response strategy based on reinforcement learning according to the present invention. DETAILED DESCRIPTION

[0044] NIMBY events can be specifically manifested as:

[0045] NIMBY incidents caused by the site selection of a waste treatment plant. In urban planning, the government proposed to build a waste treatment plant near a community. This plan aroused strong opposition from residents, who were worried that the waste treatment plant would bring environmental pollution and health risks. Variable examples: risk assessment value, the predicted air quality change value in the environmental impact assessment; public satisfaction, the attitude score of residents towards the project collected through questionnaires; decision-making efficiency: the time it takes for local governments to make a final decision from the proposal.

[0046] NIMBY time caused by the construction of high-voltage transmission lines. A power company planned to build a high-voltage transmission line near the community. Community residents were worried about the impact of high-voltage line radiation on their health and launched protests. Variable examples: electromagnetic radiation intensity, scientifically measured electromagnetic radiation values ​​around high-voltage lines; resident health risk index, health risk estimates based on existing medical research; public reaction intensity, recorded number, scale and duration of protests.

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0049] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0050] Unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0051] In the description of the embodiments of the present application, "plurality" means two or more than two. The following terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0052] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0053] Reference Figure 1 , is a flow chart of an optional example of a method for generating a NIMBY event response strategy based on reinforcement learning proposed in the present invention. The method can be applied to a computer device. The strategy generation method proposed in this embodiment may include but is not limited to the following steps:

[0054] Step S1: according to the scenario of the NIMBY event, set parameters and build a reinforcement learning model;

[0055] Step S2: obtaining an initial state, using a Q-Learning algorithm to perform reinforcement learning on the reinforcement learning model, with the maximum reward as the goal, and iteratively determining the optimal strategy.

[0056] In some feasible embodiments, the step S1 specifically includes:

[0057] The executor of the NIMBY event is set as the subject of the reinforcement learning model;

[0058] Setting the public as an environment variable of the reinforcement learning model;

[0059] Setting the public state as a state variable of the reinforcement learning model;

[0060] Setting the behavior strategy preset by the subject variable as the action variable of the reinforcement learning model;

[0061] The preset reward function is used as the reward variable of the reinforcement learning model.

[0062] Specifically, the executor is the subject of emergency response to social risks, and the executor's behavioral decisions will affect the severity and evolution direction of social risks. In the reinforcement learning model, the main agent obtains the maximum cumulative reward through continuous trial and error interaction with the environment. Based on this principle, a model is established to simulate the impact of the executor's behavioral strategy on the public's behavioral changes, and the optimal executor's behavioral tendency is screened through the feedback reward value, providing a scientific strategy reference for resolving NIMBY risks. Therefore, this paper sets the main agent variable as the NIMBY event executor.

[0063] The executor of the NIMBY event is the executor of the social risk in the NIMBY event, and his behavioral strategy plays an important role in guiding the public and resolving social risks. In the reinforcement learning of NIMBY risk evolution, on the one hand, the subject agent will observe the public's behavioral choices to determine the next decision direction; on the other hand, the public will also change their own behavior state according to the decision of the subject agent. Here, the environment is set to the public and as an environmental factor, it has an important impact on the subject's decision feedback.

[0064] The state of the public directly affects the next behavior choice of the NIMBY event executor. In this paper, the state variable is set to S. The behavior of the NIMBY event executor will weaken or enhance the public's risk perception. The public determines the initial state based on their own risk perception. Assuming that the probability of the public being in each initial state is the same, the state space at time t is set to S = {rationality, stress, wait and see, exit} in reinforcement learning. Through multiple trial-and-error interactions between the main agent and the environment, the model expects the public's target state to be "exit".

[0065] NIMBY incidents can not only alleviate the society's growing demand for public services, but also promote the development of the local economy. In the process of NIMBY incident governance, the NIMBY incident implementers are influenced by the interaction with the public, changes in the external institutional environment and other conditions, and will adjust their behavior in a timely manner to balance the relationship between economic interests and social stability.

[0066] The action variable is the action taken by the main agent when observing the environment, that is, the behavioral strategy of the NIMBY event executor. The behavioral strategy mainly refers to the governance method of the NIMBY event executor on the NIMBY event, including supportive, symbolic, compromising, and passive. The action logic of each behavioral strategy is shown in Table 1.

[0067] Table 1 Behavior logic of action variables

[0068]

[0069] Let the action variable be A, and the possible behavior space that the NIMBY event executor may choose is A = {a1, a2, a3, a4}. a1 is a supportive behavior strategy, which is mainly manifested in two aspects. First, at the project decision-making stage, the NIMBY event implementers allow the public to participate in the project site selection and review, meet the legitimate interests of residents through communication and consultation, and provide reasonable economic compensation. Second, at the conflict stage, the implementers resolve crisis events through interactive consultation. a2 is a symbolic behavior strategy. The NIMBY event implementers have the decision-making power of draft formulation, project site selection, etc., occasionally invite experts and the media for symbolic consultation. If a conflict is caused after the project is made public or started, the implementers will symbolically carry out policy communication and implementer response under the pressure of maintaining stability. a3 is a compromising behavior strategy. If citizens oppose the public project, the implementers will make slogan-like and compromising promises (such as "We will not start the project if the masses do not agree") to achieve the purpose of calming down the matter because of fear and inaction. In the end, the project is terminated or delayed. a4 is a passive behavior strategy. In passive behavior, the implementers often hinder citizens from participating in decision-making, deal with citizens' demands coldly, and the public will overact under the lack of a sense of security and the appeal of justice. The implementers deal with conflicts by means of coercive suppression.

[0070] The reward function plays a guiding role in the reinforcement learning training process. In this paper, the reward variable is set to R, and the feedback R value is used to evaluate which behavioral tendencies of the executors of the NIMBY event are conducive to the public's exit from risky behaviors.

[0071] According to the initial state obtained, the Q-Learning algorithm is used to perform reinforcement learning on the reinforcement learning model. When the main agent selects different behavior strategies, the public changes states according to the main agent behavior and rewards, and the number of the public in different states is recorded. According to the reward variable, the action variable and the state variable are changed, and the optimal strategy with the maximum reward is determined iteratively.

[0072] Reinforcement learning is inspired by the process of continuous trial and failure experienced by humans and animals in the learning process. The learning process is that the agent interacts with the environment and continuously adjusts through trial and error, and obtains the maximum cumulative reward from environmental feedback to obtain the optimal strategy.

[0073] Markov decision process is the core of reinforcement learning and its essential content. Its four basic elements include state S (State, s t ∈S), action A (Action, a t ∈A), state transition probability P(a t |s t ) and reward return R.

[0074] See also Figure 2, is a schematic diagram of the dynamic process of reinforcement learning provided by an embodiment of the present invention. It is a dynamic process of the main agent and the environment. At time t (t∈T), the main agent obtains the state S by observing the environment. t , in the state, the agent takes action according to the strategy π and takes action with probability P(s t+1 |s t , a t )Transfer to the next state s t+1 , the agent also receives a reward r(s) from the environment t |a t ).

[0075] In the iterative process, the state S0, state transition probability P0, and action a0 at time t0 are initially reinforced, and the state S1, state transition probability P1, action a1, and reward report r1 at time t1 are obtained; the state S1, state transition probability P2, action a2, and reward report r2 at time t2; the state S3, state transition probability P3, action a3, and reward report r3 at time t3 are obtained. The goal of reinforcement learning is to maximize the future accumulated rewards through strategy adjustment, and it mainly uses the estimated value function to evaluate whether a strategy is optimal.

[0076] This application proposes an agent-based in-depth enhanced evaluation model for the response strategy of NIMBY event implementers, which solves the problems of the traditional risk crisis analysis method lacking universality and consideration of dynamic uncertainty.

[0077] In some feasible embodiments, the step S2 specifically includes:

[0078] Initializing each Q value of the Q-Learning algorithm;

[0079] In the initial state, the subject uses greedy exploration to select the action that maximizes the Q value from the Q table at a preset exploration rate of 1-ε, and randomly explores other actions at an exploration rate of ε to obtain an action sample sequence, and updates the Q value according to the preset iterative update model;

[0080] Determine whether the subject has reached the preset target state;

[0081] If not, greedy exploration search is used again to obtain an updated action sample sequence, the Q value is updated according to the iterative update model, and whether the subject has reached the target state is re-judged until the judgment result is yes;

[0082] If yes, the Q-Learning algorithm returns to the initial state, and re-adopts greedy exploration search to obtain an updated action sample sequence, and updates the Q value according to the iterative update model;

[0083] Determine whether the preset discount reward function converges;

[0084] If not, re-adopt greedy exploration search to obtain an updated action sample sequence, update the Q value according to the iterative update model, and re-judge whether the discounted reward function converges until the judgment result is yes;

[0085] If so, the optimal strategy is output according to the preset optimal strategy output model.

[0086] Among them, the discounted reward is calculated based on the reward function, which can be simply understood as a reward function that introduces a discount factor. The discount factor controls the influence of future rewards on current decisions. The closer the discount factor is to 1, the more the agent values ​​the return; the closer the discounted reward is to 0, the more the agent values ​​short-term returns.

[0087] Using discounted rewards for training in the reinforcement learning process is a supplement and improvement to the reward function. By introducing a discount factor, we can better handle infinite time steps, reflect environmental uncertainty, and improve learning efficiency. It can help reinforcement learning agents consider long-term goals while also focusing on immediate benefits, thereby making more informed decisions.

[0088] When this embodiment is specifically implemented, each Q value of the Q-Learning algorithm is first initialized;

[0089] The subject is in state s t When using greedy exploration (i.e., ε-greedy, assuming ε is the exploration rate), select the action a that maximizes Q in the Q-table with a probability of 1-ε t , randomly explore other actions with probability ε to obtain a sample sequence;

[0090] Then update the Q value according to the preset iterative update model;

[0091] Determine whether the subject has reached the preset target state;

[0092] If not, greedy exploration search is used again to obtain an updated action sample sequence, the Q value is updated according to the iterative update model, and whether the subject has reached the target state is re-judged until the subject reaches the target state.

[0093] If yes, it means the iteration process is over. The algorithm returns to the initial state and starts a new round of iteration, and greedy exploration is used again to obtain an updated action sample sequence, and the Q value is updated according to the iterative update model.

[0094] Determine whether the preset discount reward function converges;

[0095] If not, re-adopt greedy exploration search to obtain an updated action sample sequence, update the Q value according to the iterative update model, and re-judge whether the discounted reward function converges until the judgment result is yes;

[0096] Q-Learning can achieve the convergence of the discounted reward function through greedy exploration, and then the optimal strategy output model outputs the optimal strategy.

[0097] Real risk is the starting point of the evolution of NIMBY risk. The transformation of real risk into perceived risk is the process of risk spreading from the objective level to the subjective level. This evolutionary stage is not only a process of physical, environmental, and economic damage, but also a process in which individuals or groups interpret risks in a new way based on known information and jointly shape new perceptions. Risk information, media exposure, group identity, and trust in executors can act as negative factors on public risk perception. Therefore, the evolution of real risk into perceived risk includes two parts: individual perception and risk amplification.

[0098] Part I: Transformation from perceived risk to social stability risk. In this stage, based on the actual damage of objective risks and the level of risk perception, the public expects to communicate effectively with executors, enterprises and other entities to eliminate individual cognitive biases about risks. Risk cognitive bias will greatly affect the individual's emotional state and thus affect the individual's attitude and behavior. Therefore, smooth risk communication will reduce the possibility of risk escalation, while failure of risk communication will easily arouse public distrust and irrational behavior, causing perceived risk to transform into social stability risk.

[0099] The characteristic of NIMBY social risk is "artificial construction", that is, the formation and transformation of public behavior will affect the generation of social stability risks and the level of risk. The public's risk perception stems from the objective risks of NIMBY projects. Under the triggering of multiple factors, the perceived risks are accumulated and amplified. However, this perception will not stay at the public's psychological level for a long time but will be externalized into the public's behavioral choices. The higher the public's group perception, the higher the intensity of the conflict, and the greater the possibility of the outbreak of NIMBY incidents. The public's behavior in NIMBY incidents is defined as three categories, namely rational behavior, stress behavior, and wait-and-see. Rational behavior refers to the public's behavior being in a restrained state. While collecting and transmitting project-related information, they will express their opinions and demands in a relatively rational way. Stress behavior refers to the public's protest behavior. When the public's demands cannot be met and the risk perception exceeds the threshold, they will transform the accumulated dissatisfaction into obstructive actions to obtain interest satisfaction and risk resolution. Wait-and-see means that the public has a neutral attitude towards NIMBY projects, has no obvious risk perception and interest demands, and has not yet taken relevant participation behaviors.

[0100] Risk perception is an important factor affecting individual behavior. NIMBY event executors are the executors of social risks in NIMBY events, and have the ability to prevent risks and deal with emergencies. The behavior of NIMBY event executors plays an important role in guiding the public and resolving social risks. The actual risks of NIMBY projects become the public's risk perception after fermentation by adverse factors. The attitude and approach of NIMBY event executors will enhance or weaken risk perception and directly affect public behavior.

[0101] Based on the behavioral strategies of the NIMBY event executors and their own risk perception, the public will generate different initial behaviors and there is a possibility of behavioral change. Therefore, this section will analyze the public's behavioral change path in detail.

[0102] When the initial behavior is on the sidelines, the behavioral paths include: wait-and-see → rational behavior, wait-and-see → stress behavior, wait-and-see → wait-and-see, wait-and-see → exit.

[0103] The public who are in a "wait-and-see" state in the early stage often have a low risk perception, and their behavior will be affected by the next behavioral strategy of the NIMBY event implementers. The active guidance and information disclosure of the NIMBY event implementers will make the public remain "wait-and-see" or choose to "exit"; the information transmission among the public and the passive response of the NIMBY event implementers will increase the risk perception of the "wait-and-see" and thus cause the behavior to change to "rational behavior" or "stress behavior."

[0104] When the initial behavior is rational, the behavior paths include: rational behavior → wait and see, rational behavior → stress behavior, rational behavior → exit. The public whose initial behavior is in the "rational" state has generated risk perception and interest demands, and the actions and feedback of the NIMBY event implementers affect the public's behavioral transformation. When the NIMBY event implementers actively respond to public demands and actively invite the public to participate in decision-making, the "rational actors" will have a better understanding of the NIMBY project due to the positive actions of the NIMBY event implementers, realize the participation needs and reduce the perceived risk, and choose to become "wait and see" or "exit"; when the NIMBY event implementers respond negatively to the public's opinions and refuse the public's demands for participation, the "rational actors" will have emotions of misunderstanding and disapproval, which will increase the risk perception and the public will enter the "stress behavior" state.

[0105] When the initial behavior is stress behavior, the behavior path includes: stress behavior → stress behavior, stress behavior → exit. The public whose initial behavior is in the "stress" state has already exceeded the acceptable range of risk perception of the NIMBY project, and has strong dissatisfaction with the NIMBY event implementers. At this time, the change of public behavior is related to the response attitude and measures of the NIMBY event implementers. If the NIMBY event implementers can promise to relocate or increase compensation, appease the emotions of the "stressed behavior", and compensate the public for the risk loss, then the public will agree to solve the problem peacefully and choose to "exit"; if the NIMBY event implementers use compulsory or repressive measures to deal with the public's stress behavior and do not actively guide the public's emotions, then the public will continue to maintain stress behavior until the problem is resolved.

[0106] In the early stages of the evolution of NIMBY risk and the transformation of public behavior, the public's perception of the actual risks of NIMBY facilities is very weak, and their psychological concerns about risks are implicit. However, the public still has a certain degree of concern about the progress of the project and the level of risk, so they will maintain a neutral attitude, that is, they will wait and see.

[0107] In the middle stage of the evolution of NIMBY risk and the transformation of public behavior, the "signal value" released by the project gradually increases over time, but it is only a change in the "quantity" of real risk, and external adverse factors have not yet produced a more obvious amplification effect. At this stage, the risk perception of some people gradually changes from a low level to a moderate level, and begins to have behavioral representation and adopt rational behavior.

[0108] In the later stage of the evolution of NIMBY risk and the transformation of public behavior, negative factors gradually intervene in the public's risk perception process. At this time, the public's risk perception is at a medium-low level, and the executors have not dealt with and responded in time. Therefore, the negative factors that have a major impact on public perception at this stage are media exposure and group identity. The untrue information differentiated by the media spreads among social individuals through various channels, and the subjective views of individuals influence each other to form a group effect. The "rationality" in NIMBY is not stable and lasting. The increase in the number of participants or the expansion of the scope of the event will trigger the transformation from "rationality" to "irrationality", and the public will change from ordinary people to risk groups. With the continuous influence of media exposure, group identity and the increase in the number of opponents, the public's risk perception level is close to the critical point. In the case of failure of risk communication mechanism and absence of emergency management, the rational boundary is broken, the public's risk perception is out of control and manifests as stressful behavior, forming a social stability risk.

[0109] In summary, the evolution of NIMBY risk is related to the formation of public behavior, and the public's behavior status can be used to characterize the stage of risk evolution. In addition, the transformation of public behavior is closely related to the degree of individual risk perception and the risk governance ability of the executors: first, the higher the risk perception of residents, the stronger their intention to participate in protests. Specifically, the high, medium and low levels of public risk perception will correspond to the selection of stress behavior, rational behavior and wait-and-see behavior respectively; second, in the entire life cycle of the NIMBY project, the executors of NIMBY events bear the responsibility of risk emergency response and maintaining social order. The risk governance methods of the executors indirectly determine the direction of public behavior transformation by affecting the risk perception of residents.

[0110] According to the model principle constructed above, the simulation environment is built using Python 3 programming, and the learning rate, exploration rate, and discount factor are initialized. In addition, assuming that the total number of the public (N) in the model is 900, and the number of onlookers, rationalists, and stressors in the initial state is 300, the model parameter settings are shown in Table 2.

[0111] Table 2 Model initial parameter settings

[0112] parameter Value Learning rate (α) 0.01 Exploration rate (ε) 0.6 Discount factor (γ) 0.9 Total number of public (N) 900 Number of spectators 300 Number of rational people 300 Number of stressors 300

[0113] By adjusting the number of iterations, the model simulates that when the main agent chooses different behavioral strategies, the public changes their states according to their behaviors and rewards, and records the number of people in different states. This is used to explain the impact of the behavior of the executors of the NIMBY event on the public's behavioral changes and the evolution of NIMBY risks. The output of the optimal behavioral strategy can effectively resolve the instability risks caused by NIMBY events.

[0114] In some feasible embodiments, the action sample sequence is: t ,a t ,s t+1 ,r t+1 >

[0115] The iterative update model is:

[0116]

[0117] Among them, s t is the state at time t, a t is the action taken at time t, s t+1 is the state at time t+1, r t+1 is the reward of the environment feedback at time t+1, α∈[0,1] is the learning rate. The larger the learning rate, the faster the information is updated. γ∈[0,1] is the discount factor. The larger the discount factor, the more the agent considers long-term benefits. Q(s t ,a t ) is state s t Take action a​t The discounted reward value obtained, Q(s t+1 ,a) is state s t+1 The discounted reward value obtained by taking action a.

[0118] The Q-Learning algorithm is a model-free, value-function-based reinforcement learning algorithm that seeks a strategy that maximizes the expected discounted reward by optimizing an iterative state-action value function.

[0119] In some feasible embodiments, the discounted reward function is expressed as follows:

[0120]

[0121] Among them, Q*(s,a) refers to the sum of the optimal discounted reward values ​​obtained by taking action a in state s, r(s,a) is the reward obtained by taking action a in state s, and T(s,a,s t ) represents the sum of all states and actions in a period of time starting from time t, Q(s t ,a t ) is state s t Next, take action a t The sum of the discounted reward values ​​obtained. In each iteration of Q-Learning, the Agent needs to examine each behavior and build a Q-table to store the Q value based on the empirical data of the state-action pair and the Bellman equation.

[0122] In this embodiment, a strategy that maximizes the expected discounted reward is found by optimizing an iterative state-action value function.

[0123] In some feasible embodiments, the optimal strategy output model is expressed as follows:

[0124]

[0125] Among them, π * is the optimal strategy for output, A is the action variable space, Q π (s t ,a t ) is state s t Next, take action a t The state-action value, γ t is the discount factor at time t.

[0126] The above formula expresses the expected E-return expression of the state value function V under the strategy ∏ when the initial state s0=s at time t.

[0127] In this embodiment, the goal of reinforcement learning is to maximize future accumulated rewards through strategy adjustment, and the estimated value function is mainly used to evaluate whether a strategy π is optimal.

[0128] In some feasible embodiments, the reward function model is expressed as follows:

[0129] R(s t , a t )=b1×R Development +b2×R Stability

[0130] Among them, R(s t ,a t ) is the state st in which action a is taken t Rewards received, R Development represents the reward of economic development factors, R Stability represents the reward of social stability factors, b1 and b2 represent the weights of economic development factors and social stability factors respectively.

[0131] In this embodiment, b1+b2=1 is set.

[0132] Economic development and social stability are two key elements in managing NIMBY incidents, and the social risk mitigation capabilities of NIMBY incident executors are mainly reflected in the completion of these two elements.

[0133] Economic development factors are mainly determined by the governance results of NIMBY conflicts. The successful construction of NIMBY facilities not only meets the needs of local economic development of the executor, but also indicates that the risk of the NIMBY project is relatively low or the original NIMBY risk has been successfully resolved. This paper divides the governance results of NIMBY conflicts into three types: termination or delay, relocation construction, and normal completion. D evelopment represents the reward function of economic development factors. Reward R D Evelopment is set to a positive value, indicating encouragement for the behavioral strategy of the NIMBY event executors, indicating that the behavioral tendency of the NIMBY event executors can effectively resolve the public's demands and guide the public to move to a risk-free "exit" behavior state, while achieving the goal of local economic development; R D When evelopment is set to a negative value, it means that the NIMBY risk has the possibility of transforming into a social risk or the public has engaged in risky behavior. Although the NIMBY event executors eventually resolved the risk, their previous behavioral strategies were not encouraged.

[0134]

[0135] Social stability is an important manifestation of the risk resolution ability of the NIMBY event implementers, which is mainly affected by the number of people involved in mass incidents. Generally speaking, without considering other factors, the more people involved, the higher the risk of the NIMBY event, and the greater the impact on the stability of social order. In the early stages of social risk evolution, if the NIMBY event implementers can take reasonable measures to meet public demands, strengthen risk communication, and guide public emotions and behaviors, it will be helpful to control the escalation of risks and avoid the outbreak of major mass incidents. The number of participants in NIMBY events is divided into four situations, and R is used to calculate the number of participants in the NIMBY event. S tability represents the reward function of social stability factors,

[0136] The reward function model in the model is as follows: R(s t ,a t )=b1×RD evelopment +b2×R Stability .

[0137] R(s t ,a t ) is the state st in which action a is taken t The rewards obtained, b1 and b2 represent the weights of economic development factors and social stability factors respectively, and b1+b2=1, the weights are randomly determined, where k i ∈[1, 2] is a random positive integer.

[0138] like Figure 3 As shown, a device for generating a NIMBY event response strategy based on reinforcement learning comprises:

[0139] Setting module, setting parameters and building reinforcement learning model according to the scenario of NIMBY event;

[0140] The simulation module obtains the initial state, uses the Q-Learning algorithm to perform reinforcement learning on the reinforcement learning model, takes the maximum reward as the goal, and iteratively determines the optimal strategy.

[0141] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] A computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the reinforcement learning-based NIMBY event response strategy generation method as described above.

[0143] The contents of the above method embodiments are all applicable to the present medium embodiments. The functions specifically implemented by the present medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0144] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for generating a response strategy for NIMBY incidents based on reinforcement learning, characterized in that: The following steps are involved: According to the scenario of the NIMBY event, set parameters and build a reinforcement learning model; The initial state is obtained, and the reinforcement learning model is reinforced by using the Q-Learning algorithm, with the maximum reward as the goal, and the optimal strategy is iteratively determined.

2. The method for generating a NIMBY event response strategy based on reinforcement learning according to claim 1, characterized in that: The step of setting parameters and building a reinforcement learning model according to the scenario of the NIMBY event specifically includes: The executor of the NIMBY event is set as the subject of the reinforcement learning model; Setting the public as an environment variable of the reinforcement learning model; Setting the public state as a state variable of the reinforcement learning model; Setting the behavior strategy preset by the subject variable as the action variable of the reinforcement learning model; The preset reward function is used as the reward variable of the reinforcement learning model.

3. The method for generating a NIMBY event response strategy based on reinforcement learning according to claim 2, characterized in that: The step of obtaining the initial state, using the Q-Learning algorithm to perform reinforcement learning on the reinforcement learning model, taking the maximum reward as the goal, and iteratively determining the optimal strategy specifically includes: Initialize each Q value in the Q-Learning algorithm; In the initial state, the subject selects the action that maximizes the Q value from the Q table at a first preset exploration rate, and randomly explores other actions at a second preset exploration rate to obtain an action sample sequence, and updates the Q value according to a preset iterative update model; Determine whether the subject has reached the preset target state; If not, greedy exploration search is used again to obtain an updated action sample sequence, the Q value is updated according to the iterative update model, and whether the subject has reached the target state is re-judged until the subject reaches the preset target state; If so, the Q-Learning algorithm returns to the initial state, re-explores, obtains an updated action sample sequence, and updates the Q value according to the iterative update model; Determine whether the preset discount reward function converges; If not, re-adopt greedy exploration search to obtain an updated action sample sequence, update the Q value according to the iterative update model, and re-judge whether the discounted reward function converges until the discounted reward function converges; If so, the optimal strategy is output according to the preset optimal strategy output model.

4. The method for generating a NIMBY incident response strategy based on reinforcement learning according to claim 3, characterized in that: The action sample sequence is t ,a t ,s t+1 ,r t+1 >​ The iterative update model is: Among them, s t is the state at time t, a t is the action taken at time t, s t+1 is the state at time t+1, r t+1 is the reward of the environment feedback at time t+1, α is the learning rate, γ is the discount factor, Q(s t ,a t ) is state s t Take action a t The discounted reward value obtained, Q(s t+1 ,a) is state s t+1 The discounted reward value obtained by taking action a.

5. The method for generating a NIMBY event response strategy based on reinforcement learning according to claim 4, characterized in that: The discounted reward function is expressed as follows: Among them, Q*(s,a) refers to the sum of the optimal discounted reward values ​​obtained by taking action a in state s, r(s,a) is the reward obtained by taking action a in state s, and T(s,a,s t ) represents the sum of all states and actions in a period of time starting from time t, Q(s t ,a t ) is state s t Next, take action a t The sum of the discount reward values ​​obtained.

6. The method for generating a NIMBY event response strategy based on reinforcement learning according to claim 5, characterized in that: The optimal strategy output model is expressed as follows: Among them, π * is the optimal strategy for output, A is the action variable space, Q π (s t ,a t ) is state s t Next, take action a t The state-action value, γ t is the discount factor at time t, and E represents the expectation.

7. The method for generating a NIMBY event response strategy based on reinforcement learning according to claim 6, characterized in that: The reward function model is expressed as follows: R(s t ,a t )=b1×R Development +b2×R Stability Among them, R(s t ,a t ) is state s t Next, take action a t Rewards received, R Development represents the reward of economic development factors, R Stability represents the reward of social stability factors, b1 and b2 represent the weights of economic development factors and social stability factors respectively.

8. A device for generating a strategy for dealing with NIMBY events based on reinforcement learning, characterized in that: include: Setting module, setting parameters and building reinforcement learning model according to the scenario of NIMBY event; The simulation module obtains the initial state, uses the Q-Learning algorithm to perform reinforcement learning on the reinforcement learning model, takes the maximum reward as the goal, and iteratively determines the optimal strategy.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method for generating a NIMBY event response strategy based on reinforcement learning as described in any one of claims 1 to 7.

Images