A reinforcement learning-based optimization method for infectious disease intervention measures

Abstract

The present invention discloses a method for optimizing infectious disease intervention measures based on reinforcement learning, comprising: adjusting parameters of an improved Covasim environment; training an intelligent agent: (1) the intelligent agent obtains state information s from the improved Covasim environment; t When the number of infected people exceeds the preset threshold, the intervention intensity value a is output according to the status information. t , according to a t Act on the environment and get the corresponding reward information r t , get the next state information s t+1 , the data group (s t , a t , r t , s t+1 ) is stored in the experience replay area; (2) when the data stored in the experience replay area reaches the preset data threshold, a Mini‑batch is randomly sampled to update the network parameters of the agent; (3) steps (1)‑(2) are repeated until the preset maximum training time is reached to obtain the trained agent; the trained agent outputs corresponding intervention measures according to the current epidemic status of the infectious disease.

Description

Technical Field

[0001] The present invention relates to the field of public health technology, and in particular to a method for optimizing infectious disease intervention measures based on reinforcement learning. Background Art

[0002] Infectious diseases have long posed a challenge to global public health. They can themselves be seen as a complex system, which includes many factors such as virus mutation and transmission paths, population susceptibility and mobility, allocation and scheduling of medical resources, and the mutual influence of social and economic activities. These factors are intertwined and interact with each other, showing a high degree of complexity, nonlinearity and dynamism.

[0003] Existing research on infectious disease control measures primarily focuses on non-pharmaceutical interventions (NPIs) and pharmacological interventions (PIs). Regarding non-pharmaceutical interventions, some scholars have studied the impact of different lockdown strategies, others have analyzed the optimal timing of intervention, and others have explored the importance of testing, contact tracing, and isolation, as well as the impact of isolation strategies on disease transmission and economic activity. Regarding pharmacological interventions, some studies have proposed new models that combine evolutionary strategies to optimize epidemic control. However, most of these studies are based on traditional optimization methods rather than reinforcement learning techniques. Traditional methods are unable to dynamically adjust strategies based on real-time epidemic data (such as new cases), exhibit significant intervention lags, and often focus on a single objective (such as minimizing the number of infections). This makes it difficult to balance multiple needs (such as economic costs, social impact, and the burden on medical resources). They also lack flexibility in optimizing the combination of multiple intervention measures (such as lockdowns) and often rely on empirical combinations rather than data-driven optimal solutions. Summary of the Invention

[0004] In view of this, the present invention provides an infectious disease intervention measure optimization method based on reinforcement learning, which is used to at least solve the problems in the existing technology of being unable to dynamically adjust according to real-time data and having a single goal.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A reinforcement learning-based method for optimizing infectious disease intervention measures includes the following steps:

[0007] Build an improved Covasim environment and adjust its parameters:

[0008] An intervention measure setting module is added to the original Covasim environment. The added intervention measure setting module is used to dynamically set the intervention intensity during the simulation process according to the intervention measure intensity value given by the agent, and to apply the intervention measure to the environment operation. Real historical data is obtained to debug the parameters in the improved Covasim environment to obtain an improved Covasim environment that conforms to the real scenario.

[0009] Train the agent:

[0010] (1) The agent obtains the state information s at the current time t from the improved Covasim environment t When the number of infected people exceeds the preset threshold, the intervention intensity value a is output according to the status information. t , according to the intervention intensity value a t Act on the environment and get the corresponding reward information r t , get the state information s of the current environment at the next moment t+1 t+1 , the data group (s t , a t , r t , s t+1 ) is stored in the experience playback area, where the reward information r t Includes health bonus H and economic incentives E ;

[0011] (2) When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the agent;

[0012] (3) Repeat steps (1)-(2) until the preset maximum training time is reached to obtain the trained agent;

[0013] The trained intelligent agent outputs corresponding intervention measures based on the current epidemic status of the infectious disease and completes the optimization of the intervention measures.

[0014] Preferably, real historical data is obtained to debug the two parameters of the initial number of infected people and the population contact rate in the improved Covasim environment.

[0015] Preferably, the intervention intensity value a t Includes: Changing the transmission rate Ch-β , change the enforcement intensity of detection measures Ch- Tp and the intensity of contact tracing measures Ch-CTp .

[0016] Preferably, the improved Covasim environment further comprises: an initialization module, a Covasim simulation model creation module, an environment operation module, a reset module and a status acquisition module;

[0017] Initialization module, used to define the environmental intervention space and state space;

[0018] Create a Covasim simulation model module to create a Covasim simulation object based on epidemic data and population parameters;

[0019] The environment operation module is used to run the environment and dynamically add intervention measures according to the number of infected people. Each time a reinforcement learning time step is executed, the Covasim simulation object runs a cycle and dynamically adds intervention measures according to the number of infected people. At each reinforcement learning time step, the reward is calculated based on the simulation information and the state of the next time step is obtained.

[0020] The reset module is used to reset the environment to its initial state to facilitate a new round of training and evaluation;

[0021] The state acquisition module is used to return the observation state of the current environment for the agent to use in decision-making.

[0022] Preferably, the status information includes: susceptible persons S, exposed persons E, infected persons I, recovered persons R, deceased persons D, cumulative number of tests CT and cumulative number of quarantines CQ.

[0023] Preferably, reward information r t The specific calculation method is:

[0024] ;

[0025] and All are hyperparameters;

[0026] Health Rewards H for:

[0027] ;

[0028] The number of new recoveries per day N R A positive unit reward will be given, and the number of new infections per day N I , number of severe infections N S and death toll N D Give a negative reward value; ω1, ω2 and ω3 are the penalty coefficients corresponding to infection, severe illness and death respectively;

[0029] Economic Rewards Efor:

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] in, L E Indicates economic losses, and is the economic coefficient, P represents the total population, It means that each healthy individual can make economic contributions every day. C T and C Q They represent the economic consumption corresponding to the number of new tests and new quarantines per day, C β represents daily economic consumption when Ch-β lockdown measures are adopted; M I 、 M Q and M D Represents the number of daily infections, quarantines, and deaths, respectively.

[0035] Preferably, the specific content of training the agent includes:

[0036] ① Initialize the intelligent agent, which includes the Actor network and the Critic network, initialize the network parameters of the Actor network and the Critic network, and initialize the preset number of people threshold, preset data threshold and preset maximum training time;

[0037] ② Both the Actor network and the Critic network obtain state information s from the improved Covasim environment t ;

[0038] ③ When the number of infected people exceeds the preset threshold, the Actor network outputs the action-intervention intensity value a according to the state information t , and obtain the intervention intensity value a t Then change the intensity of the corresponding intervention measures to act on the environment and obtain the corresponding reward information r t , and then obtain the next state information s of the current environment t+1 ;

[0039] When the number of infected people does not exceed the preset threshold, the Actor network outputs the intervention measure intensity value a according to the state information. t But no corresponding intervention measures are executed, and the reward information r corresponding to the current environment state is obtained. t ;

[0040] ④ The currently acquired data group (s t , a t , r t , s t+1 ) is stored in the experience playback area;

[0041] ⑤ When the data stored in the experience playback area does not reach the preset data threshold, continue to obtain data groups (s t , a t , r t , s t+1 ), until the preset data threshold is reached;

[0042] When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the Actor network and the Critic network;

[0043] ⑥ Repeat steps ②-⑤ until the maximum training time is reached to obtain the trained agent.

[0044] Preferably, the loss function of the Actor network is:

[0045] ;

[0046] in, For expectations, represents the probability ratio of the new and old strategies, and Represents the new and old strategies in s t Status Completed t The probability of , used for importance sampling, is the advantage function, which measures the action In state The relative value of Represents the clip function, which is used to Restricted to To prevent the policy update from being too large, is the limiting parameter;

[0047] The loss function of the Critic network is:

[0048] ;

[0049] in, Represents the current value network state information st The predicted state value of represents the value of the goal state calculated by discounted return and generalized advantage estimation.

[0050] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method for optimizing infectious disease intervention measures based on reinforcement learning, which has the following beneficial effects:

[0051] Dynamic real-time decision-making capabilities: By interacting with the environment (Covasim epidemic system), real-time data (such as daily cases and intervention effect feedback) is used to continuously update strategies, achieving a closed loop of "prediction-intervention-evaluation-optimization";

[0052] Quantify and balance multiple objectives: transform the number of infected people and economic loss targets into a unified reward function, and achieve multi-objective collaborative optimization by optimizing rewards;

[0053] Automatic exploration of optimal strategy combinations: Through trial-and-error mechanisms, the combined effects of different intervention measures (such as the synergistic effect of lockdown intensity and detection intensity) are explored, and strategy combinations that are difficult for human experts to preset are discovered. Efficient optimization based on simulation: By optimizing the epidemic simulator Covasim, rapid trial and error can be achieved in a virtual environment, avoiding the trial and error costs of real-world interventions. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 This is a flowchart of a method for optimizing infectious disease intervention measures based on reinforcement learning disclosed in the present invention;

[0056] FIG2 is a schematic diagram of the specific structures of the Actor network and the Critic network provided by an embodiment of the present invention, wherein FIG2(a) is the Actor network and FIG2(b) is the Critic network;

[0057] Figure 3 The fitting results of the improved Covasim environment provided in this embodiment using the real historical data of a certain country as input;

[0058] Figure 4 shows a comparison of the learning results of the DQN algorithm and the PPO algorithm based on prioritized experience replay in the discrete action space provided by this embodiment, where Figure 4(a) shows the DQN algorithm based on prioritized experience replay, and Figure 4(b) shows the PPO algorithm;

[0059] FIG5 is a comparison of the learning strategy effects and strategy sampling action values ​​of the DQN algorithm and the PPO algorithm based on priority experience replay in the discrete action space provided by this embodiment, wherein FIG5(a) is a schematic diagram for comparing the overall effects obtained after training the DQN algorithm based on priority experience replay, FIG5(b) is a schematic diagram for comparing the overall effects obtained after training the PPO algorithm, FIG5(c) is a schematic diagram for comparing the strategy sampling action value points obtained after training the DQN algorithm based on priority experience replay, and FIG5(d) is a schematic diagram for comparing the strategy sampling action value points obtained after training the PPO algorithm;

[0060] FIG6 shows the strategy effect obtained by the PPO algorithm in the continuous action space provided by this embodiment and the changes in the action value of the specific strategy sampling, wherein FIG6(a) is the overall effect diagram, and FIG6(b) is the strategy sampling action value point diagram;

[0061] Figure 7 This is a diagram showing the effect of applying the 7-work-7-lockdown policy provided in this embodiment;

[0062] Figure 8 is a schematic diagram comparing the real-time regeneration number Rt of different strategies provided in this embodiment. Figure 8 (a) is a schematic diagram showing the change of Rt of the strategy obtained by the PPO algorithm in the continuous action space. Figure 8 (b) is a schematic diagram showing the change of Rt of the 7-day work-7-day blockade strategy. Figure 8 (c) is a schematic diagram showing the change of Rt under the real data fitting condition. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0064] The present invention provides a method for optimizing infectious disease intervention measures based on reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0065] Build an improved Covasim environment and adjust its parameters:

[0066] An intervention measure setting module is added to the original Covasim environment. The added intervention measure setting module is used to dynamically set the intervention parameters in the simulation process according to the intervention measure intensity value given by the intelligent agent, and the intervention measures are applied to the environment operation. Real historical data is obtained to debug the parameters in the improved Covasim environment, so as to obtain an improved Covasim environment that conforms to the real scenario.

[0067] Train the agent:

[0068] (1) The agent obtains the state information s at the current time t from the improved Covasim environment t When the number of infected people exceeds the preset threshold, the intervention intensity value a is output according to the status information. t , according to the intervention intensity value a t Act on the environment and get the corresponding reward information r t , get the state information s of the current environment at the next moment t+1 t+1 , the data group (s t , a t , r t , s t+1 ) is stored in the experience playback area, where the reward information r t Includes health bonus H and economic incentives E ;

[0069] (2) When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the agent;

[0070] (3) Repeat steps (1)-(2) until the preset maximum training time is reached to obtain the trained agent;

[0071] The trained intelligent agent outputs corresponding intervention measures based on the current epidemic status of the infectious disease and completes the optimization of the intervention measures.

[0072] It should be noted that:

[0073] Historical real data include the cumulative number of diagnoses and deaths caused by the epidemic in a certain area during a certain period of time, as well as data for the time period when actual measures were taken. They are used to adjust the parameters of the improved Covasim simulation model so that the cumulative number of diagnoses and deaths output by the model matches the real data.

[0074] The original Covasim environment is an agent-based model, which means it simulates the state and behavior of each individual. The core components of this model include:

[0075] Agents: Each individual is simulated independently, including attributes such as age, gender, health status, and contact network. Diseases are transmitted through interactions between individuals, which can capture population heterogeneity and complex transmission paths.

[0076] Contact Network: defines the contact relationships between individuals, usually divided into different levels (such as family, work, community, school, etc.);

[0077] Disease States: Individuals can be in different health states, such as susceptible, exposed, infected, and recovered, which is expanded to the SEIRD model based on the SIR basic model.

[0078] The simulation process of the original Covasim environment includes the following steps:

[0079] Initialization: Initialize individuals and contact networks based on input parameters (such as population size, contact matrix, disease parameters, etc.) and predefine intervention measures.

[0080] Timestep: Simulates contact and disease transmission between individuals at each time step (one day), updates individual status (e.g., from susceptible to infected) based on the probability of disease transmission and individual health status; implements predetermined intervention measures (e.g., social distancing, quarantine, etc.) during the simulation to assess their impact on the epidemic; records individual-level (infection time, symptom status, test results, etc.) and group-level data, including daily new infections, cumulative deaths, and effective reproduction number (Rt);

[0081] Output results: time series data, including daily new cases, cumulative infections, deaths, confirmed cases, etc.; epidemic curves drawn based on time series data, such as changes in the number of infections, severe cases, and deaths over time.

[0082] It can be seen that the original Covasim environment does not support adding intervention measures during the simulation process. The execution of the intervention measures in the above steps requires the relevant parameters to be passed in as input, and thus it is impossible to add different types of intervention measures and change the set intervention measure intensity value during the simulation step process, and it is also impossible to achieve dynamic intervention. The present invention has been modified on this basis to enable it to dynamically adjust and add the intervention measure setting module to be packaged as a reinforcement learning environment.

[0083] In order to further implement the above technical solution, real historical data was obtained to debug the two parameters of initial infection number and population contact rate in the improved Covasim environment.

[0084] In order to further implement the above technical solution, the intervention intensity value a t Includes: Changing the transmission rate Ch-β , change the enforcement intensity of detection measures Ch-Tp and the intensity of contact tracing measures Ch-CTp .

[0085] It should be noted that:

[0086] Among them, the change in transmission rate Ch-β Represents the blockade intensity. A value of 1 means no blockade, and a value of 0.5 means 50% of the area is blocked.

[0087] In order to further implement the above technical solution, the improved Covasim environment also includes: an initialization module, a Covasim simulation model creation module, an environment operation module, a reset module and a status acquisition module;

[0088] Initialization module, used to define the environmental intervention space and state space;

[0089] Create a Covasim simulation model module to create a Covasim simulation object based on epidemic data and population parameters;

[0090] The environment operation module is used to run the environment and dynamically add intervention measures according to the number of infected people. Each time a reinforcement learning time step is executed, the Covasim simulation object runs a cycle and dynamically adds intervention measures according to the number of infected people. At each reinforcement learning time step, the reward is calculated based on the simulation information and the state of the next time step is obtained.

[0091] The reset module is used to reset the environment to its initial state to facilitate a new round of training and evaluation;

[0092] The state acquisition module is used to return the observation state of the current environment for the agent to use in decision-making.

[0093] In order to further implement the above technical solution, the status information includes: susceptible people S, exposed people E, infected people I, recovered people R, deceased people D, cumulative number of tests CT and cumulative number of isolations CQ.

[0094] In order to further implement the above technical solution, reward information r t The specific calculation method is:

[0095] ;

[0096] and are all hyperparameters, and Dynamic adjustment, if the epidemic is in the outbreak period, =0.7, =0.3, giving priority to controlling the epidemic; when the epidemic becomes stable, = =0.5, balancing the needs of both;

[0097] Health Rewards H for:

[0098] ;

[0099] The number of new recoveries per day N RA positive unit reward will be given, and the number of new infections per day N I , number of severe infections N S and death toll N D A negative reward value is given; ω1, ω2, and ω3 are the penalty coefficients corresponding to infection, severe illness, and death, respectively. The weight distribution follows the hazard priority of "death > severe illness > infection" (ω3 > ω2 > ω1), and the specific values ​​are ω3 = 5, ω2 = 3, and ω1 = 2;

[0100] Economic Rewards E for:

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] in, L E Indicates economic losses, and is the economic coefficient, determined by Bayesian optimization method, P represents the total population, It means that each healthy individual can make economic contributions every day. C T and C Q They represent the economic consumption corresponding to the number of new tests and new quarantines per day, C β represents daily economic consumption when Ch-β lockdown measures are adopted; M I 、 M Q and M D Represents the number of daily infections, quarantines, and deaths, respectively.

[0106] It should be noted that:

[0107] Considering that the goal is to control the epidemic and reduce the number of infections, in this example, ω1, ω2, and ω3 are set to 2, 3, and 5 respectively, depending on the severity of the infection. The Covasim model allows you to view the cumulative or new infections, serious infections, recoveries, and deaths at each time step (in days). ESpecifically, in the dimension of quantitative analysis of economic contribution, the model is constructed based on the following assumptions: assuming that each healthy individual can generate economic contribution every day , while infected individuals (regardless of whether they are in isolation) and deceased individuals have no economic contribution output. When testing and contact tracing measures are implemented, economic consumption is generated based on the number of tests and quarantined people per day. 、 If measures are taken to change the transmission rate (Ch-β) (i.e., a lockdown strategy is implemented), the economic cost is calculated based on the degree of change. , calculated as the total number of people multiplied by (1 - degree of change).

[0108] To further implement the above technical solution, the specific content of training the agent includes:

[0109] ① Initialize the intelligent agent, which includes the Actor network and the Critic network, initialize the network parameters of the Actor network and the Critic network, and initialize the preset number of people threshold, preset data threshold and preset maximum training time;

[0110] ② Both the Actor network and the Critic network obtain state information S from the improved Covasim environment t ;

[0111] ③ When the number of infected people exceeds the preset threshold, the Actor network outputs the intervention measure intensity value a according to the state information t , according to the intervention intensity value a t Act on the environment and get the corresponding reward information r t , get the next state information s of the current environment t+1 ;

[0112] When the number of infected people does not exceed the preset threshold, the Actor network outputs the intervention measure intensity value a according to the state information. t But the corresponding intervention measures are not executed, and the reward information r corresponding to the current environment state is obtained. t ;

[0113] ④ The currently acquired data group (s t , a t , r t , s t+1 ) is stored in the experience playback area;

[0114] ⑤ When the data stored in the experience playback area does not reach the preset data threshold, continue to obtain data groups (s t , a t , r t , s t+1 ), until the preset data threshold is reached;

[0115] When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the Actor network and the Critic network;

[0116] ⑥ Repeat steps ②-⑤ until the maximum training time is reached to obtain the trained agent.

[0117] It should be noted that:

[0118] In the specific application process, a data group with a mini-batch size can be randomly sampled to update the network parameters of the Actor network and the Critic network.

[0119] To further implement the above technical solution, the loss function of the Actor network is:

[0120] ;

[0121] in, For expectations, represents the probability ratio of the new and old strategies, and Represents the new and old strategies in s t Status Completed t The probability of , used for importance sampling, is the advantage function, which measures the action In state The relative value of Represents the clip function, which is used to Restricted to To prevent the policy update from being too large, is the limiting parameter;

[0122] The loss function of the Critic network is:

[0123] ;

[0124] in, Represents the current value network state information s t The predicted state value of represents the value of the goal state calculated by discounted return and generalized advantage estimation.

[0125] It should be noted that:

[0126] In this embodiment, the agent network consists of an actor and a critic network. The actor and critic have the same network architecture, consisting of an input layer, two hidden layers of 64 neurons, and an output layer. The loss function of the actor network minimizes the product of the ratio of the new and old strategies and the advantage function, ensuring that the strategy update is only carried out in the direction of increasing the reward and the amplitude is controlled; the loss function of the critic network trains the value function through the mean square error to drive the critic prediction value. Towards Convergence provides more accurate advantage estimates for policy optimization.

[0127] The present invention will be further described below through specific experiments:

[0128] Improving the Covasim model and integrating it with reinforcement learning is only the beginning. The key lies in applying reinforcement learning to control intervention parameters during the spread of infectious diseases, thereby obtaining practical strategies and effective intervention implementation plans. This presupposes that the modified model can accurately reflect the actual epidemic situation.

[0129] To this end, this example collected data on the cumulative number of confirmed cases and deaths in a certain country from January 21, 2020 to May 31, 2020. After determining some parameters (such as the number of simulated people, the number of people contacted on different networks, and the implementation time and parameters of various intervention measures that are in line with reality), the two parameters of the initial number of infections and the population contact rate were repeatedly adjusted to fit the collected real data. The final fitting result is as follows: Figure 3 shown.

[0130] Depend on Figure 3 As can be seen, after parameter adjustment, the simulation results of the improved Covasim closely match the cumulative number of confirmed cases and deaths, strongly supporting the effectiveness of this infectious disease model. Based on this, the adjusted parameters and basic parameters, excluding those related to intervention measures, can be used as basic data to reflect the actual epidemic situation. Reinforcement learning can be used to explore the parameters related to intervention measures, and the practical value of the strategy model generated by combining reinforcement learning with complex infectious disease models can be further explored.

[0131] Based on the basic parameters reflecting the real world obtained above, an epidemic model was constructed, and reinforcement learning was used to explore the effectiveness of the learned strategy in real-world scenarios. In this environment, this example first compared the learning performance and final strategy differences of different algorithms in a discrete action space, focusing on the PPO algorithm and the DQN algorithm using prioritized experience replay. The learning results are shown in Figure 4.

[0132] As shown in Figure 4, the DQN algorithm using optimized experience replay converges significantly faster, reaching convergence in approximately 250 rounds, while the PPO algorithm requires 2000 rounds to converge, with similar convergence values. Therefore, in terms of convergence speed, the DQN algorithm based on prioritized experience replay has a clear advantage. However, a deeper analysis of the policies learned by both algorithms reveals problems.

[0133] Figures 5(a) and 5(b) show that, in the discrete action space, the strategies learned by the DQN algorithm based on prioritized experience replay and the PPO algorithm outperformed the fitted strategy in controlling the epidemic, as evidenced by a significant reduction in the cumulative number of infections and deaths. Figures 5(c) and 5(d) show that the strategy learned by the DQN algorithm based on prioritized experience replay initially adopted an emergency, high-intensity lockdown strategy, followed by a low-intensity lockdown without implementing testing or quarantine measures. While this strategy achieved the goals of epidemic control and economic stability, it should be noted that this strategy is limited by the discrete action space setting. Once the action space changes, the strategy will also change accordingly. Furthermore, the DQN algorithm based on prioritized experience replay exhibits significant policy fluctuations when continuing training after convergence. The strategy generated by the PPO algorithm exhibits significant fluctuations in action values ​​and lacks stability. Given the realities of epidemic prevention and control, this strategy conflicts with the actual needs of intervention, as real-world epidemic interventions typically do not undergo frequent and drastic adjustments.

[0134] In comparison, the action values ​​in the continuous action space are more reasonable. As shown in Figure 6(b), which uses the PPO algorithm to learn the strategy, a high-intensity lockdown is immediately implemented after the outbreak is discovered, and large-scale testing is temporarily suspended. After the lockdown ends, strict testing is implemented. If the outbreak is still at a certain scale, the lockdown is continued without testing. After the lockdown ends, testing is repeated. If the outbreak is confirmed to be under control, large-scale lockdown is discontinued, and testing is continued for a period of time until all intervention measures are discontinued after the outbreak eases. This strategy fully considers the characteristics of epidemic transmission and the impact of intervention measures on the economy, effectively achieving a balance between epidemic prevention and control and economic development, and provides a valuable reference for the field of public health and safety.

[0135] exist Figure 7In this embodiment, the optimization results of the traditional 7-day work-7-day lockdown strategy on real data are further explored. By comparing with the various algorithms used in this embodiment, it is found that the strategies obtained based on agent learning are better than the traditional strategies in terms of application effect. The strategy obtained based on the PPO algorithm in the continuous action space can not only effectively control the outbreak of the epidemic, but also take into account economic stability, with an economic loss of only 10.25%; while the traditional strategy performs poorly in epidemic prevention and control. It can be observed from the relevant data that the epidemic is showing a trend of continuous deterioration. Although it is better than the strategy reflecting the real data in controlling the number of infections, due to the implementation of stricter lockdown measures, the economic losses have increased, reaching 38.01% compared to 36.01% of the strategy reflecting the real data, highlighting the advantages of the strategy explored in this embodiment in responding to the epidemic and balancing the economy.

[0136] Figure 8 compares the real-time reproduction number (Rt) under different strategies. According to epidemiological principles, when Rt is less than 1, it means that maintaining current prevention and control measures can gradually control the epidemic, resulting in a decrease in the number of cases and a mitigation of the epidemic. Conversely, if Rt is greater than 1, it indicates that the epidemic will continue to spread and is in a developing or worsening stage.

[0137] As shown in Figure 8 (a), in the continuous action space, the strategy learned by the PPO algorithm quickly reduced the Rt value to below 1 due to the implementation of strict lockdown measures in the early stage, and initially achieved the effect of epidemic control. The implementation of a series of subsequent intervention measures ensured that the Rt value did not fluctuate violently, and the number of infected people was ultimately controlled at around 300,000, as shown in Figure 6 (a). As shown in Figure 8 (b), under the 7-day work-7-day lockdown strategy, although the Rt value has been fluctuating around 1, it has achieved certain epidemic control results due to its timely control. However, the number of infected people eventually reached around 1, as shown in Figure 8 (b). Figure 7 The number of infected people under these two strategies is better than the corresponding values ​​of the real data fitting.

[0138] Although the Rt value in the real data fitting case had dropped below 1 around mid-March as shown in Figure 8(c), compared with the PPO strategy, a delay of only half a month caused the cumulative number of infections to change exponentially. This further highlights the importance of timely and reasonable intervention measures in infectious disease prevention and control, and provides a strong empirical basis for the subsequent formulation of more scientific and efficient prevention and control strategies, which helps to improve the accuracy and timeliness of public health decision-making.

[0139] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for optimizing infectious disease intervention measures based on reinforcement learning, characterized in that: The following steps are involved: Build an improved Covasim environment and adjust its parameters: An intervention measure setting module is added to the original Covasim environment. The added intervention measure setting module is used to dynamically set the intervention intensity during the simulation process according to the intervention measure intensity value given by the agent, and to apply the intervention measure to the environment operation. Real historical data is obtained to debug the parameters in the improved Covasim environment to obtain an improved Covasim environment that conforms to the real scenario. Train the agent: (1) The agent obtains the state information s at the current time t from the improved Covasim environment t When the number of infected people exceeds the preset threshold, the intervention intensity value a is output according to the status information. t , according to the intervention intensity value a t Act on the environment and get the corresponding reward information r t , get the state information s of the current environment at the next moment t+1 t+1 , the data group (s t , a t , r t , s t+1 ) is stored in the experience playback area, where the reward information r t Includes health bonus H and economic incentives E ;Intervention intensity value a t Includes: Changing the transmission rate Ch-β , change the enforcement intensity of detection measures Ch-Tp and the intensity of contact tracing measures Ch- CTp Status information includes: susceptible people S, exposed people E, infected people I, recovered people R, deceased people D, cumulative number of tests CT and cumulative number of quarantines CQ; Reward Information t The specific calculation method is: ； and All are hyperparameters; Health Rewards H for: ； The number of new recoveries per day N R A positive unit reward will be given, and the number of new infections per day N I , number of severe infections N S and death toll N D Give a negative reward value; ω1, ω2 and ω3 are the penalty coefficients corresponding to infection, severe illness and death respectively; Economic Rewards E for: ； ； ； ； in, Indicates economic losses, and is the economic coefficient, P represents the total population, It means that each healthy individual can make economic contributions every day. C T and C Q They represent the economic consumption corresponding to the number of new tests and new quarantines per day, C β represents daily economic consumption when Ch-β lockdown measures are adopted; M I 、 M Q and M D The numbers of daily infections, quarantines, and deaths are respectively; (2) When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the agent; (3) Repeat steps (1)-(2) until the preset maximum training time is reached to obtain the trained agent; The trained intelligent agent outputs corresponding intervention measures based on the current epidemic status of the infectious disease and completes the optimization of the intervention measures.

2. The method for optimizing infectious disease intervention measures based on reinforcement learning according to claim 1, characterized in that: Real historical data is obtained to debug the two parameters of initial infection number and population contact rate in the improved Covasim environment.

3. The method for optimizing infectious disease intervention measures based on reinforcement learning according to claim 1, characterized in that: The improved Covasim environment also includes: initialization module, Covasim simulation model creation module, environment operation module, reset module and status acquisition module; Initialization module, used to define the environmental intervention space and state space; Create a Covasim simulation model module to create a Covasim simulation object based on epidemic data and population parameters; The environment operation module is used to run the environment. For each reinforcement learning time step, the Covasim simulation object runs a cycle, dynamically adding intervention measures based on the number of infected people. At each reinforcement learning time step, rewards are calculated based on simulation information and the state of the next time step is obtained. The reset module is used to reset the environment to its initial state to facilitate a new round of training and evaluation; The state acquisition module is used to return the observation state of the current environment for the agent to use in decision-making.

4. The method for optimizing infectious disease intervention measures based on reinforcement learning according to claim 1, characterized in that: The specific content of training the intelligent agent includes: ① Initialize the intelligent agent, which includes the Actor network and the Critic network, initialize the network parameters of the Actor network and the Critic network, and initialize the preset number of people threshold, preset data threshold and preset maximum training time; ② Both the Actor network and the Critic network obtain state information s from the improved Covasim environment t ; ③ When the number of infected people exceeds the preset threshold, the Actor network outputs the action-intervention intensity value a according to the state information t , and obtain the intervention intensity value a t Then change the intensity of the corresponding intervention measures to act on the environment and obtain the corresponding reward information r t , and then obtain the next state information s of the current environment t+1 ; When the number of infected people does not exceed the preset threshold, the Actor network outputs the intervention measure intensity value a according to the state information. t But no corresponding intervention measures are executed, and the reward information r corresponding to the current environment state is obtained. t ; ④ The currently acquired data group (s t , a t , r t , s t+1 ) is stored in the experience playback area; ⑤ When the data stored in the experience playback area does not reach the preset data threshold, continue to obtain data groups (s t , a t , r t , s t+1 ), until the preset data threshold is reached; When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the Actor network and the Critic network; ⑥ Repeat steps ②-⑤ until the maximum training time is reached to obtain the trained agent.

5. The method for optimizing infectious disease intervention measures based on reinforcement learning according to claim 1, characterized in that: The loss function of the Actor network is: ； in, For expectations, represents the probability ratio of the new and old strategies, and Represents the new and old strategies in s t Status Completed t The probability of , used for importance sampling, is the advantage function, which measures the action In state The relative value of Represents the clip function, which is used to Restricted to To prevent the policy update from being too large, is the limiting parameter; The loss function of the Critic network is: ； in, Represents the current value network state information s t The predicted state value of represents the value of the goal state calculated by discounted return and generalized advantage estimation.

Images