A dynamic optimization method and system for emergency material reserves based on reinforcement learning

Abstract

The present invention relates to the field of emergency management technology, and in particular to a dynamic optimization method and system for emergency material reserves based on reinforcement learning, including obtaining emergency material demand forecast results through a machine learning algorithm based on disaster scenario data and emergency material consumption data in a target power facility area; based on the emergency material demand forecast results, using a Monte Carlo simulation method to quantify the risk probability distribution of emergency material reserves; constructing an emergency material reserve digital twin model based on the risk probability distribution and multi-dimensional real-time status information of an emergency material reserve system; introducing a reinforcement learning algorithm into the emergency material reserve digital twin model, and obtaining an optimal emergency material reserve strategy through interactive learning. The present invention improves the accuracy and efficiency of emergency material reserves by combining digital twin technology, reinforcement learning algorithms and other methods, and can quickly and accurately provide the required emergency materials, thereby significantly improving emergency response capabilities.

Description

Technical Field

[0001] The present invention relates to the technical field of emergency management, and in particular to a method and system for dynamic optimization of emergency material reserves based on reinforcement learning. Background Art

[0002] In the current social environment where natural disasters occur frequently, ensuring the timeliness and accuracy of emergency material reserve plans has become the key to ensuring the smooth progress of post-disaster rescue work. However, existing technologies face multiple challenges in formulating emergency material reserve plans. These problems seriously restrict the improvement of material reserve efficiency and response speed. The formulation process of traditional emergency material reserve plans often relies on static predictions and planning, and lacks dynamism. This method is difficult to effectively respond to the dynamic changes in the time, location, type and material demand of disasters, resulting in material reserves being difficult to meet actual needs when actual disasters occur, causing waste or shortage of resources; secondly, although some demand forecasting methods based on machine learning algorithms have been applied to this field, these methods are still limited by data quality in practical applications, and the accuracy of the forecast results still needs to be improved. This inaccurate demand forecast not only affects the formulation of material reserve plans, but also leads to insufficient or excessive supply of materials when disasters occur, affecting the rescue effect.

[0003] To sum up, the existing technology has problems such as insufficient dynamism and low prediction accuracy when formulating emergency material reserve plans. Therefore, it is urgent to provide a dynamic optimization method for emergency material reserves to improve the accuracy and timeliness of material reserves, ensure that the required emergency materials can be provided quickly and effectively, and provide strong support for rescue work. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a dynamic optimization method and system for emergency material reserves based on reinforcement learning.

[0005] In a first aspect, the present invention provides a method for dynamic optimization of emergency material reserves based on reinforcement learning, the method comprising the following steps:

[0006] Based on the disaster scenario data and emergency material consumption data of the target power facility area, the emergency material demand forecast results are obtained through machine learning algorithms;

[0007] Based on the emergency material demand forecast results, combined with the historical disaster frequency and material supply uncertainty, the risk probability distribution of emergency material reserves is quantified using the Monte Carlo simulation method;

[0008] Constructing a digital twin model of emergency material reserves based on the risk probability distribution and multi-dimensional real-time status information of the emergency material reserve system;

[0009] The reinforcement learning algorithm is introduced into the digital twin model of emergency material reserves, and interactive learning is carried out by simulating the material dispatch and reserve adjustment process in disaster scenarios to obtain the optimal emergency material reserve strategy.

[0010] In a further embodiment, the step of obtaining the emergency material demand forecast result by a machine learning algorithm based on the disaster scenario data and emergency material consumption data of the target power facility area includes:

[0011] Collect disaster scenario data and material consumption data in the target power facility area; disaster scenario data includes disaster occurrence time, disaster type, disaster location and disaster intensity; material consumption data includes material consumption type and material consumption quantity;

[0012] Preprocessing the disaster scene data and the material consumption data to obtain preprocessed disaster scene data and preprocessed material consumption data;

[0013] Based on the pre-processed disaster scenario data and pre-processed material consumption data, the autoregressive integrated moving average model is used to extract the material demand characteristics under different disaster types;

[0014] According to the material demand characteristics under different disaster types, the pre-built random forest prediction is used to obtain the emergency material demand prediction results.

[0015] In a further implementation scheme, the step of quantifying the risk probability distribution of emergency material reserves using a Monte Carlo simulation method based on the emergency material demand forecast results, combined with the historical disaster frequency and material supply uncertainty, includes:

[0016] Using geographic information system technology to perform spatial statistical analysis on the disaster scenario data, generate a disaster hotspot map, and determine the spatial scope of high-risk areas based on the disaster hotspot map;

[0017] According to the disaster intensity of the disaster scene data, a kernel density estimation method is used to construct probability density functions of different types of disaster intensities;

[0018] Generate disaster scenario samples based on the spatial extent of high-risk areas, probability density functions of different types of disaster intensities, historical disaster frequencies, and uncertainty in material supply;

[0019] According to the forecast results of emergency material demand and the current emergency material reserves, calculate the material gap information under each disaster scenario sample;

[0020] Based on the material gap information under multiple disaster scenario samples, the Monte Carlo simulation method is used to quantify the risk probability distribution of emergency material reserves.

[0021] In a further embodiment, the step of generating disaster scenario samples according to the spatial scope of high-risk areas, probability density functions of different types of disaster intensities, historical disaster occurrence frequencies, and uncertainty in material supply includes:

[0022] The probability density functions of different types of disaster intensity, the frequency of historical disasters and the uncertainty of material supply are divided into fuzzy sets, and the membership function is constructed for each fuzzy set using fuzzy logic algorithm;

[0023] A random number generator is used to generate random samples based on the spatial scope of high-risk areas, probability density functions of different types of disaster intensities, historical disaster frequency, and uncertainty in material supply; the random samples include disaster intensity values, disaster event distribution, and material supply status;

[0024] The generated random samples are used as input, and the probability of disaster occurrence of each random sample is calculated through the membership function;

[0025] According to the probability of disaster occurrence, disaster scenario samples that meet a preset sample probability threshold are screened out from the random samples.

[0026] In a further implementation scheme, the step of constructing a digital twin model of emergency material reserves based on the risk probability distribution and the multi-dimensional real-time status information of the emergency material reserve system includes:

[0027] Collect multi-dimensional real-time status information of the emergency material reserve system, and perform dimensionality reduction processing on the multi-dimensional real-time status information through the principal component analysis method to extract key status indicators;

[0028] According to the risk probability distribution and the key status indicators, a causal relationship model between the risk probability distribution and the emergency material reserve status is established using a structural equation model;

[0029] The causal relationship model is simulated to predict the state change of the emergency material reserve system under different risk probability distribution conditions;

[0030] According to the changes in the state of the emergency material reserve system under different risk probability distributions, a knowledge graph of emergency material reserve including the state of the emergency material reserve system, risk probability distribution and their mutual relationship is constructed;

[0031] Based on the emergency material reserve knowledge graph, a digital twin model of emergency material reserve is constructed using digital twin technology.

[0032] In a further implementation scheme, the multi-dimensional real-time status information includes the physical structure, operating status and external environment information of the emergency material reserve system.

[0033] In a further implementation scheme, the step of introducing a reinforcement learning algorithm into the emergency material reserve digital twin model, interactively learning by simulating the material dispatch and reserve adjustment process under disaster scenarios, and obtaining the optimal emergency material reserve strategy includes:

[0034] In the digital twin model of emergency material reserves, disaster uncertainty and material supply interruption risk are taken as random terms, and a state transition model is constructed using the Markov decision process;

[0035] The state transition model is used to simulate the random changes in the probability of disaster occurrence and the risk of material supply under the current environmental state, and generate a state transition probability distribution;

[0036] According to the state transition probability distribution, a next environment state is selected using a Monte Carlo sampling method;

[0037] Construct a reward function with the goal of minimizing the risk of material shortage, maximizing material utilization efficiency, and minimizing material transportation costs;

[0038] Running a reinforcement learning agent in the emergency material reserve digital twin model so that the reinforcement learning agent calculates the action probability distribution of different actions using a policy network according to the current environment state and the reward function;

[0039] An action is selected according to the action probability distribution, and the material dispatching and reserve adjustment process in a disaster scenario is simulated in combination with the next environmental state. Through multiple iterations and strategy updates, the optimal emergency material reserve strategy is obtained.

[0040] In a further implementation scheme, in the emergency material reserve digital twin model, the disaster uncertainty and material supply interruption risk are taken as random items, and the step of using the Markov decision process to construct a state transition model includes:

[0041] In the digital twin model of emergency material reserves, the state space and action space in the Markov decision process are defined; wherein the state space includes all disaster types, disaster degrees and material supply status, and the action space includes all material allocation and reserve decisions;

[0042] According to the state space and action space in the Markov decision process, the generator in the generative adversarial network is used to simulate the uncertainty of disasters and the risk of interruption of material supply, and generate state simulation samples;

[0043] According to the state simulation samples, the Markov decision process is used to simulate the state transition between different state simulation samples and build a state transition model.

[0044] In a further embodiment, the policy network is a multilayer perceptron.

[0045] In a second aspect, the present invention provides a dynamic optimization system for emergency material reserves based on reinforcement learning, the system comprising:

[0046] The demand forecasting module is used to obtain the emergency material demand forecasting results through machine learning algorithms based on the disaster scenario data and emergency material consumption data of the target power facility area;

[0047] A risk assessment module is used to quantify the risk probability distribution of emergency material reserves by using a Monte Carlo simulation method based on the emergency material demand forecast results, combined with the historical disaster frequency and material supply uncertainty;

[0048] A model building module, used to build a digital twin model of emergency material reserves based on the risk probability distribution and multi-dimensional real-time status information of the emergency material reserve system;

[0049] The optimization learning module is used to introduce the reinforcement learning algorithm into the emergency material reserve digital twin model, and interactively learn through simulating the material dispatch and reserve adjustment process under disaster scenarios to obtain the optimal emergency material reserve strategy.

[0050] The present invention provides a method and system for dynamic optimization of emergency material reserves based on reinforcement learning. The method obtains emergency material demand forecast results through a machine learning algorithm based on disaster scenario data and emergency material consumption data in a target power facility area; based on the emergency material demand forecast results, combined with the historical disaster frequency and material supply uncertainty, the risk probability distribution of emergency material reserves is quantified using a Monte Carlo simulation method; according to the risk probability distribution and multi-dimensional real-time status information of the emergency material reserve system, a digital twin model of emergency material reserves is constructed; a reinforcement learning algorithm is introduced into the digital twin model of emergency material reserves, and interactive learning is performed by simulating the material dispatch and reserve adjustment process under disaster scenarios to obtain the optimal emergency material reserve strategy. Compared with the prior art, the method improves the accuracy and efficiency of emergency material reserves by combining digital twin technology and reinforcement learning algorithms, and can quickly and accurately provide the required emergency materials, thereby significantly improving the emergency response capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of a flow chart of a dynamic optimization method for emergency material reserves based on reinforcement learning provided by an embodiment of the present invention;

[0052] Figure 2 It is a block diagram of a dynamic optimization system for emergency material reserves based on reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0054] refer to Figure 1 , the embodiment of the present invention provides a dynamic optimization method for emergency material reserves based on reinforcement learning, such as Figure 1 As shown, the method comprises the following steps:

[0055] S1. Based on the disaster scenario data and emergency material consumption data of the target power facility area, the emergency material demand forecast results are obtained through machine learning algorithms.

[0056] In this embodiment, the step of obtaining the emergency material demand prediction result by using a machine learning algorithm based on the disaster scenario data and emergency material consumption data of the target power facility area includes:

[0057] Collect disaster scenario data and material consumption data in the target power facility area; disaster scenario data includes disaster occurrence time, disaster type, disaster location and disaster intensity; material consumption data includes material consumption type and material consumption quantity;

[0058] Preprocessing the disaster scene data and the material consumption data to obtain preprocessed disaster scene data and preprocessed material consumption data;

[0059] Based on the pre-processed disaster scenario data and pre-processed material consumption data, the autoregressive integrated moving average model is used to extract the material demand characteristics under different disaster types;

[0060] According to the material demand characteristics under different disaster types, the pre-built random forest prediction is used to obtain the emergency material demand prediction results.

[0061] Specifically, this embodiment collects disaster scenario data in the target power facility area, extracts material consumption data from the operation and maintenance records of the power facilities, and performs pre-processing operations such as cleaning, conversion and integration on the collected disaster scenario data and material consumption data. Among them, data cleaning includes removing duplicate, erroneous and invalid data records, etc., and data conversion includes normalizing numerical data such as disaster intensity and material consumption quantity, and processing missing data by interpolation, mean filling or deletion. Then, this embodiment selects the autoregressive integrated moving average (ARIMA) model as a time series analysis tool to divide the data into different types according to the disaster type. Different subsets are selected, and ARIMA models are trained for each subset respectively to extract the time series of material demand under different disaster types, and the material demand characteristics are extracted from the time series of material demand under different disaster types. Finally, this embodiment selects random forest as the prediction model, and performs parameter optimization. The extracted material demand characteristics are used as input variables, and the trained model is used to predict the material demand for a period of time in the future to obtain the demand prediction results of each material under different disaster types. This embodiment can effectively predict the emergency material demand in the target power facility area when a disaster occurs through a machine learning algorithm, and provide a scientific basis for the storage and allocation of emergency materials.

[0062] S2. Based on the emergency material demand forecast results, combined with the historical disaster frequency and material supply uncertainty, the Monte Carlo simulation method is used to quantify the risk probability distribution of emergency material reserves.

[0063] In this embodiment, the step of quantifying the risk probability distribution of emergency material reserves by using the Monte Carlo simulation method based on the emergency material demand forecast result and combining the historical disaster frequency and material supply uncertainty includes:

[0064] Using geographic information system technology to perform spatial statistical analysis on the disaster scenario data, generate a disaster hotspot map, and determine the spatial scope of high-risk areas based on the disaster hotspot map;

[0065] According to the disaster intensity of the disaster scene data, a kernel density estimation method is used to construct probability density functions of different types of disaster intensities;

[0066] Generate disaster scenario samples based on the spatial extent of high-risk areas, probability density functions of different types of disaster intensities, historical disaster frequencies, and uncertainty in material supply;

[0067] According to the forecast results of emergency material demand and the current emergency material reserves, calculate the material gap information under each disaster scenario sample;

[0068] Based on the material gap information under multiple disaster scenario samples, the Monte Carlo simulation method is used to quantify the risk probability distribution of emergency material reserves.

[0069] The step of generating disaster scenario samples according to the spatial scope of high-risk areas, probability density functions of different types of disaster intensities, historical disaster occurrence frequencies and uncertainty in material supply includes:

[0070] The probability density functions of different types of disaster intensity, the frequency of historical disasters and the uncertainty of material supply are divided into fuzzy sets, and the membership function is constructed for each fuzzy set using fuzzy logic algorithm;

[0071] A random number generator is used to generate random samples based on the spatial scope of high-risk areas, probability density functions of different types of disaster intensities, historical disaster frequency, and uncertainty in material supply; the random samples include disaster intensity values, disaster event distribution, and material supply status;

[0072] The generated random samples are used as input, and the probability of disaster occurrence of each random sample is calculated through the membership function;

[0073] According to the probability of disaster occurrence, disaster scenario samples that meet a preset sample probability threshold are screened out from the random samples.

[0074] Specifically, this embodiment inputs disaster scenario data (including the time, location, type, intensity, etc. of the disaster) into a geographic information system (GIS), uses the spatial analysis function of GIS to perform statistical analysis on the disaster scenario data, identifies hot spots where disasters frequently occur, generates a disaster hot spot area map based on the identification results of the disaster hot spot areas, and determines the spatial scope of the high-risk area based on the disaster hot spot area map in combination with factors such as disaster intensity and impact range; then, extracts intensity data of each type of disaster from the disaster scenario data, uses a kernel density estimation method to perform probability density estimation on the intensity data of each type of disaster, obtains a probability density function of each disaster intensity, and verifies the obtained probability density function to ensure that it can accurately reflect the distribution of disaster intensity.

[0075] Then, this embodiment divides the disaster intensity into different fuzzy sets according to the probability density function of different types of disaster intensity, divides the frequency into different fuzzy sets according to the historical disaster frequency, divides the material supply status into different fuzzy sets according to the uncertainty of material supply, and uses the fuzzy logic algorithm to construct a membership function for each fuzzy set. The membership function describes the degree to which each sample belongs to a certain fuzzy set. For example, for the disaster intensity fuzzy set "serious", a membership function can be constructed so that when the disaster intensity value is high, the membership of the sample is close to 1; when the disaster intensity value is low, the membership is close to 0. At the same time, the location variable of the disaster occurrence is defined according to the spatial range of the high-risk area, the random variable of the disaster intensity is defined according to the probability density function of different types of disaster intensity, the random variable of the disaster occurrence time is defined according to the historical disaster frequency, and the random variable of the material supply status is defined according to the uncertainty of material supply. A random number generator is used to generate random samples according to the random variables defined above. The random samples include the disaster intensity value, the distribution of disaster occurrence events (such as time and location) and the material supply status. The generated random samples are used as inputs to the membership function constructed previously. In the example, the membership of each random sample in different fuzzy sets is calculated by the membership function. For example, the membership of a random sample in the "serious" disaster intensity fuzzy set and the membership of a random sample in the "tense" material supply status fuzzy set are calculated. The probability of disaster occurrence of each random sample is comprehensively evaluated according to the membership and the weight of the fuzzy set. According to the probability of disaster occurrence, random samples that meet the preset sample probability threshold are screened out from the random samples. These random samples will be used as the final disaster scenario samples. These disaster scenario samples can reflect the possibility and impact degree under different disaster scenarios, and provide strong support for disaster management and decision-making. In other implementation methods, this embodiment sets the parameters of the disaster scenario according to the spatial range of the high-risk area, the probability density function of the disaster intensity, the frequency of historical disasters, etc., and takes the uncertainty of material supply as a random variable and adds it to the disaster scenario. According to the set parameters and uncertainty variables, a random number generator is used to generate multiple disaster scenario samples. For example, this embodiment uses a random number generator to generate 100 disaster scenario samples, each of which contains information such as the disaster type, intensity, time and location of occurrence, and the uncertainty of material supply.

[0076] Finally, this embodiment queries the current emergency material reserves, which include the inventory quantity and distribution of various types of materials, compares the predicted material demand with the current reserves, and calculates the material gap information under each disaster scenario sample. The material gap information includes the gap quantity and gap type, etc. According to the material gap information under multiple disaster scenario samples, a Monte Carlo simulation model is constructed, and the Monte Carlo simulation model is run. Multiple simulations are performed on each disaster scenario sample, and the distribution of material gaps in the simulation results is statistically analyzed. The risk probability distribution of emergency material reserves is calculated. According to this risk probability distribution, the risks and gaps faced by emergency material reserves under different disaster scenarios can be understood, thereby providing decision support for formulating scientific emergency material reserve strategies.

[0077] S3. Construct a digital twin model of emergency material reserves based on the risk probability distribution and the multi-dimensional real-time status information of the emergency material reserve system.

[0078] In this embodiment, the step of constructing a digital twin model of emergency material reserves according to the risk probability distribution and the multi-dimensional real-time status information of the emergency material reserve system includes:

[0079] Collect multi-dimensional real-time status information of the emergency material reserve system, and perform dimensionality reduction processing on the multi-dimensional real-time status information through the principal component analysis method to extract key status indicators; wherein the multi-dimensional real-time status information includes the physical structure, operation status and external environment information of the emergency material reserve system;

[0080] According to the risk probability distribution and the key status indicators, a causal relationship model between the risk probability distribution and the emergency material reserve status is established using a structural equation model;

[0081] The causal relationship model is simulated to predict the state change of the emergency material reserve system under different risk probability distribution conditions;

[0082] According to the changes in the state of the emergency material reserve system under different risk probability distributions, a knowledge graph of emergency material reserve including the state of the emergency material reserve system, risk probability distribution and their mutual relationship is constructed;

[0083] Based on the emergency material reserve knowledge graph, a digital twin model of emergency material reserve is constructed using digital twin technology.

[0084] Specifically, this embodiment uses a sensor network to collect multi-dimensional status information of the emergency material reserve system in real time. The multi-dimensional status information includes but is not limited to the physical structure, operating status (such as inventory, material flow speed, equipment operating status) and external environmental information (such as temperature, humidity, weather conditions) of the system. Then, the collected multi-dimensional status information is cleaned, noise and outliers are removed, and the data is normalized to obtain pre-processed multi-dimensional status information. The pre-processed multi-dimensional status information is reduced in dimension using the PCA method. By calculating the covariance matrix, eigenvalues ​​and eigenvectors of each dimensional data, the first several principal components (i.e., the dimensions with larger eigenvalues ​​corresponding to the eigenvectors) are selected as key status indicators. These key status indicators can retain the information of the original data to the greatest extent while reducing the complexity of the data. Then, after extracting the key status indicators, this embodiment constructs a causal relationship model between the risk probability distribution and the emergency material reserve status based on the principle of the structural equation model (SEM). Specifically, This embodiment needs to determine the latent variables and explicit variables in the causal relationship model. In this causal relationship model, the risk level is used as the latent variable, the inventory, the material flow rate and the temperature are used as the explicit variables, the path relationship between the risk level and these explicit variables is set, and the parameter estimation and model verification are performed using statistical software. Finally, a model that can accurately reflect the causal relationship between the risk probability distribution and the emergency material reserve status is obtained. The simulation environment is built using simulation software (such as MATLAB / Simulink, AnyLogic, etc.), and the latent variables and explicit variables in the causal relationship model are converted into parameters and variables in the simulation model. The simulation parameters under different risk probability distribution conditions are set according to the actual situation. These parameters may include the specific value of the risk level, the initial value of the key status indicator and the simulation time, etc. The simulation model is run to observe the state changes of the emergency material reserve system under different risk probability distribution conditions, and the corresponding simulation results are obtained. The simulation results include time series data of the system state, the trend and law of the state change, etc.

[0085] After obtaining the simulation results, this embodiment uses Neo4j and other knowledge graph construction tools to construct an emergency material reserve knowledge graph, takes the emergency material reserve system status, risk probability distribution and their mutual relationships as entities and relationships in the graph, and assigns unique identifiers and attributes to each entity and relationship, fills the system status data, risk probability distribution data and their mutual relationship data in the simulation results into the knowledge graph, and optimizes the knowledge graph to ensure the consistency and integrity of the data, and finally obtains a complete knowledge graph containing the emergency material reserve system status, risk probability distribution and their mutual relationships. After constructing the emergency material reserve knowledge graph, this embodiment uses digital twin technology to construct an emergency material reserve digital twin model based on the emergency material reserve knowledge graph, specifically mapping the entities and relationships in the knowledge graph to the objects and relationships in the digital twin model, ensuring that the digital twin model can reflect the state changes and risk probability distribution of the emergency material reserve system in real time, and verifying and optimizing the model to ensure its accuracy and reliability. Through the emergency material reserve digital twin model, this embodiment can monitor the state changes of the system in real time, warn of potential risks, and provide scientific decision-making suggestions for decision makers.

[0086] In a specific implementation, the emergency material reserve digital twin model includes a state acquisition interface, an action execution interface, and a reward calculation interface. The functions of each interface are specifically described as follows:

[0087] The status acquisition interface is used to transmit the current key status information in the emergency material reserve digital twin model to the reinforcement learning algorithm in real time through an application programming interface (API) or direct access to a message queue, so that the algorithm can perform strategy analysis and decision-making; among which, the current key status information includes but is not limited to real-time disaster conditions, material reserves, logistics and transportation status, etc.

[0088] The action execution interface is used to execute the action instructions output by the reinforcement learning algorithm according to the optimal action instructions output by the reinforcement learning algorithm module, by calling the application programming interface (API) of the emergency material reserve digital twin model or triggering a preset specific event. This action execution interface ensures that the strategy of the reinforcement learning algorithm can be reflected in the simulation environment of the digital twin model in real time and accurately, thereby realizing real-time adjustment and optimization of material reserves and scheduling strategies.

[0089] The reward calculation interface is used to calculate the corresponding reward signal using the reward function according to the state changes in the digital twin model of emergency material reserves and the action instructions output by the reinforcement learning algorithm, and feed the reward signal back to the reinforcement learning algorithm through the interface. The reward signal can accurately reflect the contribution of the current action to achieving the optimization goal of emergency material reserves.

[0090] This embodiment realizes seamless connection and efficient interaction between the digital twin model of emergency material reserves and the reinforcement learning algorithm by designing the above-mentioned three interfaces: state acquisition interface, action execution interface and reward calculation interface, so as to adjust and optimize the emergency material reserve and scheduling strategy in real time and accurately according to the disaster situation and material reserve status.

[0091] S4. Introduce the reinforcement learning algorithm into the digital twin model of emergency material reserves, and interactively learn by simulating the material dispatch and reserve adjustment process under disaster scenarios to obtain the optimal emergency material reserve strategy.

[0092] In this embodiment, the step of introducing the reinforcement learning algorithm into the emergency material reserve digital twin model, interactively learning by simulating the material dispatch and reserve adjustment process under disaster scenarios, and obtaining the optimal emergency material reserve strategy includes:

[0093] In the digital twin model of emergency material reserves, disaster uncertainty and material supply interruption risk are taken as random terms, and a state transition model is constructed using the Markov decision process;

[0094] The state transition model is used to simulate the random changes in the probability of disaster occurrence and the risk of material supply under the current environmental state, and generate a state transition probability distribution;

[0095] According to the state transition probability distribution, a next environment state is selected using a Monte Carlo sampling method;

[0096] Construct a reward function with the goal of minimizing the risk of material shortage, maximizing material utilization efficiency, and minimizing material transportation costs;

[0097] Running a reinforcement learning agent in the emergency material reserve digital twin model so that the reinforcement learning agent uses a strategy network to calculate the action probability distribution of different actions according to the current environmental state and the reward function; in this embodiment, the strategy network is a multilayer perceptron;

[0098] An action is selected according to the action probability distribution, and the material dispatching and reserve adjustment process in a disaster scenario is simulated in combination with the next environmental state. Through multiple iterations and strategy updates, the optimal emergency material reserve strategy is obtained.

[0099] Among them, in the digital twin model of emergency material reserves, the steps of taking disaster uncertainty and material supply interruption risk as random items and using the Markov decision process to construct a state transition model include:

[0100] In the digital twin model of emergency material reserves, the state space and action space in the Markov decision process are defined; wherein the state space includes all disaster types, disaster degrees and material supply status, and the action space includes all material allocation and reserve decisions;

[0101] According to the state space and action space in the Markov decision process, the generator in the generative adversarial network is used to simulate the uncertainty of disasters and the risk of interruption of material supply, and generate state simulation samples;

[0102] According to the state simulation samples, the Markov decision process is used to simulate the state transition between different state simulation samples and build a state transition model.

[0103] Specifically, this embodiment defines the state space in the digital twin model of emergency material reserves. The state space in the digital twin model of emergency material reserves includes key parameters such as the severity of the disaster, the amount of material reserves, the material demand, and the material transportation capacity. At the same time, the influence of random factors on the state transition is considered. For example, the severity of the disaster may change randomly due to factors such as weather changes and geological conditions. This embodiment takes disaster uncertainty and the risk of material supply interruption as random items to construct a state transition model. In the process of constructing the state transition model, this embodiment introduces a generator in a generative adversarial network (GAN) to simulate disaster uncertainty and the risk of material supply interruption. The generator generates random state simulation samples based on the input state and action information. These samples reflect the uncertainty of disasters and material supply, and use the Markov decision process (MDP) to simulate the transition between different states. Specifically, this embodiment uses the principle of the Markov decision process to simulate the transition process from the current state to the possible successor state based on the generated state simulation samples. Through multiple simulations, a state transition model is constructed. The state transition model can reflect the impact of disasters and material supply changes on state transition.

[0104] This embodiment uses the constructed state transition model to simulate the random changes in the probability of disaster occurrence and material supply risks under the current environmental state. According to the simulation results, the probability of transferring from the current state to all other possible states is calculated to form a state transition probability distribution. According to the generated state transition probability distribution, the Monte Carlo sampling method is used to select the next environmental state, and the transition process from the current state to the next state is recorded for subsequent analysis and learning. Assuming that the state transition probability distribution is [0.2, 0.5, 0.3], a state is randomly selected as the next environmental state through the Monte Carlo sampling method. For example, if the sampling result is 0.4, the state with a probability of 0.5 is selected as the next state.

[0105] Next, this embodiment determines the goal of reinforcement learning, that is, minimizing the risk of material shortages, maximizing the efficiency of material utilization, and minimizing the cost of material transportation. The reward function is set according to the goal of reinforcement learning, so that a positive reward is obtained when the goal is achieved, and a negative reward is obtained when the goal is not achieved. For example, if the risk of material shortages is reduced, the efficiency of material utilization is improved, and the cost of material transportation is reduced, a positive reward is given; otherwise, a negative reward is given. In this embodiment, the reward function includes a material shortage risk sub-reward function, a material utilization efficiency sub-reward function, and a material transportation cost sub-reward function. The reward function is specifically:

[0106]

[0107] In the formula, is the reward function; N is the number of sub-reward functions contained in the reward function; i is the sub-reward function index; is the i-th sub-reward function.

[0108] Among them, the material shortage risk is usually related to the gap between the material demand and the reserve. When the material reserve cannot meet the demand, the shortage risk increases. In order to quantify this risk, this embodiment sets the material shortage risk sub-reward function as follows:

[0109]

[0110] In the formula, is the material shortage risk sub-reward function; is a positive coefficient used to adjust the sensitivity of rewards; D is the demand for materials; S is the reserve of materials.

[0111] The material utilization efficiency measures the contribution of the used materials to meeting the demand. In order to improve the utilization efficiency, this embodiment expects that the used materials can meet the demand as much as possible while reducing waste. Therefore, the material utilization efficiency sub-reward function is specifically:

[0112]

[0113] In the formula, is the material utilization efficiency sub-reward function; is a positive coefficient; The demand for materials; The amount of materials used.

[0114] Material transportation cost is an important part of emergency material storage and dispatch. In order to reduce transportation cost, this embodiment sets the material transportation cost sub-reward function as follows:

[0115]

[0116] In the formula, is the material transportation cost sub-reward function; is a positive coefficient; is the maximum transportation cost; C is the total cost of material transportation.

[0117] At the same time, this embodiment constructs and initializes a policy network. In this embodiment, the policy network adopts a multi-layer perceptron (MLP) structure. The policy network is used to calculate the action probability distribution of different actions. In this embodiment, the current environmental state is input into the policy network to use the policy network to calculate the action probability distribution of different actions. For example, this embodiment sets the policy network to be a three-layer MLP, the input layer receives the current environmental state (such as disaster type, disaster degree, material supply status, etc.), the hidden layer contains multiple neurons for feature extraction and transformation, and the output layer outputs the action probability distribution of different actions. According to the calculated action probability distribution, an action is selected as the decision of the current intelligent agent. According to the selected action, combined with the next environmental state, the material scheduling and reserve adjustment process after executing the action in the next environmental state is simulated to obtain A new environmental state. For example, if the selected action is to increase material reserves, the new environmental state after increasing the material reserves is simulated, including changes in the amount of material reserves, reduction in the risk of material shortages, etc. The rewards and state transitions in each iteration are recorded. According to the recorded rewards and state transitions, the weight parameters of the strategy network are updated using the back propagation algorithm so that the agent can learn a better strategy. The above process is repeated. Through multiple iterative learning, the agent gradually learns to gradually adjust the material reserves and scheduling strategies under different disaster scenarios to minimize the risk of material shortages, maximize material utilization efficiency, and minimize material transportation costs until the convergence condition is reached. The convergence condition can be that the change in the reward value is lower than the preset threshold or the number of iterations reaches the upper limit. At this time, the strategy obtained is the optimal emergency material reserve strategy.

[0118] The embodiment of the present invention provides a dynamic optimization method for emergency material reserves based on reinforcement learning. The method obtains the emergency material demand prediction result through a machine learning algorithm according to the disaster scenario data and emergency material consumption data of the target power facility area; based on the emergency material demand prediction result, combined with the historical disaster frequency and material supply uncertainty, the risk probability distribution of the emergency material reserve is quantified by using the Monte Carlo simulation method; according to the risk probability distribution and the multi-dimensional real-time status information of the emergency material reserve system, a digital twin model of emergency material reserves is constructed; the reinforcement learning algorithm is introduced into the digital twin model of emergency material reserves, and interactive learning is performed by simulating the material dispatching and reserve adjustment process under the disaster scenario to obtain the optimal emergency material reserve strategy. Compared with the prior art, the method not only improves the accuracy of material demand prediction, but also enhances the flexibility and response speed of the material reserve plan by combining digital twin technology and reinforcement learning algorithm, and can quickly and accurately provide the required emergency materials, avoiding waste or shortage of resources, thereby significantly improving the efficiency of rescue work.

[0119] It should be noted that the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0120] In one embodiment, Figure 2 As shown, an embodiment of the present invention provides a dynamic optimization system for emergency material reserves based on reinforcement learning, and the system includes:

[0121] The demand forecasting module 101 is used to obtain the emergency material demand forecasting result through a machine learning algorithm according to the disaster scenario data and emergency material consumption data of the target power facility area;

[0122] The risk assessment module 102 is used to quantify the risk probability distribution of emergency material reserves by using a Monte Carlo simulation method based on the emergency material demand forecast result, combined with the historical disaster frequency and material supply uncertainty;

[0123] A model building module 103 is used to build a digital twin model of emergency material reserves according to the risk probability distribution and multi-dimensional real-time status information of the emergency material reserve system;

[0124] The optimization learning module 104 is used to introduce the reinforcement learning algorithm into the emergency material reserve digital twin model, and interactively learn through simulating the material dispatch and reserve adjustment process under disaster scenarios to obtain the optimal emergency material reserve strategy.

[0125] For the specific definition of a dynamic optimization system for emergency material reserves based on reinforcement learning, please refer to the above-mentioned definition of a dynamic optimization method for emergency material reserves based on reinforcement learning, which will not be repeated here. Those of ordinary skill in the art will appreciate that the various modules and steps described in conjunction with the embodiments disclosed in this application can be implemented in hardware, software, or a combination of both. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0126] The embodiment of the present invention provides a dynamic optimization system for emergency material reserves based on reinforcement learning. The demand forecasting module of the system obtains the emergency material demand forecasting result through a machine learning algorithm according to the disaster scenario data and emergency material consumption data of the target power facility area; the risk assessment module is based on the emergency material demand forecasting result, combined with the historical disaster frequency and material supply uncertainty, and uses the Monte Carlo simulation method to quantify the risk probability distribution of emergency material reserves; the model construction module constructs an emergency material reserve digital twin model according to the risk probability distribution and the multi-dimensional real-time status information of the emergency material reserve system; the optimization learning module introduces the reinforcement learning algorithm into the emergency material reserve digital twin model, and interactively learns by simulating the material dispatching and reserve adjustment process under the disaster scenario to obtain the optimal emergency material reserve strategy. Compared with the prior art, the system not only improves the accuracy of material demand forecasting, but also enhances the flexibility and response speed of the material reserve plan by combining digital twin technology and reinforcement learning algorithm and other methods, and can quickly and accurately provide the required emergency materials, avoiding the waste or shortage of resources, thereby significantly improving the efficiency of rescue work.

[0127] The above-mentioned embodiments only express several preferred implementation modes of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in the technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be based on the protection scope of the claims.

Claims

1. A dynamic optimization method for emergency material reserves based on reinforcement learning, characterized in that: The following steps are involved: Based on the disaster scenario data and emergency material consumption data of the target power facility area, the emergency material demand forecast results are obtained through machine learning algorithms; Based on the emergency material demand forecast results, combined with the historical disaster frequency and material supply uncertainty, the risk probability distribution of emergency material reserves is quantified using the Monte Carlo simulation method; Constructing a digital twin model of emergency material reserves based on the risk probability distribution and multi-dimensional real-time status information of the emergency material reserve system; Introducing a reinforcement learning algorithm into the emergency material reserve digital twin model, interactively learning through simulating the material dispatch and reserve adjustment process under disaster scenarios, and obtaining the optimal emergency material reserve strategy; The step of obtaining the emergency material demand forecast result by machine learning algorithm based on the disaster scenario data and emergency material consumption data of the target power facility area includes: Collect disaster scenario data and material consumption data in the target power facility area; disaster scenario data includes disaster occurrence time, disaster type, disaster location and disaster intensity; material consumption data includes material consumption type and material consumption quantity; Preprocessing the disaster scene data and the material consumption data to obtain preprocessed disaster scene data and preprocessed material consumption data; Based on the pre-processed disaster scenario data and pre-processed material consumption data, the autoregressive integrated moving average model is used to extract the material demand characteristics under different disaster types; According to the material demand characteristics under different disaster types, the pre-built random forest prediction is used to obtain the emergency material demand prediction results; The step of quantifying the risk probability distribution of emergency material reserves by using a Monte Carlo simulation method based on the emergency material demand forecast result and in combination with the historical disaster frequency and material supply uncertainty includes: Using geographic information system technology to perform spatial statistical analysis on the disaster scenario data, generate a disaster hotspot map, and determine the spatial scope of high-risk areas based on the disaster hotspot map; According to the disaster intensity of the disaster scene data, a kernel density estimation method is used to construct probability density functions of different types of disaster intensities; Generate disaster scenario samples based on the spatial extent of high-risk areas, probability density functions of different types of disaster intensities, historical disaster frequencies, and uncertainty in material supply; According to the forecast results of emergency material demand and the current emergency material reserves, calculate the material gap information under each disaster scenario sample; Based on the material gap information under multiple disaster scenario samples, the Monte Carlo simulation method is used to quantify the risk probability distribution of emergency material reserves; The step of constructing a digital twin model of emergency material reserves according to the risk probability distribution and the multi-dimensional real-time status information of the emergency material reserve system includes: Collect multi-dimensional real-time status information of the emergency material reserve system, and perform dimensionality reduction processing on the multi-dimensional real-time status information through the principal component analysis method to extract key status indicators; According to the risk probability distribution and the key status indicators, a causal relationship model between the risk probability distribution and the emergency material reserve status is established using a structural equation model; The causal relationship model is simulated to predict the state change of the emergency material reserve system under different risk probability distribution conditions; According to the changes in the state of the emergency material reserve system under different risk probability distributions, a knowledge graph of emergency material reserve including the state of the emergency material reserve system, risk probability distribution and their mutual relationship is constructed; According to the emergency material reserve knowledge graph, a digital twin model of emergency material reserve is constructed using digital twin technology; The step of introducing the reinforcement learning algorithm into the emergency material reserve digital twin model, interactively learning by simulating the material dispatch and reserve adjustment process under disaster scenarios, and obtaining the optimal emergency material reserve strategy includes: In the digital twin model of emergency material reserves, disaster uncertainty and material supply interruption risk are taken as random terms, and a state transition model is constructed using the Markov decision process; The state transition model is used to simulate the random changes in the probability of disaster occurrence and the risk of material supply under the current environmental state, and generate a state transition probability distribution; According to the state transition probability distribution, a next environment state is selected using a Monte Carlo sampling method; Construct a reward function with the goal of minimizing the risk of material shortage, maximizing material utilization efficiency, and minimizing material transportation costs; Running a reinforcement learning agent in the emergency material reserve digital twin model so that the reinforcement learning agent calculates the action probability distribution of different actions using a policy network according to the current environment state and the reward function; An action is selected according to the action probability distribution, and the material dispatching and reserve adjustment process in a disaster scenario is simulated in combination with the next environmental state. Through multiple iterations and strategy updates, the optimal emergency material reserve strategy is obtained.

2. A dynamic optimization method for emergency material reserves based on reinforcement learning as claimed in claim 1, characterized in that: The step of generating disaster scenario samples according to the spatial scope of high-risk areas, probability density functions of different types of disaster intensities, historical disaster occurrence frequencies, and uncertainty in material supply includes: The probability density functions of different types of disaster intensity, the frequency of historical disasters and the uncertainty of material supply are divided into fuzzy sets, and the membership function is constructed for each fuzzy set using fuzzy logic algorithm; A random number generator is used to generate random samples based on the spatial scope of high-risk areas, probability density functions of different types of disaster intensities, historical disaster frequency, and uncertainty in material supply; the random samples include disaster intensity values, disaster event distribution, and material supply status; The generated random samples are used as input, and the probability of disaster occurrence of each random sample is calculated through the membership function; According to the probability of disaster occurrence, disaster scenario samples that meet a preset sample probability threshold are screened out from the random samples.

3. The method for dynamic optimization of emergency material reserves based on reinforcement learning as claimed in claim 1, characterized in that: The multi-dimensional real-time status information includes the physical structure, operation status and external environment information of the emergency material reserve system.

4. The method for dynamic optimization of emergency material reserves based on reinforcement learning as claimed in claim 1, characterized in that: In the digital twin model of emergency material reserves, the steps of taking disaster uncertainty and material supply interruption risk as random items and using the Markov decision process to construct a state transition model include: In the digital twin model of emergency material reserves, the state space and action space in the Markov decision process are defined; wherein the state space includes all disaster types, disaster degrees and material supply status, and the action space includes all material allocation and reserve decisions; According to the state space and action space in the Markov decision process, the generator in the generative adversarial network is used to simulate the uncertainty of disasters and the risk of interruption of material supply, and generate state simulation samples; According to the state simulation samples, the Markov decision process is used to simulate the state transition between different state simulation samples and build a state transition model.

5. The method for dynamic optimization of emergency material reserves based on reinforcement learning as claimed in claim 1, characterized in that: The policy network is a multi-layer perceptron.

6. A dynamic optimization system for emergency material reserves based on reinforcement learning, characterized in that: The system comprises: The demand forecasting module is used to obtain the emergency material demand forecasting results through machine learning algorithms based on the disaster scenario data and emergency material consumption data of the target power facility area; A risk assessment module is used to quantify the risk probability distribution of emergency material reserves by using a Monte Carlo simulation method based on the emergency material demand forecast results, combined with the historical disaster frequency and material supply uncertainty; A model building module, used to build a digital twin model of emergency material reserves based on the risk probability distribution and multi-dimensional real-time status information of the emergency material reserve system; An optimization learning module is used to introduce a reinforcement learning algorithm into the emergency material reserve digital twin model, and interactively learn by simulating the material dispatch and reserve adjustment process under disaster scenarios to obtain the optimal emergency material reserve strategy; Wherein, the demand forecasting module is specifically used for: Collect disaster scenario data and material consumption data in the target power facility area; disaster scenario data includes disaster occurrence time, disaster type, disaster location and disaster intensity; material consumption data includes material consumption type and material consumption quantity; Preprocessing the disaster scene data and the material consumption data to obtain preprocessed disaster scene data and preprocessed material consumption data; Based on the pre-processed disaster scenario data and pre-processed material consumption data, the autoregressive integrated moving average model is used to extract the material demand characteristics under different disaster types; According to the material demand characteristics under different disaster types, the pre-built random forest prediction is used to obtain the emergency material demand prediction results; The risk assessment module is specifically used for: Using geographic information system technology to perform spatial statistical analysis on the disaster scenario data, generate a disaster hotspot map, and determine the spatial scope of high-risk areas based on the disaster hotspot map; According to the disaster intensity of the disaster scene data, a kernel density estimation method is used to construct probability density functions of different types of disaster intensities; Generate disaster scenario samples based on the spatial extent of high-risk areas, probability density functions of different types of disaster intensities, historical disaster frequencies, and uncertainty in material supply; According to the forecast results of emergency material demand and the current emergency material reserves, calculate the material gap information under each disaster scenario sample; Based on the material gap information under multiple disaster scenario samples, the Monte Carlo simulation method is used to quantify the risk probability distribution of emergency material reserves; The model building module is specifically used for: Collect multi-dimensional real-time status information of the emergency material reserve system, and perform dimensionality reduction processing on the multi-dimensional real-time status information through the principal component analysis method to extract key status indicators; According to the risk probability distribution and the key status indicators, a causal relationship model between the risk probability distribution and the emergency material reserve status is established using a structural equation model; The causal relationship model is simulated to predict the state change of the emergency material reserve system under different risk probability distribution conditions; According to the changes in the state of the emergency material reserve system under different risk probability distributions, a knowledge graph of emergency material reserve including the state of the emergency material reserve system, risk probability distribution and their mutual relationship is constructed; According to the emergency material reserve knowledge graph, a digital twin model of emergency material reserve is constructed using digital twin technology; The optimization learning module is specifically used for: In the digital twin model of emergency material reserves, disaster uncertainty and material supply interruption risk are taken as random terms, and a state transition model is constructed using the Markov decision process; The state transition model is used to simulate the random changes in the probability of disaster occurrence and the risk of material supply under the current environmental state, and generate a state transition probability distribution; According to the state transition probability distribution, a next environment state is selected using a Monte Carlo sampling method; Construct a reward function with the goal of minimizing the risk of material shortage, maximizing material utilization efficiency, and minimizing material transportation costs; Running a reinforcement learning agent in the emergency material reserve digital twin model so that the reinforcement learning agent calculates the action probability distribution of different actions using a policy network according to the current environment state and the reward function; An action is selected according to the action probability distribution, and the material dispatching and reserve adjustment process in a disaster scenario is simulated in combination with the next environmental state. Through multiple iterations and strategy updates, the optimal emergency material reserve strategy is obtained.

Images