Multi-disaster coupling city comprehensive risk analysis method based on reinforcement learning

Abstract

The invention relates to a multi-disaster coupling city comprehensive risk analysis method based on reinforcement learning, in particular to the field of multi-disaster coupling city comprehensive risk analysis. According to the scheme, through multi-source heterogeneous data fusion and spatial-temporal feature extraction, a dynamic risk sensing environment is constructed, and a multi-agent reinforcement learning framework is utilized; an emergency resource allocation strategy is collaboratively optimized in a complex scene of multi-disaster coupling, and a coupling item and a hidden Markov look-ahead model are introduced, so that the system not only can analyze interaction among disasters, but also can predict a risk evolution trend, thereby guiding the strategy to avoid a high-risk mode in advance, and finally realizing conversion from passive response to active adaptation. And the accuracy, the collaboration and the robustness of emergency decision making are obviously improved.

Description

Technical Field

[0001] This invention relates to the field of multi-hazard coupled urban integrated risk analysis, and more specifically, to a multi-hazard coupled urban integrated risk analysis method based on reinforcement learning. Background Technology

[0002] In the field of urban integrated emergency management, the decision-making environment of emergency command centers becomes extremely complex and severe when faced with extreme scenarios involving the coupling of multiple disasters such as earthquakes, floods, and fires. Such scenarios typically manifest as chain reactions and superimposed effects of disasters. For example, a strong earthquake not only directly causes building collapses and casualties but may also trigger secondary floods caused by chemical plant leaks, gas pipeline explosions and fires, and dam damage. The internal constraints of this scenario are mainly reflected in the inherent limitations of the emergency resource system, including the limited absolute number and uneven geographical distribution of rescue teams, medical supplies, engineering machinery, and transport vehicles. External dynamic factors involve the real-time evolution of the disaster itself in time and space, the interruption of key transportation networks due to damage or congestion, the panic movement of affected people, and information silos caused by deteriorating communication conditions. The disaster coupling effect makes it difficult for traditional emergency resource allocation schemes based on single disaster types or static risk assessments to cope with such highly nonlinear and dynamically changing resource demands, thus creating an urgent need for intelligent decision support systems.

[0003] Currently, most technical solutions for dynamic allocation of emergency resources focus on operations research optimization algorithms and computational intelligence methods. Mainstream existing technologies include theoretical models based on linear programming or mixed-integer programming, which achieve resource allocation by solving for the extrema of the objective function under specific constraints. While these methods can obtain mathematically optimal solutions under certain assumptions, they heavily rely on precise, predefined mathematical models and parameters, making them difficult to adapt to the rapidly changing uncertainties at disaster sites. Another common type of technology is heuristic algorithms, such as genetic algorithms, simulated annealing algorithms, or greedy algorithms. These methods search for feasible solutions by simulating natural evolution or following local optima rules, demonstrating a certain flexibility in handling large-scale combinatorial optimization problems. However, regardless of whether it's a planning model or a heuristic algorithm... Their core decision-making logic is based on pre-set fixed rules, lacking the ability to continuously learn and self-optimize from massive historical rescue cases and real-time feedback data. They usually treat multiple disasters as independent events or perform simple linear superposition, failing to deeply depict the amplification or inhibition effects of the complex interaction and coupling mechanisms between different disasters on resource demand. In addition, when dealing with the influx of real-time data, these methods often cannot meet the minute or even second-level response requirements of emergency decision-making due to excessive computation time. Therefore, the core problem of existing technologies lies in their lack of adaptability and real-time performance, which cannot provide a fast, accurate and efficient resource allocation strategy for a dynamic urban system under the influence of multiple disaster coupling. This has become a key bottleneck in improving the comprehensive emergency management capabilities of cities. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a multi-hazard coupled urban integrated risk analysis method based on reinforcement learning, thereby solving the problems mentioned in the background art.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: specifically, it includes the following steps: Step S1: In response to the real-time acquisition of multi-source heterogeneous data including satellite remote sensing image data, IoT sensor data, social media text data and traffic flow data, a graph convolutional network based on attention mechanism is used to fuse the multi-source heterogeneous data, and a time series encoder is used to process historical data that are from the same source as the multi-source heterogeneous data and are arranged in chronological order, to construct a dynamic state space containing feature map information for characterizing the spatiotemporal dynamics of urban multi-hazard coupling risks. Step S2: Based on the dynamic state space constructed in Step S1, the policy network of multiple agents is pre-trained using the behavior cloning method and according to the pre-stored historical expert decision trajectory dataset. A global critic network is introduced to evaluate the value of joint actions among agents, thus completing the policy initialization of the multiple agents, where each agent corresponds to an emergency resource unit. Step S3: Place the multiple agents initialized in Step S2 into an environment defined by the dynamic state space output in Step S1, and perform online policy optimization using a multi-agent reinforcement learning framework based on the proximal policy optimization algorithm. In this framework, each agent's policy network acts as an executor and is guided by the global critic network introduced in Step S2. The global critic network contains a value function, which includes coupling terms for explicitly modeling the effects of multiple disaster interactions. Through optimization using the multi-agent reinforcement learning framework, an optimized emergency resource allocation strategy is output. Step S4: Based on the emergency resource allocation strategy and the value function of the global commentator network obtained in Step S3, the hidden Markov model is used to model the multi-hazard coupling propagation path. The predicted probability distribution of future disaster patterns is integrated into the value function of the global commentator network to form a risk-adjusted value function. Based on this, the strategy optimization process in Step S3 is fed back and guided to realize the dynamic adaptive adjustment of the emergency resource allocation strategy to the multi-hazard coupling risk. In a preferred embodiment, the specific operation of fusing the multi-source heterogeneous data using an attention-based graph convolutional network in step S1 is as follows: The target urban geographic region is discretized into multiple regular spatial grid units, each of which is defined as a node in a graph structure; based on spatial adjacency relationships and network topology connections of key urban infrastructure, connecting edges are established between the nodes to construct an undirected graph; For each node, at each time step, an initial multi-source feature concatenation vector for that node is constructed. The initial multi-source feature concatenation vector of each node is input into an attention-based graph convolutional network for computation to obtain the spatial fusion feature vector of each node; wherein, the computation process of the attention-based graph convolutional network includes: For any first node in the undirected graph that serves as the center node, calculate the attention weight between it and itself, as well as any neighboring node directly connected to it via a connecting edge. The calculation process for the attention weight is as follows: B1. Multiply a learnable first weight matrix by the initial multi-source feature concatenation vector of the first node and the initial multi-source feature concatenation vector of the neighboring node directly connected to the first node through the connecting edge, respectively, to obtain the first linear transformation vector of the first node and the second linear transformation vector of the neighboring node. B2. Concatenate the first linear transformation vector of the first node with the second linear transformation vector of the neighboring node to obtain a concatenated vector; B3. Perform a dot product operation between a learnable attention parameter vector and the concatenated vector to obtain a dot product scalar value; B4. Apply a linear rectifier function with a preset negative slope to the dot product scalar value to obtain an intermediate scalar value; B5. Calculate the value of the exponential function with the natural constant e as the base and the intermediate scalar value as the exponent; B6. Calculate the sum of the exponential function values ​​corresponding to the first node and all its neighboring nodes directly connected by connecting edges; B7. Divide the exponential function values ​​corresponding to the first node and the neighboring nodes by the sum of the exponential function values, and the resulting quotient is the attention weight between the first node and the neighboring nodes. After obtaining the attention weights between the first node and itself and all its neighboring nodes, the calculation process of the spatial fusion feature vector of the first node is as follows: C1. For the first node itself and each of its neighboring nodes directly connected by connecting edges, multiply the learnable first weight matrix with the initial multi-source feature concatenation vector of the node to obtain the linear transformation feature vector of the node. C2. Multiply the linear transformation feature vector of each node by the attention weight between that node and the first node to obtain the weighted feature vector of that node. C3. Summate all the weighted feature vectors to obtain a summed vector; C4. Apply a non-linear activation function to the summed vector to obtain the spatial fusion feature vector of the first node. Through the above process, the spatial fusion feature vector of all nodes in the undirected graph is calculated, thus completing the adaptive fusion of multi-source heterogeneous data in the spatial dimension.

[0006] In a preferred embodiment, the specific process of using a time series encoder to process historical data that shares the same source as the multi-source heterogeneous data and constructing a dynamic state space containing feature map information is as follows: For each node, its spatial fusion feature vectors output by the attention-based graph convolutional network at multiple consecutive historical time steps are arranged in chronological order to form the time-series feature vector group of that node. The time-series feature vector group of each node is sequentially input into a gated recurrent unit network; the gated recurrent unit network performs the following processing on each spatial fusion feature vector obtained in chronological order from the time-series feature vector group: The spatial fusion feature vector and the hidden state vector calculated and stored in the previous time step by the gated recurrent unit network are input together into the internal unit of the gated recurrent unit network. The internal unit outputs a new hidden state vector through its update gate and reset gate operations, and uses this new hidden state vector as the output of the current time step. At the same time, it is passed and saved as the hidden state vector of the previous time step when calculating the hidden state vector of the next time step. The hidden state vector output by the gated recurrent unit network after processing the spatial fusion feature vector of the last time step in the time series feature vector group is the temporal context encoding vector of the current time step. For each node, its spatial fusion feature vector output by the attention-based graph convolutional network at the current time step is concatenated with the temporal context encoding vector generated by the gated recurrent unit network at the current time step. Specifically, this concatenation operation involves: All elements of the spatial fusion feature vector of the current time step are arranged in their original order after all elements of the temporal context encoding vector of the current time step, thereby forming a new vector with increased dimension. This new vector is the final spatiotemporal feature vector of the node. The final spatiotemporal feature vectors of all nodes within the target area are organized one by one according to the physical arrangement order of the spatial grid units in the geographical area of ​​the target city to form a two-dimensional feature matrix. Each row of the matrix corresponds to a spatial grid node, and each column corresponds to a feature dimension of the final spatiotemporal feature vector. This two-dimensional feature matrix is ​​the specific data representation of the dynamic state space containing feature map information at the current time step, which is constructed to characterize the spatiotemporal dynamics of the city's multi-hazard coupling risk.

[0007] In a preferred embodiment, the specific process of pre-training the policy network of multiple agents using the behavior cloning method based on a pre-stored historical expert decision trajectory dataset in step S2 is as follows: Define multiple agents, each agent corresponding to an emergency resource unit; construct a policy network for each agent, the policy network of each agent takes the local observation vector of the agent as input and outputs the probability distribution of the agent choosing any one action from a set of preset actions; The specific method for constructing the local observation vector is as follows: From the two-dimensional feature matrix represented by the dynamic state space constructed in step S1, locate the row corresponding to the real-time position coordinates of the current agent, and extract the feature vectors of all rows within a preset range adjacent to this row as the center. Aggregate these feature vectors to form local area feature information that reflects the risk situation of the surrounding environment of the agent. Next, the local region feature information and the agent's own state attribute information are concatenated into a vector. The state attribute information includes the emergency resource type corresponding to the agent, the current remaining available amount of the resource, and the agent's real-time location coordinates. The vector obtained after concatenation is the agent's local observation vector. The pre-training process for a policy network involving multiple agents is as follows: Extract records from a pre-stored dataset of historical expert decision trajectories. Each record contains a historical global state, the joint action given by the expert for all agents in that historical global state, and the actual global reward signal fed back after the joint action is executed. For each record corresponding to each agent in the historical expert decision trajectory dataset, based on the method of constructing local observation vectors, the historical global state in the record is converted into the historical local observation vector corresponding to that agent, and the single action explicitly assigned to that agent in the joint action given by the expert is used as the supervision label; the internal parameters of the policy network of each agent are adjusted by optimizing a loss function. The calculation and optimization process of the loss function is as follows: Based on multiple sets of historical global states and joint action pairs given by experts, sampled from historical expert decision trajectory datasets, for each agent, the probability value of the joint action given by the sampled experts is calculated for the policy network under the condition that the input is the historical local observation vector transformed from the sampled historical global state. Then, the natural logarithm of the probability value is taken. By calculating the expected value of the above natural logarithm value on the sampled data and taking the negative value of the expected value, the loss function to be minimized is constructed. By iteratively optimizing the historical expert decision-making trajectory dataset multiple times, the policy network of each agent can initially learn to imitate the decision-making patterns of historical experts on similar emergency resource units under different environmental conditions.

[0008] In a preferred embodiment, the introduction of a global critic network to evaluate the value of joint actions among agents specifically includes: Construct a global commentator network. The input of the network consists of two parts: the first part is the current global state represented by the dynamic state space constructed in step S1, and the second part is the joint action composed of the actions output by all agent policy networks at the current moment. The output of the network is a scalar value, which represents the estimate of the long-term expected reward that can be obtained by performing the joint action in the current global state, i.e., global state-action value. The global commentator network employs a collaboratively perceived value decomposition structure. The specific process for calculating the global state-action value is as follows: First, the global commentator network contains a shared individual value evaluation subnetwork. For each agent, the subnetwork takes the current global state and the agent's single action as input and outputs an individual value item that represents the agent's individual contribution basis. Secondly, the global commentator network also contains a dynamic weight generation unit, which calculates a dynamic weight scalar for each agent based on the current global state through an attention mechanism. Furthermore, the global critic network contains an explicit cooperative evaluation subnetwork, which simultaneously receives the current global state and the aggregated representation of all agent actions as input, and outputs an explicit cooperative term. Finally, the global commentator network synthesizes the final global state-action value through the following steps: D1. Multiply the individual value of each agent by its corresponding dynamic weight scalar to obtain the weighted individual value of the agent. D2. Sum the weighted individual values ​​of all agents to obtain the total weighted individual value; D3. Multiply a preset collaboration strength coefficient by the explicit collaboration term to obtain the adjusted explicit collaboration term; D4. Add the weighted sum of individual values ​​to the adjusted explicit collaboration terms, and the resulting sum is the global state-action value.

[0009] In a preferred embodiment, the step of completing the policy initialization for multiple agents further includes: using a pre-stored historical expert decision trajectory dataset to perform supervised learning pre-training on the global commentator network as follows: Supervised learning of the global commentator network is performed using a pre-stored historical expert decision trajectory dataset. For each record in the historical expert decision trajectory dataset that includes the historical global state, the joint action given by the expert, and the actual global reward signal after executing the joint action, the historical global state and the joint action given by the expert are input into the global commentator network to obtain the global state-action value estimate calculated by the network. The actual global reward signal in the record, or the reward estimate calculated from the actual global reward signal based on the discount accumulation rule, is used as the training target value. The optimization objective of pre-training is to minimize the difference between the global state-action value estimate and the training target value. This optimization is achieved by constructing and minimizing a loss function. The loss function is calculated as follows: first, the difference between the global state-action value estimate and the target value is calculated; then, the square of the difference is calculated; finally, the average of this squared value is calculated over all relevant records in the historical expert decision trajectory dataset. The parameters of the global critic network are iteratively optimized using gradient descent, including the internal parameters of the individual value evaluation subnetwork, the dynamic weight generation unit, and the explicit collaborative evaluation subnetwork. Finally, a first set of initialization parameters consisting of the adjustable parameters of the policy network of all agents and a second set of initialization parameters consisting of the adjustable parameters of the global critic network are obtained. The first set of initialization parameters and the second set of initialization parameters together constitute the completed state of initializing the policies of multiple agents.

[0010] In a preferred embodiment, step S3 involves using a multi-agent reinforcement learning framework based on a proximal policy optimization algorithm for online policy optimization, including environmental interaction, experience collection, and advantage estimation, specifically: At each time step, the operation—namely, environmental interaction and experience collection—is executed sequentially through the following three sub-steps: E1. Input Processing: Receive the dynamic state space constructed and output by step S1, which represents the current multi-hazard coupled risk situation of the city; each agent independently selects an action to execute based on its own initialized policy network and the local observation vector extracted from the dynamic state space that reflects its surrounding environment and state; the actions executed by all agents at the current time step constitute a joint action. E2. Environmental Feedback Processing: After the joint action is executed in a simulated or actual urban emergency management environment, the environment will calculate and feed back an actual global reward signal based on a reward function designed to integrate multiple optimization objectives such as disaster loss reduction, rescue efficiency and resource utilization. At the same time, the environmental state changes, triggering step S1 to generate the dynamic state space of the next time step. E3. Experience storage processing: The data unit containing four types of information, namely the dynamic state space of the current time step, the joint action, the actual global reward signal, and the dynamic state space of the next time step, is stored as a complete experience transfer data in a preset experience playback buffer. During the optimization phase, a batch of experience transfer data is sampled from the experience replay buffer. For each record in the sampled experience transfer data, the global critic network introduced and initialized in step S2 is used to calculate the global state-action value estimate for executing the joint action in the current dynamic state space, as well as the state value estimate in the current dynamic state space and the state value estimate in the next dynamic state space. Based on the actual global reward signal, the current state value estimate, and the next state value estimate, the temporal difference error is calculated. Then, using the generalized dominance estimation algorithm, based on a series of temporal difference errors, the dominance function estimate used to evaluate the superiority or inferiority of the joint action relative to the average performance is calculated.

[0011] In a preferred embodiment, the optimization of the multi-agent reinforcement learning framework to output an optimized emergency resource allocation strategy includes policy network updates and global commentator network updates, specifically: For each piece of experience transfer data sampled from the experience replay buffer, and for each agent involved therein, the probability that the agent will output the action it performs based on its corresponding historical local observation vector under the current policy network parameters is calculated, which is the ratio of the probability that the agent will output the same action based on the same historical local observation vector under the old policy network parameters. This ratio is called the policy probability ratio. The old policy network parameters refer to the copy of the policy network parameters saved before the start of the current policy network parameter update iteration cycle. Based on the policy probability ratio and the advantage function estimate, a proximal policy optimization objective function is constructed for each agent; the objective function contains the sum of two terms: the first term is a conservative policy update term constructed based on the product of the pruning policy probability ratio and the advantage function estimate, and the second term is an entropy regularization term; The specific calculation method of the conservative strategy update term is as follows: calculate the product of the strategy probability ratio and the estimated value of the dominance function, and at the same time, calculate the product of a pruned strategy probability ratio and the estimated value of the dominance function, wherein the pruning process is as follows: restrict the strategy probability ratio to an interval centered at 1 and offset by a preset pruning hyperparameter; then, take the smaller value of the two products as the value of the conservative strategy update term. The entropy regularization term is the information entropy of the agent's current policy given its historical local observation vector, multiplied by a preset entropy coefficient, and then multiplied by a dynamic risk weight factor. The parameters of each agent's policy network are iteratively updated using the gradient ascent method by maximizing the objective function of the proximal policy of each agent. Meanwhile, the parameters of the global critic network are iteratively updated by minimizing a mean squared error loss function; Repeat the cycle of environmental interaction, experience collection, advantage estimation, policy network update, and global critic network update until any of the following preset convergence conditions are met: First, after multiple evaluation cycles are executed continuously in an independent verification environment, the average increase in reward obtained by all agents based on the latest strategy is less than a preset positive threshold. Second, the training process reaches the preset maximum number of training iterations or maximum number of time steps; Finally, the set of parameters of all agent policy networks that meet the convergence condition is used as the optimized emergency resource allocation strategy.

[0012] In a preferred embodiment, step S4 involves using a Hidden Markov Model to model the multi-hazard coupled propagation path and integrating the predicted probability distribution of future hazard patterns into the value function of the global commentator network. The specific process includes model building, training, and probability prediction. First, we construct a Hidden Markov Model, which is defined by the following five components: U1, a set of finite numbers of mutually exclusive hidden states, where each hidden state represents a distinct multi-hazard coupling mode; U2, a set of observation states, which is the dynamic state space output by step S1, characterizing the spatiotemporal dynamics of urban multi-hazard coupling risks. U3. A state transition probability matrix, which is a two-dimensional square matrix with the number of rows and columns equal to the number of hidden states. The element in the i-th row and j-th column of the state transition probability matrix represents the conditional probability of the hidden Markov model transitioning from the i-th hidden state to the j-th hidden state in any adjacent next time step. U4. An observation probability matrix, which describes the probability distribution of observing any observation state in the set of observation states given that the Hidden Markov Model is in a certain hidden state. U5. An initial state probability distribution, which is a vector whose dimension is equal to the number of hidden states, and the value of the i-th element in the vector represents the probability that the Hidden Markov Model is in the i-th hidden state at the initial time step. Secondly, the parameters of the constructed Hidden Markov Model are trained using historical disaster event data. The historical disaster event data includes a historical dynamic state space sequence generated by the method described in step S1, arranged in chronological order. A forward-backward algorithm is used to iteratively adjust the value of each element in the state transition probability matrix, the probability distribution parameters defined in the observation probability matrix, and the value of each element in the initial state probability distribution vector through the expectation-maximization method, so as to maximize the probability that the Hidden Markov Model generates the historical observation sequence, thereby completing the learning of the model parameters. Subsequently, at any current moment in online decision-making, based on the dynamic state space sequence traced back from the current moment to a past period, the probability of the Hidden Markov Model being in each hidden state at multiple future time steps is calculated using the trained Hidden Markov Model and forward algorithm, thus obtaining the probability distribution of future disaster modes.

[0013] In a preferred embodiment, the process of forming a risk-adjusted value function and using it to provide feedback and guidance to the strategy optimization process in step S3 specifically includes constructing a risk adjustment term, integrating the risk-adjusted value function, and providing strategy optimization feedback. First, a risk adjustment term is constructed, whose inputs include: the dynamic state space output by step S1 at the current moment, the joint action consisting of the actions output by all agents' policy networks at the current moment, and the probability distribution of future disaster modes; The calculation process for the risk adjustment item is as follows: W1. Initialization and parameter setting: Set a positive number less than or equal to 1 as the risk discount factor, and set a total number of future prediction steps; W2. Calculate for each future time step: For each future time step τ from 1 to the total number of future prediction steps, perform the following operations: W2.1 Expected Risk Calculation: Obtain the probability vector corresponding to the τ-th future time step in the probability distribution of future disaster patterns. This vector contains the probability of being in each hidden state at the τ-th future time step. Simultaneously, utilize a trainable conditional risk assessment function. This function takes the current dynamic state space, the current joint action, and a hidden state as input and outputs a risk estimate scalar representing the risk estimate under the assumption of that hidden state. Then, using the probability of being in each hidden state at the τ-th future time step as weight, perform a weighted summation of the conditional risk assessment values ​​for all possible hidden states. The result is the expected risk value at the τ-th future time step. W2.2 Risk Discount: Multiply the calculated expected risk value of the future step τ by the risk discount factor raised to the power of τ to obtain the discounted risk contribution value of the future step τ. W3. Cumulative Summation: All discounted risk contribution values ​​from step 1 to the total number of future prediction steps are summed, and the sum is the output value of the risk adjustment term. Then, the output value of the risk adjustment term is multiplied by a preset risk sensitivity coefficient, and then added to the original global state-action value estimate calculated by the global commentator network in step S3. The original global state-action value estimate is calculated by the global commentator network according to its value function. The sum obtained is the risk-adjusted global state-action value function. Finally, the risk-adjusted global state-action value function is fed back and applied to the strategy optimization process in step S3 to realize the dynamic adaptive adjustment of the emergency resource allocation strategy to the coupled risks of multiple disasters.

[0014] The beneficial effects of this invention are as follows: This solution constructs a dynamic risk perception environment through multi-source heterogeneous data fusion and spatiotemporal feature extraction. It utilizes a multi-agent reinforcement learning framework to collaboratively optimize emergency resource allocation strategies in complex scenarios with multiple coupled disasters. The introduced coupling terms and hidden Markov prospective models enable the system to not only analyze the interactions between disasters but also predict the trend of risk evolution, thereby guiding strategies to avoid high-risk modes in advance. Ultimately, it achieves a shift from passive response to proactive adaptation, significantly improving the accuracy, coordination, and robustness of emergency decision-making. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0018] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0019] Example 1 This embodiment provides, for example Figure 1 The method for multi-hazard coupled urban integrated risk analysis based on reinforcement learning is shown, and specifically includes the following steps: Step S1: In response to the real-time acquisition of multi-source heterogeneous data including satellite remote sensing imagery (providing land cover and thermal anomaly information), IoT sensor data (including time-series values ​​of meteorology, hydrology, and structural monitoring), social media text data (containing unstructured descriptions of disaster location, type, and impact), and traffic flow data (reflecting road network traffic status), a graph convolutional network based on an attention mechanism is used to fuse the multi-source heterogeneous data. A time-series encoder is used to process historical data that is from the same source as the multi-source heterogeneous data and is arranged in chronological order, to construct a dynamic state space containing feature map information to characterize the spatiotemporal dynamics of urban multi-hazard coupling risks. Step S2: Based on the dynamic state space constructed in Step S1, the policy network of multiple agents is pre-trained using the behavior cloning method and the pre-stored historical expert decision trajectory dataset. A global critic network is introduced to evaluate the value of joint actions among agents, thus completing the policy initialization of multiple agents, where each agent corresponds to an emergency resource unit. Step S3: Place the multiple agents initialized in Step S2 into an environment defined by the dynamic state space output in Step S1, and perform online policy optimization using a multi-agent reinforcement learning framework based on the proximal policy optimization algorithm. In this framework, each agent's policy network acts as an executor and is guided by the global critic network introduced in Step S2. The global critic network contains a value function, which includes coupling terms for explicitly modeling the effects of multiple disaster interactions. Through optimization by the multi-agent reinforcement learning framework, an optimized emergency resource allocation policy is output. Step S4: Based on the emergency resource allocation strategy and the value function of the global commentator network obtained in Step S3, a hidden Markov model is used to model the multi-hazard coupling propagation path. The predicted probability distribution of future disaster patterns is integrated into the value function of the global commentator network to form a risk-adjusted value function. Based on this, the strategy optimization process in Step S3 is fed back and guided to realize the dynamic adaptive adjustment of the emergency resource allocation strategy to the multi-hazard coupling risk.

[0020] In this embodiment, it is specifically necessary to explain the specific operation of fusing multi-source heterogeneous data using an attention-based graph convolutional network in step S1: The target city's geographical area is discretized into multiple regular spatial grid units, each defined as a node in a graph structure. Based on spatial adjacency relationships and the network topology connections of the city's critical infrastructure, connecting edges are established between nodes to construct an undirected graph. Preferably, the size of the spatial grid units is set according to the city's scale and the required risk analysis accuracy, for example, a square grid of 100m×100m to 500m×500m can be used. Critical infrastructure includes at least transportation networks, power networks, water supply networks, and communication networks, and their topology connections are used to define logical connecting edges between non-adjacent but functionally closely related grid units. For each node, at each time step, construct the initial multi-source feature concatenation vector for that node; the construction process of the initial multi-source feature concatenation vector includes: A1. First, the satellite remote sensing image data is processed, and its visual features are extracted using a convolutional neural network to form a first visual feature vector. The convolutional neural network is preferably a ResNet or VGG series network pre-trained on a large remote sensing dataset, which is used to extract deep features related to land cover, building outlines, and thermal anomaly areas. A2. Secondly, the IoT sensor data is processed and normalized to obtain the sensor numerical feature vector; the normalization process is preferably min-max normalization or Z-Score standardization to eliminate the influence of different physical dimensions. A3. Next, the social media text data is processed by using a natural language processing model to extract semantic features of the text. Based on the geographic location information contained in the text, the semantic features of the text are mapped to the corresponding spatial grid cells to form sentiment and event feature vectors. The natural language processing model is preferably BERT or RoBERTa, which is used to extract disaster keywords, public sentiment tendencies and geographic location entities in the text. The mapping is completed by matching the latitude and longitude coordinates identified in the text to the corresponding spatial grid cells. A4. Then, the traffic flow data is processed to extract the traffic speed and traffic flow features contained therein, forming a traffic state feature vector. A5. Finally, the first visual feature vector, sensor numerical feature vector, emotion and event feature vector, and traffic state feature vector are concatenated in sequence to generate the initial multi-source feature concatenation vector of the node. This concatenation operation unifies heterogeneous data into a fixed-dimensional vector representation, providing standardized input for subsequent graph convolution fusion. The initial multi-source feature concatenation vector of each node is input into an attention-based graph convolutional network for computation, resulting in a spatial fusion feature vector for each node. The computation process of the attention-based graph convolutional network includes: For any first node in an undirected graph that serves as the center node, calculate the attention weights between it and itself, as well as any neighboring node directly connected to it via an edge. The calculation process for the attention weights is as follows: B1. Multiply a learnable first weight matrix by the initial multi-source feature concatenation vector of the first node and the initial multi-source feature concatenation vector of the neighboring nodes directly connected to the first node through connecting edges, respectively, to obtain the first linear transformation vector of the first node and the second linear transformation vector of the neighboring nodes; the dimension of the first weight matrix is ​​determined according to the dimension of the initial multi-source feature concatenation vector and the expected hidden layer dimension. B2. Concatenate the first linear transformation vector of the first node with the second linear transformation vector of the neighboring nodes to obtain a concatenated vector. B3. Perform a dot product operation between a learnable attention parameter vector and the concatenated vector to obtain a dot product scalar value; B4. Apply a linear rectified function with a preset negative slope to the dot product scalar value to obtain an intermediate scalar value; the preset negative slope is preferably set between 0.01 and 0.2 to ensure that the function still has a small gradient for negative inputs and to prevent the neuron from "dying". B5. Calculate the value of an exponential function with the natural constant e as the base and an intermediate scalar value as the exponent; B6. Calculate the sum of the exponential function values ​​corresponding to the first node and all its neighboring nodes that are directly connected by connecting edges; B7. Divide the exponential function values ​​corresponding to the first node and its neighboring nodes by the sum of the exponential function values. The resulting quotient is the attention weight between the first node and its neighboring nodes. This normalization operation ensures that the sum of the attention weights associated with the first node is 1, so that the information aggregation has an interpretable weighted average form. After obtaining the attention weights between the first node and itself and all its neighboring nodes, the calculation process of the spatial fusion feature vector of the first node is as follows: C1. For the first node itself and each of its neighboring nodes directly connected by connecting edges, multiply the learnable first weight matrix with the initial multi-source feature concatenation vector of the node to obtain the linear transformation feature vector of the node; here, the first weight matrix in step B1 is reused to ensure the consistency of feature transformation and reduce the number of parameters. C2. Multiply the linear transformation feature vector of each node by the attention weight between that node and the first node to obtain the weighted feature vector of that node. C3. Summing all weighted feature vectors yields a summed vector. This summation operation aggregates information from neighboring nodes to the central node, and the aggregation weights are dynamically determined by the aforementioned attention mechanism. C4. Apply a nonlinear activation function to the summed vector. The resulting vector is the spatial fusion feature vector of the first node. The preferred nonlinear activation function is ReLU, ELU, or GELU, which introduces nonlinear transformation capabilities to enhance the model's representational ability. Through the above process, the spatial fusion feature vector of all nodes in the undirected graph is calculated, and the adaptive fusion of multi-source heterogeneous data in the spatial dimension is completed. The effect of this fusion mechanism is that it dynamically captures the mutual influence intensity of different geographical locations and different data sources under specific disaster coupling scenarios through attention weights, rather than using fixed convolution kernels, so as to more accurately model the non-uniform propagation and coupling effect of risks in space. The process involves using a time series encoder to process historical data that originates from the same source as multi-source heterogeneous data, and constructing a dynamic state space containing feature map information. The specific steps are as follows: For each node, the spatial fusion feature vectors output by the attention-based graph convolutional network over multiple consecutive historical time steps are arranged in chronological order to form the time series feature vector group of that node. The length T of multiple consecutive historical time steps is a preset hyperparameter, the value of which needs to balance the modeling depth of historical dependencies with the real-time performance of computation. For example, it can be set to 6 to 24 time steps, with each time step corresponding to an actual time interval such as 5 minutes or 1 hour. The time-series feature vector group of each node is sequentially input into a gated recurrent unit network; the gated recurrent unit network performs the following processing on each spatial fusion feature vector obtained in chronological order from the time-series feature vector group: The spatially fused feature vector and the hidden state vector calculated and stored in the previous time step by the gated recurrent unit network are input together into the internal unit of the gated recurrent unit network. This internal unit outputs a new hidden state vector through its update and reset gate operations, and uses this new hidden state vector as the output of the current time step. Simultaneously, it is passed and saved as the hidden state vector of the previous time step when calculating the hidden state vector of the next time step. The hidden state vector output by the gated recurrent unit network after processing the spatially fused feature vector of the last time step in the time series feature vector group is the temporal context encoding vector of the current time step. The dimension of the hidden state vector of the gated recurrent unit network is preset, determining its memory capacity. The update gate controls how much past information is retained, and the reset gate controls how much past information is used to calculate new candidate states. This gating mechanism enables it to effectively learn the long-term and short-term dependencies of node features over time, making it particularly suitable for time-dependent scenarios such as disaster development processes. The temporal context encoding vector encapsulates the evolution pattern and dependency information of the corresponding node features over multiple consecutive historical time steps. For each node, the spatial fusion feature vector output by the attention-based graph convolutional network at the current time step is concatenated with the temporal context encoding vector generated by the gated recurrent unit network at the current time step. This concatenation operation is specifically as follows: All elements of the spatial fusion feature vector at the current time step are arranged in their original order after all elements of the temporal context encoding vector at the current time step, thus forming a new vector with increased dimension. This new vector is the final spatiotemporal feature vector of the node. This concatenation operation organically combines the features reflecting the spatial state at the current instant with the temporal context features reflecting the historical evolution process, forming a complete description of the "state at the current moment and its origin" of each spatial grid unit. The final spatiotemporal feature vectors of all nodes within the target area are organized one by one according to the physical arrangement order of spatial grid units in the geographical region of the target city, forming a two-dimensional feature matrix. Each row of this matrix corresponds to a spatial grid node, and each column corresponds to a feature dimension of the final spatiotemporal feature vector. The physical arrangement order can be a row-first or column-first scanning order to ensure that the row index of each node in the matrix corresponds one-to-one with its spatial position. This two-dimensional feature matrix is ​​the specific data representation of the dynamic state space containing feature graph information at the current time step, which is used to characterize the spatiotemporal dynamics of the city's multi-hazard coupling risk. It fully encapsulates the spatial association of multi-source information after fusion by a graph convolutional network based on an attention mechanism, as well as the information after gated recurrent units. The temporal evolution pattern of node features in network coding, this dynamic state space, serves as the foundational environmental representation for subsequent observation and decision-making by reinforcement learning agents. Its structured and computable form enables agents to simultaneously perceive the spatial risk distribution across the entire city and its temporal trends, providing a crucial information foundation for making scientific emergency resource allocation decisions in complex dynamic environments with multiple coupled disasters. Compared to simple multi-source data stacking or independent spatiotemporal processing, this step generates a synergistic representation effect through the cascading and splicing of "graph attention space fusion" and "recurrent network temporal coding," enabling the final state space to both resolve the cross-spatial coupling of disaster risks and trace their temporal evolution. This is a key preliminary step for achieving high-precision dynamic risk analysis.

[0021] In this embodiment, the specific process of pre-training the policy network of multiple agents using the behavior cloning method based on the pre-stored historical expert decision trajectory dataset in step S2 is as follows: Define multiple agents, each agent corresponding to an emergency resource unit (such as a rescue team or an emergency supply vehicle); construct a policy network for each agent, the policy network of each agent takes the agent's local observation vector as input and outputs the probability distribution of the agent choosing any action from a set of preset actions; The specific method for constructing the local observation vector is as follows: From the two-dimensional feature matrix represented by the dynamic state space constructed in step S1, locate the row corresponding to the real-time position coordinates of the current agent, and extract the feature vectors of all rows within its adjacent preset range centered on this row. Aggregate these feature vectors to form local regional feature information reflecting the risk situation of the agent's surrounding environment. The adjacent preset range can be set according to the scale of urban blocks. For example, extract all feature vectors in a square area with a side length of 3 to 7 grids centered on the grid where the agent is located to ensure that the observation can cover the influence range of the agent's single-step action and the nearby risk sources. Next, the local area feature information and the agent's own state attribute information are concatenated into a vector. The state attribute information includes the emergency resource type corresponding to the agent, the current remaining available amount of the resource, and the agent's real-time location coordinates. The resulting vector is the agent's local observation vector. This construction method makes each agent's decision-making depend on both its local risk environment and its own unique resource status, thus realizing personalized perception of the global situation. The pre-training process for a policy network involving multiple agents is as follows: From the pre-stored historical expert decision trajectory dataset, records of multiple historical time steps are extracted. Each record contains a historical global state (corresponding to the dynamic state space of the historical moment), the joint action given by the expert for all agents in the historical global state (i.e. a complete set of cooperative operation instructions formulated and executed by the emergency command expert for all agents), and the actual global reward signal fed back after the joint action is executed. For each record corresponding to each agent in the historical expert decision trajectory dataset, based on the method of constructing local observation vectors, the historical global state in the record is converted into the historical local observation vector corresponding to that agent, and the single action explicitly assigned to that agent in the joint action given by the expert is used as the supervision label; the internal parameters of the policy network of each agent are adjusted by optimizing a loss function. The calculation and optimization process of the loss function is as follows: Based on multiple sets of historical global states and joint action pairs given by experts, sampled from historical expert decision trajectory datasets, for each agent, the probability value of the joint action given by the sampled experts is calculated for the policy network under the condition that the input is a historical local observation vector transformed from the sampled historical global state. Then, the natural logarithm of the probability value is taken. By calculating the expected value of the above natural logarithm value on the sampled data and taking the negative value of the expected value, a loss function that needs to be minimized is constructed. This loss function is essentially a maximum likelihood estimation. Its optimization process directly drives the policy network to reproduce the action choices of experts under the same observations, which is an effective means to quickly acquire basic decision-making capabilities. By iterating and optimizing the historical expert decision trajectory dataset multiple times, the policy network of each agent can initially learn to imitate the decision-making patterns of historical experts on similar emergency resource units under different environmental conditions. A global critic network is introduced to evaluate the value of joint actions among agents, specifically as follows: Construct a global commentator network. The input of the network consists of two parts: the first part is the current global state represented by the dynamic state space constructed in step S1, and the second part is the joint action composed of the actions output by all agent policy networks at the current moment. The output of the network is a scalar value, which represents the estimate of the long-term expected reward that can be obtained by performing the joint action in the current global state, i.e., global state-action value. The global commentator network employs a collaboratively perceived value decomposition structure. The specific process for calculating the global state-action value is as follows: First, the global commentator network contains a shared individual value evaluation subnetwork. For each agent, this subnetwork takes the current global state and the agent's single action as input and outputs an individual value term that represents the agent's individual contribution basis. Sharing the parameters of this subnetwork helps the model generalize and ensures that the evaluation scale of individual values ​​of different agents is consistent. Secondly, the global commentator network also includes a dynamic weight generation unit. This unit calculates a dynamic weight scalar for each agent based on the current global state through an attention mechanism. By design, the sum of the dynamic weight scalars of all agents is always one. This dynamic weight scalar is used to adjust the immediate proportion of the corresponding agent's individual value item in the total value at the global level. The attention mechanism enables the system to dynamically allocate decision weights based on real-time risk focus (such as the direction of fire spread), guiding resources to tilt towards the most critical links. Furthermore, the global commentator network contains an explicit cooperation evaluation subnetwork. This subnetwork receives the current global state and the aggregated representation of all agent actions as input, and outputs an explicit cooperation term. This explicit cooperation term is used to directly evaluate the additional cooperative value or conflict loss generated by the cooperation between multiple agent actions, which goes beyond the simple addition of individual values. This is one of the key innovations of this scheme, enabling the model to explicitly learn and evaluate the value increment brought by "cooperation" itself, such as the additional benefits generated by two rescue teams cooperating from different directions to speed up evacuation. Finally, the global commentator network synthesizes the final global state-action value through the following steps: D1. Multiply the individual value of each agent by its corresponding dynamic weight scalar to obtain the weighted individual value of the agent. D2. Sum the weighted individual values ​​of all agents to obtain the total weighted individual value; D3. Multiply a preset collaboration intensity coefficient by the explicit collaboration term to obtain the adjusted explicit collaboration term. The collaboration intensity coefficient is an adjustable hyperparameter used to balance the emphasis on individual rationality and team collaboration. Its typical initial value can be set to 0.5 and adjusted according to the needs of the task during training. D4. Add the weighted sum of individual values ​​to the adjusted explicit collaboration items, and the sum is the global state-action value; The policy initialization for multiple agents also includes: using a pre-stored dataset of historical expert decision trajectories, performing supervised learning pre-training on the global commentator network as follows: Supervised learning of the global commentator network is performed using a pre-stored historical expert decision trajectory dataset. For each record in the historical expert decision trajectory dataset that includes the historical global state, the joint action given by the expert, and the actual global reward signal after executing the joint action, the historical global state and the joint action given by the expert are input into the global commentator network to obtain the global state-action value estimate calculated by the network. The actual global reward signal in the record, or the reward estimate calculated from the actual global reward signal based on the discount accumulation rule, is used as the training target value. The discount factor in the discount accumulation rule is usually set between 0.9 and 0.99 to balance the importance of immediate reward and future reward. The optimization objective of pre-training is to minimize the difference between the global state-action value estimate and the training target value. This optimization is achieved by constructing and minimizing a loss function. The loss function is calculated as follows: first, the difference between the global state-action value estimate and the target value is calculated; then, the square of this difference is calculated; finally, the average of this squared value over all relevant records in the historical expert decision trajectory dataset is calculated. This is the mean squared error loss, a common objective in supervised learning, which enables the network's value estimate to approximate the true long-term returns corresponding to expert decisions. The optimization process iteratively optimizes all parameters of the global reviewer network using gradient descent, including the internal parameters of the individual value evaluation subnetwork, the dynamic weight generation unit, and the explicit collaborative evaluation subnetwork. The optimization process continues until the value of the loss function is lower than a preset convergence threshold or a predetermined number of training epochs is reached, for example, the loss value no longer decreases significantly for several consecutive epochs. This pre-training enables the global commentator network to learn the ability to accurately assess the value of joint actions based on historical expert demonstration data, especially to internalize the collaborative patterns inherent in expert decision-making. Combined with the pre-training of the policy network mentioned above, this step completes the initialization from "individual behavior imitation" to "team value understanding," providing a high-starting-point policy-value pair for subsequent online learning, which can significantly reduce blind exploration and accelerate overall convergence. Finally, a first set of initialization parameters consisting of the adjustable parameters of the policy networks of all agents and a second set of initialization parameters consisting of the adjustable parameters of the global critic network are obtained. The first and second sets of initialization parameters together constitute the completed state of initializing the policies of multiple agents and provide the initialized policy network and global critic network for subsequent steps.

[0022] In this embodiment, it is particularly important to explain that in step S3, a multi-agent reinforcement learning framework based on the proximal policy optimization algorithm is used for online policy optimization, including environmental interaction, experience collection, and advantage estimation, specifically as follows: At each time step, the operation—namely, environmental interaction and experience collection—is executed sequentially through the following three sub-steps: E1. Input Processing: Receive the dynamic state space constructed and output by step S1, which represents the current multi-hazard coupled risk situation of the city; each agent independently selects an action to execute based on its own initialized policy network and the local observation vector extracted from the dynamic state space that reflects its surrounding environment and state; the actions executed by all agents at the current time step constitute a joint action. E2. Environmental Feedback Processing: After the joint action is executed in a simulated or actual urban emergency management environment, the environment will calculate and feed back an actual global reward signal based on a reward function designed to integrate multiple optimization objectives such as disaster loss reduction, rescue efficiency, and resource utilization. Among them, disaster loss reduction can be quantified based on the estimated changes in the affected population or the reduction in direct economic losses; rescue efficiency can be assessed based on task response time or coverage of key areas; and resource utilization can be calculated by combining resource allocation accuracy and idle rate. At the same time, the environmental state changes, triggering step S1 to generate the dynamic state space of the next time step. E3. Experience Storage Processing: Data units containing four types of information—the dynamic state space of the current time step, the joint action, the actual global reward signal, and the dynamic state space of the next time step—are stored as complete experience transfer data in a preset experience replay buffer. The size of the experience replay buffer can be set according to the task complexity, for example, it can be set to accommodate 10,000 to 100,000 experience transfer data to ensure that there are sufficiently diverse samples for learning, and to periodically overwrite old data with new data. This experience replay buffer is used to accumulate historical interaction data to break the temporal correlation between data and provide a sampling source for subsequent optimization calculations. During the optimization phase, a batch of experience transfer data is sampled from the experience replay buffer. For each record in the sampled experience transfer data, the global critic network introduced and initialized in step S2 is used to calculate the global state-action value estimate for performing the joint action in the current dynamic state space, as well as the state value estimate in the current dynamic state space and the state value estimate in the next dynamic state space. Based on the actual global reward signal, the current state value estimate, and the next state value estimate, the temporal difference error is calculated. Then, using the generalized dominance estimation algorithm, based on a series of temporal difference errors, the dominance function estimate used to evaluate the superiority or inferiority of the joint action relative to the average performance is calculated. The time-series difference error is calculated as follows: multiply the actual global reward signal by a preset discount factor multiplied by the product of the next state value estimate, add the two together, and then subtract the current state value estimate. The difference is the time-series difference error. The discount factor is a constant between 0 and 1 (including 0 but not including 1), used to measure the value of future returns at the current moment. A typical value can be set to 0.99 to emphasize the importance of long-term returns. The Generalized Advantage (GAE) estimation algorithm introduces a GAE parameter that controls the trade-off between bias and variance. It performs an exponentially weighted summation of a series of future time-series difference errors starting from the current moment to obtain a more stable and lower-variable advantage estimate. The specific calculation process is as follows: First, the current time-series difference error is multiplied by the discount factor and the GAE parameter, then multiplied by the time-series difference error of the next moment. This is repeated, adding the square of the product of the discount factor and the GAE parameter, and multiplying by the time-series difference error of the next moment, and so on, for all future time-series difference errors. The difference is summed in an infinite number of terms, where the weight of each term is the corresponding power of the product of the discount factor and the GAE parameter. The GAE parameter is a constant between 0 and 1 (inclusive) used to adjust the balance between bias and variance in multi-step return estimation. It is usually taken between 0.92 and 0.98 to achieve a good trade-off between bias and variance. The final scalar value obtained by this weighted summation is the advantage function estimate. This advantage function estimate can more accurately reflect the long-term value of a specific joint action and provide a stable and efficient guiding signal for strategy updates. By optimizing the multi-agent reinforcement learning framework, an optimized emergency resource allocation strategy is output. This strategy involves updating the policy network and the global critic network, specifically: For each piece of experience transfer data sampled from the experience replay buffer, and for each agent involved therein, the probability that the agent will perform an action based on its corresponding historical local observation vector under the current policy network parameters is calculated as the ratio of the probability that the agent will perform the same action based on the same historical local observation vector under the old policy network parameters. This ratio is called the policy probability ratio. The old policy network parameters refer to the copy of the policy network parameters saved before the start of the current policy network parameter update iteration cycle. Based on the policy probability ratio and the advantage function estimate, a proximal policy optimization objective function is constructed for each agent. This objective function consists of the sum of two terms: the first term is a conservative policy update term constructed based on the product of the pruning policy probability ratio and the advantage function estimate, and the second term is an entropy regularization term. The conservative policy update term is calculated as follows: The product of the policy probability ratio and the estimated dominance function is calculated. Simultaneously, a product of the policy probability ratio and the estimated dominance function is calculated after pruning. The pruning process involves limiting the policy probability ratio to an interval centered at 1 and offset by a preset pruning hyperparameter. Then, the smaller of these two products is taken as the value of the conservative policy update term. The pruning hyperparameter is a preset small positive number used to control the maximum step size of the policy update, typically set to 0.1 to 0.3, for example, 0.2, to strike a balance between encouraging policy improvement and preventing performance crashes caused by overly rapid updates. The entropy regularization term is the information entropy of the agent's current policy given its historical local observation vector, multiplied by a preset entropy coefficient, and then multiplied by a dynamic risk weighting factor. The information entropy is used to measure the randomness of the policy. The entropy coefficient is a hyperparameter used to adjust the intensity of exploration incentives. Its initial value can be set according to the size of the action space and can be gradually decayed during training. The dynamic risk weighting factor is calculated based on the intensity of multi-hazard coupling risks represented by the historical dynamic state space recorded in the empirical transfer data. When the perceived coupling risk is high, the value of this factor increases, thereby enhancing the incentive effect of the entropy regularization term on exploration. For example, this factor can be positively correlated with the indicators representing the intensity or uncertainty of disaster interactions in the state space to achieve a risk-adaptive exploration strategy. By maximizing the proximal policy optimization objective function of each agent, the parameters of the policy network of each agent are iteratively updated using the gradient ascent method. This update process enables each agent to adjust its decision policy in the direction of obtaining a higher advantage estimate (i.e. better long-term returns) while ensuring that the update magnitude is controllable. Simultaneously, the parameters of the global commentator network are iteratively updated by minimizing a mean squared error loss function. The mean squared error loss function is calculated as follows: First, the difference between the global state-action value estimate output by the global commentator network and a target value is calculated. The target value is composed of the actual global reward signal plus a discount factor and the product of a target state value estimate. The target state value estimate is obtained by an independent target commentator network evaluating the dynamic state space of the next time step. The target commentator network has the same structure as the global commentator network, but its network parameters are not updated directly through gradient descent. Instead, they are periodically synchronized with the parameters of the global commentator network through a method called soft update. The specific synchronization mechanism is as follows: the parameters of the target commentator network are equal to the weighted sum of its own current parameters and the current parameters of the global commentator network. The weights are determined by a preset soft update coefficient, where the weight of the target commentator network's own current parameters is greater than the weight of the global commentator network's current parameters, making the target network parameters update slowly to maintain training stability. The soft update coefficient is usually set to a value close to 1, such as 0.995, to ensure that the target network parameters change smoothly and provide a stable learning target for value estimation. Then, the squared value of the difference is calculated; finally, the average of this squared value on a sampled batch of empirical transfer data is calculated to obtain the mean squared error loss function; by minimizing this mean squared error loss function, all parameters of the global commentator network are updated, including the parameters of the coupling term in its value function used to explicitly model the effects of multi-hazard interactions, as well as all other parameters in the global commentator network except for the coupling term; this update process enables the global commentator network, especially its coupling term, to continuously learn from the reward signals generated by actual interactions, and more accurately quantify the synergistic effects or conflicts generated by different agent actions in complex disaster coupling scenarios, thereby providing more accurate value guidance for policy optimization; Repeat the cycle of environment interaction, experience collection, advantage estimation, policy network update, and global critic network update until any of the following preset convergence conditions are met: First, if the average reward improvement obtained by all agents based on the latest policy is less than a preset positive threshold, such as 0.01, after multiple evaluation cycles are executed continuously in an independent verification environment, it indicates that the policy performance has stabilized. Second, the training process reaches the preset maximum number of training iterations or maximum number of time steps; Finally, the set of parameters of all agent policy networks that meet the convergence condition is used as the optimized emergency resource allocation strategy. This strategy is the final decision model refined through online interaction and learning. It can dynamically generate efficient and collaborative emergency resource scheduling schemes based on real-time perception of multi-hazard coupled risk situations, surpassing the initial imitation learning stage and possessing the ability to adapt to unforeseen complex scenarios.

[0023] In this embodiment, it is particularly important to explain step S4, which involves using a Hidden Markov Model to model the multi-hazard coupled propagation path and integrating the predicted probability distribution of future disaster patterns into the value function of the global commentator network. The specific process includes model building, training, and probability prediction. First, a hidden Markov model is constructed to characterize the dynamics of multi-hazard coupling. This model is defined by the following five components: U1, a set of finitely many mutually exclusive hidden states, where each hidden state represents a different multi-hazard coupling pattern, such as including but not limited to "no significant coupling", "flood-geological hazard chain", "high temperature-drought-fire" and other typical or historically common coupling scenarios. The number of hidden states K can be determined based on cluster analysis of historical disaster data, for example, set to 5 to 10. U2. A set of observation states, which is the dynamic state space output by step S1 that characterizes the spatiotemporal dynamics of urban multi-hazard coupling risk, or the characterization obtained after dimensionality reduction or feature extraction of the dynamic state space. U3. A state transition probability matrix, which is a two-dimensional square matrix with the number of rows and columns equal to the number of hidden states. The element in the i-th row and j-th column of the state transition probability matrix represents the conditional probability of the Hidden Markov Model transitioning from the i-th hidden state to the j-th hidden state in any adjacent next time step. U4. An observation probability matrix, which describes the probability distribution or probability density of any observation state in the set of observation states under the condition that the Hidden Markov Model is in a certain hidden state. For observation states with continuous values ​​(i.e., dynamic state space), the observation probability matrix can be fitted with a Gaussian mixture model to characterize the probability density distribution of the observation values ​​under a given hidden state. U5. An initial state probability distribution, which is a vector whose dimension is equal to the number of hidden states. The value of the i-th element in the vector represents the probability that the Hidden Markov Model is in the i-th hidden state at the initial time step. The initial state probability distribution can be obtained by statistically estimating the initial frequency of various coupling modes in historical data. Secondly, the parameters of the constructed Hidden Markov Model are trained using historical disaster event data. The historical disaster event data includes a historical dynamic state space sequence generated by the method in step S1, arranged in chronological order. A forward-backward algorithm is used to iteratively adjust the value of each element in the state transition probability matrix, the probability distribution parameters defined in the observation probability matrix, and the value of each element in the initial state probability distribution vector through the expectation-maximization method, so as to maximize the probability that the Hidden Markov Model generates historical observation sequences, thereby completing the learning of the model parameters. The iterative process of the expectation-maximization method continues until the improvement of the model's log-likelihood function value calculated between two adjacent iterations is less than a preset first threshold, or the preset maximum number of iterations is reached. The first threshold can be set to 0.001, for example, and the maximum number of iterations can be set to 50. Subsequently, at any given moment in online decision-making, based on the dynamic state space sequence traced back from the current moment to a past period, the probability of the Hidden Markov Model (HMM) being in each hidden state at multiple future time steps is calculated using the trained HMM and the forward algorithm, thus obtaining the probability distribution of future disaster modes; specifically: V1. Input composition: Based on the dynamic state space sequence that traces back L consecutive time steps from the current moment, an observation sequence window of length L is formed, where L is a preset positive integer. Its value needs to balance the sufficiency of historical information and the real-time performance of the calculation. For example, for data analysis with a granularity of days, L can be set to 7, which represents the data of one week. V2. Probabilistic Inference Calculation: For the τ-th time step in the future, where τ ranges from 1 to the preset total number of prediction steps T_future, and for the k-th hidden state in the set of hidden states, where k ranges from 1 to the total number of hidden states K, the forward algorithm is used to calculate the conditional probability that the Hidden Markov Model is in the k-th hidden state at the τ-th time step in the future, given all the parameters of the trained Hidden Markov Model and an observation sequence window of length L. V3. Output Generation: The conditional probabilities calculated for all τ (from 1 to T_future) and all k (from 1 to K) are organized into a two-dimensional probability matrix with T_future rows and K columns, serving as the probability distribution of future disaster modes. The total number of prediction steps T_future defines the model's look-ahead, for example, it can be set to 5, indicating a probability prediction of the evolution of disaster modes over the next 5 time steps. In this two-dimensional probability matrix, the element value in the τ-th row and k-th column is the calculated probability that the Hidden Markov Model will be in the k-th hidden state at the τ-th time step in the future. Each row of this matrix is ​​a probability distribution vector, representing the likelihood of various disaster coupling modes occurring at the corresponding future time step. A risk-adjusted value function is formed, and this function is used to provide feedback and guidance to the strategy optimization process in step S3. The specific process includes constructing risk adjustment terms, integrating the risk-adjusted value function, and providing strategy optimization feedback. First, a risk adjustment term is constructed, which is a trainable parameterized function whose inputs include: the dynamic state space output by step S1 at the current time, the joint action consisting of the actions output by all agents' policy networks at the current time, and the probability distribution of future disaster modes. This risk adjustment term is used to evaluate the additional long-term risk cost or risk aversion benefit brought about by various possible disaster coupling modes under the current state and current joint action. The calculation process for the risk adjustment item is as follows: W1. Initialization and parameter setting: Set a positive number less than or equal to 1 as the risk discount factor. Its typical value can be set to 0.95 to moderately discount the impact of future risks, and set a total number of future prediction steps. W2. Calculate for each future time step: For each future time step τ from 1 to the total number of future prediction steps, perform the following operations: W2.1 Expected Risk Calculation: Obtain the probability vector corresponding to the τ-th future time step from the probability distribution of future disaster patterns. This vector contains the probability of being in each hidden state at the τ-th future time step. Simultaneously, utilize a trainable conditional risk assessment function. This function takes the current dynamic state space, the current joint action, and a hidden state as input and outputs a risk estimate scalar representing the risk estimate under the assumption of that hidden state. This conditional risk assessment function can be constructed as a multilayer perceptron, and its parameters are initialized using the Xavier normal distribution method. Then, using the probability of being in each hidden state at the τ-th future time step as weight, the conditional risk assessment values ​​of all possible hidden states are weighted and summed. The result is the expected risk value at the τ-th future time step. W2.2 Risk Discount: Multiply the calculated expected risk value of the future step τ by the risk discount factor raised to the power of τ to obtain the discounted risk contribution value of the future step τ. W3. Cumulative Summation: The total sum of all discounted risk contribution values ​​from step 1 to the total number of future prediction steps is the output value of the risk adjustment term. Then, the output value of the risk adjustment term is multiplied by a preset risk sensitivity coefficient, which is used to balance the weight of immediate value and future risk aversion. Its value range can be, for example, between 0.5 and 2.0, and can be adjusted according to the actual risk preference. Then, it is added to the original global state-action value estimate calculated by the global commentator network in step S3. The original global state-action value estimate is calculated by the global commentator network according to its value function, which includes coupling terms for explicitly modeling the effects of multi-hazard interactions. The resulting sum is the risk-adjusted global state-action value function. Finally, the risk-adjusted global state-action value function is fed back and applied to the policy optimization process in step S3. Specifically, when calculating the advantage function estimate for policy network updates in step S3, the original global state-action value estimate of the global commentator network is no longer used to calculate the temporal difference objective. Instead, the risk-adjusted global state-action value function is used. This is the "alternative advantage estimation" mechanism, which injects forward-looking risk prediction information into policy evaluation, making the advantage estimate reflect not only the immediate benefits of the current action but also its long-term impact on potential future risks. Accordingly, the gradient of policy network parameter updates will be guided by the risk-adjusted global state-action value function. This is the "policy gradient guidance" mechanism, which drives the policy optimization process to not only pursue the maximization of immediate rewards but also encourages the selection of actions that perform well under multiple possible future risk scenarios. Robust and proactive preventative actions that can inhibit the evolution of high-risk coupling patterns are employed. Simultaneously, the trainable parameters within the risk adjustment term, along with the parameters of the global commentator network and the policy networks of each agent, are updated in the optimization loop of step S3. This is the "cooperative training" mechanism, ensuring that the goals of the risk perception module, decision-making module, and value assessment module are aligned and optimized together. When updating the policy network parameters, gradient pruning techniques can be used, such as limiting the norm of the policy gradient to within 1.0, to maintain training stability. Through the above methods, the calculated advantage function estimate will simultaneously reflect the immediate benefits of the current action and the long-term value of addressing the risk of future disaster coupling evolution. This guides the policy optimization process in step S3 towards proactively avoiding or mitigating future high-risk coupling patterns while pursuing immediate benefits, achieving dynamic adaptive adjustment of emergency resource allocation strategies to multi-hazard coupling risks.

[0024] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0025] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0026] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0027] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0028] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0029] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0030] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multi-hazard coupled urban integrated risk analysis method based on reinforcement learning, characterized in that, Specifically, the steps include the following: Step S1: In response to the real-time acquisition of multi-source heterogeneous data including satellite remote sensing image data, IoT sensor data, social media text data and traffic flow data, a graph convolutional network based on attention mechanism is used to fuse the multi-source heterogeneous data, and a time series encoder is used to process historical data that are from the same source as the multi-source heterogeneous data and are arranged in chronological order, to construct a dynamic state space containing feature map information for characterizing the spatiotemporal dynamics of urban multi-hazard coupling risks. Step S2: Based on the dynamic state space constructed in Step S1, the policy network of multiple agents is pre-trained using the behavior cloning method and according to the pre-stored historical expert decision trajectory dataset. A global critic network is introduced to evaluate the value of joint actions among agents, thus completing the policy initialization of the multiple agents, where each agent corresponds to an emergency resource unit. Step S3: Place the multiple agents initialized in Step S2 into an environment defined by the dynamic state space output in Step S1, and perform online policy optimization using a multi-agent reinforcement learning framework based on the proximal policy optimization algorithm. In this framework, each agent's policy network acts as an executor and is guided by the global critic network introduced in Step S2. The global critic network contains a value function, which includes coupling terms for explicitly modeling the effects of multiple disaster interactions. Through optimization using the multi-agent reinforcement learning framework, an optimized emergency resource allocation strategy is output. Step S4: Based on the emergency resource allocation strategy and the value function of the global commentator network obtained in Step S3, a hidden Markov model is used to model the multi-hazard coupling propagation path. The predicted probability distribution of future disaster patterns is integrated into the value function of the global commentator network to form a risk-adjusted value function. Based on this, the strategy optimization process in Step S3 is fed back and guided to realize the dynamic adaptive adjustment of the emergency resource allocation strategy to the multi-hazard coupling risk.

2. The multi-hazard coupled urban integrated risk analysis method based on reinforcement learning according to claim 1, characterized in that: In step S1, the specific operation of fusing the multi-source heterogeneous data using a graph convolutional network based on an attention mechanism is as follows: The target urban geographic region is discretized into multiple regular spatial grid units, each of which is defined as a node in a graph structure; based on spatial adjacency relationships and network topology connections of key urban infrastructure, connecting edges are established between the nodes to construct an undirected graph; For each node, at each time step, an initial multi-source feature concatenation vector for that node is constructed. The initial multi-source feature concatenation vector of each node is input into an attention-based graph convolutional network for computation to obtain the spatial fusion feature vector of each node; wherein, the computation process of the attention-based graph convolutional network includes: For any first node in the undirected graph that serves as the center node, calculate the attention weight between it and itself, as well as any neighboring node directly connected to it via a connecting edge. The calculation process for the attention weight is as follows: B1. Multiply a learnable first weight matrix by the initial multi-source feature concatenation vector of the first node and the initial multi-source feature concatenation vector of the neighboring node directly connected to the first node through the connecting edge, respectively, to obtain the first linear transformation vector of the first node and the second linear transformation vector of the neighboring node. B2. Concatenate the first linear transformation vector of the first node with the second linear transformation vector of the neighboring node to obtain a concatenated vector; B3. Perform a dot product operation between a learnable attention parameter vector and the concatenated vector to obtain a dot product scalar value; B4. Apply a linear rectifier function with a preset negative slope to the dot product scalar value to obtain an intermediate scalar value; B5. Calculate the value of the exponential function with the natural constant e as the base and the intermediate scalar value as the exponent; B6. Calculate the sum of the exponential function values ​​corresponding to the first node and all its neighboring nodes directly connected by connecting edges; B7. Divide the exponential function values ​​corresponding to the first node and the neighboring nodes by the sum of the exponential function values, and the resulting quotient is the attention weight between the first node and the neighboring nodes. After obtaining the attention weights between the first node and itself and all its neighboring nodes, the calculation process of the spatial fusion feature vector of the first node is as follows: C1. For the first node itself and each of its neighboring nodes directly connected by connecting edges, multiply the learnable first weight matrix with the initial multi-source feature concatenation vector of the node to obtain the linear transformation feature vector of the node. C2. Multiply the linear transformation feature vector of each node by the attention weight between that node and the first node to obtain the weighted feature vector of that node. C3. Summate all the weighted feature vectors to obtain a summed vector; C4. Apply a non-linear activation function to the summed vector to obtain the spatial fusion feature vector of the first node. Through the above process, the spatial fusion feature vector of all nodes in the undirected graph is calculated, thus completing the adaptive fusion of multi-source heterogeneous data in the spatial dimension.

3. The multi-hazard coupled urban integrated risk analysis method based on reinforcement learning according to claim 2, characterized in that: The specific process of using a time series encoder to process historical data that shares the same source as the multi-source heterogeneous data and constructing a dynamic state space containing feature map information is as follows: For each node, its spatial fusion feature vectors output by the attention-based graph convolutional network at multiple consecutive historical time steps are arranged in chronological order to form the time-series feature vector group of that node. The time-series feature vector group of each node is sequentially input into a gated recurrent unit network; the gated recurrent unit network performs the following processing on each spatial fusion feature vector obtained in chronological order from the time-series feature vector group: The spatial fusion feature vector and the hidden state vector calculated and stored in the previous time step by the gated recurrent unit network are input together into the internal unit of the gated recurrent unit network. The internal unit outputs a new hidden state vector through its update gate and reset gate operations, and uses this new hidden state vector as the output of the current time step. At the same time, it is passed and saved as the hidden state vector of the previous time step when calculating the hidden state vector of the next time step. The hidden state vector output by the gated recurrent unit network after processing the spatial fusion feature vector of the last time step in the time series feature vector group is the temporal context encoding vector of the current time step. For each node, its spatial fusion feature vector output by the attention-based graph convolutional network at the current time step is concatenated with the temporal context encoding vector generated by the gated recurrent unit network at the current time step. Specifically, this concatenation operation involves: All elements of the spatial fusion feature vector of the current time step are arranged in their original order after all elements of the temporal context encoding vector of the current time step, thereby forming a new vector with increased dimension. This new vector is the final spatiotemporal feature vector of the node. The final spatiotemporal feature vectors of all nodes in the target area are organized one by one according to the physical arrangement order of the spatial grid units in the geographical area of ​​the target city to form a two-dimensional feature matrix, wherein each row of the matrix corresponds to a spatial grid node and each column corresponds to a feature dimension of the final spatiotemporal feature vector. This two-dimensional feature matrix is ​​the specific data representation of the dynamic state space, which contains feature map information and is used to characterize the spatiotemporal dynamics of urban multi-hazard coupling risks, at the current time step.

4. The multi-hazard coupled urban integrated risk analysis method based on reinforcement learning according to claim 3, characterized in that: In step S2, the specific process of pre-training the policy network of multiple agents based on a pre-stored historical expert decision trajectory dataset using the behavior cloning method is as follows: Define multiple agents, each agent corresponding to an emergency resource unit; construct a policy network for each agent, the policy network of each agent takes the local observation vector of the agent as input and outputs the probability distribution of the agent choosing any one action from a set of preset actions; The specific method for constructing local observation vectors is as follows: From the two-dimensional feature matrix represented by the dynamic state space constructed in step S1, locate the row corresponding to the real-time position coordinates of the current agent, and extract the feature vectors of all rows within a preset range adjacent to this row as the center. Aggregate these feature vectors to form local area feature information that reflects the risk situation of the surrounding environment of the agent. Next, the local region feature information and the agent's own state attribute information are concatenated into a vector. The state attribute information includes the emergency resource type corresponding to the agent, the current remaining available amount of the resource, and the agent's real-time location coordinates. The vector obtained after concatenation is the agent's local observation vector. The pre-training process for a policy network involving multiple agents is as follows: Extract records from a pre-stored dataset of historical expert decision trajectories. Each record contains a historical global state, the joint action given by the expert for all agents in that historical global state, and the actual global reward signal fed back after the joint action is executed. For each record corresponding to each agent in the historical expert decision trajectory dataset, based on the method of constructing local observation vectors, the historical global state in the record is converted into the historical local observation vector corresponding to that agent, and the single action explicitly assigned to that agent in the joint action given by the expert is used as the supervision label; the internal parameters of the policy network of each agent are adjusted by optimizing a loss function. The calculation and optimization process of the loss function is as follows: Based on multiple sets of historical global states and joint action pairs given by experts, sampled from historical expert decision trajectory datasets, for each agent, the probability value of the joint action given by the sampled experts is calculated for the policy network under the condition that the input is the historical local observation vector transformed from the sampled historical global state. Then, the natural logarithm of the probability value is taken. By calculating the expected value of the above natural logarithm value on the sampled data and taking the negative value of the expected value, the loss function to be minimized is constructed. By iteratively optimizing the historical expert decision-making trajectory dataset multiple times, the policy network of each agent can initially learn to imitate the decision-making patterns of historical experts on similar emergency resource units under different environmental conditions.

5. The multi-hazard coupled urban integrated risk analysis method based on reinforcement learning according to claim 4, characterized in that: The introduction of a global critic network to evaluate the value of joint actions among agents specifically involves: Construct a global commentator network. The input of the network consists of two parts: the first part is the current global state represented by the dynamic state space constructed in step S1, and the second part is the joint action composed of the actions output by all agent policy networks at the current moment. The output of the network is a scalar value, which represents the estimate of the long-term expected reward that can be obtained by performing the joint action in the current global state, i.e., global state-action value. The global commentator network employs a collaboratively perceived value decomposition structure. The specific process for calculating the global state-action value is as follows: First, the global commentator network contains a shared individual value evaluation subnetwork. For each agent, the subnetwork takes the current global state and the agent's single action as input and outputs an individual value item that represents the agent's individual contribution basis. Secondly, the global commentator network also contains a dynamic weight generation unit, which calculates a dynamic weight scalar for each agent based on the current global state through an attention mechanism. Furthermore, the global critic network contains an explicit cooperative evaluation subnetwork, which simultaneously receives the current global state and the aggregated representation of all agent actions as input, and outputs an explicit cooperative term. Finally, the global commentator network synthesizes the final global state-action value through the following steps: D1. Multiply the individual value of each agent by its corresponding dynamic weight scalar to obtain the weighted individual value of the agent. D2. Sum the weighted individual values ​​of all agents to obtain the total weighted individual value; D3. Multiply a preset collaboration strength coefficient by the explicit collaboration term to obtain the adjusted explicit collaboration term; D4. Add the weighted sum of individual values ​​to the adjusted explicit collaboration terms, and the resulting sum is the global state-action value.

6. The multi-hazard coupled urban integrated risk analysis method based on reinforcement learning according to claim 5, characterized in that: The process of initializing policies for multiple agents further includes: using a pre-stored dataset of historical expert decision trajectories to perform supervised learning pre-training on the global commentator network as follows: Supervised learning of the global commentator network is performed using a pre-stored historical expert decision trajectory dataset. For each record in the historical expert decision trajectory dataset that includes the historical global state, the joint action given by the expert, and the actual global reward signal after executing the joint action, the historical global state and the joint action given by the expert are input into the global commentator network to obtain the global state-action value estimate calculated by the network. The actual global reward signal in the record, or the reward estimate calculated from the actual global reward signal based on the discount accumulation rule, is used as the training target value. The optimization objective of pre-training is to minimize the difference between the global state-action value estimate and the training target value. This optimization is achieved by constructing and minimizing a loss function. The loss function is calculated as follows: first, the difference between the global state-action value estimate and the target value is calculated; then, the square of the difference is calculated; finally, the average of this squared value is calculated over all relevant records in the historical expert decision trajectory dataset. The parameters of the global critic network are iteratively optimized using gradient descent, including the internal parameters of the individual value evaluation subnetwork, the dynamic weight generation unit, and the explicit collaborative evaluation subnetwork. Finally, a first set of initialization parameters consisting of the adjustable parameters of the policy networks of all agents and a second set of initialization parameters consisting of the adjustable parameters of the global critic network are obtained. The first set of initialization parameters and the second set of initialization parameters together constitute the completed state of initializing the policies of multiple agents.

7. The multi-hazard coupled urban integrated risk analysis method based on reinforcement learning according to claim 6, characterized in that: In step S3, an online policy optimization is performed using a multi-agent reinforcement learning framework based on the proximal policy optimization algorithm, including environmental interaction, experience collection, and advantage estimation, specifically as follows: At each time step, the operation—namely, environmental interaction and experience collection—is executed sequentially through the following three sub-steps: E1. Input Processing: Receive the dynamic state space constructed and output by step S1, which represents the current multi-hazard coupled risk situation of the city; each agent independently selects an action to execute based on its own initialized policy network and the local observation vector extracted from the dynamic state space that reflects its surrounding environment and state; the actions executed by all agents at the current time step constitute a joint action. E2. Environmental Feedback Processing: After the joint action is executed in a simulated or actual urban emergency management environment, the environment will calculate and feed back an actual global reward signal based on a reward function designed to integrate multiple optimization objectives such as disaster loss reduction, rescue efficiency and resource utilization. At the same time, the environmental state changes, triggering step S1 to generate the dynamic state space of the next time step. E3. Experience storage processing: The data unit containing four types of information, namely the dynamic state space of the current time step, the joint action, the actual global reward signal, and the dynamic state space of the next time step, is stored as a complete experience transfer data in a preset experience playback buffer. During the optimization phase, a batch of experience transfer data is sampled from the experience replay buffer. For each record in the sampled experience transfer data, the global commentator network introduced and initialized in step S2 is used to calculate the global state-action value estimate of performing the joint action in the current dynamic state space, and at the same time, the state value estimate in the current dynamic state space and the state value estimate in the next dynamic state space are calculated. Based on the actual global reward signal, the current state value estimate, and the next state value estimate, the time-series difference error is calculated; Furthermore, using the generalized advantage estimation algorithm, based on a series of temporal difference errors, an estimated value of the advantage function used to evaluate the superiority or inferiority of the joint action relative to the average performance is calculated.

8. The multi-hazard coupled urban integrated risk analysis method based on reinforcement learning according to claim 7, characterized in that: The optimization of the multi-agent reinforcement learning framework outputs an optimized emergency resource allocation strategy. This strategy involves updating the policy network and the global commentator network, specifically: For each piece of experience transfer data sampled from the experience replay buffer, and for each agent involved therein, the probability that the agent will output the action it performs based on its corresponding historical local observation vector under the current policy network parameters is calculated, which is the ratio of the probability that the agent will output the same action based on the same historical local observation vector under the old policy network parameters. This ratio is called the policy probability ratio. The old policy network parameters refer to the copy of the policy network parameters saved before the start of the current policy network parameter update iteration cycle. Based on the policy probability ratio and the advantage function estimate, a proximal policy optimization objective function is constructed for each agent; the objective function contains the sum of two terms: the first term is a conservative policy update term constructed based on the product of the pruning policy probability ratio and the advantage function estimate, and the second term is an entropy regularization term; The specific calculation method of the conservative strategy update term is as follows: calculate the product of the strategy probability ratio and the estimated value of the dominance function, and at the same time, calculate the product of a pruned strategy probability ratio and the estimated value of the dominance function, wherein the pruning process is as follows: restrict the strategy probability ratio to an interval centered at 1 and offset by a preset pruning hyperparameter; then, take the smaller value of the two products as the value of the conservative strategy update term. The entropy regularization term is the information entropy of the agent's current policy given its historical local observation vector, multiplied by a preset entropy coefficient, and then multiplied by a dynamic risk weight factor. The parameters of each agent's policy network are iteratively updated using the gradient ascent method by maximizing the objective function of the proximal policy of each agent. Meanwhile, the parameters of the global critic network are iteratively updated by minimizing a mean squared error loss function; Repeat the cycle of environmental interaction, experience collection, advantage estimation, policy network update, and global critic network update until any of the following preset convergence conditions are met: First, after multiple evaluation cycles are executed continuously in an independent verification environment, the average increase in reward obtained by all agents based on the latest strategy is less than a preset positive threshold. Second, the training process reaches the preset maximum number of training iterations or maximum number of time steps; Finally, the set of parameters of all agent policy networks that meet the convergence condition is used as the optimized emergency resource allocation strategy.

9. The multi-hazard coupled urban integrated risk analysis method based on reinforcement learning according to claim 8, characterized in that: In step S4, a hidden Markov model is used to model the multi-hazard coupled propagation path, and the predicted probability distribution of future disaster patterns is integrated into the value function of the global commentator network. The specific process includes model building, training, and probability prediction. First, we construct a Hidden Markov Model, which is defined by the following five components: U1, a set of finite numbers of mutually exclusive hidden states, where each hidden state represents a distinct multi-hazard coupling mode; U2, a set of observation states, which is the dynamic state space output by step S1, characterizing the spatiotemporal dynamics of urban multi-hazard coupling risks. U3. A state transition probability matrix, which is a two-dimensional square matrix with the number of rows and columns equal to the number of hidden states. The element in the i-th row and j-th column of the state transition probability matrix represents the conditional probability of the Hidden Markov Model transitioning from the i-th hidden state to the j-th hidden state in any adjacent next time step. U4. An observation probability matrix, which describes the probability distribution of observing any observation state in the set of observation states given that the Hidden Markov Model is in a certain hidden state. U5. An initial state probability distribution, which is a vector whose dimension is equal to the number of hidden states. The value of the i-th element in the vector represents the probability that the Hidden Markov Model is in the i-th hidden state at the initial time step. Secondly, the parameters of the constructed Hidden Markov Model are trained using historical disaster event data. The historical disaster event data includes a historical dynamic state space sequence generated by the method in step S1, arranged in chronological order. The forward-backward algorithm is used to iteratively adjust the value of each element in the state transition probability matrix, the probability distribution parameters defined in the observation probability matrix, and the value of each element in the initial state probability distribution vector through the expectation-maximization method, so as to maximize the probability of the Hidden Markov Model generating historical observation sequences, thereby completing the learning of the model parameters. Subsequently, at any current moment in online decision-making, based on the dynamic state space sequence traced back from the current moment to a past period, the probability of the Hidden Markov Model being in each hidden state at multiple future time steps is calculated using the trained Hidden Markov Model and forward algorithm, thus obtaining the probability distribution of future disaster modes.

10. The multi-hazard coupled urban integrated risk analysis method based on reinforcement learning according to claim 9, characterized in that: The process of forming a risk-adjusted value function and using it to provide feedback and guidance to the strategy optimization process in step S3 includes constructing a risk adjustment term, integrating the risk-adjusted value function, and providing strategy optimization feedback. First, a risk adjustment term is constructed, whose inputs include: the dynamic state space output by step S1 at the current moment, the joint action consisting of the actions output by all agents' policy networks at the current moment, and the probability distribution of future disaster modes; The calculation process for the risk adjustment item is as follows: W1. Initialization and parameter setting: Set a positive number less than or equal to 1 as the risk discount factor, and set a total number of future prediction steps; W2. Calculate for each future time step: For each future time step τ from 1 to the total number of future prediction steps, perform the following operations: W2.1 Expected Risk Calculation: Obtain the probability vector corresponding to the τ-th future time step in the probability distribution of future disaster patterns. This vector contains the probability of being in each hidden state at the τ-th future time step. Simultaneously, utilize a trainable conditional risk assessment function. This function takes the current dynamic state space, the current joint action, and a hidden state as input and outputs a risk estimate scalar representing the risk estimate under the assumption of that hidden state. Then, using the probability of being in each hidden state at the τ-th future time step as weight, perform a weighted summation of the conditional risk assessment values ​​for all possible hidden states. The result is the expected risk value at the τ-th future time step. W2.2 Risk Discount: Multiply the calculated expected risk value of the future step τ by the risk discount factor raised to the power of τ to obtain the discounted risk contribution value of the future step τ. W3. Cumulative Summation: The total sum of all discounted risk contribution values ​​from step 1 to the total number of future prediction steps is the output value of the risk adjustment term. Then, the output value of the risk adjustment term is multiplied by a preset risk sensitivity coefficient, and then added to the original global state-action value estimate calculated by the global commentator network in step S3. The original global state-action value estimate is calculated by the global commentator network according to its value function. The sum is the risk-adjusted global state-action value function. Finally, the risk-adjusted global state-action value function is fed back and applied to the strategy optimization process in step S3 to realize the dynamic adaptive adjustment of emergency resource allocation strategy to multi-hazard coupled risks.

Images