Emergency data fusion and decision-making method under cross-modal dynamic routing mechanism

Abstract

The invention discloses an emergency data fusion and decision-making method under a cross-modal dynamic routing mechanism, and relates to the field of emergency management and disaster response, and the method comprises the following steps: collecting multi-modal emergency data; carrying out feature extraction by adopting a lightweight grouping separable convolutional network, and generating an attribute capsule of a corresponding mode; fusing the attribute capsules of different modals through a cross-modal dynamic routing mechanism, and generating a decision capsule representing a global emergency situation; the decision capsules are simultaneously input into an attribute reconstruction channel and a reinforcement learning channel for parallel processing, and an emergency strategy is generated through joint optimization; and issuing the emergency strategy to the edge device for execution, and updating the weight matrix in the routing mechanism based on execution effect feedback data. According to the method, lightweight edge feature extraction, disaster stage adaptive cross-modal dynamic routing fusion and edge and cloud collaborative decision closed loop are utilized, so that the accuracy and decision efficiency of multi-source data fusion in emergency response are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of emergency management and disaster response, and more specifically to an emergency data fusion and decision-making method under a cross-modal dynamic routing mechanism. Background Technology

[0002] In modern urban emergency management and disaster response, the efficient integration of heterogeneous multimodal data and the rapid generation of decision-making instructions are crucial for improving rescue efficiency and reducing losses. This is especially true in disaster environments such as earthquakes, floods, and hazardous chemical leaks, where fluctuating network conditions, massive amounts of data, complex information dimensions, and extremely high timeliness requirements pose severe challenges to data processing and decision-making systems.

[0003] Currently, existing technologies typically rely on centralized cloud processing of raw multimodal data or employ computationally complex models for preliminary analysis at the edge. The former faces the risk of data transmission delays or failures when network bandwidth is limited or interrupted due to disasters, severely impacting timeliness; the latter is constrained by the limited computing power of edge devices, making it difficult to efficiently process high-dimensional visual, temporal, and audio data. Traditional convolutional network models have a large number of parameters and high inference latency, making it difficult to meet the real-time feature extraction requirements at the edge, thus making information processing a bottleneck and failing to provide timely, lightweight, high-quality feature representations for subsequent decision-making.

[0004] At the data fusion level, existing methods mostly employ static weights or simple concatenation and pooling to integrate information from different modalities, lacking effective modeling of the dynamic relationships between modalities. This rigid fusion mechanism struggles to adapt to the changing information priorities at different stages of a disaster, resulting in an inaccurate situational awareness capsule after fusion. Simultaneously, decision generation often relies on a single model, lacking consideration for the interpretability of the decision-making process and the ability to dynamically optimize policies using reinforcement learning, making it difficult to generate robust policies that balance efficiency, resources, and risk.

[0005] Furthermore, existing technical solutions generally lack effective closed-loop optimization mechanisms. After decision-making instructions are issued and executed, their actual effects are difficult to systematically feed back to the upstream fusion and decision-making modules. Routing parameters are usually fixed or manually preset, and cannot be dynamically and adaptively adjusted according to the stage of disaster evolution or real-time network conditions.

[0006] Therefore, how to design an emergency data fusion and decision-making method under a cross-modal dynamic routing mechanism to significantly improve the accuracy of multi-source data fusion, decision-making efficiency and system adaptability in emergency response is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides an emergency data fusion and decision-making method under a cross-modal dynamic routing mechanism. By integrating multimodal data processing, an edge-cloud collaborative architecture, and a self-optimization mechanism, it constructs an emergency decision-making system capable of rapid and accurate response, adapting to different disaster stages, and continuously self-optimizing, thereby significantly improving the efficiency and robustness of emergency response.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] An emergency data fusion and decision-making method under a cross-modal dynamic routing mechanism includes the following steps:

[0010] S1. Collect multimodal emergency data through edge devices; the multimodal emergency data includes visual modal data, text modal data, sensor time-series data, and audio modal data;

[0011] S2. A lightweight grouped separable convolutional network is used to extract features from each modality of data and generate attribute capsules for the corresponding modality.

[0012] S3. Transmit the attribute capsules of each modality to the cloud, and merge the attribute capsules of different modalities through a cross-modal dynamic routing mechanism to generate a decision capsule that represents the global emergency situation.

[0013] S4. Simultaneously input the decision capsule into the attribute reconstruction channel and the reinforcement learning channel for parallel processing, and generate emergency strategies through joint optimization.

[0014] S5. Distribute the emergency strategy to the edge devices for execution, and update the weight matrix in the routing mechanism based on the execution effect feedback data.

[0015] Preferably, in S2, the lightweight grouped separable convolutional network adopts a four-level hierarchical structure;

[0016] The first, second, and third levels are each composed of two BasicBlock modules connected in series, while the fourth level is composed of three BasicBlock modules connected in series.

[0017] Each BasicBlock module contains two convolutional layers with a kernel size of 3×3 and a stride of 1, as well as a skip connection bridging these two convolutional layers; the convolutional layers employ grouped separable convolution operations, with each group of convolutions followed by a ReLU activation function.

[0018] Preferably, in S2, the attribute capsule is represented as a binary tuple C = (v, p);

[0019] Where p represents the probability of existence, v represents the feature vector, and v = [(x,y,z),t,A,L] s ] T(x,y,z) represents spatial coordinates, t represents a timestamp, A represents the normalized amplitude value, and L represents the normalized amplitude value. s Represents semantic tags.

[0020] Preferably, in step S3, the execution process of the cross-modal dynamic routing mechanism includes:

[0021] Initialize the number of routing pairs b between capsule i and capsule j ij Set to zero, and define the number of iterations N. iter For each iteration r∈[1,N] iter ], calculate the prediction vectors sequentially Coupling coefficient High-level input Compressed output Update item and update the number of route pairs

[0022] After the iteration is complete, compress the output vector. As a decision capsule;

[0023] Where i, j, and k represent capsule indices, Let v represent the weight matrix in the r-th iteration. i This represents the output vector of capsule i. Let $\mathbf{r}$ represent the output vector of capsule $j$ in round $r$, and $\mathbf{r}$ denote the compression function. Let represent the number of route pairs between capsule i and capsule j in the r-th iteration. Let t represent the number of route logs between capsule i and capsule k in the r-th iteration, α, β, and γ represent the dynamic weight coefficients, λ represent the time decay factor, and t represent the time decay factor. i The timestamp of capsule i is represented by t. j p represents the timestamp of capsule j. i p j This represents the probability value of the existence of capsule i and capsule j.

[0024] Preferably, routing parameters are dynamically configured according to the disaster stage:

[0025] Initial stage of disaster: α = 0.7, β = 0.2, γ = 0.1, λ = 0.2;

[0026] Mid-stage of the disaster: α = 0.1, β = 0.8, γ = 0.1, λ = 0.8;

[0027] Post-disaster reconstruction phase: α = 0.3, β = 0.3, γ = 0.4, λ = 0.5.

[0028] Preferably, the number of iteration rounds N iter Dynamically adjusted based on real-time network bandwidth:

[0029]

[0030] Among them, B w Indicates real-time network bandwidth. This represents the floor function.

[0031] Preferably, in step S4, the attribute reconstruction channel processing includes:

[0032] Select the key attribute dimensions of the decision capsule and apply perturbations to generate perturbed vectors. δ represents the disturbance intensity, e q Represents unit basis vectors;

[0033] Reconstructed data R is generated using a self-attention mechanism, and the interpretability loss L is calculated. recon =||RR orig ||2,R orig This represents the raw, perceived data.

[0034] Preferably, in step S4, the reinforcement learning channel processing includes flattening the decision capsule into a state vector using the TD3 model and outputting the optimal action in the current state; the reward function R of the TD3 model is expressed as:

[0035]

[0036] in, A represents the weighting coefficient. c T represents the precision of the action. r ξ represents the response delay, ξ represents the time sensitivity coefficient, and C represents the response latency. r This represents the resource depletion rate, and Risk represents the risk assessment value.

[0037] Preferably, in step S4, joint optimization is performed by minimizing the total loss function L. total accomplish:

[0038] L total =L task +μ·L recon

[0039] Among them, L task This represents task loss, including strategy execution performance loss and TD loss; L recon denoted as interpretability loss, and μ represents the weighting coefficient.

[0040] Preferably, in step S5, dynamically updating the routing weight matrix based on execution effect data includes:

[0041] Based on the effect data of emergency strategy execution by edge devices, the deviation between the strategy execution effect and the expected goal is calculated, and the routing weight matrix is ​​updated through backpropagation; the effect data includes: resource scheduling accuracy, path planning overlap, and situation prediction error rate.

[0042] As can be seen from the above technical solution, compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0043] 1. By using a lightweight grouping separable convolutional network to extract capsule features of heterogeneous modal data at the edge, and combining it with an adaptive dynamic routing mechanism for disaster stages, the speed and quality of cross-modal information fusion are significantly optimized, generating decision capsules with more accurate global situational awareness, and providing a reliable basis for emergency strategies.

[0044] 2. A layered architecture is adopted, consisting of edge feature extraction and cloud-based fusion decision-making. Edge devices perform lightweight feature compression, while the cloud executes dynamic routing fusion and reinforcement learning decisions, dynamically adapting to network bandwidth through iterative rounds. This reduces transmission overhead and ensures that emergency strategies can be generated and deployed in real time even in low-bandwidth disaster environments.

[0045] 3. By jointly optimizing the strategy through attribute reconstruction channels and reinforcement learning channels, and dynamically updating the routing weight matrix in conjunction with edge execution feedback, a decision-execution-feedback-optimization closed loop is formed; the routing parameters are dynamically configured according to the disaster stage, enabling the system to continuously adapt to the evolution of the disaster and improve response robustness. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 This is a flowchart of an emergency data fusion and decision-making method under a cross-modal dynamic routing mechanism provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram illustrating the execution process of the cross-modal dynamic routing mechanism provided in this embodiment of the invention;

[0049] Figure 3 A flowchart illustrating the emergency data fusion and decision-making method for earthquake disaster emergency response scenarios provided in this embodiment of the invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, this embodiment provides an emergency data fusion and decision-making method under a cross-modal dynamic routing mechanism, including the following steps:

[0052] S1. Collect multimodal emergency data through edge devices; the multimodal emergency data includes visual modal data, text modal data, sensor time-series data, and audio modal data;

[0053] S2. A lightweight grouped separable convolutional network is used to extract features from each modality of data and generate attribute capsules for the corresponding modality.

[0054] S3. Transmit the attribute capsules of each modality to the cloud, and merge the attribute capsules of different modalities through a cross-modal dynamic routing mechanism to generate a decision capsule that represents the global emergency situation.

[0055] S4. Simultaneously input the decision capsule into the attribute reconstruction channel and the reinforcement learning channel for parallel processing, and generate emergency strategies through joint optimization.

[0056] S5. Distribute the emergency strategy to the edge devices for execution, and update the weight matrix in the routing mechanism based on the execution effect feedback data.

[0057] This method significantly improves emergency response efficiency. By integrating lightweight edge feature extraction with adaptive cross-modal dynamic routing at disaster stages, it achieves accurate and rapid situational awareness of multi-source heterogeneous data. The edge-cloud collaborative architecture ensures efficient decision generation and distribution in low-resource environments. Furthermore, by jointly optimizing attribute reconstruction and reinforcement learning, and dynamically updating routing weights based on execution feedback, a closed-loop self-optimizing emergency decision-making system is formed, continuously improving response robustness.

[0058] The following provides a further detailed explanation of each step of the above method;

[0059] In this embodiment S1, multimodal emergency data is collected through an edge device; the multimodal emergency data includes visual modal data, text modal data, sensor time-series data, and audio modal data;

[0060] Specifically, edge devices play a crucial role in emergency management, responsible for collecting various types of data in real time. Visual modal data, such as satellite remote sensing imagery, provides large-scale disaster monitoring; drone aerial video captures more detailed on-site conditions; and surveillance camera footage is used to monitor dynamic information such as pedestrian and vehicle traffic in real time. Text modal data, such as on-site reports, typically contains firsthand disaster information; social media information reflects public sentiment and the impact of disasters; and emergency response documents provide guidance for disaster response. Sensor time-series data, such as data from seismic, meteorological, and hydrological sensors, continuously records changes in disaster-related physical parameters. Audio modal data, such as alarm telephone recordings and on-site intercom communications, provides real-time voice information, helping to understand the specific situation at the disaster site and the needs of personnel.

[0061] Furthermore, edge devices employ a tiered caching strategy, with high-priority data being directly transmitted to the cloud and regular data being temporarily stored locally before being uploaded in batches. The multimodal emergency data collected by edge devices forms the information cornerstone of the emergency management system, providing comprehensive and real-time disaster site information, which is crucial for accurately assessing the disaster situation and formulating effective emergency strategies.

[0062] In this embodiment S2, a lightweight grouped separable convolutional network is used to extract features from each modality of data and generate attribute capsules for the corresponding modality.

[0063] The lightweight grouped separable convolutional network employs a four-level hierarchical structure. The first, second, and third levels each consist of two BasicBlock modules cascaded together, while the fourth level comprises three BasicBlock modules cascaded together. Each BasicBlock module contains two convolutional layers with a kernel size of 3×3 and a stride of 1, along with skip connections bridging these two layers. The convolutional layers utilize grouped separable convolution operations, with each group of convolutions followed by a ReLU activation function. This network structure not only reduces computational resource requirements but also improves model efficiency. The design of the two 3×3 convolutional layers and their skip connections within each BasicBlock module ensures the effectiveness of feature extraction and the stability of the deep learning model.

[0064] Furthermore, the feature maps extracted by the lightweight grouped separable convolutional network are input into a pre-defined fully connected layer, which outputs a feature vector. Probabilistic modeling is performed on the feature vector to output the existence probability. The feature vector and the existence probability are combined to form an attribute capsule C = (v, p); where p represents the existence probability, v represents the feature vector, and v = [(x, y, z), t, A, L]. s ] T (x,y,z) represents spatial coordinates, t represents a timestamp, A represents the normalized amplitude value, and L represents the normalized amplitude value. s Represents semantic tags.

[0065] In this embodiment, S3, the attribute capsules of each modality are transmitted to the cloud, and the attribute capsules of different modalities are fused through a cross-modal dynamic routing mechanism to generate a decision capsule representing the global emergency situation.

[0066] like Figure 2 As shown, the execution process of the cross-modal dynamic routing mechanism includes:

[0067] 1) Initialization phase;

[0068] After multimodal attribute capsules (such as visual building damage capsules, sensor aftershock capsules, and text-based casualty location capsules) are transmitted to the cloud, the system initializes routing parameters, including:

[0069] The logarithm b of the routes between capsule i (each modal input) and capsule j (decision capsule) ij Set all values ​​to zero; based on real-time network bandwidth B w Dynamically set the number of iteration rounds N iter Load the dynamic weights for the current disaster stage (e.g., in the initial stage of a disaster, α = 0.7, β = 0.2, γ = 0.1, λ = 0.2).

[0070] 2) Iterative fusion stage;

[0071] Visual capsules of building damage are projected onto the decision space using a weight matrix to generate prediction vectors.

[0072] The coupling strength between the text wounded capsule and the decision capsule is calculated based on the route logarithm. Coupling strength The larger the value, the more critical the text information is to the decision-making process;

[0073] The weighted contributions from all modalities are aggregated to generate the input vector for the decision capsule.

[0074] The decision capsule is processed by a compression function and output as a normalized result.

[0075] Calculate route update items Dynamically adjust routing weights; if the timestamp difference between the audio distress capsule and the decision capsule is |t i -t j |>2 hours (if the timestamp difference between capsule i and capsule j is |t) i -t j If the preset time window is exceeded, the coupling calculation of the capsule pair will be skipped. Then γ·exp(-λ×|t) i -t j |) Each item approaches 0;

[0076] Update route logs: Increase the routing priority of key modalities (such as sensor data in the middle of a disaster) based on the update items to ensure that the next iteration focuses more on high-value information.

[0077] 3) Decision output stage;

[0078] Complete N iter After rounds of iteration, the final decision capsule is output. Decision Capsule It integrates multimodal core information, including: in the spatial dimension, it fuses the building coordinates of the visual capsule with the location of the wounded in the text capsule; in the temporal dimension, it combines weighted sensor time-series data with the latest audio information; and in the semantic dimension, it generates labels such as "high-risk area / resource gap".

[0079] This mechanism can automatically identify and integrate relevant information from different modalities to form a comprehensive emergency situation representation. For example, in an earthquake disaster, visual modal data may provide information on building damage caused by the earthquake, while sensor time-series data records the earthquake's magnitude and focal depth. Through a cross-modal dynamic routing mechanism, this information can be effectively integrated to generate a decision capsule containing comprehensive information about the earthquake disaster. The attribute capsules from each modality are transmitted to the cloud for fusion to generate the decision capsule, which achieves effective integration and comprehensive representation of multimodal information, providing timely and accurate decision support for emergency response.

[0080] In this embodiment S4, the decision capsule is simultaneously input into the attribute reconstruction channel and the reinforcement learning channel for parallel processing, and an emergency strategy is generated through joint optimization; wherein, the attribute reconstruction channel processing includes:

[0081] Select the key attribute dimensions of the decision capsule and apply perturbations to generate perturbed vectors. δ represents the disturbance intensity, e q Represents unit basis vectors;

[0082] Reconstructed data R is generated using a self-attention mechanism, and the interpretability loss L is calculated. recon =||RR orig ||2,R orig This represents the raw, perceived data.

[0083] Furthermore, the reinforcement learning channel processing includes flattening the decision capsule into a state vector using the TD3 model and outputting the optimal action in the current state; the reward function R of the TD3 model is expressed as:

[0084]

[0085] in, A represents the weighting coefficient. cT represents the precision of the action. r ξ represents the response delay, ξ represents the time sensitivity coefficient, and C represents the response latency. r This represents the resource depletion rate, and Risk represents the risk assessment value.

[0086] Furthermore, joint optimization minimizes the total loss function L. total accomplish:

[0087] L total =L task +μ·L recon

[0088] Among them, L task This represents task loss, including strategy execution performance loss and TD loss; L recon denoted as interpretability loss, and μ represents the weighting coefficient.

[0089] During each training iteration, the task loss L is calculated synchronously. task With interpretability loss L recon According to the total loss L total The gradient direction is determined, and the parameters of the reinforcement learning policy network and the routing weight matrix are updated simultaneously.

[0090] In the attribute reconstruction channel, by reconstructing and interpreting the key attributes of the decision capsule, easy-to-understand emergency strategy descriptions can be generated, helping rescue personnel quickly grasp the core content of the strategy. Simultaneously, in the reinforcement learning channel, deep learning and optimization of the decision capsule using an advanced TD3 model can generate more intelligent and efficient emergency strategies. This parallel processing approach not only improves the efficiency of emergency strategy formulation but also ensures the practicality of the strategies.

[0091] In embodiment S5, the emergency strategy is distributed to the edge device for execution, and the weight matrix in the routing mechanism is updated based on the execution effect feedback data; wherein, dynamically updating the routing weight matrix based on the execution effect data includes:

[0092] Based on the effect data of emergency strategy execution by edge devices, the deviation between the strategy execution effect and the expected goal is calculated, and the routing weight matrix is ​​updated through backpropagation; the effect data includes: resource scheduling accuracy, path planning overlap, and situation prediction error rate.

[0093] like Figure 3 As shown, the implementation process of this method will be further explained below in conjunction with a specific earthquake disaster emergency response scenario;

[0094] 1) Multimodal emergency data acquisition (edge ​​end);

[0095] Following the earthquake, edge devices deployed at and around the site immediately activated, collecting multimodal emergency data. Drones quickly took to the air to capture real-time images of collapsed building structures and road damage; rescue teams used PDAs to input text reports of the locations and conditions of injured people; vibration sensors deployed in the rubble continuously monitored aftershocks, while life detectors captured faint vital signs of survivors; and audio from emergency calls and rescue walkie-talkies was also recorded in real time.

[0096] To optimize transmission efficiency, critical data will be prioritized for direct transmission to the cloud, while regular video streams will be temporarily stored at edge nodes and uploaded in batches later, ensuring that the information base is both comprehensive and timely.

[0097] 2) Lightweight feature extraction (edge ​​end);

[0098] Once the data reaches the edge computing nodes, it is efficiently processed by lightweight, grouped, separable convolutional networks. For example, video streams captured by drones are extracted layer by layer by the network to generate attribute capsules that describe key spatial features such as crack width and road interruption status at specific coordinate points; time-series data from life detectors are transformed into attribute capsules containing coordinates, timestamps, heart rate intensity, and semantic labels of "vital signs".

[0099] 3) Cross-modal dynamic routing fusion (cloud);

[0100] All modal attribute capsules are transmitted to the cloud processing center; a cross-modal dynamic routing mechanism plays a key role, dynamically adjusting the fusion strategy according to different stages of the earthquake disaster. In the early stages of the disaster, the mechanism prioritizes associating location capsules from text reports and audio distress calls to quickly locate rescue targets; in the middle stages of the disaster, it focuses more on fusing sensor data and visual data to assess risks and adjust plans. In addition, the mechanism strictly adheres to time window constraints, such as automatically ignoring old image capsules from 72 hours after the earthquake, to ensure that the fusion results reflect the latest situation.

[0101] Ultimately, information from different sources and dimensions is precisely integrated to output a decision capsule representing the overall situation, which includes: the coordinates of the core disaster area, areas with a high probability of survivors, the current aftershock risk level, and information on key bottleneck road sections affecting rescue efforts.

[0102] 4) Dual-channel strategy generation (cloud-based);

[0103] The generated decision capsules are simultaneously fed into an attribute reconstruction channel and a reinforcement learning channel for parallel analysis and optimization. The attribute reconstruction channel focuses on the key attributes in the decision capsule, applying small perturbations and utilizing a self-attention mechanism to generate intuitive visualizations and output clear and easy-to-understand policy descriptions. Meanwhile, the reinforcement learning channel acts as a policy optimization engine, flattening the decision capsule into a state vector and outputting the current optimal action command through deep computation. Its reward function is designed to closely align with the rescue objectives, with a particular emphasis on reducing response delays and the risk of casualties.

[0104] The results from the two channels are fed back to each other through a joint optimization mechanism, ensuring that the final emergency strategy is not only efficient and intelligent, but also that the key decision-making basis is transparent and explainable, making it easy for rescue personnel to understand and implement.

[0105] 5) Strategy execution and feedback (edge-cloud closed loop);

[0106] The cloud-based system distributes optimized emergency strategies to edge devices of frontline rescue teams for execution. The effectiveness of strategy execution is fed back to the cloud in real time. Successful strategy execution increases the weight of relevant text modal capsules in the routing mechanism; problems exposed during execution trigger dynamic updates to the routing weight matrix, and the attribute reconstruction channel adjusts its focus accordingly. This allows the entire system to continuously learn and optimize during ongoing rescue operations, improving the accuracy and robustness of subsequent decisions. Ultimately, within the limited timeframe of earthquake rescue, a precise rescue heatmap is generated, effectively guiding resource allocation priorities and improving overall rescue efficiency and success rate.

[0107] This embodiment presents an emergency data fusion and decision-making method based on a cross-modal dynamic routing mechanism. In emergency response scenarios such as earthquake disasters, it achieves efficient end-to-end processing from multimodal data acquisition, lightweight feature extraction, cross-modal fusion, dual-channel policy generation to policy execution and feedback. It not only achieves accurate fusion of multi-source heterogeneous data and comprehensive situational awareness, but also ensures efficient decision generation and execution in low-resource environments through an edge-cloud collaborative architecture. Furthermore, the dynamic update mechanism based on execution feedback enables the system to continuously optimize and adapt, significantly improving the efficiency and robustness of emergency response and providing strong technical support for disaster emergency management.

[0108] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0109] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An emergency data fusion and decision-making method under a cross-modal dynamic routing mechanism, characterized in that, The method comprises the following steps: S1, collecting multi-modal emergency data through an edge device; the multi-modal emergency data comprises visual modal data, text modal data, sensor time series data and audio modal data; S2, performing feature extraction on each modal data respectively by using a lightweight grouped separable convolutional network to generate attribute capsules of the corresponding modal; S3, transmitting the attribute capsules of each modal to the cloud, fusing the attribute capsules of different modalities through a cross-modal dynamic routing mechanism to generate a decision capsule representing the global emergency situation; S4, inputting the decision capsule into an attribute reconstruction channel and a reinforcement learning channel for parallel processing, and generating an emergency strategy through joint optimization; S5, issuing the emergency strategy to the edge device for execution, and updating the weight matrix in the routing mechanism based on the execution effect feedback data.

2. The emergency data fusion and decision method under the cross-modal dynamic routing mechanism according to claim 1, characterized in that, In the S2, the lightweight grouped separable convolutional network adopts a four-level hierarchical structure; The first, second and third levels are each composed of two BasicBlock modules connected in series, and the fourth level is composed of three BasicBlock modules connected in series; Each BasicBlock module includes two convolution layers with a convolution kernel size of 3x3 and a stride of 1, and a jump connection connecting the two convolution layers; the convolution layers use grouped separable convolution operation, and each group is followed by a ReLU activation function.

3. The emergency data fusion and decision method under the cross-modal dynamic routing mechanism according to claim 1, characterized in that, In the S2, the attribute capsule is represented as a binary tuple C=(v, p); where p represents a presence probability, v represents a feature vector, v = [(x, y, z), t, A, L s ] T , (x, y, z) represents a spatial coordinate, t represents a time stamp, A represents a normalized amplitude value, L s represents a semantic label.

4. The emergency data fusion and decision method under the cross-modal dynamic routing mechanism according to claim 1, characterized in that, In the S3, the execution process of the cross-modal dynamic routing mechanism comprises: Initialize the routing log2b between capsules i and j ij Zero-out, define iteration round N iter For each iteration round r ∈ [1, N iter ], in turn, compute the prediction vector Coupling coefficient High-level input Compressed output Update term And update the routing log2b After the iteration ends, the compressed output vector As a decision capsule; where i, j, k represent capsule indices, denotes the weight matrix at the rth iteration, v i denotes the output vector of capsule i, denotes the output vector of capsule j at the rth iteration, squash(·) denotes the squashing function, denotes the routing score between capsule i and capsule j at the rth iteration, denotes the routing score between capsule i and capsule k at the rth iteration, a, b, g represent dynamic weight coefficients, l represents a time decay factor, t i denotes the timestamp of capsule i, t j denotes the timestamp of capsule j, p i , p j denotes the probability value of the existence of capsule i, capsule j.

5. The emergency data fusion and decision method under the cross-modal dynamic routing mechanism according to claim 4, characterized in that, The routing parameters are dynamically configured according to the disaster phase: In the early stage of disaster: α=0.7, β=0.2, γ=0.1, λ=0.2; In the middle stage of disaster: α=0.1, β=0.8, γ=0.1, λ=0.8; In the post-disaster reconstruction stage: α=0.3, β=0.3, γ=0.4, λ=0.

5.

6. The emergency data fusion and decision method under the cross-modal dynamic routing mechanism according to claim 4, characterized in that, The iteration number of rounds N iter According to real-time network bandwidth dynamic adjustment: where B w represents real-time network bandwidth, represents a floor function.

7. The method of claim 1, wherein, In the S4, the attribute reconstruction channel processing comprises: Select the key attribute dimension of the decision capsule, apply disturbance to generate the disturbed vector δ represents the disturbance intensity, e q represents the unit basis vector; The reconstruction data R is generated by a self-attention mechanism, and an explainability loss L is calculated recon = ||R-R orig ||2, R orig represents the original perception data.

8. The emergency data fusion and decision method under the cross-modal dynamic routing mechanism according to claim 1, characterized in that, In the S4, the reinforcement learning channel processing comprises flattening the decision capsule into a state vector by using a TD3 model to output the optimal action under the current state; the TD3 model reward function R is represented as: wherein, denotes a weight coefficient, A c denotes an action accuracy, T r denotes a response delay, ξ denotes a time sensitivity coefficient, C r denotes a resource depletion rate, Risk denotes a risk evaluation value.

9. The emergency data fusion and decision method under the cross-modal dynamic routing mechanism according to claim 1, characterized in that, In the S4, the joint optimization is performed by minimizing the total loss function L total Implementation: L total = L task + μ · L recon wherein L task represents the task loss, including the policy execution effect loss and the TD loss; L recon represents the explainability loss, and μ represents a weight coefficient.

10. The emergency data fusion and decision method under the cross-modal dynamic routing mechanism according to claim 1, characterized in that, In the S5, the dynamic updating of the routing weight matrix based on the execution effect data comprises: Based on the effect data of the edge device executing the emergency strategy, the deviation of the strategy execution effect from the expected target is calculated, and the routing weight matrix is updated through back propagation; the effect data includes: resource scheduling accuracy, path planning overlap degree and situation prediction error rate.

Images