Intelligent fire-fighting command method and system based on multi-modal AI large model

Abstract

The invention relates to the technical field of multi-source information integration, and discloses an intelligent fire-fighting command method and system based on a multi-modal AI large model.According to the intelligent fire-fighting command method and system, fire scene data are obtained in real time through multi-terminal information collection, scene reconnaissance feedback is conducted through flame and smoke recognition and dangerous goods recognition through an image recognition algorithm, and the fire-fighting commanding efficiency is improved. A pre-trained multi-mode live restoration model is utilized to carry out restoration simulation on a fire scene, a multi-source live simulation model is generated to analyze and formulate a personalized rescue plan, positioning and state information of a rescue unit is collected in real time, a rescue unit simulation model is constructed, and specific rescue execution steps are analyzed according to the personalized plan. The steps help commanders to realize accurate commanding and efficient rescue, the rescue decision accuracy is improved, the response time is shortened, the resource allocation is optimized, the emergency response capability and the successful rescue rate of a fire scene are remarkably improved, and the problem that in the prior art, the efficiency is low when multi-source information is manually integrated to restore the fire situation is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source information integration, and in particular to an intelligent fire command method and system based on a multimodal AI large model. Background Art

[0002] The scene of a fire is very complex and in dynamic change. The conventional fire rescue method is to collect on-site information in multiple dimensions and integrate the information to obtain the true restoration of the fire scene. In traditional technical means, this step is usually performed manually, which makes the intelligence processing less efficient and easily misses the best time for rescue. Summary of the invention

[0003] The purpose of the present invention is to provide an intelligent fire command method and system based on a multimodal AI large model, aiming to solve the problem of low efficiency in restoring fire conditions by manually integrating multi-source information in the prior art.

[0004] The present invention is implemented in this way. In a first aspect, the present invention provides an intelligent fire command method based on a multimodal AI large model, comprising: Perform multi-terminal information collection on the fire scene to obtain multi-terminal scene information of the fire scene, and perform a live restoration simulation of the fire scene according to the multi-terminal scene information using a pre-trained multimodal live restoration model to obtain a multi-source live simulation model of the fire scene; wherein the multi-terminal scene information includes text end information, image end information, video end information, and voice end information; Analyzing the real-time rescue strategy of the multi-source real-time simulation model through the pre-trained rescue decision-making intelligent model to obtain a personalized rescue plan corresponding to the fire scene; Collecting real-time information of rescue units to obtain location information and status information of each rescue unit, and constructing a rescue unit simulation model based on the multi-source live simulation model according to the location information and status information of each rescue unit; According to the personalized rescue plan, the multi-source live simulation model with the rescue unit simulation model is analyzed for specific rescue execution steps to obtain specific rescue execution steps; wherein, the specific rescue execution steps are used to assist the rescue commander to conduct real-time firefighting command for each rescue unit.

[0005] In a second aspect, the present invention provides an intelligent fire command system based on a multimodal AI big model, which is used to implement an intelligent fire command method based on a multimodal AI big model as described in any one of the first aspects, including: A live simulation module is used to collect multi-terminal information of the fire scene to obtain multi-terminal scene information of the fire scene, and to perform live restoration simulation of the fire scene according to the multi-terminal scene information through a pre-trained multimodal live restoration model to obtain a multi-source live simulation model of the fire scene; wherein the multi-terminal scene information includes text terminal information, image terminal information, video terminal information, and voice terminal information; A plan analysis module, used to analyze the real-time rescue strategy of the multi-source real-time simulation model through a pre-trained rescue decision-making intelligent model, so as to obtain a personalized rescue plan corresponding to the fire scene; A rescue simulation module is used to collect real-time information of rescue units to obtain the location information and status information of each rescue unit, and to construct a rescue unit simulation model based on the multi-source live simulation model according to the location information and status information of each rescue unit; The specific command module is used to analyze the specific rescue execution steps of the multi-source live simulation model with the rescue unit simulation model according to the personalized rescue plan to obtain the specific rescue execution steps; wherein, the specific rescue execution steps are used to assist the rescue commander to conduct real-time firefighting command for each rescue unit.

[0006] The present invention provides an intelligent fire command method based on a multimodal AI large model, which has the following beneficial effects: The present invention acquires fire scene data in real time through multi-terminal information collection, and uses a pre-trained multimodal real-time restoration model to restore and simulate the fire scene, generating a multi-source real-time simulation model, and based on this model analysis, uses an intelligent decision-making model to formulate a personalized rescue plan, and constructs a rescue unit simulation model by collecting the location and status information of the rescue unit in real time, and analyzes the specific rescue execution steps according to the personalized plan. These steps help command personnel to dispatch rescue forces in real time, achieve precise command and efficient rescue, improve the accuracy of rescue decision-making, reduce response time, optimize resource allocation, and ensure the efficiency, coordination and safety of rescue operations, thereby significantly improving the emergency response capability and successful rescue rate of the fire scene, and solving the problem of low efficiency in restoring fire conditions by manually integrating multi-source information in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a schematic diagram of the steps of an intelligent fire command method based on a multimodal AI large model provided by an embodiment of the present invention; Figure 2 It is a structural schematic diagram of an intelligent fire command system based on a multimodal AI large model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0008] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0009] The implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0010] Reference Figure 1 , Figure 2 As shown, a preferred embodiment of the present invention is provided.

[0011] In a first aspect, the present invention provides an intelligent fire command method based on a multimodal AI large model, comprising: S1: Collect multi-terminal information of the fire scene to obtain multi-terminal scene information of the fire scene, and perform a live restoration simulation of the fire scene according to the multi-terminal scene information through a pre-trained multimodal live restoration model to obtain a multi-source live simulation model of the fire scene; wherein the multi-terminal scene information includes text end information, image end information, video end information, and voice end information; S2: Analyzing the real-time rescue strategy of the multi-source real-time simulation model through the pre-trained rescue decision-making intelligent model to obtain a personalized rescue plan corresponding to the fire scene; S3: collecting real-time information of the rescue units to obtain the location information and status information of each rescue unit, and constructing a rescue unit simulation model based on the multi-source real-time simulation model according to the location information and status information of each rescue unit; S4: Analyze the specific rescue execution steps of the multi-source live simulation model with the rescue unit simulation model according to the personalized rescue plan to obtain the specific rescue execution steps; wherein the specific rescue execution steps are used to assist the rescue commander to conduct real-time firefighting command for each rescue unit.

[0012] Specifically, in step S1 of the embodiment provided by the present invention, at the fire scene, different types of information are collected through various devices and sensors to fully understand the situation on the scene, and these information sources include text, images, videos and voices.

[0013] More specifically, real-time fire reports, weather data, sensor feedback and other information are collected through on-site staff, communication equipment, automated monitoring systems, etc. Text data may include fire reports, fire levels, meteorological information, etc., and natural language processing (NLP) technology is used to parse text information, extract key information, and update event logs in real time.

[0014] More specifically, on-site cameras, drones and other equipment are deployed to obtain static image information of the fire, such as the source of the fire, building damage, fire spread, etc., and computer vision (CV) technology is used for fire source identification, smoke monitoring, building structure analysis, etc.

[0015] More specifically, real-time video streams are transmitted through on-site surveillance cameras and drones to capture dynamic information such as the fire development process, personnel evacuation routes, and fire spread, through video analysis technology (such as object detection, motion tracking, and fire dynamic monitoring).

[0016] More specifically, the speech recognition system collects voice information from on-site communication equipment, broadcasting systems, etc., such as commanders' orders, alarms, personnel reports, etc., and converts voice signals into text through automatic speech recognition (ASR) technology, and extracts useful information for further processing.

[0017] It is understandable that through the real-time collection of multiple information sources, comprehensive data about the fire scene can be obtained, covering multimodal information such as text, images, videos, and voice. This information helps to establish a complete data set of the fire scene for subsequent analysis and decision-making.

[0018] More specifically, the collected multi-terminal information is combined with a pre-trained multimodal real-time restoration model to perform a real-time restoration simulation of the fire scene. This process can accurately reconstruct the current situation of the fire scene through the fusion of multimodal data, providing a basis for subsequent rescue decisions. The information from text, images, videos and voice is preprocessed, including denoising, formatting, time synchronization, etc. Then, multimodal data fusion technology is used to effectively integrate different types of information to provide more accurate input for the model. Multimodal learning methods in deep learning are used, such as the combination of Transformer architecture, convolutional neural network (CNN) and recurrent neural network (RNN).

[0019] More specifically, a multimodal real-time restoration model based on deep learning is used for training in combination with historical data and real-time data on site, so that the model can extract key features from multi-source information and restore the dynamic changes of the fire scene. Convolutional neural network (CNN) is used to process image and video data, recurrent neural network (RNN) or long short-term memory network (LSTM) is used to process time series data, and the Transformer model is combined for information fusion. The fire scene is restored based on multimodal information, the current state of the fire (such as the location of the fire source, the spread trend, the damage to the building, etc.) is generated, and the fire development process is simulated. Based on technologies such as generative adversarial networks (GAN) or reinforcement learning (RL), real-time simulation of fire scenes is generated, and the evolution and change prediction of dynamic scenes are supported. Through the restoration and simulation of multimodal information, the current situation of the fire scene can be accurately reproduced, helping decision makers understand the dynamic process of fire spread in real time. The multimodal restoration model provides data support for the subsequent formulation of personalized rescue plans, making decisions more real-time and accurate.

[0020] More specifically, based on multi-terminal information collection and multi-modal real-time restoration simulation, a multi-source real-time simulation model of the fire scene is constructed. The model can combine different information sources to reconstruct the actual situation of the fire scene in real time, and integrate information obtained from multiple data sources such as text, images, videos, and voice to form complete fire scene data. Through the spatiotemporal modeling method, multimodal data are combined to construct a dynamic, real-time updated fire simulation scene. Spatiotemporal data modeling and fusion algorithms (such as weighted average method, deep fusion network, etc.) are used to perform multi-level and multi-dimensional modeling of multimodal data. Combined with the dynamic information of the fire (such as fire source, building structure, meteorological conditions, etc.), physical models and machine learning algorithms are used to simulate the spread of fire, building damage, personnel flow, etc. Physical simulation (such as fluid dynamics model, fire spread model) is combined with machine learning model to predict the development trend of fire. Through three-dimensional simulation technology, the restoration effect of the fire scene is displayed in real time. Combined with multi-dimensional information such as thermodynamics and optics, realistic on-site restoration effects are provided.

[0021] Specifically, in step S2 of the embodiment provided by the present invention, the rescue decision-making intelligent model is trained based on historical fire rescue data, fire scene analysis and deep learning methods. Its main goal is to learn the rescue modes of different fire scenes, automatically analyze the fire scene information provided by the multi-source real-life simulation model, and predict the optimal rescue strategy.

[0022] More specifically, a large amount of historical fire rescue data is collected, including fire types, rescue resources (such as fire brigades, medical teams, rescue tools, etc.), rescue strategies, casualties, etc., and model training is carried out using methods such as reinforcement learning (RL), deep neural networks (DNN), decision trees, and graph neural networks (GNN). During the training process, the model learns how to select the optimal rescue decision based on the specific situation of the fire and on-site conditions (such as fire intensity, wind speed, personnel distribution, etc.). Reinforcement learning (RL) is used for multi-stage decision-making processes, deep neural networks (DNN) are used for pattern recognition and feature extraction, and graph neural networks (GNN) can be used to process complex fire rescue task networks.

[0023] More specifically, after receiving the real-time data from the multi-source live simulation model, the rescue decision intelligent model generates and analyzes the rescue strategy. Specifically, the model will analyze the most suitable rescue plan based on the fire development, personnel distribution, resource conditions, etc. at the scene. The model infers and calculates based on the input fire scene information to evaluate the effects of different rescue strategies, such as how to choose the best fire-fighting route, how to guide personnel to evacuate safely, how to allocate rescue resources, etc. The image recognition and speech recognition results assist the model reasoning, the reinforcement learning strategy generates the optimal plan, and the path planning algorithm is used to calculate the best evacuation route. For different rescue strategies, the model evaluates through a variety of performance indicators (such as rescue time, casualty rate, resource consumption, etc.). Finally, the model selects the strategy that best suits the current fire situation.

[0024] It is understandable that targeted rescue plans can be quickly generated based on the characteristics of different fire scenes. As the situation at the fire scene changes, the rescue plan can be adjusted in real time to ensure the implementation of the optimal strategy. Through intelligent scheduling and resource optimization, the utilization efficiency of rescue resources is improved and the waste of resources in the rescue process is reduced.

[0025] Specifically, in step S3 of the embodiment provided by the present invention, it is necessary to collect real-time information of each rescue unit through various sensors and communication systems. This information includes the positioning information of each rescue unit (such as GPS coordinates, motion status, speed, etc.) and its status information (such as resource usage, task execution progress, personnel health status, etc.). This step is the basis for building a rescue unit simulation model.

[0026] More specifically, use GPS equipment or a positioning system based on a ground base station (such as RTK positioning technology, Bluetooth positioning, Wi-Fi positioning, etc.) to collect the location of each rescue unit, and update the location information of the rescue unit in real time to ensure that its dynamic location can be accurately tracked.

[0027] More specifically, through the Internet of Things technology, the use status of rescue equipment (such as fire trucks, rescue tools, etc.) is monitored in real time, such as remaining fuel, power, equipment failure, etc., and the health status, fatigue level, injury status, etc. of each rescuer are obtained through wearable smart bracelets and vital signs monitoring equipment, and the status information of each rescue unit's mission execution (such as fire extinguishing progress, personnel evacuation status, etc.) is collected.

[0028] It is understandable that real-time and accurate acquisition of the positioning information and status information of each rescue unit ensures the comprehensiveness and timeliness of the information and provides data support for the subsequent construction of the simulation model.

[0029] More specifically, the real-time data obtained from different sources (GPS positioning, sensor data, mission execution information, etc.) are fused to generate a unified rescue unit information set. The key to this step is how to effectively integrate these data so that they can be provided to subsequent simulation and decision-making models. The real-time data from different sources are standardized and converted into a unified format for further analysis and modeling. Missing data or abnormal data are processed, and data is filled and corrected through interpolation, filtering and other technologies. The multi-source data such as the rescue unit's positioning information, status information, mission progress, etc. are integrated into an efficient information set through data fusion algorithms.

[0030] More specifically, based on the integrated real-time information, simulation technology is used to construct a simulation model of the rescue unit to simulate the behavior, task execution and resource allocation of the rescue unit. This model can reflect the status and action trajectory of each rescue unit in real time at the fire scene, simulate its action process and predict possible dynamic changes in the future.

[0031] More specifically, an independent model is built for each rescue unit (such as fire trucks, rescue teams, medical teams, etc.) to simulate its location changes, task execution, resource consumption, etc. Based on positioning information and status information, the specific actions of the rescue unit at the fire scene are simulated, such as entering and exiting the fire area, performing fire extinguishing, personnel evacuation, equipment supply, etc. The model simulates how the rescue unit can be dynamically adjusted according to task requirements. For example, when a fire spreads in a certain area, the model can automatically adjust the rescue unit's tasks and resource allocation.

[0032] It can be understood that the constructed rescue unit simulation model can accurately reflect the real-time actions and status changes of each rescue unit, help commanders foresee the results of rescue operations, provide real-time simulation support, and be able to adjust the rescue unit's action plan in a timely manner according to changes in the fire scene and mission.

[0033] Specifically, in step S4 of the embodiment provided by the present invention, based on the personalized rescue plan, a multi-source live simulation model is started to simulate the action trajectory, task progress and resource status of each rescue unit. The model adjusts the action plan and task allocation of each unit in real time according to the positioning, fire spread, resource consumption, task completion and other data of the fire fighting unit, and uses real-time sensor data (such as GPS positioning, sensor feedback, meteorological information, etc.) to update the simulation model to ensure the real-time and accuracy of the model. The system automatically adjusts the resource scheduling of the rescue unit according to the simulation results, such as adjusting the fire truck route, medical resource scheduling, etc., and dynamically optimizes the rescue mission to ensure that each rescue unit can perform its tasks efficiently while avoiding excessive concentration or excessive dispersion of resources.

[0034] More specifically, when executing real-time rescue operations, as the disaster site changes, the rescue unit's mission, location, status and other information are also constantly changing. Therefore, commanders need to monitor and optimize the rescue execution steps in real time to ensure the smooth progress of the rescue work.

[0035] More specifically, based on the feedback from the simulation model, specific tasks of the rescue units are assigned in real time. For example, when the fire intensifies in a certain area, more firefighting forces are automatically deployed, the priority of the tasks is adjusted, and high-priority tasks are assigned to units with sufficient resources. The location and task progress of the rescue units are monitored in real time, and the optimal path is calculated to avoid time loss caused by road blockages or traffic congestion. The rescue team's action route is dynamically optimized to ensure timely response. Through the real-time monitoring platform, the status of each rescue unit, such as location, task completion, resource consumption, etc., is tracked to ensure that the rescue work is carried out according to the predetermined plan. Artificial intelligence technology is used to analyze real-time data, propose emergency plans, and remind and suggest rescue commanders.

[0036] The present invention provides an intelligent fire command method based on a multimodal AI large model, which has the following beneficial effects: The present invention acquires fire scene data in real time through multi-terminal information collection, and uses a pre-trained multimodal real-time restoration model to restore and simulate the fire scene, generating a multi-source real-time simulation model, and based on this model analysis, uses an intelligent decision-making model to formulate a personalized rescue plan, and constructs a rescue unit simulation model by collecting the location and status information of the rescue unit in real time, and analyzes the specific rescue execution steps according to the personalized plan. These steps help command personnel to dispatch rescue forces in real time, achieve precise command and efficient rescue, improve the accuracy of rescue decision-making, reduce response time, optimize resource allocation, and ensure the efficiency, coordination and safety of rescue operations, thereby significantly improving the emergency response capability and successful rescue rate of the fire scene, and solving the problem of low efficiency in restoring fire conditions by manually integrating multi-source information in the prior art.

[0037] Preferably, the steps of collecting multi-terminal information on the fire scene to obtain multi-terminal scene information of the fire scene, and performing a live restoration simulation of the fire scene according to the multi-terminal scene information using a pre-trained multimodal live restoration model to obtain a multi-source live simulation model of the fire scene include: S11: Collecting information from the fire scene through multiple types of information collection terminals that arrive at the fire scene, so as to obtain on-site information from various types of information terminals, where the on-site information from various types of information terminals together constitutes multi-terminal on-site information from the fire scene; S12: Retrieving a model portion corresponding to the fire scene from a pre-built urban three-dimensional simulation platform as an original simulation model; S13: The multi-end scene information is disassembled and analyzed in terms of information feedback areas through a pre-trained multi-modal real-time restoration model to obtain the real-time features of each model area of ​​the original simulation model, and the model real-time correction is performed on each model area according to the real-time features to obtain a multi-source real-time simulation model of the fire scene.

[0038] Specifically, comprehensive information collection is carried out on the fire scene through various types of information collection terminals (such as drones, smart sensors, surveillance cameras, mobile terminals, etc.). Different types of information terminals obtain different types of data, such as temperature, humidity, smoke concentration, fire images, audio (such as explosion sounds), degree of structural damage, meteorological information, etc.

[0039] More specifically, various sensors (such as temperature and humidity sensors, CO2 sensors, gas detectors), surveillance cameras, drones, robots and other collection equipment are deployed at the fire scene. These terminals collect data on fire spread, fire source location, smoke diffusion, building structure damage, etc. in real time. Each information collection terminal transmits the information collected on site to the data processing center or cloud platform in real time through a wireless network or wired connection. All collected on-site information is integrated through a multi-terminal information transmission protocol to form multi-terminal on-site information of the fire scene. Through multiple types of information collection terminals, comprehensive monitoring of the fire scene is achieved to ensure the comprehensiveness and real-time nature of the information, and the timely transmission and integration of multi-source data, to provide accurate and comprehensive data support for subsequent real-time restoration and simulation.

[0040] More specifically, the model parts related to the fire scene are retrieved from the pre-built urban 3D simulation platform. These models include building structures, road networks, environmental geographic information, etc., and based on the spatial structure information provided by these models, basic data is provided for the real-time restoration simulation. According to the geographical location and scope of the fire, the building models, road traffic models, firefighting facilities and other information of the relevant areas are retrieved from the urban 3D simulation platform. The original model can reflect the spatial structure and environment of the fire scene, and provide the basis for simulations such as fire spread and resource allocation. The multi-terminal information collected from the scene (such as temperature, smoke, and fire intensity) is integrated with the 3D model data to provide data support for subsequent real-time restoration and simulation. The precise model obtained through the 3D simulation platform provides accurate geographic information for the multi-source real-time simulation of the fire scene, and effectively integrates the on-site information with the urban model data to ensure the spatial accuracy of the simulation model.

[0041] More specifically, the multi-terminal on-site information is divided into multiple feedback areas (such as fire source areas, damaged building areas, evacuation passages, etc.), and each area corresponds to different types of data feedback. The on-site data of each area is analyzed, such as fire spread speed, temperature distribution, smoke concentration, etc. Based on multimodal learning, the system will extract the actual characteristics of each area, such as fire intensity, spread mode, smoke propagation path, building damage, etc., and improve the understanding and accuracy of the fire scene through multimodal data fusion (such as images, temperature, sound, etc.). Through the multimodal real-time restoration model, it can comprehensively consider data in multiple dimensions such as images, temperature, gas concentration, etc., to improve the accuracy of fire scene restoration. According to the data feedback from different areas, it can finely analyze the details of the fire scene, such as the location of the fire source, the trend of fire spread, damage to the building structure, etc., to optimize the simulation accuracy.

[0042] More specifically, after extracting the real-time features of the fire scene, the original three-dimensional urban model is modified based on these features to generate a realistic and accurate multi-source real-time simulation model. This simulation model will reflect the real situation of the fire scene and provide a basis for subsequent command decisions and resource allocation. According to the real-time features extracted from multi-terminal on-site information, the original model of each area is modified in real time. This includes modifying the location of the fire source, temperature distribution, fire spread speed, etc., dynamically updating information such as the degree of damage to the building, road traffic conditions, and personnel evacuation status, and splicing the modified model areas to finally form a multi-source real-time simulation model that can reflect the situation at the fire scene in real time.

[0043] More specifically, after the simulation model is completed, the fire scene can be visualized through a multi-source live simulation model to assist commanders in adjusting rescue strategies in real time. By using a multi-source live simulation model to visualize the fire scene in real time, commanders can view the spread of fire, resource allocation, and personnel evacuation in a dynamic environment. The system analyzes changes in the fire scene in real time based on the simulation results and gives optimized rescue suggestions, such as adjusting rescue routes, increasing rescue forces, and activating backup channels. Commanders adjust rescue strategies based on system feedback, and the system automatically updates the simulation model to ensure the real-time and accuracy of each decision.

[0044] It is understandable that through multimodal data fusion and original model correction, high-precision real-time restoration of the fire scene can be achieved. Through multi-terminal information collection and simulation model feedback, it is ensured that the dynamic changes of the fire scene can be reflected and corrected in a timely manner. Based on the accurate multi-source real-time simulation model, reliable decision-making support is provided for rescue commanders, optimizing resource allocation and task execution. The real-time update and precise feedback of the simulation model improve the speed and accuracy of emergency response, and minimize the losses caused by fire.

[0045] Preferably, the step of pre-training the multimodal reality restoration model includes: S101: Construct a multimodal reality restoration model consisting of a multi-source channel architecture, an intermediate fusion layer, a common mode analysis layer, and an image convolution layer; S102: collecting corresponding forms of source data for a training target according to various types of source channels in the multiple source channel architecture to obtain a first part of training data; wherein the training target is a fire scene for collecting training data; S103: collecting data of actual characteristics of the training target to obtain a second part of training data; S104: receiving the first part of training data through the multiple information source channel architecture, and causing the intermediate fusion layer to perform preliminary data fusion analysis on the received first part of training data to obtain multimodal features corresponding to the first part of training data; S105: performing common space mapping of feature representation content on the multimodal features through the common mode analysis layer, so as to express the information feedback correlation of each source channel of the multimodal features in the same space, so as to obtain the common mode correlation feature of the first part of training data; S106: Substituting the common mode correlation features into the image convolution layer, and performing supervised training on the common mode correlation features in the image convolution layer according to the second part of training data, so as to obtain a mapping relationship between the common mode correlation features and the second part of training data; S107: configuring the model parameters of the image convolution layer according to the mapping relationship, so that the image convolution layer has the performance of analyzing the real-time features of the common-mode correlation features according to the mapping relationship, thereby completing the pre-training of the multi-modal real-time restoration model.

[0046] Specifically, a multimodal scene reconstruction model consisting of a multi-source channel architecture, an intermediate fusion layer, a common mode analysis layer, and an image convolution layer is constructed. The model is designed to process and fuse multimodal data from different sources (such as temperature sensors, images, gas sensors, etc.) and perform accurate fire scene reconstruction.

[0047] More specifically, multiple source channels are constructed, each of which receives different types of input data, such as visual data, sensor data, audio data, etc. Each source channel is processed according to the characteristics of its input type. For example, visual data is extracted through a convolutional neural network (CNN), temperature data is processed through a fully connected layer, and audio data is analyzed through a sound processing network.

[0048] More specifically, after preliminary processing, the data in the multi-source channels are initially fused through the intermediate fusion layer, which integrates the data from different sources to form a unified multimodal feature representation.

[0049] More specifically, the common mode analysis layer further processes the fused multimodal features and maps these features to a common space. The goal of this layer is to spatially align different types of information sources in order to analyze the correlation between the information sources in the same space.

[0050] More specifically, the image convolution layer is one of the core parts of the model, which is used to extract deep features in image data and convert multimodal information into an actual fire scene restoration model.

[0051] It can be seen that the multi-source channel architecture can flexibly handle complex data types from different sources and enhance the processing capabilities of the model. The intermediate fusion layer and the common mode analysis layer effectively fuse and align data from different sources, thereby improving the model's ability to process and understand multimodal information.

[0052] More specifically, according to the design of the multiple-source channel architecture, the first part of the training data is different types of source data related to the fire scene. These data include temperature, humidity, smoke concentration, sound, video images, etc., covering the multi-dimensional information in the fire scene. A variety of sensors and equipment (such as temperature sensors, gas detectors, cameras, drones, etc.) are used to collect data on the fire scene. These data include temperature changes when the fire occurs, the location of the fire source, smoke diffusion, sound data, image data, etc. These data from different sources are calibrated and preprocessed to ensure that they can be used as unified input data in subsequent training processes.

[0053] More specifically, the second part of training data (actual feature data) is collected. The second part of training data is the actual feature data for the fire scene, which mainly includes the speed of fire spread, the temperature distribution of the fire source, the spread of smoke, the degree of damage to the building, etc. Through simulation of the fire scene or actual fire events, real-time data including fire spread, damage assessment, and personnel evacuation are collected. These data are usually collected in real time through intelligent video surveillance systems, temperature sensors, fire detectors and other equipment. The collected actual feature data are calibrated to ensure that these data can be effectively connected with the first part of the training data of the model. The calibration process involves manual annotation of key features of the fire scene to ensure the accuracy of the data.

[0054] More specifically, during the model training process, the first part of the training data (source data) is combined with the second part of the training data (actual feature data) through the intermediate fusion layer and the common mode analysis layer to extract common mode correlation features. Multiple source channels receive the first part of the training data and send the data to the intermediate fusion layer for preliminary fusion analysis. The intermediate fusion layer combines data from different sources (such as temperature, images, audio, etc.) to form multimodal features. These multimodal features are mapped to the same space through the common mode analysis layer to analyze the relationship and mutual influence between different sources. The goal of this step is to align data from different sources (such as images, sounds, temperature, etc.) in the same space, identify their common features and perform correlation analysis.

[0055] More specifically, through the intermediate fusion layer and the common mode analysis layer, data from different sources can be effectively fused and their common mode features in the same space can be extracted. The common mode analysis layer enables data from different sources to be associated in space, improving the comprehensive understanding ability of the model. The common mode correlation features are input into the image convolution layer, and the image convolution layer is supervised and trained using the second part of the training data to learn the ability to extract actual features of the fire scene from the common mode correlation features.

[0056] More specifically, the common-mode correlation features are input into the image convolution layer together with the second part of training data (actual feature data). During the training process, the image convolution layer adjusts the weights and parameters through back propagation to achieve the mapping between the common-mode correlation features and the actual features. The convolution layer parameters are configured according to the training results to optimize its performance in fire scene restoration. After the configuration is completed, the convolution layer can efficiently extract accurate actual features from the common-mode correlation features to achieve fire scene restoration.

[0057] It can be understood that through supervised training, the image convolution layer enables the model to efficiently extract the actual features of the fire scene from the fused features. Through this training process, the image convolution layer can accurately identify and reflect the real characteristics of the fire in the fire restoration simulation, improving the practicality and effectiveness of the model.

[0058] Preferably, the step of analyzing the real-time rescue strategy of the multi-source real-time simulation model by using a pre-trained rescue decision intelligent model to obtain a personalized rescue plan corresponding to the fire scene includes: S21: Obtain unit characteristic information of available rescue units, and perform format conversion and statistical summarization on the unit characteristic information of available rescue units to obtain rescue deployment elements applicable to the fire scene; wherein the rescue units include personnel units, equipment units and vehicle units; S22: generating a rescue plan for the multi-source real-time simulation model according to the rescue deployment elements through the pre-trained rescue decision intelligent model, so as to obtain several preliminary rescue plans for the fire scene; S23: performing multi-dimensional plan evaluation processing on each of the preliminary rescue plans to obtain a plan evaluation label for each of the preliminary rescue plans; wherein the plan evaluation label includes a time efficiency feature, a resource efficiency feature, and a success probability feature; S24: Displaying the plan evaluation labels of each of the preliminary rescue plans to the rescue commander to receive a plan determination instruction from the rescue commander for the preliminary rescue plan, and determining the designated preliminary rescue plan as the plan to be optimized from among the preliminary rescue plans according to the plan determination instruction; S25: Performing real-time optimization processing on the plan to be optimized according to the multi-source real-time simulation model to obtain a personalized rescue plan.

[0059] Specifically, information such as the capabilities, skills, number, available time, and health status of rescue personnel is collected; characteristic information of equipment units including fire trucks, rescue equipment (such as fire extinguishers, fire pumps, etc.), medical equipment, etc., their available status, storage location, capacity, effectiveness, etc.; characteristic information of vehicle units such as rescue vehicles, drones, etc., their load capacity, range, speed and other parameters are collected.

[0060] More specifically, the characteristic information of rescue units is formatted in a unified manner to facilitate subsequent processing and application. This information is summarized and statistically analyzed to extract the key parameters of each rescue unit in a specific fire scenario, such as response speed, rescue efficiency, etc. By collecting and standardizing the characteristic information of rescue units in detail, it is possible to ensure that there is a reliable resource basis for subsequent rescue decisions.

[0061] More specifically, by processing the unit characteristic information of the available rescue units, rescue deployment elements related to the fire scene are generated. These elements will affect the subsequent formulation of rescue strategies. Combined with multi-source real-time data at the fire scene (such as fire spread, dangerous areas, locations of trapped people, etc.), it is determined which types of rescue units are most suitable. According to factors such as the type of fire, the affected area, and the on-site environment, specific deployment elements are extracted, such as how many firefighters are needed, what type of equipment, and what kind of vehicle support are needed.

[0062] It can be seen that by acquiring and processing fire scene data in real time, the deployment elements of rescue resources can be dynamically adjusted to ensure that rescue resources match the on-site needs to the maximum extent. The generated rescue deployment elements can accurately allocate necessary resources according to the specific conditions of the fire scene to avoid excessive or insufficient resource investment.

[0063] More specifically, after obtaining the rescue deployment elements, a pre-trained rescue decision-making intelligent model is used to generate several preliminary rescue plans. These plans are based on the results of resource allocation, task allocation and real-time scenario simulation, and the generated rescue deployment elements are input into the pre-trained rescue decision-making intelligent model. The model has been trained based on historical data and expert experience, and can process complex multi-source data and generate reasonable rescue plans.

[0064] More specifically, based on the multi-source real-time simulation data and rescue resources at the fire scene, the model will automatically generate several preliminary rescue plans, covering different strategies and plans, for rescue commanders to choose and adjust. With the help of the rescue decision-making intelligent model, it can combine multimodal data (such as fire spread maps, rescue resource distribution, personnel location, etc.) to automatically generate preliminary rescue plans with high rationality. The model can generate a variety of preliminary rescue plans based on different variables to cope with the complexity and changing conditions of the fire scene.

[0065] More specifically, after each preliminary rescue plan is generated, it needs to undergo a multi-dimensional evaluation to ensure that the most suitable rescue plan for the current fire scene is selected. The evaluation indicators include time efficiency characteristics: evaluating the execution time of the plan, including the overall time from receiving the alarm to completing the rescue, resource efficiency characteristics: evaluating the efficiency of resource utilization, such as whether resources such as personnel, equipment, and vehicles are reasonably allocated and whether waste is minimized, success probability characteristics: evaluating the probability of success of the plan, which can be measured through historical data, simulation and deduction, etc. According to these evaluation indicators, each preliminary rescue plan is scored and ranked to obtain an evaluation label for the plan.

[0066] More specifically, through comprehensive evaluation of multiple dimensions such as time efficiency, resource efficiency and probability of success, we can fully understand the pros and cons of each rescue plan. The evaluation labels provide clear decision-making support basis for rescue commanders, which helps to quickly select the optimal plan. The evaluation labels of each preliminary rescue plan are displayed to the rescue commanders, and their opinions on the selection or adjustment of the plan are received.

[0067] More specifically, according to the decision instructions of the rescue commander, the rescue plan to be optimized is selected, and the optimization process is performed through a multi-source live simulation model to finally obtain a personalized rescue plan.

[0068] More specifically, a multi-source live simulation model is used to simulate and optimize the rescue plan to be optimized. This process combines on-site data (such as changes in fire intensity, trapped personnel, etc.) and known resource configurations, adjusts the rescue strategy, and generates a personalized rescue plan that best suits the current fire scene through optimization processing. Through live optimization processing, the rescue strategy can be dynamically adjusted according to the latest changes at the fire scene to ensure the accuracy and flexibility of decision-making. According to the specific scene requirements, a highly personalized rescue plan is generated to ensure that each fire scene can obtain the best rescue plan.

[0069] Preferably, the pre-training step of the rescue decision intelligent model includes: S221: Constructing a rescue decision intelligent model consisting of an input unit, a decision algorithm unit and an output unit; wherein the input unit is used to input rescue deployment elements and on-site actual characteristics, the on-site actual characteristics are characteristic descriptions of the multi-source real-time simulation model, the decision algorithm unit is used to perform rescue decision processing on the rescue deployment elements and on-site actual characteristics according to a preset decision algorithm to obtain a number of preliminary rescue plans, and the output unit is used to output the plurality of the preliminary rescue plans; S222: The decision algorithm applied by the decision algorithm unit is derived from a model training result of a deep learning network model, and the deep learning network model is a feedforward neural network model, a convolutional neural network model, or a generative adversarial network model; The model training steps of the deep learning network model include: S223: Pre-collecting training data for model training; wherein the training data includes a first data group and a second data group in a corresponding relationship in a historical case library, an expert knowledge base, and a plan library, wherein the first data group includes rescue deployment elements and on-site actual situation characteristics, and the second data group includes a preliminary rescue plan corresponding to the first data group; S224: Substitute the training data into the deep learning network model, and allow the deep learning network model to perform model training according to the training data to obtain the association relationship between the first data group and the second data group in the training data; wherein the association relationship between the first data group and the second data group is the model training result of the deep learning network model, that is, the decision algorithm applied by the decision algorithm unit.

[0070] Specifically, the core of the rescue decision-making intelligent model is composed of an input unit, a decision algorithm unit and an output unit, and can make intelligent decisions based on on-site conditions and resource allocation.

[0071] More specifically, the input unit mainly receives two types of information: rescue deployment elements: including status information and deployment requirements of available rescue resources (such as personnel, equipment, vehicles, etc.); on-site real-time characteristics: including on-site data characteristics obtained from multi-source real-time simulation models, such as the spread of fire, disaster-stricken areas, distribution of trapped people, etc.

[0072] More specifically, the decision algorithm unit is used to process the input rescue deployment elements and on-site actual characteristics according to a preset decision algorithm, and generate a preliminary rescue plan. The core of the decision algorithm unit is to train through a deep learning model to learn how to generate effective plans based on different scenarios and resource configurations.

[0073] More specifically, the output unit displays the preliminary rescue plan processed by the decision algorithm unit to the rescue commander. The preliminary plan is a preliminary decision made based on on-site data and available resources, and there are usually multiple options to choose from.

[0074] It can be understood that the design of the model architecture ensures that the process from input to output is clear and efficient, can achieve effective data flow transmission and decision support, modularizes different tasks so that each module can focus on a specific task, and improves the scalability and maintainability of the system.

[0075] More specifically, in order to enable the decision algorithm unit to handle complex rescue decision-making problems, the deep learning model used (such as a feedforward neural network, a convolutional neural network, or a generative adversarial network) needs to be pre-trained based on the training data to establish the correlation between the actual on-site characteristics and the rescue plan.

[0076] More specifically, training data collection includes a historical case library: collecting historical rescue cases and recording resource allocation and rescue plans in different scenarios; an expert knowledge base: collecting best practices and rules for disaster rescue through expert experience; and a plan library: collecting and organizing historical or simulated plans, including successful rescue plans in different scenarios.

[0077] More specifically, the training data format includes a first data group: including rescue deployment elements (resource allocation, personnel and equipment, etc.) and on-site actual characteristics (fire spread, on-site environment, etc.), and a second data group: containing the corresponding preliminary rescue plan, that is, the rescue plan generated under given rescue deployment elements and on-site actual characteristics.

[0078] More specifically, the collected first data group and second data group are substituted into the deep learning network model for training. A large-scale labeled data set is usually used to train the model. The model type can be a feedforward neural network (FNN): suitable for processing simple input-output relationships, a convolutional neural network (CNN): if the on-site data is an image or two-dimensional data (such as a fire spread map), CNN can effectively extract local features and perform spatial analysis, and a generative adversarial network (GAN): used to generate more complex or realistic rescue plans, suitable for generating plans for novel and unprecedented disaster scenarios.

[0079] More specifically, during the model training process, the deep learning model adjusts the weights through the back-propagation algorithm so that the model can learn the association between the first data group and the second data group. Specifically, the model will learn to generate reasonable rescue plans under given rescue deployment elements and on-site characteristics.

[0080] It is understandable that deep learning models can learn the complex relationship between scene features and resource allocation through large amounts of data, so that the model can make reasonable decisions when faced with new or unknown scenarios. By using historical data, expert knowledge and plan libraries, the model can make full use of existing resources and quickly learn effective decision-making rules. Different neural network architectures can be selected according to different tasks and scenarios, allowing the model to cope with various complex rescue environments.

[0081] Preferably, the step of performing real-time optimization processing on the plan to be optimized according to the multi-source real-time simulation model to obtain a personalized rescue plan includes: S251: performing a live execution simulation on the plan to be optimized according to the multi-source live simulation model to obtain a plan effect distribution; wherein the plan effect distribution is used to describe the plan execution effect of each specific position of the multi-source live simulation model corresponding to the plan to be optimized; S252: Analyzing the key optimization positions and position optimization effects of the plan effect distribution according to the multi-source live simulation model to obtain a target effect distribution; wherein the target effect distribution is used to describe the plan execution effect theoretically required for each specific position of the multi-source live simulation model; S253: performing a cyclic optimization process on the plan to be optimized according to the target effect distribution, so as to obtain a personalized rescue plan whose execution effect meets the preset conditions of the target effect distribution.

[0082] Specifically, the plan to be optimized is usually one of the multiple rescue plans generated through the preliminary decision-making stage. In this stage, the model will be further optimized according to the actual scenario and resource allocation.

[0083] More specifically, a multi-source live simulation model is used to simulate the optimized plan and the execution effect of the plan in different scenarios. These scenarios usually involve different geographical locations, environmental characteristics and changes at different time nodes of the disaster site. The multi-source live simulation model may integrate multiple data sources, such as satellite images, drone monitoring, sensor data, historical cases, on-site real-time feedback, etc.

[0084] More specifically, the results of the simulation will be represented by the distribution of plan effects. This distribution describes the execution effect of the plan to be optimized at each specific location and time point. The plan effect usually includes multiple dimensions, such as resource scheduling efficiency, time response, personnel safety, disaster spread control, etc. Multi-source live simulation provides a comprehensive execution evaluation perspective, taking into account the dynamic changes of the on-site environment and the multiple influencing factors of the plan. Through simulation, the performance of the plan in the actual environment can be obtained in real time, providing data support for subsequent optimization.

[0085] More specifically, after obtaining the distribution of plan effects, an analytical algorithm is used to conduct an in-depth analysis of these data to find out the key optimization positions in the plan execution process, that is, which locations or resource configurations have optimization space or deficiencies, and to conduct position optimization effect analysis on the key optimization positions, and evaluate the impact of different optimization strategies on these positions. The goal of this stage is to determine which specific locations or resource configurations can be optimized to further improve the rescue effect, and to form a target effect distribution through target effect analysis. The target effect distribution is derived from theoretical models, historical data, and expert experience. It is the ideal execution effect that should be achieved in theory at each specific location. This distribution reflects the optimal rescue resource allocation, personnel scheduling, and execution timing.

[0086] It is understandable that through analysis and evaluation, it is possible to accurately determine which locations and links have insufficient execution or room for optimization under the existing plan, so as to carry out targeted optimization. The target effect distribution has both theoretical basis and actual scenario requirements to ensure the feasibility and efficiency of the optimization plan.

[0087] More specifically, according to the target effect distribution, a cyclic optimization algorithm is used to adjust the key optimization positions of the plan to be optimized. This process converges the execution effect of the plan at these key positions towards the ideal state of the target effect distribution. The cyclic optimization may include multiple iterations, and each round of iteration is adjusted according to the gap between the target effect and the actual effect. The optimization process may involve resource allocation, personnel deployment, timing adjustment and other aspects. In each round of optimization, the model will be adjusted based on the difference between the actual simulation results and the target effect distribution. Iterative algorithms (such as genetic algorithms, particle swarm optimization, gradient descent, etc.) are usually used to find the optimal solution. After multiple rounds of optimization and adjustment, a personalized rescue plan that is closest to the target effect distribution is finally obtained. This plan has the best effect both in theory and in actual implementation, and adapts to the complex situation and dynamic changes on the scene.

[0088] It is understandable that through loop optimization, the plan can be fine-tuned according to the needs of each specific location, ensuring that every resource allocation and every action decision are in line with actual needs. The optimized personalized plan can respond to disasters more efficiently in actual scenarios, improve rescue efficiency, reduce resource waste, and ensure the safety of rescuers and disaster victims.

[0089] Preferably, real-time information collection is performed on the rescue units to obtain the location information and status information of each rescue unit, and the steps of constructing a rescue unit simulation model based on the multi-source live simulation model according to the location information and status information of each rescue unit include: S31: Acquire unit characteristic information of a rescue unit, and construct a rescue unit simulation model for performing digital simulation feedback on the rescue unit according to the unit characteristic information of the rescue unit; S32: collecting real-time information of the rescue unit to obtain location information and status information of the rescue unit; S33: Setting the rescue unit simulation model according to the positioning information, and annotating the rescue unit simulation model in real time according to the status information, so as to construct the rescue unit simulation model based on the multi-source real-time simulation model.

[0090] Specifically, determine the type of each rescue unit, such as medical team, fire brigade, search and rescue team, etc. Each type of unit undertakes different tasks in the rescue process, collect information such as the unit's equipment, supplies, staffing, technical capabilities, etc., and determine the response capabilities and operational limitations, including each unit's operating time, response speed, effective operating range and other characteristics.

[0091] More specifically, based on the above unit characteristic information, a rescue unit simulation model for digital simulation feedback is established. The model will specifically consider the resource capabilities, task execution logic and interactive relationships of each unit to ensure that the simulation model can accurately reflect the behavior of the unit in the actual scenario. The rescue unit simulation model is not only used to display the task execution status of the unit, but also to provide feedback based on the simulation results to help decision makers predict and optimize their future behavior.

[0092] More specifically, real-time information collection is carried out on rescue units to obtain positioning information and status information. The geographic location information of the rescue units is collected in real time through GPS, drones, ground sensors, etc. The positioning information includes the current location of the rescue units, action path, mission area, etc. The status information of each rescue unit is collected, such as: mission execution status (such as whether the mission is being performed, mission progress, completion degree, etc.), resource status (such as whether the equipment is normal, whether the materials are sufficient, whether the personnel are healthy, etc.), time schedule (such as the scheduled time to complete the mission, real-time response time, etc.), and the positioning information and status information are transmitted to the central command platform through wireless communication networks (such as 5G, satellite communications, VHF radio, etc.).

[0093] It is understandable that the location information of each unit is obtained in real time to ensure that the dynamic changes in the rescue process can be reflected in a timely manner, and the real-time status of the rescue units (resources, task progress, etc.) is comprehensively collected to provide accurate basic data for subsequent decision-making and simulation.

[0094] More specifically, the real-time collected positioning information is input into the rescue unit simulation model, and the position parameters in the model are adjusted according to the current position of each unit. This step helps simulate the actual action path of the unit in the disaster area, ensuring that the behavior of each unit in the model is consistent with the actual situation. According to different rescue tasks and geographical locations, the task area and action range in the model are set. The model will update the path of the rescue unit in real time according to the positioning information, determine the task area, action direction and expected arrival time of each unit, etc. For example, the fire brigade is assigned to a specific fire scene based on the real-time positioning information, and the starting point, path, expected end point and other information of the task are updated in real time in the simulation model.

[0095] More specifically, the real-time collected status information (such as task progress, resource status, etc.) is annotated into the rescue unit simulation model. The status of each unit can be reflected in the simulation in real time through different labels (such as "standby", "executing", "resource shortage", "task completed", etc.). According to the changes in status information, such as equipment damage, resource shortage, etc., the parameters in the model are dynamically adjusted to reflect the unit's current capabilities and task execution status. The model updates the behavior of the rescue unit according to the real-time changes in status information. For example, if a unit's equipment is damaged or personnel are injured or killed, the simulation model will mark the unit as "restricted" and adjust the task execution strategy or dispatch other units to replace it accordingly.

[0096] Understandably, the real-time status of each unit is clearly displayed through the model’s annotation system, allowing the command center to quickly grasp the current status of each rescue unit. The real-time status annotation can reflect the immediate situation of the rescue unit (such as resource changes, task progress, etc.), allowing rescue decisions to be flexibly adjusted based on this real-time information.

[0097] More specifically, real-time data from different sources (such as meteorological data, earthquake monitoring data, traffic conditions, etc.) are integrated with the positioning information and status information of the rescue units. Through the fusion of multi-source data, a more comprehensive and accurate rescue unit simulation model is constructed. Based on the multi-source real-time simulation model, the behavior of the rescue unit simulation model is combined with a variety of external environmental factors (such as weather changes, disaster site dynamics, etc.) for simulation to ensure that the model reflects a comprehensive and dynamic rescue scenario. The model also needs to consider the coordinated operations between units, for example, resource scheduling and task coordination when multiple units jointly rescue an area.

[0098] Preferably, the steps of analyzing the specific rescue execution steps of the multi-source live simulation model having the rescue unit simulation model according to the personalized rescue plan to obtain the specific rescue execution steps include: S41: analyzing the starting execution position of the plan on the multi-source live simulation model according to the personalized rescue plan, so as to obtain the starting position of each rescue unit executing the personalized rescue plan corresponding to the multi-source live simulation model; S42: according to the real-time positioning of each rescue unit simulation model in the multi-source live simulation model, performing a step analysis of the initial deployment of each rescue unit at the initial position to obtain the initial deployment step of each rescue unit, and taking each initial deployment step as the specific rescue execution step to be executed first; S43: analyzing the theoretical execution steps of the multi-source real-time simulation model for each rescue unit simulation model according to the personalized rescue plan to obtain the theoretical execution steps of each rescue unit simulation model; S44: Perform rescue command processing on each rescue unit simulation model in turn according to the theoretical execution steps. During the rescue command processing, the multi-source live simulation model is updated in real time by collecting multi-terminal information from the fire scene. The rescue unit simulation model is updated in real time by collecting real-time information from the rescue units. The theoretical execution steps are corrected in real time on the real-time updated multi-source live simulation model and the rescue unit simulation model to obtain specific rescue execution steps.

[0099] Specifically, personalized rescue plans are formulated based on different rescue needs (such as fire, earthquake, flood, etc.) and the characteristics of rescue units (such as fire, medical, police, etc.). These plans usually include: mission objectives, priority rescue areas, resource allocation, role allocation of each unit, etc.

[0100] More specifically, the starting execution position of each rescue unit is analyzed in the multi-source live simulation model to determine the initial deployment position of each unit. At this time, the simulation model will take into account multiple factors, such as the geographical location of the disaster site, traffic conditions, disaster distribution, etc., and determine the starting mission area of ​​each unit based on the priority set in the rescue plan. For example, the fire unit may need to approach the source of the fire first, and the medical team should be deployed according to the distribution points of the wounded.

[0101] More specifically, using the function of the multi-source live simulation model, the starting execution position of each rescue unit is simulated and positioned according to the rescue plan. This step is the basis for ensuring that the rescue units can start to perform tasks at the correct time, place and sequence. Through personalized plan analysis, the starting position of each unit is ensured to be reasonable and accurate. Avoid waste of resources or ineffective deployment, improve rescue efficiency, optimize resource scheduling based on the situation at the disaster site and the characteristics of the units, and ensure that each unit can quickly carry out rescue operations at the most appropriate location.

[0102] More specifically, based on the real-time positioning information of each rescue unit, combined with the geographical information of the disaster area and the priority of the rescue mission, the starting position of each rescue unit is further optimized, the position changes of each unit are monitored in real time, and the starting deployment position is dynamically adjusted taking into account factors such as traffic conditions and the urgency of the rescue mission.

[0103] More specifically, the deployment analysis is performed on the starting position of each rescue unit to determine the optimal deployment steps. For example, the fire brigade can choose the shortest path deployment according to the traffic conditions, and the medical team can adjust the position according to the distribution points of the wounded. After analysis, the specific deployment plan of each unit is obtained and used as the initial step of the rescue execution.

[0104] More specifically, a multi-source live simulation model is used to simulate the initial deployment steps of each rescue unit to confirm whether the deployment plan is feasible. During the simulation, if potential problems are found (such as traffic congestion, insufficient resources, etc.), adjustments can be made. Real-time positioning information allows rescue units to flexibly adjust their starting positions to ensure efficient deployment. Through the analysis of deployment steps, the task execution path of each unit is ensured to be the shortest and fastest, thereby improving rescue efficiency and effectiveness.

[0105] More specifically, based on the personalized rescue plan and the multi-source live simulation model, the theoretical execution steps of each rescue unit are analyzed. These steps include each unit's mission objectives, action paths, resource usage, collaborative relationships, etc. For example, the theoretical execution steps of the fire brigade include extinguishing the fire source, rescuing trapped people, evacuating the surrounding people, etc.; the theoretical steps of the medical team may include classification of the wounded, medical first aid, and transfer of the wounded.

[0106] More specifically, the theoretical execution steps of each rescue unit are simulated in combination with the specific conditions of the disaster site (such as the speed of fire spread, the number of injured, traffic conditions, etc.). This step ensures that in the actual rescue process, the unit can perform the task according to the predetermined steps, avoid deviation from the task objectives, ensure that the task steps of each unit are clear and operational, avoid confusion or task omissions during the execution process, and through the simulation and analysis of the theoretical steps, discover potential problems and optimize them in advance to ensure a smooth rescue process.

[0107] More specifically, during the rescue execution process, multi-terminal information from the disaster site (such as fire spread, status of the wounded, weather changes, etc.) and the real-time status of the rescue units (such as location, resource consumption, task progress, etc.) are collected in real time, and this information is fed back to the multi-source live simulation model and the rescue unit simulation model in real time to ensure that the information in the model is always consistent with the actual situation. According to the real-time feedback data, the execution steps of the rescue units are adjusted in time. Through the real-time update of the simulation model, the execution steps of each rescue unit are corrected. For example, if a unit encounters equipment failure, the model will automatically adjust the execution task of the unit based on this information, or dispatch other units for support.

[0108] More specifically, based on real-time data and the correction of simulation models, the command center can make dynamic decisions. For example, when the rescue progress in a certain area is slow, the command center can adjust the tasks of other units to ensure the optimal allocation of resources and maximize the rescue effect. The information in the model is updated and corrected in real time, so that the task execution of each unit is always in line with the actual situation, improving the flexibility and response speed of rescue operations. Through real-time information collection and simulation model updates, the command personnel can promptly discover problems and make adjustments to ensure that the rescue mission is carried out along the optimal path.

[0109] More specifically, under the joint action of the multi-source live simulation model and the rescue unit simulation model, the real-time status, task progress, resource usage and other information of each unit are comprehensively analyzed to further optimize the rescue execution steps. The simulation model will provide real-time feedback on the actual execution of each unit, combined with the strategy adjustment of the command center to obtain the most suitable rescue execution steps. After multiple rounds of analysis and real-time adjustment, the specific rescue execution steps of each rescue unit are finally determined. These steps will serve as the basis for the implementation of the rescue operation to ensure the smooth progress of the mission. Through the combination of simulation and real-time information feedback, the execution steps of each rescue unit are always optimal, and the execution steps of each unit are effectively optimized to ensure smoother coordination between different units and improve the overall rescue effect.

[0110] Reference Figure 2 As shown, in a second aspect, the present invention provides an intelligent fire command system based on a multimodal AI big model, which is used to implement an intelligent fire command method based on a multimodal AI big model as described in any one of the first aspects, including: A live simulation module is used to collect multi-terminal information of the fire scene to obtain multi-terminal scene information of the fire scene, and to perform live restoration simulation of the fire scene according to the multi-terminal scene information through a pre-trained multimodal live restoration model to obtain a multi-source live simulation model of the fire scene; wherein the multi-terminal scene information includes text terminal information, image terminal information, video terminal information, and voice terminal information; A plan analysis module, used to analyze the real-time rescue strategy of the multi-source real-time simulation model through a pre-trained rescue decision-making intelligent model, so as to obtain a personalized rescue plan corresponding to the fire scene; A rescue simulation module is used to collect real-time information of rescue units to obtain the location information and status information of each rescue unit, and to construct a rescue unit simulation model based on the multi-source live simulation model according to the location information and status information of each rescue unit; The specific command module is used to analyze the specific rescue execution steps of the multi-source live simulation model with the rescue unit simulation model according to the personalized rescue plan to obtain the specific rescue execution steps; wherein, the specific rescue execution steps are used to assist the rescue commander to conduct real-time firefighting command for each rescue unit.

[0111] In this embodiment, for the specific implementation of each module in the above system embodiment, please refer to the above method embodiment, which will not be repeated here.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent fire command method based on a multimodal AI large model, characterized in that: include: Perform multi-terminal information collection on the fire scene to obtain multi-terminal scene information of the fire scene, and perform a live restoration simulation of the fire scene according to the multi-terminal scene information using a pre-trained multimodal live restoration model to obtain a multi-source live simulation model of the fire scene; wherein the multi-terminal scene information includes text end information, image end information, video end information, and voice end information; Analyzing the real-time rescue strategy of the multi-source real-time simulation model through the pre-trained rescue decision-making intelligent model to obtain a personalized rescue plan corresponding to the fire scene; Collecting real-time information of rescue units to obtain location information and status information of each rescue unit, and constructing a rescue unit simulation model based on the multi-source live simulation model according to the location information and status information of each rescue unit; According to the personalized rescue plan, the multi-source live simulation model with the rescue unit simulation model is analyzed for specific rescue execution steps to obtain specific rescue execution steps; wherein, the specific rescue execution steps are used to assist the rescue commander to conduct real-time firefighting command for each rescue unit.

2. The intelligent fire command method based on the multimodal AI large model as claimed in claim 1 is characterized in that: The steps of collecting multi-terminal information of a fire scene to obtain multi-terminal scene information of the fire scene, and performing a real-time restoration simulation of the fire scene according to the multi-terminal scene information by using a pre-trained multi-modal real-time restoration model to obtain a multi-source real-time simulation model of the fire scene include: Collecting information from the fire scene through multiple types of information collection terminals that arrive at the fire scene, so as to obtain on-site information from various types of information terminals, the on-site information from various types of information terminals together forming multi-terminal on-site information from the fire scene; Retrieving the model portion corresponding to the fire scene from the pre-built urban three-dimensional simulation platform as the original simulation model; The multi-end scene information is disassembled and analyzed in terms of information feedback areas through a pre-trained multi-modal real-time restoration model to obtain the real-time characteristics of each model area of ​​the original simulation model, and the model real-time correction is performed on each model area based on the real-time characteristics to obtain a multi-source real-time simulation model of the fire scene.

3. The intelligent fire command method based on multimodal AI large model as claimed in claim 2 is characterized in that: The steps for pre-training the multimodal ground truth restoration model include: Construct a multimodal reality restoration model consisting of a multi-source channel architecture, an intermediate fusion layer, a common mode analysis layer, and an image convolution layer; According to various types of information source channels in the multiple information source channel architecture, corresponding forms of information source data are collected for the training target to obtain a first part of training data; wherein the training target is a fire scene for collecting training data; Collecting data of actual characteristics of the training target to obtain a second part of training data; Receiving the first part of training data through the multiple information source channel architecture, and causing the intermediate fusion layer to perform preliminary data fusion analysis on the received first part of training data to obtain multimodal features corresponding to the first part of training data; Performing common spatial mapping of feature representation content on the multimodal features through the common mode analysis layer, so as to express the information feedback correlation of each source channel of the multimodal features in the same space, so as to obtain the common mode correlation features of the first part of the training data; Substituting the common mode correlation features into the image convolution layer, and performing supervised training on the common mode correlation features in the image convolution layer according to the second part of training data to obtain a mapping relationship between the common mode correlation features and the second part of training data; The model parameters of the image convolution layer are configured according to the mapping relationship so that the image convolution layer has the performance of analyzing the real-time features of the common-mode correlation features according to the mapping relationship, thereby completing the pre-training of the multi-modal real-time restoration model.

4. The intelligent fire command method based on multimodal AI large model as claimed in claim 1 is characterized in that: The steps of analyzing the real-time rescue strategy of the multi-source real-time simulation model by using the pre-trained rescue decision intelligent model to obtain a personalized rescue plan corresponding to the fire scene include: Acquire unit characteristic information of available rescue units, and perform format conversion and statistical summarization on the unit characteristic information of available rescue units to obtain rescue deployment elements applicable to the fire scene; wherein the rescue units include personnel units, equipment units and vehicle units; Generate a rescue plan for the multi-source live simulation model according to the rescue deployment elements through a pre-trained rescue decision-making intelligent model to obtain several preliminary rescue plans for the fire scene; Performing multi-dimensional plan evaluation processing on each of the preliminary rescue plans to obtain a plan evaluation label for each of the preliminary rescue plans; wherein the plan evaluation label includes a time efficiency feature, a resource efficiency feature, and a success probability feature; By showing the plan evaluation labels of each of the preliminary rescue plans to the rescue commander, the rescue commander receives a plan determination instruction for the preliminary rescue plan, and determines the designated preliminary rescue plan as the plan to be optimized from each of the preliminary rescue plans according to the plan determination instruction; The plan to be optimized is subjected to real-time optimization processing according to the multi-source real-time simulation model to obtain a personalized rescue plan.

5. The intelligent fire command method based on multimodal AI large model as claimed in claim 4 is characterized in that: The pre-training steps of the rescue decision intelligent model include: Constructing a rescue decision intelligent model consisting of an input unit, a decision algorithm unit and an output unit; wherein the input unit is used to input rescue deployment elements and on-site actual characteristics, the on-site actual characteristics are characteristic descriptions of the multi-source real-time simulation model, the decision algorithm unit is used to perform rescue decision processing on the rescue deployment elements and on-site actual characteristics according to a preset decision algorithm to obtain a number of preliminary rescue plans, and the output unit is used to output a number of the preliminary rescue plans; The decision algorithm applied by the decision algorithm unit is derived from a model training result of a deep learning network model, and the deep learning network model is a feedforward neural network model, a convolutional neural network model, or a generative adversarial network model; The model training steps of the deep learning network model include: Pre-collect training data for model training; wherein the training data includes a first data group and a second data group in a corresponding relationship in a historical case library, an expert knowledge base, and a plan library, the first data group includes rescue deployment elements and on-site actual characteristics, and the second data group includes a preliminary rescue plan corresponding to the first data group; Substitute the training data into the deep learning network model, and allow the deep learning network model to perform model training according to the training data to obtain the association relationship between the first data group and the second data group in the training data; wherein the association relationship between the first data group and the second data group is the model training result of the deep learning network model, that is, the decision algorithm applied by the decision algorithm unit.

6. The intelligent fire command method based on multimodal AI large model as claimed in claim 4 is characterized in that: The step of performing real-time optimization processing on the plan to be optimized according to the multi-source real-time simulation model to obtain a personalized rescue plan includes: Performing a live execution simulation on the plan to be optimized according to the multi-source live simulation model to obtain a plan effect distribution; wherein the plan effect distribution is used to describe the plan execution effect of each specific position of the multi-source live simulation model corresponding to the plan to be optimized; According to the multi-source live simulation model, the key optimization positions and position optimization effects of the plan effect distribution are analyzed to obtain the target effect distribution; wherein the target effect distribution is used to describe the plan execution effect theoretically required for each specific position of the multi-source live simulation model; The plan to be optimized is subjected to a cyclic optimization process according to the target effect distribution to obtain a personalized rescue plan whose execution effect meets the preset conditions of the target effect distribution.

7. The intelligent fire command method based on multimodal AI large model as claimed in claim 1 is characterized in that: The steps of collecting information of rescue units in real time to obtain the location information and status information of each rescue unit, and constructing a rescue unit simulation model based on the multi-source live simulation model according to the location information and status information of each rescue unit include: Acquire unit characteristic information of a rescue unit, and construct a rescue unit simulation model for performing digital simulation feedback on the rescue unit according to the unit characteristic information of the rescue unit; Collecting real-time information of the rescue unit to obtain the location information and status information of the rescue unit; The rescue unit simulation model is set up according to the positioning information, and the real-time status of the rescue unit simulation model is annotated according to the status information, so as to realize the construction of the rescue unit simulation model based on the multi-source real-time simulation model.

8. The intelligent fire command method based on multimodal AI large model as claimed in claim 1 is characterized in that: The steps of analyzing the specific rescue execution steps of the multi-source live simulation model having the rescue unit simulation model according to the personalized rescue plan to obtain the specific rescue execution steps include: Analyzing the starting execution position of the plan on the multi-source live simulation model according to the personalized rescue plan to obtain the starting position of each rescue unit executing the personalized rescue plan corresponding to the multi-source live simulation model; According to the real-time positioning of each rescue unit simulation model in the multi-source live simulation model, the starting position of each rescue unit is analyzed for the steps of initial deployment to obtain the initial deployment steps of each rescue unit, and each initial deployment step is used as the specific rescue execution step to be executed first; Analyzing the theoretical execution steps of the multi-source live simulation model for each of the rescue unit simulation models according to the personalized rescue plan to obtain the theoretical execution steps of each of the rescue unit simulation models; According to the theoretical execution steps, rescue command processing is performed on each rescue unit simulation model in turn. During the rescue command processing, the multi-source live simulation model is updated in real time through multi-terminal information collection at the fire scene, and the rescue unit simulation model is updated in real time through real-time information collection of the rescue unit. The real-time updated multi-source live simulation model and the rescue unit simulation model are corrected in real time according to the theoretical execution steps to obtain specific rescue execution steps.

9. An intelligent fire command system based on a multimodal AI large model, characterized in that: An intelligent fire command method based on a multimodal AI large model for implementing any one of claims 1 to 8, comprising: A live simulation module is used to collect multi-terminal information of the fire scene to obtain multi-terminal scene information of the fire scene, and to perform live restoration simulation of the fire scene according to the multi-terminal scene information through a pre-trained multimodal live restoration model to obtain a multi-source live simulation model of the fire scene; wherein the multi-terminal scene information includes text terminal information, image terminal information, video terminal information, and voice terminal information; A plan analysis module, used to analyze the real-time rescue strategy of the multi-source real-time simulation model through a pre-trained rescue decision-making intelligent model, so as to obtain a personalized rescue plan corresponding to the fire scene; A rescue simulation module is used to collect real-time information of rescue units to obtain the location information and status information of each rescue unit, and to construct a rescue unit simulation model based on the multi-source live simulation model according to the location information and status information of each rescue unit; The specific command module is used to analyze the specific rescue execution steps of the multi-source live simulation model with the rescue unit simulation model according to the personalized rescue plan to obtain the specific rescue execution steps; wherein, the specific rescue execution steps are used to assist the rescue commander to conduct real-time firefighting command for each rescue unit.

Images