A multimodal perception and UAV-assisted ship damage control and firefighting system

Abstract

This invention relates to the field of ship safety, specifically to a multimodal perception and UAV-assisted ship damage control and firefighting system. The system includes an intelligent perception network subsystem; a multimodal perception data stream acquisition subsystem; a central intelligent decision-making subsystem; a decision-making scheme generation and dynamic adjustment subsystem; a UAV-assisted firefighting subsystem; and a collaborative execution and data feedback system for fire response. This invention utilizes a data fusion engine to perform layered fusion of multi-dimensional perception data, integrating environmental parameters, images, videos, vibration signals, and other information. Combined with a digital twin baseline model of the ship's entire lifecycle, it can accurately determine the type, severity, core location of the fire source, and spread trend, clarify the protection priority and avoidance areas for critical equipment, and provide comprehensive and multi-dimensional data support for decision-making.

Description

Technical Field

[0001] This invention relates to the field of ship safety, specifically to a ship damage control and firefighting system that combines multimodal perception with unmanned aerial vehicle (UAV) collaboration. Background Technology

[0002] As core equipment for maritime transportation and operations, ships have complex compartment structures and highly diverse functional zones. The environmental characteristics, equipment density, and material types vary significantly between different compartments, placing extremely high demands on the targeted and timely nature of fire monitoring and emergency response. Existing ship damage control and firefighting systems have numerous technical limitations, making it difficult to meet the precise response needs in complex compartment environments: Fire perception is limited in scope, with existing systems relying heavily on single-type sensors for data collection, lacking collaborative verification of multimodal data, and failing to establish differentiated perception standards for the environmental characteristics of different compartments. This makes them susceptible to environmental interference such as ship vibration and fluctuations in compartment temperature and humidity, leading to frequent false alarms and missed alarms. Furthermore, the traditional communication links used by sensor nodes are unstable; when nodes fail or transmission links are blocked, data interruptions easily occur, affecting the timeliness of early fire warnings. Moreover, the accuracy of fire assessment is insufficient; existing systems rely solely on isolated monitoring parameters to determine fire situations, lacking in-depth fusion analysis of multi-source data such as environmental parameters, image information, and equipment vibration. This makes it difficult to accurately identify the type of fire, its core location, and its spread trend, resulting in a lack of scientific basis in decision-making and blind resource allocation. Summary of the Invention

[0003] This invention addresses the technical problems existing in the prior art by providing a ship damage control and firefighting system that combines multimodal perception with UAV collaboration.

[0004] The technical solution of this invention to solve the above-mentioned technical problems is as follows: a ship damage control and firefighting system that combines multimodal perception and unmanned aerial vehicle (UAV) collaboration, the system comprising: Intelligent sensing network subsystem; used to collect sensing data from ship cabins through multimodal sensor nodes connected by a Mesh self-organizing network, and to verify and make preliminary judgments on the collected sensing data to obtain multimodal sensing data stream; The central intelligent decision-making subsystem is used to receive multimodal sensing data streams transmitted by the intelligent sensing network subsystem, classify and sort the data streams and perform hierarchical fusion processing through the data fusion engine, generate multi-source fusion data reports, and conduct analysis. The decision command module builds models to calculate the number of drones dispatched and the amount of fire extinguishing agent used and generates decision plans. At the same time, it receives operational feedback data from the drone collaborative fire extinguishing subsystem and dynamically adjusts the decision plans. The UAV-assisted firefighting subsystem receives decision plans from the central intelligent decision-making subsystem, activates its onboard visual, acoustic, and multimodal environmental sensors, plans the optimal flight path based on ship cabin structure, environmental parameters, and wind field data, constructs a distributed cluster collaborative communication link, dynamically allocates reconnaissance and firefighting tasks, with reconnaissance UAVs performing dynamic fire monitoring and data feedback, and firefighting UAVs performing firefighting operations, simultaneously transmitting operational status and environmental change data to the central intelligent decision-making subsystem, thus completing the coordinated execution and data feedback of fire response.

[0005] In a preferred embodiment, after the intelligent sensing network subsystem has deployed all multimodal sensor nodes, it continuously collects all sensing data under the operating status of the ship's cabins and establishes an environmental baseline database containing benchmark values. Initial sensing thresholds are set for different ship cabins where the multimodal sensor nodes are deployed. The thresholds for smoke particle concentration and combustible gas detection in the electronic equipment cabin are higher than those in the cargo hold, and the thresholds for temperature and sound field vibration in the engine room are higher than those in the living quarters. This ensures that the initial sensing thresholds are set in accordance with the actual environmental requirements of the cabins. The benchmark value is the average value of each sensor node in daily data collection. In this application, it represents a characteristic of the sensor node collecting normal data. All multimodal sensor nodes complete the communication link construction through a Mesh self-organizing network. After each multimodal sensor node collects the corresponding sensing data, an environmental interference adaptive filtering algorithm is used to remove invalid data, and noise reduction processing is performed on all the remaining valid sensing data. All the valid sensing data currently collected by the intelligent sensing network subsystem are compared one by one with the benchmark values ​​in the regional baseline database in real time, and the first deviation value between all valid sensing data and the corresponding benchmark value is calculated. When the first deviation value exceeds the initial sensing threshold without meeting the fire judgment standard, the corresponding valid sensing data is determined to be failed sensing data, and the multimodal sensor node corresponding to the failed sensing data is recorded as an abnormal node. When the first deviation value reaches the fire judgment standard, the ship compartment where the corresponding valid data multimodal sensor node is deployed is identified and designated as the suspected fire area.

[0006] In a preferred embodiment, the intelligent sensing network subsystem sends a coordination signal to all multimodal sensor nodes in the same compartment of the abnormal node and / or suspected fire area through a Mesh self-organizing network. After receiving the coordination signal, all multimodal sensor nodes in the same compartment increase the data acquisition frequency and construct a temporary local sensing subnet. Multimodal sensor nodes within the temporary local sensing subnet employ data cross-validation; If at least one multimodal sensor node detects failure sensing data, and the failure sensing data is determined to be valid sensing data, the intelligent sensing network subsystem resumes its normal operating mode. If any multimodal sensor node fails to detect failure sensing data, the abnormal node is determined to be a failure node. If there is a first deviation between the effective sensing data detected by at least one multimodal sensor node and the corresponding benchmark value, and the deviation reaches the fire judgment standard, the area is judged as a fire area, and the collected sensing data is verified and preliminarily judged. In a preferred embodiment, the intelligent sensing network subsystem collects the sensing data from all temporary local sensing subnets, uses a dynamic balancing algorithm to adjust the communication links and transmission priorities of the multimodal sensor nodes, and combines the verification and preliminary judgment results with the collected sensing data into a multimodal sensing data stream.

[0007] In a preferred embodiment, the central intelligent decision-making subsystem further includes a data fusion engine. The central intelligent decision-making subsystem uses the data fusion engine to classify and organize the multimodal sensing data streams, dividing them into environmental parameter data, image and video data, and vibration data according to data type. Then, it uses a multi-source data fusion algorithm to perform layered fusion processing on the data of each data type, including: By fusing similar environmental parameter data from the same ship compartment, the average values ​​of multiple sets of temperature, smoke particle concentration, and combustible gas concentration data from the same ship compartment are calculated. The coordinates of high-temperature areas in thermal imaging frames are analyzed, and corresponding environmental parameter concentration data including temperature, smoke particle concentration, and combustible gas concentration are matched. Combined with the cabin scene in the visible light video stream of the image and video data, the specific scene of the fire area is determined, and the fusion of environmental and image parameter data is completed. The fused environmental-image parameter data is supplemented and fused with vibration data. The sound field vibration data in the vibration data is used to determine the vibration anomaly caused by equipment failure. Finally, a multi-source fusion data report is generated.

[0008] In a preferred embodiment, the central intelligent decision-making subsystem connects the generated multi-source fusion data report to the digital twin module, wherein the digital twin module has a built-in digital twin baseline model of the ship's entire life cycle, including preset data such as ship design drawings, cabin layout, equipment distribution, pipeline routing, and environmental benchmark parameters of each cabin. Through the matching algorithm of the digital twin module, the coordinates of abnormal areas in the multi-source fusion data report are matched with the coordinates of the ship's cabins in the digital twin model. Based on the preset environmental baseline parameters of each compartment in the digital twin model, the fused environmental parameter values ​​in the multi-source fusion data report are compared with the baseline parameters of the corresponding compartment to calculate the second deviation value. In some specific implementations, the comparison standards for smoke particle concentration and combustible gas detection accuracy in the electronic equipment compartment are higher than those in the cargo compartment, and the comparison standards for engine room temperature and sound field vibration are higher than those in the living compartment, which are in line with the actual environmental requirements of different compartments. Combined with the image recognition results and vibration anomaly judgment results in the multi-source fusion data report, the fire situation assessment results are obtained. It should be noted that the analysis logic of the digital twin module accurately determines four core aspects: First, the fire type, clearly identifying it as either an electrical fire or a flammable material fire, based on the composition of combustible gases, the rate of temperature change, and abnormal vibration. Second, the fire level, categorized into different levels based on temperature deviation, smoke particle concentration deviation, and combustible gas concentration fusion values, with corresponding handling standards clearly defined. Third, the core location of the fire source, calculated by combining the coordinates of the high-temperature area in the thermal imaging frame with the layout of the digital twin cabin. Fourth, the fire spread trend, predicting the direction and speed of fire spread by combining preset data on cabin structure and airflow direction in the digital twin model, identifying key equipment, flammable materials, and escape routes around the abnormal area, clarifying equipment protection priorities and avoidance areas. After the analysis is completed, a detailed fire analysis report is generated, and the analysis results are simultaneously mapped to the digital twin model in real time, achieving visualization of the fire situation.

[0009] In a preferred embodiment, the central intelligent decision-making subsystem further includes a decision instruction module, which determines the fire level coefficient based on the fire level classification in the fire assessment results, and exports the corresponding ship compartment volume where the fire occurred through the digital twin module. Using ship cabin volume and fire severity coefficient as independent variables and the number of drones dispatched as dependent variables, a preliminary linear correlation model between the basic number of drones dispatched and the fire severity coefficient and cabin volume parameters is constructed through fitting calculations. A nonlinear least squares fitting algorithm is used to correct the deviation of the preliminary linear model. The correction is based on the difference between the historical dispatch number and the predicted dispatch number output by the preliminary linear model. By minimizing the sum of squared differences, the linear correlation model is obtained, and the basic number of drones dispatched is calculated through the linear correlation model. The number of reconnaissance drones and firefighting drones is allocated according to a fixed ratio based on the basic drone deployment volume. Based on the area of ​​the fire zone inside the ship's cabin obtained through image recognition and the unit fire extinguishing coverage of the fire extinguishing agent, the amount of fire extinguishing agent to be used is obtained, and a decision plan is finally generated. In some other specific ways, the central intelligent decision-making subsystem continuously receives feedback data from the drone-assisted firefighting subsystem, sorts it into environmental parameter change data, equipment status data, and operation progress data, compares it with the initial data in the detailed fire situation assessment report, and calculates the third deviation value: if the third deviation value does not exceed the preset threshold, the original decision plan remains unchanged, and the fire situation and drone operation status in the digital twin model are updated synchronously for continuous visual monitoring. If the third deviation value exceeds the preset threshold, the dynamic adjustment mechanism of the decision-making scheme is activated: the data fusion engine re-integrates real-time feedback data with the original multi-source fusion data, updates the environmental parameter fusion values, dynamic coordinates of the fire area, fire spread trend and other core data; the digital twin module corrects the fire situation visualization results based on the updated data, adjusts the equipment protection area and escape route markings; the decision command module re-substitutes the optimized linear correlation model, combines the real-time remaining power of the drone, the remaining fire extinguishing agent and operation efficiency data, corrects the basic dispatch volume and model ratio of drones, calculates the amount of supplementary fire extinguishing agent, and generates supplementary decision commands.

[0010] In a preferred embodiment, the central intelligent decision-making subsystem further includes a decision instruction module, which determines the fire level coefficient based on the fire level classification in the fire assessment results, and exports the corresponding ship compartment volume where the fire occurred through the digital twin module. Using ship cabin volume and fire severity coefficient as independent variables and the number of drones dispatched as dependent variables, a preliminary linear correlation model between the basic number of drones dispatched and the fire severity coefficient and cabin volume parameters is constructed through fitting calculations. A nonlinear least squares fitting algorithm is used to correct the deviation of the preliminary linear model. The correction is based on the difference between the historical dispatch number and the predicted dispatch number output by the preliminary linear model. By minimizing the sum of squared differences, the linear correlation model is obtained, and the basic number of drones dispatched is calculated through the linear correlation model. The number of reconnaissance drones and firefighting drones is allocated according to a fixed ratio based on the basic drone deployment volume. Based on the area of ​​the fire zone inside the ship's cabin obtained through image recognition and the unit fire extinguishing coverage of the fire extinguishing agent, the amount of fire extinguishing agent to be used is obtained, and a decision plan is finally generated. In some other specific ways, the central intelligent decision-making subsystem continuously receives feedback data from the drone-assisted firefighting subsystem, sorts it into environmental parameter change data, equipment status data, and operation progress data, compares it with the initial data in the detailed fire situation assessment report, and calculates the third deviation value: if the third deviation value does not exceed the preset threshold, the original decision plan remains unchanged, and the fire situation and drone operation status in the digital twin model are updated synchronously for continuous visual monitoring. If the third deviation value exceeds the preset threshold, the dynamic adjustment mechanism of the decision-making scheme is activated: the data fusion engine re-integrates real-time feedback data with the original multi-source fusion data, updates the environmental parameter fusion values, dynamic coordinates of the fire area, fire spread trend and other core data; the digital twin module corrects the fire situation visualization results based on the updated data, adjusts the equipment protection area and escape route markings; the decision command module re-substitutes the optimized linear correlation model, combines the real-time remaining power of the drone, the remaining fire extinguishing agent and operation efficiency data, corrects the basic dispatch volume and model ratio of drones, calculates the amount of supplementary fire extinguishing agent, and generates supplementary decision commands.

[0011] In a preferred embodiment, after the reconnaissance drone arrives at the fire area, the following settings are also included: Activate visual sensors and multimodal environmental sensors to continuously collect real-time image and video data of the fire area, capture the spread dynamics and movement trajectory of high-temperature areas through frame difference method, and mark the coordinates of uncontrolled local fire areas; Real-time environmental parameter data is acquired at a preset acquisition frequency, and the rate of change of environmental parameters is calculated after processing by a data smoothing algorithm. Mark the areas not covered by the extinguishing agent spray, and transmit all monitoring data synchronously to the cluster control module and the central intelligent decision-making subsystem; After the firefighting drone arrives at the fire area, it also has the following settings: The adaptive adjustment algorithm dynamically adjusts the flight altitude, pitch angle, and fire extinguishing agent spray angle based on the wind field inside the cabin. Based on the spray intensity standard corresponding to the fire level and the calculated amount of extinguishing agent; During the spraying process, environmental parameter changes in the work area are continuously collected and fed back to the cluster control module for work performance evaluation.

[0012] The beneficial effects of this invention are as follows: By using a data fusion engine to perform layered fusion of multi-dimensional perception data, integrating information such as environmental parameters, images and videos, and vibration signals, and combining it with a digital twin baseline model of the ship's entire life cycle, it can accurately determine the type, level, core location of the fire source, and spread trend of the fire, clarify the protection priority and avoidance areas of key equipment, and provide comprehensive and three-dimensional data support for decision-making. Moreover, the decision-making instructions calculate the number of drones dispatched and the amount of fire extinguishing agent used through an optimized linear correlation model, so as to achieve reasonable allocation of resources and avoid resource waste or insufficient supply. By employing multimodal sensor nodes to collaboratively collect data, setting differentiated initial sensing thresholds based on the environmental characteristics of different cabins, and combining an adaptive filtering algorithm for environmental interference with a cross-validation mechanism of a temporary local sensing subnet, invalid data can be effectively eliminated and abnormal sensor nodes can be accurately identified. The UAV-assisted firefighting subsystem plans the optimal flight path based on cabin structure, environmental parameters, and wind field data, avoiding critical equipment and obstacles. It achieves real-time data interaction and dynamic task allocation among UAVs through a distributed cluster collaborative communication link. The reconnaissance UAV provides real-time feedback on fire dynamics and operational effectiveness, while the firefighting UAV adaptively adjusts its flight attitude and extinguishing agent spraying parameters according to wind field changes, ensuring comprehensive firefighting coverage and precise targeting. This solves the problems of limited coverage and insufficient coordination in traditional firefighting methods, and significantly improves firefighting efficiency. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall system architecture of the present invention.

[0014] Figure 2 This is a schematic diagram of the node structure of the intelligent sensing network subsystem of the present invention.

[0015] Figure 3 This is a flowchart of the central intelligent decision-making subsystem of the present invention.

[0016] Figure 4 This is a schematic diagram of the working scenario of the UAV-assisted firefighting subsystem of the present invention.

[0017] Figure 5 This is a system closed-loop workflow diagram of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] As attached Figure 1 - Appendix Figure 5 As shown, this embodiment provides: a ship damage control and firefighting system that combines multimodal perception and unmanned aerial vehicle (UAV) collaboration, the system comprising: Intelligent sensing network subsystem; used to collect sensing data from ship cabins through multimodal sensor nodes connected by a Mesh self-organizing network, and to verify and make preliminary judgments on the collected sensing data to obtain multimodal sensing data stream; The central intelligent decision-making subsystem is used to receive multimodal sensing data streams transmitted by the intelligent sensing network subsystem, classify and sort the data streams and perform hierarchical fusion processing through the data fusion engine, generate multi-source fusion data reports, and conduct analysis. The decision command module builds models to calculate the number of drones dispatched and the amount of fire extinguishing agent used and generates decision plans. At the same time, it receives operational feedback data from the drone collaborative fire extinguishing subsystem and dynamically adjusts the decision plans. The UAV-assisted firefighting subsystem receives decision plans from the central intelligent decision-making subsystem, activates its onboard visual, acoustic, and multimodal environmental sensors, plans the optimal flight path based on ship cabin structure, environmental parameters, and wind field data, constructs a distributed cluster collaborative communication link, dynamically allocates reconnaissance and firefighting tasks, with reconnaissance UAVs performing dynamic fire monitoring and data feedback, and firefighting UAVs performing firefighting operations, simultaneously transmitting operational status and environmental change data to the central intelligent decision-making subsystem, thus completing the coordinated execution and data feedback of fire response.

[0020] After all the multimodal sensor nodes are deployed, the intelligent sensing network subsystem continuously collects all sensing data under the operating status of the ship's cabins, establishes an environmental baseline database containing reference values, sets initial sensing thresholds for different ship cabins where the multimodal sensor nodes are deployed, and establishes communication links for all multimodal sensor nodes through a Mesh self-organizing network. After each multimodal sensor node collects the corresponding sensing data, an environmental interference adaptive filtering algorithm is used to remove invalid data, and noise reduction processing is performed on all the remaining valid sensing data. All the valid sensing data currently collected by the intelligent sensing network subsystem are compared one by one with the benchmark values ​​in the regional baseline database in real time, and the first deviation value between all valid sensing data and the corresponding benchmark value is calculated. When the first deviation value exceeds the initial sensing threshold without meeting the fire judgment standard, the corresponding valid sensing data is determined to be failed sensing data, and the multimodal sensor node corresponding to the failed sensing data is recorded as an abnormal node. When the first deviation value reaches the fire judgment standard, the ship compartment where the corresponding valid data multimodal sensor node is deployed is identified and designated as the suspected fire area.

[0021] The intelligent sensing network subsystem sends a coordination signal to all multimodal sensor nodes in the same compartment of the abnormal node and / or suspected fire area through a Mesh self-organizing network. After receiving the coordination signal, all multimodal sensor nodes in the same compartment increase the data acquisition frequency and construct a temporary local sensing subnet. Multimodal sensor nodes within the temporary local sensing subnet employ data cross-validation; If at least one multimodal sensor node detects failure sensing data, and the failure sensing data is determined to be valid sensing data, the intelligent sensing network subsystem resumes its normal operating mode. If any multimodal sensor node fails to detect failure sensing data, the abnormal node is determined to be a failure node. If there is a first deviation between the effective sensing data detected by at least one multimodal sensor node and the corresponding benchmark value, and the deviation reaches the fire judgment standard, the area is judged as a fire area, and the collected sensing data is verified and preliminarily judged. The intelligent sensing network subsystem collects sensing data from all temporary local sensing subnets, uses a dynamic balancing algorithm to adjust the communication links and transmission priorities of multimodal sensor nodes, and combines the verification and preliminary judgment results with the collected sensing data into a multimodal sensing data stream.

[0022] The central intelligent decision-making subsystem also includes a data fusion engine. This engine categorizes and organizes the multimodal sensing data streams, classifying them into environmental parameter data, image and video data, and vibration data based on data type. Furthermore, it employs a multi-source data fusion algorithm to perform layered fusion processing on each data type, including: By fusing similar environmental parameter data from the same ship compartment, the average values ​​of multiple sets of temperature, smoke particle concentration, and combustible gas concentration data from the same ship compartment are calculated. The coordinates of high-temperature areas in thermal imaging frames are analyzed, and corresponding environmental parameter concentration data including temperature, smoke particle concentration, and combustible gas concentration are matched. Combined with the cabin scene in the visible light video stream of the image and video data, the specific scene of the fire area is determined, and the fusion of environmental and image parameter data is completed. The fused environmental-image parameter data is supplemented and fused with vibration data. The sound field vibration data in the vibration data is used to determine the vibration anomaly caused by equipment failure. Finally, a multi-source fusion data report is generated, which includes the coordinates of the abnormal area, the fused values ​​of various environmental parameters, the image recognition results, and the vibration anomaly determination results. The core characteristics of the abnormal area are clearly defined, providing comprehensive fused data support for subsequent fire situation assessment.

[0023] The central intelligent decision-making subsystem connects the generated multi-source fusion data report to the digital twin module. The digital twin module has a built-in digital twin baseline model of the ship's entire life cycle, which includes preset data such as ship design drawings, cabin layout, equipment distribution, pipeline routing, and environmental benchmark parameters of each cabin. Through the matching algorithm of the digital twin module, the coordinates of abnormal areas in the multi-source fusion data report are matched with the coordinates of the ship's cabins in the digital twin model. Based on the preset environmental baseline parameters of each compartment in the digital twin model, the fused environmental parameter values ​​in the multi-source fusion data report are compared with the baseline parameters of the corresponding compartment to calculate the second deviation value. In some specific implementations, the comparison standards for smoke particle concentration and combustible gas detection accuracy in the electronic equipment compartment are higher than those in the cargo compartment, and the comparison standards for engine room temperature and sound field vibration are higher than those in the living compartment, which are in line with the actual environmental requirements of different compartments. Combined with the image recognition results and vibration anomaly judgment results in the multi-source fusion data report, the fire situation assessment results are obtained.

[0024] It should be noted that the analysis logic of the digital twin module accurately determines four core aspects: First, the fire type, clearly identifying it as either an electrical fire or a flammable material fire, based on the composition of combustible gases, the rate of temperature change, and abnormal vibration. Second, the fire level, categorized into different levels based on temperature deviation, smoke particle concentration deviation, and combustible gas concentration fusion values, with corresponding handling standards clearly defined. Third, the core location of the fire source, calculated by combining the coordinates of the high-temperature area in the thermal imaging frame with the layout of the digital twin cabin. Fourth, the fire spread trend, predicting the direction and speed of fire spread by combining preset data on cabin structure and airflow direction in the digital twin model, identifying key equipment, flammable materials, and escape routes around the abnormal area, clarifying equipment protection priorities and avoidance areas. After the analysis is completed, a detailed fire analysis report is generated, and the analysis results are simultaneously mapped to the digital twin model in real time, achieving visualization of the fire situation.

[0025] The central intelligent decision-making subsystem also includes a decision instruction module, which determines the fire level coefficient based on the fire level classification in the fire assessment results, and exports the corresponding ship compartment volume of the fire through the digital twin module. Using ship cabin volume and fire severity coefficient as independent variables and the number of drones dispatched as dependent variables, a preliminary linear correlation model between the basic number of drones dispatched and the fire severity coefficient and cabin volume parameters is constructed through fitting calculations. A nonlinear least squares fitting algorithm is used to correct the deviation of the preliminary linear model. The correction is based on the difference between the historical dispatch number and the predicted dispatch number output by the preliminary linear model. By minimizing the sum of squared differences, the linear correlation model is obtained, and the basic number of drones dispatched is calculated through the linear correlation model. The number of reconnaissance drones and firefighting drones is allocated according to a fixed ratio based on the basic drone deployment volume. Based on the area of ​​the fire zone inside the ship's cabin obtained through image recognition and the unit fire extinguishing coverage of the fire extinguishing agent, the amount of fire extinguishing agent to be used is obtained, and a decision plan is finally generated. In some other specific ways, the central intelligent decision-making subsystem continuously receives feedback data from the drone-assisted firefighting subsystem, sorts it into environmental parameter change data, equipment status data, and operation progress data, compares it with the initial data in the detailed fire situation assessment report, and calculates the third deviation value: if the third deviation value does not exceed the preset threshold, the original decision plan remains unchanged, and the fire situation and drone operation status in the digital twin model are updated synchronously for continuous visual monitoring. If the third deviation value exceeds the preset threshold, the dynamic adjustment mechanism of the decision-making scheme is activated: the data fusion engine re-integrates real-time feedback data with the original multi-source fusion data, updates the environmental parameter fusion values, dynamic coordinates of the fire area, fire spread trend and other core data; the digital twin module corrects the fire situation visualization results based on the updated data, adjusts the equipment protection area and escape route markings; the decision command module re-substitutes the optimized linear correlation model, combines the real-time remaining power of the drone, the remaining fire extinguishing agent and operation efficiency data, corrects the basic dispatch volume and model ratio of drones, calculates the amount of supplementary fire extinguishing agent, and generates supplementary decision commands.

[0026] After receiving the decision plan, the UAV collaborative firefighting subsystem obtains the static data of the ship's cabin structure and the environmental parameters of the cabin based on the fire situation assessment results and data fusion results of the central intelligent decision-making subsystem, and simultaneously activates the visual and acoustic sensors carried by all UAVs. The current position of the UAV is determined by extracting structural feature points of the ship's cabins from visual images captured by the visual sensor using image recognition algorithms. Furthermore, the acoustic sensor collects sound field data and calculates the propagation time and reflection distance of the sound waves in the sound field data; The calculated and collected data are synchronously transmitted to the UAV cluster control module in the UAV collaborative firefighting subsystem. The airflow data within the cabin is processed using variational mode decomposition technology to generate an internal wind field. Combined with the internal three-dimensional wind field data, the UAV's current position, and the ship's cabin structure static data, an improved... The path planning algorithm independently plans the optimal flight path for each UAV. It should be noted that the path planning must avoid the critical equipment protection areas, structural obstacles, and escape routes marked in the digital twin model. At the same time, it must conform to the predicted fire spread trend to ensure that the path points to the core area of ​​the fire source or the predetermined operation location. The real-time attitude angle and flight speed data of the UAV are incorporated into the path planning process. The path curvature, flight altitude and turning radius are corrected by dynamic obstacle avoidance algorithm to avoid collisions between UAVs and scratches with the cabin structure. Furthermore, the aforementioned UAV-assisted firefighting subsystem, based on the UAV number allocation results in the decision-making scheme, relies on Mesh self-organizing network and short-range communication technology between UAVs to construct a distributed cluster collaborative communication link. This distributed cluster collaborative communication link enables real-time data interaction among UAVs within the cluster, including location information, remaining battery power, remaining fire extinguishing agent, and collected real-time data. A distributed model predictive control algorithm is employed to dynamically allocate operational tasks to each UAV. Reconnaissance UAVs prioritize reaching the edge of the fire area to perform monitoring tasks, while firefighting UAVs are prioritized according to the core location of the fire source and proceed sequentially to designated work points to perform firefighting tasks.

[0027] After the reconnaissance drone arrives at the fire area, it also has the following settings: Activate visual sensors and multimodal environmental sensors to continuously collect real-time image and video data of the fire area, capture the spread dynamics and movement trajectory of high-temperature areas through frame difference method, and mark the coordinates of uncontrolled local fire areas; Real-time environmental parameter data is acquired at a preset acquisition frequency, and after being processed by a data smoothing algorithm, the rate of change of environmental parameters is calculated to monitor the development trend of the fire. Mark the areas not covered by the extinguishing agent spray, and transmit all monitoring data synchronously to the cluster control module and the central intelligent decision-making subsystem; After the firefighting drone arrives at the fire area, it also has the following settings: The adaptive adjustment algorithm dynamically adjusts the flight altitude, pitch angle, and fire extinguishing agent spray angle based on the wind field inside the cabin. Based on the spray intensity standard corresponding to the fire level and the calculated amount of extinguishing agent; During the spraying process, environmental parameter changes in the work area are continuously collected and fed back to the cluster control module for work performance evaluation.

Claims

1. A ship damage control and firefighting system that combines multimodal perception and unmanned aerial vehicle (UAV) collaboration, characterized in that, The system includes: Intelligent sensing network subsystem; used to collect sensing data from ship cabins through multimodal sensor nodes connected by a Mesh self-organizing network, and to verify and make preliminary judgments on the collected sensing data to obtain multimodal sensing data stream; The central intelligent decision-making subsystem is used to receive multimodal sensing data streams transmitted by the intelligent sensing network subsystem, classify and sort the data streams and perform hierarchical fusion processing through the data fusion engine, generate multi-source fusion data reports, and conduct analysis. The decision command module builds models to calculate the number of drones dispatched and the amount of fire extinguishing agent used and generates decision plans. At the same time, it receives operational feedback data from the drone collaborative fire extinguishing subsystem and dynamically adjusts the decision plans. The UAV-assisted firefighting subsystem receives decision plans from the central intelligent decision-making subsystem, activates its onboard visual, acoustic, and multimodal environmental sensors, plans the optimal flight path based on ship cabin structure, environmental parameters, and wind field data, constructs a distributed cluster collaborative communication link, dynamically allocates reconnaissance and firefighting tasks, with reconnaissance UAVs performing dynamic fire monitoring and data feedback, and firefighting UAVs performing firefighting operations, simultaneously transmitting operational status and environmental change data to the central intelligent decision-making subsystem, thus completing the coordinated execution and data feedback of fire response.

2. The ship damage control and firefighting system based on multimodal perception and UAV collaboration according to claim 1, characterized in that, After all the multimodal sensor nodes are deployed, the intelligent sensing network subsystem continuously collects all sensing data under the operating status of the ship's cabins, establishes an environmental baseline database containing reference values, sets initial sensing thresholds for different ship cabins where the multimodal sensor nodes are deployed, and establishes communication links for all multimodal sensor nodes through a Mesh self-organizing network. After each multimodal sensor node collects the corresponding sensing data, an environmental interference adaptive filtering algorithm is used to remove invalid data, and noise reduction processing is performed on all the remaining valid sensing data. All the valid sensing data currently collected by the intelligent sensing network subsystem are compared one by one with the benchmark values ​​in the regional baseline database in real time, and the first deviation value between all valid sensing data and the corresponding benchmark value is calculated. When the first deviation value exceeds the initial sensing threshold without meeting the fire judgment standard, the corresponding valid sensing data is determined to be failed sensing data, and the multimodal sensor node corresponding to the failed sensing data is recorded as an abnormal node. When the first deviation value reaches the fire judgment standard, the ship compartment where the corresponding valid data multimodal sensor node is deployed is identified and designated as the suspected fire area.

3. The ship damage control and firefighting system based on multimodal perception and UAV collaboration according to claim 2, characterized in that, The intelligent sensing network subsystem sends a coordination signal to all multimodal sensor nodes in the same compartment of the abnormal node and / or suspected fire area through a Mesh self-organizing network. After receiving the coordination signal, all multimodal sensor nodes in the same compartment increase the data acquisition frequency and construct a temporary local sensing subnet. Multimodal sensor nodes within the temporary local sensing subnet employ data cross-validation; If at least one multimodal sensor node detects failure sensing data, and the failure sensing data is determined to be valid sensing data, the intelligent sensing network subsystem resumes its normal operating mode. If any multimodal sensor node fails to detect failure sensing data, the abnormal node is determined to be a failure node. If there is a first deviation between the effective sensing data detected by at least one multimodal sensor node and the corresponding benchmark value, and this deviation reaches the fire determination standard, the area is determined to be a fire zone, thus completing the verification and preliminary judgment of the collected sensing data.

4. A ship damage control and firefighting system based on multimodal perception and UAV collaboration as described in claim 3, characterized in that, The intelligent sensing network subsystem collects sensing data from all temporary local sensing subnets, uses a dynamic balancing algorithm to adjust the communication links and transmission priorities of multimodal sensor nodes, and combines the verification and preliminary judgment results with the collected sensing data into a multimodal sensing data stream.

5. A ship damage control and firefighting system based on multimodal perception and UAV collaboration according to claim 1, characterized in that, The central intelligent decision-making subsystem also includes a data fusion engine. This engine categorizes and organizes the multimodal sensing data streams, classifying them into environmental parameter data, image and video data, and vibration data based on data type. Furthermore, it employs a multi-source data fusion algorithm to perform layered fusion processing on each data type, including: By fusing similar environmental parameter data from the same ship compartment, the average values ​​of multiple sets of temperature, smoke particle concentration, and combustible gas concentration data from the same ship compartment are calculated. The coordinates of high-temperature areas in thermal imaging frames are analyzed, and corresponding environmental parameter concentration data including temperature, smoke particle concentration, and combustible gas concentration are matched. Combined with the cabin scene in the visible light video stream of the image and video data, the specific scene of the fire area is determined, and the fusion of environmental and image parameter data is completed. The fused environmental-image parameter data is supplemented and fused with vibration data. The sound field vibration data in the vibration data is used to determine the vibration anomaly caused by equipment failure. Finally, a multi-source fusion data report is generated.

6. A ship damage control and firefighting system based on multimodal perception and UAV collaboration according to claim 5, characterized in that, The central intelligent decision-making subsystem will connect the generated multi-source fusion data report to the digital twin module. Through the matching algorithm of the digital twin module, the coordinates of abnormal areas in the multi-source fusion data report will be matched with the coordinates of ship compartments in the digital twin model. Based on the preset environmental baseline parameters of each compartment in the digital twin model, the fused environmental parameter values ​​in the multi-source fusion data report are compared with the baseline parameters of the corresponding compartment to calculate the second deviation value. Combined with the image recognition results and vibration anomaly judgment results in the multi-source fusion data report, the fire situation assessment result is obtained.

7. A ship damage control and firefighting system based on multimodal perception and UAV collaboration according to claim 1, characterized in that, The central intelligent decision-making subsystem also includes a decision instruction module, which determines the fire level coefficient based on the fire level classification in the fire assessment results, and exports the corresponding ship compartment volume of the fire through the digital twin module. Using ship cabin volume and fire severity coefficient as independent variables and the number of drones dispatched as dependent variables, a preliminary linear correlation model between the basic number of drones dispatched and the fire severity coefficient and cabin volume parameters is constructed through fitting calculations. A nonlinear least squares fitting algorithm is used to correct the deviation of the preliminary linear model. The correction is based on the difference between the historical dispatch number and the predicted dispatch number output by the preliminary linear model. By minimizing the sum of squared differences, the linear correlation model is obtained, and the basic number of drones dispatched is calculated through the linear correlation model. The number of reconnaissance drones and firefighting drones is allocated according to a fixed ratio based on the basic drone deployment volume. Based on the area of ​​the fire zone inside the ship's cabin obtained through image recognition and the unit fire extinguishing coverage of the fire extinguishing agent, the amount of fire extinguishing agent to be used is obtained, and finally a decision plan is generated.

8. A ship damage control and firefighting system based on multimodal perception and UAV collaboration according to claim 1, characterized in that, After receiving the decision plan, the UAV collaborative firefighting subsystem obtains the static data of the ship's cabin structure and the environmental parameters of the cabin based on the fire situation assessment results and data fusion results of the central intelligent decision-making subsystem, and simultaneously activates the visual and acoustic sensors carried by all UAVs. The current position of the UAV is determined by extracting structural feature points of the ship's cabins from visual images captured by the visual sensor using image recognition algorithms. Furthermore, acoustic sensors collect sound field data and calculate the propagation time and reflection distance of sound waves in the sound field data.

9. A ship damage control and firefighting system based on multimodal perception and UAV collaboration according to claim 1, characterized in that, The calculated and collected data are synchronously transmitted to the UAV cluster control module in the UAV collaborative firefighting subsystem. The airflow data within the cabin is processed using variational mode decomposition technology to generate an internal wind field. Combined with the internal three-dimensional wind field data, the UAV's current position, and the ship's cabin structure static data, an improved... The path planning algorithm independently plans the optimal flight path for each drone; Furthermore, the UAV collaborative firefighting subsystem, based on the UAV number allocation result in the decision scheme, relies on Mesh self-organizing network and short-range communication technology between UAVs to construct a distributed cluster collaborative communication link. The constructed distributed cluster collaborative communication link realizes real-time data interaction between UAVs within the cluster. The interaction content includes location information, remaining power, remaining fire extinguishing agent, and collected real-time data. A distributed model predictive control algorithm is used to dynamically allocate the operation tasks of each UAV.

10. A ship damage control and firefighting system based on multimodal perception and UAV collaboration according to claim 8, characterized in that, After the reconnaissance drone arrives at the fire area, it also has the following settings: Activate visual sensors and multimodal environmental sensors to continuously collect real-time image and video data of the fire area, capture the spread dynamics and movement trajectory of high-temperature areas through frame difference method, and mark the coordinates of uncontrolled local fire areas; Real-time environmental parameter data is acquired at a preset acquisition frequency, and the rate of change of environmental parameters is calculated after processing by a data smoothing algorithm. Mark the areas not covered by the extinguishing agent spray, and transmit all monitoring data synchronously to the cluster control module and the central intelligent decision-making subsystem; After the firefighting drone arrives at the fire area, it also has the following settings: The adaptive adjustment algorithm dynamically adjusts the flight altitude, pitch angle, and fire extinguishing agent spray angle based on the wind field inside the cabin. Based on the spray intensity standard corresponding to the fire level and the calculated amount of extinguishing agent; During the spraying process, environmental parameter changes in the work area are continuously collected and fed back to the cluster control module for work performance evaluation.

Images