Multi-modal space-time fusion and distributed cluster fire extinguishing robot cooperative control system and method

Abstract

The invention discloses a multi-modal space-time fusion and distributed cluster fire extinguishing robot cooperative control system, aims to solve the problems of low efficiency and poor precision of multi-machine cooperative fire extinguishing in wide-area fire investigation and flammable and explosive scenes, and provides a whole set of unmanned fire extinguishing technical solution for the multi-mode space-time fusion and distributed cluster fire extinguishing robot cooperative control system. The system comprises four core hardware architectures, namely a cloud dispatching center (A), an unmanned aerial vehicle cluster (B), an unmanned resource guarantee vehicle (C) and a crawler-type fire extinguishing robot cluster (D), and has the following core functions: 1, the cloud dispatching center (A) updates a fire scene model through distributed dispatching by fusing multi-modal sensor data, dynamically allocates tasks and plans a safe approach path; 2, the unmanned aerial vehicle cluster (B) executes the instruction, completes fire global positioning and cluster cooperative sensing, generates a high-precision fire topographic map and feeds back a signal; thirdly, the unmanned resource support vehicle (C) autonomously says for help, and has the functions of automatic mechanical arm water feeding and power supply, environment detection, on-site cleaning after fire extinguishing and communication relay; and fourthly, the crawler-type fire extinguishing robot cluster (D) autonomously navigates and positions the root of a fire source, detects and predicts the drop point of a water column, carries out fire extinguishing feedback evaluation, and interacts with the dispatching center cooperatively to dynamically adjust the strategy, so that closed-loop control is formed, and the fire extinguishing efficiency and accuracy are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and emergency equipment, specifically involving: a perception system for tracked firefighting robots based on dynamic fusion of multimodal sensors, focusing on solving the problem of wide-area rapid search by unmanned aerial vehicles and the problem of multi-robot collaborative firefighting in dense smoke, high temperature, and flammable and explosive environments; a hierarchical collaborative decision-making mechanism for tasks with multiple ignition points, integrating distributed bee colony algorithms to achieve dynamic scheduling of multiple vehicles; and a precise firefighting execution control system, encompassing the automatic water and power supply functions of the robotic arm, water spray trajectory prediction and energy consumption optimization, firefighting feedback mechanism, a complete unmanned system solution, and a complete technology chain for multi-robot collaborative firefighting. Background Technology

[0002] The rapid development of autonomous driving and artificial intelligence technologies has made interdisciplinary scientific research an essential path for industrial digital transformation, especially for high-end enterprises. The integration of autonomous driving and AI technologies with firefighting equipment to create a new generation of firefighting robots is a current trend, reshaping the underlying logic of the firefighting industry. Its importance lies not only in technological iteration but also in pioneering a new paradigm of "digital twin firefighting." Traditional firefighting technologies suffer from several drawbacks: First, over-reliance on manual patrols, monitoring, and response is significantly delayed. Command and firefighting deployment rely on personal experience, lacking systematic and scientific decision-making. Equipment is often simplistic and prone to independent operation, creating isolated operational zones and leading to significant resource waste. Furthermore, for fires involving high-risk flammable, explosive, or chemical products, weak human-machine coordination and resistance can easily trigger secondary accidents, endangering the lives of firefighters. Therefore, this patent addresses these industry pain points and conducts in-depth research on existing technologies. Specific technological comparisons and the patent's technological value are as follows:

[0003] Traditional fire prevention relies on intelligent monitoring and regular manual inspections, which are costly, have limited detection range, and slow response times. After a fire is detected, factors such as dense smoke and high temperatures can interfere with the rapid and accurate location of the fire source, delaying the optimal time for firefighting. Early unmanned firefighting technologies had limitations in sensor application, relying on single sensors (such as monocular or binocular cameras) for fire source detection, which are easily interfered with in complex environments, leading to misjudgments. Robot patrols have a small field of view and low fire detection efficiency. This patent proposes a distributed unmanned aerial vehicle (UAV) global search system equipped with a long-range infrared camera and GPS positioning system, which can quickly obtain the global location of the fire source. Based on the fire situation, an appropriate number of firefighting robots can be dispatched to the scene for rescue and firefighting.

[0004] Traditional firefighting methods have significant shortcomings in dealing with complex fire scenarios. In emergency firefighting situations involving valuable equipment, traditional methods rely on manual judgment and operation. Because valuable equipment has stringent requirements for firefighting conditions, it is difficult for personnel to accurately select the appropriate firefighting method in a short time, easily leading to equipment damage and huge economic losses. In high-risk areas of flammable and explosive materials, such as chemical warehouses, traditional firefighting requires firefighters to risk entering, their lives seriously threatened by explosions and toxic gas leaks. Furthermore, traditional equipment is ineffective in handling complex and dangerous environments. In harsh natural fire environments, such as forest fires, traditional firefighting relies on firefighters carrying equipment on foot, resulting in high physical exertion. Facing large fires and complex terrain, firefighting efficiency is low, and it is difficult to control the spread of the fire.

[0005] While traditional firefighting technologies have made some progress, they also have many shortcomings. Equipment often only offers simple remote control and water spraying functions, lacking the ability to perceive and make decisions in complex fire environments. Single-unit operations are inadequate in the face of large-scale fires, and collaborative combat capabilities are poor, failing to form an effective united front against fires. This patent discloses a multi-sensor spatiotemporal fusion decision-making and distributed collaborative control method based on swarm firefighting robots, providing a comprehensive and highly innovative solution to the aforementioned problems of traditional and early unmanned firefighting technologies. This patent innovatively proposes a distributed swarm algorithm, which can quickly balance tasks based on the fire scale and the number of robots, enabling multiple firefighting robots to form an organic whole, achieving highly efficient collaborative combat, demonstrating significant advantages in complex large-scale emergency rescue scenarios. This significantly improves overall firefighting efficiency, allowing fires to be controlled in a shorter time and reducing fire damage.

[0006] This patent integrates multiple sensors, including LiDAR, cameras, RTK, and IMU. Multi-sensor fusion leverages the advantages of different sensors to compensate for the shortcomings of a single sensor. The drone performs global positioning, while the robot quickly and accurately identifies the local location of the fire source using a Camera-Lidar dynamic fusion depth estimation method, significantly improving the efficiency and accuracy of fire source detection. Furthermore, while early unmanned firefighting technologies achieved a degree of automation, they still had shortcomings in fully automated water filling, requiring manual assistance for some robots. This patent designs a mobile robotic arm specifically to assist the robot in completing the automated water filling function. It proposes a target locking and automatic verification algorithm based on Camera-Lidar fusion, making a fully unmanned firefighting process a reality. This innovation greatly improves the safety and efficiency of firefighting operations.

[0007] In traditional firefighting methods, water spray trajectories are often judged and operated by firefighters based on experience, making it difficult to adjust and automatically switch between different firefighting modes in real time according to the actual situation at the fire scene. This can lead to water waste and fail to ensure optimal firefighting results. Early unmanned firefighting technologies were also relatively simple in terms of water spray trajectory control, lacking the ability to accurately analyze and dynamically adjust the fire source. This patent combines fire source location and 3D physical modeling to propose a 3D water spray trajectory prediction method with dual-mode automatic switching. This method can adjust the robot's posture and water spray trajectory in real time according to the actual situation at the fire scene, achieving the dual goals of precise firefighting and energy-saving control. During the firefighting process, the robot can automatically select the appropriate water spray mode and trajectory based on the size, location, and combustion state of the fire source, avoiding water waste while ensuring firefighting effectiveness. This precise and energy-saving firefighting method has significant advantages in scenarios with limited resources or high environmental impact requirements, and better meets the needs of modern fire prevention and control.

[0008] In summary, traditional firefighting methods have numerous limitations when facing complex fire scenarios, and early unmanned firefighting technologies have also failed to fully solve these problems. This patent proposes a series of innovative technologies, including unmanned aerial vehicle (UAV) fire reconnaissance and positioning, distributed dynamic scheduling, L4-level autonomous driving of robots to reach the vicinity of the fire and automatically supply water, multi-sensor fusion for local fire source localization, 3D water spray trajectory prediction and nozzle attitude adjustment, firefighting effect feedback evaluation, and orderly withdrawal. These technologies effectively overcome the shortcomings of traditional unmanned firefighting technologies, significantly improving the accuracy, safety, efficiency, and energy saving of firefighting. This method is well-chosen, closely integrated with current technological development trends and actual firefighting needs, possessing extremely high practical value and broad development prospects. With continuous technological advancements and the expansion of application scenarios, this patented technology is expected to play a vital role in the field of fire prevention and control, making a positive contribution to protecting people's lives and property and the ecological environment. Summary of the Invention

[0009] The core objective of this invention is to construct a complete, unmanned firefighting solution that uses robots and artificial intelligence to replace firefighters in dangerous fire scenes involving flammable and explosive materials and harsh environments, ultimately ensuring personnel safety and improving firefighting efficiency and success rate. It provides a multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method to address the bottlenecks of existing unmanned firefighting systems.

[0010] This addresses several key issues: reducing safety risks for firefighters by enabling fully automated, collaborative firefighting in extremely dangerous environments such as flammable and explosive environments (e.g., chemical tank areas), high-temperature smoke, and collapse zones. It also addresses insufficient fire scene perception capabilities, resolving the inability of a single sensor (e.g., a regular camera) to effectively detect and accurately locate fire sources in environments with reduced visible light, such as dense smoke or darkness. Furthermore, it addresses firefighting efficiency and coordination, resolving the inability of multiple firefighting units to intelligently allocate tasks and coordinate effectively in scenarios with multiple fire points, as well as the overall inefficiency caused by single-point failures. Finally, it addresses the issues of firefighting accuracy and resource waste, specifically the low hit rate of water spray due to factors such as water pressure, distance, and wind direction, avoiding significant waste of firefighting resources (water, foam). Finally, it addresses the issue of fully automated operation throughout the entire firefighting chain, from receiving the alarm, deployment, resupply, firefighting, to evacuation, eliminating the "breakpoints" that still require human intervention and achieving truly autonomous firefighting.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method, comprising a hardware architecture consisting of four main parts, namely A, B, C, and D:

[0012] Cloud-based Dispatch Center A: Computing and Processing Layer (servers, data storage systems), Networking and Communication Layer (firewalls, routers, switches, 5G private network), Visualization and Monitoring Layer (monitors, monitoring workstations)

[0013] Drone Swarm B: The swarm consists of multiple intelligent drones, each containing a unified environmental perception sensor (long-range infrared camera), 5G communication module, and GPS positioning module.

[0014] Unmanned Resource Support Vehicle C: The vehicle's cargo compartment stores a large tonnage of water, a high-power multi-interface fast-charging power supply, a pressure pump, an intelligent water supply robotic arm, and includes sensing equipment (low-light camera, blind spot radar), positioning equipment (RTK), computing equipment (single-domain control), communication networking (signal relay), environmental monitoring sensors, etc.

[0015] Tracked firefighting robot cluster D: Composed of multiple tracked firefighting robots, each containing standardized sensing devices (low-light cameras, lidar, RTK, IMU), computing devices (industrial control computers or high-performance domain controllers), and integrated environmental sensing sensors. The functions to be implemented and the new methods of use among the four major hardware modules can then be introduced and explained module by module.

[0016] The cloud-based dispatch center A has three main functions: distributed cluster drone dispatch (A1), distributed tracked firefighting robot cluster dispatch (A2), and fire extinguishing channel generation (A3). Data collected by the drone cluster can immediately generate a fire situation map and automatically plan safe entry paths for the firefighting robots.

[0017] The selected drone swarm B has two main functions: global fire location and fire source positioning (B1) and signal feedback and command transmission (B2). Following instructions from the dispatch center, each drone flies to a gridded map and begins a "bow-shaped" carpet search. If a fire is detected, the drone sends its location information and high-resolution images to the dispatch center. The dispatch center then mobilizes more nearby drones to rescan the accurate fire terrain map and dispatches a specific number of firefighting robots to extinguish the fire based on its size.

[0018] The preferred unmanned resource support vehicle C mainly includes six functions: receiving cloud tasks and autonomous assistance (driving unmanned from the garage to the vicinity of the fire point) function C1; automatic water supply function of the robotic arm C2; automatic power supply function of the robotic arm C3; environmental detection and monitoring function C4; cleaning and recycling function C5 (after completing the automatic water supply and power supply tasks, after the fire is extinguished, the water bag is removed, and all resource support accessories are transported back to the vehicle compartment, awaiting the return command); and communication relay function C6 (to prevent weak or interrupted signals).

[0019] The selected firefighting robot cluster D has five main functions: autonomous driving (D1), enabling it to drive to the vicinity of a designated fire point (Level 4 autonomous driving); precise local location of the fire source (D2); image detection of water jet impact point and 3D modeling prediction of water jet impact point (D3); fire extinguishing feedback and effect evaluation mechanism (D4); and signal coordination and interaction with the dispatch center (D5).

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] 1. Fully Unmanned Operation: From fire reconnaissance and resource allocation to post-fire logistics replenishment, the integrated "vehicle-arm" design enables water and power supply. The automatic power supply function utilizes a combination of visual detection and strong magnetic self-attraction to achieve precise insertion. The water supply function combines visual-Lidar pose estimation with physical verification of current signals to form a dual-confirmation mechanism for automatic water supply, achieving truly unmanned intervention and thoroughly ensuring the safety of firefighters.

[0022] 2. Multimodal Fusion Perception: By integrating multiple sensor sources such as visible light, infrared, and lidar in spatiotemporal mode, the accuracy and robustness of fire source location are greatly improved in harsh environments such as dense smoke and darkness. "Perception-Decision-Control" Closed Loop: The 5G network connects front-end perception (drones), intelligent decision-making (dispatch center), and end-point execution (firefighting robots, water supply trucks), forming a highly efficient automated closed loop that significantly improves the speed of cluster emergency response.

[0023] 3. Distributed cloud-based scheduling: The combination of "grid-based partitioning" and "bow-shaped path" achieves full coverage of the search range and optimal flight efficiency. The dynamic response mechanism of "single-point alarm - multi-point collaboration" and the closed-loop process of "reconnaissance - assessment - scheduling" enable the information acquired by UAV cluster A to directly provide decision-making basis for the accurate and efficient scheduling of firefighting robot cluster B, forming a complete air-ground collaborative combat chain.

[0024] 4. Precise and efficient fire suppression: Through 3D water jet trajectory prediction and visual detection closed-loop feedback control, the fire suppression effect of "precise spraying and optimal water consumption" is achieved, which improves efficiency and saves resources.

[0025] 5. High system integration: A complete integrated technology chain of "air-ground-cloud" has been constructed, with each module being highly efficient, intelligent, and collaborative, achieving an operational effectiveness of "1+1>2". A comprehensive solution applicable to high-risk and complex environments has been formed.

[0026] In summary, this patent constructs a complete multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system, encompassing four major hardware modules: a drone swarm, a tracked firefighting robot swarm, an intelligent water-filled robotic arm, and a cloud-based dispatch center. These modules complement each other, forming a highly efficient overall firefighting solution. Through close cooperation and information exchange between the modules, the entire process—from fire source search and location, firefighting robot deployment, precise firefighting, to effect evaluation—is automated and collaborative, significantly improving firefighting efficiency and accuracy, and laying a solid foundation for large-scale unmanned firefighting applications. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 Functional structure diagram of the collaborative control system for multimodal spatiotemporal fusion and distributed cluster firefighting robots;

[0029] Figure 2 Distributed UAV swarm fire detection process and information interaction diagram with dispatch center

[0030] Figure 3 Distributed adaptive map grid and UAV reconnaissance trajectory planning

[0031] Figure 4 Numerical simulation rendering of unmanned collaborative firefighting using multiple devices (ground-air-cloud)

[0032] Figure 5 Schematic diagram of the multi-functional coverage and fire extinguishing process of the unmanned resource support vehicle

[0033] Figure 6 Flowchart of Intelligent Decision-Making and Dynamic Feedback Mechanism for Tracked Firefighting Robot

[0034] Figure 7 Multi-robot "flat-projectile" collaborative fire extinguishing 3D water column prediction effect diagram

[0035] Figure 8 Multi-robot "oblique upward throw" collaborative fire extinguishing 3D water column prediction effect diagram Detailed Implementation

[0036] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0037] Please see Figure 1 and Figure 4 A multimodal spatiotemporal fusion and distributed swarm firefighting robot collaborative control system and method is proposed, comprising four core components: unmanned aerial vehicle (UAV) swarm (A), firefighting robot swarm (B), unmanned resource support vehicle (C), and cloud-based dispatch center (D). These four intelligent devices achieve two-way real-time data interaction between air, ground, and cloud, forming a closed-loop system of perception, decision-making, and control, laying the foundation for batch unmanned collaborative operation of heterogeneous swarm devices.

[0038] Please see Figure 2 Drone Swarm A: Each drone contains a unified environmental perception sensor (long-range infrared camera), a 5G communication module, and a GPS positioning module. Drone Swarm A has two main functions: global fire location and fire source positioning (A1), and signal feedback and command transmission (A2). In this invention, Drone Swarm A is used to achieve rapid reconnaissance, accurate positioning, and initial fire situation generation for large-scale fires (forests, chemical plants, large warehouses, logistics sites). The specific implementation steps are as follows:

[0039] Cloud-based dispatch center A serves as the command center for the entire system, responsible for global information processing, intelligent decision-making, and scheduling all unmanned units for collaborative operations. Its core function is to achieve cross-platform and cross-cluster collaborative command. Dispatch center A primarily has three functions: distributed cluster drone dispatching (A1), fire-fighting path generation (A2), automatically planning safe entry paths for fire-fighting robots, and distributed tracked fire-fighting robot cluster dispatching (A3). The specific implementation steps are as follows:

[0040] A1S1: Please refer to Figure 3Task initialization and grid partitioning: Global search map gridding refers to the scheduling algorithm of scheduling center D being based on the number of UAVs (let's say k) and their effective reconnaissance range. Given i = 1, 2, ..., k, divide the region to be searched into k sub-regions Ω. i ..., i = 1, 2... Assuming all drones have the same battery power consumption, the area of ​​each sub-region decreases progressively with each subsequent region being assigned to a single drone. Assume the total flight distance of the drones is L km, s is the scan area of ​​a single drone flight in a sub-region, w is the width of the uniformly spaced grid in the global search map, and h... i These are the lengths of different sub-regions, and the formula for calculating the sub-region is shown in (1).

[0041]

[0042] A1S2: Each drone searches sub-region Ω i The flight trajectory is planned within the range of i=1,2..., generating an efficient "bow-shaped" coverage path, ensuring that the total length of the path is within the range of a single UAV (30km), achieving a comprehensive search without omissions.

[0043] A1S3: Distributed UAV swarm scheduling (corresponding function D1)

[0044] Task allocation and path planning: Initial grid division: Upon receiving a fire alarm, the reconnaissance area is dynamically divided into multiple grids based on the forest area and the number of drones, assigning an initial search area to each drone. "Bow-shaped" path distribution: An optimal "bow-shaped" coverage search path is generated for each grid and distributed to each drone.

[0045] A1S4: Dynamic Cooperative Scheduling, Hotspot Response. When a drone detects a suspected fire, the dispatch center no longer relies solely on the information transmitted back by that drone, but immediately initiates a dynamic rescheduling algorithm (such as one based on distributed constraint optimization).

[0046] A1S5: Cluster Focus: Commands multiple drones near the fire point to change their original paths and converge on the suspected fire point, performing cross-scanning and confirmation from different angles to quickly build a high-precision local fire scene model and eliminate false alarms.

[0047] A2S3: Fire Sentiment Awareness and Situation Building (Function A2 Pre-input)

[0048] Multi-source data aggregation: The dispatch center receives data transmitted back from drone cluster A in real time through a dedicated 5G network, including image data: visible light and infrared video streams and infrared thermal imaging images.

[0049] Pose data: GPS / RTK high-precision position, attitude, and velocity of each UAV.

[0050] Fire metadata: The drone AI module autonomously identifies the coordinates, temperature, and size of the fire.

[0051] A2S4: Real-time generation of fire situation maps (Function initially implemented in A2)

[0052] Data fusion: By integrating reconnaissance data from multiple drones, blind spots from a single perspective are eliminated, and all fire point information is unified onto a single global digital map through spatiotemporal registration technology.

[0053] Situational awareness: Utilizing computer vision and deep learning models, a fire situation map is automatically generated and updated in real time. This map not only marks the location of fire points but also includes key information such as fire boundaries, fire spread direction, intensity level (represented by different colors), and high-temperature core areas.

[0054] A2S5: Fire Extinguishing Lane Generation (Core manifestation of Function A2)

[0055] Global path planning: The scheduling center plans a globally optimal path for each robot from its current location to its area of ​​responsibility. This path is based on a high-precision map and prioritizes wide and flat passages.

[0056] Safety constraint embedding: Path planning algorithms (such as the A* algorithm or its variants) will avoid extreme danger zones marked on the fire situation map (such as areas of intense burning or areas that may collapse) and known obstacles, thereby generating a safe "fire extinguishing path".

[0057] Command issuance: The planned "fire extinguishing routes" (i.e., the sequence of path points) are issued to the corresponding fire extinguishing robot D.

[0058] A3S1: Distributed Firefighting Robot Cluster Scheduling (corresponding function A3) Firefighting Task Package Generation: Based on the generated fire situation map, the scheduling center divides the fire scene into multiple firefighting responsibility zones, and each responsibility zone is defined as a firefighting task package. The task package contains: Target coordinates: The core GPS coordinates of the responsibility zone.

[0059] Fire information: estimated fire intensity and required amount of extinguishing agent (water or foam).

[0060] Task Priority: Assign different priorities to each task package based on the direction and speed of fire spread.

[0061] A3S2 robot allocation and global optimal path planning:

[0062] Optimal allocation: Using Hungarian and bee colony algorithms, fire extinguishing task packages are allocated to the most suitable fire extinguishing robot (considering factors such as the robot's current location, remaining water / electricity, and travel time to the fire scene) to achieve optimal global efficiency.

[0063] A3S3: Full-process monitoring and adaptive adjustment; the dispatch center's large screen displays the real-time location, status (power, water volume), and task execution progress of all unmanned units. It also receives fire extinguishing effectiveness evaluation feedback from the fire extinguishing robots.

[0064] A3S4: Dynamic Rescheduling: If a robot malfunctions or the fire suddenly intensifies, the dispatch center will immediately recalculate and dynamically assign the robot's task to other robots, or instruct water supply truck C to provide support. The fire situation map is updated in real-time based on the latest data continuously transmitted back by the drone swarm. If the fire spreads beyond expectations, new task packages and firefighting routes are immediately regenerated, and resources are allocated accordingly.

[0065] A3S5: Mission terminated and system recalled. The dispatch center, based on the combined reconnaissance of the drones and feedback from the firefighting robot, confirmed that all open flames at the fire site had been extinguished and there was no risk of reignition.

[0066] A3S6: Issues an evacuation command and a mission termination command to all unmanned units. Each robot returns to the resupply point along the safe path planned by the dispatch center, the drones return to base and land, and the system enters standby mode.

[0067] The aforementioned drone swarm B has two main functions: a global fire location and search function (B1) and a signal feedback and command transmission function (B2). Following instructions from the dispatch center, each drone flies to a gridded map. If a fire is detected, the drone will send its location information and high-resolution images back to the dispatch center.

[0068] B1S1: Autonomous Takeoff and Cruise. The drone swarm takes off autonomously in sequence according to instructions and navigates to its assigned search sub-area using the onboard GPS positioning module.

[0069] B1S2: Each drone performs a carpet scan of its respective sub-area in a "bow-shaped" pattern. Once the drone reaches the designated airspace, it activates its long-range infrared camera. The infrared camera continuously scans the ground, utilizing its thermal sensitivity to penetrate thin fog and detect high-temperature heat sources.

[0070] B1S3: If the drone detects a fire, it will promptly transmit the data back in real time. The drone continuously transmits its flight status, real-time location, and infrared video stream data back to the cloud dispatch center A via the 5G communication module.

[0071] B1S4: Fire Confirmation and Dynamic Collaborative Reconnaissance. When the onboard processing unit of a UAV identifies a suspected fire point exceeding the temperature threshold by analyzing infrared image data, it immediately sends an alarm signal to the dispatch center A. This signal contains the precise GPS coordinates and infrared snapshot of the suspected fire point.

[0072] in

[0073] f x ,f y It is focal length (in pixels), c x ,c y Principal point coordinates (image center offset).

[0074] R: 3x3 rotation matrix (rotation from world coordinate system to camera coordinate system).

[0075] t: 3x1 translation vector (the position of the world coordinate system origin in the camera coordinate system), Zc: depth scaling factor (the value of the z coordinate in the camera coordinate system).

[0076] B1S5: The camera is responsible for identifying the fire source, and GPS achieves global positioning. Formula (2) realizes the fire source location in the camera coordinate system. After switching to the global coordinate system, we can obtain

[0077]

[0078] B1S6: After receiving the alarm, dispatch center A immediately activates the dynamic dispatch algorithm: the drone that detected the fire hovers over the suspected fire point for continuous monitoring.

[0079] B1S7: Dispatch Center A dispatches additional drones from the surrounding area, rapidly converging on the suspected fire zone. Multi-angle collaborative scanning: the assembled drone swarm conducts cross-scanning and focused reconnaissance of the suspected fire zone from different angles, eliminating false alarms from a single sensor through multi-view data fusion.

[0080] B1S8: Fire situation map generation: The dispatch center A integrates infrared images and positioning data transmitted from multiple drones to quickly construct a preliminary fire situation map. This map clearly marks the fire boundary, fire point distribution, and fire intensity (estimated through temperature distribution).

[0081] The detailed steps of the signal feedback and command transmission function B2 are as follows:

[0082] B2S1: Task handover and fire-fighting resource scheduling. The dispatch center A packages the generated fire situation map, precise coordinates of the fire site, fire scale assessment results and other information into a complete fire-fighting task package.

[0083] Based on the scale of the fire, the dispatch center A calculates the required number of firefighting robots and issues the firefighting task package to the corresponding number of tracked firefighting robot clusters B via the 5G network, instructing them to go to the firefighting site.

[0084] B2S3: Continuous Monitoring and Guidance: Some units in the drone swarm B continue to hover over the fire site, providing real-time information on changes in the fire and visual guidance to the approaching firefighting robot D until the firefighting task is handed over.

[0085] In this invention, the unmanned resource support vehicle C is a mobile supply platform integrating autonomous driving, warehousing, and precision operations. Its core function is to provide unmanned, uninterrupted water hose connection and resupply for the forward firefighting robot cluster. It mainly includes six functions: receiving cloud tasks and autonomous deployment (driving unmanned from the garage to the vicinity of the fire point) C1; automatic water supply to the robotic arm C2; automatic power supply to the robotic arm C3; environmental monitoring and detection C4; cleaning and recycling C5 (after completing the automatic water supply and power supply tasks, once the firefighting is over, the water hose is removed, and all resource support components are transported back to the vehicle's compartment, awaiting a return command); and communication relay C6.

[0086] For the preferred unmanned resource support vehicle C, please refer to [link / reference]. Figure 5 It mainly includes six functions: receiving cloud tasks and autonomous assistance (driving unmanned from the garage to the vicinity of the fire) function C1; automatic water supply function of the robotic arm C2; automatic power supply function of the robotic arm C3; environmental detection and monitoring function C4; cleaning and recycling function C5 (after completing the automatic water supply and power supply tasks, wait for the fire to end, remove the water bag, transport all resource support parts back to the vehicle compartment, and wait for the return command); and communication relay function C6 (to prevent weak or interrupted signals).

[0087] The specific steps are as follows:

[0088] C1S1: Task Reception and Autonomous Water Supply Function. The water supply truck is parked at the robot's water supply point, and its onboard control unit continuously listens for instructions from the cloud-based dispatch center A. When the firefighting robot cluster D needs water replenishment, dispatch center A generates a water replenishment task package and sends it to the unmanned water supply truck via the 5G network.

[0089] C1S2: Path planning and driving. After receiving the precise GPS coordinates of the target fire-fighting robot, the water supply truck's autonomous driving system (with L4 level capability) plans the optimal safe path to the target based on high-precision maps and real-time road conditions.

[0090] C1S3: Autonomous Approach. The water supply truck relies on its own lidar, camera, and RTK / IMU integrated navigation system to achieve autonomous driving and obstacle avoidance, and finally safely and accurately park near the target firefighting robot (usually maintaining a safe operating distance of 3-5 meters).

[0091] The automatic water supply function C2 of the unmanned resource support vehicle C is implemented in the following steps:

[0092] C2S1: After receiving a specific water supply task, the resource support vehicle performs a power-on self-check to confirm that the hydraulic system, robotic arm, and all sensors are functioning normally. Subsequently, it controls the automatic opening of the control compartment door (or hatch), releasing the movable intelligent robotic arm to prepare for installing the water bag onto the interface of the fire-fighting robot.

[0093] C2S2: The robotic arm is removed and mounted on a wheeled device, ensuring the robot moves from inside the compartment to the optimal working position outside the vehicle. This process allows for automatic water replenishment of the firefighting robot via a lightweight, retractable / mobile robotic arm support platform.

[0094] C3S1: Employing a vision-guided and precision docking solution, it first performs coarse positioning and target search. Then, the vision system (low-light camera + blind-spot lidar) at the end of the robotic arm begins to work. The camera scans within a wide field of view and uses a target detection algorithm to initially identify the water inlet (concentric circular pipe opening) on ​​the firefighting robot.

[0095] C3S2: Precise localization and pose estimation, switching to near-field perception, fusing LiDAR and camera data. LiDAR point cloud data provides the precise 3D coordinates and surface normals of the interface, while visual data provides texture and feature confirmation. The precise pose of the interface relative to the robotic arm's end effector is calculated using calibrated extrinsic parameters M.

[0096] C3S3: The robotic arm control algorithm plans a collision-free, smooth motion trajectory based on the target pose, driving the quick connector carried at the end of the robotic arm to approach the interface of the fire extinguishing robot.

[0097] C3S4: Physical docking and multimodal verification. In the final stage of the approach (centimeter level), image-based visual servoing (IBVS) is used for fine-tuning to ensure alignment of the docking centerline.

[0098] C3S5: Force / Position Hybrid Control: The robotic arm is equipped with a force / torque sensor, which switches to force control mode at the moment of contact to insert the connector in a compliant manner, preventing damage to the interface due to positional errors.

[0099] C3S6: Current Signal Verification (Key Innovation): After docking, the robotic arm applies a safe 48V voltage to the contact surface. Real-time monitoring of the robotic arm's current changes verifies the success and seal of the pipe connection. If the connection is good, the current will stabilize within the expected load range; the robotic arm will then issue a water supply command to the water truck, and the system will immediately supply water. If the current falls below the safe range, the robotic arm will be instructed to retract and retry.

[0100] C3S7: After the verification is passed, maintain the water pump pressure and start to supply water to the fire extinguishing robot stably until its water tank is full or a stop command is received.

[0101] The automatic power-on function C3 of the unmanned resource support vehicle C is implemented in the following steps:

[0102] C3S1: A low-light camera scans the surface of the firefighting robot to locate the charging port and identify visual markers;

[0103] C3S2: The robotic arm presses the mechanical trigger area of ​​the charging port to open the charging cover;

[0104] C3S3: The robotic arm carries the power plug and moves it to the vicinity of the charging port, using strong magnetic material to attract the plug and complete the initial docking.

[0105] C3S4: Force / torque sensor monitors insertion resistance value; if abnormal, it reverts to S3.

[0106] C3S5: Sequentially perform electrical parameter verification, communication protocol verification, and physical status confirmation;

[0107] C3S6: If all checks pass, lock the plug and start charging; otherwise, revert to S3.

[0108] The unmanned resource support vehicle C and the environmental monitoring function C4 are implemented in the following steps:

[0109] C4S1: Sensor Deployment and Initialization. Deploy various types of environmental sensors on the unmanned resource support vehicle C, including but not limited to: temperature sensors (for real-time monitoring of ambient temperature), toxic gas sensors (for detecting the concentration of specific toxic gases such as carbon monoxide and hydrogen sulfide), and explosive gas sensors (for detecting the concentration level of flammable and explosive gases such as methane and hydrogen).

[0110] C4S2: Initialize each sensor to ensure it is in normal working condition and calibrate it to the preset accuracy range.

[0111] C4S3: Data Acquisition and Real-time Transmission. The sensor sampling frequency is set, such as collecting data once per second or per minute, depending on monitoring requirements and the rate of environmental change. Wireless communication technologies (such as 5G, Wi-Fi, etc.) are used to transmit the collected environmental data to a remote monitoring center or local control terminal in real time.

[0112] C4S4: Data Analysis and Processing. At the remote monitoring center or local control terminal, preprocessing of received environmental data includes data cleaning and outlier removal. Statistical analysis methods (such as moving averages and exponential smoothing) are used to smooth the environmental data to reduce the impact of random errors.

[0113] C4S5: Based on preset safety thresholds (such as upper temperature limit, toxic gas concentration threshold, explosive gas concentration threshold, etc.), it compares and analyzes the processed data.

[0114] C4S6: Risk Assessment and Decision Support. Based on data analysis results, assess the safety status of the on-site environment, including but not limited to: whether the temperature exceeds the safe range, potentially causing a fire or equipment damage; whether the concentration of toxic gases reaches a dangerous level, posing a threat to personnel health; and whether the concentration of explosive gases is close to the explosion limit, posing an explosion risk. Based on the risk assessment results, generate corresponding decision support information, such as: whether a secondary support team needs to be dispatched to strengthen on-site monitoring or handle emergencies; and whether an emergency evacuation order needs to be issued to ensure the safe evacuation of personnel from the danger zone.

[0115] The unmanned resource protection vehicle C, with its cleaning and recycling function C5, is implemented using the following steps:

[0116] C5S1: Mission Cancellation and Return to Base (corresponding function C5)

[0117] C5S2: Disconnection and Retrieval. Upon receiving a "water replenishment complete" signal from dispatch center A or firefighting robot D, the water pump stops working. The robotic arm performs the reverse operation: depressurizes, unlocks, disconnects the hose connector, and then precisely places the connector back into the storage rack on the water supply truck.

[0118] C5S3: The robotic arm or a dedicated reel mechanism on the vehicle automatically winds up the hose neatly, preventing tangling.

[0119] C5S4: Robotic Arm Return to Loading Position: The robotic arm returns to its initial position inside the loading compartment, and the loading compartment door closes. The water supply truck reports task completion to dispatch center A and receives the next instructions (such as refilling water for the next robot or autonomously returning to base to stand by).

[0120] The unmanned resource support vehicle C and the communication relay function C6 are implemented in the following steps:

[0121] C6S1: Communication relay mode initialization and parameter configuration. When the unmanned resource support vehicle C detects insufficient on-site communication signal coverage (such as RSSI value below the threshold, packet loss rate exceeding the preset value, etc.), or receives a relay request instruction from the remote command center or terminal equipment, the communication relay function C6 is automatically activated.

[0122] C6S2: Parameter configuration: Configure the relay frequency band (such as public network frequency band, private network frequency band, or custom frequency band) to ensure compatibility with the frequency bands of field equipment and command center; set parameters such as relay bandwidth, transmission power, and encryption method (such as AES-256) to balance transmission efficiency and security; dynamically allocate relay channels to avoid frequency band conflicts with other equipment (such as environmental monitoring sensors and other relay nodes).

[0123] C6S3: Signal reception and multi-source fusion processing. Through the vehicle-mounted multi-protocol communication module (supporting 4G / 5G, LoRa, Wi-Fi, Zigbee, satellite communication, etc.), it receives signals from different terminal devices, including communication requests from other unmanned devices (such as drones and robots).

[0124] C6S4: Signal Fusion: Performs timing alignment, deduplication, and error correction on received multi-source signals to generate relay data packets in a unified format, ensuring data integrity and real-time performance.

[0125] C6S5: Relay Forwarding Strategy (P1-P4) Formulation and Implementation

[0126] P1: Forwarding Path Planning: Dynamically select the optimal forwarding path based on the site topology (e.g., obstacle distribution, device location) and signal quality (e.g., RSSI, SNR).

[0127] P2: Multi-hop relay: Relay transmission via other unmanned resource support vehicles or fixed relay nodes (e.g., when signal obstruction is severe or the distance is too far).

[0128] P3: Priority scheduling: Relay data packets are sorted by priority (e.g., emergency evacuation instructions > environmental monitoring data > routine communication), and high-priority data is forwarded first to ensure the real-time nature of critical information.

[0129] P4: Dynamic bandwidth allocation: Dynamically adjusts the bandwidth of each channel based on the relay task load (such as data volume and number of devices) to avoid congestion.

[0130] C6S6: Real-time signal quality monitoring (Q1-Q5), continuously monitors the signal strength, packet loss rate, latency and other indicators of the relay link, and generates a communication quality assessment report.

[0131] Q1: Adaptive adjustment strategy: Dynamically optimize relay parameters based on evaluation results.

[0132] Q2: Frequency band switching: If the current frequency band is severely interfered with, it will automatically switch to the backup frequency band;

[0133] Q3: Power Adjustment: Adjust the transmission power according to distance and obstacles to balance coverage and energy consumption;

[0134] Q4: Path Reconfiguration: If a relay node fails, replan the forwarding path (e.g., switch to other unmanned vehicles or satellite links).

[0135] Q5: Linkage with environmental monitoring function C4: If environmental data (such as the concentration of explosive gases) triggers an emergency evacuation command, priority will be given to ensuring the relay bandwidth and transmission reliability of the command (such as suspending the forwarding of low-priority data).

[0136] Please refer to the described tracked firefighting robot cluster D. Figure 6 It has five main functions: autonomous walking (D1), precise local location of the fire source (D2), image detection of water jet impact point and 3D modeling prediction of water jet impact point (D3), fire extinguishing feedback and effect evaluation mechanism (D4), and signal coordination and interaction with the dispatch center (D5).

[0137] The autonomous walking function D1 of the tracked firefighting robot is implemented in the following steps:

[0138] D1S1: Task Reception and Path Planning. The firefighting robot receives a firefighting task package from the cloud-based dispatch center D, which contains the core GPS coordinates of the fire site. Based on the built-in high-precision map and real-time positioning information (integrating RTK centimeter-level positioning and IMU inertial navigation), the onboard computing equipment plans a safe and optimal path to the fire site.

[0139] D2S2: The firefighting robot activates its Level 4 autonomous driving function and travels along the planned path. During the journey, it uses onboard LiDAR and low-light cameras to perceive the surrounding environment in real time, dynamically avoids obstacles (such as trees and rocks), and maintains continuous communication with the dispatch center A, reporting its own position and status until it safely arrives at the designated assembly point near the fire scene.

[0140] The specific implementation steps of the fire-fighting robot's precise local fire source positioning function D2 are as follows:

[0141] D2S1: The camera and lidar are installed on the fire extinguishing nozzle. The low-light camera identifies open flames and smoke in visible light mode and initially locates the visual position of the fire source.

[0142] D2S2: Local fire source positioning and adjustment of the nozzle to face the fire source, switching to the fire extinguishing robot coordinate system, Camera-Lidar fusion, converting 2D pixel coordinates to 3D world coordinates through inverse perspective transformation, which is a process involving camera imaging principles and geometric calculations, as shown in formula (1). Then, switch the Lidar to the vehicle coordinate system:

[0143]

[0144] D2S3: Adjust the tracked robot's front end to face the fire source, issue initial motion commands, adjust the robot's front end to face the fire source, and take the center of the robot's rear track axle as the origin. If the fire source's coordinates in the 3D physical world are (x, y, z), the robot's motion θ heading The rule is that the left side is positive and the right side is negative.

[0145] D2S4: If The robot rotates to the right by β units of angles. The size of this angle is mainly limited by the camera's field of view.

[0146] D2S5: If The robot rotates to the left by β units of angle, so that it faces the fire source directly.

[0147] The function D4, which involves detecting the water column's landing point using images and predicting its landing point using 3D modeling, is implemented through the following steps:

[0148] D4S1: The cross-sectional diameter of the water column is approximately 15cm, and air resistance cannot be ignored. If F Air Represented as air resistance, C d Here, A is the drag coefficient (obtained through experimental testing), and A is the area of ​​the water column. Air resistance formula:

[0149]

[0150] The revised equation of motion requires numerical integration to solve. Air resistance causes the water flow velocity to decrease, and the velocity change formula is shown in formula (5):

[0151]

[0152] According to the principles of kinematics, the velocity of the water column is expressed as: A projectile motion formula can be established for a water column; please refer to [link / reference]. Figure 7 By decomposing the water column into horizontal and vertical directions, we can obtain a system of differential equations, the specific expression of which is shown in formula (6):

[0153]

[0154] For projectile motion at an angle, please refer to Figure 8 Before the water column reaches its highest point, the second equation of formula (6) becomes

[0155]

[0156] The fourth-order Runge-Kutta iteration was used to solve the differential equations, with a time step Δt = 0.01, until the water column hit the ground. The first-order differential equation system was then solved. The RK4 iteration formula is:

[0157]

[0158] Here, the state variable is y = (x, v) x ,y,v y ) T In formula (8), k is a constant. The second-order equations of motion need to be transformed into a system of first-order equations:

[0159]

[0160] With x and v x Taking this as an example, each component can be represented as, v * It is an intermediate variable:

[0161] k 1,1 =v x,n k 1,2 =-kv n v x,n (10)

[0162] k 2,1 =v x,n +k 1,2 Δt / 2 k 2,2 =-kv * (v x,n +k 1,2 Δt / 2) (11)

[0163] The remaining components k i,3 ,k i,4 Corresponding to (y,v) y A similar calculation process is used to update the formulas for the four k vectors mentioned above, resulting in the component iteration formula:

[0164]

[0165] If the water pressure P of the unmanned resource support vehicle is constant, the nozzle height h and angle θ of the tracked fire extinguishing robot can be adjusted. The adjustment of the nozzle attitude will affect the initial velocity of the water column. The formula for estimating the initial velocity is as follows.

[0166] For an incompressible ideal fluid (water), the sum of pressure energy, kinetic energy, and potential energy is constant along a streamline, according to Bernoulli's equation. After simplification, the initial velocity of the water column can be obtained:

[0167]

[0168] Considering the influence of air resistance, and combining formula (5), by adjusting the nozzle height and angle, we can obtain the initial velocity components:

[0169]

[0170] By following the steps above, starting from the initial state, given the initial time t0, and adjusting the nozzle height h of the fire extinguishing robot and the horizontal angle θ, the water jet range can generally be extended by setting θ ∈ [30°, 45°]. Initial state y0 = (x0, v x,0 ,y0,y y,0 ) T Given a time step Δt and a drag coefficient k, arbitrary time step Δt is obtained through iterative steps. n =tn-1 The state y at time +Δt n =(x n ,v x,n ,y n ,v y,n ) T .

[0171] The specific implementation steps of the fire extinguishing feedback and effect evaluation mechanism D5 are as follows:

[0172] D5S1: Fire Source Detection and Localization. Utilizing target detection models such as YOLO and Faster R-CNN, it continuously identifies and locates fire sources in visible light and infrared images, outputting their pixel coordinates. f ,v f ] and detection box fire .

[0173] D5S2: Fire Intensity Estimation: The size of the fire source detection frame area detected by the camera on the robot nozzle, the highest temperature, average temperature, and area of ​​the high-temperature zone reported by the drone reconnaissance are used as quantitative evaluation indicators.

[0174] D5S3: Water Column Impact Point Detection: Using image segmentation models or traditional image processing algorithms (such as motion and texture-based detection), identify the water splash and steam regions generated when a water column impacts a target in a visible light image, and calculate its centroid pixel coordinates [u water ,v water ].

[0175] D5S4: Effect Evaluation and Decision-Making. This patent will provide specific iteration stopping conditions as follows:

[0176] D5S5: Condition (1): Water column-fire source overlap test, calculate Euclidean distance, calculate centroid of water column landing point [u] in image coordinate system. water ,v water ] and the fire source center [u f ,v f pixel distance

[0177] Set an overlap threshold: Define a threshold T overlap (Overlap of the fire source and water column detection frames). Judgment: If T IOU >T overlap If the target is hit precisely, the system will maintain its current thrust posture. If T... IOU <T overlap If the condition is determined to be "needs fine-tuning of the nozzle", then based on the 3D water column prediction model, the nozzle height and angle are readjusted until T... IOU >T overlap .

[0178] D5S6: Condition (2): Fire weakening trend test, fire source detection box size: continuously record fire source detection box fire Area A fire (t) (in pixels). Calculate A within a time period T (e.g., the last 10 seconds). fire The trend of (t) change. If its area continues to decrease significantly (calculated once per second, the slope of the linear fit is negative, and the current area is less than 50% of the initial area), then it is determined that the fire is being effectively suppressed.

[0179] D5S:7: Multimodal verification of fire extinguishing effect. During fire extinguishing, the UAV infrared data should show that the core temperature of the fire site continues to decrease and the area of ​​the high-temperature zone shrinks, which is consistent with the visual judgment conclusion.

[0180] D5S8: Comprehensive Decision Making and Iterative Stopping Criteria.

[0181] Continue firefighting: If neither condition (1) nor condition (2) is met, generate adjustment instructions (such as fine-tuning the angle and height of the robotic arm, or increasing the water pump pressure) and continue spraying.

[0182] Transfer / Stop: If conditions (1) and (2) are met simultaneously and last for a period of time (e.g., 5 seconds), then the current fire point is determined to be extinguished.

[0183] D5S9: Immediately sends a "fire extinguished" signal to the cloud-based dispatch center D. Stop the current spraying operation.

[0184] Waiting for new instructions from the dispatch center (e.g., proceed to the next fire point or return to the starting point).

[0185] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system, characterized in that, The hardware architecture comprises four main parts: a cloud-based dispatch center (A), a drone swarm (B), an unmanned resource support vehicle (C), and a tracked firefighting robot swarm (D); its core functionality includes four main steps: Step 1: Cloud-based Dispatch Center (A): First, the distributed dispatch of drones begins wide-area search (A1) and real-time fire monitoring (A2), dynamically allocating tasks between drones and firefighting robots to optimize resource utilization. Based on the fire situation map fed back by the drones, a safe entry path is planned for the robots, and fire extinguishing channels are generated (A3). Multimodal perception data is integrated to update the fire scene model in real time, ensuring the timeliness and accuracy of dispatch decisions. Step Two: Drone Swarm B is primarily responsible for executing cloud-based commands, real-time fire reconnaissance and global positioning (B1), gridded airspace mapping, and a "H"-shaped carpet search. It detects heat sources using long-range infrared cameras and generates precise coordinates of the fire point using GPS positioning. Upon detecting a fire, it immediately uploads location information and high-resolution images to the dispatch center. Signal feedback and swarm collaborative sensing functions (B2) are implemented, and the dispatch center mobilizes nearby drones for secondary scanning, fusing multi-angle data to generate a high-precision fire topographic map, transmitting the reconnaissance data packet back to the dispatch center. The dispatch center calculates the required number of firefighting robots based on the fire's scale and plans the global path in the cloud. Step 3: Unmanned resource support vehicle (C), receiving cloud tasks, autonomously rushing to the fire (driving unmanned from the garage to the vicinity of the fire) function (C1). Automatic water supply function of robotic arm (C2). Automatic power supply function of robotic arm (C3), environmental monitoring function (C4), cleaning and recycling function (C5) (after completing the automatic water supply and power supply tasks, wait for the fire to end, remove the water bag, transport all resource support accessories back to the vehicle compartment, and wait for the return command) Communication relay function (C6) (to prevent weak or interrupted signals). Step 4: The tracked firefighting robot swarm (D) receives instructions from the cloud and autonomously navigates to the vicinity of the fire point (D1), and accurately locates the base of the fire source (D2). It also performs image detection of the water jet's impact point and 3D modeling to predict the water jet's impact point (D3). A firefighting feedback and effectiveness evaluation mechanism (D4) is established, along with signal coordination and interaction with the dispatch center (D5). The firefighting strategy is dynamically adjusted to form a closed loop of "perception-decision-control".

2. The multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method according to claim 1, characterized in that, Step 1: Distributed scheduling of UAVs begins wide-area search (A1) and real-time fire monitoring (A2). The key feature is the proposed dynamic grid adaptive partitioning method. Based on the number of UAVs, their flight range, and the real-time fire situation, the area of ​​the search sub-region is dynamically adjusted (due to the fixed flight range, the area of ​​subsequent regions gradually decreases), achieving coordinated optimization of full coverage and energy consumption balance. A distributed constraint optimization algorithm is used. When a UAV detects a suspected fire, the dispatch center immediately activates a dynamic rescheduling mechanism based on distributed constraint optimization. This instructs multiple UAVs around the fire point to break through the initial grid boundary, cross-scan from multiple angles, and construct a local high-precision fire scene model. The algorithm ensures that there are no conflicts in the cluster paths.

3. The multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method according to claim 1, characterized in that, Step 1: The dispatch center (A) plans a safe entry path for the robot and generates a fire extinguishing channel (A3). Its key features include: intelligent task package generation and dynamic priority allocation: Fire extinguishing responsibility zones are automatically divided based on the fire situation map. Each task package includes target coordinates, fire intensity, extinguishing agent requirements, and a priority label based on the fire spread rate, supporting real-time dynamic adjustment. A Hungarian-swarm hybrid optimization algorithm combines the Hungarian algorithm (for static task-robot matching) with the swarm algorithm (for dynamic cluster collaboration), using the robot's current position, remaining water / electricity, and travel time as constraints to achieve globally optimal task allocation. Multi-source heterogeneous data fusion integrates UAV visible / infrared video streams, infrared thermal imaging images, GPS pose data, and AI fire metadata. Spatiotemporal registration technology unifies all fire point information onto a global digital map, eliminating blind spots from a single perspective. Dynamic hazard avoidance path planning employs an improved A* algorithm (or a variant thereof), embedding extreme danger zones (such as intensely burning zones and collapse risk zones) and static obstacle constraints from the fire situation map in real time to generate globally optimal "fire extinguishing channels" with adjustable safety margins. Anomaly handling and adaptive scheduling based on fire evolution: When a robot malfunctions or the fire situation changes abruptly, the scheduling center immediately triggers a dynamic rescheduling mechanism to recalculate task allocation and fire extinguishing channels, and ensures seamless resource integration through coordinated support from water supply trucks.

4. The multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method according to claim 1, characterized in that, Step two hardware devices include: The drone swarm (A) consists of multiple intelligent drones with autonomous flight capabilities. Each drone integrates a uniformly configured environmental perception sensor (infrared camera), a 5G communication module, and a GPS positioning module. The environmental perception sensor is a long-range infrared camera used to detect heat source signals in real time in complex environments or under smoky conditions. The communication module is a 5G communication module, supporting low-latency, high-bandwidth data transmission to enable real-time interaction between the drones and the dispatch center and other swarm members. The positioning module is a GPS positioning module, providing high-precision three-dimensional spatial location information to support accurate navigation of the drones on a gridded map.

5. The multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method according to claim 1, characterized in that, Step Two: The UAV swarm (A) global search and fire source global positioning function (A1) is characterized by: according to the instructions of the dispatch center, the UAV swarm (A) adopts a "bow-shaped" carpet search strategy to perform collaborative search tasks within a preset grid area; when any UAV detects a suspected fire point through an infrared camera, the fire source positioning process is immediately triggered, and the precise geographical coordinates of the fire point are generated by combining GPS positioning data; the positioning information and real-time high-definition images are uploaded to the dispatch center through the 5G communication module to support the initial assessment of the fire.

6. The multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method according to claim 1, characterized in that, Step Two: The signal feedback and command transmission (A2) function of the UAV swarm (A) is characterized by the following: The UAV swarm (A) acts as an aerial relay node, constructing a dynamic communication network to ensure unimpeded command transmission between the dispatch center and the ground firefighting robots. After detecting a fire, the initial detection UAV continuously feeds back dynamic fire data to the dispatch center, while simultaneously receiving commands from the dispatch center to mobilize nearby UAVs for reinforcement. Upon arrival of the reinforcement UAVs, they collaborate with the initial detection UAV to perform a secondary scan, generating a high-precision fire topographic map through multi-angle data fusion, and calculating the required number and deployment locations of firefighting robots based on the fire scale. The UAV swarm A continuously monitors the firefighting site, transmitting firefighting progress data in real time via the 5G network, providing decision support to the dispatch center, and forming a closed-loop control system of "perception-decision-execution-feedback".

7. The multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method according to claim 1, characterized in that, Step 3: Upon receiving instructions from the cloud, the robot autonomously rushes to the fire site and releases its intelligent water and electricity supply robotic arm, which is the automatic water supply function (C2) of the fire-fighting robot. Its key feature is that, based on multimodal fusion positioning technology, the robotic arm's end effector integrates a low-light camera and a blind-spot lidar. Through a two-level perception strategy of "coarse positioning + fine positioning," it achieves rapid identification of the fire-fighting robot's water supply interface (both ends are metal). Coarse positioning: The low-light camera scans within a large field of view, and a target detection algorithm identifies the visual markers of the interface (concentric circular pipe openings). Precise Positioning: The system switches to a fusion perception system combining LiDAR and a high-resolution camera. LiDAR point clouds provide the interface's 3D coordinates and surface normals, while visual data provides texture features. Extrinsic parameter calibration calculates the precise pose of the interface relative to the robotic arm's end effector (error ≤ 1cm). Force / Displacement Hybrid Control and Fine-Tuning Mechanism: The robotic arm is equipped with force / torque sensors. Upon contact with the fire extinguishing robot interface, it switches to force control mode for compliant insertion, preventing interface damage due to pose errors. During the approach phase (centimeter-level), image-based visual servoing (IBVS) is used for fine-tuning to ensure alignment of the docking centerline, achieving collision-free, high-precision physical docking. After docking, current signal verification and closed-loop feedback are implemented. The robotic arm applies a safe 48V voltage to the contact surface, and real-time monitoring of current changes verifies the docking seal. If the current stabilizes within the expected load range, the connection is considered successful, and the robotic arm sends a water supply command to the water truck. If the current falls below the safe value, the robotic arm automatically retracts and retryes, forming a "perception-control-verification" closed-loop system.

8. The multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method according to claim 1, characterized in that, Step 3: Upon receiving instructions from the cloud, the robotic arm autonomously rushes to the fire site and releases its intelligent power-supplying robotic arm. Based on visual guidance and magnetic adaptive automatic power-on function (C3), its key feature is: The visual positioning module is equipped with a low-light camera at the end of the robotic arm, used to scan and identify the location of the charging port and visual markers of the fire extinguishing robot; the charging cover opening module, linked with the visual positioning module, drives the robotic arm to press the mechanical trigger area around the charging port to open the charging cover; the magnetic docking module includes a power plug and a strong magnetic material inside the charging interface of the fire extinguishing robot (the strong magnetic material is a neodymium iron boron magnet with a surface magnetic field strength ≥300mT, ensuring that the plug is automatically attracted within a distance of 5cm), which guides the plug to automatically suck into the interface through magnetic attraction, and the insertion resistance value is monitored by a force / torque sensor; the resistance fluctuation exceeds ±20% or continues to exceed the preset safety range (0.5-2N) for more than 2 seconds. Based on abnormal resistance values, the robotic arm is controlled to retract and re-connect to the dynamic adjustment module; the multi-level verification module includes an electrical parameter verification unit that mainly monitors whether the voltage, current, and insulation resistance are within safe ranges; a communication protocol verification unit that verifies the encrypted signature and data frame integrity of the charging control command; and a physical status confirmation unit that recognizes the fit between the plug and the interface through image recognition and controls the electromagnetic latch to lock the plug; the control unit triggers the formal charging or retraction process based on the results of the multi-level verification.

9. The multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method according to claim 1, characterized in that, Step four: After the tracked firefighting robot receives instructions from the cloud and autonomously rushes to the fire site and completes pre-treatment, the key 3D water column physical prediction model (D3) is characterized by: D3S1 constructs a set of differential equations for the motion of the water column, integrating Bernoulli's equation, air resistance formula, and projectile / oblique projectile kinematic model, and defines the input parameters as nozzle height h, horizontal angle θ, water pressure P, and air resistance coefficient k; D3S2 uses the fourth-order Runge-Kutta method to numerically iterate and solve the differential equation system, outputting the three-dimensional spatial coordinates (x(t), y(t), z(t)) of the water column at any time t; D3S3 obtains the centroid coordinates (x actual, y actual) of the actual landing point of the water column through image detection, and calculates the deviation Δd between the actual centroid coordinates (x predicted, y predicted) and the model predicted coordinates. D3S4: If Δd exceeds the preset threshold (e.g., 5cm), dynamically adjust the water pressure P and nozzle attitude parameters (h, θ), and re-execute D2S2-D2S3 until Δd meets the accuracy requirements.

10. The multimodal spatiotemporal fusion and distributed cluster firefighting robot collaborative control system and method according to claim 1, characterized in that, Step four, the dual feedback closed-loop control function D5 of the tracked firefighting robot, is characterized by: D5S1: Visual Feedback: The coordinates of the fire source center (x fire source, y fire source) and the centroid coordinates of the water column landing point (x water column, y water column) are detected by the YOLO / Faster R-CNN model, and the Euclidean distance d between the two in the image coordinate system is calculated. D5S2: Thermal Imaging Feedback: Combining UAV infrared data, extracting the core temperature Tcore and the area of ​​the high-temperature region Shigh-temperature, and assessing the trend of fire weakening. D5S3: Define the iteration stopping condition Condition 1 (Precise Hit): If d ≤ d threshold (e.g., 10 pixels) and the duration is t (e.g., 5 seconds), then the water column is determined to precisely cover the fire source; Condition 2 (Fire Suppression): If High temperature, initial and T core ≤T safe If the temperature reaches 100℃, it is determined that the fire has been effectively suppressed. D5S4: If both conditions 1 and 2 are met, send a "fire extinguished" signal to the dispatch center and stop spraying; otherwise, adjust the nozzle attitude or water pump pressure according to the 3D water column prediction model.

Images