Multi-modal fire-fighting robot cluster cooperation method and system, medium and product

Abstract

The invention provides a multi-modal fire-fighting robot cluster cooperation method and system, a medium and a product, and relates to the technical field of fire-fighting robots, and the method comprises the steps: obtaining own multi-modal sensing data collected by a fire-fighting robot cluster in real time, and generating a corresponding local environment model for each robot in the cluster; the working state of the current master control fire-fighting robot is monitored, if abnormity is found, the master control performance of a non-master control robot cluster (except the current master control robot) is analyzed according to the local environment models, and a new master control fire-fighting robot is determined; acquiring associated multi-mode sensing data collected by a new master controller from the local environment model of the slave robot cluster through a wireless ad hoc network, and generating a new global collaborative operation strategy according to the associated multi-mode sensing data; and distributing the new global collaborative operation strategy to each robot of the non-master-control robot cluster, and controlling the robots to carry out fire-fighting operation according to the strategy. The technical problem that a fire-fighting robot cluster is difficult to collaboratively carry out fire-fighting operation under the condition that a master control node fails in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of firefighting robot technology, and in particular to a multimodal firefighting robot swarm collaboration method, system, medium and product. Background Technology

[0002] With the rapid development of intelligent technologies and the increasing complexity of fire rescue needs, multi-robot collaborative operations have become an important technical means in high-risk scenarios such as fire fighting, hazardous chemical spill response, and building collapse rescue. Faced with the dynamic changes in the fire scene environment and the urgency of rescue missions, there is an urgent need to establish an efficient and stable robot cluster collaborative mechanism to achieve intelligent operations in complex firefighting scenarios.

[0003] In related technologies, a fixed master-slave cluster control architecture is typically adopted. In practice, during the deployment phase, a high-performance robot is pre-designated as the fixed master control node. This master robot is responsible for collecting sensor data from the entire cluster. Then, the master robot centrally stores the raw sensor data uploaded by each slave robot, including information such as temperature, smoke concentration, and images, and creates an index in a local database. Next, based on a preset task allocation algorithm, the master robot calculates and generates a unified sequence of operation instructions according to the position information and hardware configuration parameters of each slave robot. Finally, the specific action instructions are sent to each slave robot for execution via point-to-point communication. While this cluster architecture with a fixed master control node at its core can achieve basic multi-robot collaboration, the operation of the entire system is highly dependent on the continuous and normal operation of the single master control node.

[0004] However, when using the above-mentioned fixed master-slave collaborative method, once the preset master robot fails due to high temperature damage, communication interruption or energy depletion, the entire robot cluster will lose its coordination and control capabilities. Each slave robot can only operate independently based on its own limited perception range, making it difficult to form effective collaboration. Consequently, in the case of a failure of the master control node, the fire-fighting robot cluster in the relevant technology will find it difficult to coordinate fire-fighting operations. Summary of the Invention

[0005] This application provides a multimodal fire-fighting robot cluster collaboration method, system, medium, and product, which enables the fire-fighting robot cluster to continue collaborative fire-fighting operations even in the event of a failure of the master control node.

[0006] Firstly, this application provides a multimodal firefighting robot swarm collaboration method, applied to the aforementioned multimodal firefighting robot swarm collaboration system. The method includes: acquiring real-time multimodal perception data collected by the firefighting robot swarm, and generating corresponding local environment models for each firefighting robot in the swarm based on the multimodal perception data; monitoring the working status of the current master firefighting robot in the swarm, so that when an abnormal working status of the current master firefighting robot is detected, a master control performance analysis is performed on the non-master firefighting robot swarm (excluding the current master firefighting robot) based on each local environment model, to determine a new master firefighting robot from the non-master firefighting robot swarm; acquiring associated multimodal perception data collected by the new master firefighting robot from the local environment models corresponding to the subordinate robot swarm via a wireless ad hoc network, and generating a new global collaborative operation strategy based on the associated multimodal perception data, wherein the subordinate robot swarm is a set of robots in the non-master firefighting robot swarm excluding the new master firefighting robot; and distributing the new global collaborative operation strategy to each firefighting robot in the non-master firefighting robot swarm, to control each firefighting robot to perform firefighting operations according to the new global collaborative operation strategy.

[0007] By adopting the above technical solution, multimodal perception data collected in real time by each fire-fighting robot in the fire-fighting robot cluster through its onboard sensors is acquired. This multimodal perception data is processed to generate a corresponding local environment model for each fire-fighting robot, enabling each robot to independently grasp its surrounding environment information. The current master fire-fighting robot's operating status is continuously monitored. Once an abnormal operating status is detected, the master performance of the non-master robot cluster is immediately analyzed using the local environment models generated by each fire-fighting robot, selecting the most suitable new master fire-fighting robot. The new master fire-fighting robot can collect associated multimodal perception data from the subordinate robot cluster via a wireless ad hoc network, generate a new global collaborative operation strategy based on this data, and distribute this new global collaborative operation strategy to each fire-fighting robot in the non-master robot cluster for execution. This dynamic master switching mechanism ensures that even if the current master fire-fighting robot fails, the entire fire-fighting robot cluster can maintain the continuity of collaborative operation by quickly electing a new master node, thereby effectively avoiding the single point of failure problem under a fixed master architecture. This solves the technical problem that fire-fighting robot clusters are unable to coordinate fire-fighting operations when the main control node fails, and achieves the technical effect that fire-fighting robot clusters can continue to coordinate fire-fighting operations even when the main control node fails.

[0008] Optionally, the working status of the current master fire-fighting robot in the fire-fighting robot cluster is monitored. When an abnormal working status of the current master fire-fighting robot is detected, the master control performance of the non-master fire-fighting robot cluster (excluding the current master fire-fighting robot) in the fire-fighting robot cluster is analyzed based on the local environment model to determine a new master fire-fighting robot from the non-master fire-fighting robot cluster. Specifically, this includes: obtaining the real-time operating status parameters of the current master fire-fighting robot, and determining an abnormal working status when the real-time operating status parameters do not meet the preset parameter conditions; extracting scene performance factors from the local environment model corresponding to each fire-fighting robot; performing performance fusion processing on the scene performance factors to determine the master control performance index value of each fire-fighting robot; and determining a new master fire-fighting robot from the non-master fire-fighting robot cluster based on the master control performance index value.

[0009] By adopting the above technical solution, and comparing the real-time operating status parameters of the current master control firefighting robot with preset parameter conditions, abnormal states of the master control node can be detected in a timely manner. Once an abnormal working state is determined, scene performance factors are extracted from the local environment model corresponding to each firefighting robot. These scene performance factors reflect the operational capabilities of each firefighting robot in the current environment. Through performance fusion processing of the scene performance factors, the master control performance index value of each firefighting robot is determined. This master control performance index value comprehensively considers multiple dimensions such as the environmental adaptability and task execution capability of each firefighting robot. Based on the master control performance index value, the optimal robot is selected from the non-master control robot cluster as the new master control firefighting robot, ensuring that the newly selected master control node has the best overall performance. This master control selection mechanism based on real-time performance evaluation, compared to the method of preset fixed master control, can dynamically select the most suitable master control node according to the actual operating environment and robot status, thereby improving the adaptability and reliability of the entire cluster system.

[0010] Optionally, a new master firefighting robot can be determined from the non-master robot cluster based on the master control performance index value. Specifically, this includes: sorting each firefighting robot in descending order of performance based on the master control performance index value to obtain a master control candidate list; selecting the first-ranked candidate robot in the master control candidate list as the candidate master control robot; sending the first master control permission transfer request to the candidate master control robot; and, after receiving the confirmation response message returned by the candidate master control robot, determining the candidate master control robot as the new master firefighting robot.

[0011] By adopting the above technical solution, the firefighting robots are sorted in descending order of performance based on their master control performance indicators to generate a master control candidate list, ensuring that the robot with the best performance is placed first. The first-ranked candidate robot in the master control candidate list is selected as the candidate master control robot, and an initial master control authority transfer request is sent to it. This request-confirmation mechanism ensures the reliability of the authority transfer. Only after receiving a confirmation response message from the candidate master control robot is it confirmed as the new master firefighting robot, effectively avoiding uncertainty in the authority transfer process. This master control selection process based on performance sorting and handshake confirmation not only ensures that the robot with the best performance is selected as the new master control robot, but also ensures through the confirmation mechanism that the selected robot truly has the ability and willingness to take over master control authority, effectively preventing master control authority transfer failure due to communication failures or sudden changes in robot state, thereby improving the success rate and reliability of the master control switching process.

[0012] Optionally, after sending a master control permission transfer request to the candidate master control robot, the method further includes: if the candidate master control robot does not respond or returns a master control permission rejection message within a preset time period, marking the preferred candidate robot as a master control transfer failure state; selecting the second-ranked secondary candidate robot from the master control candidate list as the new candidate master control robot; obtaining real-time multimodal perception data from the local environment model corresponding to the new candidate master control robot, and determining whether the current master control performance index value of the new candidate master control robot meets the preset master control conditions based on the real-time multimodal perception data; if it is determined that the current master control performance index value meets the preset master control conditions, sending a second master control permission transfer request to the new candidate master control robot; if the master control permission transfer request fails N times consecutively for the sequential candidate robot in the master control candidate list, then activating the emergency collaboration mode, which selects multiple firefighting robots that meet the preset collaboration conditions from the master control candidate list as the new master firefighting robots.

[0013] By adopting the above technical solution, when a candidate master robot fails to respond or returns a master control permission rejection message within a preset time period, the preferred candidate robot is marked as failing to transfer master control, avoiding indefinite waiting. The second-ranked candidate robot is immediately selected from the master control candidate list as the new candidate master robot, and real-time multimodal perception data is obtained from the corresponding local environment model. The real-time performance of the candidate robot is ensured by judging whether the current master control performance index value meets the preset master control conditions. After N consecutive failed master control permission transfers, an emergency collaboration mode is activated, selecting multiple firefighting robots that meet the preset collaboration conditions from the master control candidate list to jointly serve as the new master firefighting robot, achieving distributed master control. This multi-level fault-tolerant mechanism, through a step-by-step degradation strategy and the final multi-master collaboration mode, ensures that basic collaborative operation capabilities can be maintained even in extreme situations, greatly improving the survivability and task completion rate of the firefighting robot cluster in complex and harsh environments.

[0014] Optionally, scene performance factors are extracted from the local environment models corresponding to each firefighting robot. Specifically, this includes: extracting 3D point cloud data and first thermal imaging data from the local environment model; performing spatial accessibility analysis on the 3D point cloud data to obtain spatial accessibility parameters, and performing temperature field analysis on the first thermal imaging data to obtain thermal barrier hazard parameters; generating environmental accessibility factors based on the spatial accessibility parameters and thermal barrier hazard parameters; extracting second thermal imaging data and gas concentration data from the local environment model; performing fire situation analysis on the second thermal imaging data to obtain fire characteristic parameters, and performing gas distribution analysis on the gas concentration data to obtain spread risk parameters; generating fire resistance factors based on the fire characteristic parameters and spread risk parameters; acquiring the body state data of each firefighting robot and performing state fusion processing on the body state data to obtain resource availability parameters; generating self-state factors based on resource availability parameters; and using the environmental accessibility factors, fire resistance factors, and self-state factors as scene performance factors.

[0015] By employing the aforementioned technical solution, 3D point cloud data and first thermal imaging data are extracted from the local environment model. Spatial accessibility parameters and thermal barrier hazard parameters are obtained through spatial accessibility analysis and temperature field analysis, respectively. These are combined to generate an environmental accessibility factor, comprehensively evaluating the robot's environmental navigation capabilities. Simultaneously, second thermal imaging data and gas concentration data are extracted. Fire situation analysis and gas distribution analysis yield fire characteristic parameters and spread risk parameters, generating a fire resistance factor that reflects the degree of fire threat faced by the robot. The robot's own state data is acquired and fused to obtain resource availability parameters, generating its own state factor, reflecting the robot's operational capabilities. The environmental accessibility factor, fire resistance factor, and own state factor are used as scenario performance factors to achieve a multi-dimensional quantitative evaluation of the comprehensive performance of firefighting robots in specific scenarios. This multi-source data fusion performance factor extraction method fully utilizes the complementarity of different types of sensor data, providing a more comprehensive and accurate basis for performance evaluation.

[0016] Optionally, the scene performance factors are subjected to performance fusion processing to determine the master control performance index value of each firefighting robot. Specifically, this includes: constructing a multi-dimensional performance vector including environmental access factors, fire situation resistance factors, and self-state factors; determining the fire scene complexity level based on fire characteristic parameters, and selecting the corresponding adaptive fusion model from a preset fusion model library based on the fire scene complexity level; inputting the multi-dimensional performance vector into the adaptive fusion model and obtaining the comprehensive performance feature value output by the adaptive fusion model; obtaining the historical collaborative operation records and current relative position relationships of the non-master control robot cluster, and determining the collaborative correlation degree of each firefighting robot based on the historical collaborative operation records and current relative position relationships; and performing adaptive aggregation processing on the collaborative correlation degree and comprehensive performance feature value to generate the master control performance index value.

[0017] By employing the aforementioned technical solution, a multi-dimensional performance vector is constructed, comprising environmental accessibility factors, fire situation resistance factors, and self-state factors, integrating scattered performance indicators into a unified data structure. The fire complexity level is determined based on fire characteristic parameters, and a corresponding adaptive fusion model is selected from a pre-set fusion model library accordingly, enabling the fusion strategy to dynamically adjust based on the actual fire situation. The multi-dimensional performance vector is input into the adaptive fusion model to obtain comprehensive performance feature values, achieving intelligent fusion of multi-dimensional performance. Furthermore, historical collaborative operation records and current relative positions of the non-master control robot cluster are acquired to determine the collaborative correlation degree of each firefighting robot, reflecting the closeness of cooperation between robots. Master control performance index values ​​are generated through adaptive aggregation processing of the collaborative correlation degree and comprehensive performance feature values, considering both individual performance and collaborative capabilities. This adaptive performance fusion mechanism can dynamically adjust evaluation weights according to the fire situation and collaborative needs, generating master control performance evaluation results that better meet actual operational requirements.

[0018] Optionally, the newly appointed master fire-fighting robot acquires associated multimodal perception data collected from the local environment model corresponding to the subordinate robot cluster via a wireless ad hoc network, and generates a new global collaborative operation strategy based on the associated multimodal perception data. Specifically, this includes: acquiring the associated multimodal perception data collected by the newly appointed master fire-fighting robot during the following operations: the newly appointed master fire-fighting robot sends data acquisition requests to each subordinate robot in the subordinate robot cluster via a wireless ad hoc network; the newly appointed master fire-fighting robot receives the associated multimodal perception data returned by each subordinate robot based on the received data acquisition requests; and the newly appointed master fire-fighting robot performs spatiotemporal alignment processing on the associated multimodal perception data to obtain... The new master firefighting robot constructs a global fire situation map based on the spatiotemporally synchronized perception data. The global fire situation map includes information on fire source distribution, temperature field distribution, smoke concentration distribution, and obstacle distribution. The new master firefighting robot determines the fire control points and priority control areas from the global fire situation map, and generates a task allocation plan based on the fire control points, priority control areas, and the current operating status information of each subordinate robot. The new master firefighting robot generates individual operation instructions for each subordinate robot according to the task allocation plan. The new global collaborative operation strategy generated by the new master firefighting robot by combining the task allocation plan with the individual operation instructions is obtained.

[0019] By adopting the above technical solution, the newly appointed master firefighting robot sends data acquisition requests to the subordinate robot cluster via a wireless ad hoc network and receives the returned associated multimodal perception data, achieving centralized aggregation of distributed data. The associated multimodal perception data undergoes spatiotemporal alignment processing to obtain spatiotemporally synchronized perception data, eliminating spatiotemporal differences in data acquisition from different firefighting robots. Based on the spatiotemporally synchronized perception data, a global fire situation map is constructed, including information on fire source distribution, temperature field distribution, smoke concentration distribution, and obstacle distribution, achieving comprehensive perception of the entire fire environment. Fire control points and priority control areas are determined from the global fire situation map. A task allocation scheme is generated by combining the current operational status information of each subordinate robot, and individual operational instructions are generated for each subordinate robot. The task allocation scheme and individual operational instructions are combined to generate a new global collaborative operation strategy, realizing a complete closed loop from global perception to collaborative decision-making to task allocation, ensuring that the firefighting robot cluster can perform efficient collaborative operations based on real-time global information.

[0020] Secondly, embodiments of this application provide a multimodal fire-fighting robot swarm collaborative system, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, which includes computer instructions, and the one or more processors invoke the computer instructions to cause the multimodal fire-fighting robot swarm collaborative system to perform the methods described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a multimodal fire-fighting robot cluster collaborative system, cause the multimodal fire-fighting robot cluster collaborative system to execute the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a multimodal fire-fighting robot cluster collaborative system, cause the multimodal fire-fighting robot cluster collaborative system to perform the method described in the first aspect and any possible implementation thereof. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a multimodal firefighting robot cluster collaboration method in an embodiment of this application. Figure 2 This is a schematic diagram of the physical device structure of an electronic device in an embodiment of this application. Detailed Implementation

[0024] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0025] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0026] This application provides a method for multimodal firefighting robot swarm collaboration, see reference. Figure 1 , Figure 1 This is a flowchart illustrating a multimodal firefighting robot swarm collaboration method in an embodiment of this application, including the following steps: Step S101: Obtain the self-contained multimodal perception data collected in real time by the fire-fighting robot cluster, and generate a corresponding local environment model for each fire-fighting robot in the fire-fighting robot cluster based on the self-contained multimodal perception data. Step S102: Monitor the working status of the current master fire robot in the fire robot cluster. When an abnormal working status of the current master fire robot is detected, perform master control performance analysis on the non-master fire robot cluster in the fire robot cluster other than the current master fire robot according to each local environment model, so as to determine the new master fire robot from the non-master fire robot cluster. Step S103: Obtain the associated multimodal perception data collected by the new master fire-fighting robot from the local environment model corresponding to the subordinate robot cluster through the wireless self-organizing network, and generate a new global collaborative operation strategy based on the associated multimodal perception data. The subordinate robot cluster is a set of robots in the non-master robot cluster except for the new master fire-fighting robot. Step S104: Distribute the new global collaborative operation strategy to each firefighting robot in the non-master control robot cluster, so as to control each firefighting robot to perform firefighting operations according to the new global collaborative operation strategy.

[0027] Among them, the firefighting robot cluster refers to a collaborative group composed of multiple firefighting robots with autonomous movement and operation capabilities, specifically including tracked firefighting robots, wheeled fire reconnaissance robots, flying fire monitoring robots, serpentine pipeline detection robots, and water cannon fire extinguishing robots; multimodal perception data represents the environmental information independently collected by each firefighting robot through its own onboard sensors, specifically including lidar point cloud data, infrared thermal imaging data, visible light image data, toxic and harmful gas concentration data, and environmental temperature and humidity data; local environment models are used to represent the digital representation of the local scene constructed by a single firefighting robot based on its perception range, specifically including a 3D obstacle map. The system includes: temperature field distribution model, smoke concentration gradient map, and passable area marking map; the current master control firefighting robot refers to the central control node in the cluster responsible for global decision-making and task allocation; abnormal working status indicates that the master control robot has a fault condition that affects its normal performance of master control duties, specifically including communication link interruption, processor overheat protection, severe battery power shortage, and large-scale sensor failure; non-master control robot cluster refers to all subordinate execution robots other than the current master control firefighting robot; the newly appointed master control firefighting robot refers to the robot newly selected through performance evaluation to replace the original master control robot; wireless ad hoc network refers to the network topology autonomously constructed by robots based on wireless communication, specifically including multi-hop network based on WiFi Mesh, long-distance communication network based on LoRa, and low-latency network based on 5G; associated multimodal perception data is used to represent perception information collected and associated from multiple subordinate robots; the new global collaborative operation strategy represents the cluster collaboration scheme formulated based on global information, specifically including task allocation matrix, path planning scheme, resource scheduling plan, and collaborative timing arrangement.

[0028] Specifically, this step operates continuously throughout the entire process of the firefighting robot swarm performing fire scene operations, especially in scenarios requiring coordinated swarm operations in complex fire environments. First, various sensors on each firefighting robot collect multimodal perception data of the surrounding environment in real time. This raw data undergoes preprocessing, feature extraction, and data fusion to construct a local environment model for each robot reflecting the environmental conditions within its perception range. Simultaneously, the operational status of the current master firefighting robot is continuously monitored, including hardware health, communication quality, and computing resource utilization. When an abnormality is detected in the master robot, rendering it unable to continue fulfilling its master control responsibilities, a master control switchover process is immediately initiated. By analyzing the local environment models of each non-master robot, the comprehensive performance indicators of their environmental perception capabilities, operational execution capabilities, and collaborative capabilities are evaluated, selecting the most suitable new master firefighting robot to take over master control. Once the new master is determined, it sends data requests to all subordinate robots via a wireless ad hoc network, collects and fuses local perception data, and generates new collaborative operation strategies based on the fused global environmental information. These strategies include reallocating firefighting tasks, adjusting operational routes, and optimizing resource allocation. Finally, the new strategy is distributed to each execution robot to ensure that the entire cluster can quickly resume collaborative operation after the master switch.

[0029] In some embodiments, dynamic master control switching and collaborative operation of firefighting robot clusters can be achieved in a variety of ways: Optionally, a distributed processing architecture based on edge computing is adopted. Each firefighting robot is equipped with an edge computing unit, which processes sensor data such as LiDAR, depth camera, and infrared thermal imager in real time through convolutional neural networks to generate a local environment model containing semantic information. The master control status monitoring adopts a heartbeat packet mechanism combined with state vector broadcasting. Each robot periodically sends a state vector containing information such as CPU utilization, memory usage, battery level, and sensor status. Other robots evaluate the master control fitness score of each node through a distributed consensus algorithm. When the master control fails, the node with the highest score is automatically elected as the new master control. The new master control collects global data through a publish-subscribe model and uses a deep reinforcement learning algorithm to generate the optimal collaborative strategy. Optionally, a hierarchical architecture with cloud-edge-device collaboration can be adopted. The firefighting robot, as the end device, is responsible for raw data acquisition; the edge server handles local data processing and model generation; and the cloud server is responsible for global decision-making and master control management. When a master control anomaly is detected, the cloud server calculates the master control priority of each robot based on stored historical operation data and real-time performance indicators using a fuzzy comprehensive evaluation method. After selecting a new master control, a master control switching command is pushed through a reliable multicast protocol. The new master control collects environmental data from each robot through a RESTful API interface, constructs a global situational awareness map using a graph neural network, and generates a collaborative operation plan considering factors such as task priority, resource constraints, and safety distance through a multi-objective optimization algorithm. It is understood that other methods can also be used to achieve collaborative control of a firefighting robot cluster based on dynamic master control switching; this is not limited here.

[0030] Through the above steps, multimodal perception data collected in real time by each fire-fighting robot in the fire-fighting robot cluster via its onboard sensors is acquired. This multimodal perception data is processed to generate a corresponding local environment model for each fire-fighting robot, enabling each robot to independently grasp its surrounding environment. The current master fire-fighting robot's operating status is continuously monitored. If an abnormality is detected, the master performance of the non-master robot cluster is immediately analyzed using the local environment models generated by each fire-fighting robot, selecting the most suitable new master fire-fighting robot. The new master fire-fighting robot can collect associated multimodal perception data from the subordinate robot cluster via a wireless ad hoc network. Based on this associated multimodal perception data, a new global collaborative operation strategy is generated and distributed to each fire-fighting robot in the non-master robot cluster for execution. This dynamic master switching mechanism ensures that even if the current master fire-fighting robot fails, the entire fire-fighting robot cluster can maintain the continuity of collaborative operation by quickly electing a new master node, effectively avoiding the single point of failure problem under a fixed master architecture. This solves the technical problem that fire-fighting robot clusters are unable to coordinate fire-fighting operations when the main control node fails, and achieves the technical effect that fire-fighting robot clusters can continue to coordinate fire-fighting operations even when the main control node fails.

[0031] The entity performing the above steps may be a control system, a control device, a controller or processor in the device or system, a standalone controller or processor, or other processing devices or processing units with similar processing functions, but is not limited to these.

[0032] In an optional embodiment, the working status of the current master fire-fighting robot in the fire-fighting robot cluster is monitored. When an abnormal working status of the current master fire-fighting robot is detected, the master control performance of the non-master fire-fighting robot cluster (excluding the current master fire-fighting robot) in the fire-fighting robot cluster is analyzed based on the local environment model to determine a new master fire-fighting robot from the non-master fire-fighting robot cluster. Specifically, this includes: obtaining the real-time operating status parameters of the current master fire-fighting robot, and determining an abnormal working status when the real-time operating status parameters do not meet preset parameter conditions; extracting scene performance factors from the local environment model corresponding to each fire-fighting robot; performing performance fusion processing on the scene performance factors to determine the master control performance index value of each fire-fighting robot; and determining a new master fire-fighting robot from the non-master fire-fighting robot cluster based on the master control performance index value.

[0033] Among them, real-time operating status parameters represent quantitative indicators reflecting the current operating status of the main control firefighting robot, specifically including processor temperature, memory usage, network communication latency, sensor data update frequency, remaining battery percentage, and vibration amplitude of mechanical components; preset parameter conditions refer to the threshold standards for judging whether the main control robot is working properly, such as CPU temperature continuously exceeding the preset threshold of 85 degrees Celsius, battery level below the warning value of 20%, and continuous interruption of heartbeat communication with the cluster management node for more than 3 seconds; scenario performance factors are used to represent the robot's operational capabilities in specific fire scene environments, specifically including environmental complexity factors, fire threat factors, mobility factors, and perception coverage factors; performance fusion processing refers to the process of integrating multiple performance factors into a comprehensive evaluation index through a specific algorithm; the main control performance index value represents a numerical value that quantitatively evaluates the suitability of the robot to assume the main control role, and the value range is usually normalized to between 0 and 1.

[0034] Specifically, this step is performed in scenarios where a main control robot's health assessment and alternative main control robot selection are required during the execution of tasks by the firefighting robot cluster. Monitoring modules deployed on the current main control firefighting robot collect various parameters reflecting its operational status in real time. These parameters cover physical quantities at the hardware level such as temperature, power consumption, and vibration; system indicators at the software level such as CPU utilization, memory usage, and process response time; and performance indicators at the functional level such as sensor accuracy, communication quality, and decision latency. The collected real-time operational status parameters are compared with pre-set multi-level threshold conditions. When any key parameter exceeds the safe range or multiple parameters simultaneously approach the critical value, the current main control robot's operating status is determined to be abnormal. After confirming the abnormality, scene performance factors reflecting the robot's operational capabilities in the current scenario are extracted from the local environment models already constructed by each non-main control robot. These scene performance factors include not only the robot's own hardware performance and remaining resources but also environmental factors such as the environmental characteristics of its location, its relative relationship to the fire source, and the distribution of surrounding obstacles. By employing fusion algorithms such as weighted summation, fuzzy inference, or neural networks, multi-dimensional scene performance factors are comprehensively processed into a single master control performance index value. The higher this value, the more suitable the robot is for the master control role. Finally, based on the calculated master control performance index value, the robot with the highest score and meeting the minimum performance requirements is selected as the new master control firefighting robot.

[0035] In some embodiments, master controller status monitoring and new master controller selection can be implemented in multiple ways: Optionally, a threshold-based detection and weighted scoring method is adopted. Multiple monitoring processes are deployed on the master robot to collect status parameters such as CPU temperature, GPU temperature, battery voltage, current, available memory, and network packet loss rate once per second. The moving average of each parameter is calculated using a sliding window algorithm to eliminate instantaneous fluctuations. When any parameter exceeds the preset threshold range within three consecutive sampling periods, an abnormal alarm is triggered. Then, environmental features such as obstacle density, temperature gradient, and visibility, as well as the robot's own features such as moving speed, turning flexibility, and sensor integrity rate, are extracted from the local environment model of each robot as scene performance factors. The weight of each factor is determined by the analytic hierarchy process, and the comprehensive score is calculated using a linear weighted method as the master robot's performance index value. The robot with the highest score is selected as the new master robot. Optionally, a machine learning-based intelligent evaluation method can be adopted. A Long Short-Term Memory (LSTM) network is used to analyze the historical state sequence of the master robot to predict the probability of failure within a future period. When the predicted failure probability exceeds a set threshold, the master robot switching process is initiated in advance. Simultaneously, a convolutional neural network is used to automatically extract deep features from the local environment models of each robot as scene performance factors, including spatial accessibility features, thermal distribution features, and smoke diffusion features. These features are input into a pre-trained deep neural network model, outputting a master robot fitness score for each robot. The score is converted into a probability distribution using a softmax function, and the robot with the highest probability is selected as the new master robot. It is understood that other methods can also be used to achieve master robot anomaly detection and alternative master robot evaluation; this is not limited here.

[0036] In an optional embodiment, a new master firefighting robot is determined from the non-master robot cluster based on the master control performance index value. Specifically, this includes: sorting each firefighting robot in descending order of performance based on the master control performance index value to obtain a master control candidate list; selecting the first preferred candidate robot in the master control candidate list as the candidate master control robot; sending a first master control authority transfer request to the candidate master control robot; and, after receiving a confirmation response message returned by the candidate master control robot, determining the candidate master control robot as the new master firefighting robot.

[0037] Among them, performance descending order sorting refers to the process of arranging robots from high to low according to the master control performance index value; the master control candidate list represents the robot sequence after performance sorting, specifically including candidate robot identifier, performance score, current status, communication address and other information; the first-choice candidate robot is used to represent the best candidate ranked first in the master control candidate list; the candidate master control robot is the target robot selected to take over master control authority; the first master control authority transfer request is the control message used to initiate the handover of master control authority, specifically including fields such as request type identifier, sender identity, timestamp, and permission scope description; the confirmation response message is the reply message of the candidate robot agreeing to take over master control authority, specifically including response code, robot status information, estimated switchover time and other content.

[0038] Specifically, this step is executed after the performance evaluation of each robot's master controller is completed, in scenarios where a new master controller needs to be determined. First, the master controller performance index values ​​of all non-master controller robots are quickly sorted or heap-sorted to generate a master controller candidate list ranked from highest to lowest performance. This list not only records the performance score of each robot but also includes auxiliary information such as the robot's current position, task status, and communication quality. The robot ranked first in the master controller candidate list is selected as the primary candidate robot and designated as the candidate master controller robot. A first master controller transfer request containing detailed master controller permission descriptions, switchover time requirements, and data synchronization needs is sent to the candidate master controller robot via a reliable communication protocol. Upon receiving the request, the candidate master controller robot performs a self-check to confirm its takeover capability, including checking whether computing resources are sufficient, communication links are stable, and key modules are functioning correctly. After confirming everything is correct, it returns a confirmation response message containing an acceptance flag, preparation completion time, and data receiving port. Upon receiving the confirmation response message, the original master controller or cluster management module officially updates the candidate master controller robot's role to the new master firefighting robot and notifies other members in the cluster to adjust communication routes and control permissions accordingly.

[0039] In some embodiments, performance-based master control selection and permission transfer can be implemented in a variety of ways: Optionally, an implementation based on a priority queue and a three-way handshake is adopted. A max-heap data structure is used to maintain the master control candidate list. The top element of the heap is the preferred candidate robot with the best performance. A SYN request packet containing the sequence number, master control permission level, and validity period is sent to the candidate robot as the first master control permission transfer request. After verifying the legality of the request, the candidate robot returns a SYN-ACK packet containing its own sequence number and acknowledgment number. After receiving the SYN-ACK packet, the original master control sends an ACK packet to complete the three-way handshake and starts the data synchronization process. The data is transmitted to transmit key information such as the global task queue, robot status table, and environment map. After the data synchronization is completed, all robots are notified to update the master control address through a broadcast message to complete the master control switch. Optionally, a decentralized implementation based on blockchain and smart contracts can be adopted. The master control performance index and robot identifier are written into the blockchain. The smart contract automatically executes the sorting and selection logic. After determining the preferred candidate robot, a smart contract containing the master control transfer conditions, permission scope, and effective time is created as the first master control permission transfer request. The candidate robot confirms acceptance of the contract by signing with its private key. The signature information is broadcast to the entire network as a confirmation response message. After other nodes verify the signature validity, they automatically identify the robot as the new master firefighting robot. The entire process is transparent, traceable, and tamper-proof. It is understood that other methods can also be used to achieve performance-prioritized master control selection and secure and reliable permission transfer; this is not limited here.

[0040] In an optional embodiment, after sending a master control permission transfer request to the candidate master control robot, the method further includes: if the candidate master control robot does not respond or returns a master control permission rejection message within a preset time period, marking the preferred candidate robot as a master control transfer failure state; selecting the second-ranked secondary candidate robot from the master control candidate list as the new candidate master control robot; obtaining real-time multimodal perception data from the local environment model corresponding to the new candidate master control robot, and determining whether the current master control performance index value of the new candidate master control robot meets the preset master control conditions based on the real-time multimodal perception data; if it is determined that the current master control performance index value meets the preset master control conditions, sending a second master control permission transfer request to the new candidate master control robot; if the master control permission transfer request fails N times consecutively for the sequential candidate robot in the master control candidate list, then activating an emergency collaboration mode, wherein the emergency collaboration mode selects multiple firefighting robots that meet the preset collaboration conditions from the master control candidate list as the new master firefighting robots.

[0041] The preset time period represents the timeout limit for waiting for a candidate robot's response (e.g., 3-10 seconds), specifically including a time window set after comprehensively considering network round-trip latency, processing latency, and buffering latency. The master control permission rejection message is the response from a candidate robot that cannot take over master control for various reasons, including rejection reason codes such as insufficient resources, task conflicts, and hardware failures. The master control transfer failure status is used to mark the status of a candidate robot that failed to successfully take over master control. The second-choice candidate robot represents the second-ranked alternative robot in the master control candidate list. The new candidate master control robot refers to the candidate selected again after the first choice failed. The current master control performance index value represents the real-time performance evaluation value of the candidate robot at the current moment. The preset master control conditions refer to the minimum standards for judging whether a robot is still suitable to serve as master, specifically including performance index thresholds and key function integrity requirements. The second master control permission transfer request represents a permission transfer request initiated to the new candidate master control robot. The emergency collaboration mode refers to a distributed control mode in which multiple robots jointly assume master control responsibilities. The preset collaboration conditions are used to screen robots that can participate in distributed master control, specifically including minimum performance requirements, communication capability requirements, and collaboration experience requirements.

[0042] Specifically, this step is executed when the primary candidate robot fails to successfully take over master control authority, providing a multi-level fault tolerance mechanism. If no response is received from the candidate master robot within a preset time period, or if a clear master control authority rejection message is received, the primary candidate robot's status in the master control candidate list is immediately marked as master control transfer failure, and the reason for failure and timestamp are recorded. The second-ranked secondary candidate robot is extracted from the master control candidate list and set as the new candidate master robot. Considering the potential time lag between performance evaluation and actual transfer, it is necessary to reacquire real-time multimodal perception data of the new candidate master robot, including the latest sensor readings, system resource usage, task execution status, etc., and recalculate the current master control performance index based on this real-time data. The recalculated performance index is compared with the preset master control conditions to ensure that the robot still has the capability to act as master. After confirming that the conditions are met, a second master control authority transfer request is sent to the new candidate master robot. This request may contain a higher priority identifier or a more detailed switching plan. If the master control transfer fails N times consecutively (N can be set to 3-5 times depending on the cluster size and real-time communication status), it means that the single master control mode is no longer applicable. It will automatically switch to the emergency collaboration mode, select multiple robots that meet the preset collaboration conditions from the master control candidate list, and jointly exercise the master control function through a distributed consensus algorithm to ensure the basic collaboration capability of the cluster.

[0043] In some embodiments, a multi-level fault-tolerant master control switching mechanism can be implemented in a variety of ways: Optionally, a state machine-based and recursive selection approach is adopted. A finite state machine is used to manage the master control transfer process, including request sending state, waiting for response state, success state, and failure retry state. In the waiting for response state, a timer is started. After the timeout, it automatically jumps to the failure retry state, adds the failed robot to the blacklist, and recursively calls the selection function to obtain the next candidate. Before each retry, the reachability and responsiveness of the candidate robot are quickly detected by sending probe packets. The calculation weight of the current master control performance index value is dynamically adjusted according to the probe results, with emphasis on communication stability and response timeliness. When the number of consecutive failures reaches a threshold, an emergency collaboration mode is triggered. Distributed consensus algorithms such as Raft or Paxos are used to elect 3 or 5 nodes from the robots that meet the minimum performance requirements to form a master control committee, and make decisions together through a voting mechanism. Optionally, an implementation based on prediction models and proactive switching can be adopted. A time-series prediction model is used to analyze the historical response records of candidate robots, predicting their response probability and expected delay. Based on this, the length of the preset time period is dynamically adjusted. For candidates with low predicted response probabilities, the waiting time is skipped or shortened. Simultaneously, preparatory notifications are sent to multiple candidates in parallel, allowing them to prepare for takeover. When the primary candidate fails, the system immediately switches to a ready backup. Before each attempt at a new candidate, its real-time status is quickly evaluated via edge computing nodes, including task queue length, CPU load, and memory pressure. Only candidates whose overall score still exceeds a preset threshold receive a formal request for master control transfer. When an emergency collaboration mode is triggered, different coordination responsibilities are automatically assigned based on the robot's expertise and location, such as perception coordination, decision coordination, and communication relay, achieving distributed master control with separation of responsibilities. It is understood that other methods can also be used to implement a fault-tolerant master control switching process; this is not limited here.

[0044] In an optional embodiment, scene performance factors are extracted from the local environment model corresponding to each firefighting robot. Specifically, this includes: extracting 3D point cloud data and first thermal imaging data from the local environment model; performing spatial accessibility analysis on the 3D point cloud data to obtain spatial accessibility parameters, and performing temperature field analysis on the first thermal imaging data to obtain thermal barrier hazard parameters; generating an environmental accessibility factor based on the spatial accessibility parameters and thermal barrier hazard parameters; extracting second thermal imaging data and gas concentration data from the local environment model; performing fire situation analysis on the second thermal imaging data to obtain fire characteristic parameters, and performing gas distribution analysis on the gas concentration data to obtain spread risk parameters; generating a fire resistance factor based on the fire characteristic parameters and spread risk parameters; acquiring the body state data of each firefighting robot and performing state fusion processing on the body state data to obtain resource availability parameters; generating a self-state factor based on the resource availability parameters; and using the environmental accessibility factor, fire resistance factor, and self-state factor as scene performance factors.

[0045] Among them, 3D point cloud data represents spatial point set information acquired by LiDAR or depth camera, specifically including attributes such as the 3D coordinates, reflection intensity, and normal vector of the points; first thermal imaging data refers to infrared images used for environmental temperature analysis, specifically including information such as temperature matrix, hotspot distribution, and temperature gradient; spatial mobility analysis refers to the process of assessing the feasibility of robot movement in 3D space; spatial mobility parameters represent quantified mobility indicators, specifically including the area of ​​the traversable region, obstacle density, terrain undulation, and minimum passage width; temperature field analysis is the process of identifying high-temperature areas and temperature distribution characteristics; thermal barrier hazard parameters represent quantified indicators of the degree of threat posed by high temperatures to robot passage, specifically including the proportion of high-temperature areas, the highest temperature value, and the steepness of the temperature gradient; environmental mobility factor is a comprehensive reflection of the robot's performance in the physical environment. The data includes: indicators of mobility; secondly, thermal imaging data, which refers to infrared images used for fire identification; fire situation analysis, which describes the process of assessing the development of a fire; fire characteristic parameters, which describe the intensity and spread characteristics of a fire, including flame height, burning area, spread rate, and heat release rate; gas concentration data, which represents the concentration values ​​measured by various gas sensors, including the concentrations of carbon monoxide, carbon dioxide, smoke particles, and toxic gases; spread risk parameters, which represent an assessment indicator of the likelihood of fire spread; fire resistance factors, which measure the robot's ability to respond to fires; body state data, which represents the robot's own operating parameters, including battery power, water tank capacity, equipment temperature, and actuator status; resource availability parameters, which represent a quantitative indicator of the robot's available resources; and self-state factors, which reflect a comprehensive indicator of the robot's continuous operating capability.

[0046] Specifically, this step is performed when a comprehensive assessment of the integrated operational capabilities of each firefighting robot in a specific scenario is required. First, different types of perception data are extracted from the local environment models of each robot for specialized analysis. For 3D point cloud data, algorithms such as voxelization, plane fitting, and obstacle clustering are used to identify passable areas and obstacle distribution, calculating spatial passability parameters such as channel width, slope, and obstacle density. The first thermal imaging data undergoes temperature threshold segmentation, hotspot detection, and isotherm extraction to identify high-temperature hazard areas, calculating the statistical characteristics of temperature distribution as thermal barrier hazard parameters. The parameters from both spatial and temperature dimensions are normalized and weighted to generate an environmental passability factor. Simultaneously, flame recognition and plume detection algorithms are used to analyze fire distribution and intensity from the second thermal imaging data, extracting fire characteristic parameters. Spatiotemporal interpolation and diffusion model prediction are performed on gas concentration data to assess the distribution range and concentration gradient of harmful gases, generating spread risk parameters. Based on the analysis results of both fire and gas aspects, a fire suppression factor is calculated. In addition, real-time data from the battery management system, liquid level sensors, and temperature sensors of each robot are collected. Through state estimation and remaining lifespan prediction algorithms, the robot's continuous operating time and available resources are assessed, generating resource availability parameters and thus its own state factor. Finally, the environmental accessibility factor, fire suppression factor, and its own state factor are combined to form a scene performance factor that comprehensively reflects the robot's scene adaptability.

[0047] In some embodiments, multi-dimensional scenario performance factors can be extracted in various ways: Optionally, a deep learning-based feature extraction method is adopted, using the PointNet++ network to process 3D point cloud data, automatically learning the local and global features of the point cloud, identifying categories such as ground, walls, and obstacles through semantic segmentation, and calculating the spatial proportion and distribution density of each category as spatial accessibility parameters. At the same time, the U-Net network is used to analyze thermal imaging data, identifying normal temperature zone, high temperature zone, and flame zone through pixel-level classification, and statistically analyzing the area ratio and boundary complexity of each region as thermal barrier hazard parameters. The two types of parameters are input into a fully connected network to generate an environmental accessibility factor. For fire analysis, the YOLO detection network is used to identify flame and smoke areas, and the direction and speed of fire spread are tracked through optical flow. Combined with gas sensor data, the LSTM network is used to predict the gas diffusion trend. The detection and prediction results are fused into a fire resistance factor. For the robot's status, the Kalman filter is used to fuse multi-sensor data, estimate the remaining battery capacity, predict the probability of equipment failure, and generate a self-state factor reflecting the robot's health. Optionally, a rule-based and fuzzy logic-based analysis approach is employed. The RANSAC algorithm is used to extract the ground from the 3D point cloud, and the height histogram of the point cloud above the ground is calculated to identify obstacles. Connected components of passable areas are extracted through morphological operations, and the narrowest and average widths are measured as spatial passability parameters. Otsu's adaptive threshold segmentation method is used on thermal imaging data to identify hazardous areas exceeding a set temperature. The centroid, area, and perimeter of these hazardous areas are calculated as thermal barrier hazard parameters. A fuzzy inference system is used to generate environmental passability factors based on rules such as larger passability width and lower temperature indicating better environmental passability. For fire assessment, color and texture features are used to identify flames, frame difference is used to detect dynamically changing areas, and the spatial gradient of gas concentration is combined to determine the direction of fire development. Fire control difficulty is inferred based on fire dynamics. For the self-state, a mathematical model including power consumption and equipment aging models is established. Model parameters are fitted based on historical data to predict the sustainable operating time under the current load. All predictions are combined to generate a self-state factor. It is understood that other methods can also be used to achieve multi-dimensional extraction and analysis of scene performance factors; this is not limited here.

[0048] In the above embodiments, in actual firefighting scenarios, fire situation analysis not only needs to identify the current fire situation but also predict the fire spread trend. Specifically, a series of temperature field sequences from multiple consecutive frames are extracted from the second thermal imaging data. A time-series difference algorithm is used to calculate the temperature change rate matrix, identifying the area with the fastest temperature rise as the fire spread front. Combined with the building's BIM (Building Information Modeling) data, key information such as combustible material distribution, ventilation opening locations, and fire compartment boundaries is identified. Using a physically constrained fire dynamics model, parameters such as the current temperature field, combustible material load, and ventilation conditions are input. The heat conduction equation and combustion chemical reaction equation are solved using the finite element method to predict the fire development path within the next 5-10 minutes. The prediction results are fused with real-time thermal imaging data using Kalman filtering to dynamically correct the prediction model parameters, generating a set of fire characteristic parameters including a fire spread velocity vector field, a temperature contour evolution sequence, and flashover risk area markings. This in-depth analysis provides firefighting robots with forward-looking fire information, enabling them to be deployed in advance to key locations along the fire spread path, achieving preventative control rather than reactive response.

[0049] In an optional embodiment, the scene performance factors are subjected to performance fusion processing to determine the master control performance index value of each firefighting robot. Specifically, this includes: constructing a multi-dimensional performance vector including environmental access factors, fire situation resistance factors, and self-state factors; determining the fire scene complexity level based on fire characteristic parameters, and selecting a corresponding adaptive fusion model from a preset fusion model library based on the fire scene complexity level; inputting the multi-dimensional performance vector into the adaptive fusion model and obtaining the comprehensive performance feature value output by the adaptive fusion model; obtaining the historical collaborative operation records and current relative position relationships of the non-master control robot cluster, and determining the collaborative correlation degree of each firefighting robot based on the historical collaborative operation records and current relative position relationships; and performing adaptive aggregation processing on the collaborative correlation degree and comprehensive performance feature value to generate the master control performance index value.

[0050] Among them, the multidimensional performance vector represents a data structure that organizes multiple performance factors into a vector form, specifically a numerical vector of [environmental access factor, fire situation resistance factor, and self-state factor]; the fire scene complexity level refers to the graded assessment of the complexity of the fire scene, specifically including simple fire scene, medium complex fire scene, highly complex fire scene, and extremely complex fire scene levels; the pre-set fusion model library represents a set of multiple pre-trained and stored fusion models, specifically including linear fusion model, multinomial fusion model, neural network fusion model, fuzzy inference fusion model, etc.; the adaptive fusion model refers to a model that can dynamically adjust the fusion strategy according to scene characteristics; the comprehensive performance feature value represents the comprehensive performance score after fusing multiple factors; the historical collaborative operation record is used to represent the robot's past collaborative experience information, specifically including collaborative task type, collaboration duration, collaboration effect evaluation, communication stability record, etc.; the current relative position relationship represents the spatial distribution of robots, specifically including relative distance, relative orientation, and whether there is occlusion, etc.; the collaborative correlation degree refers to the index for evaluating the degree of collaboration between robots; and the adaptive aggregation processing represents the process of dynamically adjusting the aggregation weight according to the scene.

[0051] Specifically, this step is performed under the scenario of comprehensively evaluating the robot's master control capabilities after obtaining the performance factors of each dimension of the scene. First, the extracted environmental access factors, fire combat factors, and self-state factors are arranged in a fixed order to construct a multi-dimensional performance vector. This vector comprehensively describes the robot's multifaceted capabilities in the current scenario. Based on indicators such as flame area, peak temperature, and spread rate in the fire characteristic parameters, the complexity level of the current fire scene is determined through preset grading rules or clustering algorithms. Based on the identified complexity level, the most suitable adaptive fusion model for the current scenario is selected from a preset fusion model library. For example, a linear model is selected for simple fire scenes, and a deep neural network model is selected for complex fire scenes. The multi-dimensional performance vector is input into the selected adaptive fusion model. The model performs nonlinear transformation and combination of each dimension factor according to its internal parameters and structure, and outputs a comprehensive performance feature value. At the same time, the database is queried to obtain historical collaborative operation records of the non-master control robot cluster. Historical information such as the frequency of collaboration, success rate, and tacit understanding between robots is analyzed. Combined with the current relative positional relationships, such as the distance between robots, communication signal strength, and whether they are in the same work area, the collaborative correlation degree of each robot with other robots is calculated. Finally, an adaptive aggregation algorithm is used to dynamically adjust the aggregation weights of the comprehensive performance feature value and the collaborative correlation degree according to the current task requirements, generating the final master control performance index value. This value not only reflects the robot's individual capabilities but also considers its collaborative potential in the cluster.

[0052] In some embodiments, intelligent fusion of performance factors and master control performance evaluation can be achieved in a variety of ways: Optionally, a dynamic fusion approach based on deep reinforcement learning is adopted. A deep Q-network (DQN) is used to learn the optimal fusion strategy under different fire complexity levels. The fire complexity level is encoded as a state, the fusion model selection is used as an action, and the task completion effect is used as a reward. The policy network is obtained through extensive simulation training. In practical applications, a multi-dimensional performance vector containing environmental access factors, fire combat factors, and self-state factors, along with the fire complexity, is input into the policy network. The network output selects a specific fusion model, such as a Transformer model with enhanced attention mechanism. This model learns the correlation between different factors through self-attention mechanism and captures the matching degree between factors and the scene through cross-attention mechanism, outputting a comprehensive performance feature value. At the same time, a graph neural network is used to process the collaborative relationship between robots. The robot is used as a node, and the historical collaboration record and current distance are used as the attributes of the edge. The collaborative embedding vector of each node is calculated through message passing mechanism. The collaborative embedding vector and the comprehensive performance feature value are adaptively fused through a gating mechanism to generate a master control performance index value that considers individual capabilities and collaborative potential. Optionally, an adaptive fusion approach based on Bayesian optimization is adopted to construct a Gaussian process model to model performance under different fusion parameters. Environmental accessibility factors, fire situation control factors, and self-state factors are used as input features. Bayesian optimization is used to search for the optimal fusion weights and nonlinear transformation parameters. The kernel function of the Gaussian process is adjusted according to the fire complexity level; a linear kernel is used for simple scenarios, and an RBF kernel or Matern kernel is used for complex scenarios. The next set of fusion parameters is selected for evaluation using the Expectation Improvement (EI) criterion. Iterative optimization continues until convergence, yielding the optimal fusion model and comprehensive performance feature values ​​for the current scenario. For calculating the collaborative correlation degree, a Hidden Markov Model is used to model the collaborative patterns between robots. State transition probabilities are learned based on historical collaborative operation records. The Viterbi algorithm is used to infer the most likely collaborative state sequence based on the current relative position. The collaborative correlation degree is calculated based on the collaborative states. Finally, an adaptive weighted least squares method is used to dynamically adjust the weight ratio of individual performance and collaborative ability according to the urgency of the task, generating the final master control performance index value. It is understood that other methods can also be used to achieve intelligent fusion and comprehensive evaluation of multi-dimensional performance; this is not limited here.

[0053] In the above embodiments, in complex fire environments, the calculation of collaborative correlation needs to consider the task dependencies and collaborative modes between robots. Specifically, a fire-fighting task dependency graph is constructed, where nodes represent various fire-fighting sub-tasks (such as fire source location, smoke emission, personnel search and rescue, and firefighting operations), and edges represent the dependencies and collaborative requirements between tasks. Based on the functional configuration of each fire-fighting robot (such as reconnaissance, firefighting, and rescue), a robot-task capability matrix is ​​established to quantify the efficiency value of each robot in performing various tasks. When analyzing historical collaborative operation records, not only is the frequency of collaboration counted, but also collaborative mode features are extracted, including the success rate and completion time of master-slave collaboration, parallel collaboration, and relay collaboration. Combined with the current fire situation, the most needed collaborative mode is identified, such as multi-robot ring collaboration for fire source containment and echelon relay collaboration for in-depth firefighting. The path cost and time cost of forming a specific collaborative formation are calculated based on the robot's current position, and a graph convolutional network is used to learn the importance weights of robots in the collaborative network. By dynamically integrating multiple dimensions such as task matching degree, collaborative mode adaptability, and formation cost through an attention mechanism, a collaborative correlation degree reflecting the robot's specific task requirements in the current fire scene is generated, ensuring that the selected new master controller is not only highly capable individually, but also able to effectively coordinate other robots to form an efficient collaborative operation mode.

[0054] In an optional embodiment, the newly appointed master fire-fighting robot acquires associated multimodal perception data collected from the local environment model corresponding to the subordinate robot cluster via a wireless ad hoc network, and generates a new global collaborative operation strategy based on the associated multimodal perception data. Specifically, this includes: acquiring the associated multimodal perception data collected by the newly appointed master fire-fighting robot during the following operations: the newly appointed master fire-fighting robot sends data acquisition requests to each subordinate robot in the subordinate robot cluster via the wireless ad hoc network; the newly appointed master fire-fighting robot receives the associated multimodal perception data returned by each subordinate robot based on the received data acquisition requests; and the newly appointed master fire-fighting robot performs spatiotemporal alignment processing on the associated multimodal perception data. The system obtains spatiotemporally synchronized perception data; the newly appointed master firefighting robot constructs a global fire situation map based on the spatiotemporally synchronized perception data, which includes information on fire source distribution, temperature field distribution, smoke concentration distribution, and obstacle distribution; the newly appointed master firefighting robot determines fire control points and priority control areas from the global fire situation map, and generates a task allocation plan based on the fire control points, priority control areas, and the current operating status information of each subordinate robot; the newly appointed master firefighting robot generates individual operation instructions for each subordinate robot according to the task allocation plan; and obtains a new global collaborative operation strategy generated by the newly appointed master firefighting robot by combining the task allocation plan with the individual operation instructions.

[0055] Among them, the data acquisition request represents the data acquisition instruction sent by the new master controller to the subordinate robot, specifically including parameters such as request identifier, data type, time range, and compression format; spatiotemporal alignment processing refers to the process of unifying data from different time and space reference systems to the same coordinate system; spatiotemporal synchronous sensing data represents the unified data after alignment processing; the global fire situation map is used to represent the comprehensive visualization of the entire fire scene, specifically including a 3D fire scene model, multi-level temperature cloud map, smoke flow vector field, and hazard level distribution map; fire source distribution information represents the location information of the fire ignition point and active combustion area; temperature field distribution information refers to the temperature value distribution at various points in space; smoke... Concentration distribution information indicates the density distribution of smoke in space; obstacle distribution information describes the location of objects affecting passage and operations; fire control points refer to locations that play a key role in controlling the spread of fire, specifically including firebreaks, main passage entrances, and areas near load-bearing structures; priority control areas indicate fire zones that require priority handling; current operation status information indicates the real-time working status of subordinate robots, specifically including task execution progress, equipment load, and position coordinates; task allocation scheme refers to the specific task plan assigned to each robot; individual operation instructions represent detailed control commands for a single robot, specifically including parameters such as movement path, water spray angle, and operation duration.

[0056] Specifically, this step is performed in scenarios where the ability to quickly restore cluster collaborative operation is needed after the establishment of a new master firefighting robot. The new master firefighting robot first broadcasts a data acquisition request via a wireless ad hoc network, requesting each robot in the subordinate robot cluster to report the latest perceived data from its local environmental model. The request explicitly specifies the required data type, format, and timestamp requirements. Each subordinate robot responds to the request, packaging and sending multimodal data, including point clouds, images, temperature, and gas data, to the new master. Upon receiving this associated multimodal perceived data, the new master first performs spatiotemporal alignment processing, including converting data from different robot coordinate systems to a unified global coordinate system, aligning data with different sampling rates through timestamp interpolation, and eliminating data inconsistencies caused by robot movement and sensor latency. Based on the spatiotemporally synchronized perceived data, the new master constructs a global fire situation map, integrating the local observations of all robots through a data fusion algorithm to generate a global environmental representation containing multi-level information such as fire source location, temperature distribution, smoke diffusion, and obstacle distribution. By employing fire spread models and risk assessment algorithms, key fire control points and priority control areas requiring immediate attention are identified from the global fire situation map. Combining the operational status information reported by each subordinate robot, including current task execution status, remaining resources, and location distribution, a task allocation optimization algorithm generates a globally optimal task allocation scheme to ensure timely handling of critical areas and rational resource utilization. Based on the task allocation scheme, individual operational instructions are generated for each subordinate robot, including specific path planning, operational parameter settings, and time scheduling. Finally, the global task allocation scheme and individual operational instructions are integrated into a structured new global collaborative operation strategy, which is then distributed to each subordinate robot for execution via a reliable communication protocol.

[0057] In some embodiments, global awareness fusion and collaborative strategy generation can be achieved in a variety of ways: Optionally, an implementation based on distributed SLAM and multi-objective optimization is adopted. The new master robot sends data acquisition requests containing synchronization clock, data format requirements, and QoS levels to all subordinate robots via a publish / subscribe model. Each robot packages laser point clouds, visual features, IMU data, etc., into ROS message format and returns them. The new master robot uses a distributed SLAM framework to achieve multi-robot map fusion through feature matching and graph optimization, generating a unified 3D raster map. Scalar field information such as temperature field and smoke concentration field is overlaid on the raster map. A GPU-accelerated ray casting algorithm is used to render the global fire situation map, and computational fluid dynamics is used to... The computer-generated dynamics (CFD) simulation predicts the fire spread trend, identifies key nodes in the fire spread as fire control points, uses Voronoi diagrams to divide the responsibility areas of each robot to ensure coverage of all areas that need to be controlled, constructs a multi-objective optimization problem, with objective functions including minimizing task completion time, minimizing resource consumption, and maximizing safety margin, and constraints including robot kinematic limitations, obstacle avoidance requirements, and cooperative timing constraints, and uses genetic algorithms or particle swarm optimization to solve the problem and obtain a task allocation scheme. A specific path is planned for each robot using A or RRT algorithms, generating individual operation instructions containing path point sequences, velocity curves, and operation action sequences. Optionally, a deep learning-based, hierarchical decision-making approach is adopted. The new master controller uses a RESTful API to request structured data in JSON format from subordinate robots, including raw sensor data and preprocessed feature vectors. A Transformer-based multimodal fusion network is used to process the received associated data, achieving spatiotemporal alignment through positional and temporal encoding. An attention mechanism automatically learns the importance weights of different data sources. A 3D convolutional neural network is used to generate a voxel-based global fire situation map from the fused features. Each voxel contains multiple attributes such as temperature, smoke, and combustibles. A trained fire prediction model is used to predict the fire development over a period of time based on the current situation. A reinforcement learning agent is used to identify the optimal fire control point. The agent's reward function considers control effectiveness and resource consumption. A hierarchical decision-making architecture is adopted. The upper layer uses Monte Carlo Tree Search (MCTS) for strategic task allocation to determine which robot is responsible for which area. The lower layer uses Model Predictive Control (MPC) to generate tactical-level individual operation instructions, including real-time trajectory adjustment and operation parameter optimization. The hierarchical decision results are encoded into a new global collaborative operation strategy in Protocol Buffer format and efficiently distributed to each subordinate robot through the gRPC framework. It is understandable that other methods can be used to generate and distribute collaborative work strategies based on global awareness, and no restrictions are imposed here.

[0058] In the above embodiments, the determination of fire control points directly affects the fire extinguishing effect in actual firefighting operations. Specifically, the global fire situation map is first converted into a fire topology map, where nodes represent different spatial areas (such as rooms, corridors, and stairwells), edges represent the connectivity between areas, and edge weights indicate the ease or difficulty of fire spread. An improved betweenness centrality algorithm is used to identify key nodes in the topology; if these nodes are occupied by fire, the fire will rapidly spread to multiple areas. Combined with the building's fire protection facility distribution map, the locations of facilities such as fire doors, sprinkler systems, and fire-resistant roller shutters are identified as potential defense points. The minimum set of nodes required to control and block fire spread is calculated using a minimum cut set algorithm; these nodes are the highest priority fire control points. For each control point, its defense difficulty is analyzed, including temperature level, smoke concentration, obstacle density along the path, and required extinguishing agent dosage. A multi-objective optimization algorithm is used to solve for the optimal allocation scheme of control points under constraints on the number and capabilities of robots, ensuring that high-priority control points receive sufficient defensive support. The generated control points include not only location coordinates, but also recommended defense strategies (such as water curtain barriers, foam coverage, and dry powder suppression), the required robot type and quantity, and the expected control time, providing precise guidance for task allocation.

[0059] It should also be noted that the examples of actual values ​​for the various parameters mentioned above are merely exemplary embodiments, and the examples of actual values ​​for the various parameters are not limited to the examples mentioned above.

[0060] Through the embodiments of this application, the multimodal perception data collected in real time by the fire-fighting robot cluster is used to generate corresponding local environment models for each fire-fighting robot, and the working status of the current master fire-fighting robot is monitored. When an abnormal working status is detected, the scene performance factors in the local environment models of each robot are extracted and the performance is fused to generate the master control performance index value. After that, the new master fire-fighting robot is determined according to the descending order of performance. Then, the new master control collects the associated multimodal perception data of the subordinate robot cluster through a wireless ad hoc network, constructs a global fire situation map after spatiotemporal alignment processing, and determines the fire control points and priority control areas from it. Combined with the current working status information of each subordinate robot, a new global collaborative operation strategy is generated and distributed for execution. When the master control node fails, the best replacement can be quickly elected and the cluster collaborative operation capability can be restored, which comprehensively improves the fault tolerance and task execution reliability of the fire-fighting robot cluster in complex fire environment.

[0061] The multimodal firefighting robot cluster collaborative system in the embodiments of this invention is described below from the perspective of hardware processing. (See attached document.) Figure 2 , Figure 2 This is a schematic diagram of the physical device structure of a multimodal fire-fighting robot cluster collaborative system in the embodiments of this application.

[0062] It should be noted that, Figure 2 The structure of the multimodal firefighting robot cluster collaborative system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0063] like Figure 2 As shown, the multimodal firefighting robot swarm collaborative system includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 202 or a program loaded from storage section 208 into random access memory (RAM) 203, such as performing the methods described in the above embodiments. The RAM 203 also stores... It contains various programs and data required for system operation. CPU 201, ROM 202, and RAM 203 are interconnected via bus 204. Input / output (I / O) interface 205 is also connected to bus 204.

[0064] The following components are connected to I / O interface 205: input section 206 including audio input devices, push-button switches, etc.; output section 207 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 208 including a hard disk, etc.; and communication section 209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 209 performs communication processing via a network such as the Internet. Drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0065] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs the various functions defined in the present invention.

[0066] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0068] Specifically, the multimodal firefighting robot swarm collaboration system of this embodiment includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the multimodal firefighting robot swarm collaboration method provided in the above embodiment.

[0069] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the multimodal fire-fighting robot swarm collaboration system described in the above embodiments; or it may exist independently and not assembled into the multimodal fire-fighting robot swarm collaboration system. The storage medium carries one or more computer programs, which, when executed by a processor of the multimodal fire-fighting robot swarm collaboration system, cause the multimodal fire-fighting robot swarm collaboration system to implement the multimodal fire-fighting robot swarm collaboration method provided in the above embodiments.

[0070] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0071] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for collaborative multimodal firefighting robot swarms, characterized in that, include: The system acquires real-time multimodal perception data collected by the fire-fighting robot cluster and generates corresponding local environment models for each fire-fighting robot in the cluster based on the multimodal perception data. The working status of the current master fire robot in the fire robot cluster is monitored. When the working status of the current master fire robot is detected to be abnormal, the master control performance of the non-master robot cluster in the fire robot cluster other than the current master fire robot is analyzed according to the local environment model, so as to determine the new master fire robot from the non-master robot cluster. The newly appointed master fire-fighting robot acquires associated multimodal perception data collected from the local environment model corresponding to the subordinate robot cluster via a wireless ad hoc network, and generates a new global collaborative operation strategy based on the associated multimodal perception data. The subordinate robot cluster is a set of robots in the non-master robot cluster other than the newly appointed master fire-fighting robot. The new global collaborative operation strategy is distributed to each firefighting robot in the non-master control robot cluster to control each firefighting robot to perform firefighting operations according to the new global collaborative operation strategy.

2. The method according to claim 1, characterized in that, The monitoring of the working status of the current master fire-fighting robot in the fire-fighting robot cluster, and the analysis of the master control performance of the non-master fire-fighting robot cluster (excluding the current master fire-fighting robot) based on the local environment model when an abnormal working status of the current master fire-fighting robot is detected, in order to determine a new master fire-fighting robot from the non-master fire-fighting robot cluster, specifically includes: The system acquires the real-time operating status parameters of the current main control fire robot, and determines that the working status is abnormal if the real-time operating status parameters do not meet the preset parameter conditions. Extract scene performance factors from the local environment models corresponding to each firefighting robot; The scene performance factors are subjected to performance fusion processing to determine the main control performance index values ​​of each fire-fighting robot; The new master firefighting robot is determined from the non-master robot cluster based on the master control performance index value.

3. The method according to claim 2, characterized in that, The step of determining the new master firefighting robot from the non-master robot cluster based on the master control performance index value specifically includes: The firefighting robots are sorted in descending order of performance based on the main control performance index values ​​to obtain a main control candidate list. Select the first-ranked candidate robot in the candidate list of master controllers as the candidate master controller robot; A first master control authority transfer request is sent to the candidate master control robot, and after receiving a confirmation response message from the candidate master control robot, the candidate master control robot is determined as the new master control firefighting robot.

4. The method according to claim 3, characterized in that, After sending the master control transfer request to the candidate master robot, the method further includes: If the candidate master robot does not respond or returns a master control permission rejection message within a preset time period, the preferred candidate robot will be marked as a master control transfer failure state. Select the second-ranked candidate robot from the list of candidate master robots as the new candidate master robot; Real-time multimodal perception data is obtained from the local environment model corresponding to the new candidate master robot, and the current master control performance index value of the new candidate master robot is determined based on the real-time multimodal perception data to determine whether it meets the preset master control conditions. If the current master control performance index value is determined to meet the preset master control conditions, a second master control authority transfer request is sent to the new candidate master control robot. If the request to transfer master control authority to the candidate robot in the master control candidate list fails N times in a row, an emergency collaboration mode is activated. The emergency collaboration mode selects multiple firefighting robots that meet preset collaboration conditions from the master control candidate list as the new master firefighting robot.

5. The method according to claim 2, characterized in that, The scene performance factors extracted from the local environment model corresponding to each firefighting robot specifically include: Extract three-dimensional point cloud data and first thermal imaging data from the local environment model; Spatial accessibility analysis is performed on the three-dimensional point cloud data to obtain spatial accessibility parameters, and temperature field analysis is performed on the first thermal imaging data to obtain thermal barrier hazard parameters. An environmental accessibility factor is generated based on the spatial accessibility parameters and the thermal barrier hazard parameters; Extract the second thermal imaging data and gas concentration data from the local environment model; Fire situation analysis is performed on the second thermal imaging data to obtain fire characteristic parameters, and gas distribution analysis is performed on the gas concentration data to obtain spread risk parameters; Fire control factors are generated based on the fire characteristic parameters and the spread risk parameters; The system acquires the body status data of each firefighting robot and performs state fusion processing on the body status data to obtain resource availability parameters. Generate its own state factors based on the resource availability parameters; The environmental accessibility factor, the fire suppression factor, and the self-state factor are used as the scene performance factors.

6. The method according to claim 5, characterized in that, The process of performing performance fusion processing on the scene performance factors to determine the main control performance index values ​​of each fire-fighting robot specifically includes: Construct a multidimensional performance vector that includes the environmental access factor, the fire resistance factor, and the self-state factor; The fire complexity level is determined based on the fire characteristic parameters, and a corresponding adaptive fusion model is selected from the preset fusion model library based on the fire complexity level. The multidimensional performance vector is input into the adaptive fusion model, and the comprehensive performance feature value output by the adaptive fusion model is obtained. Obtain the historical collaborative operation records and current relative position relationships of the non-master control robot cluster, and determine the collaborative correlation degree of each firefighting robot based on the historical collaborative operation records and the current relative position relationships; The collaborative correlation degree and the comprehensive performance feature value are adaptively aggregated to generate the master control performance index value.

7. The method according to claim 1, characterized in that, The process of acquiring the associated multimodal perception data collected by the newly appointed master fire-fighting robot from the local environment model corresponding to the subordinate robot cluster via a wireless ad hoc network, and generating a new global collaborative operation strategy based on the associated multimodal perception data, specifically includes: The associated multimodal perception data collected by the newly appointed master fire-fighting robot in performing the following operations is obtained: The newly appointed master fire-fighting robot sends data collection requests to each subordinate robot in the subordinate robot cluster through the wireless self-organizing network; The newly appointed master fire-fighting robot receives the associated multimodal perception data returned by each subordinate robot according to the received data acquisition request; The newly appointed main control firefighting robot performs spatiotemporal alignment processing on the associated multimodal perception data to obtain spatiotemporally synchronized perception data; The newly appointed main control firefighting robot constructs a global fire situation map based on the spatiotemporal synchronous perception data. The global fire situation map includes fire source distribution information, temperature field distribution information, smoke concentration distribution information, and obstacle distribution information. The newly appointed master firefighting robot determines the fire control point and priority control area from the global fire situation map, and generates a task allocation scheme based on the fire control point, the priority control area and the current operation status information of each subordinate robot. The newly appointed master fire-fighting robot generates individual operation instructions for each subordinate robot according to the task allocation scheme; The new global collaborative operation strategy is generated by combining the task allocation scheme with the individual operation instructions of the newly appointed master fire-fighting robot.

8. A multimodal firefighting robot swarm collaborative system, characterized in that, The multimodal firefighting robot swarm collaborative system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the multimodal firefighting robot swarm collaborative system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the multimodal firefighting robot cluster collaborative system, the multimodal firefighting robot cluster collaborative system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the multimodal fire-fighting robot cluster collaborative system, the multimodal fire-fighting robot cluster collaborative system performs the method as described in any one of claims 1-7.

Images