A heterogeneous multi-robot intelligent cooperative scheduling system and method based on a large model

Abstract

This invention relates to the field of intelligent robot collaborative scheduling technology, specifically disclosing a heterogeneous multi-robot intelligent collaborative scheduling system and method based on a large model, including a cloud-based intelligent cognitive platform, an edge collaborative scheduling platform, and an autonomous mobile platform. The cloud platform handles large model training, knowledge management, and advanced cognitive functions, constructing an intelligent hub; the edge platform is responsible for task parsing and allocation, establishing a heterogeneous multi-agent system, constructing a closed-loop link for semantic understanding, task decomposition, global allocation, dynamic replanning, and local obstacle avoidance, and realizing the coupling of a large language model with multiple algorithms; the autonomous mobile platform contains at least one heterogeneous robot, performing terminal operations and perception operations. This invention has advantages such as flexible scheduling and efficient collaboration, avoids single points of failure, and effectively solves existing problems such as rigid scheduling architecture and semantic gaps.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot collaborative scheduling technology, specifically to a heterogeneous multi-robot intelligent collaborative scheduling system and method based on a large model. Background Technology

[0002] In recent years, with the rapid development of robotics technology, multi-robot systems have been increasingly widely used in complex industrial scenarios such as substation inspection and factory equipment maintenance. In particular, heterogeneous multi-robot collaborative systems integrating different types of robots, such as drones, quadruped robots, and wheeled robots, have become a research hotspot due to their adaptability to diverse operating environments. However, existing technologies still face a series of profound challenges in constructing such heterogeneous robot collaborative scheduling systems.

[0003] The scheduling architecture suffers from severe rigidity. Traditional centralized control schemes are prone to single-point failure risks, and the computational complexity increases sharply with the scale of robots, making it difficult to support the real-time scheduling needs of large-scale heterogeneous teams. On the other hand, distributed control schemes lack the ability to integrate global information, making it impossible to achieve global optimization of task allocation. At the same time, the task allocation process relies too much on pre-programmed rules, resulting in insufficient adaptability of the system when task requirements or environmental conditions change, and limited scalability.

[0004] Significant semantic gap exists in human-computer interaction. Existing systems only support structured instruction input and cannot directly parse task descriptions in natural language form. High-level task decomposition must be done manually, which not only greatly increases the operational threshold but also severely restricts work efficiency, especially in emergency scenarios.

[0005] The heterogeneous collaboration capability is significantly weak. Most systems are designed for homogeneous robots and fail to effectively coordinate the differences in capabilities between different types of robots. For example, drones have advantages and limitations in high-altitude operations, quadruped robots have limitations in adaptability and speed in complex terrain, and wheeled robots have limited efficiency and obstacle-crossing ability on flat roads. This results in low overall collaboration efficiency and makes it difficult to leverage the comprehensive effectiveness of heterogeneous teams.

[0006] The system suffers from a severe lack of dynamic adaptability. When encountering sudden environmental changes such as the appearance of obstacles or worsening weather, or abnormal robot states such as sudden power loss or sensor failure, the system lacks an autonomous planning mechanism and relies heavily on human intervention to resume task execution, fundamentally limiting its autonomous operation capabilities. Although large language model technology offers a potential solution to these problems, current research is mostly limited to instruction conversion in single-robot scenarios, failing to construct a systematic architecture that supports collaborative scheduling of heterogeneous multi-robot systems. Some improved solutions only use large language models as surface-level instruction interfaces, with the core scheduling logic still adhering to traditional rule frameworks. This fails to overcome core bottlenecks such as rigid scheduling and insufficient semantic understanding, resulting in the system's intelligence level always being limited by preset task flows and equipment combinations, unable to meet the urgent needs of practical applications for flexibility and adaptability. Summary of the Invention

[0007] The purpose of this invention is to provide a heterogeneous multi-robot intelligent collaborative scheduling system and method based on a large model, which can flexibly schedule, efficiently coordinate, have strong dynamic adaptability, avoid single points of failure, and effectively solve problems such as rigid scheduling architecture, semantic gap in human-computer interaction, weak heterogeneous collaborative capability, and lack of dynamic adaptability.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a heterogeneous multi-robot intelligent collaborative scheduling system based on a large model, comprising: Cloud-based intelligent cognitive platform, edge collaborative scheduling platform, and autonomous mobile platform; A cloud-based intelligent cognitive platform is used to perform large model training, knowledge management, and advanced cognitive functions, building an intelligent hub. The edge collaborative scheduling platform is used to perform task parsing, allocation, scheduling and coordination control, and establish a heterogeneous multi-agent assistance system. The edge collaborative scheduling platform constructs a closed-loop collaborative link of semantic understanding, task decomposition, global allocation, dynamic replanning and local obstacle avoidance, and realizes the coupling of large language model with hierarchical task network, auction algorithm, Monte Carlo tree search and multi-agent reinforcement learning. An autonomous mobile platform, including at least one heterogeneous robot, is used to perform tasks and sensing operations at a task terminal.

[0009] Furthermore, the cloud-based intelligent cognitive platform adopts a multimodal Transformer architecture, which includes: The input layer is used to acquire multi-source information, including visible light images, infrared thermal imaging, LiDAR point clouds, robot status data, and natural language commands. The feature extraction and alignment layer is used to extract visual features using a visual Transformer architecture and achieve synchronous fusion of multi-source data through spatiotemporal alignment and cross-modal attention mechanisms. The knowledge-enhanced fusion layer is used to inject a device knowledge graph containing equipment ledgers, historical defects, and physical mechanisms into the Transformer unified representation space to achieve semantic alignment. The output layer adopts a dual-path parallel structure. One path is based on DS evidence theory to perform multimodal information fusion, and the other path is based on a combination of hierarchical task network and Monte Carlo tree search to output task planning scheme.

[0010] This invention also proposes an edge collaborative scheduling platform, comprising: The natural language instruction parsing module is used to perform semantic parsing of natural language instructions using a large language model and extract task constraints. The hierarchical task decomposition module is used to recursively decompose high-level tasks into atomic subtasks using a hierarchical task network. Each atomic subtask includes task type, execution window, spatial location, resource requirements, risk level, and dependencies. The intelligent task allocation module is used to generate bidding costs based on the robot's capability vector, remaining power, current position, load status and link quality, and calculate the globally optimal task allocation result based on the auction algorithm. The collaborative execution control module is used to achieve distributed collaborative obstacle avoidance and path optimization by employing multi-agent reinforcement learning. The dynamic replanning module is used to perform real-time dynamic task replanning using Monte Carlo tree search.

[0011] Furthermore, in the closed-loop collaborative link: Large language models are used to perform semantic parsing, constraint completion, and conflict interpretation, and to build an interpretable scheduling front-end; The hierarchical task network is coupled with the auction algorithm, and the task structure constraints are explicitly injected into the bidding cost function; Monte Carlo tree search and multi-agent reinforcement learning work together; the former performs global sequence optimization, while the latter performs local continuous control. Algorithm switching is achieved through a unified state machine, avoiding system interruption caused by the failure of a single algorithm.

[0012] Furthermore, autonomous mobile platforms include: Drones are used for high-level equipment inspection, rapid cross-regional verification, initial screening of thermal anomalies, and relay communication. Quadruped robots are used to perform inspection tasks on steps, narrow passages, complex ground environments, and areas with weak structures. Wheeled robots are used to perform high-frequency routine inspections of main roads and to load materials.

[0013] Furthermore, the workflow of an autonomous mobile platform includes: Environmental and status data are collected through multiple source sensors, including LiDAR, RGB camera, infrared thermal imager, IMU, wheel speed meter, and GNSS. Real-time semantic segmentation is performed using object detection and visual language models, and a semantic map is constructed by combining geometric information. A factor graph multi-sensor tight coupling method is used for localization to achieve semantically constrained relocalization. Path planning is performed based on semantic tags, and semantic navigation is executed. Terrain-adaptive gait switching is achieved through model predictive control or reinforcement learning controllers to complete robot motion control.

[0014] Furthermore, the cloud-based intelligent cognitive platform also includes a defect template library, which includes: Defect templates for substation equipment include hot spots, discharge traces, leaks, abnormal noises, and abnormal vibrations; Defect templates for factory equipment include bearing deterioration, belt misalignment, valve malfunction, and abnormal cabinet temperature rise.

[0015] Furthermore, the edge collaborative scheduling platform is also used to dynamically switch the robot's movement mode based on terrain labels, including wheeled robot detours, quadruped robot traversal, and drone take-off for retesting. In response to rain, fog, backlight at night, and electromagnetic interference environments, the cloud-based intelligent cognitive platform also introduces a perception confidence weighting and redundancy verification mechanism.

[0016] Furthermore, the triggering conditions for the dynamic replanning module include: robot disconnection, battery level below the threshold, critical sensor failure, task timeout, appearance of dynamic obstacles, update of restricted areas, sudden weather changes, and deterioration of communication quality.

[0017] Furthermore, the dynamic replanning module is also used to perform hierarchical replanning: For minor anomalies, perform local replanning, adjusting only the affected tasks and local paths; For moderate anomalies, regional replanning is performed, tasks in the corresponding regions are reassigned, and security redundancy is improved. In response to severe anomalies, a global replanning was implemented, non-critical tasks were suspended, and priority was given to reviewing and safely evacuating high-risk equipment.

[0018] Furthermore, the system also includes an automatic degradation protection strategy, which includes: When the large model is unavailable, switch to the rule base and lightweight model while retaining basic scheduling capabilities; When the main sensor fails, switch to a redundant sensor and reduce the operating speed; When the cloud-edge link is interrupted, the edge collaborative scheduling platform takes over task scheduling, and the autonomous mobile platform executes local security policies. When the robot experiences consecutive failures, task migration and nearby takeover are triggered.

[0019] Furthermore, the system also includes a security closed-loop and recovery mechanism, including: Each replanning process records the triggering reason, strategy version, execution result, and rollback point; After the anomaly is resolved, the original task plan will be gradually restored in the order of first local and then global. An online experience base is created for frequently occurring anomalies, and the threshold and cost function parameters are continuously optimized.

[0020] The present invention further proposes a heterogeneous multi-robot intelligent cooperative scheduling method based on a large model, comprising: Receive natural language commands; Semantic analysis of natural language instructions is performed through a cloud-based intelligent cognitive platform to extract task constraints; The edge collaborative scheduling platform uses a hierarchical task network to recursively decompose high-level tasks into atomic subtasks. The atomic subtasks are allocated to the corresponding heterogeneous robots based on the auction algorithm. During the allocation process, the bidding cost is generated based on the robot's capability vector, remaining power, current position, load status and link quality. Control heterogeneous robots to perform assigned tasks and achieve distributed cooperative obstacle avoidance through multi-agent reinforcement learning; Real-time monitoring of execution status; when replanning conditions are triggered, hierarchical dynamic task replanning is performed through Monte Carlo tree search. Once the task is completed, an inspection report will be generated and feedback will be provided.

[0021] Compared with existing technologies, this invention provides a heterogeneous multi-robot intelligent collaborative scheduling system and method based on a large model. By constructing an intelligent hub and closed-loop collaborative link through a cloud-based intelligent cognitive platform, an edge collaborative scheduling platform, and an autonomous mobile platform, it realizes the coupling of a large language model with multiple algorithms. This effectively solves problems such as rigid scheduling architecture, semantic gap in human-computer interaction, weak heterogeneous collaborative capabilities, and lack of dynamic adaptability. It can flexibly schedule, efficiently collaborate, dynamically adapt, and avoid single points of failure. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a heterogeneous multi-robot intelligent collaborative scheduling system based on a large model, provided as an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of a heterogeneous multi-robot intelligent collaborative scheduling method based on a large model, provided as an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of this invention. The components of this invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0026] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] Existing multi-robot scheduling systems suffer from problems such as rigid scheduling architecture, semantic gaps in human-robot interaction, insufficient heterogeneous collaboration capabilities, and poor dynamic adaptability when facing complex scenarios. Current applications of large language models in multi-robot collaborative scheduling primarily focus on single-robot instruction translation, failing to form a systematic architecture. This results in collaborative capabilities being limited by preset task and device combinations, leading to insufficient intelligence.

[0028] like Figure 1 As shown, this embodiment provides a heterogeneous multi-robot intelligent collaborative scheduling system based on a large model. Its architecture is designed to improve the intelligence level and collaborative efficiency of multi-robot systems. The system mainly consists of a cloud-based intelligent cognitive platform, an edge collaborative scheduling platform, and an autonomous mobile platform.

[0029] In one embodiment, the cloud-based intelligent cognitive platform is configured to perform large-scale model training, knowledge management, and advanced cognitive functions, and to build the intelligent hub of the system. For example, large-scale model training can be performed using a distributed computing cluster, with model parameters repeatedly optimized to improve performance; knowledge management can be performed using a database or document management system with a specific structure to store and retrieve relevant operating procedures, device parameters, and other information; advanced cognitive functions can be implemented by an expert system or a rule-based inference engine to make initial judgments on situations.

[0030] In one embodiment, the edge collaborative scheduling platform is configured to perform task parsing, allocation, scheduling, and coordination control to establish a heterogeneous multi-agent collaborative system. For example, task parsing can use keyword matching or template filling to convert user instructions into a task description that the system can recognize; task allocation can use pre-set priority rules or basic cyclic allocation strategies; scheduling and coordination control can be based on time slice rotation or predetermined path planning to avoid conflicts between robots.

[0031] In one embodiment, the edge collaborative scheduling platform also constructs a closed-loop collaborative link encompassing semantic understanding, task decomposition, global allocation, dynamic replanning, and local obstacle avoidance, and couples a large language model with a hierarchical task network, auction algorithms, Monte Carlo tree search, and multi-agent reinforcement learning. Specifically, semantic understanding can employ dictionary-based or grammatical rule-based methods to perform preliminary analysis of natural language instructions; task decomposition can be achieved by human operators breaking down high-level tasks into sub-tasks based on experience; global allocation can utilize local optimum algorithms or traversal search (for small-scale tasks); dynamic replanning can be triggered by human intervention or pre-set contingency plans; and local obstacle avoidance can employ basic obstacle avoidance strategies based on distance sensors, such as stopping or turning when an obstacle is detected. These functional modules can operate independently and exchange data via message queues.

[0032] In one embodiment, the autonomous mobile platform includes at least one heterogeneous robot for performing tasks and sensing operations at the task terminal. For example, the heterogeneous robot may include various types of wheeled robots, each equipped with a camera or temperature sensor, for performing pre-set route inspection tasks and collecting environmental data. These robots can operate according to a predetermined schedule or route and upload the collected data to a central server for processing.

[0033] In some possible embodiments, a cloud-based intelligent cognitive platform is proposed to employ a multimodal Transformer architecture. This architecture can effectively process and fuse data from different modalities and inject domain knowledge, thereby improving the system's cognitive capabilities and decision-making quality. The multimodal Transformer architecture includes an input layer, a feature extraction and alignment layer, a knowledge enhancement and fusion layer, and an output layer.

[0034] Specifically, the input layer is used to acquire multi-source information, including visible light images, infrared thermal imaging, LiDAR point clouds, robot state data, and natural language commands. Visible light images provide rich texture and color information for object and scene recognition; infrared thermal imaging captures temperature distribution, which is crucial for detecting abnormal heat points in equipment; LiDAR point clouds provide accurate 3D geometric structure information for environmental modeling and obstacle detection; robot state data includes the robot's position, posture, battery level, and load, reflecting the robot's real-time operating status; and natural language commands convey the user's task intent and high-level planning. This multi-source information undergoes preliminary preprocessing at the input layer, such as image normalization, point cloud filtering, and text segmentation, to adapt to the input requirements of the Transformer model.

[0035] The feature extraction and alignment layer is used to extract visual features using a visual Transformer architecture and achieves synchronous fusion of multi-source data through spatiotemporal alignment and cross-modal attention mechanisms. The visual Transformer architecture can effectively extract high-level semantic features from image data, capturing key information within the image. The spatiotemporal alignment mechanism ensures that data collected by different sensors at different times can be accurately correlated, for example, through timestamp synchronization or motion compensation. The cross-modal attention mechanism allows the model to pay attention to related information in other modalities while processing data from one modality, thereby achieving deep interaction and fusion of features from different modalities. For example, when analyzing visible light images, the model can simultaneously refer to depth information provided by LiDAR point clouds to more accurately understand the scene.

[0036] The knowledge-enhanced fusion layer injects a device knowledge graph, containing device records, historical defects, and physical mechanisms, into the Transformer's unified representation space, achieving semantic alignment. A device knowledge graph is a structured knowledge representation that organizes relationships between devices, device attributes, historical failure modes, and related physical principles in a graph form. By injecting this domain knowledge into the Transformer's unified representation space, the model can leverage this prior knowledge for reasoning and judgment during feature learning and information fusion. For example, when a specific visual feature of a device is detected, combining the device's record information and historical defect records in the knowledge graph can more accurately determine its health status or potential failures. Semantic alignment ensures consistency between the model's internal feature representations and the concepts in the external knowledge graph, thereby enabling effective knowledge utilization.

[0037] The output layer employs a dual-path parallel structure. One path is based on DS evidence theory for multimodal information fusion. DS evidence theory is a mathematical framework for handling uncertainty and conflicting information. It can weight and fuse perception results from different modalities to obtain a more reliable and robust comprehensive judgment. For example, when there is uncertainty or slight conflict in the judgment of a device malfunction based on visible light images, infrared thermal imaging, and robot state data, DS evidence theory can provide a fusion result with higher confidence. The other path outputs a task planning scheme based on a combination of hierarchical task networks and Monte Carlo tree search. The hierarchical task network decomposes complex tasks into a series of executable subtasks, forming a hierarchical task structure; Monte Carlo tree search is a heuristic search algorithm used to find the optimal task execution sequence in a complex decision space. Through the combination of the two, the system can generate a refined task planning scheme that meets both high-level task objectives and considers low-level execution details.

[0038] In some possible embodiments, an edge collaborative scheduling platform is proposed, comprising: a natural language instruction parsing module, a hierarchical task decomposition module, an intelligent task allocation module, a collaborative execution control module, and a dynamic replanning module.

[0039] The natural language instruction parsing module aims to transform user-input natural language instructions into structured task information that the system can understand and execute. Specifically, it employs a large language model to perform deep semantic parsing on unstructured natural language instructions, identifying and extracting key constraints such as task objectives, execution conditions, time windows, spatial scope, and resource limitations. For example, when the user instruction is "Please inspect all equipment in area A, and prioritize checking temperature anomalies," the module can parse "inspect" as the task type, "all equipment in area A" as the spatial scope and target object, and "prioritize checking temperature anomalies" as the priority constraint. In this process, the large language model can leverage its powerful language understanding and reasoning capabilities to handle ambiguity and vagueness in the instructions, and perform necessary common-sense reasoning, thereby generating a clear and complete set of task constraints, providing accurate input for subsequent task decomposition and scheduling.

[0040] The hierarchical task decomposition module is responsible for recursively decomposing the high-level task output by the natural language instruction parsing module into a series of atomic subtasks that can be directly executed by the robot through a hierarchical task network. The hierarchical task network is a planning method based on domain knowledge, which progressively refines complex tasks through predefined task decomposition rules (methods). Each atomic subtask is the smallest indivisible execution unit, detailing its task type, execution window, spatial location, resource requirements, risk level, and dependencies on other subtasks. This decomposition method ensures appropriate task granularity, satisfying both the overall goals of the high-level task and adapting to the specific execution capabilities of the underlying robot, while providing fine-grained operational units for subsequent task allocation and scheduling.

[0041] The core function of the intelligent task allocation module is to efficiently allocate the decomposed atomic subtasks to the most suitable heterogeneous robots based on the current system state and robot capabilities. It employs an auction algorithm for task allocation, simulating a market bidding mechanism where each robot competes for tasks based on its own "bid price." The generation of the bid price comprehensively considers multiple dimensions: robot capability vector, remaining battery power, current position, load status, and link quality between the robot and the task point. By calculating the bid price for each robot for each task and employing a global optimization strategy, this module can determine a globally optimal task allocation result, maximizing overall system efficiency or minimizing total cost while avoiding task conflicts and resource waste.

[0042] The collaborative execution control module focuses on the task execution phase, ensuring that multiple heterogeneous robots work together safely and efficiently in a shared environment. It employs a multi-agent reinforcement learning approach, enabling robots to learn optimal distributed collaborative obstacle avoidance and path optimization strategies through interaction with the environment. Each robot is treated as an agent, perceiving the state of its surroundings (including other robots and obstacles) and selecting actions (such as adjusting speed and direction) based on learned strategies to avoid collisions and optimize its own path. The multi-agent reinforcement learning model can be trained offline or fine-tuned online during actual operation, allowing it to adapt to dynamically changing and complex environments. This distributed control approach reduces reliance on central control, improves system robustness and response speed, and is particularly suitable for large-scale, highly dynamic robot swarms.

[0043] The dynamic replanning module is responsible for adjusting and optimizing task planning in real time during task execution when unexpected events or environmental changes occur. It employs the Monte Carlo Tree Search algorithm to perform real-time dynamic task replanning. Monte Carlo Tree Search explores possible future states and action sequences through extensive simulations and evaluations in the search space, thereby finding near-optimal decisions in uncertain environments. When the robot encounters sudden obstacles, changes in task priorities, robot malfunctions, or environmental changes, this module can quickly activate, re-evaluate the task sequence and path based on the current system state and remaining tasks, and generate new planning schemes adapted to the current situation. This real-time replanning capability significantly enhances the system's adaptability and robustness, ensuring successful task completion even in highly dynamic and unpredictable operating environments.

[0044] In some possible embodiments, a large language model is proposed to perform semantic parsing, constraint completion, and conflict interpretation in a closed-loop collaborative link, constructing an interpretable scheduling front end; a hierarchical task network is coupled with an auction algorithm to explicitly inject task structure constraints into the bidding cost function; Monte Carlo tree search and multi-agent reinforcement learning are coordinated, with the former performing global sequence optimization and the latter performing local continuous control; and algorithm switching is achieved through a unified state machine to avoid system interruption due to the failure of a single algorithm.

[0045] Specifically, the large language model plays the role of an intelligent front-end in the closed-loop collaborative chain. Its main functions include semantic parsing of natural language instructions. This means the system can understand complex, unstructured user input and accurately extract core tasks, objectives, and implicit constraints. For example, a user might input, "Please have the robot check all equipment in workshop number three, prioritizing areas with anomalies." The large language model can parse this as "checking equipment in workshop number three" being the core task and "prioritizing areas with anomalies" being the priority constraint. Furthermore, the large language model can perform constraint completion, automatically supplementing necessary but undefined constraints in user instructions based on context and domain knowledge, ensuring the completeness of task definitions. For example, if the user only says "check equipment," the large language model might automatically complete "check frequency," "check items," etc., based on the equipment type. Moreover, when task conflicts or resource contention occur during scheduling, the large language model can interpret these conflicts, providing users or operators with the reasons for the conflict, the resources involved, and possible solutions, thereby building an interpretable scheduling front-end and enhancing system transparency and user trust. This can be achieved by fine-tuning a pre-trained large language model with domain knowledge, or by guiding a general large language model to complete these tasks through carefully designed prompt engineering.

[0046] The coupling of hierarchical task networks with auction algorithms aims to integrate the structured constraints of tasks more deeply into resource allocation decisions. Hierarchical task networks decompose high-level tasks into a series of atomic subtasks with dependencies, which constitute the inherent structure of the tasks. In traditional auction algorithms, robots typically bid on individual tasks based on their capabilities and current state. Through this coupling mechanism, when calculating the bidding cost, not only the direct cost (such as time and energy) of the robot executing the current atomic subtask is considered, but the position of the atomic subtask in the entire task network, its impact on subsequent tasks, and its dependencies are explicitly injected as weights or penalties into the bidding cost function. For example, if completing an atomic subtask unlocks multiple high-priority subsequent tasks, the robot's bidding cost may be appropriately reduced; conversely, if completing the task hinders the execution of other critical tasks, its bidding cost will be increased accordingly. This approach ensures that the task allocation result is not only locally optimal but also better meets the logical and efficiency requirements of the overall task flow.

[0047] The synergy between Monte Carlo Tree Search (MCTS) and Multi-Agent Reinforcement Learning (MARL) enables an effective combination of global planning and local control. MCTS, a powerful decision-making algorithm, performs global sequence optimization. By simulating and evaluating a large number of possible task execution sequences, it explores the future state space, generating a long-term optimal or suboptimal macroscopic task execution plan for a swarm of heterogeneous robots, including the task order, approximate path, and resource allocation strategy. This global plan provides strategic guidance for the robots. Building upon this, Multi-Agent Reinforcement Learning (MARL) performs local continuous control. MARL agents (i.e., individual robots) learn and execute fine-grained actions in a real-time, dynamic environment based on the global plan provided by MCTS. This includes, but is not limited to, distributed cooperative obstacle avoidance, dynamic obstacle avoidance, local path optimization, and adaptive responses to environmental changes. Through continuous interaction with the environment and reward feedback, MARL enables robots to autonomously adjust their motion strategies, ensuring smooth, efficient, and safe motion control while adhering to the global plan and flexibly responding to unexpected local situations.

[0048] In one embodiment, the unified state machine defines multiple states of the system under different operational phases and abnormal conditions (e.g., initialization, task planning, task execution, exception handling, replanning, etc.). Each state corresponds to a specific combination of algorithms or processing logic. When the system detects that an algorithm (e.g., large language model parsing failure, auction algorithm non-convergence, MCTS planning timeout) has failed, its performance has degraded, or it cannot meet the current task requirements, the state machine can automatically trigger state transitions according to preset rules and conditions, and switch to a backup algorithm, a degradation strategy, or an exception handling process. For example, if MCTS fails to generate a valid plan within a specified time, the state machine may switch to using a simpler heuristic algorithm for local replanning, or trigger manual intervention. This mechanism ensures that even if a core algorithm fails, the system can maintain basic functionality by switching to other algorithms or strategies, avoiding a complete system crash due to the failure of a single algorithm, thereby significantly improving the system's fault tolerance and continuous operational stability.

[0049] In some possible embodiments, autonomous mobile platforms are proposed, including drones, quadruped robots, and wheeled robots. Drones are robots capable of aerial flight, characterized by high mobility, wide field of vision, and high speed, enabling them to quickly reach high altitudes or inaccessible areas. In practical applications, drones can be equipped with sensors such as high-definition cameras and infrared thermal imagers for inspecting high-altitude equipment, such as high-voltage power lines, high-rise building structures, or the external condition of large storage tanks. Their rapid verification capability allows for quick secondary confirmation after anomalies are detected, improving response speed. Furthermore, drones can also serve as relay communication nodes, providing temporary network coverage in areas with weak communication signals, ensuring uninterrupted communication between other robots or systems and cloud-based intelligent cognitive platforms and edge collaborative scheduling platforms. Quadruped robots mimic animal walking patterns, possessing excellent obstacle-crossing capabilities and terrain adaptability. Their multi-jointed leg structure allows them to move and operate stably in complex ground environments such as uneven ground, steps, narrow passages, and areas with weak structures (such as construction sites and abandoned factories), including gravel and mud. These robots are typically equipped with sensors such as LiDAR and depth cameras, enabling them to perform sophisticated environmental perception and local path planning. They are suitable for inspection tasks requiring close contact with or traversing obstacles, such as inspecting equipment bases, checking for pipe leaks, or detecting hazardous areas. Wheeled robots are known for their high speed and large load capacity, making them particularly suitable for flat, open terrain. Their relatively simple structure and low energy consumption make them suitable for long-duration, high-frequency routine inspection tasks, such as monitoring equipment status and collecting environmental parameters in factory workshops, warehouses, or main roads of industrial parks. Furthermore, wheeled robots usually have a large payload capacity and can be used to carry various materials, such as spare parts, tools, or additional sensor modules, facilitating material transportation and auxiliary operations, thereby improving overall operational efficiency.

[0050] In one embodiment, the autonomous mobile platform includes various heterogeneous robots for performing diverse tasks and sensing operations. However, in real-world complex environments, ensuring that these heterogeneous robots can efficiently and accurately perceive their environment, achieve high-precision positioning, perform intelligent path planning, and complete refined motion control to cope with changing terrain and task requirements is a key challenge in realizing their intelligent collaborative scheduling.

[0051] In some possible embodiments, a workflow for an autonomous mobile platform is proposed. This workflow involves acquiring environmental and state data through multiple source sensors, including LiDAR, RGB cameras, infrared thermal imagers, IMU, wheel speedometers, and GNSS; performing real-time semantic segmentation through target detection and a visual language model, and constructing a semantic map by combining geometric information; using factor graph multi-sensor tight coupling for localization to achieve semantically constrained relocalization; performing path planning based on semantic labels to execute semantic navigation; and achieving terrain-adaptive gait switching through model predictive control or reinforcement learning controllers to complete robot motion control.

[0052] Specifically, multi-source sensors are fundamental to a robot's environmental perception. By integrating different types of sensors, more comprehensive and robust environmental information and the robot's own state data can be acquired. LiDAR provides high-precision 3D point cloud data for environmental modeling and obstacle detection; RGB cameras provide rich texture and color information for target recognition and semantic understanding; infrared thermal imagers are used to detect temperature anomalies, especially suitable for hotspot identification in equipment inspection; IMU (Inertial Measurement Unit) provides the robot's attitude, angular velocity, and acceleration information; wheel speedometers provide odometer readings for wheeled robots; and GNSS (Global Navigation Satellite System) provides the robot's global position information. These sensors can be physically mounted on different heterogeneous robots on an autonomous mobile platform and transmit data to an edge collaborative scheduling platform or the onboard computing unit of the autonomous mobile platform for preprocessing and fusion via a unified data interface or bus.

[0053] In one embodiment, real-time semantic segmentation is performed using object detection and a visual language model, and a semantic map is constructed by combining geometric information. Real-time semantic segmentation is the process of identifying different object categories in the environment and classifying them at the pixel level. Combining this with a visual language model allows for the understanding of more complex scene semantics. By combining semantic information with geometric information acquired by sensors (such as LiDAR point clouds or depth maps), a semantic map containing information such as object category, location, and shape can be constructed, providing a higher level of environmental understanding for subsequent path planning and navigation. For example, a deep learning-based object detection model and a visual language model can be used to perform real-time inference on RGB images to identify different semantic regions such as roads, walls, devices, and obstacles. Simultaneously, the geometric boundaries and 3D positions of these semantic regions are obtained using LiDAR point cloud or depth camera data. These semantic labels and geometric information are then fused, for example, by overlaying semantic labels onto an occupancy grid map or point cloud map, to form a queryable and understandable semantic map.

[0054] To achieve high-precision positioning, this embodiment employs factor graph multi-sensor tight coupling for positioning, realizing semantic constraint relocalization. Factor graph is a graph optimization method that models measurements from different sensors (such as LiDAR, IMU, vision, GNSS) and robot state variables as nodes and edges in a graph. By optimizing the entire graph, high-precision robot pose estimation is obtained. Semantic constraint relocalization refers to using known semantic information (such as specific equipment, walls, roads, etc.) in the semantic map as additional constraints during the localization process, improving the robustness and accuracy of relocalization, especially in scenarios where GPS signals are lost or environmental features are sparse. For example, the robot can run a factor graph-based SLAM (Simultaneous Localization and Mapping) algorithm, adding LiDAR odometry, visual odometry, IMU pre-integration, GNSS measurements, and semantic feature matching results extracted from the semantic map as factors to the graph. When the robot enters an area with a constructed semantic map, semantic constraints are generated by matching the currently perceived semantic features with corresponding features in the semantic map, and these constraints are added as new factors to the factor graph for optimization, thereby achieving more accurate relocalization.

[0055] In one embodiment, path planning is performed based on semantic tags, executing semantic navigation. Semantic tags provide a high-level understanding of the environment, enabling path planning to go beyond just geometric obstacles and consider more complex semantic information, such as "through a corridor," "avoiding dangerous areas," and "heading to the inspection point of equipment A." Semantic navigation refers to the robot's ability to understand and follow these semantic instructions, improving the intelligence and efficiency of navigation. For example, on a constructed semantic map, the path planning algorithm can utilize semantic tags to generate smarter paths. The planner can prioritize "road" areas, avoiding "restricted areas" or "dangerous equipment" areas. When receiving the instruction "heading to the substation control room," the navigation system can query the semantic map, find the semantic tags and location of the control room, and plan a path through semantic areas such as "corridor" and "passage."

[0056] In one embodiment, terrain-adaptive gait switching is achieved through model predictive control (MPC) or a reinforcement learning controller to complete the robot's motion control. MPC is an advanced control strategy that uses a system model to predict future behavior and optimizes control inputs at each time step to meet constraints and objectives. Reinforcement learning controllers, on the other hand, learn the optimal control strategy through interaction with the environment. Terrain-adaptive gait switching is crucial for multi-legged robots or robots with multiple motion modes, allowing them to select the most suitable movement mode on different terrains (such as flat ground, steps, and rough surfaces) to improve motion efficiency, stability, and maneuverability. For example, for quadruped robots, multiple gait modes can be preset, and the appropriate gait can be selected based on terrain information perceived by sensors. The model predictive controller can calculate the optimal joint torque or velocity commands in real time based on the current terrain and the robot's dynamics model to maintain robot stability and task execution efficiency.

[0057] In some possible embodiments, the cloud-based intelligent cognitive platform is proposed to also include a defect template library. This defect template library, as an important component of the cloud-based intelligent cognitive platform, is a structured knowledge storage unit specifically used to store and manage predefined patterns and characteristics of various equipment defects. Its role is to provide the cloud-based intelligent cognitive platform with targeted defect identification knowledge, enabling the system to quickly and accurately match and identify specific equipment fault types from massive amounts of sensor data, thereby elevating general anomaly detection capabilities to precise defect diagnosis capabilities.

[0058] Specifically, the defect template library includes defect templates for substation equipment. These templates are customized based on the operating characteristics and common fault modes of substation equipment, including specific defect types such as hot spots, discharge traces, leaks, abnormal noises, and abnormal vibrations. For example, the "hot spot" template may define temperature thresholds, thermal imaging characteristics, or temperature change trends for specific equipment parts; the "discharge trace" template may include pattern recognition features of traces such as arc burning and insulation breakdown in visual images; the "leak" template may involve visual or chemical sensor data features such as liquid flow and oil stain diffusion; the "abnormal noise" template uses acoustic sensor data to identify abnormal frequencies, volumes, or periodic noise patterns; and the "abnormal vibration" template analyzes the vibration frequency, amplitude, or waveform changes of the equipment through vibration sensor data to determine whether a mechanical fault exists.

[0059] In one embodiment, the defect template library also includes defect templates for factory equipment. These templates are designed specifically for the characteristics of factory production lines and mechanical equipment, and include specific defect types such as bearing deterioration, belt misalignment, valve malfunction, and cabinet temperature rise. Specifically, the "bearing deterioration" template may identify characteristic frequencies such as bearing wear and ball bearing damage based on vibration spectrum analysis; the "belt misalignment" template may detect belt position deviation or wear during transmission through visual image analysis; the "valve malfunction" template may combine pressure and flow sensor data or acoustic characteristics to determine whether a valve is stuck, leaking, or not properly switched; and the "cabinet temperature rise" template monitors whether the internal temperature of electrical cabinets, control cabinets, etc., exceeds the safe range through infrared thermal imaging or temperature sensor data to warn of overload or poor heat dissipation problems.

[0060] In some possible embodiments, the edge collaborative scheduling platform is proposed to dynamically switch the robot's motion mode based on terrain labels, including wheeled robot circling, quadruped robot crossing, and drone take-off for retesting; for rain, fog, nighttime backlight, and electromagnetic interference environments, the cloud-based intelligent cognitive platform is also used to introduce a perception confidence weighting and redundancy verification mechanism.

[0061] The edge collaborative scheduling platform dynamically switches the robot's movement mode based on terrain tags, aiming to enable heterogeneous robots to fully leverage their respective strengths and flexibly cope with different terrain challenges. Specifically, terrain tags are obtained by acquiring environmental data in real time through multi-source sensors such as LiDAR and RGB cameras mounted on the autonomous mobile platform, and combining this data with object detection and visual language models for semantic segmentation to construct a semantic map containing terrain information. When the edge collaborative scheduling platform receives the terrain tag information for the current area, it intelligently determines the suitability of the current terrain for different types of robots based on preset robot capability models and terrain adaptability rules. For example, if a wheeled robot encounters a rugged surface or steps unsuitable for its passage, the edge collaborative scheduling platform will plan an alternative route to avoid the terrain. If a quadruped robot encounters steps or small obstacles, it will be instructed to switch to a straddling or climbing gait to overcome the obstacle. When ground robots (such as wheeled or quadruped robots) encounter areas that are impassable or uncertain, the edge collaborative scheduling platform can dispatch drones to conduct aerial reconnaissance, path verification, or provide relay communication to ensure mission continuity.

[0062] In one embodiment, to address adverse environments such as rain, fog, nighttime backlighting, and electromagnetic interference, the cloud-based intelligent cognitive platform introduces a perception confidence weighting and redundancy verification mechanism to improve the reliability of perception data and the accuracy of decision-making in complex environments. Specifically, the cloud-based intelligent cognitive platform receives multi-source sensor data from the autonomous mobile platform in real time and assesses the perception confidence of each sensor data based on current environmental conditions (such as data obtained through environmental sensors or weather forecasts). For example, in rainy or foggy weather, the confidence of visible light images decreases, while the confidence of infrared thermal imaging may be relatively high. When performing multimodal information fusion, the cloud-based intelligent cognitive platform weights different sensor data or their extracted features based on the assessed confidence, giving higher-confidence information a larger proportion in the fusion result, thereby reducing the negative impact of low-confidence data on overall perception. Simultaneously, the redundancy verification mechanism further ensures perception accuracy through multi-source information cross-validation and multi-robot collaborative verification. When the confidence of a certain sensor data or its processing result is low, the system actively uses other sensor data for comparison and correction, such as combining visual information and inertial measurement unit data. In addition, the edge collaborative scheduling platform can also schedule other robots to verify the information from different angles or using different sensors to confirm the accuracy of the information, thereby effectively filtering out noise and correcting errors.

[0063] In some possible embodiments, the triggering conditions for the dynamic replanning module are proposed to include: robot disconnection, battery level below a threshold, critical sensor failure, task timeout, appearance of dynamic obstacles, update of restricted areas, sudden weather changes, and deterioration of communication quality.

[0064] Specifically, when a robot goes offline, it means the communication link between the robot in the autonomous mobile platform and the edge collaborative scheduling platform is interrupted, or the robot itself malfunctions, making it unable to respond to commands or send status data. This is usually determined through heartbeat mechanisms, communication timeout detection, or the robot's internal fault diagnosis system. Once a robot goes offline, its original task cannot continue, requiring immediate replanning to reallocate tasks or adjust the paths of other robots. When the battery level is below a threshold, it means the battery level of the robot in the autonomous mobile platform is below a preset safety threshold. This threshold is usually set based on factors such as robot type, task load, remaining task volume, and energy consumption required to return to the charging station. When the battery level is below this threshold, the robot may not be able to complete the current or subsequent tasks, requiring replanning to arrange its return to charging or battery replacement, and assigning its unfinished tasks to other capable robots. When a critical sensor fails, it means that the critical sensors used by the robot to perceive the environment, locate itself, or perform tasks (such as LiDAR, RGB cameras, IMU, etc.) malfunction, causing it to be unable to obtain accurate environmental information or its own status. For example, a failure of a positioning sensor may cause the robot to get lost, and a failure of an obstacle avoidance sensor may lead to collision risks. In this situation, the robot cannot perform the task safely or accurately, necessitating a replanning process. This may include isolating the robot, switching to redundant sensors (if available), or transferring the task. When a task times out, it means the robot spends more time performing a specific task or sub-task than the preset maximum allowed time. This can be caused by various reasons, such as the robot getting stuck, encountering unknown obstacles, insufficient computing resources, or task complexity exceeding expectations. A task timeout indicates an abnormal progress in the current task, requiring replanning to assess the task status and decide whether to continue waiting, retry, assign to another robot, or cancel the task. When dynamic obstacles appear, it means a moving obstacle suddenly appears on the robot's planned path or within the work area, not initially considered in the planning. This could be a moving person, vehicle, other robot, or temporarily placed equipment. These dynamic obstacles may hinder the robot's progress or even pose a collision risk. To ensure safety and task continuity, immediate replanning is required to calculate a new obstacle avoidance path or adjust the task execution order. When restricted areas are updated, it means that certain areas in the work environment are temporarily or permanently marked as areas where robots are prohibited from entering due to safety, maintenance, or other operational needs. These updates may originate from human commands, safety system linkages, or environmental perception. When the robot's current path or planned task involves restricted areas, replanning must be triggered to avoid entering these areas and to find alternative paths or adjust task assignments. Sudden weather changes refer to drastic changes in weather conditions within the operating environment, such as sudden strong winds, heavy rain, hail, sandstorms, or extreme high / low temperatures. These weather conditions can affect the robot's sensor performance (e.g., rain and fog affecting vision), communication quality, mobility, and even battery life.For weather-sensitive robots such as drones, sudden weather changes can directly render them unable to fly. Therefore, replanning is necessary, which may include pausing the mission, returning to base, or switching to a robot better suited to severe weather. Deteriorating communication quality refers to a decrease in wireless communication signal strength, increased latency, or higher packet loss rate between the robot and the edge collaborative scheduling platform or cloud-based intelligent cognitive platform. Deteriorating communication quality can lead to command transmission failures and untimely updates of status data, thus affecting the robot's real-time control and collaborative capabilities. When communication quality falls below a preset threshold, replanning is required, which may include adjusting communication strategies, finding areas with better signal strength, switching communication links, or reducing task complexity to decrease reliance on real-time communication.

[0065] In one embodiment, a strategy for hierarchical replanning to be performed by the dynamic replanning module is proposed. Specifically, this hierarchical replanning mechanism divides the replanning task into different levels according to the severity of the anomaly and takes corresponding handling measures.

[0066] In one embodiment, for minor anomalies, such as temporary obstacles on a local path or brief communication fluctuations, the dynamic replanning module performs local replanning, adjusting only the affected tasks and local paths. This local replanning typically involves fine-tuning the robot's current path, avoiding temporary obstacles, or adjusting the execution order of individual tasks to minimize the impact on the overall task plan. For example, when sensors detect a small, dodgeable dynamic obstacle ahead, the system quickly calculates a new local path, allowing the robot to avoid the obstacle and continue its current task without replanning the entire task sequence.

[0067] In one embodiment, for moderate anomalies, such as a robot temporarily going offline or encountering persistent but limited obstacles in a specific area, the dynamic replanning module performs area replanning, reallocating tasks to the corresponding area and increasing safety redundancy. In this case, the system reassesses all available robot resources in the affected area and, based on their capability vectors, remaining battery power, current location, load status, and link quality, uses an auction algorithm to reallocate tasks to ensure that tasks in that area can continue to execute. Simultaneously, the system increases the safety redundancy of the area, for example, by adding backup robots, increasing the safe distance between robots, or extending the task execution window, thereby enhancing the system's ability to cope with potential risks.

[0068] In one embodiment, for severe anomalies, such as widespread system failures, sudden environmental disasters, or simultaneous failures of multiple critical robots, the dynamic replanning module performs a global replanning, suspending non-critical tasks and prioritizing the review and safe evacuation of high-risk equipment. In such extreme cases, the system will reassess all unfinished tasks, identify those critical to system safety or key objectives, and immediately activate corresponding emergency plans. For example, it may guide robots to quickly evacuate dangerous areas or dispatch specific robots to conduct emergency inspections of high-risk equipment to minimize losses and ensure the safety of personnel and equipment.

[0069] In a heterogeneous multi-robot intelligent collaborative scheduling system based on a large model, a cloud-based intelligent cognitive platform, an edge collaborative scheduling platform, and an autonomous mobile platform work together to achieve complex task scheduling and execution. However, in actual operation, the large model, key sensors, cloud-edge communication links, or individual robots may face unavailability or failure. Without an effective response mechanism, the overall stability of the system, the continuity of task execution, and security will be severely affected, even leading to task interruption or system crash, thus limiting the reliable application of the system in complex dynamic environments.

[0070] In some possible embodiments, an automatic degradation protection strategy is proposed, which includes: when the large model is unavailable, switching to the rule base and lightweight model while retaining basic scheduling capabilities; when the main sensor fails, switching to redundant sensors and reducing the running speed; when the cloud-edge link is interrupted, the edge collaborative scheduling platform takes over task scheduling and the autonomous mobile platform executes local safety policies; when the robot experiences continuous failures, triggering task migration and nearby takeover.

[0071] This automatic degradation protection strategy aims to enhance the system's robustness, ensuring a certain level of operational capability even when some components or functions fail, thus preventing a complete system crash. Specifically, when a large model in the cloud-based intelligent cognitive platform becomes unavailable due to various reasons (such as computing resource limitations, network latency, or model failure), the system can respond quickly and automatically switch to a pre-configured rule base and lightweight model. The rule base contains preset scheduling logic for common tasks and scenarios, while the lightweight model is a computationally less expensive and faster-responding model, typically deployed locally on an edge collaborative scheduling platform to handle urgent or basic scheduling needs. Through this switching mechanism, the system can retain basic scheduling capabilities, preventing the entire scheduling system from shutting down due to a large model failure.

[0072] For autonomous mobile platforms, perception capabilities are crucial for task execution. When the primary sensor (e.g., LiDAR for high-precision positioning or an RGB camera for environmental perception) fails, the system detects the fault through a sensor health monitoring module and automatically switches to backup or redundant sensors. These redundant sensors provide alternative perception data, ensuring the robot can still acquire necessary environmental information. Simultaneously, to mitigate the risks of potential degradation in perception capabilities, the autonomous mobile platform proactively reduces its operating speed, providing the system with more processing time to handle uncertainties and ensuring safe operation of the robot under conditions of limited perception.

[0073] In one embodiment, the communication link between the cloud and the edge is a crucial guarantee for achieving intelligent collaborative scheduling. When the cloud-edge link is interrupted due to network failure, signal interference, or other reasons, the edge collaborative scheduling platform immediately initiates a local takeover mode. Because the edge collaborative scheduling platform possesses a certain degree of autonomous decision-making capability and local task caching, it can utilize existing task information and local computing resources to continue task parsing, allocation, and scheduling, maintaining task execution within the area. Simultaneously, the autonomous mobile platform activates preset local security policies, such as pausing the current task, returning to a predetermined safe area, entering standby mode, or performing simple local patrol tasks, to avoid danger in the absence of cloud instructions, ensuring the basic operation and security of the system in offline mode.

[0074] When a heterogeneous robot in an autonomous mobile platform experiences consecutive failures due to hardware malfunctions, software errors, or energy depletion, rendering it unable to continue performing tasks, the edge collaborative scheduling platform continuously monitors the status of each robot. Once a robot's consecutive failures reach a preset threshold, the system removes the tasks it is currently executing or about to execute from that robot. Subsequently, based on information such as the capability vectors, remaining battery power, current location, load status, and link quality of other available robots in the current task pool, the system uses mechanisms such as auction algorithms or Monte Carlo tree search to reassign these tasks to other nearby robots with matching capabilities and normal status, achieving seamless task takeover and ensuring the overall task completion rate.

[0075] In some possible embodiments, a safety closed-loop and recovery mechanism is proposed. This mechanism aims to provide a robust and adaptive anomaly handling framework for heterogeneous multi-robot intelligent collaborative scheduling systems, ensuring that the system can operate stably, recover effectively, and continuously learn in the face of various emergencies. It is not merely simple error handling, but a complete closed-loop management process encompassing information recording, policy execution, state recovery, and experience accumulation.

[0076] Specifically, this mechanism includes recording the triggering reason, strategy version, execution result, and rollback point for each replanning operation. The triggering reason refers to the specific event or condition that causes the dynamic replanning module to start. For example, the system records whether it's due to an autonomous mobile platform 100's battery level falling below a preset threshold, a critical sensor malfunctioning, the detection of a new dynamic obstacle, or the receipt of a new restricted area update command. This information is typically stored in structured logs, containing detailed data such as timestamps, event types, relevant robot IDs, and sensor readings, for subsequent fault analysis and tracing. The strategy version refers to the specific version identifier of the Monte Carlo tree search algorithm or related scheduling strategy used during replanning. This is crucial for evaluating the effectiveness of different replanning strategies, conducting A / B testing, and backtracking to a specific strategy configuration when problems occur. For example, the system can maintain a unique version number for different replanning algorithms or parameter configurations and record it for each replanning operation. The execution result describes the changes in the system or task state after the replanning operation is completed. This includes whether the replanning successfully generated new task sequences or paths, new task allocation schemes, changes in task completion times, resource consumption, and whether obstacles were successfully avoided or anomalies were resolved. These results provide a quantitative basis for evaluating the effectiveness and efficiency of the replanning. A rollback point is a snapshot or set of key parameters representing a stable state of the system before the replanning operation. When replanning fails to achieve the expected results or leads to new problems, the system can use this rollback point to restore the task plan or robot state to a known good state before replanning, thereby avoiding the system falling into an unrecoverable error loop. Rollback points can include information such as task queues, robot positions, sensor configurations, and environmental maps.

[0077] In one embodiment, the mechanism further includes gradually restoring the original task plan in a local-to-global order after the anomaly is resolved. Anomaly resolution means that the abnormal conditions that caused the replanning have disappeared or been effectively addressed. Examples include a stable communication link, replenished robot battery, removal of dynamic obstacles, or repair of sensor malfunctions. The system continuously monitors relevant status parameters to determine whether the anomaly has been resolved. The local-to-global sequential recovery is a phased, gradual recovery strategy designed to minimize risks and impact on the overall system operation during the recovery process. Local recovery typically involves the robot or task unit least affected by the anomaly. For example, if communication of an autonomous mobile platform 100 is briefly interrupted and then restored, the system will first attempt to restore the robot's pre-interruption task or path and verify the safety of its local environment. Global recovery, after local recovery has stabilized, gradually restores the entire system's task plan to its pre-anomaly state, or makes minor adjustments based on the latest environmental information. This step-by-step recovery avoids large-scale adjustments while the overall system state is unstable, thereby reducing the risk of secondary failures and improving the success rate and efficiency of recovery.

[0078] In one embodiment, the mechanism further includes forming an online experience base for frequently occurring anomalies, continuously optimizing threshold and cost function parameters. The online experience base is a dynamically updated knowledge base used to store and analyze data on historical replanning events. It gathers information such as the triggering reasons, strategy versions, execution results, and recovery processes of multiple replanning events, and identifies common anomaly types, the success rate of replanning strategies, and recovery efficiency through data mining and pattern recognition. For example, the experience base can record path blocking events frequently encountered by the wheeled robot 100 under specific terrain conditions, and the performance of the Monte Carlo tree search algorithm in these situations. The analysis results of the experience base are used to guide the adaptive adjustment of system parameters. Threshold optimization may include adjusting the conditions for triggering replanning. For example, based on historical data, if it is found that a certain autonomous mobile platform 100 is more prone to task interruption when the battery level is below 25%, the system can automatically adjust the replanning trigger threshold of that robot to 30% to intervene in advance. Cost function parameter optimization is applied to the auction algorithm or Monte Carlo tree search algorithm in the intelligent task allocation module 100. For example, if the experience base shows that the link quality of a certain autonomous mobile platform 100 fluctuates significantly under specific environments, the system can increase its communication stability weight in the bidding cost function, making it more inclined to select tasks with better communication conditions during task allocation, or to avoid communication blind spots during path planning. This continuous self-learning and adaptive capability enables the system to continuously improve its scheduling performance and robustness in complex dynamic environments.

[0079] In some possible embodiments, such as Figure 2As shown, a heterogeneous multi-robot intelligent cooperative scheduling method based on a large model is proposed, including receiving natural language instructions; performing semantic parsing of the natural language instructions through a cloud-based intelligent cognitive platform to extract task constraints; recursively decomposing high-level tasks into atomic subtasks through a hierarchical task network using an edge cooperative scheduling platform; allocating the atomic subtasks to the corresponding heterogeneous robots based on an auction algorithm, generating bidding costs based on robot capability vectors, remaining battery power, current position, load status, and link quality during the allocation process; controlling the heterogeneous robots to execute the allocated tasks, achieving distributed cooperative obstacle avoidance through multi-agent reinforcement learning; monitoring the execution status in real time, and performing hierarchical dynamic task replanning through Monte Carlo tree search when replanning conditions are triggered; and generating and feeding back an inspection report after task completion.

[0080] In one embodiment, the cloud-based intelligent cognitive platform first receives natural language instructions input by the operator, such as "Immediately check whether the No. 3 main transformer in the substation is overheating or leaking." The platform uses a large language model to perform deep semantic analysis of the instruction, identifying constraints such as task objectives, detection types, and urgency levels. It also combines equipment knowledge graphs to complete implicit information, such as automatically linking the equipment ledger and historical defect data of the No. 3 main transformer. Furthermore, the edge collaborative scheduling platform uses a hierarchical task network to recursively decompose high-level tasks into atomic subtasks. For example, "checking the No. 3 main transformer" is decomposed into specific steps such as "moving to the location of the No. 3 main transformer," "collecting infrared thermal imaging data," and "performing leak detection." Each atomic subtask includes task type, execution window, spatial location, resource requirements, risk level, and dependencies.

[0081] During the task allocation phase, based on an auction algorithm, each robot generates a bid price according to its own capability vector, remaining battery power, current position, load status, and link quality. For example, when performing high-level equipment inspection, a drone generates a lower bid price due to its flight capability; while for tasks in narrow passages, a quadruped robot receives allocation priority due to its terrain adaptability. When controlling this heterogeneous robot to perform tasks, a multi-agent reinforcement learning algorithm is configured to achieve distributed cooperative obstacle avoidance. Each robot dynamically adjusts its trajectory by sharing local environmental information to avoid collisions and optimize overall path efficiency.

[0082] The system monitors the execution status in real time. When triggering conditions such as robot disconnection, battery level below a threshold, critical sensor failure, task timeout, appearance of dynamic obstacles, update of restricted areas, sudden weather changes, or deterioration of communication quality are detected, the dynamic replanning module is activated. This module performs hierarchical dynamic task replanning through Monte Carlo tree search: for minor anomalies, local replanning is performed, adjusting only the affected tasks and local paths; for moderate anomalies, regional replanning is performed, reallocating tasks in the corresponding area and improving safety redundancy; for severe anomalies, global replanning is performed, suspending non-critical tasks and prioritizing the verification and safe evacuation of high-risk equipment. After the task is completed, the system automatically generates an inspection report containing detection results, anomaly records, and suggested measures, and feeds it back to the operator through a cloud-based intelligent cognitive platform.

[0083] In one embodiment, by combining a large language model with a hierarchical task network, an auction algorithm, Monte Carlo tree search, and multi-agent reinforcement learning in a closed-loop collaborative manner, automatic parsing of natural language instructions, intelligent decomposition and allocation of tasks, and autonomous replanning in dynamic environments are achieved. This enhances the intelligence level and collaborative efficiency of heterogeneous multi-robot systems in complex scenarios such as substation inspection and factory maintenance. Specifically, the application of the large language model eliminates the semantic gap in human-computer interaction; the coupling of the hierarchical task network and the auction algorithm explicitly injects task structure constraints into the bidding process; the collaboration between Monte Carlo tree search and multi-agent reinforcement learning achieves the unification of global sequence optimization and local continuous control; and a unified state machine ensures the smoothness of algorithm switching, avoiding system interruption due to single algorithm failure.

[0084] The embodiments and examples presented herein are provided to best illustrate embodiments of the invention and its particular applications, thereby enabling those skilled in the art to practice and use the invention. However, those skilled in the art will understand that the above description and examples are provided merely for ease of illustration and example. The descriptions presented are not intended to cover all aspects of the invention or to limit the invention to the precise forms disclosed.

[0085] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A heterogeneous multi-robot intelligent collaborative scheduling system based on a large model, characterized in that, include: Cloud-based intelligent cognitive platform, edge collaborative scheduling platform, and autonomous mobile platform; The cloud-based intelligent cognitive platform is used to perform large model training, knowledge management, and advanced cognitive functions, and to build an intelligent hub. The edge collaborative scheduling platform is used to perform task parsing, allocation, scheduling and coordination control, and establish a heterogeneous multi-agent assistance system. The edge collaborative scheduling platform constructs a closed-loop collaborative link of semantic understanding, task decomposition, global allocation, dynamic replanning and local obstacle avoidance, and realizes the coupling of large language model with hierarchical task network, auction algorithm, Monte Carlo tree search and multi-agent reinforcement learning. The autonomous mobile platform includes at least one heterogeneous robot for performing tasks and sensing operations on the task terminal.

2. The system according to claim 1, characterized in that, The cloud-based intelligent cognitive platform adopts a multimodal Transformer architecture, which includes: The input layer is used to acquire multi-source information, including visible light images, infrared thermal imaging, LiDAR point clouds, robot status data, and natural language commands. The feature extraction and alignment layer is used to extract visual features using a visual Transformer architecture and achieve synchronous fusion of multi-source data through spatiotemporal alignment and cross-modal attention mechanisms. The knowledge-enhanced fusion layer is used to inject a device knowledge graph containing equipment ledgers, historical defects, and physical mechanisms into the Transformer unified representation space to achieve semantic alignment. The output layer adopts a dual-path parallel structure. One path is based on DS evidence theory to perform multimodal information fusion, and the other path is based on a combination of hierarchical task network and Monte Carlo tree search to output task planning scheme. The cloud-based intelligent cognitive platform also includes a defect template library, which includes: Defect templates for substation equipment include hot spots, discharge traces, leaks, abnormal noises, and abnormal vibrations; Defect templates for factory equipment include bearing deterioration, belt misalignment, valve malfunction, and abnormal cabinet temperature rise.

3. The system according to claim 2, characterized in that, The edge collaborative scheduling platform includes: The natural language instruction parsing module is used to perform semantic parsing of natural language instructions using a large language model and extract task constraints. The hierarchical task decomposition module is used to recursively decompose high-level tasks into atomic subtasks using a hierarchical task network. The atomic subtasks include task type, execution window, spatial location, resource requirements, risk level, and dependencies. The intelligent task allocation module is used to generate bidding costs based on the robot's capability vector, remaining power, current position, load status and link quality, and calculate the globally optimal task allocation result based on the auction algorithm. The collaborative execution control module is used to achieve distributed collaborative obstacle avoidance and path optimization by employing multi-agent reinforcement learning. The dynamic replanning module is used to perform real-time dynamic task replanning using Monte Carlo tree search.

4. The system according to claim 3, characterized in that, In the closed-loop cooperative link: Large language models are used to perform semantic parsing, constraint completion, and conflict interpretation, and to build an interpretable scheduling front-end; The hierarchical task network is coupled with the auction algorithm, and the task structure constraints are explicitly injected into the bidding cost function; Monte Carlo tree search and multi-agent reinforcement learning work together; the former performs global sequence optimization, while the latter performs local continuous control. Algorithm switching is achieved through a unified state machine, avoiding system interruption caused by the failure of a single algorithm.

5. The system according to claim 1, characterized in that, The autonomous mobile platform includes: Drones are used for high-level equipment inspection, rapid cross-regional verification, initial screening of thermal anomalies, and relay communication. Quadruped robots are used to perform inspection tasks on steps, narrow passages, complex ground environments, and areas with weak structures. Wheeled robots are used to perform high-frequency routine inspections of main roads and to load materials. The workflow of the autonomous mobile platform includes: Environmental and status data are collected through multiple source sensors, including LiDAR, RGB camera, infrared thermal imager, IMU, wheel speedometer, and GNSS. Real-time semantic segmentation is performed using object detection and visual language models, and a semantic map is constructed by combining geometric information. A factor graph multi-sensor tight coupling method is used for localization to achieve semantically constrained relocalization. Path planning is performed based on semantic tags, and semantic navigation is executed. Terrain-adaptive gait switching is achieved through model predictive control or reinforcement learning controllers to complete robot motion control. The edge collaborative scheduling platform is also used to dynamically switch the robot's movement mode according to the terrain label, including wheeled robot detour, quadruped robot traversal, and drone take-off for retesting. In response to rain, fog, backlight at night, and electromagnetic interference environments, the cloud-based intelligent cognitive platform also introduces a perception confidence weighting and redundancy verification mechanism.

6. The system according to claim 3, characterized in that, The triggering conditions for the dynamic replanning module include: robot disconnection, battery level below a threshold, critical sensor failure, task timeout, appearance of dynamic obstacles, update of restricted areas, sudden weather changes, and deterioration of communication quality.

7. The system according to claim 6, characterized in that, The dynamic replanning module is also used to perform hierarchical replanning: For minor anomalies, perform local replanning, adjusting only the affected tasks and local paths; For moderate anomalies, perform regional replanning, reallocate tasks to the corresponding regions, and enhance security redundancy; In response to severe anomalies, a global replanning was implemented, non-critical tasks were suspended, and priority was given to reviewing and safely evacuating high-risk equipment.

8. The system according to claim 1, characterized in that, The system also includes an automatic degradation protection strategy, which includes: When the large model is unavailable, switch to the rule base and lightweight model while retaining basic scheduling capabilities; When the main sensor fails, switch to a redundant sensor and reduce the operating speed; When the cloud-edge link is interrupted, the edge collaborative scheduling platform takes over task scheduling, and the autonomous mobile platform executes local security policies. When the robot experiences consecutive failures, task migration and nearby takeover are triggered.

9. The system according to claim 7, characterized in that, The system also includes a security closed-loop and recovery mechanism, including: Each replanning process records the triggering reason, strategy version, execution result, and rollback point; After the anomaly is resolved, the original task plan will be gradually restored in the order of first local and then global. An online experience base is created for frequently occurring anomalies, and the threshold and cost function parameters are continuously optimized.

10. A heterogeneous multi-robot intelligent cooperative scheduling method based on a large model, characterized in that, include: Receive natural language commands; The natural language instructions are semantically parsed using a cloud-based intelligent cognitive platform to extract task constraints. The edge collaborative scheduling platform uses a hierarchical task network to recursively decompose high-level tasks into atomic subtasks. The atomic subtasks are allocated to corresponding heterogeneous robots based on an auction algorithm. During the allocation process, bidding costs are generated based on the robot's capability vector, remaining power, current position, load status, and link quality. The heterogeneous robots are controlled to perform assigned tasks, and distributed cooperative obstacle avoidance is achieved through multi-agent reinforcement learning. Real-time monitoring of execution status; when replanning conditions are triggered, hierarchical dynamic task replanning is performed through Monte Carlo tree search. Once the task is completed, an inspection report will be generated and feedback will be provided.

Images