Cross-platform heterogeneous unmanned equipment integrated command system

Abstract

This invention discloses a cross-platform heterogeneous unmanned equipment integrated command system, including a platform side and an equipment side, which communicate with each other through an interconnected patrol operation network. The platform side is equipped with a platform-side command module and a cloud-based intelligent planning module. The equipment side is equipped with a unified robot access node, which is implemented in the form of a software package. The platform side also has a controller and listener mechanism, which achieves closed-loop cooperation through a unified state space field and a unified event semantics. This invention solves the problems mentioned in the background art, such as the difficulty in unified scheduling of collaborative tasks, control source switching conflicts, and discontinuous control and feedback under link changes, by cooperating the platform-side command module, the unified robot access node on the equipment side, and the cloud-based intelligent planning module. It achieves unified scheduling and collaboration of tasks without relying on vendor-specific closed-loop systems.

Description

Technical Field

[0001] This invention relates to the field of unmanned equipment access and collaborative control technology, specifically to a cross-platform heterogeneous unmanned equipment integrated command system for inspection and surveillance scenarios. Background Technology

[0002] In outdoor inspection and surveillance scenarios such as energy and power plants, common unmanned equipment includes drones, quadruped robots, unmanned surface vessels, and fixed cameras and sensor terminals. Equipment from different manufacturers varies significantly in terms of communication bearers, control protocols, status data structures, and sensor data formats, and these devices may be distributed across public networks, private networks, local area networks, or short-range private networks. Existing technologies typically require the repeated development of access and adaptation logic for different devices and the maintenance of multiple sets of data parsing and control protocols on the platform side. This leads to a series of problems, such as difficulties in unifying the orchestration of collaborative tasks, conflicts in control source switching, and discontinuities in control and data transmission under link changes. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a cross-platform heterogeneous unmanned equipment integrated command system to solve the problems mentioned in the background art, such as difficulty in unified scheduling of collaborative tasks, conflict of control source switching, and discontinuity of control and feedback under link changes, so as to achieve unified scheduling and collaboration of tasks without relying on the closed loop of the manufacturer.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows.

[0005] A cross-platform heterogeneous unmanned equipment integrated command system includes a platform side and an equipment side, which communicate with each other through an interconnected patrol operation network. The platform side is equipped with a platform-side command module and a cloud-based intelligent planning module. The equipment side is equipped with a unified robot access node, which is implemented in the form of a software package. The cloud-based intelligent planning module receives instructions from the platform-side command module and generates multi-device path planning results and task allocation results. The platform side receives the generated results and converts them into atomic operation sequences, which are then sent to the equipment side. The equipment side executes the corresponding actions after receiving the instructions. The platform side also has a controller and listener mechanism, which achieve closed-loop cooperation through a unified state space field and a unified event semantics.

[0006] The aforementioned cross-platform heterogeneous unmanned equipment integrated command system establishes equipment templates and generates template identifiers on the platform side. After the unmanned equipment instance on the device side is connected to the unified robot access node, the platform side obtains the device identifier and binds it with the generated template identifier. The unified robot access node loads the corresponding configuration and plugins according to the binding relationship.

[0007] The aforementioned cross-platform heterogeneous unmanned equipment integrated command system includes a platform-side command module that issues unified control commands, control switching signals, and task sequences, receives and stores status snapshots, events, task progress, and results reported by the unified robot access node, and negotiates with the unified robot access node on the carrier protocol preferences and constraints.

[0008] The aforementioned cross-platform heterogeneous unmanned equipment integrated command system includes a platform-side command module comprising a data uplink / downlink path, a protocol message converter, a device and gateway status maintenance layer, an equipment control and data circulation link layer, an automated inspection task layer, an equipment implementation remote control layer, a streaming media processing / environmental sensor reporting layer, an equipment control and management service layer, and a data persistence layer. The data uplink / downlink path enables the transmission and reception of uplink data with the interconnected patrol operation network. The protocol message converter facilitates the mutual conversion of data formats, semantics, and transmission rules between different communication protocols. The equipment control and data circulation link layer carries message information obtained from the protocol message converter, forwards operational control information and sensor information to the equipment implementation remote control layer and the streaming media processing / environmental sensor reporting layer, and processes data from each layer before sending it to the protocol message converter for corresponding conversion. The automated inspection task layer sends received user requests to the cloud-based intelligent planning module for processing and generates corresponding instructions from the results processed by the cloud-based intelligent planning module, which are then sent to the device layer. Controllers and listeners are deployed within the automated inspection task layer.

[0009] The aforementioned cross-platform heterogeneous unmanned equipment integrated command system includes a cloud-based intelligent planning module comprising an edge device layer, a data format conversion layer, a hardware support layer, a core algorithm layer, a strategy selection layer, and a service configuration and control input layer. The service configuration and control input layer receives inspection request instructions from the automated inspection task layer and outputs corresponding planning schemes, which are then sent to the strategy selection layer for confirmation. The strategy selection layer selects the optimal planning scheme and transmits it to the data format conversion layer for conversion. The edge device layer receives the task instructions converted by the data format conversion layer and sends them to the device side for task execution. The edge device side receives information feedback from the device layer and transmits it to the data format conversion layer. The converted data is received by the core algorithm layer. The core algorithm layer performs comprehensive analysis and reasoning based on the information transmitted from the edge device layer to generate a multi-device collaborative planning scheme. The strategy selection layer receives the multi-device system planning scheme output by the core algorithm layer and compares it with the planning scheme output by the service configuration and control output module. The selected optimal execution strategy is sent to the data format conversion layer for format conversion, ensuring that the edge device layer always receives and issues task instructions based on the optimal planning scheme.

[0010] The aforementioned cross-platform heterogeneous unmanned equipment integrated command system further includes a planning / decision algorithm service layer, a data mapping layer, and an action execution layer on the equipment side. The planning / decision algorithm service layer is the intelligent execution center on the equipment side, completing local path replanning and multi-agent behavior decisions. The planning / decision algorithm service layer receives task instructions issued by the edge device layer and transmits the instructions to the data mapping layer for conversion. The action execution layer receives the converted task instructions and executes the corresponding actions.

[0011] The aforementioned cross-platform heterogeneous unmanned equipment integrated command system comprises a core package, a motion control package, a sensing package, a media package, and an automatic task package. The core package establishes a unified state space, maintains a protocol path table, handles control switching, and negotiates with the cloud to determine the primary uplink protocol. When degradation criteria are met, it triggers a primary uplink protocol switch and reports the switch event. The core package provides real-time device status data to the planning / decision algorithm service layer as the basis for local decision-making on the edge. The motion control package receives unified motion control commands from the platform side, selects and executes deterministic selection rules based on template identifiers, device types, and plugins issued by the platform side, loads target plugins, converts unified commands into vendor-specific motion control commands, and executes them. The automatic task package receives task allocation and path results from the automated inspection task layer. The planning / decision algorithm service layer is responsible for edge-side path tracking and local replanning, and reports progress and abnormal events to the platform side via the core package. The raw data collected by the sensing package is normalized by the data mapping layer and written into the unified state space. The media package directly transmits media streams to the platform side through the action execution layer by calling the audio / video hardware interface.

[0012] The aforementioned cross-platform heterogeneous unmanned equipment integrated command system supports local key mapping configuration on the device side, which is used to map key events of handheld terminals, remote controls or consoles into unified control commands or task trigger commands, and execute and transmit them back under the condition that the control authority allows.

[0013] The aforementioned cross-platform heterogeneous unmanned equipment integrated command system includes a cloud-based intelligent planning module comprising two complementary planning links: one is an operational planning link oriented towards standard task package input and platform callback, used to receive standard task packages generated by external systems and quickly output task allocation results, global path results, and equipment scheduling files; the other is an interactive planning link oriented towards inspection business orchestration, used to directly convert manual selection, area selection, or natural language inspection intentions into multi-device collaborative inspection schemes.

[0014] The aforementioned cross-platform heterogeneous unmanned equipment integrated command system, wherein the cloud-based intelligent planning module completes task objective parsing through semantic planning. After the task objective is determined, it performs unified modeling of the equipment's initial pose, home pose, road network cost, planning budget, task template, and target set. Through operational planning links or interactive planning links, it executes multi-device task allocation and path sequence optimization to generate task allocation results. The final planning results are output externally as a unified result package and unified scheduling information.

[0015] Due to the adoption of the above technical solutions, the technical progress achieved by this invention is as follows.

[0016] This invention provides a cross-platform heterogeneous unmanned equipment integrated command system. Through the cooperation of a platform-side command module, a unified robot access node on the equipment side, and a cloud-based intelligent planning module, it solves the problems mentioned in the background technology, such as difficulty in unified scheduling of collaborative tasks, conflicts in control source switching, and discontinuity in control and feedback under link changes. It achieves unified scheduling and collaboration of tasks without relying on vendor-specific closed loops. The core package establishes a unified state space on the equipment side, centrally storing operational data and control status. By binding the unified state space with templates, it ensures unified state fields and consistent reporting. Local key mapping on the equipment side unifies the input behavior of different control terminals and provides traceable event records. Furthermore, the platform side issues unified control commands, achieving control command reuse and multi-vendor adaptation through unified command input and plug-in conversion output, and improving controllability through unified security semantics. New equipment access is mainly completed through equipment-side plug-in expansion, reducing redundant modifications on the platform side.

[0017] In this invention, when the device executes instructions, it will automatically report progress and events, enabling the cloud-based intelligent planning module to re-plan, thereby enhancing the programmability of multi-device collaborative tasks. Through protocol path tables, negotiation and degradation switching mechanisms, the continuity of link change scenarios is improved, and it can flexibly adapt to the interruption recovery or migration needs of multi-device collaborative tasks.

[0018] The closed loop between the cloud-based intelligent planning module and the multi-agent execution on the device side forms a full-link collaborative mechanism that includes "task semantic parsing, target point parsing, device capability constraint modeling, multi-machine task allocation, path sequence optimization, result file distribution, device execution feedback, and anomaly trigger replanning." The cloud can complete multi-device collaborative decision-making based on the device's initial pose, home pose, speed constraints, planned flight budget, and graph road network cost matrix. The device side then executes the plan based on the task allocation file, path file, and scheduling file, thereby improving the completion rate, load balancing, and stability of multi-device path planning and collaborative inspection. Attached Figure Description

[0019] Figure 1 This is a system architecture diagram of the present invention; Figure 2-6 This is a reference diagram showing the interface changes during path planning in this invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0021] A cross-platform, heterogeneous, integrated command system for unmanned equipment, such as Figures 1 to 6 As shown, it includes a platform side and an equipment side. The platform side is equipped with a platform-side command module and a cloud-based intelligent planning module, while the equipment side is equipped with a unified robot access node. The platform side and the equipment side communicate with each other through an interconnected patrol operation network.

[0022] The cloud-based intelligent planning module receives instructions from the platform-side command module and generates multi-device path planning results and task allocation results. The platform side receives the generated results and converts them into atomic operation sequences, which are then sent to the device side. The device side receives the instructions and executes the corresponding actions.

[0023] The interconnected patrol network serves as the interconnection carrier between the platform and the equipment side. It achieves cross-network segment connectivity through direct connection or gateway transfer. For short-range private protocol devices, the gateway node can complete the protocol conversion before connecting to the platform.

[0024] The platform establishes equipment templates and generates template identifiers. The template identifiers include equipment type identifiers, capability descriptions, status field mapping rules, control parameter boundaries, and operation and control conversion plug-in selection rules.

[0025] After the unmanned equipment instance is connected to the device side, the platform side obtains the device identifier and binds it with the template identifier. The unified robot access node loads the corresponding configuration and plugins according to the binding relationship.

[0026] The platform-side command module is used to manage equipment templates and equipment instances. It issues unified control commands, control switching signals and task sequences, receives and stores status snapshots, events, task progress and results reported by the unified robot access node, and negotiates with the unified robot access node on the carrier protocol preferences and constraints.

[0027] The platform-side command module includes data uplink and downlink paths, protocol message converters, equipment and gateway status maintenance layer, equipment control and data flow link layer, automated inspection task layer, equipment remote control layer, streaming media processing / environmental sensor reporting layer, equipment control and management service layer, and data persistence layer.

[0028] Protocol message converters are used to parse the message content from network transmission protocols such as MQTT, WebSocket, CoAP, and HTTP after they are transmitted to the cloud platform, and convert them into unified Java POJO class objects for use by various interfaces of the cloud platform backend program. When sending to devices or other control terminals, the Java POJO class objects are then converted back into the format applicable to each protocol message.

[0029] The device and gateway status maintenance layer is used to record the online status of each device and its gateway on the access platform. The gateway will also record the online information of other devices connected to its lower layer. This is used to record the online status and connection relationship between the devices in the star topology network that constitutes the system.

[0030] The equipment control and data flow link layer is used to carry message information obtained from the protocol message converter, forward operation control information, sensor information, etc. to other layers, and send the processed data from each layer to the protocol message converter for corresponding conversion.

[0031] The automated inspection task layer receives user requests, processes them using large-scale model scheduling and planning techniques, reinforcement learning algorithms, listeners, and controllers, and then sends waypoint information and formation information of each device to the device layer.

[0032] The equipment implementation remote control layer can establish a direct control channel between terminal equipment, Web console and field equipment, and realize the direct transmission of operation and control commands and image transmission through protocols such as MQTT, WebSocket and WebRTC.

[0033] The streaming media processing / environmental sensor reporting layer is dedicated to processing audio and video data from the device and transmitting it via the WebRTC protocol. Sensor data is reported to the cloud console via the WebSocket protocol.

[0034] The equipment control and management service layer is used to maintain and record the equipment's power level, GPS location information, equipment status (normal / fault / alarm), and other data that users can customize and record. This data is stored in the persistent layer for equipment management.

[0035] The data persistence layer uses a combination of databases such as PostgreSQL, Redis, MongoDB, and MySQL, as well as message queues such as Kafka and RabbitMQ, to store the various data generated in the above layers.

[0036] The data uplink and downlink paths enable the transmission and reception of uplink data with the interconnected patrol operation network. The protocol message converter enables the mutual conversion of data formats, semantics and transmission rules between different communication protocols, solving the communication compatibility problem between heterogeneous systems and enabling devices that could not communicate directly to exchange data smoothly.

[0037] The automated inspection task layer is configured with a controller and listener mechanism. The controller is used to issue control and tasks. Specifically, the controller issues unified control commands, control switching signals or task sequences to the equipment, and can maintain the desired state and control strategy on the platform side.

[0038] Listeners are used to subscribe to state snapshots and event streams, and drive data recording and subsequent control processes according to trigger conditions. After listening to the data information reported by each piece of equipment, the listener processes and judges the data, and can automatically call the relevant controllers to complete the operation.

[0039] The triggering conditions for a listener include at least threshold triggering, state change triggering, time window aggregation triggering, and event type triggering.

[0040] The actions that can be performed after the listener is triggered include at least data storage, alarm push, calling the intelligent planning module to generate new tasks, and calling the controller to issue a new atomic operation sequence through the MCP interface.

[0041] The controller and listener achieve closed-loop coordination through unified state space fields and unified event semantics. After the controller issues a command, the listener tracks the relevant state fields and event reports. When it detects an execution failure event, rejection event, runaway protection event, protocol switching event, or path deviation event, the listener triggers compensation actions and, through the controller, performs retries, rollbacks, or task migrations, thereby improving the reliability and traceability of multi-device collaborative tasks. This closed-loop coordination between the controller and listener improves the observability of control execution, the efficiency of anomaly handling, and the consistency of data feedback.

[0042] The unified robot access node is set on the device side and implemented in the form of a software package. The device side is also configured with a planning / decision algorithm service layer, a data mapping layer, and an action execution layer.

[0043] The software package includes a core package and an operation and control package. The core package is used to establish a unified state space, maintain a protocol path table, handle control switching, negotiate with the cloud to determine the primary uplink protocol, and trigger the primary uplink protocol switching and report the switching event when the degradation judgment conditions are met.

[0044] The software package layer (core package, operation and control package, sensor package, media package, and automatic task package) is located at the top layer on the device side and communicates directly with the platform side. The software package layer is passed down layer by layer through the planning / decision algorithm service layer, data mapping layer, and action execution layer, ultimately driving hardware execution.

[0045] The core package maintains a unified state space and provides real-time device status data to the planning / decision algorithm service layer as the basis for local decisions on the edge. The motion control package passes unified motion control commands down to the data mapping layer for format conversion, and then the action execution layer calls the vendor's SDK for execution. After receiving the task allocation and path results from the cloud, the automatic task package is responsible for edge path tracking and local replanning by the planning / decision algorithm service layer, and reports progress and abnormal events to the platform side through the core package. The raw data collected by the sensing package is normalized by the data mapping layer and written into the unified state space. The media package directly transmits media streams to the platform side by calling the audio and video hardware interface through the action execution layer.

[0046] The planning / decision algorithm service layer is the intelligent execution hub on the device side, responsible for completing local path replanning and multi-agent behavior decisions on the device. When the device encounters local obstacles or deviates from the path during execution, it can adjust the path directly on the device side without requesting the cloud every time. At the same time, it can autonomously decide the execution method based on the task allocation file issued by the cloud and the real-time status of the local machine, and continuously send the task progress and abnormal events back to the core package to support the cloud listener to trigger replanning.

[0047] The data mapping layer is the device-side protocol adaptation and format standardization layer, whose core responsibility is to shield the differences in data structures between hardware from different manufacturers. In the uplink direction, it normalizes the raw data reported by hardware from various manufacturers into a unified structured format and writes it into a unified state space. In the downlink direction, it reverse-maps unified control commands into a proprietary command format that specific manufacturer hardware can recognize, and then hands it over to the action execution layer to drive the hardware. Its essential function is to decouple the upper-layer software package from the lower-layer hardware, so that when a new device is connected, only adaptation rules need to be added at this layer, without modifying the upper-layer software package logic.

[0048] The motion control package is used to receive unified motion control commands from the platform side, execute deterministic selection rules based on template identifier, device type, and plugin selection configuration issued by the platform side, load the target plugin, convert the unified commands into vendor-specific motion control commands and execute them, and perform filtering based on the control rights, modes and speed limit parameters of the unified state space, outputting unified safety semantics and events such as speed limit, rejection and runaway protection.

[0049] The core package establishes a unified state space on the device side to centrally store runtime data and control status, and distinguishes between persistent and non-persistent fields. Examples of standard fields in the same state space are shown in Table 1.

[0050] Table 1 field name type persistence illustrate device_id string yes Equipment instance primary key template_id string yes Equipment Template Primary Key device_type string yes Equipment type online_status bool no Current online status current_controller string no Current controller control_mode string no Inspection mode primary_uplink_protocol string no Current main uplink protocol protocol_table object no Protocol Path Table battery_level float no Battery percentage position object no Location information velocity object no Speed ​​information attitude object no Attitude information linear_vel_limit float yes Combined speed limit angular_vel_limit float yes Yaw rate limit error_code int no Error code task_id string no Current task identifier task_stage string no Current task phase uptime_seconds int no runtime failsafe_active bool no runaway protection status failsafe_action_executed string no Recent protective actions The core package serves as the unified entry point for writing to the state space, and other software packages write to the state space through the interfaces provided by the core package. For update events from multiple sources for the same field, the core package uses timestamp overwriting or source priority adjudication rules to ensure state consistency. Persistent fields are written to disk under change-triggered or periodic-triggered conditions, and are restored and a state snapshot is reported after a restart.

[0051] The core package is used to maintain the protocol path table. Specifically, for at least two uplink bearer protocols, it records the path identifier, most recent probe time, availability flag, round-trip time metric, packet loss rate metric, available bandwidth estimate, number of consecutive failures, and path quality score. The core package performs link probe or passive statistics on each path at a configurable period and generates path quality scores in a configurable weighted manner.

[0052] The platform distributes a set of candidate protocols and protocol preferences or constraints. The core packet combines the protocol path table score to determine the primary uplink protocol and writes it into the unified state space before it takes effect.

[0053] The core packet settings can be configured with degradation criteria to trigger the switching of the primary uplink protocol. Degradation criteria include primary path unavailability, primary path latency exceeding a threshold for a duration window, primary path packet loss rate exceeding a threshold for a duration window, and primary path bandwidth falling below a threshold for a duration window.

[0054] When the degradation criteria are met, the core packet selects the path with the highest score and that meets the constraints from the candidate path set as the new primary uplink protocol and reports the protocol switching event.

[0055] The core package can be configured with a minimum hold time or a switch cooldown mechanism to suppress frequent switching.

[0056] The motion control package receives unified motion control commands from the platform side and then outputs manufacturer commands. Examples of unified motion control fields are shown in Table 2.

[0057] Table 2 Fields type unit illustrate msg_version string - Message Version device_id string - Equipment identification timestamp_ns uint64 ns Timestamp controller string - Source of instructions linear_vel_x float64 m / s forward and backward speed linear_vel_y float64 m / s Lateral velocity linear_vel_z float64 m / s Vertical velocity angular_vel_z float64 rad / s Yaw angular velocity target_roll float64 rad Target roll angle target_pitch float64 rad Target pitch angle target_yaw float64 rad Target yaw angle action string - Extended Actions extra_json string - Extended parameters The instruction translation logic of the operation control package is provided in the form of extensible plug-in packages. Each plug-in package contains at least one manifest file, which includes at least the plug-in identifier and version number, the scope of adaptation, the capability declaration, the dependency information, and the optional verification information.

[0058] The operation and control package executes deterministic selection rules and loads the target plugin based on the template identifier, device type, and plugin selection configuration issued by the platform. It prioritizes precise matching based on the template identifier; if no template match exists, it matches based on the device type; if multiple candidate plugins exist, it selects based on version priority or priority specified by the platform; if a unique selection cannot be made, it reports a plugin selection failure event and waits for a clear selection from the platform.

[0059] When the platform issues a plugin switching command or the template binding changes, the operation and control package uninstalls the current plugin and loads the new plugin, writes the plugin identifier, version number and effective time before and after the switch into the unified state space and reports the plugin switching event.

[0060] Before executing commands, the operation control package performs filtering and safety control based on a unified state space. It limits or rejects execution based on the control mode, current controller, runaway protection status and speed limit parameters, and reports the truncation, rejection or abnormal results with unified event semantics to achieve consistent safety control behavior across devices.

[0061] The device supports local key mapping configuration, which maps key events from handheld terminals, remote controls, or consoles to unified control commands or task trigger commands, and executes and transmits them back when control is permitted.

[0062] The key event is mapped to a configurable data structure, which includes at least the key identifier, trigger condition, target instruction type and instruction parameters. The trigger condition includes single click, double click, long press, combination key and joystick axis threshold trigger.

[0063] Upon receiving a button event, the device determines its operation based on the current controller and control mode. Only when control authority permits will the button event be converted into a unified control command and executed by the operation and control package. If a button-triggered command is truncated or rejected, the device reports the corresponding event for recording and playback.

[0064] Key event mapping supports runtime updates. After an update, a mapping change event is generated and the mapping version number is recorded.

[0065] The platform sends control switching signaling to adjust the controller, control mode and bearer preference. After receiving the core packet, it updates the state space and sends back confirmation. See Table 3 for examples of control switching signaling fields.

[0066] Table 3 Fields type Optional values illustrate msg_type string control_switch Message Type msg_version string 1.0 Version device_id string - Target equipment timestamp_ms int - Timestamp switch.control_mode string or null manual or auto or null Mode switching switch.controller string or null cloud, field, auto, or null Controller switching switch.primary_uplink_protocol string or null MQTT, WebSocket, TCP, or null Carrying preferences switch.media.action string start, stop, or switch Media activities switch.media.protocol string or null WebRTC or RTSP or null Media Agreement The software package also includes a sensing package, a media package, and an automated task package. The sensing package is used to acquire raw data from the external environment, the media package is responsible for processing and transmitting multimedia information, such as audio and video capture, media storage and playback, and real-time streaming, and the automated task package is responsible for orchestration and automation logic.

[0067] The platform can generate multi-device path planning results and task allocation results through the cloud-based intelligent planning module, and convert them into atomic operation sequences for distribution. The device side can form multi-agent collaborative behavior through automatic task packages, execute path segments and report progress and anomalies, triggering cloud-based replanning or device-side local replanning.

[0068] Specifically, the platform breaks down the inspection task into a sequence of atomic operations and issues them out. The device executes these operations automatically and reports task progress, abnormal events, and results. Each atomic operation includes at least the operation type, execution parameters, expected result, and rollback strategy.

[0069] When a device is offline, malfunctions, or lacks sufficient resources, the platform selects a replacement device based on the unified state space information of each device, and migrates the unfinished atomic operation sequence to the replacement device for continued execution, thereby achieving continuity and traceability of multi-device collaborative tasks.

[0070] The cloud-based intelligent planning module includes an edge device layer, a data format conversion layer, a hardware support layer, a core algorithm layer, a strategy selection layer, and a service configuration and control input layer.

[0071] Lightweight small intelligent agents are deployed at the edge device layer. They are responsible for receiving structured instructions or natural language commands issued by the platform side. The locally deployed small intelligent agents parse these instructions and convert them into a unified command format. Then, they call the underlying interface of the device through the MCP protocol to send the instructions to the planning / decision algorithm service layer on the device side to complete the closed-loop execution of specific actions.

[0072] At the same time, the edge device layer continuously collects device operating status, environmental perception data, and action execution results, and transmits them back to the core algorithm layer in real time, providing data support for the platform to dynamically adjust the planning scheme.

[0073] The core responsibility of the data format conversion layer is to shield the differences between heterogeneous data structures and achieve standardized communication across layers. In the uplink direction, this layer normalizes the raw streaming data reported by edge devices and transforms it into a unified structured data format; in the downlink direction, it reverse-maps the decision logic generated by the platform side into a private instruction format that edge devices can recognize, thereby building a transparent and consistent data exchange channel between the cloud and edge devices.

[0074] The hardware support layer undertakes two core functions: first, to uniformly manage heterogeneous computing resources in the cloud and realize the reasonable scheduling and resource allocation of computing tasks; second, to maintain the network connection channel between the cloud and edge devices, and to be responsible for handling link protection strategies such as communication reconnection, data caching, message retransmission and protocol degradation, so as to ensure the continuity and reliability of communication in complex network environments.

[0075] The core algorithm layer is responsible for running a large-scale intelligent agent in the cloud. This agent gathers the status and perception data continuously reported by edge devices. After comprehensively analyzing and reasoning about the above information, it generates candidate multi-device collaborative planning schemes and distributes the schemes to the strategy selection layer according to user instructions or system trigger conditions.

[0076] The strategy selection layer evaluates and optimizes the candidate planning schemes output by the core algorithm layer and the service configuration and control input layer from multiple dimensions based on the system default strategy or user-defined strategy. After selecting the optimal execution strategy, it sends it to the data format conversion layer for data format conversion. Then, it ensures that the final execution scheme achieves overall optimization in terms of efficiency, security and resource consumption through the edge device layer to the device layer.

[0077] The service configuration and control input layer is responsible for encapsulating the finalized strategies and plans into standardized external service interfaces. After receiving the requirement instructions from the automated inspection task layer, it automatically retrieves the internally encapsulated strategies and plans, and supports external output in multiple protocol formats such as RESTful API, gRPC, and MQTT to adapt to the integration needs of different business systems.

[0078] The service configuration and control input layer also undertakes operation and maintenance management functions such as log recording, security monitoring and audit tracking, providing complete data support for system anomaly location, behavior backtracking and compliance review.

[0079] In this embodiment, the strategy selection layer of the cloud-based intelligent planning module includes two complementary planning links: one is an operational planning link oriented towards standard task package input and platform callback, and the other is an interactive planning link oriented towards inspection business orchestration. The operational planning link is used to receive standard task packages generated by external systems and quickly output task allocation results, global path results, and equipment scheduling files. The interactive planning link is used to directly convert manual point selection, area selection, or natural language inspection intentions into multi-device collaborative inspection schemes. Examples of relevant interfaces and main fields are shown in Table 4.

[0080] Table 4 The cloud-based intelligent planning module first completes the task target parsing through semantic planning. Specifically, semantic planning parses the inspection intent in the order of template matching, numbered building matching, building range matching, building alias matching, anchor point proximity matching, category matching, independent navigation point matching, and large model matching, and uniformly converts the targets from different sources into a set of navigation points to be assigned.

[0081] The target parsing results adopt a unified structure, and field examples are shown in Table 5.

[0082] Table 5 field name type illustrate resolution_mode string Target parsing method resolved_target_set_ids array List of target set identifiers obtained by parsing resolved_nav_point_ids array List of navigation point identifiers obtained by parsing matched_building_ids array List of matched building identifiers matched_building_names array List of matched building names matched_nav_point_ids array List of directly matched navigation point identifiers notes string Analysis and Explanation query string Original semantic query text llm_attempted bool Should we try calling the large model parsing? After the task objectives are determined, the cloud-based intelligent planning module performs unified modeling of the device's initial pose, home pose, road network cost, planning budget, task template, and target set. Examples of the core asset files and key fields used in formal multi-robot planning are shown in Table 6.

[0083] Table 6 file name Key fields illustrate planner_problem.json planning_slot_id, hardware_id, start_nav_point_id, home_nav_point_id, start_pose, home_pose, planner_limits Describe the equipment involved in the planning, the origin and destination points, and the budget constraints. robot_to_nav_costs.json start_to_nav_costs, nav_to_home_costs, reachable, distance_m, estimated_time_s, path_node_ids, path_edge_ids Describe the cost from the device's origin to the task point, and the cost of returning home from the task point. nav_to_nav_shortest_paths.json pairs, reachable, distance_m, estimated_time_s, path_node_ids, path_edge_ids Describe the cost of the shortest path between navigation points mission_request_templates.json template_id, natural_language, target_set_ids, candidate_nav_point_ids, default_constraints Describe the templated inspection task semantic_target_sets.json target_set_id, display_name, selector_type, building_ids, nav_point_ids Describes the mapping relationship between building categories or building instances and the set of navigation points. For task-based planning, the cloud-based intelligent planning module receives standard task packages as input for rapid planning. Examples of task package files and key fields are shown in Table 7.

[0084] Table 7 file name Key fields illustrate manifest.json package_id, entry_files, etc. Describe the task package metadata and entry file semantic.json Task semantic related fields Describe the semantics of the task and the business context route_graph.json nodes, edges, node_id, from, to, edge_id, cost, length_m, surface Describe the weighted road network structure goals.json goals, goal_id, position Describe the target points to be inspected robots.json robots, robot_id, start, max_speed_mps Describe the equipment and speed capabilities involved in the execution. constraints.json formation_required, formation_type, spacing_m Describe formation and cooperative execution constraints The task-oriented planning link uses fast nearest neighbor priority allocation and shortest path calculation on the graph to complete batch planning. That is, it first completes the initial task allocation based on the distance between the current location of the device and the target point, and then finds the shortest path for each path segment on the weighted road network. When the task constraint requires group execution, the cloud further generates the group configuration so that each device maintains the predetermined master-slave relationship and relative offset during execution.

[0085] When allocating tasks and optimizing path order for multiple devices, a heuristic minimum completion time planning method with budget constraints is adopted, and the specific operation is as follows.

[0086] First, the task difficulty is calculated based on the single point reachability time and the number of devices capable of performing the objective, and difficult-to-assign objectives are prioritized.

[0087] Subsequently, under various construction modes and different device sequences, the feasible positions for inserting the target into the current path of each device are enumerated. The task completion time vector after insertion is compared with the total distance, with the goal of minimizing the longest task completion time, while also considering reducing the total distance.

[0088] After the initial allocation is completed, path reversal optimization, cross-device migration optimization, cross-device exchange optimization, and unassigned task repair are performed to improve task coverage, load balancing, and overall completion efficiency under explicit range budget constraints.

[0089] The final planning results are output externally as a unified result package and unified scheduling information. Examples of result files and key contents are shown in Table 8.

[0090] Table 8 file name Key fields illustrate planner_manifest.json planner_job_id, mission_id, status, files Description of the result package list and file index task_assignment.json mission_id, assignments, robot_id, task_sequence Describe the task sequence allocation results for each device. global_paths.json mission_id, paths, robot_id, goal_id, path_type, estimated_length_m, estimated_duration_s, waypoints Describe the global path results for each device. planner_summary.json status, message, reachable_goals, unreachable_goals, warnings Description of planning summary information formation_plan.json formation_required, formation_type, leader_robot_id, members, spacing_m Describe the formation execution constraints ros_dispatch.json dispatches, robot_id, namespace, controller_mode, path_file, task_file Describe the device-side scheduling mapping relationship After cloud-based planning is completed, the upper-level platform can continuously obtain the planning status through the job query interface and download the result package after the job is completed. If a callback address is configured, the cloud-based intelligent planning module can also actively send back the job identifier, task identifier, execution status, result address, and message description to the upper-level business system.

[0091] The automatic task package on the device side executes its respective path segment based on the task allocation, path results and scheduling information in the result package, and continuously reports the location, speed, task progress and abnormal events.

[0092] The following closed loop is formed between the cloud and the device side: The cloud first generates multi-device task allocation and path results based on task semantics, map semantics, device start and end poses, road network cost matrix and remaining budget, and then maps the results to path tracking tasks that each device can execute.

[0093] The device executes according to the scheduling relationship and returns the execution status through the unified state space and event mechanism mentioned above in this invention.

[0094] When the upper-level platform detects that a device is offline, partially unreachable, blocked by obstacles, disrupted in formation, or has insufficient remaining budget, it can resubmit the latest task set, device status, and constraints to the planning interface. The cloud then regenerates the task allocation results, path results, and scheduling information, which are then distributed to the devices for continued execution. This forms a closed loop of cloud-based intelligent planning and multi-device, multi-agent path planning, consisting of "semantic decision-making, multi-machine allocation, graph-based path planning, result packaging, device execution, status feedback, and replanning."

[0095] The command method based on the command system described in this invention is as follows: S1. On the platform side, an equipment template is established and a template identifier is generated. On the equipment side, a unified robot access node is deployed to complete the registration. After registration, the equipment identifier is obtained and the equipment identifier is bound to the template identifier.

[0096] S2, the platform-side listener subscribes to state snapshots and event streams and drives the controller to issue controls and tasks according to trigger conditions.

[0097] S3. The platform sends out unified motion control commands. The motion control package selects the plug-in based on the template identifier or device type and converts it into the manufacturer's command for execution. When the configurable degradation judgment condition is met, the core package switches the main uplink protocol and reports the switching event.

[0098] S4. The platform generates multi-device path planning results through the cloud-based intelligent planning module and converts them into atomic operation sequences for distribution. The automatic task package is executed and progress and events are reported. The device triggers local replanning or requests the platform to replan. The platform generates updated path segments or atomic operation sequences and distributes them.

[0099] When a device malfunctions or resources are insufficient, the platform-side migration fails to complete the atomic operation and is transferred to another device for continued execution. The listener continuously tracks the execution result and triggers the compensation control process.

Claims

1. A cross-platform heterogeneous unmanned equipment integrated command system, characterized in that: The system includes a platform side and a device side, which communicate with each other through an interconnected patrol operation network. The platform side is equipped with a platform-side command module and a cloud-based intelligent planning module. The device side is equipped with a unified robot access node, which is implemented in the form of a software package. The cloud-based intelligent planning module receives instructions from the platform-side command module and generates multi-device path planning results and task allocation results. The platform side receives the generated results and converts them into atomic operation sequences, which are then sent to the device side. The device side executes the corresponding actions after receiving the instructions. The platform side also has a controller and listener mechanism, which achieve closed-loop cooperation through a unified state space field and a unified event semantics.

2. The cross-platform heterogeneous unmanned equipment integrated command system according to claim 1, characterized in that: The platform establishes equipment templates and generates template identifiers. After the unmanned equipment instance on the device side is connected to the unified robot access node, the platform side obtains the device identifier and binds it with the generated template identifier. The unified robot access node loads the corresponding configuration and plugins according to the binding relationship.

3. The cross-platform heterogeneous unmanned equipment integrated command system according to claim 1, characterized in that: The platform-side command module issues unified control commands, control switching signals and task sequences, receives and stores status snapshots, events, task progress and results reported by the unified robot access node, and negotiates with the unified robot access node on the carrier protocol preferences and constraints.

4. The cross-platform heterogeneous unmanned equipment integrated command system according to claim 1, characterized in that: The platform-side command module includes data uplink and downlink paths, protocol message converters, device and gateway status maintenance layer, equipment control and data circulation link layer, automated inspection task layer, equipment remote control implementation layer, streaming media processing / environmental sensor reporting layer, equipment control and management service layer, and data persistence layer. The data uplink and downlink paths enable the transmission and reception of uplink data with the interconnected patrol operation network. The protocol message converter enables the mutual conversion of data formats, semantics, and transmission rules between different communication protocols. The equipment control and data circulation link layer is used to carry the message information obtained from the protocol message converter, forward the operation control information, sensor information, etc. to the equipment remote control layer and the streaming media processing / environmental sensor reporting layer, and send the processed data from each layer to the protocol message converter for corresponding conversion. The automated inspection task layer sends the received user requests to the cloud-based intelligent planning module for processing, and generates corresponding instructions from the results processed by the cloud-based intelligent planning module and sends them to the device layer; the controller and the listener are deployed in the automated inspection task layer.

5. The cross-platform heterogeneous unmanned equipment integrated command system according to claim 4, characterized in that: The cloud-based intelligent planning module includes an edge device layer, a data format conversion layer, a hardware support layer, a core algorithm layer, a strategy selection layer, and a service configuration and control input layer. The service configuration and control input layer receives inspection requirement instructions from the automated inspection task layer and outputs the corresponding planning scheme, which is then sent to the strategy selection layer for confirmation. The strategy selection layer selects the optimal planning scheme and transmits it to the data format conversion layer for conversion. The edge device layer receives the task instructions after they have been converted by the data format conversion layer and sends them to the device side for task execution; The edge device receives information from the device layer and transmits it to the data format conversion layer. The converted data is received by the core algorithm layer. The core algorithm layer performs comprehensive analysis and reasoning based on the information transmitted by the edge device layer to generate a multi-device collaborative planning scheme. The strategy selection layer receives the multi-device system planning scheme output by the core algorithm layer and compares it with the planning scheme output by the service configuration and control output module. The optimal execution strategy is then sent to the data format conversion layer for format conversion, so that the edge device layer always receives and issues the task instructions of the optimal planning scheme.

6. The cross-platform heterogeneous unmanned equipment integrated command system according to claim 5, characterized in that: The device side is also equipped with a planning / decision algorithm service layer, a data mapping layer, and an action execution layer. The planning / decision algorithm service layer is the intelligent execution center on the device side, which completes local path replanning and multi-agent behavior decision-making on the device side. The planning / decision algorithm service layer receives task instructions issued by the edge device layer and transmits the instructions to the data mapping layer for conversion. The action execution layer receives the converted task instructions and executes the corresponding actions.

7. The cross-platform heterogeneous unmanned equipment integrated command system according to claim 6, characterized in that: The software package includes a core package, an operation and control package, a sensor package, a media package, and an automatic task package. The core package is used to establish a unified state space, maintain a protocol path table, handle control switching, negotiate with the cloud to determine the primary uplink protocol, and trigger the primary uplink protocol switching and report the switching event when the degradation judgment conditions are met. The core package provides real-time device status data to the planning / decision algorithm service layer as the basis for local decision-making on the edge side; the motion control package is used to receive unified motion control commands from the platform side, execute deterministic selection rules based on template identifier, device type and plugin selection configuration issued by the platform side, load the target plugin, convert the unified commands into vendor-specific motion control commands and execute them. After receiving the task allocation and path results from the automated inspection task layer, the automatic task package is responsible for edge path tracking and local replanning by the planning / decision algorithm service layer, and reports the progress and abnormal events to the platform side through the core package; the raw data collected by the sensor package is normalized by the data mapping layer and written into the unified state space; the media package directly calls the audio and video hardware interface through the action execution layer to transmit the media stream to the platform side.

8. The cross-platform heterogeneous unmanned equipment integrated command system according to claim 1, characterized in that: The device supports local key mapping configuration, which is used to map key events of handheld terminals, remote controls or consoles to unified control commands or task trigger commands, and execute and transmit them back when control is permitted.

9. The cross-platform heterogeneous unmanned equipment integrated command system according to claim 1, characterized in that: The cloud-based intelligent planning module includes two complementary planning links: one is a job-oriented planning link that receives standard task packages as input and platform callbacks, which is used to receive standard task packages generated by external systems and quickly output task allocation results, global path results, and device scheduling files. Secondly, there is an interactive planning link for inspection business orchestration, which is used to directly convert manual selection, area selection or natural language inspection intentions into multi-device collaborative inspection solutions.

10. The cross-platform heterogeneous unmanned equipment integrated command system according to claim 9, characterized in that: The cloud-based intelligent planning module completes task objective parsing through semantic planning. After the task objective is determined, it performs unified modeling of the device's starting pose, returning pose, road network cost, planning budget, task template, and target set. It then performs multi-device task allocation and path sequence optimization through a task-oriented planning link or an interactive planning link, generating task allocation results. The final planning results are output externally as a unified result package and unified scheduling information.

Images