Heterogeneous robot collaborative control method based on large model

Abstract

The application relates to the field of robot cooperative control, in particular to a heterogeneous robot cooperative control method based on a large model. The method comprises the following steps: a center node generates a semantic state tree by combining a large model with a capability parameter and issues a cache, an edge robot calculates the similarity between environment perception and trigger condition text by using a local model, activates a target tree node and converts an action label into a control instruction to drive an actuator, when communication is abnormal, each robot selects a pseudo center node based on hardware capability, network connectivity and node semantic complexity, the pseudo center node prunes and re-plans a state tree according to a node identifier and synchronously updates. Through the end-side autonomous semantic matching closed loop and the offline dynamic takeover mechanism, the application improves the single point failure problem of the centralized system depending on the cloud real-time scheduling, effectively reduces the local high-frequency cooperative communication overhead, and improves the self-healing and anti-destroying capability and task cooperative stability of the heterogeneous cluster in the limited extreme environment.

Description

Technical Field

[0001] This invention relates to the field of robot cooperative control, and in particular to a method for cooperative control of heterogeneous robots based on a large model. Background Technology

[0002] With the development of artificial intelligence and automatic control technology, heterogeneous robot swarms composed of different types of equipment such as drones and unmanned vehicles are playing an important role in scenarios such as disaster relief, field exploration, and complex industrial inspection. These heterogeneous systems currently widely adopt a centralized cloud control architecture, which uses a large-scale model or high-performance scheduling center deployed in the cloud to perform unified cognitive analysis, path planning, and task decomposition of global tasks. Then, macro-scheduling instructions are distributed to each edge execution node through a wireless communication network, thereby ensuring that devices with different physical characteristics and sensor configurations can work according to predetermined logic, realizing unified collaborative operation of large-scale swarms.

[0003] However, existing centralized collaborative control methods rely excessively on real-time and stable wireless networks. When faced with environments where the network is disrupted, such as underground spaces or environments with strong communication interference, the edge devices not only struggle to acquire new instructions but also lack dynamic takeover and task replanning mechanisms based on local states. This makes them highly susceptible to paralyzing the entire heterogeneous cluster's collaborative operation system due to a single point of failure, such as the loss of connection to the central scheduling hub. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a heterogeneous robot collaborative control method based on a large model, which aims to improve the problem that centralized architecture lacks dynamic takeover and replanning mechanisms when the network is down, and is prone to cluster collaborative paralysis due to single point of failure.

[0005] This invention provides the following technical solution: a cooperative control method for heterogeneous robots based on a large model, comprising:

[0006] S1. The central node uses a large model to combine the capability parameters of multiple edge heterogeneous robots to decompose the task, generate and distribute the corresponding semantic state tree to the cache of each edge heterogeneous robot. The semantic state tree contains multiple tree nodes with semantic triggering conditions, action tags and node identifiers.

[0007] S2. The edge heterogeneous robot uses a local lightweight model to calculate the similarity between real-time environmental perception features and text features corresponding to semantic triggering conditions in each unexecuted tree node;

[0008] S3. The edge heterogeneous robot activates the tree node whose similarity meets a preset threshold as the target tree node, and converts the action label in the target tree node into a control command to drive the actuator of the edge heterogeneous robot.

[0009] S4. When an abnormal communication with the central node is detected, each edge heterogeneous robot calculates the takeover weight based on its own capability parameters, local network connectivity, and semantic complexity of the unexecuted tree node, and recommends a pseudo-central node.

[0010] S5. Each of the edge heterogeneous robots reports the node identifier of the target tree node to the pseudo-central node through a local communication network. The pseudo-central node performs pruning and replanning on the semantic state tree according to the node identifier and updates the semantic state tree synchronously.

[0011] Preferably, in step S1, the step of decomposing the task includes:

[0012] Receive global task instructions and use an entity extraction model to transform the global task instructions into a task graph that includes environmental and spatial constraints.

[0013] The task graph and the capability parameters are fused into a prompt word matrix, which is then input into the large model inference to generate a temporalized subtask sequence.

[0014] The subtask sequence is hierarchically compiled using a graph structure model to construct a semantic state tree that includes the semantic triggering conditions, the action tags, and the node identifiers.

[0015] Preferably, in step S2, the similarity calculation step includes:

[0016] The real-time environment perception features are extracted using a multimodal pre-trained encoder and mapped to generate a high-dimensional perception feature vector.

[0017] The semantic triggering conditions in each unexecuted tree node are embedded and mapped using a text encoder to generate a high-dimensional text feature vector of the corresponding dimension.

[0018] Perform a dot product operation between the high-dimensional perceptual feature vector and the high-dimensional text feature vector, and then normalize the result to quantify the similarity between the high-dimensional perceptual feature vector and the high-dimensional text feature vector.

[0019] Preferably, in step S3, the step of activating the tree node whose similarity satisfies the preset threshold as the target tree node includes:

[0020] Obtain the current environmental interference parameters of the edge heterogeneous robot, and dynamically calculate and generate the preset threshold using a compensation function;

[0021] The similarity of each of the unexecuted tree nodes is compared with the preset threshold, and candidate tree nodes with similarity greater than the preset threshold are selected.

[0022] When there are multiple candidate tree nodes, the optimal candidate tree node with the highest similarity is selected and locked as the target tree node, and other candidate tree nodes with mutually exclusive branch relationships are frozen.

[0023] Preferably, in step S3, the step of converting the action label within the target tree node into a control command includes:

[0024] The action labels within the target tree node are parsed to extract basic action identifiers and associated spatial coordinate parameters;

[0025] Index the local skill library of the edge heterogeneous robot and match the underlying control model corresponding to the basic motion identifier;

[0026] The spatial coordinate parameters are injected into the underlying control model, and the driving parameters of the actuator are output using a kinematics solving algorithm to generate the control command.

[0027] Preferably, in step S4, the step of calculating the takeover weight includes:

[0028] Traverse the local semantic state tree, count the hierarchical features of the unexecuted tree nodes, and use an accumulation algorithm to quantize and generate the semantic complexity;

[0029] Send probe packets to other edge heterogeneous robots within the local communication network and calculate round-trip delays to construct an adjacency matrix to evaluate the connectivity of the local network;

[0030] Extract the hardware resource margin from the capability parameters, perform normalized fusion calculations with the hardware resource margin, the local network connectivity, and the semantic complexity, and output the takeover weight.

[0031] Preferably, in step S4, the step of recommending the pseudo-center node includes:

[0032] Each of the aforementioned edge heterogeneous robots is controlled to encapsulate the takeover weight into an election frame and broadcast it to other edge heterogeneous robots within the local communication network via an ad hoc networking protocol;

[0033] Within a preset time window, all campaign frames within the local communication network are received and parsed to construct a global ranking table sorted by weight values.

[0034] Extract the network address identifier of the edge heterogeneous robot with the highest weight in the global ranking table, and set the edge heterogeneous robot holding the network address identifier as the pseudo-center node.

[0035] Preferably, in step S5, the step of pruning the semantic state tree according to the node identifier includes:

[0036] Extract the received node identifiers at the pseudo-central node to construct a global progress set representing the task execution status;

[0037] The local semantic state tree is traversed using a graph search algorithm to match and locate the activated branch that contains the corresponding node identifier in the global progress set;

[0038] Remove the hierarchical dependency references of the activated branch and simultaneously delete the alternative logical branches that have an exclusive relationship with the activated branch, generating a pruned semantic state tree.

[0039] Preferably, in step S5, the step of synchronously updating the semantic state tree includes:

[0040] Extract all unexecuted tree nodes from the pruned semantic state tree, and combine the real-time physical state of each edge heterogeneous robot with the capability parameters to construct a replanning optimization model;

[0041] The replanning optimization model is solved using a planning algorithm to reconstruct the execution sequence between the unexecuted tree nodes and synthesize the updated semantic state tree.

[0042] The updated semantic state tree is converted into incremental data packets and distributed to the corresponding edge heterogeneous robots for cache overlay using a multicast protocol.

[0043] The present invention has the following beneficial effects:

[0044] 1. In this invention, when communication is abnormal, a pseudo-central node is dynamically elected based on features such as hardware resources and semantic complexity. The aggregation progress identifier of the pseudo-central node is used to perform pruning and replanning on the semantic state tree, which improves the single point of failure problem of traditional multi-machine systems that rely on cloud scheduling. This greatly enhances the self-healing and resilience of heterogeneous clusters in network-disconnected and limited environments and the stability of continuous task takeover and coordination.

[0045] 2. In this invention, a large model is used to decompose global instructions into semantic state tree-based caches, enabling edge heterogeneous robots to rely on local lightweight models to map real-time multimodal perception and text instructions to a unified high-dimensional space for similarity measurement and activation. This transforms abstract language intent into edge-side autonomous closed-loop logic, reducing the high-frequency communication overhead in the cluster collaboration process.

[0046] 3. In this invention, an instruction conversion architecture based on skill library index and kinematic solution is constructed to address node differences. At the same time, environmental interference parameters are introduced to dynamically compensate feature matching thresholds, which effectively decouples high-level semantic control logic from low-level heterogeneous hardware features. Under the premise of ensuring macro-level semantic scheduling consistency, micro-level physical action control is achieved, enhancing the generalization deployment capability in complex environments. Attached Figure Description

[0047] Figure 1 The flowchart shows the heterogeneous robot cooperative control method based on a large model proposed in this invention.

[0048] Figure 2 This is a flowchart of the task decomposition and semantic state tree generation proposed in this invention;

[0049] Figure 3 This is a flowchart of the multimodal semantic matching and similarity calculation proposed in this invention;

[0050] Figure 4 This is a flowchart illustrating the target activation and underlying control transformation process proposed in this invention.

[0051] Figure 5 This is a flowchart of the communication monitoring and pseudo-central node election process proposed in this invention;

[0052] Figure 6 This is a flowchart of the state tree pruning and dynamic replanning process proposed in this invention. Detailed Implementation

[0053] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1

[0055] In a first embodiment of the present invention, the present invention provides a cooperative control method for heterogeneous robots based on a large model, such as... Figure 1 As shown, it includes:

[0056] S1. The central node uses a large model to combine the capability parameters of multiple edge heterogeneous robots to decompose the task, generate and distribute the corresponding semantic state tree to the cache of each edge heterogeneous robot. The semantic state tree contains multiple tree nodes with semantic trigger conditions, action labels and node identifiers.

[0057] Furthermore, step S1, the task decomposition step, includes:

[0058] Receive global task instructions and use an entity extraction model to transform the global task instructions into a task graph that includes environmental and spatial constraints.

[0059] The task graph and capability parameters are fused into a prompt word matrix, which is then input into a large model inference to generate a temporal sequence of subtasks.

[0060] A hierarchical compilation of the subtask sequence is performed using a graph structure model to construct a semantic state tree containing semantic triggering conditions, action labels, and node identifiers.

[0061] Specifically, the central node receives global task instructions from external operators via a wireless communication interface. These instructions are unstructured natural language text. Internally, the central node deploys an entity extraction model based on a self-attention mechanism. This model performs sequence labeling and feature extraction on the global task instructions, identifying target location entities and physically constrained entities within the instructions, and converting them into a task graph. Let the task graph be represented as... , where the vertex set The spatial constraint information represents the three-dimensional coordinates of the target that the edge heterogeneous robot needs to reach in the workspace, and the edge set. This represents environmental constraint information, namely the path passage threshold and obstacle attributes connecting two target coordinate points.

[0062] The central node aggregates capability parameters from multiple heterogeneous edge robots in real time. These parameters include each robot's maximum physical payload, maximum speed, and the type of sensors it carries. The central node flattens the topology of the task graph into a one-dimensional graph feature vector and quantifies the capability parameters of each heterogeneous edge robot into a capability feature matrix. Let the total number of participating heterogeneous edge robots be... The capability feature dimension of each robot is: In this scheme, the capability characteristic dimension The preferred value is 128, then the capability feature matrix is ​​represented as follows: The central node performs a concatenation operation on the one-dimensional graph feature vector and the capability feature matrix, and generates a prompt word matrix through a linear mapping network. Let the prompt word matrix be... The splicing operator is The linear mapping weights are The bias term is The mapping process satisfies the formula:

[0063] ;

[0064] in the formula This is the feature vector of the one-dimensional graph after the task graph has been flattened. The center node will contain the prompt word matrix. The input is fed into a pre-trained large model, which calculates the conditional probability distribution based on an autoregressive decoding mechanism, generating a temporalized subtask sequence corresponding to the highest probability. Let the generated subtask sequence be... ,in Indicating timing steps Subtasks assigned to specific edge heterogeneous robots Given the total number of subtasks, the inference and decoding objective of the large model is to maximize the joint conditional probability:

[0065] ;

[0066] in the formula Indicate steps All previously generated historical subtask sequences.

[0067] The central node inputs the subtask sequence output by the large model into a graph structure model based on a directed acyclic graph (DAG) algorithm for hierarchical compilation. The graph structure model constructs a tree-like topology based on the temporal execution dependencies of each subtask in the subtask sequence, thereby generating a unique semantic state tree for each edge-heterogeneous robot. During the hierarchical compilation process, the graph structure model generates feature tuples for each node in the semantic state tree. Let the tree node be... Its characteristic tuple structure is defined as:

[0068] ;

[0069] in the formula A unique node identifier, generated based on a hash algorithm, is used to distinguish different execution nodes. The semantic triggering condition is the preceding multimodal environment feature vector that needs to be matched to activate the node. The action label maps to the desired linear and angular velocity command parameters of the underlying actuator. After compilation, the central node sends the corresponding semantic state tree to the local storage chip of each edge heterogeneous robot via wireless communication link for caching and retrieval, based on the network physical address.

[0070] This step combines the reasoning capabilities of large models with the physical properties of heterogeneous hardware to automatically decompose macroscopic and fuzzy natural language instructions into underlying structured and distributable control logic.

[0071] S2. The edge heterogeneous robot uses a local lightweight model to calculate the similarity between real-time environmental perception features and text features corresponding to semantic triggering conditions in each unexecuted tree node.

[0072] Furthermore, in step S2, the similarity calculation steps include:

[0073] A multimodal pre-trained encoder is used to extract features from real-time environment-aware features and map them to generate high-dimensional perception feature vectors.

[0074] By using a text encoder to embed and map the semantic triggering conditions in each unexecuted tree node, a high-dimensional text feature vector of the corresponding dimension is generated.

[0075] Perform a dot product operation between the high-dimensional perceptual feature vector and the high-dimensional text feature vector, and then normalize the result to quantify the similarity between the high-dimensional perceptual feature vector and the high-dimensional text feature vector.

[0076] Specifically, the edge heterogeneous robot uses its onboard vehicle-mounted vision camera and depth sensor to acquire real-time color images and depth / distance streams of its surrounding working environment, inputting these as real-time environmental perception features into its internal embedded computing unit. The edge heterogeneous robot then invokes a lightweight multimodal pre-trained encoder deployed locally on the computing unit to perform downsampling convolution and feature extraction on the real-time environmental perception features. Let the input color image tensor be... ,in The pixel height of the image. The pixel width of the image. Let be the number of color channels. A multimodal pre-trained encoder performs multi-layer nonlinear mapping on the color image tensor, outputting a high-dimensional perceptual feature vector representing the global semantics of the current physical environment. Let this high-dimensional perceptual feature vector be... ,in To embed features into a unified dimension as preset, in this scheme, feature embedding uses a unified dimension. The preferred value is 512.

[0077] The edge heterogeneous robot traverses the locally cached semantic state tree, extracting the semantic triggering conditions contained in each unexecuted tree node. These semantic triggering conditions are sequences of linguistic text describing the physical appearance or geometric attributes of the target object. The edge heterogeneous robot then uses a locally weighted text encoder to perform word segmentation and word embedding mapping on the linguistic text sequences. Let the... The text sequence corresponding to the semantic triggering conditions in the unexecuted tree nodes is: The text encoder maps the text sequence into a high-dimensional text feature vector. Let the generated high-dimensional text feature vector be... The dimension of the vector The dimensions are strictly consistent with those of the high-dimensional perceptual feature vectors to ensure that data from both the image and text modalities are projected into the same shared semantic metric space.

[0078] The computational unit of the edge heterogeneous robot calls the underlying tensor acceleration library to perform a dot product operation on the generated high-dimensional perceptual feature vector and high-dimensional text feature vector, and then performs L2 normalization processing to quantify the semantic overlap between the current real-time field of view and the node triggering conditions. Let's assume that for the... The similarity calculated from the unexecuted tree nodes is: The numerical calculation process of this similarity satisfies the formula:

[0079] ;

[0080] in the formula This represents the inner product of the high-dimensional perceptual feature vector and the high-dimensional text feature vector. The L2 norm of the high-dimensional perceptual feature vector is represented by the length of the L2 norm. This represents the L2 norm of the high-dimensional text feature vector. The similarity of the output is calculated. The numerical domain is normalized and restricted to between negative one and positive one. The closer the value is to positive one, the higher the semantic matching degree between the current robot's real environment and the preset triggering scenario of the node.

[0081] This step enables edge heterogeneous robots to achieve localized scene understanding and target recognition without relying on cloud computing power by aligning heterogeneous image sensing data and abstract text instructions into a unified high-dimensional space for distance measurement.

[0082] S3. The edge heterogeneous robot activates tree nodes whose similarity meets a preset threshold as target tree nodes, and converts the action labels within the target tree nodes into control commands to drive the actuators of the edge heterogeneous robot.

[0083] Further, in step S3, the step of activating tree nodes whose similarity meets a preset threshold as target tree nodes includes:

[0084] Obtain the current environmental disturbance parameters of the edge heterogeneous robot, and dynamically calculate and generate a preset threshold using a compensation function;

[0085] The similarity of each unexecuted tree node is compared with a preset threshold, and candidate tree nodes with similarity greater than the preset threshold are selected.

[0086] When there are multiple candidate tree nodes, the optimal candidate tree node with the highest similarity is selected and locked as the target tree node, and other candidate tree nodes with mutually exclusive branch relationships are frozen.

[0087] Furthermore, step S3, the step of converting the action labels within the target tree node into control commands, includes:

[0088] Perform syntax parsing on the action labels within the target tree node to extract basic action identifiers and associated spatial coordinate parameters;

[0089] Index the local skill library of the edge heterogeneous robot and match the underlying control model corresponding to the basic motion identifier;

[0090] Spatial coordinate parameters are injected into the underlying control model, and the driving parameters of the actuator are output using kinematic solving algorithms to generate control commands.

[0091] Specifically, the edge heterogeneous robot reads the underlying operating status of the low-level drive bus and vision sensor through its internal diagnostic module, obtains the average pixel illumination variance and body vibration frequency of the current field of view, and then weights and fuses these two data into environmental interference parameters. The edge heterogeneous robot uses an internally stored dynamic compensation function to calculate and generate a floating preset threshold. Let the basic matching threshold be... Environmental interference parameters are The preset threshold is The compensation coefficient is The calculation process of this compensation function satisfies the formula:

[0092] ;

[0093] Compensation coefficient in the formula The settings are based on the sensor hardware's factory calibration error. In this solution, the basic matching threshold... The preferred value is 0.80, compensation coefficient The preferred value is 0.05. The edge heterogeneous robot compares the similarity of each unexecuted tree node calculated in the previous steps with a preset threshold. Perform numerical comparison operations. The edge heterogeneous robot establishes a candidate memory queue, prioritizing those with similarity greater than a preset threshold. Tree node objects are pushed into a candidate memory queue to generate a candidate tree node set. When the candidate memory queue contains two or more candidate tree nodes, the edge heterogeneous robot calls a sorting algorithm to sort the queue in descending order according to the similarity value, directly pops the first node of the sequence (i.e., the optimal candidate tree node with the highest similarity value), and locks it as the target tree node for the current control cycle. The edge heterogeneous robot queries the topological adjacency table of the local state tree to find other candidate tree nodes that have the same parent node as the target tree node and have a logical mutual exclusion relationship, and overwrites the associated memory address status flag to a frozen locked state to avoid action conflicts in subsequent physical coordination logic.

[0094] The edge heterogeneous robot reads the action label string encapsulated in the target tree node's memory block and calls its built-in regular expression parsing engine to perform syntax parsing on the string. The edge heterogeneous robot segments the string according to a preset delimiter, extracting the basic action identifier representing the action category and the spatial coordinate parameters representing the target working point. Using the basic action identifier as a hash key, the edge heterogeneous robot performs a hash index lookup in the robot's underlying skill library built into the local storage medium, matching and retrieving the corresponding underlying control model and associated underlying driver script as the current instance control object.

[0095] The edge heterogeneous robot injects the extracted spatial coordinate parameters into the underlying instance control object and calls the built-in inverse kinematics algorithm to perform matrix calculation. Let the spatial coordinate parameters be the desired end-effector pose matrix in the three-dimensional workspace. The current actual end-effector pose matrix of the edge heterogeneous robot actuator is: The current servo joint state vector of the system is The kinematic Jacobian matrix of the edge heterogeneous robot is The kinematics solution algorithm calculates the pseudo-inverse of the Jacobian matrix and outputs the driving velocity vectors of each joint of the actuator as driving parameters. Let the driving velocity vector be... The proportional control gain diagonal matrix is In this scheme, the proportional control gain diagonal matrix The diagonal element is preferably 0.5, and its solution formula satisfies:

[0096] ;

[0097] in the formula This is the Morpence pseudo-inverse of the Jacobian matrix under the current joint state vector. The computational unit of the edge heterogeneous robot will solve for the obtained drive velocity vector. The control command is generated by encapsulating a data frame that conforms to the Controller Area Network Bus protocol and sending it to the lower-level servo motor driver through the internal communication bus, thereby driving the corresponding mechanism to perform displacement or rotation.

[0098] This step enables the transformation of abstract semantic decisions into control signals for underlying physical devices, ensuring the accuracy of motion response and the safety of collision avoidance for edge heterogeneous robots under complex environmental interference.

[0099] S4. When an abnormal communication with the central node is detected, each edge heterogeneous robot calculates the takeover weight based on its own capability parameters, local network connectivity, and semantic complexity of the unexecuted tree nodes, and recommends a pseudo-central node.

[0100] Further, step S4, the step of calculating the takeover weight, includes:

[0101] Traverse the local semantic state tree, count the hierarchical features of the unexecuted tree nodes, and use an accumulation algorithm to quantize and generate semantic complexity;

[0102] Send probe packets to other edge heterogeneous robots within the local communication network and calculate round-trip delays to construct an adjacency matrix to evaluate the connectivity of the local network;

[0103] Extract the hardware resource margin from the capability parameters, perform normalized fusion calculation with the local network connectivity and semantic complexity, and output the takeover weight.

[0104] Furthermore, step S4, the step of electing a pseudo-center node, includes:

[0105] Each edge heterogeneous robot is controlled to encapsulate the takeover weight into an election frame and broadcast it to other edge heterogeneous robots in the local communication network via an ad hoc networking protocol;

[0106] Within a preset time window, all campaign frames within the local communication network are received and parsed to construct a global ranking table sorted by weight values.

[0107] Extract the network address identifier of the edge heterogeneous robot with the highest weight in the global ranking table, and set the edge heterogeneous robot holding the network address identifier as the pseudo-central node.

[0108] Specifically, the edge heterogeneous robot continuously monitors heartbeat data packets with the central node through its underlying communication module. When the heartbeat data packet loss time exceeds a safety threshold (preferably 3 seconds in this scheme), the edge heterogeneous robot triggers a local weight calculation process. The edge heterogeneous robot traverses the locally cached semantic state tree, extracting all tree nodes marked as "not executed" to generate a set to be processed. The edge heterogeneous robot then calculates the depth level of each "not executed" tree node in the tree topology and the number of its branch child nodes, using an accumulation algorithm to quantify and generate the semantic complexity representing the remaining computational burden of the current task. Let the semantic complexity be... The set of unexecuted tree nodes is ,node The depth level is ,node The number of branches is The depth penalty factor is The branch penalty factor is The calculation process of this semantic complexity satisfies the formula:

[0109] ;

[0110] In this scheme, the depth penalty factor The preferred value is 0.2, the branch penalty factor. The preferred value is 0.5.

[0111] Edge heterogeneous robots use their underlying wireless radio frequency modules to send probe packets to other edge heterogeneous robots within the local communication network and record the system timestamp of the data packets. Upon receiving an acknowledgment packet from the corresponding node, the edge heterogeneous robot calculates the time difference between the two to obtain the round-trip delay. Based on the reciprocal relationship of the round-trip delay, the edge heterogeneous robot constructs an adjacency matrix reflecting the communication link quality to evaluate the local network connectivity. Let the local network connectivity be... The set of successfully connected neighbor nodes is , with neighboring nodes The round-trip delay is The calculation process of the connectivity of this local network satisfies the formula:

[0112] ;

[0113] The edge heterogeneous robot reads the underlying operating system bus interface and extracts the current remaining battery power percentage and CPU idle rate from the capability parameters as hardware resource reserves. The edge heterogeneous robot performs minimax normalization on the hardware resource reserves, local network connectivity, and semantic complexity, and then performs a weighted fusion calculation to output the takeover weight. To prioritize nodes with sufficient resources and smooth network connectivity, semantic complexity is included as a negative reduction term in the calculation. Let the takeover weight be... The normalized hardware resource margin is The normalized local network connectivity is The normalized semantic complexity is The forward fusion weight coefficients are respectively and The negative penalty coefficient is The output formula for the takeover weight satisfies:

[0114] ;

[0115] In this scheme, the forward fusion weight coefficient The preferred value is 0.4. The preferred value is 0.4, a negative penalty coefficient. The preferred value is 0.2.

[0116] The edge heterogeneous robot encapsulates its calculated takeover weight and its own network address identifier into a campaign frame containing a payload, and broadcasts it to other edge heterogeneous robots within the local communication network by invoking the underlying ad hoc network routing protocol. The edge heterogeneous robot starts an internal timer to define a preset time window; in this scheme, the preset time window is preferably 500 milliseconds. During the duration of the time window, it continuously receives and parses all campaign frames within the local communication network. The edge heterogeneous robot extracts the takeover weight values ​​from the payloads of all campaign frames and constructs a global ranking table in its local memory, sorted in descending order by weight value. When the timer overflows and the time window ends, the edge heterogeneous robot extracts the network address identifier of the edge heterogeneous robot at the top of the global ranking table, i.e., the edge heterogeneous robot with the highest weight. The system determines that the edge heterogeneous robot currently holding the network address identifier has the highest takeover priority, and thus sets this robot as the pseudo-central node for taking over control of the local communication network.

[0117] This step enables dynamic election by integrating multi-dimensional physical characteristics such as node communication capabilities and task load, ensuring seamless switching of the control center and system robustness in the extreme environment of central network outage in heterogeneous clusters.

[0118] S5. Each edge heterogeneous robot reports the node identifier of the target tree node to the pseudo-central node through the local communication network. The pseudo-central node performs pruning and replanning on the semantic state tree according to the node identifier and updates the semantic state tree synchronously.

[0119] Further, in step S5, the step of pruning the semantic state tree based on the node identifier includes:

[0120] Extract the received node identifiers at the pseudo-central node and construct a global progress set representing the task execution status;

[0121] The local semantic state tree is traversed using a graph search algorithm to match and locate the active branch that contains the corresponding node identifier in the global progress set.

[0122] Remove hierarchical dependency references from the activated branch and simultaneously delete alternative logical branches that have an exclusive relationship with the activated branch, generating a pruned semantic state tree.

[0123] Furthermore, step S5, the step of synchronizing and updating the semantic state tree, includes:

[0124] Extract all unexecuted tree nodes from the pruned semantic state tree, and combine them with the real-time physical state and capability parameters of each edge heterogeneous robot to construct a replanning optimization model;

[0125] The replanning optimization model is solved using a planning algorithm, and the execution sequence between unexecuted tree nodes is reconstructed to synthesize the updated semantic state tree;

[0126] The updated semantic state tree is converted into incremental data packets and distributed to the corresponding edge heterogeneous robots using a multicast protocol for cache overlay.

[0127] Specifically, each edge heterogeneous robot sends the node identifier of the currently active target tree node to the pseudo-central node in real time through the underlying local area network data distribution service. The pseudo-central node receives and extracts all the above node identifiers in its local memory buffer, and concatenates and combines these identifier fields to construct a global progress set representing the overall task execution progress of the current system.

[0128] The central processing unit of the pseudo-central node invokes a breadth-first graph search algorithm to traverse the topology of the locally cached semantic state tree layer by layer. The pseudo-central node performs a hash lookup match between the identifier of each traversed tree node and the identifier in the global progress set, accurately locating the active branch containing the matching identifier. The pseudo-central node then invokes underlying memory pointer manipulation instructions to set the dependency reference pointers of the parent nodes pointing to the active branch to null values, thus removing the dependency. The pseudo-central node further reads the mutex node vectors in the active branch attribute mapping table, locates and synchronously releases the storage space occupied by all candidate logical branches with mutually exclusive relationships, and generates a pruned semantic state tree after removing redundant structures and completed tasks.

[0129] The pseudo-central node traverses the pruned semantic state tree, extracting all tree nodes marked as unexecuted to generate a set of tasks to be assigned. The pseudo-central node collects the current 3D spatial coordinates of each edge heterogeneous robot as its real-time physical state through a local communication network and reads the limit movement speed from the capability parameters. The pseudo-central node constructs a replanning optimization model with the objective of minimizing the maximum task completion time of the entire cluster. Let the total number of unexecuted tree nodes in the set of tasks to be assigned be... The total number of edge heterogeneous robots participating in the collaboration is Boolean decision variables Indicate whether to... The unexecuted tree node is assigned to the first... For a heterogeneous edge robot, the objective function constructed by the pseudo-central node satisfies the following formula:

[0130] ;

[0131] in the formula For the first The current three-dimensional spatial coordinate vector of an edge heterogeneous robot For the first The target space coordinate vector bound to the unexecuted tree node. This represents the Euclidean linear distance between two coordinate vectors. For the first The scalar of the ultimate moving speed of an edge heterogeneous robot.

[0132] The pseudo-central node uses an internally deployed heuristic programming algorithm to iteratively solve the aforementioned replanning optimization model, outputting the optimal allocation matrix that converges the objective function. In this scheme, the maximum number of iterations of the heuristic programming algorithm is preferably 200. Based on the optimal allocation matrix, the pseudo-central node reconstructs the execution sequence and pre-triggered dependencies of each unexecuted tree node on the time axis, writing the new topology connections into the data structure to synthesize the updated semantic state tree. The pseudo-central node uses a differential coding algorithm to calculate the difference binary stream between the updated semantic state tree and the original version, converting it into lightweight incremental data packets. The pseudo-central node calls the underlying multicast communication protocol stack to broadcast the incremental data packets to the corresponding edge heterogeneous robots. After receiving the packets, the edge heterogeneous robots directly decompress them and overwrite the old version of the state tree cache in their local memory.

[0133] This step enables progress synchronization of the edge cluster and secondary optimal allocation of residual tasks when the main network is disconnected, ensuring the task continuity of the heterogeneous system under conditions of no central control.

[0134] Example 2

[0135] Taking a search and rescue mission in an earthquake-stricken area as an example, the disaster area deployed a command vehicle as the central node and three heterogeneous robots, including drones, robotic-armed unmanned vehicles, and robotic dogs. After inputting instructions to search for trapped personnel, clear rubble, and drop first-aid kits, the central node, combining the capabilities of the drones (high speed), the unmanned vehicles (heavy load capacity), and the robotic dogs (flexibility), used a large model to generate and distribute three semantic state trees, assigning tasks for high-altitude reconnaissance, heavy object clearing, and first-aid kit drop, respectively.

[0136] The drone uses a local model to process images. When it detects trapped personnel and rubble, the similarity between perceived features and textual features reaches 0.85, exceeding the pre-set threshold of 0.7 after compensation. The drone immediately activates the target tree node, drives the hovering marker, and synchronizes the coordinate status with the unmanned vehicle and robot dog.

[0137] Aftershocks caused three robots to lose network access from the central node, triggering local takeover. When calculating weights: the drone had only 20% battery remaining and low network connectivity; the unmanned vehicle had 90% battery, a high CPU idle rate, and was in a communication relay position, resulting in the highest connectivity. After normalized fusion calculation, the unmanned vehicle had the highest takeover weight and was nominated as the pseudo-central node through a broadcast election frame.

[0138] After receiving the marker indicating a casualty, the autonomous vehicle acting as a pseudo-central node prunes the state tree and removes invalid global search branches. It then extracts two unexecuted tasks: clearing debris and distributing first-aid kits. Based on the current coordinates and speed, it re-plans: assigning the nearest and most load-matching autonomous vehicle to clear the debris, and the fastest robot dog to distribute the first-aid kit. The updated state tree is then converted into incremental data packet multicast. Finally, under complete network outage conditions, the autonomous vehicle removes the debris, and the robot dog distributes the first-aid kit, achieving cluster self-healing and collaborative closed-loop operation even in network outages.

[0139] The table below shows a quantitative performance comparison between the method of this invention and a traditional centralized cloud-based collaborative control method based on a master-slave control architecture of a robot operating system under the same heterogeneous cluster size and network outage test environment:

[0140] Table 1. Comparison of Quantitative Performance Tests of Heterogeneous Robot Collaborative Control Systems

[0141] Performance evaluation indicators Traditional centralized collaborative control methods The collaborative control method of the present invention Task completion rate under network outage at the center Shutdown between 0% and 12% 94.5% Self-healing and reconfiguration time after network outage It cannot heal on its own and requires manual intervention to reposition. 1.35 seconds Heterogeneous device instruction conversion error rate 18.4% poor physical parameter adaptation 1.2%

[0142] As shown in the table above, traditional centralized control methods suffer from a sharp drop in task completion rate to 0% to 12% when communication is interrupted due to the complete loss of the cloud scheduling hub, and the system struggles to self-heal. Furthermore, the instruction translation error rate is as high as 18.4% due to the differences in heterogeneous hardware. In contrast, this invention introduces a pseudo-central node dynamic election mechanism based on multi-dimensional hardware weights and a local replanning mechanism for the state tree. This allows the cluster to seamlessly transfer control and self-heal in just 1.35 seconds after a complete network outage, significantly increasing the task completion rate to 94.5% in extreme environments. Moreover, by combining the edge-side semantic state tree with the local underlying skill library and kinematic solutions, the underlying physical adaptation problem of heterogeneous devices is effectively improved, reducing the instruction translation error rate to 1.2%. This fully demonstrates the superiority of this method in breaking cloud dependence and ensuring high stability and high self-healing collaboration of heterogeneous clusters.

[0143] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A heterogeneous robot cooperative control method based on a large model, characterized in that, include: S1. The central node uses a large model to combine the capability parameters of multiple edge heterogeneous robots to decompose the task, generate and distribute the corresponding semantic state tree to the cache of each edge heterogeneous robot. The semantic state tree contains multiple tree nodes with semantic triggering conditions, action tags and node identifiers. S2. The edge heterogeneous robot uses a local lightweight model to calculate the similarity between real-time environmental perception features and text features corresponding to semantic triggering conditions in each unexecuted tree node; S3. The edge heterogeneous robot activates the tree node whose similarity meets a preset threshold as the target tree node, and converts the action label in the target tree node into a control command to drive the actuator of the edge heterogeneous robot. S4. When an abnormal communication with the central node is detected, each edge heterogeneous robot calculates the takeover weight based on its own capability parameters, local network connectivity, and semantic complexity of the unexecuted tree node, and recommends a pseudo-central node. S5. Each of the edge heterogeneous robots reports the node identifier of the target tree node to the pseudo-central node through a local communication network. The pseudo-central node performs pruning and replanning on the semantic state tree according to the node identifier and updates the semantic state tree synchronously.

2. The heterogeneous robot cooperative control method based on a large model according to claim 1, characterized in that, In step S1, the step of decomposing the task includes: Receive global task instructions and use an entity extraction model to transform the global task instructions into a task graph that includes environmental and spatial constraints. The task graph and the capability parameters are fused into a prompt word matrix, which is then input into the large model inference to generate a temporalized subtask sequence. The subtask sequence is hierarchically compiled using a graph structure model to construct a semantic state tree that includes the semantic triggering conditions, the action tags, and the node identifiers.

3. The heterogeneous robot cooperative control method based on a large model according to claim 1, characterized in that, In step S2, the similarity calculation step includes: The real-time environment perception features are extracted using a multimodal pre-trained encoder and mapped to generate a high-dimensional perception feature vector. The semantic triggering conditions in each unexecuted tree node are embedded and mapped using a text encoder to generate a high-dimensional text feature vector of the corresponding dimension. Perform a dot product operation between the high-dimensional perceptual feature vector and the high-dimensional text feature vector, and then normalize the result to quantify the similarity between the high-dimensional perceptual feature vector and the high-dimensional text feature vector.

4. The heterogeneous robot cooperative control method based on a large model according to claim 1, characterized in that, Step S3, the step of activating the tree node whose similarity satisfies the preset threshold as the target tree node, includes: Obtain the current environmental interference parameters of the edge heterogeneous robot, and dynamically calculate and generate the preset threshold using a compensation function; The similarity of each of the unexecuted tree nodes is compared with the preset threshold, and candidate tree nodes with similarity greater than the preset threshold are selected. When there are multiple candidate tree nodes, the optimal candidate tree node with the highest similarity is selected and locked as the target tree node, and other candidate tree nodes with mutually exclusive branch relationships are frozen.

5. The heterogeneous robot cooperative control method based on a large model according to claim 1, characterized in that, Step S3, the step of converting the action label within the target tree node into a control command, includes: The action labels within the target tree node are parsed to extract basic action identifiers and associated spatial coordinate parameters; Index the local skill library of the edge heterogeneous robot and match the underlying control model corresponding to the basic motion identifier; The spatial coordinate parameters are injected into the underlying control model, and the driving parameters of the actuator are output using a kinematics solving algorithm to generate the control command.

6. The heterogeneous robot cooperative control method based on a large model according to claim 1, characterized in that, Step S4, the step of calculating the takeover weight, includes: Traverse the local semantic state tree, count the hierarchical features of the unexecuted tree nodes, and use an accumulation algorithm to quantize and generate the semantic complexity; Send probe packets to other edge heterogeneous robots within the local communication network and calculate round-trip delays to construct an adjacency matrix to evaluate the connectivity of the local network; Extract the hardware resource margin from the capability parameters, perform normalized fusion calculations on the hardware resource margin, the local network connectivity, and the semantic complexity, and output the takeover weight.

7. The heterogeneous robot cooperative control method based on a large model according to claim 1, characterized in that, Step S4, the step of recommending a pseudo-center node, includes: Each of the aforementioned edge heterogeneous robots is controlled to encapsulate the takeover weight into an election frame and broadcast it to other edge heterogeneous robots within the local communication network via an ad hoc networking protocol; Within a preset time window, all campaign frames within the local communication network are received and parsed to construct a global ranking table sorted by weight values. Extract the network address identifier of the edge heterogeneous robot with the highest weight in the global ranking table, and set the edge heterogeneous robot holding the network address identifier as the pseudo-center node.

8. The heterogeneous robot cooperative control method based on a large model according to claim 1, characterized in that, In step S5, the step of pruning the semantic state tree according to the node identifier includes: Extract the received node identifiers at the pseudo-central node to construct a global progress set representing the task execution status; The local semantic state tree is traversed using a graph search algorithm to match and locate the activated branch that contains the corresponding node identifier in the global progress set; Remove the hierarchical dependency references of the activated branch and simultaneously delete the alternative logical branches that have an exclusive relationship with the activated branch, generating a pruned semantic state tree.

9. The heterogeneous robot cooperative control method based on a large model according to claim 1, characterized in that, Step S5, the step of synchronizing and updating the semantic state tree, includes: Extract all unexecuted tree nodes from the pruned semantic state tree, and combine the real-time physical state of each edge heterogeneous robot with the capability parameters to construct a replanning optimization model; The replanning optimization model is solved using a planning algorithm to reconstruct the execution sequence between the unexecuted tree nodes and synthesize the updated semantic state tree. The updated semantic state tree is converted into incremental data packets and distributed to the corresponding edge heterogeneous robots for cache overlay using a multicast protocol.

Images