A distributed heterogeneous multi-robot cooperative exploration method based on visual language model

Abstract

The application discloses a distributed heterogeneous multi-robot cooperative exploration method based on a visual language model, relates to the technical field of robot autonomous exploration, and aims to solve the following problems in the prior art: centralized global map communication and storage overhead is large; differences in the capabilities of heterogeneous robots are not fully used for task decomposition and cooperation; a visual language model is difficult to simultaneously process exploration, navigation and operation in a large-scale unknown environment; multi-robots are prone to inconsistent situation awareness under asynchronous communication conditions; and there is a lack of an effective assistance mechanism when a robot that discovers a task target cannot complete target operation.

Description

Technical Field

[0001] This invention relates to the field of autonomous robot exploration technology, specifically to a distributed heterogeneous multi-robot collaborative exploration method based on a visual language model. Background Technology

[0002] With the development of large language models and visual language models, robots are gradually acquiring the ability to understand task semantics based on natural language instructions, understand environmental states based on visual images, and generate high-level decisions. Compared with traditional rule-based robot systems, visual language models can associate fuzzy instructions such as "find drinking water," "find survivors," and "check if there are any operable targets in the room" with actual visual scenes, thereby providing semantic reasoning capabilities for robots to perform tasks in complex environments.

[0003] However, relying solely on a single robot or centralized planning approach remains significantly insufficient in large-scale, unknown environments. First, single-robot systems exhibit low exploration efficiency and long completion times in long-term tasks, and struggle to simultaneously cover multiple potential target areas. Second, centralized multi-robot systems typically require maintaining a complete global map and collecting all sensor data, leading to high consumption of communication bandwidth, storage space, and computing resources; centralized mapping and decision-making can easily become bottlenecks when the network is unstable or the environmental scale increases. Third, existing visual-language model-based multi-robot methods often prioritize navigation or target search, lacking a mechanism to unify long-distance exploration with local physical operations. For example, drones can quickly observe high places and traverse narrow areas, but cannot perform contact operations such as opening doors or carrying things; mobile robotic arms possess operational capabilities, but their movement speed and field of view are limited. If task allocation and assistance requests cannot be based on heterogeneous capabilities, the system is prone to problems such as "detecting a target but being unable to process it" or "capable robots not knowing the target's location."

[0004] Furthermore, existing methods often rely on pre-constructed global maps or complete semantic maps. While complete semantic maps are rich in information, they require the continuous storage of large amounts of point clouds, meshes, images, and semantic descriptions in unknown large-scale environments. This increases both the communication burden and the input length and inference latency of the visual language model. Directly inputting large amounts of historical images and point clouds into the visual language model also increases the probability of misidentification and hallucination decisions. Therefore, how to enable heterogeneous robots to form a sufficiently consistent situational awareness under limited communication and storage conditions, and to utilize visual language models to complete exploration, navigation, and operational decisions, remains a problem that current technology has not yet fully solved. Summary of the Invention

[0005] This invention aims to provide a distributed heterogeneous multi-robot collaborative exploration method based on a visual language model to address the following problems in existing technologies: centralized global map communication and storage overhead is large; the differences in capabilities of heterogeneous robots are not fully utilized for task decomposition and collaboration; visual language models have difficulty handling exploration, navigation, and manipulation simultaneously in large-scale unknown environments; multiple robots are prone to inconsistent situational awareness under asynchronous communication conditions; and there is a lack of effective assistance mechanisms when a robot that has discovered the task target is unable to complete the target operation.

[0006] The technical solution of this invention is as follows: a distributed heterogeneous multi-robot cooperative exploration method based on a visual language model, wherein each robot maintains a local observation space, an internal memory state space, and a high-level semantic action space; specifically including the following steps:

[0007] S1: Each robot acquires local observations at the current moment, which include at least panoramic visual images, geometric observations, and robot poses;

[0008] S2: Each robot maintains its internal memory state locally, which includes the overall task, the set of subtasks obtained by decomposition, historical interaction information, robot swarm capability description, and minimap.

[0009] S3: Based on the overall task, the set of subtasks, the robot swarm capability description, and the initial local observations of each robot, perform task decomposition and capability matching through a visual language model or a language model, and distribute the subtask set that has been confirmed by feedback to each robot.

[0010] S4: On each robot, the internal memory state and the current local observation are input into the visual language model, and the visual language model outputs a high-level semantic action. The high-level semantic action includes at least a decision mode, a task target detection flag, a task completion flag, a target description, a target global position, a target point sent to the lower-level planner, and an assistance request.

[0011] S5: When the visual language model determines that there is a task-related target or a task-related scene, the global position of the target area is calculated based on the current robot's local observation; when the current robot has the ability to perform the corresponding operation, the global position is converted into a task-oriented target point; when the current robot does not have the ability to perform the corresponding operation, an assistance request including a target description and target position is generated and sent asynchronously to other robots with the corresponding capabilities.

[0012] S6: When the visual language model does not detect a task-related target or task-related scene, the visual language model selects the next exploration target point based on the minimap, historical interaction information, geometric exploration candidate points, and semantic exploration candidate points.

[0013] S7: The underlying motion planning module converts the task-oriented target point or the next exploration target point into continuous executable motion commands, and drives the corresponding robot to complete navigation, exploration or physical operation;

[0014] S8: Each robot asynchronously broadcasts its small map increment, scene language description, exploration status, assistance requests, and task progress to other robots through the communication unit, and integrates the asynchronous updates of other robots, so that each robot can form a nearly consistent global situational awareness without the need for a centralized complete global map.

[0015] The heterogeneous multi-robot ensemble includes at least two aerial robots and one ground mobile robot with navigation capabilities. Each robot includes a visual perception unit, a geometric perception unit, a computing unit, a communication unit, and a motion execution unit.

[0016] The local observation is defined as:

[0017]

[0018] in, Indicates the robot number. Indicates time, Represents robots At any moment The captured panoramic visual images, This represents geometric observations in the form of laser point clouds, depth maps, or locally occupied grids. This indicates the robot's pose.

[0019] The internal memory state is defined as follows:

[0020]

[0021] in, Indicates the overall task. Represents a set of subtasks. Represents robots Historical missions and communication information This describes the capabilities of a robot swarm. Represents robots The small map being maintained.

[0022] The internal memory state is updated in the following manner:

[0023]

[0024] in, Represents robots The memory update function, Indicates a high-level semantic action. Represents robots Received asynchronous communication information from other robots.

[0025] The high-level semantic action is defined as:

[0026]

[0027] in, As a decision-making model, For detecting markers related to task objectives or scenarios, This is a sign that the task is complete. For target description, For the target global position, To send the target point to the underlying planner, To assist with the request.

[0028] The target point sent to the underlying planning module satisfies:

[0029]

[0030] in, To guide the target point of the task. This will be the next target point for exploration.

[0031] The minimap is a two-dimensional image, a two-dimensional grid map, or a combination thereof, and includes at least one of the following: explored traversable areas, obstacle areas, room segmentation and semantic description, unexplored areas, robot swarm positions, geometric exploration candidate points, semantic exploration candidate points, and the next exploration target point; the minimap is encoded using color blocks, symbol blocks, text labels, or a combination thereof, enabling the visual language model to use it as image input for global situational understanding.

[0032] The fusion update of the small map includes at least one or more of the following relationships:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] in, This indicates that a passable area has been explored. Indicates the area of ​​obstacles. Indicates room area, Indicates unexplored areas. Indicates the location of the robot swarm. Indicates the set of next target points for the robot swarm; superscript Indicates local observation. This indicates information from other robots.

[0039] The geometric exploration candidate points are calculated from the boundary relationships between the locally occupied grid, the explored passable area, the obstacle boundary, the room segmentation area, and the unexplored area. The semantic exploration candidate points are identified from the panoramic visual image by the visual language model. The semantic exploration candidate points include doors, corridor entrances, occluded areas, potential task-related objects, or scene structures that can lead to unknown areas.

[0040] The visual language model performs motion decisions locally on each robot and satisfies:

[0041]

[0042] in, The visual language model strategy is represented; the underlying motion planning module converts the high-level semantic actions into continuous motion instructions.

[0043]

[0044] in, Indicates a continuous motion command. Represents robots Experts in low-level, non-learning actions.

[0045] When performing task decomposition, a task decomposition unit first generates a set of candidate subtasks based on the overall task, the capability descriptions of each robot, and initial visual observations. Then, the set of candidate subtasks is sent to each robot for feedback confirmation. When the feedback results from each robot meet the consistency condition, the final set of subtasks is determined. When the feedback results do not meet the consistency condition, the set of candidate subtasks is regenerated or modified.

[0046] In the assistance request processing, the robot that initiates assistance retains the target location that cannot be completed as a semantic exploration candidate point on the minimap and broadcasts the assistance request to robots with corresponding capabilities; the robot that receives the assistance request converts the target location into its own task-oriented target point, and reaches the target location to perform the operation through global minimap path search and local motion planning.

[0047] The asynchronous broadcast includes an incremental message with robot ID, timestamp, coordinate transformation information, and update type; the receiver performs fusion, deduplication, and conflict resolution on the received updates based on at least one of timestamp, spatial distance threshold, semantic category consistency (observed environmental semantic description), and coordinate transformation consistency (cluster global position).

[0048] The heterogeneous multi-robot system includes at least one aerial robot with navigation capabilities and at least one mobile robot with navigation capabilities.

[0049] The beneficial effects of this invention are:

[0050] 1) This invention significantly reduces storage and communication overhead by replacing the complete semantic map or complete point cloud map with a lightweight minimap. The minimap only retains the exploration area, obstacles, room semantics, robot position, geometric exploration candidate points, and semantic exploration candidate points that are useful for decision-making, enabling the visual language model to understand the global situation in a shorter context, thereby reducing inference latency and the risk of illusion.

[0051] 2) This invention employs a distributed asynchronous update mechanism, allowing each robot to independently complete perception, memory updates, visual language model decisions, and underlying motion planning locally. Even with communication delays or packet loss, the robot can still continue performing tasks based on its local memory. Upon receiving incremental updates from other robots, the updates are then fused and deduplicated, thus the system exhibits good scalability and robustness.

[0052] 3) This invention projects both geometric and semantic exploration points onto a small map. Geometric exploration points reflect accessible and informative unknown boundaries, while semantic exploration points reflect semantically meaningful areas such as doors, corridor entrances, occluded areas, or potential task targets. The visual language model can integrate geometric accessibility, task semantics, and robot swarm status to select exploration targets, thereby improving exploration efficiency in unknown environments.

[0053] 4) This invention introduces a capability matching and assistance request mechanism. The robot that discovers the target does not necessarily have to complete the operation independently, but rather determines whether it can handle it based on its own capabilities. If it cannot handle it, it sends the target location, target description, and operational requirements to a robot with the capability to operate it. This enables heterogeneous collaboration, such as UAVs handling rapid search and target localization, and mobile robotic arms handling opening doors or manipulating objects.

[0054] 5) This invention unifies task decomposition, visual understanding, exploration point selection, assistance requests, and operational decisions within a high-level decision-making framework driven by a visual language model, while entrusting continuous motion to a mature planner for execution, thus possessing both semantic reasoning capabilities and engineering feasibility.

[0055] 6) Typical application scenarios of the present invention include, but are not limited to: item search and door operation in home or office environments, multi-robot inspection and navigation in complex indoor scenarios such as hospitals, factories, and underground spaces, survivor search and material search in post-disaster ruins or dangerous areas, and autonomous collaboration of heterogeneous robot swarms under conditions of limited communication. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the overall architecture for distributed heterogeneous multi-robot collaborative exploration in the embodiment.

[0057] Figure 2 This is a schematic diagram of the information elements and projection relationships of the small map in the embodiment.

[0058] Figure 3 The flowchart for motion decision-making, assistance request, and asynchronous update based on the visual language model is shown in the embodiment. Detailed Implementation

[0059] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Any equivalent substitutions or improvements made to robot types, sensor types, visual language model types, communication methods, planner types, and small map resolutions within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0060] Example 1: System Composition

[0061] Construct a robot swarm consisting of multiple heterogeneous robots. The swarm includes two or more drones with navigation capabilities, one or more mobile robotic arms, and one or more legged robots. Drones are suitable for rapid aerial observation, traversing narrow areas, and acquiring panoramic images; mobile robotic arms are suitable for performing contact operations such as opening doors, pushing and pulling, grasping, pressing, or moving objects; legged robots are suitable for mid-range navigation, close-range observation, and localized operations in complex ground environments. Each robot is equipped with a visual perception unit, a geometric perception unit, a computing unit, a communication unit, and a motion execution unit. The visual perception unit can be a panoramic camera, a fisheye camera, a surround-view camera, or a camera group consisting of multiple ordinary cameras; the geometric perception unit can be a LiDAR, a depth camera, a stereo camera, a local occupancy grid generation module, or a combination thereof. The computing unit is used to run the visual language model interface, memory update algorithm, minimap construction algorithm, and motion planning algorithm; the communication unit is used to send and receive minimap increments, scene descriptions, exploration status, assistance requests, and task progress.

[0062] Figure 1 This is a schematic diagram of the overall architecture for distributed heterogeneous multi-robot collaborative exploration. At the start of the task, a visual language model or language model is used to perform a hybrid task decomposition based on the overall task, robot swarm capability description, and initial visual observations. Feedback from each robot is used to confirm and reduce task decomposition errors.

[0063] During the task execution phase, each robot does not rely on a centralized complete global map, but maintains a lightweight small map and structured memory state locally. Each robot periodically inputs its visual images, geometric observations, task memory, and small map into the visual language model locally, and outputs high-level semantic actions. The underlying planning module generates continuous executable control based on the high-level semantic actions. Robots exchange small maps, scene descriptions, exploration points, assistance requests, and task progress through asynchronous incremental communication, thereby forming a nearly consistent global situational awareness.

[0064] Lightweight minimaps are used as a shared visual cognitive interface for robot swarms, enabling visual language models to directly understand exploration areas, obstacles, room semantics, robot states, geometric exploration points, and semantic exploration points. Distributed asynchronous updates replace centralized full map transmission. Capability matching and assistance request mechanisms connect navigation robots with manipulating robots. Visual language models simultaneously perform task target detection, semantic exploration point recognition, exploration point selection, capability assessment, and assisted decision-making.

[0065] The heterogeneous robot set is represented as The system state space is represented as Each robot maintains a local observation space, an internal memory state space, and a high-level semantic action space. The system can be represented as:

[0066]

[0067] in, Represents the environment state transition function, This represents the memory update function. This represents a function that evaluates the benefits or costs of a task. Indicates the discount factor. This represents the maximum task duration. This model is used to characterize the state, memory, decision-making, and communication relationships of multi-robot collaborative exploration and operation; in practice, high-level strategies are implemented using a visual language model and cue word templates, while low-level continuous motion is implemented using a learning-free planner.

[0068] The local observation is defined as:

[0069]

[0070] The internal memory state is defined as follows:

[0071]

[0072] The high-level semantic action is defined as:

[0073]

[0074] The visual language model strategy satisfies:

[0075]

[0076] Since high-level semantic actions cannot be directly executed by the robot, they are converted into continuous motion instructions by low-level motion experts or motion planners.

[0077]

[0078] The system can optimize between task completion rate and execution cost, and its objective function can be expressed as:

[0079]

[0080] in, Given the joint environment state, the payoff function for a single robot can be expressed as:

[0081]

[0082] in, Indicates the cost of execution. This indicates the weight of the reward for completing the subtask. Indicates an indicator function, Indicates the first Each subtask is in status The following has been completed or satisfied.

[0083] Example 2: Task Decomposition and Capability Matching

[0084] When the task begins, the system receives the overall task. The overall task can be vague natural language instructions such as "find survivors", "find water or food", "check all rooms for dangerous objects", or "find and open the door to the target area".

[0085] The system first obtains the capability descriptions of each robot. This includes each robot's movement method, reachable area, sensor type, whether it has a robotic arm, whether it can open doors, whether it can move objects, maximum speed, maximum load, and safety limitations. Subsequently, the task decomposition unit inputs the overall task, capability description, and initial visual observations into a visual language model or language model to obtain a set of candidate subtasks. For example, for the task of "finding survivors", candidate subtasks could include "drones quickly scanning corridors and room entrances from above", "ground mobile robots inspecting obstructed areas", and "mobile robotic arms moving to the door and operating the doorknob after receiving a door opening request".

[0086] To reduce illusory decisions caused by single-step model inference, this embodiment employs a feedback confirmation mechanism. The candidate subtask set is not executed immediately but is sent to each robot for reflection and feedback. Each robot returns an agreement, modification suggestion, or reason for rejection based on its own capability description, initial observations, and task reachability. When the feedback results meet the consistency condition, the final subtask set is determined. The state is written into the internal memory of each robot; when the feedback result does not meet the consistency condition, the task decomposition unit regenerates or corrects the subtask set based on the feedback.

[0087] Example 3: Lightweight Smallmap Construction

[0088] like Figure 2 As shown, each robot maintains a lightweight minimap. The minimap can be a fixed-resolution 2D RGB image, or a combination of a 2D raster structure and a visualized image. In this embodiment, the minimap uses color blocks or symbol blocks to express different information: explored traversable areas. Obstacle Area Room segmentation and semantic description Unexplored areas robot swarm location Geometric exploration candidate points Semantic exploration candidate points And the next exploration target of the robot swarm .

[0089] The minimap merging and updating can be performed as follows:

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096] in, and Obtained from the robot's local geometric observations. and Received via broadcast from other robots. Room partitioning. This can be achieved through obstacle boundaries, connected components, morphological segmentation, or geometric region division methods on the minimap. For each room region, the robot inputs its panoramic visual image into the visual language model to generate a concise natural language description, and attaches the description most recently or most relevant to that room region.

[0097] Geometric exploration candidate points Candidate points for semantic exploration can be extracted from the boundary between explored and unexplored areas, and filtered based on obstacle distance, accessibility, distance to other robot-explored targets, room connectivity, and the area of ​​unknown regions. Objects that can be identified in panoramic images by a visual language model include doors, corridor entrances, room entrances, areas behind obstructions, half-open passages, potential task objects, or structures that may lead to unexplored areas. Geometric exploration candidate points and semantic exploration candidate points are used together for subsequent exploration target selection.

[0098] Example 4: Motion Decision Making Using Visual Language Model

[0099] like Figure 3 As shown, within each decision cycle, the robot The internal memory state and local observations are input into the visual language model. To reduce the complexity of one-time reasoning, motion decisions can be broken down into multiple stages.

[0100] In the first stage, the visual language model is based on the overall task. Subtask set Historical Information Ability Description and the current panoramic image Determine if any task-related targets or scenarios have been detected. If detected, output a detection flag. and target description If not found, output .

[0101] The second stage, when At this time, the robot performs visual localization of the target region in the image. Visual localization can be achieved through bounding boxes, selected regions, segmentation masks, or text region descriptions output by a visual language model. This visual region is then aligned with LiDAR or depth observation to estimate the target depth, and the robot's pose is then considered. Convert to global coordinates .

[0102] In the third stage, the robot determines whether it can handle the target based on the capability description. If the robot possesses the corresponding capability, then... The execution mode is set up, and a task-oriented target point is generated from the target's global location. If the robot does not currently possess the corresponding capability, then set... In the request-for-help mode, generate a help request. And add the target location to the semantic exploration candidate point set. They await the arrival and processing of robots with the corresponding capabilities.

[0103] In the fourth stage, when At this time, the robot enters exploration mode. The visual language model reads the minimap. Task information and historical information are used to explore candidate points geometrically. With semantic exploration candidate points Select the next exploration target point When making a selection, the visual language model can consider the following factors: whether the target point is located at the boundary of an unknown area, whether it is close to a door or corridor entrance that may lead to other rooms, whether it overlaps with the targets of other robots, whether it is semantically related to the current task, and whether it meets the current robot's reachability and capability constraints.

[0104] The above motion decision-making process can be summarized as follows:

[0105]

[0106] in, This indicates the generation of task-oriented target points. Indicates the selection of exploration points. This indicates the generation of an assistance request. An assistance request itself does not necessarily become the current robot's motion target; instead, it is sent to other robots as an asynchronous message and a semantic exploration candidate point.

[0107] Example 5: Underlying Planning and Execution

[0108] The visual language model outputs high-level semantic actions, which are not directly used as input for robot control. The underlying motion planning module determines the target point... Local geometric observation and minimap Generate continuous executable motion instructions.

[0109] For distant targets, this embodiment first displays them on the minimap. A global path search is performed, using methods such as A-star search, D-star search, topological path search, or sampled graph search, to obtain a global reference path composed of several intermediate waypoints. The robot then uses a local motion planner combined with real-time geometric observations for obstacle avoidance and trajectory generation. For UAVs, the local planner can output the position, velocity, acceleration, and yaw angle trajectory; for mobile robotic arms, it can output the chassis path and the robotic arm's operating trajectory; for legged robots, it can output the footing point, center of mass trajectory, and torso posture target.

[0110] When a task involves physical manipulation, the system invokes the corresponding operation control module based on the target type and the robot's capabilities. For example, when a drone detects a closed door but cannot open it, it generates an assistance request. Upon receiving the request, the mobile robotic arm plans a path to the door based on its global position and performs actions such as door handle positioning, end effector alignment, gripping, rotation, or pushing / pushing. After the operation is completed, the mobile robotic arm broadcasts the task progress and environmental changes to other robots, who then update their minimaps and continue exploring.

[0111] Example 6: Asynchronous Communication, Fusion and Deduplication

[0112] This embodiment does not require all robots to wait synchronously for the same global clock. Each robot performs perception, memory updates, visual language model inference, and motion planning at its own frequency, and broadcasts incremental information asynchronously. Each message includes robot number, timestamp, coordinate system identifier, local-to-global coordinate transformation, minimap increment, room semantic description, geometric exploration point, semantic exploration point, robot's current position, next exploration target, assistance request, and task completion status.

[0113] When the receiver merges messages, it can determine the age of the information based on the timestamp, project other robots' information onto its own minimap using coordinate transformation, remove duplicate exploration points, duplicate obstacles, and duplicate target locations based on spatial distance thresholds, and merge identical doors, rooms, or objects based on semantic category consistency. For conflicting information, such as different robots having inconsistent judgments on whether the same area is passable, it can be handled using methods such as prioritizing the most recent observation, prioritizing obstacles, weighting confidence, or voting based on multiple observations.

[0114] The asynchronously merged global minimap approximately satisfies:

[0115]

[0116] This means that different robots do not need to be completely synchronized with limited communication latency, but their understanding of explored areas, obstacles, exploration points, and task objectives remains approximately consistent. This avoids the single-point bottleneck of centralized maps and reduces the probability of robots repeatedly exploring or interfering with each other due to inconsistent situations.

[0117] Example 7: Task Completion Determination

[0118] After updating their internal memory, each robot checks its task completion status based on the overall task, the set of subtasks, and historical information. Task completion can be determined by a rule module, a visual language model, or a combination of both. For example, for the task of "finding survivors," once any robot confirms finding a survivor and broadcasts their location, the corresponding subtask can be marked as completed. For the task of "finding drinking water," once a robot identifies a water bottle, cup, or water dispenser in the room and locates its position, the target search subtask can be marked as completed. For the task of "opening a door to a target area," once the robot opens the door and confirms that the passage is passable, the operation subtask can be marked as completed.

[0119] Task completion sign High-level semantic actions and historical information can be written into the data and broadcast to other robots via asynchronous communication. When all necessary subtasks are completed, or the overall task meets the preset termination conditions, the robot swarm stops exploring and outputs the final result.

[0120] In this embodiment, the visual language model can be a locally deployed model, a cloud model, or an edge server model. Model input can include a single panoramic image, a multi-view image sequence, an image stitched with a small map, a textual small map summary, or a combination thereof. The small map resolution can be adaptively set according to the robot swarm size and environmental area, and is not limited to a fixed-size image. Small map encoding methods can include color encoding, symbol encoding, layer encoding, sparse raster encoding, or topological graph encoding. Geometric exploration candidate points can be obtained through frontal detection, ray casting, room boundary detection, topological node expansion, or probabilistic information gain calculation; semantic exploration candidate points can be obtained through a visual language model, object detection model, segmentation model, or multi-model fusion. The underlying motion planner is not limited to A-star, EGO-type local planning, trajectory optimization, or mobile robotic arm planning; it can also be replaced by sampling planning, model predictive control, topological path planning, or a learning-based low-level controller. Asynchronous communication can employ broadcast, point-to-point, publish-subscribe, opportunistic communication, or relay communication methods; message fusion can employ recent update priority, confidence weighting, multi-robot voting, or graph optimization methods. The above are merely exemplary embodiments of the present invention and do not constitute any limitation on the scope of protection of the present invention. All technical solutions formed by equivalent exchange or substitution fall within the scope of protection of this invention.

Claims

1. A distributed heterogeneous multi-robot collaborative exploration method based on a visual language model, characterized in that, Each robot maintains a local observation space, an internal memory state space, and a high-level semantic action space; specifically, this includes the following steps: S1: Each robot acquires its local observations at the current moment. S2: Each robot maintains its internal memory state locally, which includes the overall task, the set of subtasks obtained by decomposition, historical interaction information, robot swarm capability description, and minimap. S3: Based on the overall task, the set of subtasks, the robot swarm capability description, and the initial local observations of each robot, perform task decomposition and capability matching through a visual language model or a language model, and distribute the subtask set that has been confirmed by feedback to each robot. S4: On each robot, the internal memory state and the current local observation are input into the visual language model, and the visual language model outputs high-level semantic actions. S5: When the visual language model determines that there is a task-related target or a task-related scene, the global position of the target area is calculated based on the current robot's local observation; when the current robot has the ability to perform the corresponding operation, the global position is converted into a task-oriented target point; when the current robot does not have the ability to perform the corresponding operation, an assistance request including a target description and target position is generated and sent asynchronously to other robots with the corresponding capabilities. S6: When the visual language model does not detect a task-related target or task-related scene, the visual language model selects the next exploration target point based on the minimap, historical interaction information, geometric exploration candidate points, and semantic exploration candidate points. S7: The underlying motion planning module converts the task-oriented target point or the next exploration target point into continuous executable motion commands, and drives the corresponding robot to complete navigation, exploration or physical operation.

2. The distributed heterogeneous multi-robot collaborative exploration method based on a visual language model according to claim 1, characterized in that, The local observation includes at least panoramic visual images, geometric observations, and robot pose; specifically defined as: ； in, Indicates the robot number. Indicates time, Represents robots At any moment The captured panoramic visual images This represents geometric observations in the form of laser point clouds, depth maps, or locally occupied grids. This indicates the robot's pose.

3. The distributed heterogeneous multi-robot collaborative exploration method based on a visual language model according to claim 2, characterized in that, The internal memory state is defined as follows: ； in, Indicates the overall task. Represents a set of subtasks. Represents robots Historical missions and communication information This describes the capabilities of a robot swarm. Represents robots The small map being maintained.

4. The distributed heterogeneous multi-robot collaborative exploration method based on a visual language model according to claim 3, characterized in that, The internal memory state is updated in the following manner: ； in, Represents robots The memory update function, Indicates a high-level semantic action. Represents robots Received asynchronous communication information from other robots.

5. The distributed heterogeneous multi-robot collaborative exploration method based on a visual language model according to claim 4, characterized in that, The high-level semantic action is defined as: ； in, As a decision-making model, For detecting markers related to task objectives or scenarios, This is a sign that the task is complete. For target description, For the target global position, To send the target point to the underlying planner, To assist with the request.

6. The distributed heterogeneous multi-robot collaborative exploration method based on a visual language model according to claim 5, characterized in that, The target point sent to the underlying planning module satisfies: ； in, To guide the target point of the task. This will be the next target point for exploration.

7. The distributed heterogeneous multi-robot cooperative exploration method based on a visual language model according to claim 6, characterized in that, The minimap is a two-dimensional image, a two-dimensional raster map, or a combination thereof, and includes at least one of the following: explored traversable areas, obstacle areas, room segmentation and semantic description, unexplored areas, robot swarm positions, geometric exploration candidate points, semantic exploration candidate points, and the next exploration target point.

8. The distributed heterogeneous multi-robot cooperative exploration method based on a visual language model according to claim 7, characterized in that, The fusion update of the small map includes at least one or more of the following relationships: ； ； ； ； ； in, This indicates that a passable area has been explored. Indicates the area of ​​obstacles. Indicates room area, Indicates unexplored areas. Indicates the location of the robot swarm. Indicates the set of next target points for the robot swarm; superscript Indicates local observation. This indicates information from other robots.

9. The distributed heterogeneous multi-robot collaborative exploration method based on a visual language model according to claim 8, characterized in that, The visual language model performs motion decisions locally on each robot and satisfies: ； in, The visual language model strategy is represented; the underlying motion planning module converts the high-level semantic actions into continuous motion instructions. ； in, Indicates a continuous motion command. Represents robots Experts in low-level, non-learning actions.

10. The distributed heterogeneous multi-robot cooperative exploration method based on a visual language model according to claim 9, characterized in that, In the assistance request processing, the robot that initiates assistance retains the target location that cannot be completed as a semantic exploration candidate point on the minimap and broadcasts the assistance request to robots with corresponding capabilities; the robot that receives the assistance request converts the target location into its own task-oriented target point, and reaches the target location to perform the operation through global minimap path search and local motion planning.

Images