A multi-robot cooperative navigation method and system based on a visual language large model
By combining a large visual language model with a multi-robot system, efficient task allocation and collaboration in complex dynamic environments are achieved. This solves the problem of collaborative navigation and task allocation in complex environments for multi-robot systems in existing technologies, and improves the system's adaptability and task execution efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TONGJI UNIV
- Filing Date
- 2024-12-25
- Publication Date
- 2026-07-24
AI Technical Summary
Existing multi-robot collaborative navigation systems struggle to achieve efficient task allocation and collaboration in complex and dynamic environments. They lack the ability to intelligently analyze and flexibly execute complex task instructions, and the interaction between robots and operators is particularly limited in dynamic environments.
A method based on a large visual language model is adopted. The large language model is used to parse user task instructions, and combined with the robot semantic map and visual perception module, multimodal feature fusion is achieved, environmental changes are monitored in real time, and dynamic task adjustments are made through human-computer interaction.
It improves the collaborative efficiency and task execution flexibility of multi-robot systems in complex environments, achieves highly accurate navigation and task allocation in complex environments, reduces operational difficulty, and improves the system's adaptability and timeliness.
Smart Images
Figure CN119756375B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multi-robot cooperative navigation technology, and in particular to a multi-robot cooperative navigation method and system based on a large visual language model. Background Technology
[0002] With the rapid development of artificial intelligence and robotics, multi-robot systems have received widespread attention and application in many fields, such as smart manufacturing, smart warehousing, indoor cleaning, logistics and distribution, and disaster relief. Multi-robot collaborative work can accomplish tasks that are difficult for a single robot to complete independently, significantly improving task execution efficiency and system adaptability. However, it also presents challenges in achieving efficient collaboration, task allocation, and navigation planning among robots in complex and dynamic environments.
[0003] Traditional multi-robot cooperative navigation and task allocation methods largely rely on predetermined rules, static maps, and limited sensor information. These methods are typically limited to fixed task scenarios and lack real-time response capabilities to dynamic environments, resulting in poor system flexibility and adaptability when facing changes in tasks or environments. Furthermore, existing multi-robot systems often depend on simple sensor data (such as LiDAR and ultrasonic sensors) and hard-coded task allocation strategies, exhibiting limited understanding of complex or unknown environments and lacking the ability to intelligently interpret and flexibly execute complex task instructions. Especially in complex task scenarios, understanding and executing natural language instructions from operators, and dynamically adjusting and allocating tasks based on the capabilities, states, and environmental changes of different robots, are key challenges in current multi-robot cooperative systems.
[0004] Although deep learning and natural language processing technologies have made significant progress in recent years, research on combining natural language with visual information to form efficient and intelligent large-scale visual-language models and applying them to multi-robot cooperative navigation and task allocation is still in its early stages. Most existing systems can only process simple, fixed instructions when parsing task commands, making it difficult to cope with the complex and varied command requirements put forward by users in dynamic environments, and failing to fully leverage their advantages in multimodal information fusion.
[0005] To address these issues, recent research has attempted to introduce visual language models and human-computer interaction mechanisms to enhance the intelligence and flexibility of robotic systems. By combining visual information and natural language understanding, robots can more accurately understand the operator's intentions and make real-time task adjustments and path planning. However, current technologies still have the following shortcomings: Firstly, the application of existing visual language models in multi-robot collaborative tasks is still immature, making it difficult to achieve real-time and effective task allocation and collaboration in complex task environments; secondly, the interaction between robots and operators remains relatively limited, making it difficult to efficiently support dynamic task allocation and real-time feedback.
[0006] Therefore, there is an urgent need for a new method to improve the collaborative efficiency and task execution flexibility of multi-robot systems in complex dynamic environments by integrating large-scale visual language models and human-computer interaction mechanisms. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a multi-robot cooperative navigation method and system based on a large visual language model.
[0008] The objective of this invention can be achieved through the following technical solutions: As a first aspect of the present invention, a multi-robot cooperative navigation method based on a large visual language model is provided, comprising the following steps: Based on the task instructions input by the user, combined with the semantic map constructed by each robot, the large language model parses the instructions and assigns the sub-task instructions to the robot. Each robot acquires environmental images in real time and maps them into visual embeddings using the visual perception module. At the same time, the assigned sub-task language instructions are mapped into language embeddings. The visual embedding and language embedding are aligned across modal features to obtain visual-language multimodal fusion features. These features are then input into a large visual-language navigation model specific to each robot for analysis, predicting and generating navigation actions for each robot, which are then executed by the robot. The robot monitors task progress and environmental changes in real time. If a new task or environmental change is detected, the operator can adjust the task and handle the anomaly. If the task instructions change, the robot can dynamically adjust its execution strategy based on the operator's feedback and update its status. Repeat the above steps until each robot successfully completes its assigned sub-task and reaches the target location.
[0009] As a preferred technical solution, the specific steps for each robot to construct a semantic map are as follows: In the initial state, the robot rotates and captures RGB images and depth maps in real time through the camera every 30 degrees. For RGB images, RGB features are extracted using a pre-trained ResNet50-based visual feature extraction network. For depth maps, depth features are extracted through point cloud generation. The RGB features and depth features are then jointly modeled by point cloud projection to form RGB-D joint features. Based on RGB-D joint features, the decoder of a pre-trained semantic segmentation network upsamples the RGB-D joint features layer by layer, classifies each pixel of the RGB-D joint features, and generates a semantic segmentation map. The label map and depth map after pixel classification are transformed into 3D point cloud, and the semantic point clouds of multiple frames are fused to generate a 3D scene model with semantic labels. A semantic map is created by projecting a semantically labeled 3D scene model onto a 2D grid map using 2D grid projection processing.
[0010] As a preferred technical solution, the steps of parsing the user-input task instructions and assigning sub-task instructions to the most suitable robot by the large language model are as follows: The large language model GPT3.5 is used to replace the scene prior. Based on the robot's position, the scattered semantic maps are merged to obtain a pieced semantic map, which includes robot position, scene structure, scene objects and target category information, and is made into prompts in a fixed format. Based on the prompts and the current state of each robot, the large language model GPT3.5 acts as a global planner according to the task instructions input by the user, breaking down the total task into the most suitable sub-tasks for each robot.
[0011] As a preferred technical solution, the robot maps real-time acquired environmental images into visual embeddings using a visual perception module, including: A visual encoder using DINOv2 and SigLIP pre-trained features extracts RGB visual embeddings. Extracting deep visual embeddings using the InceptionNet network; The assigned subtask language instructions are mapped to language embeddings. Specifically, the task instructions are converted into word embeddings of the corresponding global vectors, and the language embeddings are extracted using a pre-trained BERT language model.
[0012] As a preferred technical solution, the construction steps of the visual-language multimodal fusion features are as follows: For the RGB visual embedding and deep visual embedding in the visual embedding, a cross-attention mechanism is used to align the cross-modal features with the language embedding to obtain RGB-language multimodal features. and deep-language multimodal features The RGB-language multimodal features and the deep-language multimodal features are concatenated together and then input into the Llama2 large language model to predict the robot's next navigation action.
[0013] As a preferred technical solution, the visual language navigation large model is based on the Llama2 large language model and adopts joint training of multiple modalities. The specific training process is as follows: The historical video information and current frame video information of the robot navigation video, as well as the natural language instructions of the subtask, are used as inputs to the Llama2 large language model; Based on the processing of robot navigation video by image encoder, special lexical units are used to identify different information in order to distinguish which modality the current input lexical unit sequence comes from; The collected robot visual language navigation data is fed into the robot visual language navigation big model for training: the continuous measurement range of robot navigation actions is discretized into multi-dimensional action lexical units, covering the lexical units used in the Llama vocabulary; the visual language big model is trained using the standard next lexical unit prediction target, and the cross-entropy loss on the predicted action lexical units is evaluated. Based on different task scenarios, robot visual language navigation trajectories are collected, and LoRA is used to fine-tune the large model of robot visual language navigation.
[0014] As a preferred technical solution, the visual language navigation data is collected in the Habitat simulator, as follows: Semantic segmentation is performed in the Habitat simulator. The path planning algorithm is used to find the shortest path from the starting point to the target point. Based on the semantic information on the path, the navigation trajectory is labeled with natural language using GPT3.5 to obtain video-language data pairs. Given a command, collect the image sequence in the corresponding navigation trajectory and answer the corresponding next action; or given an image of a trajectory, answer the corresponding command.
[0015] As a preferred technical solution, the real-time monitoring of task progress and environmental changes allows the operator to adjust tasks and handle anomalies if new tasks or environmental changes are detected, as detailed below: The robot detects its current status and environmental information, and transmits the current observations to the host computer. The host computer displays the observation data and current task execution progress of each robot in real time. When new obstacles are detected in the environment, changes in the target position, or abnormal situations such as delays, failures, or sudden task requirements occur in the task, a warning is issued and the environmental change information and task abnormality information are fed back to the operator. The operator can intervene by inputting new instructions or adjusting the existing task plan, and the operator's modifications are synchronized to the robot system in real time. The robot's visual language navigation big model receives feedback from the operator, parses and generates new task strategies through the big language model, adjusts the execution logic according to the new task strategies, and updates its own status in real time during the execution of the adjusted tasks; the system synchronously updates the task status and semantic map of all robots.
[0016] As a second aspect of the present invention, a multi-robot cooperative navigation system based on a large visual language model is provided, employing the above-described method for multi-robot cooperative navigation based on a large visual language model, the system comprising: The multi-robot task allocation module is used to parse high-level task instructions input by users based on visual language multimodal representation information, analyze the task instructions through a large language model and decompose them into multiple sub-tasks, and allocate the sub-task instructions to the most suitable robot. The visual perception and semantic understanding module is used to perform visual perception and semantic understanding on multimodal input data based on environmental information collected by the robot, including RGBD visual images and natural language instructions corresponding to sub-tasks, and to extract key feature information. The multimodal information fusion module is used to fuse and align visual and linguistic features to generate a unified multimodal representation, and to encode and represent heterogeneous input data in a unified manner. The robot visual language navigation module is used to analyze and predict the navigation actions of each robot based on the fused visual language multimodal representation information and the robot's own large-scale visual language navigation model. The robot then executes the action and updates its state. The dynamic feedback and real-time optimization module is used to monitor the robot status and task progress in real time based on the status of each robot. When a new task or environmental change is detected, it prompts the operator to adjust the task and handle the anomaly. The robot dynamically adjusts its execution strategy and updates its status based on the operator's feedback. The real-world deployment verification module is used to verify and optimize the performance of multi-robot systems in actual working environments based on the developed multi-robot collaborative navigation method.
[0017] As a preferred technical solution, the visual perception and semantic understanding module includes: The visual feature extraction unit is used to extract RGB visual features from the RGBD visual images acquired by the robot using DINOv2 and SigLIP, and to extract depth visual features using the InceptionNet network for subsequent navigation and task decision-making. The language embedding generation unit is used to perform semantic parsing and representation of task instructions based on user input and sub-task instructions parsed and split by GPT3.5. It uses the pre-trained language model BERT to generate language embedding information with high semantic consistency that can be recognized by the model.
[0018] Compared with the prior art, the present invention has the following beneficial effects: 1) This invention introduces a large-scale visual-language model, achieving deep fusion of visual and linguistic multimodal data, thus endowing robots with stronger environmental perception and task understanding capabilities. Through the combination of vision and language, collaboration and dynamic task allocation among multiple robots are realized, and real-time monitoring and adjustment are possible via human-computer interaction. Compared to traditional systems based on a single sensor, robots can comprehensively understand the environment and task requirements through multimodal perception, thereby improving their adaptability to complex environments and achieving high accuracy in navigation and task allocation.
[0019] 2) The task allocation module of this invention can analyze task requirements in real time, dynamically adjust task allocation strategies, and prioritize the completion of critical tasks based on real-time environmental information, task priorities, and robot status. Compared with traditional static or predefined rule-based task allocation methods, this invention achieves intelligent allocation of task resources, avoids resource waste and task conflicts, and effectively improves the timeliness and accuracy of the overall system task completion.
[0020] 3) This invention provides a human-computer interaction module based on natural language and a visual interface, introducing a safety monitoring and dynamic adjustment mechanism. When the robot encounters obstacles, path conflicts, or other emergencies, it can issue real-time alarms and execute emergency strategies. Simultaneously, operators can monitor and operate the robot in real-time through voice commands, text input, or a graphical interface. Compared to existing technologies that rely on fixed commands or limited feedback, the human-computer interaction of this invention is more intuitive, efficient, and reliable, significantly reducing operational difficulty. Attached Figure Description
[0021] Figure 1 A flowchart illustrating a multi-robot cooperative navigation method based on a large visual language model, provided in an embodiment of the present invention; Figure 2 A flowchart illustrating another multi-robot cooperative navigation method based on a large visual language model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a multi-robot cooperative navigation system based on a large visual language model, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of another multi-robot cooperative navigation system based on a large visual language model provided in an embodiment of the present invention; In the attached diagram, 10 is the multi-robot task allocation module, 20 is the visual perception and semantic understanding module, 21 is the visual feature extraction unit, 22 is the language embedding and generation unit, 30 is the multimodal information fusion module, 40 is the robot visual language navigation module, 50 is the dynamic feedback and real-time optimization module, and 60 is the actual deployment and verification module. Detailed Implementation
[0022] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0023] Example 1 This embodiment relates to a method for multi-robot task allocation and cooperative navigation based on a large visual language model. This method is applied to multi-robot cooperative navigation tasks with dynamic task allocation, such as... Figure 1 As shown, the method specifically includes the following steps: Step S101: Based on the task instructions input by the user, and combined with the semantic map constructed by each robot, the large language model parses the sub-task instructions and assigns them to the most suitable robot.
[0024] The task instructions input by the user are natural language instructions, which need to be semantically understood and converted into instructions by the natural language processing module. Specifically, the user can input task instructions in the form of text or voice through the human-computer interaction interface. The final input to the model is a natural language instruction, such as "transport item A from location X to location Y".
[0025] The semantic map constructed by the robot refers to the semantic map generated by the robot after rotating 360° at the initial moment based on the RGBD visual image captured by the camera, through feature extraction, SegNet network semantic segmentation, 3D point cloud transformation, and 2D mesh projection processing.
[0026] In this embodiment of the invention, at the initial state, the robot rotates 360° and acquires RGB images and depth maps in real time every 30 degrees using a camera. The RGB images contain color information of the scene, and the depth maps provide distance information from each pixel to the camera. For the RGB images, RGB features are extracted using a pre-trained ResNet50 visual feature extraction network. For the depth images, depth features are extracted using point cloud generation, and the RGB and depth information are jointly modeled using point cloud projection to form a more robust feature representation. Based on these RGB-D joint features, a pre-trained SegNet semantic segmentation network is used to classify each pixel in the image, generating a semantic segmentation map. These processes can be represented as follows: , (1) in, and These represent the RGB image and depth map captured by the robot, respectively. The RGB image features can be extracted using the ResNet50 network. Depth map features can be extracted by processing the depth map through the PointNet network. .
[0027] (2) in, This indicates concat feature concatenation, which combines RGB image features. and depth map features After splicing, the RGB-D joint features can be obtained. .
[0028] (3) in, Representing image pixels, the decoder of the SegNet semantic segmentation network processes the RGB-D joint features. Upsampling layer by layer yields the results for each pixel. Category tags .
[0029] By combining the labeled map and depth map after pixel classification, the semantic point clouds of multiple frames are fused and mapped to generate a 3D scene model with semantic labels. This model is then projected onto a 2D grid map to obtain the required semantic map.
[0030] After parsing the user's input task instructions, the large language model assigns sub-task instructions to the most suitable robot. This means that the GPT3.5 large language model, which combines the semantic map built for each robot and has global information, can assign the most suitable sub-task instructions to each robot based on its current position and state.
[0031] In this embodiment of the invention, considering that learning-based robot visual language navigation methods typically require significant computational resources to learn and utilize environmental priors, and that large language models themselves contain rich knowledge about the world, the large language model GPT3.5 is chosen to replace scene priors. Based on the robot's position, fragmented semantic maps are merged to obtain a complete semantic map, which includes information such as robot position, scene structure, scene objects, and target categories, and is formatted into prompts. Based on these prompts, and the current state of each robot (battery level, task load, etc.), the large language model acts as a global planner according to the user's input task instructions, breaking down the overall task into the most suitable sub-tasks for each robot, such as "Robot 1 is responsible for moving item A to the transfer location," and "Robot 2 is responsible for moving it from the transfer location to the exit."
[0032] In step S102, each robot acquires environmental images in real time and maps them into visual embeddings using the visual perception module, while simultaneously mapping the assigned subtask language instructions into language embeddings.
[0033] Real-time acquisition of environmental images and mapping them to visual embeddings using the visual perception module refers to: inputting the RGB visual images acquired by the robot into the visual encoder of DINOv2 and SigLIP pre-trained features to extract RGB visual embeddings, and inputting the depth images into the InceptionNet network to extract depth visual embeddings; mapping the assigned subtask language instructions to language embeddings refers to extracting language embeddings after semantic understanding by the BERT pre-trained natural language processing module.
[0034] In this embodiment of the invention, based on the RGBD visual images collected by the robot and the assigned subtask instructions, the corresponding image signals and text signals are extracted to word units using an image encoder and a text encoder, respectively.
[0035] Specifically, in robot visual-language navigation tasks, the model needs to utilize historical information to understand which part of the instruction it has completed. For the current frame, it not only needs to provide the latest scene information but also predict the next reasonable action that aligns with the instruction. This significant difference in the role of image signals requires us to encode historical video information and current frame video information separately. Regarding image encoding, a visual encoder using DINOv2 and SigLIP pre-trained features extracts RGB visual embeddings. The fusion of low-level spatial information from DINOv2 and high-level semantics from SigLIP helps achieve visual generalization; the InceptionNet network extracts deep visual embeddings. Regarding text encoding, a BERT pre-trained language model extracts language embeddings. These processes can be represented as: (4) The RGB visual embeddings are extracted using visual encoders that utilize DINOv2 and SigLIP pre-trained features, respectively. [ ] indicates concat feature concatenation to obtain the RGB visual embeddings. .
[0036] (5) The InceptionNet network is used to extract deep visual embeddings. .
[0037] (6) First, the task instructions are converted into corresponding GLoVE embeddings (word embeddings of global vectors), encoded as follows: Where T represents the instruction length, the extracted language embeddings are then obtained using the BERT language model. .
[0038] Step S103: The visual and language embeddings are input into the large language model specific to each robot for parsing, and the navigation actions of each robot are predicted and generated. The robot then executes the actions.
[0039] The Transformer-based cross-attention mechanism aligns and concatenates visual and linguistic embeddings to obtain multimodal features. When input into a large robot visual-linguistic navigation model using the Llama2 large language model, it can generate the robot's next navigation action, such as moving forward or turning.
[0040] The robot's visual embeddings are mapped to the same format as the language embeddings through a multi-layer perceptron network. These multimodal visual-language data are first aligned across modalities using a Transformer-based cross-attention mechanism. , (7) in, This indicates that the cross-attention mechanism based on Transformer can be used to obtain RGB-language multimodal features. Deep-language multimodal features Finally, these multimodal features are concatenated and input into the Llama2 large language model to predict the robot's next navigation action.
[0041] In this embodiment of the invention, by using robot navigation video (historical video information and current frame video information) and subtask natural language instructions as input to the Llama2 large language model, the robot's underlying execution commands can be directly output.
[0042] Specifically, based on the image encoder processing of robot navigation videos, special lexical units are used to identify different information, such as historical information ([HIS] and [ / HIS]), the current frame ([OBS] and [ / OBS]), and the instruction [NAV], so that the model can still understand data from common modalities. These identifiers help the large model distinguish which modality the current input lexical sequence comes from, enabling joint training of multiple modalities. This joint training method can avoid the problem of catastrophic forgetting, maintaining an understanding of the natural world while learning navigation data.
[0043] First, considering the difficulty, high cost, and limited diversity of collecting real-world data, visual-language navigation data is collected in the Habitat simulator: given a command, the system collects the image sequence of the corresponding navigation trajectory and responds with the next action; or given an image of a trajectory, the system responds with the corresponding command. Specifically, semantic segmentation is first performed in the Habitat simulator, and the A* path planning algorithm is used to find the shortest path from the starting point to the target point. Based on the semantic information on the path, GPT3.5 is used to perform natural language tagging on these navigation trajectories, resulting in video-language data pairs.
[0044] Then, the collected robot visual language navigation trajectories are fed into a large visual language model for training. The DINOv2 and SigLIP visual encoders, BERT language model, and Llama2 large language model are all pre-trained, and fine-tuning is performed on these pre-trained models instead of training from scratch. The continuous metric range of robot navigation actions (forward, turn) is discretized into 256-dimensional action lexical units, covering the 256 least used lexical units in the Llama vocabulary. The large visual language model is trained using a standard next-next-lexical prediction objective, evaluating only the cross-entropy loss on the predicted action lexical units. The robot navigation "stop" action is directly represented by a single lexical unit (0 for no stop, 1 for stop).
[0045] Finally, 50-100 robot visual language navigation trajectories are collected based on different task scenarios, and LoRA is used to fine-tune the large visual language model. This allows the large visual language model to fuse multimodal information, analyze the environment and task requirements, and generate the next navigation action. The robot performs motion control based on the generated navigation instructions and simultaneously provides feedback on its execution status.
[0046] Step S104: The system monitors task progress and environmental changes in real time. If a new task or environmental change is detected, the system will prompt the operator to adjust the task and handle the anomaly, including: First, the system continuously collects the robot's current status (such as position, battery level, and task completion rate) and environmental information (such as obstacles and dynamic changes) through a sensor network and a visual perception module. The system displays the task and environmental status dynamically updated on the user interface in real time and intuitively presents the robot's position and environmental changes. Then, the system has a built-in anomaly detection module. When anomalies are detected, such as the addition of obstacles in the environment, changes in the target position, or sudden task requirements, the system will issue reminders to the operator in the form of sound, images, and text, prompting the operator to adjust the task and handle the anomaly. Finally, the operator intervenes through the system interface to adjust task objectives, priorities, or replan the path. The system supports natural language input and click operations to ensure that the adjustment process is fast and efficient. The operator's modifications will be synchronized to the robot system in real time to update task allocation and navigation strategies and avoid execution interruption.
[0047] Specifically, these indoor robots can transmit their current observations to a host computer via WiFi wireless communication technology. The host computer is a human-computer interaction interface written using the MFC application framework, which can display the observation data of each robot in real time (such as first-person visual images, battery level, any load, etc.) and the current task execution progress. The robot can detect new obstacles, dynamic targets, or path blockages in the environment. If delays, failures, or other abnormalities occur in the task, the system will issue warning prompts and provide feedback on environmental changes or task abnormalities to the operator through the interface or voice. The operator can intervene by inputting new commands or adjusting the existing task plan.
[0048] Step S105: If a change occurs, the robot dynamically adjusts its execution strategy based on operator feedback and updates its status. This includes: if the system detects a change in the task or environment, it synchronizes the new task objective to the affected robots, which then dynamically adjust their strategies based on operator feedback. For example, the robot may replan its path, adjust task priorities, or reassign tasks to other robots. During execution, the robot updates its status based on feedback and continues to perform new navigation actions.
[0049] Specifically, the robot's visual language navigation big data model receives feedback from the operator (such as reassigning task objectives and adjusting paths), parses it using the big data language model, and generates new task strategies. Based on the new task strategies, the robot adjusts its execution logic, such as replanning navigation paths and modifying task priorities. During the execution of the adjusted tasks, the robot updates its own status in real time, including position, battery level, and task completion progress. The system synchronously updates the task status and semantic maps of all robots to ensure consistency in collaboration among multiple robots.
[0050] Example 2 This embodiment also relates to a method for multi-robot task allocation and cooperative navigation based on a large visual language model. It introduces a novel multi-robot task allocation method based on a large language model, which can assign the most suitable sub-tasks according to the different states of each robot. Simultaneously, this invention introduces a novel robot navigation large model based on visual language multimodality as an action decision strategy. This strategy can be fine-tuned with a small amount of data and generalized to different robot navigation scenarios. Furthermore, this invention introduces a human-robot task cooperative decision-making mechanism and a dynamic task adjustment mechanism, aiming to further improve the execution flexibility and task completion efficiency of the robot system in complex environments. This embodiment is applicable to more complex multi-robot collaborative scenarios, such as disaster relief, industrial production, and logistics distribution, further expanding the scope of application and practical application value of this invention.
[0051] Figure 2 This is a flowchart of another method for multi-robot task allocation and cooperative navigation based on a large visual language model according to an embodiment of the present invention. Figure 2 As shown, the method specifically includes the following steps: Step S201: Obtain the current state of each robot in the scene, and the user inputs natural language instructions in the form of text or voice.
[0052] In step S202, based on the semantic map constructed by each robot, the multi-robot task allocation module can parse the task instructions into sub-task instructions and allocate them to the most suitable robot.
[0053] Step S203: Input the visual images and subtask instructions acquired by the robot into the visual perception and semantic understanding module to obtain visual embedding and language embedding.
[0054] Step S204: Input the visual embedding and language embedding into the multimodal information fusion module to obtain the visual-language multimodal fusion features.
[0055] In step S205, based on the visual language multimodal fusion features, the robot visual language navigation module can predict and generate the robot's next action.
[0056] In step S206, the robot interacts with the environment based on decision-making actions and updates its current state.
[0057] Step S207: The system obtains the latest status of the robot and displays it on the screen.
[0058] Step S208: Determine whether each robot has completed the sub-task. If not, proceed to step S209; if completed, proceed to step S212.
[0059] Step S209: Determine whether the task or environment has changed. If it has not changed, proceed to step S203 to perform iterative calculations. If it has changed, proceed to step S210.
[0060] In step S210, the operator can adjust tasks and handle exceptions through the dynamic feedback and real-time optimization module, thereby updating the overall task instructions or sub-tasks.
[0061] Step S211: Determine whether the overall task has changed. If it has changed, jump to step S201 to perform iterative calculation; if it has not changed, jump to step S203 to perform iterative calculation.
[0062] In step S212, the system terminates the current multi-robot collaborative navigation task and generates a task report and execution log.
[0063] Example 3 This embodiment also relates to a multi-robot task allocation and cooperative navigation system based on a large visual language model, which is applied to multi-robot cooperative navigation tasks with dynamic task allocation.
[0064] like Figure 3 As shown, the system includes a multi-robot task allocation module 10, a visual perception and semantic understanding module 20, a multimodal information fusion module 30, a robot visual language navigation module 40, a dynamic feedback and real-time optimization module 50, and a real-machine deployment and verification module 60.
[0065] The multi-robot task allocation module 10 is used to parse high-level task instructions input by the user based on visual language multimodal representation information, analyze the task instructions through a large language model and decompose them into multiple sub-tasks, and allocate the sub-task instructions to the most suitable robot.
[0066] Specifically, the visual perception and semantic understanding module 20 is used to perform visual perception and semantic understanding on multimodal input data based on environmental information collected by the robot, including RGBD visual images and natural language instructions corresponding to sub-tasks, and to extract key feature information.
[0067] The multimodal information fusion module 30 is used to perform fusion and alignment based on visual features and language features to generate a unified multimodal representation. Its main purpose is to encode and represent heterogeneous input data in a unified manner so that information from different modalities can complement and enhance each other.
[0068] The robot visual language navigation module 40 is used to parse and predict navigation actions for each robot based on the fused visual language multimodal representation information through a dedicated large language model (robot visual language navigation large model), and the robot executes the action and updates its state.
[0069] The dynamic feedback and real-time optimization module 50 is used to monitor the robot status and task progress in real time based on the status of each robot. When a new task or environmental change is detected, it will prompt the operator to adjust the task and handle the anomaly. The robot will dynamically adjust its execution strategy and update its status based on the operator's feedback information.
[0070] The actual deployment verification module 60 is used to verify and optimize the performance of the multi-robot system in a real working environment based on the developed multi-robot task allocation and cooperative navigation method. This ensures the feasibility, robustness, and efficiency of the multi-robot task allocation and cooperative navigation method based on the visual language large model in real application scenarios, providing a reliable basis for further industrial deployment and promotion.
[0071] Example 4 This embodiment also relates to a system for multi-robot task allocation and cooperative navigation based on a large visual language model. It introduces a novel multi-robot task allocation method based on a large language model, which can assign the most suitable sub-task to each robot based on its state. Simultaneously, this invention introduces a novel robot navigation large model based on visual language multimodality as an action decision strategy. This strategy can be fine-tuned with a small amount of data and generalized to different robot navigation scenarios. Furthermore, this invention introduces a human-robot task collaborative decision-making mechanism and a dynamic task adjustment mechanism, aiming to further improve the robot system's execution flexibility and task completion efficiency in complex environments. This embodiment is applicable to more complex multi-robot collaborative scenarios, such as disaster relief, industrial production, and logistics distribution, further expanding the scope of application and practical value of this invention.
[0072] Optionally, Figure 4 This is a schematic diagram of another multi-robot task allocation and cooperative navigation system based on a large visual language model according to an embodiment of the present invention. Figure 4 As shown, the visual perception and semantic understanding module 20 includes a visual feature extraction unit 21 and a language embedding generation unit 22.
[0073] Specifically, the visual feature extraction unit 21 is used to extract RGB visual features based on the RGBD visual images acquired by the robot, using DINOv2 and SigLIP, and to extract depth visual features using the InceptionNet network. These high-dimensional visual features can be used for subsequent navigation and task decision-making.
[0074] The language embedding generation unit 22 is used to perform semantic parsing and representation of the task instructions based on user input and the sub-task instructions after parsing and splitting by GPT3.5, and to generate language embedding information with high semantic consistency that can be recognized by the model.
[0075] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0076] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0077] The processing unit executes the various methods and processes described above, such as methods S101-S105 and S201-S210. For example, in some embodiments, methods S101-S105 and S201-S210 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S101-S105 and S201-S212 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S101-S105 and S201-S212 by any other suitable means (e.g., by means of firmware).
[0078] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0079] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0080] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0081] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A multi-robot cooperative navigation method based on a large visual language model, characterized by the following steps: include: Based on the task instructions input by the user, combined with the semantic map constructed by each robot, the large language model parses the instructions and assigns the sub-task instructions to the robot. Each robot acquires environmental images in real time and maps them into visual embeddings using the visual perception module. At the same time, the assigned sub-task language instructions are mapped into language embeddings. Visual embeddings and language embeddings are aligned across modalities to obtain visual-language multimodal fusion features. The construction of these features is as follows: RGB visual embeddings and deep visual embeddings are aligned across modalities with language embeddings using a cross-attention mechanism to obtain RGB-language multimodal features. and deep-language multimodal features The RGB-language multimodal features and the deep-language multimodal features are concatenated together and then input into the visual language navigation big model exclusive to each robot for analysis, predicting and generating the navigation action of each robot, and the robot executes the navigation action. The robot monitors task progress and environmental changes in real time. If a new task or environmental change is detected, the operator can adjust the task and handle the anomaly. If the task instructions change, the robot can dynamically adjust its execution strategy based on the operator's feedback and update its status. Repeat the above steps until each robot successfully completes its assigned sub-task and reaches the target location.
2. The multi-robot cooperative navigation method based on a large visual language model according to claim 1, characterized in that, The specific steps for each robot to construct a semantic map are as follows: In the initial state, the robot rotates and captures RGB images and depth maps in real time through the camera; For RGB images, RGB features are extracted using a pre-trained visual feature extraction network. For depth maps, depth features are extracted using point cloud generation. The RGB features and depth features are then jointly modeled using point cloud projection to form RGB-D joint features. Based on RGB-D joint features, the decoder of a pre-trained semantic segmentation network upsamples the RGB-D joint features layer by layer, classifies each pixel of the RGB-D joint features, and generates a semantic segmentation map. The label map and depth map after pixel classification are transformed into 3D point cloud, and the semantic point clouds of multiple frames are fused to generate a 3D scene model with semantic labels. A semantic map is created by projecting a semantically labeled 3D scene model onto a 2D grid map using 2D grid projection processing.
3. The multi-robot cooperative navigation method based on a large visual language model according to claim 1, characterized in that, The steps by which the large language model parses the user-input task instructions and assigns the sub-task instructions to the most suitable robot are as follows: The large language model GPT3.5 is used to replace the scene prior. Based on the robot's position, the scattered semantic maps are merged to obtain a complete semantic map, which includes robot position, scene structure, scene objects and target category information, and is made into prompts in a fixed format. Based on the prompts and the current state of each robot, the large language model GPT3.5 acts as a global planner according to the task instructions input by the user, breaking down the total task into the most suitable sub-tasks for each robot.
4. The multi-robot cooperative navigation method based on a large visual language model according to claim 1, characterized in that, The robot uses a visual perception module to map real-time acquired environmental images into visual embeddings, including: A visual encoder using DINOv2 and SigLIP pre-trained features extracts RGB visual embeddings. Extracting deep visual embeddings using the InceptionNet network; The assigned subtask language instructions are mapped to language embeddings. Specifically, the task instructions are converted into word embeddings of the corresponding global vectors, and the language embeddings are extracted using a pre-trained BERT language model.
5. A multi-robot cooperative navigation method based on a large visual language model according to claim 1, characterized in that, The visual language navigation big model is based on the Llama2 big language model and uses joint training of multiple modalities. The training process is as follows: The historical video information and current frame video information of the robot navigation video, as well as the natural language instructions of the subtask, are used as inputs to the Llama2 large language model; Based on the processing of robot navigation video by image encoder, special lexical units are used to identify different information in order to distinguish which modality the current input lexical unit sequence comes from; The collected robot visual language navigation data is fed into the robot visual language navigation big model for training: the continuous measurement range of robot navigation actions is discretized into multi-dimensional action lexical units, covering the lexical units used in the Llama vocabulary; the visual language big model is trained using the standard next lexical unit prediction target, and the cross-entropy loss on the predicted action lexical units is evaluated. Based on different task scenarios, robot visual language navigation trajectories are collected, and LoRA is used to fine-tune the large model of robot visual language navigation.
6. A multi-robot cooperative navigation method based on a large visual language model according to claim 5, characterized in that, The visual language navigation data is collected in the Habitat simulator, as follows: Semantic segmentation is performed in the Habitat simulator. The path planning algorithm is used to find the shortest path from the starting point to the target point. Based on the semantic information on the path, the navigation trajectory is labeled with natural language using GPT3.5 to obtain video-language data pairs. Given a command, collect the image sequence in the corresponding navigation trajectory and answer the corresponding next action; or given an image of a trajectory, answer the corresponding command.
7. The multi-robot cooperative navigation method based on a large visual language model according to claim 1, characterized in that, The real-time monitoring of task progress and environmental changes allows the operator to adjust tasks and handle anomalies if new tasks or environmental changes are detected, as detailed below: The robot detects its current status and environmental information, and transmits the current observations to the host computer. The host computer displays the observation data and current task execution progress of each robot in real time. When new obstacles are detected in the environment, changes in the target position, or abnormal situations such as delays, failures, or sudden task requirements occur in the task, a warning is issued and the environmental change information and task abnormality information are fed back to the operator. The operator can intervene by inputting new instructions or adjusting the existing task plan, and the operator's modifications are synchronized to the robot system in real time. The robot's visual language navigation big model receives feedback from the operator, parses and generates new task strategies through the big language model, adjusts the execution logic according to the new task strategies, and updates its own status in real time during the execution of the adjusted tasks; the system synchronously updates the task status and semantic map of all robots.
8. A multi-robot cooperative navigation system based on a large visual language model, employing the multi-robot cooperative navigation method based on a large visual language model as described in claim 1, characterized in that, The system includes: The multi-robot task allocation module (10) is used to parse the high-level task instructions input by the user based on the visual language multimodal representation information, analyze the task instructions through the large language model and decompose them into multiple sub-tasks, and allocate the sub-task instructions to the most suitable robot. The visual perception and semantic understanding module (20) is used to perform visual perception and semantic understanding on multimodal input data based on environmental information collected by the robot, including RGBD visual images and natural language instructions corresponding to sub-tasks, and to extract key feature information. The multimodal information fusion module (30) is used to perform fusion alignment based on visual features and language features to generate a unified multimodal representation and to encode and represent heterogeneous input data in a unified manner. The robot visual language navigation module (40) is used to analyze and predict the navigation action of each robot based on the fused visual language multimodal representation information and the robot's dedicated robot visual language navigation big model. The robot executes the action and updates its state. The dynamic feedback and real-time optimization module (50) is used to monitor the robot status and task progress in real time based on the status of each robot. When a new task or environmental change is detected, the operator is prompted to adjust the task and handle the abnormality. The robot dynamically adjusts its execution strategy and updates its status based on the operator's feedback information. The real-machine deployment verification module (60) is used to verify and optimize the performance of the multi-robot system in the actual working environment based on the developed multi-robot collaborative navigation method.
9. The multi-robot cooperative navigation system based on a large visual language model according to claim 8, characterized in that, The visual perception and semantic understanding module (20) includes: The visual feature extraction unit (21) is used to extract RGB visual features based on the RGBD visual images collected by the robot, through DINOv2 and SigLIP, and extract depth visual features through the InceptionNet network for subsequent navigation and task decision-making. The language embedding generation unit (22) is used to perform semantic parsing and representation of the task instructions based on user input and the sub-task instructions after parsing and splitting by GPT3.5, and to generate language embedding information with high semantic consistency that can be recognized by the model.