Multi-robot collaborative navigation method and system based on visual language large model

Abstract

The invention relates to a multi-robot collaborative navigation method and system based on a visual language large model, and the method comprises the steps: carrying out the analysis according to a task instruction inputted by a user, a semantic map constructed by each robot, and a large language model, and then distributing a subtask instruction to the robot; each robot acquires an environment image in real time, analyzes and predicts the next action by using the visual language navigation large model according to the distributed subtask instruction, and executes the corresponding action and updates the state; an operator monitors the state of the robot and checks the task progress in real time through the man-machine interaction interface, and adjusts the task or deals with abnormity in the execution process when needed, and the robot dynamically adjusts an execution strategy according to feedback information, optimizes the task and completes navigation. Compared with the prior art, the multi-modal data is deeply fused, the robot is endowed with higher environment perception and task understanding ability, the cooperation efficiency of the multi-robot system is improved, and the dynamic environment adaptability is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-robot collaborative navigation, and in particular to a multi-robot collaborative navigation method and system based on a visual language large model. Background Art

[0002] With the rapid development of artificial intelligence and robotics, multi-robot systems have received widespread attention and applications in many application fields, such as intelligent manufacturing, intelligent warehousing, indoor cleaning, logistics distribution, post-disaster rescue, etc. The collaborative work of multiple robots can complete tasks that are difficult for a single robot to complete independently, significantly improving the execution efficiency of tasks and the adaptability of the system. However, at the same time, it also faces the challenge of how to achieve efficient collaboration, task allocation and navigation planning between robots in a complex and dynamic environment.

[0003] Traditional multi-robot collaborative navigation and task allocation methods mostly rely on predetermined rules, static maps, and limited sensor information. These methods are usually limited to fixed task scenarios and lack the ability to respond to dynamic environments in real time, resulting in poor flexibility and adaptability of the system when facing task changes or environmental changes. At the same time, existing multi-robot systems usually rely on simple sensor data (such as lidar, ultrasonic sensors, etc.) and hard-coded task allocation strategies, with limited understanding of complex or unknown environments, and lack the ability to intelligently parse and flexibly execute complex task instructions. Especially in complex task scenarios, how to understand and execute natural language instructions from operators, and how to dynamically adjust and allocate tasks according to the capabilities, states, and environmental changes of different robots are key difficulties in current multi-robot collaborative systems.

[0004] Although deep learning and natural language processing technologies have made significant progress in recent years, the research on combining natural language with visual information to form an efficient and intelligent large visual language model and apply it to multi-robot collaborative navigation and task allocation is still in its infancy. Most existing systems can only process simple fixed instructions when parsing task instructions, and it is difficult to cope with the complex and changeable instruction requirements put forward by users in dynamic environments, and fail to give full play to their advantages in multimodal information fusion.

[0005] In order to solve these problems, in recent years, some studies have tried to introduce visual language models and human-computer interaction mechanisms to enhance the intelligence and flexibility of robot systems. By combining visual information and natural language understanding, robots can understand the operator's intentions more accurately and make task adjustments and path planning in real time. However, the current technology still has the following shortcomings: on the one hand, the application of existing visual language models in multi-robot collaborative tasks is not yet mature, and it is difficult to perform real-time and effective task allocation and collaboration in complex task environments; on the other hand, the interaction between robots and operators is still relatively limited, and it is difficult to efficiently support dynamic task allocation and real-time feedback. Therefore, a new method is urgently needed to improve the collaborative efficiency and task execution flexibility of multi-robot systems in complex dynamic environments by integrating large visual language models and human-computer interaction mechanisms. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multi-robot collaborative navigation method and system based on a large visual language model.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] As a first aspect of the present invention, a multi-robot collaborative navigation method based on a visual language large model is provided, the steps comprising:

[0009] Based on the task instructions input by the user, combined with the semantic map built by each robot, the large language model parses and assigns the sub-task instructions to the robot;

[0010] Each robot collects environmental images in real time and maps them into visual embeddings using the visual perception module, while also mapping the assigned subtask language instructions into language embeddings;

[0011] The visual embedding and language embedding are cross-modal feature aligned to obtain the visual language multimodal fusion feature, which is input into the visual language navigation model exclusive to each robot for parsing, and the navigation action of each robot is predicted and generated, and the robot executes the navigation action;

[0012] Monitor task progress and environmental changes in real time. If a new task or environmental change is detected, the operator will adjust the task and handle exceptions. If the task instructions change, the robot will dynamically adjust the execution strategy based on the operator's feedback and update the status.

[0013] The above steps are repeated until each robot successfully completes the assigned subtask and reaches the target location.

[0014] As a preferred technical solution, the steps for each robot to construct a semantic map are as follows:

[0015] At the initial state, the robot rotates and collects RGB images and depth maps in real time through the camera every 30 degrees;

[0016] For RGB images, RGB features are extracted through a pre-trained ResNet50-based visual feature extraction network. For depth images, depth features are extracted through point cloud generation, and RGB features and depth features are jointly modeled through point cloud projection to form RGB-D joint features.

[0017] Based on the RGB-D joint features, the decoder of the pre-trained semantic segmentation network is used to upsample the RGB-D joint features layer by layer, classify each pixel of the RGB-D joint features, and generate a semantic segmentation map;

[0018] The label map and depth map after pixel classification are converted into 3D point cloud, and the semantic point clouds of multiple frames are fused and mapped to generate a 3D scene model with semantic labels.

[0019] The semantic map is obtained by projecting the semantically labeled 3D scene model onto a 2D grid map through 2D grid projection.

[0020] As a preferred technical solution, the steps of the large language model parsing the task instructions input by the user and assigning the subtask instructions to the most suitable robot are as follows:

[0021] The large language model GPT3.5 is used to replace the scene prior. According to the robot's position, the scattered semantic maps are merged to obtain a patchwork semantic map, which contains the robot's position, scene structure, scene objects and target category information, and is made into prompts in a fixed format;

[0022] Based on the prompts and the current state of each robot, the large language model GPT3.5 acts as a global planner according to the task instructions entered by the user, splitting the total task into the most appropriate subtasks for each robot.

[0023] As a preferred technical solution, the robot uses a visual perception module to map the real-time collected environmental image into a visual embedding, including:

[0024] Extract RGB visual embeddings using the visual encoder with DINOv2 and SigLIP pre-trained features;

[0025] Extract deep visual embeddings using the InceptionNet network;

[0026] The assigned subtask language instructions are mapped to language embeddings, specifically: the task instructions are converted into word embeddings of the corresponding global vector, and the language embeddings are extracted using the pre-trained BERT language model.

[0027] As a preferred technical solution, the steps for constructing the visual-language multimodal fusion feature are as follows: the RGB visual embedding and the deep visual embedding in the visual embedding are respectively aligned with the language embedding using a cross-attention mechanism to obtain the RGB-language multimodal feature E RGB-L And the deep-language multimodal feature E RGB-L ; The RGB-language multimodal features and the depth-language multimodal features are spliced ​​together and input into the Llama2 large language model to predict the robot's next navigation action.

[0028] As a preferred technical solution, the visual language navigation large model is based on the Llama2 large language model and adopts joint training of multiple modalities. The training process is as follows:

[0029] The historical video information and current frame video information of the robot navigation video and the natural language instructions of the subtask are used as the input of the Llama2 large language model;

[0030] Based on the processing of robot navigation video by the image encoder, special words are used to identify different information to distinguish which modality the current input word sequence comes from;

[0031] The collected robot vision-language navigation data is fed into the robot vision-language navigation model for training: the continuous measurement range of the robot navigation action is discretized into multi-dimensional action tokens, covering the tokens used in the Llama vocabulary; the vision-language model is trained using the standard next token prediction objective, and the cross entropy loss on the predicted action tokens is evaluated;

[0032] The robot vision and language navigation trajectories are collected according to different mission scenarios, and LoRA is used to fine-tune the robot vision and language navigation model.

[0033] As a preferred technical solution, the visual language navigation data is collected in the Habitat simulator, as follows:

[0034] Semantic segmentation is performed in the Habitat simulator, and the path planning algorithm is used to find the shortest path from the starting point to the target point. GPT3.5 is used to perform natural language tagging on the navigation trajectory based on the semantic information on the path to obtain video-language data pairs;

[0035] Given an instruction, collect the image sequence in the navigation trajectory corresponding to this instruction and answer the corresponding next action; or given an image of a trajectory, answer the corresponding instruction.

[0036] As a preferred technical solution, the real-time monitoring of task progress and environmental changes, if a new task or environmental change is detected, the operator will adjust the task and handle the exception, as follows:

[0037] The robot detects the current state and environmental information, and transmits the current observation to the host computer, which displays the observation data of each robot and the current task execution progress in real time;

[0038] When new obstacles in the environment, changes in the target position, or abnormal delays, failures, and sudden task requirements are detected, a warning prompt is issued and the environmental change information and task abnormalities are fed back to the operator. The operator intervenes by entering new instructions or adjusting the existing task plan, and the operator's changes are synchronized to the robot system in real time;

[0039] The robot vision language navigation large model receives feedback from the operator, parses and generates new task strategies through the large language model, adjusts the execution logic according to the new task strategy, and updates its own status in real time during the execution of the adjusted task; the system synchronously updates the task status and semantic map of all robots.

[0040] As a second aspect of the present invention, there is provided a multi-robot collaborative navigation system based on a visual language large model using the multi-robot collaborative navigation method based on a visual language large model, the system comprising:

[0041] The multi-robot task assignment module is used to parse the high-level task instructions input by the user based on the multimodal representation information of the visual language, analyze the task instructions through the large language model and decompose them into multiple subtasks, and assign the subtask instructions to the most suitable robot;

[0042] The visual perception and semantic understanding module is used to perform visual perception and semantic understanding of multimodal input data based on the environmental information collected by the robot, including RGBD visual images and natural language instructions corresponding to subtasks, and extract key feature information;

[0043] The multimodal information fusion module is used to fuse and align visual features and language features, generate a unified multimodal representation, and uniformly encode and represent the input heterogeneous data;

[0044] The robot vision language navigation module is used to parse and predict the navigation action of each robot based on the fused visual language multimodal representation information through the robot vision language navigation model exclusive to each robot. The robot executes the action and updates its status.

[0045] Dynamic feedback and real-time optimization module, which is used to monitor the robot status and task progress in real time based on the status of each robot. When a new task or environmental change is detected, the operator is prompted to adjust the task and handle exceptions. The robot dynamically adjusts the execution strategy according to the operator's feedback information and updates the status.

[0046] The real-machine deployment verification module is used to verify and optimize the performance of the multi-robot system in the actual working environment based on the developed multi-robot collaborative navigation method.

[0047] As a preferred technical solution, the visual perception and semantic understanding module includes:

[0048] The visual feature extraction unit is used to extract RGB visual features through DINOv2 and SigLIP based on the RGBD visual images collected by the robot, and extract deep visual features through the InceptionNet network for subsequent navigation and task decision-making;

[0049] The language embedding generation unit is used to perform semantic analysis and representation of task instructions based on the task instructions input by the user and the sub-task instructions after GPT3.5 parsing and splitting, using the pre-trained language model BERT, and generate language embedding information with high semantic consistency that can be recognized by the model.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1) This invention achieves a deep fusion of visual and language multimodal data by introducing a large visual language model, giving the robot stronger environmental perception and task understanding capabilities. By combining vision and language, collaboration and dynamic task allocation between multiple robots are achieved, and real-time monitoring and adjustment can be performed through human-computer interaction. Compared with traditional systems based on a single sensor, robots can fully understand the environment and task requirements through multimodal perception, thereby improving the robot's adaptability to complex environments and achieving high accuracy in navigation and task allocation.

[0052] 2) The task allocation module of the present invention can analyze task requirements in real time, dynamically adjust task allocation strategies and give priority to key tasks based on real-time environmental information, task priority and robot status. Compared with the traditional static or predefined rule task allocation method, the present invention realizes the intelligent allocation of task resources, avoids resource waste and task conflicts, and effectively improves the timeliness and accuracy of the overall task completion of the system.

[0053] 3) The present invention provides a human-computer interaction module based on natural language and visual interface, and introduces a safety monitoring and dynamic adjustment mechanism. When the robot encounters obstacles, path conflicts or other emergencies, it can issue an alarm in real time and execute emergency strategies. At the same time, the operator can monitor and operate the robot in real time through voice commands, text input or graphical interface. Compared with the interaction mode based on fixed instructions or limited feedback in the prior art, the human-computer interaction of the present invention is more intuitive, more efficient, and more reliable, and can significantly reduce the difficulty of operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flowchart of a multi-robot collaborative navigation method based on a large visual language model provided by an embodiment of the present invention;

[0055] Figure 2 A flowchart of another multi-robot collaborative navigation method based on a large visual language model provided by an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of a multi-robot collaborative navigation system based on a visual language large model provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of another multi-robot collaborative navigation system based on a large visual language model provided by an embodiment of the present invention;

[0058] In the accompanying drawings, 10 is a multi-robot task allocation module, 20 is a visual perception and semantic understanding module, 21 is a visual feature extraction unit, 22 is a language embedding generation unit, 30 is a multimodal information fusion module, 40 is a robot visual language navigation module, 50 is a dynamic feedback and real-time optimization module, and 60 is a real machine deployment verification module. DETAILED DESCRIPTION

[0059] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0060] Example 1

[0061] This embodiment relates to a method for multi-robot task allocation and collaborative navigation based on a large visual language model. The method is applied to multi-robot collaborative navigation tasks with dynamic task allocation, such as Figure 1 As shown, the method specifically comprises the following steps:

[0062] Step S101, according to the task instructions input by the user, combined with the semantic map constructed by each robot, the large language model parses and assigns the sub-task instructions to the most suitable robot.

[0063] The task instruction input by the user is a natural language instruction, which needs to be semantically understood and converted by the natural language processing module. Specifically, the user can input the task instruction in the form of text or voice through the human-computer interaction interface, and the final input to the model is a natural language instruction, such as "transport item A from location X to location Y".

[0064] The semantic map constructed by the robot refers to the semantic map generated by the RGBD visual image captured by the camera when the robot rotates 360° at the initial moment after feature extraction, SegNet network semantic segmentation, three-dimensional point cloud conversion, and two-dimensional grid projection processing.

[0065] In an embodiment of the present invention, at the initial state, the robot rotates 360° and collects RGB images and depth maps in real time every 30 degrees through the camera, wherein the RGB image contains the color information of the scene, and the depth map provides the distance information from each pixel to the camera. For RGB images, RGB features are extracted through the pre-trained ResNet50 visual feature extraction network, and for depth images, depth features are extracted through point cloud generation, and RGB and depth information are jointly modeled through point cloud projection to form a more robust feature representation. Based on these RGB-D joint features, the pre-trained SegNet semantic segmentation network is used to classify each pixel in the image and generate a semantic segmentation map. These processes can be expressed as:

[0066] F RGB =ResNet50(RGB), F Depth =PointNet(Depth) (1)

[0067] Among them, RGB and Depth represent the RGB image and depth map collected by the robot respectively. The RGB image can be extracted through the ResNet50 network to obtain the RGB image feature F RGB , the depth map can be extracted through the PointNet network to obtain the depth map feature F Depth .

[0068] F fusion =[F RGB ,F DeptH ] (2)

[0069] Among them, [] represents concat feature concatenation, which converts the RGB image feature F RGB and depth map feature F Depth After splicing, we can get the RGB-D joint feature F fusion .

[0070] S(x,y)=SegNet decoder (Ffusion (x,y)) (3)

[0071] Among them, (x, y) represents the image pixel, and the decoder of the SegNet semantic segmentation network is used to decode the RGB-D joint feature F fusion By upsampling layer by layer, we can get the category label S(x,y) of each pixel (x,y).

[0072] Through the label map and depth map after pixel classification, the semantic point clouds of multiple frames are fused and mapped to generate a three-dimensional scene model with semantic labels, which is then projected onto a two-dimensional grid map to obtain the required semantic map.

[0073] The large language model parses the task instructions input by the user and assigns the sub-task instructions to the most suitable robot. This means that the GPT3.5 large language model with global information can assign the most suitable sub-task instructions to each robot based on the location status of each robot.

[0074] In an embodiment of the present invention, considering that the learning-based robot visual language navigation method usually requires a large amount of computing resources to learn and utilize environmental priors, and the large language model itself contains rich knowledge about the world, the large language model GPT3.5 is selected to replace the scene prior. According to the position of the robot, the scattered semantic maps are merged to obtain a complete semantic map, which contains information such as the robot position, scene structure, scene objects and target categories, and prompts are made in a fixed format. Based on these prompts, as well as the current state of each robot (power, task load, etc.), the large language model acts as a global planner according to the task instructions input by the user, and splits the total task into the most suitable subtasks for each robot, such as "Robot 1 is responsible for carrying item A to the transfer location", "Robot 2 is responsible for carrying from the secondary location to the exit".

[0075] In step S102, each robot collects environmental images in real time and maps them into visual embeddings using a visual perception module, and maps the assigned subtask language instructions into language embeddings.

[0076] Collecting environmental images in real time and mapping them into visual embeddings using the visual perception module means: inputting the RGB visual images collected by the robot into the visual encoder with pre-trained features of DINOv2 and SigLIP to extract the RGB visual embedding, and inputting the deep image into the InceptionNet network to extract the deep visual embedding; mapping the assigned subtask language instructions into language embeddings means extracting the language embeddings after semantic understanding through the BERT pre-trained natural language processing module.

[0077] In the embodiment of the present invention, based on the RGBD visual image collected by the robot and the assigned subtask instructions, an image encoder and a text encoder are used to extract corresponding image signals and text signals into word units respectively.

[0078] Specifically, in the robot vision and language navigation task, the model needs to use historical information to understand the part of the instruction that it has completed. For the current frame, it not only needs to provide the latest scene information, but also needs to predict the next reasonable action that complies with the instruction. This significant difference in the role of image signals requires us to encode historical video information and current frame video information separately. Regarding image encoding, the visual encoder using DINOv2 and SigLIP pre-trained features extracts RGB visual embeddings, which fuses low-level spatial information from DINOv2 and high-level semantics from SigLIP to help achieve visual generalization; the InceptionNet network extracts deep visual embeddings; regarding text encoding, the BERT pre-trained language model is used to extract language embeddings. These processes can be expressed as:

[0079] E RGB =[DINOv2(RGB),SigLIP(RGB)] (4)

[0080] Among them, the visual encoders of DINOv2 and SigLIP pre-trained features are used to extract RGB visual embedding, [] represents concat feature splicing, and the RGB visual embedding E can be obtained. RGB .

[0081] E Depth =InceptionNet(Depth) (5)

[0082] Among them, the deep visual embedding E is extracted using the InceptionNet network Depth .

[0083] E L =BERT(w1,…,w T ) (6)

[0084] The task instructions are first converted into the corresponding GLoVE embedding (global vector word embedding), encoded as (w1,…,w T ), where T represents the instruction length, and then the BERT language model is used to extract the language embedding E L .

[0085] Step S103, embedding the vision and language into each robot's exclusive large language model for parsing, predicting and generating the navigation action of each robot, and the robot executes the action.

[0086] Based on the Transformer's cross-attention mechanism, the visual embedding and the language embedding are aligned and spliced ​​to obtain multimodal features. The input is used into the robot visual language navigation model using the Llama2 large language model to generate the robot's next navigation action, such as moving forward or turning.

[0087] The robot vision embedding is mapped into the same format as the language embedding through a multi-layer perceptron network. The multimodal data of these visual languages ​​are first aligned through the Transformer-based cross-attention mechanism to achieve cross-modal feature alignment:

[0088] E RGB- =Attn(E RGB ,E L ), E RGB-L =Attn(E Dept ,E L ) (7)

[0089] Among them, Attn represents the cross attention mechanism based on Transformer, which can obtain RGB-language multimodal features E respectively. RGB-L , Deep-Language Multimodal Features E RGB-L Finally, these multimodal features are concatenated and input into the Llama2 large language model to predict the robot’s next navigation action.

[0090] In the embodiment of the present invention, by taking the robot navigation video (historical video information and current frame video information) and the subtask natural language instructions as the input of the Llama2 large language model, the underlying execution commands of the robot can be directly output.

[0091] Specifically, based on the processing of robot navigation videos by the image encoder, in order to allow the model to still understand data of common modalities, special words are used to identify different information, such as historical information ([HIS] and [ / HIS]), current frame ([OBS] and [ / OBS]) and instructions [NAV]. These identifiers can help the large model distinguish which modality the current input word sequence comes from, and realize joint training of multiple modalities. This joint training method can avoid the problem of catastrophic forgetting and maintain an understanding of the natural world while learning navigation data.

[0092] First of all, considering the difficulty, high cost and limited diversity of real data collection. Therefore, visual language navigation data is collected in the Habitat simulator: given an instruction, collect the image sequence in the navigation trajectory corresponding to this instruction, and answer the corresponding next action; or given an image of a trajectory, answer the corresponding instruction. Specifically, first perform semantic segmentation in the Habitat simulation simulator, use the A* path planning algorithm to find the shortest path from the starting point to the target point, and use GPT3.5 to mark these navigation trajectories with natural language based on the semantic information on the path to obtain video-language data pairs.

[0093] Then, the collected robot visual language navigation trajectory is sent to the visual language large model for training. The DINOv2 and SigLIP visual encoders, BERT language model, and Llama2 large language model are all pre-trained, and fine-tuned on this basis instead of training from scratch. The continuous measurement range of the robot navigation action (forward, turn) is discretized into 256-dimensional action tokens, covering the 256 least used tokens in the Llama vocabulary. The visual language large model will use the standard next token prediction target for training, and only the cross-entropy loss on the predicted action token is evaluated. As for the robot navigation "stop" action, it can be directly represented by a token (0 means do not stop, 1 means stop).

[0094] Finally, according to different mission scenarios, 50-100 robot visual language navigation trajectories are collected, and the visual language model is fine-tuned using LoRA. The visual language model can be used to fuse multimodal information, analyze the environment and mission requirements, and generate the next navigation action. The robot performs motion control based on the generated navigation instructions and also feedbacks the execution status.

[0095] Step S104: The system monitors the progress of tasks and environmental changes in real time. If a new task or environmental change is detected, the system will prompt the operator to adjust the task and handle the exception, including:

[0096] First, the system continuously collects the robot's current status (such as position, power, task completion) and environmental information (such as obstacles, dynamic changes, etc.) through the sensor network and visual perception module, and dynamically updates the task and environmental status on the user interface in real time, and intuitively presents the robot's position and environmental changes;

[0097] Then, the system has a built-in anomaly detection module. When it detects abnormal situations such as new obstacles in the environment, changes in the target position, or sudden task requirements, the system will send reminders to the operator through sound, images, and text, prompting the operator to adjust the task and handle the abnormality;

[0098] Finally, the operator intervenes through the system interface to adjust task objectives, priorities, or re-plan paths. The system supports natural language input and click operations to ensure that the adjustment process is fast and efficient. The operator's modifications will be synchronized to the robot system in real time, updating task allocation and navigation strategies to avoid execution interruptions.

[0099] Specifically, these robots in indoor scenes can transmit their current observations to the host computer through WiFi wireless communication technology. The host computer is a human-computer interaction interface written using the MFC application framework, which can display the observation data of each robot (such as first-person visual images, power, any load, etc.) and the current task execution progress in real time. The robot can detect new obstacles, dynamic targets or path blockages in the environment; if there are delays, failures or other abnormalities in the task, the system will issue a warning prompt and feedback the environmental change information or task abnormalities to the operator through the interface or voice. The operator can intervene by entering new instructions or adjusting the existing task plan.

[0100] Step S105, if changes occur, the robot dynamically adjusts the execution strategy according to the operator's feedback information and updates the status, including: if the system detects a change in the task or environment, the system synchronizes the new task target to the affected robots, and these robots dynamically adjust the strategy according to the operator's feedback information. For example, the robot can re-plan the path, adjust the task priority, or reallocate the task to other robots. The robot updates the status according to the feedback information during the execution process and continues to execute the new navigation action.

[0101] Specifically, the robot vision language navigation model receives feedback from the operator (such as reallocating task objectives and adjusting paths), parses and generates new task strategies through the large language model. According to the new task strategy, the robot adjusts the execution logic, such as replanning the navigation path and modifying the task priority. While executing the adjusted task, the robot updates its own status in real time, including location, power, task completion, etc. The system synchronously updates the task status and semantic map of all robots to ensure the consistency of collaboration among multiple robots.

[0102] Example 2

[0103] This embodiment also relates to a method for multi-robot task allocation and collaborative navigation based on a large visual language model, which introduces a new multi-robot task allocation method based on a large language model, which can allocate the most suitable subtasks according to the different states of each robot; at the same time, the present invention introduces a new robot navigation large model based on visual language multimodality as an action decision strategy, which can be generalized to different robot navigation scenarios through fine-tuning with a small amount of data; at the same time, the present invention introduces a human-machine task collaborative decision-making mechanism and a dynamic task adjustment mechanism, aiming to further improve the execution flexibility and task completion efficiency of the robot system in complex environments. This embodiment is suitable for more complex multi-robot collaborative scenarios, such as disaster relief, industrial production, and logistics distribution, further expanding the scope of application and practical application value of the present invention.

[0104] Figure 2 FIG. 1 is a flowchart of another multi-robot task allocation and collaborative navigation method based on a large visual language model provided according to an embodiment of the present invention. Figure 2 As shown, the method specifically comprises the following steps:

[0105] Step S201, obtaining the current state of each robot in the scene, and the user inputs a natural language instruction task in the form of text or voice.

[0106] Step S202: According to the semantic map constructed by each robot, the multi-robot task allocation module can parse the task instruction into sub-task instructions and allocate them to the most suitable robot.

[0107] Step S203: input the visual images and subtask instructions acquired by the robot into the visual perception and semantic understanding module to obtain visual embedding and language embedding.

[0108] Step S204: input the visual embedding and the language embedding into a multimodal information fusion module to obtain a visual-language multimodal fusion feature.

[0109] Step S205: Based on the visual language multimodal fusion features, the robot visual language navigation module can predict and generate the next action of the robot.

[0110] Step S206: The robot interacts with the environment based on the decision action and updates the current state.

[0111] Step S207, the system obtains the latest status of the robot and displays it on the screen.

[0112] Step S208, determine whether each robot has completed the subtask, if not, execute step S209; if completed, execute step S212.

[0113] Step S209, determine whether the task or environment has changed. If not, jump to step S203 to perform iterative calculations; if changed, execute step S210.

[0114] In step S210, the operator can adjust the task and handle exceptions through the dynamic feedback and real-time optimization module, thereby updating the overall task instructions or subtasks.

[0115] Step S211, determine whether the overall task has changed, if so, jump to step S201, perform iterative calculation; if not, jump to step S203, perform iterative calculation.

[0116] Step S212: The system terminates the multi-robot collaborative navigation task and generates a task report and execution log.

[0117] Example 3

[0118] This embodiment also relates to a multi-robot task allocation and collaborative navigation system based on a visual language large model, which is applied to multi-robot collaborative navigation tasks with dynamic task allocation.

[0119] like Figure 3 As shown, the system includes a multi-robot task allocation module 10, a visual perception and semantic understanding module 20, a multimodal information fusion module 30, a robot visual language navigation module 40, a dynamic feedback and real-time optimization module 50 and a real machine deployment verification module 60.

[0120] The multi-robot task assignment module 10 is used to parse the high-level task instructions input by the user based on the visual language multimodal representation information, analyze the task instructions through a large language model and decompose them into multiple subtasks, and assign the subtask instructions to the most suitable robot.

[0121] Specifically, the visual perception and semantic understanding module 20 is used to perform visual perception and semantic understanding of multimodal input data based on the environmental information collected by the robot, including RGBD visual images and natural language instructions corresponding to subtasks, and extract key feature information.

[0122] The multimodal information fusion module 30 is used to perform fusion and alignment based on visual features and language features to generate a unified multimodal representation. The main purpose is to uniformly encode and represent input heterogeneous data so that information of different modalities can complement and enhance each other.

[0123] The robot visual language navigation module 40 is used to parse and predict the navigation action of each robot based on the fused visual language multimodal representation information through a large language model exclusive to each robot (robot visual language navigation large model), and the robot executes the action and updates the state.

[0124] The dynamic feedback and real-time optimization module 50 is used to monitor the robot status and task progress in real time based on the status of each robot. When a new task or environmental change is detected, the operator will be prompted to adjust the task and handle exceptions. The robot dynamically adjusts the execution strategy according to the operator's feedback information and updates the status.

[0125] The actual machine deployment verification module 60 is used to verify and optimize the performance of the multi-robot system in the actual working environment based on the developed multi-robot task allocation and collaborative navigation method, to ensure the feasibility, robustness and efficiency of the multi-robot task allocation and collaborative navigation method based on the visual language large model in actual application scenarios, and to provide a reliable basis for further industrial deployment and promotion and application.

[0126] Example 4

[0127] This embodiment also relates to a multi-robot task allocation and collaborative navigation system based on a visual language large model, which introduces a new multi-robot task allocation method based on a large language model, which can allocate the most suitable subtasks according to the different states of each robot; at the same time, the present invention introduces a new robot navigation large model based on visual language multimodality as an action decision strategy, which can be generalized to different robot navigation scenarios through fine-tuning with a small amount of data; at the same time, the present invention introduces a human-machine task collaborative decision-making mechanism and a dynamic task adjustment mechanism, aiming to further improve the execution flexibility and task completion efficiency of the robot system in complex environments. This embodiment is suitable for more complex multi-robot collaborative scenarios, such as disaster relief, industrial production, and logistics distribution, further expanding the scope of application and practical application value of the present invention.

[0128] Optionally, Figure 4 FIG. 1 is a schematic diagram of another multi-robot task allocation and collaborative navigation system based on a large visual language model provided according to an embodiment of the present invention. Figure 4 As shown, the visual perception and semantic understanding module 20 includes a visual feature extraction unit 21 and a language embedding generation unit 22.

[0129] Specifically, the visual feature extraction unit 21 is used to extract RGB visual features through DINOv2 and SigLIP based on the RGBD visual images collected by the robot, and extract deep visual features through the InceptionNet network. These high-dimensional visual features can be used for subsequent navigation and task decision-making.

[0130] The language embedding generation unit 22 is used to perform semantic analysis and representation on the task instructions input by the user and the sub-task instructions after parsing and splitting by GPT3.5, using the pre-trained language model BERT, and generate language embedding information with high semantic consistency that can be recognized by the model.

[0131] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0132] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0133] The processing unit performs the various methods and processes described above, such as methods S101 to S105, S201 to S210. For example, in some embodiments, methods S101 to S105, S201 to S210 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S101 to S105, S201 to S212 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S101 to S105, S201 to S212 in any other appropriate manner (e.g., by means of firmware).

[0134] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0135] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A multi-robot collaborative navigation method based on a large visual language model, characterized in that the steps include: Based on the task instructions input by the user, combined with the semantic map built by each robot, the large language model parses and assigns the sub-task instructions to the robot; Each robot collects environmental images in real time and maps them into visual embeddings using the visual perception module, while also mapping the assigned subtask language instructions into language embeddings; The visual embedding and language embedding are cross-modal feature aligned to obtain the visual language multimodal fusion feature, which is input into the visual language navigation model exclusive to each robot for parsing, and the navigation action of each robot is predicted and generated, and the robot executes the navigation action; Monitor task progress and environmental changes in real time. If a new task or environmental change is detected, the operator will adjust the task and handle exceptions. If the task instructions change, the robot will dynamically adjust the execution strategy based on the operator's feedback and update the status. The above steps are repeated until each robot successfully completes the assigned subtask and reaches the target location.

2. According to claim 1, a multi-robot collaborative navigation method based on a visual language large model is characterized in that: The specific steps for each robot to construct a semantic map are as follows: At the initial state, the robot rotates and collects RGB images and depth maps in real time through the camera; For RGB images, RGB features are extracted through a pre-trained visual feature extraction network. For depth images, depth features are extracted through point cloud generation, and RGB features and depth features are jointly modeled through point cloud projection to form RGB-D joint features. Based on the RGB-D joint features, the decoder of the pre-trained semantic segmentation network is used to upsample the RGB-D joint features layer by layer, classify each pixel of the RGB-D joint features, and generate a semantic segmentation map; The label map and depth map after pixel classification are converted into 3D point cloud, and the semantic point clouds of multiple frames are fused and mapped to generate a 3D scene model with semantic labels. The semantic map is obtained by projecting the semantically labeled 3D scene model onto a 2D grid map through 2D grid projection.

3. The multi-robot collaborative navigation method based on a visual language large model according to claim 1 is characterized in that: The steps of the large language model parsing the task instruction input by the user and assigning the subtask instruction to the most suitable robot are as follows: The large language model GPT3.5 is used to replace the scene prior. According to the robot's position, the scattered semantic maps are merged to obtain a complete semantic map, which contains the robot's position, scene structure, scene objects and target category information, and is made into prompts in a fixed format; Based on the prompts and the current state of each robot, the large language model GPT3.5 acts as a global planner according to the task instructions entered by the user, splitting the total task into the most appropriate subtasks for each robot.

4. The multi-robot collaborative navigation method based on a visual language large model according to claim 1 is characterized in that: The robot uses a visual perception module to map the real-time collected environment image into a visual embedding, including: Extract RGB visual embeddings using the visual encoder with DINOv2 and SigLIP pre-trained features; Extract deep visual embeddings using the InceptionNet network; The assigned subtask language instructions are mapped to language embeddings, specifically: the task instructions are converted into word embeddings of the corresponding global vector, and the language embeddings are extracted using the pre-trained BERT language model.

5. The multi-robot collaborative navigation method based on a visual language large model according to claim 4 is characterized in that: The steps of constructing the visual-language multimodal fusion feature are as follows: the RGB visual embedding and the deep visual embedding in the visual embedding are respectively aligned with the language embedding using a cross-attention mechanism to obtain the RGB-language multimodal feature E RGB-L And the deep-language multimodal feature E RGB-L ; The RGB-language multimodal features and the depth-language multimodal features are spliced ​​together and input into the Llama2 large language model to predict the robot's next navigation action.

6. The multi-robot collaborative navigation method based on a visual language large model according to claim 1 is characterized in that: The visual language navigation model is based on the Llama2 large language model and adopts joint training of multiple modalities. The training process is as follows: The historical video information and current frame video information of the robot navigation video and the natural language instructions of the subtask are used as the input of the Llama2 large language model; Based on the processing of robot navigation video by the image encoder, special words are used to identify different information to distinguish which modality the current input word sequence comes from; The collected robot vision-language navigation data is fed into the robot vision-language navigation model for training: the continuous measurement range of the robot navigation action is discretized into multi-dimensional action tokens, covering the tokens used in the Llama vocabulary; the vision-language model is trained using the standard next token prediction objective, and the cross entropy loss on the predicted action tokens is evaluated; The robot vision and language navigation trajectories are collected according to different mission scenarios, and LoRA is used to fine-tune the robot vision and language navigation model.

7. The multi-robot collaborative navigation method based on a visual language large model according to claim 6 is characterized in that: The visual language navigation data is collected in the Habitat simulator as follows: Semantic segmentation is performed in the Habitat simulator, and the path planning algorithm is used to find the shortest path from the starting point to the target point. GPT3.5 is used to perform natural language tagging on the navigation trajectory based on the semantic information on the path to obtain video-language data pairs; Given an instruction, collect the image sequence in the navigation trajectory corresponding to this instruction and answer the corresponding next action; or given an image of a trajectory, answer the corresponding instruction.

8. The multi-robot collaborative navigation method based on a visual language large model according to claim 1 is characterized in that: The real-time monitoring of task progress and environmental changes, if a new task or environmental change is detected, the operator will adjust the task and handle the exception, as follows: The robot detects the current state and environmental information, and transmits the current observation to the host computer, which displays the observation data of each robot and the current task execution progress in real time; When new obstacles in the environment, changes in the target position, or abnormal delays, failures, and sudden task requirements are detected, a warning prompt is issued and the environmental change information and task abnormalities are fed back to the operator. The operator intervenes by entering new instructions or adjusting the existing task plan, and the operator's changes are synchronized to the robot system in real time; The robot vision language navigation large model receives feedback from the operator, parses and generates new task strategies through the large language model, adjusts the execution logic according to the new task strategy, and updates its own status in real time during the execution of the adjusted task; the system synchronously updates the task status and semantic map of all robots.

9. A multi-robot collaborative navigation system based on a visual language large model using the multi-robot collaborative navigation method based on a visual language large model according to claim 1, characterized in that: The system comprises: A multi-robot task assignment module (10) is used to parse the high-level task instructions input by the user based on the multimodal representation information of the visual language, analyze the task instructions through a large language model and decompose them into multiple subtasks, and assign the subtask instructions to the most suitable robot; A visual perception and semantic understanding module (20), which is used to perform visual perception and semantic understanding on multimodal input data based on environmental information collected by the robot, including RGBD visual images and natural language instructions corresponding to subtasks, and extract key feature information; A multimodal information fusion module (30), used for fusing and aligning visual features and language features, generating a unified multimodal representation, and uniformly encoding and representing input heterogeneous data; A robot visual language navigation module (40) is used to parse and predict the navigation action of each robot based on the fused visual language multimodal representation information through a robot visual language navigation large model exclusive to each robot, and the robot executes the action and updates the state; A dynamic feedback and real-time optimization module (50) is used to monitor the robot status and task progress in real time based on the status of each robot. When a new task or environmental change is detected, the operator is prompted to adjust the task and handle the exception. The robot dynamically adjusts the execution strategy according to the operator's feedback information and updates the status. The real machine deployment verification module (60) is used to verify and optimize the performance of the multi-robot system in an actual working environment based on the developed multi-robot collaborative navigation method.

10. The multi-robot collaborative navigation system based on the visual language large model according to claim 9 is characterized in that: The visual perception and semantic understanding module (20) comprises: A visual feature extraction unit (21), used for extracting RGB visual features through DINOv2 and SigLIP based on the RGBD visual images collected by the robot, and extracting deep visual features through the InceptionNet network for subsequent navigation and task decision-making; The language embedding generation unit (22) is used to perform semantic analysis and representation on the task instructions based on the task instructions input by the user and the sub-task instructions after parsing and splitting by GPT3.5, using the pre-trained language model BERT, and generate language embedding information with high semantic consistency that can be recognized by the model.

Images