Heterogeneous animal robot cooperative navigation system based on large language model

Abstract

The invention provides a heterogeneous animal robot collaborative navigation system based on a large language model, which comprises a centralized control center, and is provided with a visual perception conversion module and an instruction generation module. The visual perception conversion module extracts semantic information by using a visual large model SAM, generates a bird's-eye view (BEV) representation in combination with a space-time attention mechanism, constructs a global semantic map through a semantic segmentation decoder, and converts the global semantic map into natural language description by means of a BLIP-2 model. The instruction generation module inputs visual description, navigation information, historical tracks and human expert instructions as cues into the large language model, outputs a navigation thinking chain including task allocation and instruction selection, and includes four intermediate steps of state prediction, filtering, action prediction and instruction output. According to the system, through a centralized control structure and a cooperative sensing control method, the sensing, cooperation and communication capabilities of the heterogeneous animal robot are improved, and efficient navigation and decision execution of the heterogeneous animal robot in a complex environment are ensured.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of animal robots and cooperative navigation, and particularly relates to a heterogeneous animal robot cooperative navigation system based on a large language model. BACKGROUND

[0002] In modern rescue and search tasks, animal robots have become the focus of researchers due to their flexible movement ability, long endurance time, and good concealment. For example, in narrow and complex environments such as post-earthquake ruins and underground pipelines, rat robots can complete many tasks that humans and mechanical robots cannot achieve due to their small size, excellent climbing ability, and adaptability to dim environments. However, single-species animal robots also face significant limitations and challenges when performing these tasks.

[0003] In order to complement the capabilities of different animal robots, we propose a heterogeneous animal robot cooperative navigation system based on a large language model. This system adopts a centralized control architecture, collects and processes information from each robot through a centralized control center, and then transmits the processed information to the operator. The operator makes decisions based on this information to generate a globally optimal navigation plan.

[0004] The centralized control center includes a visual perception conversion module and an instruction generation module, which collect and process information perceived by each robot and generate control instructions. The visual perception conversion module uses the visual large model SAM to extract semantic information from different perspectives, generates a bird's eye view (BEV) representation with temporal self-attention and spatial cross-attention mechanisms, constructs a global semantic map using a semantic segmentation decoder, and converts the global semantic map into a natural language description using the BLIP-2 model. The instruction generation module inputs visual description information, navigation system information, historical trajectory information, and instructions from human control experts as prompt words into a large language model, and the large model outputs a navigation thought chain to generate control instructions. The thought chain includes four intermediate reasoning steps: state prediction, state filtering, action prediction, and instruction output, to improve the collaboration efficiency between animal robots. Through the centralized control structure and effective collaborative perception and control methods, the application improves the perception, collaboration, and communication capabilities of heterogeneous animal robots, ensuring efficient navigation and decision execution in complex environments. SUMMARY

[0005] The purpose of the present application is to propose a heterogeneous animal robot cooperative navigation system based on a large language model, which significantly improves the perception and navigation capabilities of multi-animal robot navigation systems in complex environments, and solves the problems of limited perception range and movement ability, low control and navigation efficiency in the prior art.

[0006] In order to solve the above technical problems, the application provides a heterogeneous animal robot cooperative navigation system based on a large language model, comprising the following steps: S1, the centralized control architecture collects and processes the information provided by each animal robot through a centralized control center to implement unified control decisions. First, the local observation information obtained by each robot is fused into a global semantic map. Then, the global semantic map is converted into a natural language description through a BLIP-2 model. The observation information description, navigation system information, historical trajectory information, and instructions of a human control expert are input into a large language model as prompt words to effectively guide the large model to generate control instructions for each animal robot, promote efficient cooperation between the animal robots, optimize the task execution process, and significantly improve the stability and overall performance of the system. S2, a visual perception conversion module uses a visual large model SAM to extract semantic information under different perspectives and generates a bird's eye view (BEV) representation with spatiotemporal features based on time self-attention and spatial cross-attention mechanisms. A global semantic map is constructed through a semantic segmentation decoder, and the global semantic map is converted into a natural language description using a BLIP-2 model. S3, an instruction generation module inputs visual description information, navigation system information, historical trajectory information, and instructions of a human control expert into a large language model as prompt words. The visual description information is the language description of the global language map. The navigation system information includes the overall navigation task, the capabilities of each animal robot, the task, and the control method. The historical trajectory information records previous observations and actions. The human instruction information includes the description of the state and control instructions of each animal robot by a human. The above prompt words are combined with the thinking chain technology to let the large model output a navigation thinking chain. The thinking chain contains four intermediate reasoning steps: state prediction, state filtering, action prediction, and instruction output, to improve the cooperation efficiency between animal robots.

[0007] The navigation system includes the following parts: a variety of animal robot expert control systems. The animal robot includes an electric stimulation backpack, a miniature camera, and an animal body. The electric stimulation backpack includes a wireless communication module and an electric stimulation module. The expert control system includes a human control expert, a wireless communication module, a computer, and a remote control interface. The miniature camera carried by the animal robot is used to shoot the navigation environment. These videos are transmitted to the computer in the expert control system in real time. The human control expert can view these videos in real time on the computer. The human expert performs a navigation control method and sends control instructions to each animal robot through the wireless communication module and the remote control interface to control it to complete the predetermined navigation task.

[0008] The S1 includes the following steps: S11, the multi-animal robot navigation system comprises N different animal robots, all the animal robots are in a shared environment, the state space is S, each agent has an observation space O i and an action space A i , the animal robots cooperatively complete a given task T S12, the multi-animal robot navigation system adopts a centralized control architecture, receives the capability information, observation data and navigation task description of all agents through a centralized control center, the centralized control center comprises a visual perception conversion module and an instruction generation module, and the centralized control center centrally collects and processes the information perceived by each robot and generates control instructions, the visual perception conversion module fuses the local observation information obtained by each robot into a global semantic map, and converts the global semantic map into a natural language description by using a BLIP-2 model, and the instruction generation module inputs the visual description information, navigation system information, historical trajectory information and instructions of a human control expert into a large language model as prompt words, so as to output a navigation thinking chain and generate control instructions

[0009] The S2 comprises the following steps: S21, obtaining an observation image of the animal robot i at time t obtaining a feature map F in sequence through a visual large model SAM i t , creating a BEV query firstly using the BEV feature of the last time and the BEV query to do time self-attention, and then doing spatial cross-attention with the feature map of the current time to generate a BEV feature B of the current time t ; S22, inputting the obtained BEV feature into a pre-trained MaskDecoder to output a semantic map, and using a pixel-level cross-entropy loss function L pixel Model training: wherein, represents a predicted semantic map, y represents a real label of a pixel, represents a probability that the model predicts the pixel to be the real label S23, inputting the output global semantic map into a BLIP-2 model to convert it into a natural language description E about the environment t ;

[0010] The S3 comprises the following steps: S31, the prompt words of the large language model comprise the following parts, visual description information E t converted and generated by BLIP-2 in the visual perception conversion stage, navigation system information G tThe historical trajectory information H includes the overall navigation task, the capability of each animal robot, the task, and the control mode t Record the observation and action before, the historical trajectory information, and the instruction I of the human control expert t ; S32, by combining the above prompt words with the thinking chain technology, the large model outputs the navigation thinking chain, and the thinking chain includes four intermediate reasoning steps: (1) state prediction, the large language model generates the imagination U of the next time state according to the instruction and visual observation t , (2) state filtering, the large language model selects the candidate observation V most matched with the predicted state t , (3) action prediction, the large language model performs animal robot action prediction a according to the state filtering result t , (4) instruction output, the observation information and the action prediction result are combined to output the control instruction; S33, based on the existing navigation data, the real labels of the navigation thinking chain are collected, and the large language model is pre-trained in the domain, and the pre-training target is to define the loss functions of the three navigation reasoning tasks: L1=-U*log(p LLM (U|E,G,H,I)) L2=-V*log(p LLM (V|E,G,H,I)) L3=-a*log(p LLM (a|E,G,H,I)) Wherein, L1, L2 and L3 represent the loss functions of the state prediction, state filtering and action prediction reasoning tasks respectively, and the total loss function of the pre-training is the sum of the three. S34, by constructing a diversified strategy group, output sampling is carried out, and the large model weight is adjusted according to the relative performance evaluation, the exploration and utilization are balanced, and the decoupling and dynamic sampling mechanism of the cutting parameter are introduced, and the overall optimization formula is as follows: Wherein, The importance sampling ratio of the new strategy relative to the old strategy is represented by The estimated value of the advantage function is represented by

[0011] The beneficial effects of the present application are: The system adopts a centralized control architecture, collects and processes information of each robot through a centralized control center, and provides the processing results to the operator for decision-making, thereby generating a globally optimal navigation scheme. The centralized control center includes a cooperative perception module and a cooperative control module. The cooperative perception module includes three core parts: image super-resolution, BEV feature extraction, and semantic segmentation. The image super-resolution part is used to improve image quality and solve problems caused by low-resolution images. The BEV feature extraction part fuses BEV queries with BEV features at the previous time and image features from multiple perspectives to generate BEV features with good spatiotemporal expression capability. The semantic segmentation part uses the fused high-quality images to generate a global semantic map. The cooperative control module adopts a multi-agent reinforcement learning algorithm with individualized training and distillation execution. The model is pre-trained through massive prior data to ensure that it can be quickly generalized to each agent and generate customized instructions, thereby improving the reliability of the instructions. The system realizes functional complementation among different types of animal robots, optimizes their cooperative work capability, and improves the success rate of navigation tasks: (1) Visual perception conversion extracts semantic information at different perspectives through a visual large model SAM, generates bird's eye view (BEV) representation with spatiotemporal features based on temporal self-attention and spatial cross-attention mechanisms, constructs a global semantic map through a semantic segmentation decoder, and converts the global semantic map into a natural language description using a BLIP-2 model, thereby expanding the perception range and generating high-quality observation descriptions. (2) The instruction generation module inputs visual description information, navigation system information, historical trajectory information, and instructions of human control experts as prompt words into a large language model, and outputs a navigation thought chain from the large model to generate control instructions. The thought chain includes four intermediate reasoning steps: state prediction, state filtering, action prediction, and instruction output, thereby improving the collaboration efficiency among animal robots. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, illustrate the application, and are used to explain the application, and do not constitute a limitation on the application.

[0013] Figure 1 A flowchart of a heterogeneous animal robot cooperative navigation system based on a large language model is provided.

[0014] Figure 2 An algorithm schematic diagram of a visual perception conversion module is provided.

[0015] Figure 3 A schematic diagram of an instruction generation module based on a large language model is provided. DETAILED DESCRIPTION

[0016] The application is further illustrated in combination with the drawings and examples.

[0017] Example 1

[0018] As Figure 1 shown, the system of the application adopts a centralized control architecture, which collects and processes information of each robot through a centralized control center, and the centralized control center includes a collaborative perception module and a collaborative control module: S1, the centralized control architecture collects and processes information provided by each animal robot through a centralized control center, and implements unified control decisions. First, the local observation information obtained by each robot is fused into a global semantic map. Then, the global semantic map is converted into natural language description through the BLIP-2 model. The observation information, navigation system information, historical trajectory information and instructions of human control experts are input into the large language model as prompt words to effectively guide the large model to generate control instructions for each animal robot, promote efficient cooperation between animal robots, optimize the task execution process, and significantly improve the stability and overall performance of the system. S2, the visual perception conversion module uses the visual large model SAM to extract semantic information under different perspectives, and generates bird's eye view (BEV) representation with spatio-temporal features based on time self-attention and spatial cross-attention mechanism. A global semantic map is constructed through a semantic segmentation decoder, and the BLIP-2 model is used to convert the global semantic map into natural language description. S3, the instruction generation module inputs the visual description information, navigation system information, historical trajectory information and instructions of human control experts into the large language model as prompt words, wherein the visual description information is the language description of the global language map, the navigation system information includes the overall navigation task, the ability of each animal robot, the task and the control mode, the historical trajectory information records the previous observation and action, and the human instruction information includes the description of the state and control instruction of each animal robot by human. Through the above prompt words combined with the thinking chain technology, the large model outputs the navigation thinking chain, which contains four intermediate reasoning steps: state prediction, state filtering, action prediction and instruction output, so as to improve the cooperation efficiency among animal robots.

[0019] As Figure 2 shown, the visual perception conversion module has the following steps: S21, the observation image of the animal robot i at time t is sequentially input into the visual large model SAM to obtain the feature map F i t , and a BEV query is created First, the BEV features of the previous time and the BEV query are used to do time self-attention, and then the spatial cross-attention is done with the feature map of the current time to generate the BEV feature B t; S22, input the obtained BEV feature into the pre-trained MaskDecoder, output the semantic map, and use the pixel-level cross-entropy loss function L pIxel Model training is performed: wherein, represents the predicted semantic map, y represents the true label of the pixel, represents the probability that the model predicts the pixel to be the true label; S23, input the output global semantic map into the BLIP-2 model, and convert it into a natural language description E about the environment t .

[0020] As Figure 3 shown, the instruction generation process based on the large model is as follows: S31, the prompt words of the large language model include the following parts, visual description information E t converted and generated by BLIP-2 according to the global semantic map in the visual perception conversion stage, navigation system information G t including the overall navigation task, the ability of each animal robot, the task and the control mode, historical trajectory information H t record the previous observation and action, historical trajectory information and the instruction I of the human control expert t ; S32, through the above prompt words combined with the thinking chain technology, the large model outputs the navigation thinking chain, and the thinking chain contains four intermediate reasoning steps: (1) state prediction, the large language model generates the imagination U of the next time state according to the instruction and visual observation t , (2) state filtering, the large language model selects the candidate observation V most matched with the predicted state t , (3) action prediction, the large language model performs action prediction a of the animal robot according to the result of state filtering t , (4) instruction output, combined with the observation information and the action prediction result, the control instruction is output; S33, collect the true labels of the navigation thinking chain based on the existing navigation data, and pre-train the large language model in the domain. The pre-training target is to define the loss functions of the three navigation reasoning tasks: L1=-U*log(p LLM (U|E,G,H,I)) L2=-V*log(p LLM (V|E,G,H,I)) L3=-a*log(p LLM (a|E,G,H,I)) Wherein, L1, L2, L3 respectively represent the loss function of state prediction, state filtering and action prediction reasoning task, and the total loss function of pre-training is the sum of the three; S34, by constructing a diversified strategy group, output sampling is carried out and the relative performance evaluation adjusts the large model weight, balances exploration and utilization, and simultaneously introduces decoupling and dynamic sampling mechanism of pruning parameters, and the overall optimization formula is as follows: Wherein, The importance sampling ratio of the new strategy relative to the old strategy is represented by The estimated value of the advantage function is represented by, which evaluates the good and bad of a certain action relative to the average situation.

[0021] In summary, the heterogeneous animal robot cooperative navigation method based on the large language model proposed in the application significantly improves the cooperative perception ability of the multi-animal robot system. Through this method, we can not only effectively expand the perception range of the navigation system, but also realize efficient cooperative navigation between multiple animal robots. This method provides strong support for the cooperative work of animal robots in complex environments.

[0022] Example 2

[0023] We use the heterogeneous animal robot navigation method based on the large model proposed in the application to perform cooperative navigation of pigeon robots and rat robots, and realize efficient search and navigation tasks in complex environments. In the experiment, we use multiple real rats with implanted electrodes and pigeons, equipped with camera sensors, and coordinate their actions through a human control system. First, the rat robot is released at the starting point of the experimental scene and starts moving in the experimental scene. The camera sensor carried by the rat robot collects real-time visual signals in front of it, and transmits them to the centralized control center through the wireless transmission module. At the same time, the pigeon robot flies from the air and stays on the platform set, uses its camera sensor to provide overall environmental visual information from above, and also transmits the visual signals to the centralized control center through the wireless transmission module.

[0024] The observation images of the rat robot and the pigeon robot at time t are sequentially passed through the visual large model SAM to obtain feature maps Create BEV query First, use the BEV features and BEV query of the last time to do time self-attention, and then do spatial cross-attention with the feature map of the current time to generate the BEV feature B of the current time t .

[0025] The obtained BEV features are input into the pre-trained MaskDecoder to output the semantic map, and the output global semantic map is input into the BLIP-2 model to convert it into a natural language description of the environment E t .

[0026] The prompt words for inputting the large model include the following parts: visual description information e t In the visual perception conversion stage, BLIP-2 generates the navigation system information G according to the global semantic map conversion. t Including the overall navigation task, the capabilities, tasks and control methods of the rat robot and the pigeon robot, and the historical trajectory information H t Record previous observations and actions, historical trajectory information, and instructions from human control experts I t ;

[0027] By navigating the thought chain prompt words, the large model outputs (1) state predictions in turn, and the large language model generates an imagination U of the observation state at the next moment based on the instructions and visual observations. t ,(2) State filtering, the large language model selects the candidate observation V that best matches the predicted state t ,(3) Action prediction, the large language model predicts the action of the animal robot based on the results of state filtering. t ,(4) command output, combining the observation information and the action prediction results to output the control commands of rats and pigeons;

[0028] To validate the effectiveness of this navigation method, we conducted two comparative experiments: a method based on the present invention and a method based on human expert control. The experimental sites consisted of a maze-slope and a maze-tunnel, each with different obstacle and passage designs. The maze-slope environment presented a challenge because the angles of the ramp and stairs were too steep for the rat robot, and the space at the end of the stairs for the rat robot to stand and turn was relatively narrow. The maze-tunnel environment also presented a challenge because the rat robot's position and state information could not be obtained during control.

[0029] We performed n search and navigation tasks in each area, recording the time required to complete each task, path deviation, and success rate. The experimental results are shown in Table 1 below:

[0030] Table 1 Experimental data comparison table

[0031] Area 1: The method of the present invention: the average completion time is 12.5 minutes, the path deviation is 1.2 meters, and the success rate is 61. The traditional method: the average completion time is 18.4 minutes, the path deviation is 2.8 meters, and the success rate is 54%.

[0032] Region 2: the method of the present application: the average completion time is 13.2 minutes, the path deviation is 1.1 meters, and the success rate is 64. The traditional method: the average completion time is 20.7 minutes, the path deviation is 3.1 meters, and the success rate is 60%.

[0033] From the above experimental data, it can be seen that the navigation performance of the method of the present application in a complex environment is significantly better than that of the animal robot system based on human control.

Claims

1. A heterogeneous animal-robot collaborative navigation system based on a large language model, characterized by: This is achieved by following these steps: S1. Adopt a centralized control system architecture, collect and process the information provided by each animal robot through the centralized control center to achieve collaborative perception and decision-making; S2, the visual perception conversion module, uses the visual large model SAM to extract semantic information from different perspectives, and generates a bird's-eye view (BEV) representation based on the multi-view information. It constructs a global semantic map through the semantic segmentation decoder and uses the BLIP-2 model to convert the global semantic map into a natural language description. S3, the instruction generation module, inputs visual description information, navigation system information, historical trajectory information, and instructions from human control experts as prompt words into the large language model, and combines the thought chain technology to guide the large model to output control quality; The navigation system includes the following parts: an expert control system for multiple animal robots, wherein the animal robots include an electric stimulation backpack, a micro camera, and an animal body, the electric stimulation backpack includes a wireless communication module and an electric stimulation module, the expert control system includes a human control expert, a wireless communication module, a computer and a remote control interface, the micro camera carried by the animal robots is used to shoot the navigation environment, and these videos are transmitted to the computer in the expert control system in real time. The human control expert can view these videos in real time on the computer, the human expert executes the navigation control method, and sends control instructions to each animal robot through the wireless communication module and the remote control interface to control it to complete the predetermined navigation task.

2. A heterogeneous animal-robot collaborative navigation system based on a large language model according to claim 1, characterized in that: Said S1 comprises the following steps: S11. The multi-animal robot navigation system consists of N different animal robots. All animal robots are in a shared environment with a state space of S. Each agent has an observation space O. i and action space A i , the animal robots collaboratively complete the given task T; S12. The multi-animal robot navigation system adopts a centralized control architecture. The centralized control center receives the ability information, observation data and navigation task description of the intelligent body. The control center contains a visual perception conversion module and an instruction generation module. The visual perception conversion module integrates local observation information into a global semantic map and converts it into a natural language description using the BLIP-2 model. The instruction generation module inputs the visual description, navigation information, historical trajectory and human expert instructions into a large language model to generate navigation thought chains and control instructions.

3. The heterogeneous animal-robot collaborative navigation system based on a large language model according to claim 1, characterized in that: The S2 comprises the following steps: S21, the observation image of the animal robot i at time t The feature map is obtained by sequentially passing the visual large model SAM Create a BEV query First, use the BEV features and BEV query of the previous moment to do temporal self-attention, and then do spatial cross-attention with the feature map of the current time to generate the BEV feature B at the current moment. t ; S22, input the obtained BEV features into the pre-trained MaskDecoder, output the semantic map, and use the pixel-level cross entropy loss function L pixel Perform model training: in, represents the predicted semantic map, y represents the true label of the pixel, Represents the probability that the model predicts the pixel to be the true label; S23. Input the output global semantic map into the BLIP-2 model and convert it into a natural language description of the environment E t .

4. The heterogeneous animal-robot collaborative navigation system based on a large language model according to claim 1, wherein S3 comprises the following steps: S31. The prompt words of the large language model include the following parts: visual description information E t In the visual perception conversion stage, BLIP-2 generates the navigation system information G according to the global semantic map conversion. t Including the overall navigation task, the capabilities of each animal robot, and the control method, historical trajectory information H t Record historical trajectory information and instructions from human control experts I t ; S32. By combining the above prompt words with the thought chain technology, the large model outputs the navigation thought chain. The thought chain includes four intermediate reasoning steps: (1) State prediction. The large language model generates an imagination of the next state based on the instructions and visual observations. t ,(2) State filtering, the large language model selects the candidate observation V that best matches the predicted state t ,(3) Action prediction, the large language model predicts the action of the animal robot based on the results of state filtering. t ,(4) command output, combining the observation information and the action prediction results to output the control command; S33. Based on existing navigation data, we collect the true labels of navigation thought chains and pre-train the large language model in the domain. The goal of pre-training is to define the loss functions for three navigation reasoning tasks: L1=-U*log(p LLM (U|E,G,H,I)) L2=-V*log(p LLM (V|E,G,H,I)) L3=-a*log(p LLM (a|E,G,H,I)) Among them, L1, L2, and L3 represent the loss functions of state prediction, state filtering, and action prediction reasoning tasks respectively, and the total loss function of pre-training is the sum of the three; S34. By building a diverse strategy group, performing output sampling and adjusting the large model weights based on relative performance evaluation, we balance exploration and utilization. At the same time, we introduce the decoupling of trimming parameters and the dynamic sampling mechanism. The overall optimization formula is as follows: in, represents the importance sampling ratio of the new strategy relative to the old strategy, Represents an estimate of the advantage function, which evaluates how good an action is relative to the average.

Images