Heterogeneous animal robot cooperative navigation system based on large language model

Abstract

This invention proposes a collaborative navigation system for heterogeneous animal robots based on a large language model. The system includes a centralized control center, a visual perception conversion module, and a command generation module. The visual perception conversion module extracts semantic information using a large visual model (SAM), generates a bird's-eye view (BEV) representation using a spatiotemporal attention mechanism, constructs a global semantic map through a semantic segmentation decoder, and converts it into a natural language description using a BLIP-2 model. The command generation module inputs visual descriptions, navigation information, historical trajectories, and human expert commands as prompts into the large language model, outputting a navigation thought chain that includes task allocation and command selection, comprising four intermediate steps: state prediction, filtering, action prediction, and command output. This system, through a centralized control structure and collaborative perception control method, enhances the perception, collaboration, and communication capabilities of heterogeneous animal robots, ensuring efficient navigation and decision-making in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of animal robot and cooperative navigation technology, and particularly relates to a heterogeneous animal robot cooperative navigation system based on a large language model. Background Technology

[0002] In modern rescue and search missions, animal robots have become a focus of research attention due to their flexible movement, long endurance, and good stealth capabilities. For example, in confined and complex environments such as earthquake ruins and underground pipes, rat robots can accomplish many tasks that are difficult for humans and mechanical robots due to their small size, excellent climbing ability, and adaptability to dim lighting. However, single-species animal robots also face significant limitations and challenges when performing these tasks.

[0003] To enable complementary capabilities among different animal robots, we propose a heterogeneous animal robot cooperative navigation system based on a large language model. This system employs a centralized control architecture, where a central control center collects and processes information from each robot, then transmits the processed information to the operator. The operator uses this information to make control decisions, thereby generating a globally optimal navigation plan.

[0004] The centralized control center comprises a visual perception conversion module and an instruction generation module. It centrally collects and processes information perceived by each robot and generates control instructions. The visual perception conversion module uses a large visual model (SAM) to extract semantic information from different perspectives and generates a bird's-eye view (BEV) representation with spatiotemporal characteristics based on temporal self-attention and spatial cross-attention mechanisms. A global semantic map is constructed using a semantic segmentation decoder, and the BLIP-2 model is used to convert the global semantic map into a natural language description. The instruction generation module inputs visual description information, navigation system information, historical trajectory information, and instructions from human control experts as prompts into the large language model. The large model outputs a navigation thought chain to generate control instructions. This thought chain includes four intermediate reasoning steps: state prediction, state filtering, action prediction, and instruction output, thereby improving the collaborative efficiency between animal robots. This invention, through a centralized control structure and effective collaborative perception and control methods, improves the perception, collaboration, and communication capabilities of heterogeneous animal robots, ensuring their efficient navigation and decision-making execution in complex environments. Summary of the Invention

[0005] The purpose of this invention is to propose a heterogeneous animal robot cooperative navigation system based on a large language model, which significantly improves the perception and navigation capabilities of multi-animal robot navigation systems in complex environments, and solves the problems of limited perception range and motion capabilities, as well as low control and navigation efficiency in existing technologies.

[0006] To address the aforementioned technical problems, this invention provides a heterogeneous animal robot cooperative navigation system based on a large language model, comprising the following steps:

[0007] S1. The centralized control architecture gathers and processes information provided by each animal robot through a centralized control center, and implements unified control decisions. This architecture first integrates the local observation information acquired by each robot into a global semantic map. Then, the global semantic map is converted into a natural language description through the BLIP-2 model. The observation information description, navigation system information, historical trajectory information, and instructions from human control experts are used as prompt words to input into the large language model, which effectively guides the large model to generate control instructions for each animal robot, promotes efficient collaboration among the animal robots, optimizes the task execution process, and significantly improves the stability and overall performance of the system.

[0008] S2, the visual perception conversion module, uses the visual large model SAM to extract semantic information from different perspectives, and generates a bird's-eye view (BEV) representation with spatiotemporal features based on temporal self-attention and spatial cross-attention mechanisms. It constructs a global semantic map through a semantic segmentation decoder and uses the BLIP-2 model to convert the global semantic map into a natural language description.

[0009] S3, the instruction generation module, inputs visual description information, navigation system information, historical trajectory information, and instructions from human control experts as prompts into the large language model. The visual description information is the language description of the global language map; the navigation system information includes the overall navigation task, the capabilities of each animal robot, the task itself, and the control method; the historical trajectory information records previous observations and actions; and the human instruction information includes descriptions of the state and control instructions given to each animal robot. By combining these prompts with thought chain technology, the large model outputs a navigation thought chain. This thought chain includes four intermediate reasoning steps: state prediction, state filtering, action prediction, and instruction output, thereby improving the collaboration efficiency between animal robots.

[0010] The navigation system comprises the following components: an expert control system for multiple animal robots. Each animal robot includes an electrostimulation backpack, a miniature camera, and the animal itself. The electrostimulation backpack includes a wireless communication module and an electrostimulation module. The expert control system includes a human control expert, a wireless communication module, a computer, and a remote control interface. The miniature camera carried by the animal robot is used to capture images of the navigation environment. These videos are transmitted in real-time to the computer in the expert control system. The human control expert can view these videos in real-time on the computer. The human expert executes navigation control methods and sends control commands to each animal robot via the wireless communication module and the remote control interface, controlling it to complete the predetermined navigation task.

[0011] S1 includes the following steps:

[0012] S11. This multi-animal robot navigation system contains N different animal robots. All animal robots are in a shared environment with a state space of S, and each agent has an observation space of O. i and action space A i Animal robots work together to complete a given task T;

[0013] S12. This multi-animal robot navigation system adopts a centralized control architecture. It receives the capability information, observation data, and navigation task descriptions of all intelligent agents through a centralized control center. The centralized control center includes a visual perception conversion module and an instruction generation module. It centrally collects and processes the information perceived by each robot and generates control instructions. The visual perception conversion module integrates the local observation information acquired by each robot into a global semantic map and uses the BLIP-2 model to convert the global semantic map into a natural language description. The instruction generation module takes the visual description information, navigation system information, historical trajectory information, and instructions from human control experts as prompt words and inputs them into the large language model, so that it can output the navigation thought chain and generate control instructions.

[0014] S2 includes the following steps:

[0015] S21. The observation image of animal robot i at time t. Feature maps F are obtained sequentially through the large visual model SAM. i t Create BEV query First, perform temporal self-attention using the BEV features and BEV query from the previous time step. Then, perform spatial cross-attention with the feature map from the current time step to generate the BEV features B at the current time step. t ;

[0016] S22. Input the obtained BEV features into the pre-trained MaskDecoder to output a semantic map, and use the pixel-level cross-entropy loss function L. pixel Model training:

[0017]

[0018] in, This represents the predicted semantic map, where y represents the true label of the pixel. This indicates the probability that the model predicts a pixel to be the true label;

[0019] S23. Input the output global semantic map into the BLIP-2 model and convert it into a natural language description E of the environment. t ;

[0020] S3 includes the following steps:

[0021] S31. The prompt words of the large language model include the following parts: visual descriptive information E t During the visual perception conversion phase, navigation system information G is generated by BLIP-2 based on the global semantic map. t This includes the overall navigation task, the capabilities of each animal robot, its tasks and control methods, and historical trajectory information H. t Record previous observations and actions, historical trajectory information, and instructions from human control experts. t ;

[0022] S32. By combining the above prompts with the thinking chain technology, the large model outputs a navigation thinking chain, which includes four intermediate reasoning steps: (1) State prediction: The large language model generates an imagination of the state at the next moment based on the instructions and visual observations. t (2) State filtering: The large language model selects the candidate observation V that best matches the predicted state. t (3) Action prediction: The large language model predicts the actions of the animal robot based on the results of state filtering. t (4) Output commands, combining observation information and action prediction results to output control commands;

[0023] S33. Based on existing navigation data, collect real labels for the navigation thought chain and perform domain-specific pre-training on the large language model. The goal of the pre-training is to define the loss functions for three navigation inference tasks:

[0024] L1 = -U*log(p LLM (U|E,G,H,I))

[0025] L2 = -V*log(p) LLM (V|E,G,H,I))

[0026] L3 = -a*log(p) LLM (a|E,G,H,I))

[0027] Here, L1, L2, and L3 represent the loss functions for state prediction, state filtering, and action prediction reasoning tasks, respectively, and the total loss function for pre-training is the sum of the three.

[0028] S34. By constructing a diverse policy group, sampling the output, and adjusting the weights of the large model based on relative performance evaluation, the exploration and exploitation are balanced. Simultaneously, a pruning parameter decoupling and dynamic sampling mechanism are introduced. The overall optimization formula is as follows:

[0029]

[0030] in, This represents the sampling ratio indicating the importance of the new strategy relative to the old strategy. It represents an estimate of the advantage function, which assesses the merits of a particular action relative to the average situation.

[0031] The beneficial effects of this invention are:

[0032] This system adopts a centralized control architecture, collecting and processing information from each robot through a central control center. The processing results are then provided to the operator for decision-making, thereby generating a globally optimal navigation plan. The central control center includes a collaborative perception module and a collaborative control module. The collaborative perception module comprises three core parts: image super-resolution, BEV feature extraction, and semantic segmentation. The image super-resolution part improves image quality and addresses the problems caused by low image resolution. The BEV feature extraction part fuses BEV queries with BEV features from the previous time step and image features from multiple perspectives to generate BEV features with good spatiotemporal representation capabilities. The semantic segmentation part uses the fused high-quality image to generate a global semantic map. The collaborative control module employs a multi-agent reinforcement learning algorithm with personalized training and distillation execution. It pre-trains the model using massive amounts of prior data to ensure rapid generalization to various agents and generation of customized instructions, thereby improving the reliability of the instructions. This system achieves functional complementarity between different types of animal robots, optimizes their collaborative operation capabilities, and improves the success rate of navigation tasks.

[0033] (1) Visual perception conversion: The semantic information from different perspectives is extracted by the visual big model SAM, and a bird’s-eye view (BEV) representation with spatiotemporal characteristics is generated based on the temporal self-attention and spatial cross-attention mechanism. A global semantic map is constructed by the semantic segmentation decoder, and the global semantic map is converted into a natural language description using the BLIP-2 model, which expands the perception range and generates a high-quality observation description.

[0034] (2) The instruction generation module inputs visual description information, navigation system information, historical trajectory information and instructions from human control experts as prompt words into the large language model. The large model outputs a navigation thought chain to generate control instructions. The thought chain includes four intermediate reasoning steps: state prediction, state filtering, action prediction and instruction output, thereby improving the collaboration efficiency between animal robots. Attached Figure Description

[0035] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0036] Figure 1 This is a flowchart of a heterogeneous animal robot cooperative navigation system based on a large language model, as proposed in this invention.

[0037] Figure 2 This is a schematic diagram of the visual perception conversion module algorithm.

[0038] Figure 3 This is a schematic diagram of an instruction generation module based on a large language model. Detailed Implementation

[0039] The present invention will be further described in conjunction with the accompanying drawings and embodiments.

[0040] Example 1

[0041] like Figure 1 As shown, the system of this invention adopts a centralized control architecture, which collects and processes information from each robot through a centralized control center. The centralized control center includes a collaborative perception module and a collaborative control module.

[0042] S1. The centralized control architecture gathers and processes information provided by each animal robot through a centralized control center, and implements unified control decisions. This architecture first integrates the local observation information acquired by each robot into a global semantic map. Then, the global semantic map is converted into a natural language description through the BLIP-2 model. The observation information description, navigation system information, historical trajectory information, and instructions from human control experts are used as prompt words to input into the large language model, which effectively guides the large model to generate control instructions for each animal robot, promotes efficient collaboration among the animal robots, optimizes the task execution process, and significantly improves the stability and overall performance of the system.

[0043] S2, the visual perception conversion module, uses the visual large model SAM to extract semantic information from different perspectives, and generates a bird's-eye view (BEV) representation with spatiotemporal features based on temporal self-attention and spatial cross-attention mechanisms. It constructs a global semantic map through a semantic segmentation decoder and uses the BLIP-2 model to convert the global semantic map into a natural language description.

[0044] S3, the instruction generation module, inputs visual description information, navigation system information, historical trajectory information, and instructions from human control experts as prompts into the large language model. The visual description information is the language description of the global language map; the navigation system information includes the overall navigation task, the capabilities of each animal robot, the task itself, and the control method; the historical trajectory information records previous observations and actions; and the human instruction information includes descriptions of the state and control instructions given to each animal robot. By combining these prompts with thought chain technology, the large model outputs a navigation thought chain. This thought chain includes four intermediate reasoning steps: state prediction, state filtering, action prediction, and instruction output, thereby improving the collaboration efficiency between animal robots.

[0045] like Figure 2 As shown, the steps of the visual perception conversion module are as follows:

[0046] S21. The observation image of animal robot i at time t. Feature maps F are obtained sequentially through the large visual model SAM. i t Create BEV query First, perform temporal self-attention using the BEV features and BEV query from the previous time step. Then, perform spatial cross-attention with the feature map from the current time step to generate the BEV features B at the current time step. t ;

[0047] S22. Input the obtained BEV features into the pre-trained MaskDecoder to output a semantic map, and use the pixel-level cross-entropy loss function L. pIxel Model training:

[0048]

[0049]

[0050] in, This represents the predicted semantic map, where y represents the true label of the pixel. This indicates the probability that the model predicts a pixel to be the true label;

[0051] S23. Input the output global semantic map into the BLIP-2 model and convert it into a natural language description E of the environment. t .

[0052] like Figure 3 As shown, the steps of the instruction generation process based on the large model are as follows:

[0053] S31. The prompt words of the large language model include the following parts: visual descriptive information E t During the visual perception conversion phase, navigation system information G is generated by BLIP-2 based on the global semantic map. t This includes the overall navigation task, the capabilities of each animal robot, its tasks and control methods, and historical trajectory information H. t Record previous observations and actions, historical trajectory information, and instructions from human control experts. t ;

[0054] S32. By combining the above prompts with the thinking chain technology, the large model outputs a navigation thinking chain, which includes four intermediate reasoning steps: (1) State prediction: The large language model generates an imagination of the state at the next moment based on the instructions and visual observations. t (2) State filtering: The large language model selects the candidate observation V that best matches the predicted state. t(3) Action prediction: The large language model predicts the actions of the animal robot based on the results of state filtering. t (4) Output commands, combining observation information and action prediction results to output control commands;

[0055] S33. Based on existing navigation data, collect real labels for the navigation thought chain and perform domain-specific pre-training on the large language model. The goal of the pre-training is to define the loss functions for three navigation inference tasks:

[0056] L1 = -U*log(p LLM (U|E,G,H,I))

[0057] L2 = -V*log(p) LLM (V|E,G,H,I))

[0058] L3 = -a*log(p) LLM (a|E,G,H,I))

[0059] Where L1, L2, and L3 represent the loss functions for the state prediction, state filtering, and action prediction reasoning tasks, respectively, and the total loss function for pre-training is the sum of the three.

[0060] S34. By constructing a diverse policy group, sampling the output, and adjusting the weights of the large model based on relative performance evaluation, the exploration and exploitation are balanced. Simultaneously, a pruning parameter decoupling and dynamic sampling mechanism are introduced. The overall optimization formula is as follows:

[0061]

[0062] in, This represents the sampling ratio indicating the importance of the new strategy relative to the old strategy. It represents an estimate of the advantage function, which assesses the merits of a particular action relative to the average situation.

[0063] In summary, the proposed heterogeneous animal robot cooperative navigation method based on a large language model significantly improves the cooperative perception capability of multi-animal robot systems. This method not only effectively expands the perception range of the navigation system but also enables efficient cooperative navigation among multiple animal robots. This approach provides strong support for collaborative operations of animal robots in complex environments.

[0064] Example 2

[0065] We used the proposed large-model-based heterogeneous animal robot navigation method to perform cooperative navigation on pigeon and rat robots, achieving efficient search and navigation in complex environments. In the experiment, we selected multiple real rat and pigeon robots with implanted electrodes in their brains, each equipped with camera sensors, and coordinated their actions through a human control system. First, the rat robots were deployed to the starting point of the experimental scene and began moving within it. The camera sensors on the rat robots collected visual signals from in front of them in real time and transmitted them to the central control center via a wireless transmission module. Simultaneously, the pigeon robots flew from the air and landed on a pre-set platform, using their camera sensors to provide overall environmental visual information from above, also transmitting visual signals to the central control center via a wireless transmission module.

[0066] The observation images of the rat robot and the pigeon robot at time t are sequentially processed through the large visual model SAM to obtain feature maps. Create BEV query First, perform temporal self-attention using the BEV features and BEV query from the previous time step. Then, perform spatial cross-attention with the feature map from the current time step to generate the BEV features B at the current time step. t .

[0067] The obtained BEV features are input into a pre-trained MaskDecoder, which outputs a semantic map. This global semantic map is then input into the BLIP-2 model to convert it into a natural language description (E) of the environment. t .

[0068] The input prompts for large models include the following parts: visual descriptive information e t During the visual perception conversion phase, navigation system information G is generated by BLIP-2 based on the global semantic map. t This includes the overall navigation task, the capabilities, tasks, and control methods of the rat and pigeon robots, and historical trajectory information H. t Record previous observations and actions, historical trajectory information, and instructions from human control experts. t ;

[0069] Through navigation thought chain prompts, the large model sequentially outputs (1) state predictions, and the large language model generates an imagined state U for the next moment based on instructions and visual observations. t (2) State filtering: The large language model selects the candidate observation V that best matches the predicted state. t (3) Action prediction: The large language model predicts the actions of the animal robot based on the results of state filtering. t (4) Output instructions: Combine observation information and action prediction results to output control instructions for rats and pigeons;

[0070] To verify the effectiveness of this navigation method, we set up two control groups: a comparative experiment using the method of this invention and a method based on human expert control. The experimental sites included a maze-ramp and a maze-tunnel, with different obstacle and passage designs. The challenge of the maze-ramp environment lies in the steep angles of the ramps and stairs for the rat robot, and the limited space at the end of the stairs for the rat robot to stand and turn. The maze-tunnel environment presents the challenge of not being able to obtain the rat robot's position and status information during control.

[0071] We conducted n search and navigation tasks in each area, recording the time, path deviation, and success rate for each task. The experimental results are shown in Table 1 below:

[0072] Table 1 Comparison of Experimental Data

[0073]

[0074] Area 1: The method of this invention: average completion time is 12.5 minutes, path deviation is 1.2 meters, and success rate is 61%. Traditional method: average completion time is 18.4 minutes, path deviation is 2.8 meters, and success rate is 54%.

[0075] Area 2: The method of this invention: average completion time is 13.2 minutes, path deviation is 1.1 meters, and success rate is 64%. Traditional method: average completion time is 20.7 minutes, path deviation is 3.1 meters, and success rate is 60%.

[0076] As can be seen from the experimental data above, the method of the present invention has significantly better navigation performance in complex environments than animal robot systems based on human control.

Claims

1. A heterogeneous animal robot cooperative navigation system based on a large language model, characterized in that, This can be achieved through the following steps: S1. A centralized control architecture is adopted, which collects and processes information provided by each animal robot through a centralized control center to achieve collaborative perception and decision-making; S2, the visual perception conversion module, uses the visual large model SAM to extract semantic information from different perspectives, and generates a bird's-eye view representation based on multi-view information. It constructs a global semantic map through a semantic segmentation decoder and uses the BLIP-2 model to convert the global semantic map into a natural language description. S3, the instruction generation module, takes visual description information, navigation system information, historical trajectory information, and instructions from human control experts as prompt words and inputs them into the large language model. Combined with the thought chain technology, it guides the large model to output control instructions. The navigation system comprises the following components: an expert control system for multiple animal robots, each animal robot including an electric stimulation backpack, a miniature camera, and the animal itself; the electric stimulation backpack including a wireless communication module and an electric stimulation module; and the expert control system including a human control expert, a wireless communication module, a computer, and a remote control interface. The miniature camera carried by the animal robot is used to capture the navigation environment, and these videos are transmitted in real time to the computer in the expert control system. The human control expert can view these videos in real time on the computer. The human expert executes the navigation human control method and sends control commands to each animal robot through the wireless communication module and the remote control interface to control it to complete the predetermined navigation task. S1 includes the following steps: S11. A multi-animal robot navigation system contains N different animal robots. All animal robots are in a shared environment with a state space of S. Each agent has an observation space O_i and an action space A_i. The animal robots cooperate to complete a given task T. S12. This multi-animal robot navigation system adopts a centralized control architecture. The centralized control center receives the agent's capability information, observation data, and navigation task description. The control center includes a visual perception conversion module and an instruction generation module. The visual perception conversion module fuses local observation information into a global semantic map and converts it into a natural language description using a BLIP-2 model. The instruction generation module inputs the visual description, navigation information, historical trajectory, and human control expert instructions into a large language model to generate navigation thought chains and control instructions. S3 includes the following steps: S31. The prompt words of the large language model include the following parts: visual description information E_t is generated by BLIP-2 based on the global semantic map during the visual perception conversion stage; navigation system information G_t includes the overall navigation task, the capabilities of each animal robot, and the control method; historical trajectory information H_t records historical trajectory information and instructions I_t from human control experts. S32. By combining the above prompts with the thinking chain technology, the large model outputs the navigation thinking chain, which includes four intermediate reasoning steps: (1) State prediction: the large language model generates the imagination of the state at the next moment U_t based on the instructions and visual observations; (2) State filtering: the large language model selects the candidate observation V_t that best matches the predicted state; (3) Action prediction: the large language model performs the action prediction a_t of the animal robot based on the state filtering results; (4) Instruction output: the control instructions are output by combining the observation information and the action prediction results. S33. Based on existing navigation data, collect real labels for the navigation thought chain and perform domain-specific pre-training on the large language model. The goal of the pre-training is to define the loss functions for three navigation inference tasks: L_1=-U* log(p_LLM (U|E,G,H,I)) L_2=-V*log(p_LLM (V│E,G,H,I)) L_3=-a* log(p_LLM (a|E,G,H,I)) Where L_1, L_2, and L_3 represent the loss functions for the state prediction, state filtering, and action prediction reasoning tasks, respectively, and the total loss function for pre-training is the sum of the three. S34. By constructing a diverse policy group, sampling the output, and adjusting the weights of the large model based on relative performance evaluation, the exploration and exploitation are balanced. Simultaneously, a pruning parameter decoupling and dynamic sampling mechanism are introduced. The overall optimization formula is as follows: ; where, represents the importance sampling ratio of the new policy relative to the old policy, represents the estimated value of the advantage function, assessing how good a certain action is relative to the average.

2. The heterogeneous animal robot cooperative navigation system based on a large language model according to claim 1, wherein, S2 includes the following steps: S21. The observation image of animal robot i at time t. Feature maps are obtained sequentially through the large visual model SAM. Create BEV query First, perform temporal self-attention using the BEV features and BEV query from the previous time step, then perform spatial cross-attention with the feature map of the current time step to generate the BEV features of the current time step. ; S22, input the obtained BEV feature into the pre-trained MaskDecoder, output a semantic map, and use a pixel-level cross-entropy loss function Model training is performed: ; in, This represents the predicted semantic map, where y represents the true label of the pixel. This indicates the probability that the model predicts a pixel to be the true label; S23. Input the output global semantic map into the BLIP-2 model and convert it into a natural language description of the environment. .

Images