Exhibition hall robot visual language navigation method based on large model

Abstract

The invention discloses an exhibition hall robot visual language navigation method based on a large model, belongs to the field of exhibition hall robots, and aims at solving the problems existing in an existing exhibition hall robot. According to the method, an exhibition hall robot multi-mode scene sensing system senses a current scene, and candidate target objects are obtained through fusion features of RGB images and laser point clouds; extracting image features of all candidate target objects in the scene; according to the exhibition hall robot multi-mode environment understanding system, an exhibition hall robot receives a voice instruction, a visual language multi-mode large model is used for understanding a target object and a user intention, then high-dimensional semantic features of a target object text are extracted from the system, and the target object is found out from candidate target objects; according to the exhibition hall robot intelligent navigation system based on the multi-modal large model, an exhibition hall robot carries out path planning and autonomous movement according to a pose sequence generated by an intention, and self behaviors are dynamically adjusted according to a real-time changing scene provided by the exhibition hall robot multi-modal scene sensing system.

Description

Technical Field

[0001] The invention relates to robot navigation technology and belongs to the field of exhibition hall robots. Background Art

[0002] Exhibition hall robots have been widely used in various exhibitions, museums and commercial places, becoming an important tool to improve operational efficiency and reduce labor costs. However, despite the significant progress in related technologies, most existing robot systems still rely on predefined navigation tasks and path planning, which leads to certain limitations of these systems when facing complex and changing exhibition hall environments. Especially in terms of human-computer interaction, most of the current systems adopt coding-based interaction methods, which not only increases the complexity of robot operation, but also puts high requirements on the operator's professional knowledge, thus limiting the wide application of these methods in practical scenarios. In order to overcome these limitations, in recent years, Visual Language Navigation (VLN) technology has gradually become a solution with broad application prospects. VLN allows users to communicate with robots through natural language, and the robot performs the corresponding navigation tasks through visual understanding of the scene and parsing of language instructions. However, the current VLN causes navigation uncertainty due to insufficient observation of the environment and vague understanding of instructions.

[0003] Exhibition hall robots have been gradually applied to various complex and changeable exhibition hall environments due to their strong flexibility and reliability. The core task of exhibition hall robots is to sense the environment through their own sensors without human intervention, obtain information about unknown environments, and perform real-time modeling and analysis of the environment; and then complete tasks such as guiding visitors to visit the exhibition hall, providing route guidance, introducing exhibit information, and answering visitors' questions.

[0004] Current navigation technologies are mainly divided into two categories: navigation methods based on traditional SLAM algorithms and navigation methods based on visual language models. Simultaneous Localization And Mapping (SLAM) is a traditional technical solution to achieve this task of exhibition hall robots. SLAM technology was first used in the field of robotics, hoping that without any prior knowledge, the robot could build a map of the surrounding environment in real time based on the information from the sensors, and at the same time, infer its own position based on this map. Depending on the type of sensor used, SLAM can be divided into radar-based SLAM and vision-based SLAM. A complete SLAM system is mainly composed of sensor data streams, front-end tracking modules (visual odometers), back-end optimization modules, loop detection modules, and map construction modules.

[0005] Current SLAM includes vision-based SLAM and SLAM based on LiDAR point cloud data. Montemerlo et al. proposed the FastSLAM framework (A factored solution to the simultaneous localization and mapping problem [J]. Aaai / iaai, 2002, 593598: 593-598.), in which Kalman filtering is used to estimate the position of landmarks and particle filtering is used to estimate the position and posture of the system, so that the robot path and landmark distribution over time are relatively independent. During back-end optimization, the posterior is decomposed into the product of conditional landmark distribution and path distribution according to the principle of conditional independence, which simplifies the complexity of the calculation process compared to EKF SLAM. GMapping is an improvement on FastSLAM (Improved techniques for grid mapping with rao-blackwellized particle filters[J].IEEE transactions on Robotics,2007,23(1):34-46.). It solves the memory explosion problem that may exist in FastSLAM by reducing the number of particles; it uses a selective resampling method to sort the particles by importance weight and resample the examples with low weights. The GMapping algorithm has good real-time performance and high accuracy in indoor environments. Zhang et al. proposed the LOAM algorithm using the feature point method (Lidar odometry and mapping in real-time[C] / / Robotics:Science and systems.2014,2(9):1-9.). It uses points in a local range to calculate the curvature of the points and classifies the feature points into edge feature points and plane feature points according to the size of the curvature; the algorithm makes a uniform speed assumption for the lidar and uses linear interpolation for motion compensation.Shan et al. added loop detection function based on Loam (Lightweight and ground-optimized lidar odometry and mapping on variable terrain [C] / / 2018IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018: 4758-4765.), introduced the concept of keyframes, and used keyframes and data frames in their local range to form loop-submaps, which can greatly reduce the amount of calculation, filter redundant data, and match the current data frame with the loop-submap to achieve the purpose of loop detection. However, traditional SLAM technology relies on pre-established scene maps and cannot cope with dynamically changing environments or natural interactions with humans. In addition, traditional SLAM methods mainly rely on the geometric features of the environment, ignoring the fusion of semantic information and natural language, which limits the ability of robots to perform tasks in complex environments.

[0006] In view of the limitations of the above-mentioned traditional navigation methods, navigation methods based on visual language have gradually become a research hotspot in recent years. Such methods aim to enable robots to understand and perform more complex and humanized navigation tasks by integrating visual information with natural language instructions. For example, exhibition hall robots can realize autonomous navigation and task execution by processing information such as images, videos, and point clouds, combined with natural language instructions. This method can not only adapt to changes in dynamic environments, but also interact with humans in a more natural and intuitive way.

[0007] Visual language navigation methods enable mobile robots to navigate autonomously in unknown environments, that is, there is no need to build maps in advance, and the robot makes decisions in real time based on each perception module. Peter et al. created the Room-to-Room (R2R) dataset (Interpreting visually-grounded navigation instructions in real environments [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 3674-3683.), which provides rich natural language instructions and corresponding real environment panoramic images for visual language navigation tasks. However, this work mainly uses RGB sensors to perceive and understand the environment, and does not include depth information of the scene or other high-level visual features that may be helpful for navigation tasks. Wang et al. combined model-free and model-based reinforcement learning methods to solve the visual language navigation task (Neural machine translation by jointly learning to align and translate [J]. arXiv preprint arXiv: 1409.0473, 2014.). By combining the forward-looking module with the decision model, the best action for the next step is predicted, and the next state and reward are predicted through an environmental model, and the model is verified on the R2R dataset. Similar to Peter et al., this study mainly relies on RGB images to perceive and understand the environment, using the color information of the scene to guide the navigation task. Although RGB images provide rich visual details, this approach does not integrate depth information or other high-level visual features, which to some extent limits the mobile robot's comprehensive understanding of the three-dimensional structure of the environment. Li et al. use a large-scale pre-trained language model to learn text representation (Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation[C] / / Proceedings ofthe European Conference on Computer Vision(ECCV).2018:37-53.) to better generalize to unlearned instructions, and propose a random sampling scheme to reduce the error between expert actions during training and sampled actions during testing, so that the mobile robot can learn to correct its own mistakes during long-term sequence action decoding.Hao et al. proposed a new pre-training method that enables the model to better capture the relationship between vision and language, thereby enhancing its generalization ability, by performing multi-task learning on large-scale and diverse datasets (Towards learning a generic agent for vision-and-language navigation via pre-training [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 13137-13146.). This method combines visual representation learning, language understanding, and multimodal alignment to construct an agent with generalization characteristics in the pre-training stage. To construct the dataset, the authors used a variety of scenes and path generation methods to simulate the diversity in the actual environment. At the same time, the article proposed an improved multimodal alignment mechanism, which enables visual features to match language descriptions more accurately, thereby improving navigation accuracy. Chen et al. proposed a history-aware multimodal Transformer, which uses the historical visual, language, and action information obtained by the agent throughout the navigation process (Historyaware multimodal transformer for vision-and-language navigation[J]. Advances in neural information processing systems, 2021, 34: 5834-5847.), encodes it into a contextual representation, and then uses a multimodal Transformer for temporal modeling to capture the dependency between historical information and the current navigation state. Although these methods have made significant progress in processing visual and language navigation tasks, they still have certain shortcomings when faced with ambiguous instructions. These shortcomings are mainly reflected in the difficulty of the model to effectively parse and execute unclear or ambiguous instructions, which affects its robustness and practicality in practical applications. Summary of the invention

[0008] In view of the problems existing in the existing exhibition hall robots, the present invention provides a visual language navigation method for the exhibition hall robot based on a large model.

[0009] The present invention discloses a large-model-based exhibition hall robot visual language navigation method, which comprises an exhibition hall robot multimodal scene perception system, an exhibition hall robot multimodal environment understanding system and an exhibition hall robot intelligent navigation system based on a multimodal large model;

[0010] Exhibition hall robot multimodal scene perception system: perceive the current scene, where the camera is used to collect RGB images, the radar is used to collect laser point clouds, and then the candidate target objects are obtained by the fusion features of the RGB images and the laser point clouds; at the same time, the image features of all candidate target objects in the scene are extracted, and the image features include visual features and spatial features;

[0011] Exhibition hall robot multimodal environment understanding system: The exhibition hall robot receives voice commands and uses a large multimodal model of visual language to understand the target object and user intention. It then extracts high-dimensional semantic features of the target object text from the system, aligns these high-dimensional semantic features with the image features of each candidate target in the current scene perceived by the exhibition hall robot multimodal scene perception system, and finds the target object from the candidate target objects.

[0012] Intelligent navigation system for exhibition hall robots based on multimodal large models: The exhibition hall robot generates a posture sequence based on the intention understood by the exhibition hall robot's multimodal environment understanding system. The exhibition hall robot performs path planning and autonomous movement according to the generated posture sequence, and dynamically adjusts its own behavior according to the real-time changing scene provided by the exhibition hall robot's multimodal scene perception system.

[0013] Preferably, during the navigation process, when the recognition confidence of the visual language multimodal large model for the target object is lower than a set threshold, the system actively initiates multiple rounds of dialogue with the user for interactive confirmation to adjust the posture sequence, thereby adjusting its positioning and path planning.

[0014] Preferably, the fusion features of the RGB image and the laser point cloud are obtained through a multi-branch heterogeneous multi-source feature fusion module, specifically:

[0015] Using a camera to capture RGB images Collecting laser point clouds using radar N is the number of point clouds;

[0016] S1. Extract image features F I and point cloud features F P ;

[0017] F I =FeatExtra I (I), where I is an RGB image, and H is the height of the image, W is the width of the image; FeatExtra I is an image feature extractor;

[0018] F P =FeatExtra P (P), where P is the laser point cloud, and N is the number of point clouds; FeatExtra PPoint cloud feature extractor

[0019] S2, the image feature F I and point cloud features F P Serialization;

[0020] Serialized image features S I :S I = ImagetoSeq(F I ), where ImagetoSeq means converting an image into a sequence;

[0021] Serialized point cloud features S P :S P = ImagetoSeq(F P );

[0022] S3. Obtain the fused sequence feature F seq : F seq =SelfAtten(CrossAtten(S I ,S P ),CrossAtten(S I ,S P ))

[0023] Among them, SelfAtten means performing self-attention operation between sequences, and CrossAtten means performing cross-attention operation between sequences;

[0024] S4, obtain the fusion feature F of RGB image and laser point cloud: F = SeqtoImage(F seq ), where SeqtoImage means converting a sequence into an image.

[0025] Preferably, the candidate target objects are obtained by utilizing the fusion features of the RGB image and the laser point cloud and a detection object occupancy probability prediction network is adopted. The detection object occupancy probability prediction network takes the fusion feature F as input, estimates the probability that each pixel in the image and its corresponding point cloud are occupied by a specific object, obtains the correspondence between the semantic identification and the spatial distribution of the object in the image, generates candidate target objects, and after identification, encodes each candidate target object and assigns a unique ID.

[0026] Preferably, the process of using the visual language multimodal large model to understand the target object and intention is:

[0027] First, the voice command is converted into command text. The visual language multimodal large model deeply understands the command text. It first parses the semantic structure of the sentence, identifies the text involving actions and the text of the target object, and then reasoning based on the context semantics to understand the user's intention.

[0028] Preferably, after the visual language multimodal large model understands the target object, the system extracts high-dimensional semantic features of the target object text, wherein the high-dimensional semantic features include concepts, shapes, colors, and usage information related to the target object.

[0029] Preferably, the process of finding the target object from the candidate target objects is:

[0030] The target object understood by the visual language multimodal large model calculates the similarity between the high-dimensional semantic features corresponding to the target object text and the image features of the candidate target object in the current scene, and selects the candidate target object with the highest similarity as the target object.

[0031] Preferably, the process of dynamically adjusting the robot's behavior in navigation according to the real-time changing scene provided by the exhibition hall robot's multimodal scene perception system is as follows:

[0032] Initially, the exhibition hall robot's multimodal environment understanding system generates a posture sequence based on the intention understood by the voice command, and the exhibition hall robot starts to move according to the path planning generated by the posture sequence;

[0033] During the movement of the exhibition hall robot, the exhibition hall robot's multimodal scene perception system continuously perceives the spatial distribution in the scene. The spatial distribution includes the positioning of candidate target objects and the spatial relationship with other obstacles. The exhibition hall robot adjusts its own positioning and path planning according to the scene changes.

[0034] Beneficial effects of the invention: This paper proposes a visual language navigation method for exhibition hall robots based on a large language model. This method makes full use of the powerful capabilities of the current large language model (LLM) in language understanding and reasoning, enabling the robot to accurately understand and execute complex instructions based on natural language, providing a more natural and intuitive user interaction experience. The system not only improves the navigation flexibility of the robot, but also greatly reduces the user's operating threshold, allowing non-professionals to easily operate the robot.

[0035] In view of the shortcomings of VLN in environmental observation, the system first uses a high-resolution image segmentation network to accurately segment the exhibition hall scene, thereby extracting key environmental features. Then, the system fuses these semantic segmentation results with the point cloud data obtained by the lidar through a feature matching algorithm to construct a detailed three-dimensional semantic map. This fusion process enables the robot to have a three-dimensional perception of the exhibition hall environment, so that it can understand the environment more accurately and navigate effectively. In view of the fuzzy understanding of VLN's instructions, with the help of the powerful parsing ability of the large language model, a multi-label decision model is constructed to convert complex natural language into target-oriented instructions and path-oriented instructions. By generating specific pose sequences, the robot can move autonomously according to the user's instructions and dynamically adjust its behavior according to the real-time changing scene. This navigation method not only enhances the robot's adaptability in the dynamic exhibition hall environment, but also significantly improves the robot's execution efficiency in complex tasks.

[0036] The visual language navigation adopted by the present invention has the following advantages: first, it enables the robot to understand and execute human instructions more naturally. By combining visual information and natural language processing, the robot can directly extract the core information of the navigation task, such as the target location or route requirements, from the voice or text instructions. This capability not only simplifies the interaction process with the robot, but also enables the robot to better meet the personalized needs of visitors without cumbersome pre-programming or specific instruction formats. Secondly, visual language navigation improves the robot's environmental adaptability. In the dynamically changing exhibition hall environment, traditional navigation methods often rely on pre-built static maps, while visual language navigation allows the robot to perceive and understand changes in the surrounding environment in real time. For example, the robot can identify newly placed exhibits, temporarily built display areas, and even the distribution of people in the exhibition hall. This dynamic environment perception capability enables the robot to flexibly adjust its navigation path, avoid obstacles, and ensure the safety and accuracy of the navigation process.

[0037] Secondly, visual language navigation also enhances the robot's interactive capabilities. By engaging in natural language dialogue with humans, the robot can obtain more environmental context information or clarify uncertain instructions, thereby making more informed decisions in complex or ambiguous situations. This not only improves the robot's success rate in performing tasks, but also provides visitors with a more friendly and interactive experience.

[0038] Finally, visual language navigation helps improve the overall service level of the exhibition hall. Through visual language navigation technology, the robot can provide visitors with accurate guided services, answer questions, and recommend the most suitable tour route based on the visitors' points of interest. This intelligent service can greatly improve visitor satisfaction, while also reducing the burden on exhibition hall staff and making exhibition hall management more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a principle block diagram of a large-model-based exhibition hall robot visual language navigation method described in the present invention;

[0040] Figure 2 It is the point cloud-semantic map feature matching flowchart. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0043] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0044] Specific implementation method 1: The following is combined Figure 1 and Figure 2 The present embodiment is described. The present embodiment describes a large-model-based exhibition hall robot visual language navigation method, which includes an exhibition hall robot multimodal scene perception system, an exhibition hall robot multimodal environment understanding system, and an exhibition hall robot intelligent navigation system based on a multimodal large model.

[0045] Exhibition hall robot multimodal scene perception system: perceive the current scene, where the camera is used to collect RGB images, the radar is used to collect laser point clouds, and then the candidate target objects are obtained by the fusion features of the RGB images and the laser point clouds; at the same time, the image features of all candidate target objects in the scene are extracted, and the image features include visual features and spatial features;

[0046] Exhibition hall robot multimodal environment understanding system: The exhibition hall robot receives voice commands and uses a large multimodal model of visual language to understand the target object and user intention. It then extracts high-dimensional semantic features of the target object text from the system, aligns these high-dimensional semantic features with the image features of each candidate target in the current scene perceived by the exhibition hall robot multimodal scene perception system, and finds the target object from the candidate target objects.

[0047] Intelligent navigation system for exhibition hall robots based on multimodal large models: The exhibition hall robot generates a posture sequence based on the intention understood by the exhibition hall robot's multimodal environment understanding system. The exhibition hall robot performs path planning and autonomous movement according to the generated posture sequence, and dynamically adjusts its own behavior according to the real-time changing scene provided by the exhibition hall robot's multimodal scene perception system.

[0048] During the navigation process, when the recognition confidence of the target object by the visual language multimodal large model is lower than the set threshold, the system actively initiates multiple rounds of dialogue with the user for interactive confirmation to adjust the posture sequence, thereby adjusting its positioning and path planning.

[0049] Exhibition hall robot multimodal scene perception system: Exhibition hall robots can perform two types of tasks according to instructions: target navigation tasks and route navigation tasks. Target navigation tasks require the robot to use its own perception system (camera, radar, etc.) to use the scene perception module to locate the target in the complex exhibition hall environment (identify all objects in the exhibition hall, including fixed exhibits, dynamic objects (such as pedestrians) and temporary objects (such as equipment boxes, display props, etc.). Each object after identification will be assigned a unique ID, which is not only used to identify the object, but also plays a key role in the subsequent navigation process.) to provide support for finding the best path to the target. Route navigation tasks require the robot to visit multiple locations in sequence according to the specified route sequence; for example, the robot is required to go to booth 3 to confirm whether the exhibits are intact and then go to booth 5 to complete the explanation task.

[0050] Different modalities provide different focuses on environmental information. RGB images can capture rich color and texture features, while LiDAR can provide accurate spatial position and distance information. Figure 2 As shown in the figure, the exhibition hall robot's multimodal scene perception integrates laser point cloud and RGB image to make up for the shortcomings of a single sensor. For example, the performance may be unstable in poor light, while the laser radar can still accurately perceive the distance in this situation; on the contrary, the laser radar cannot distinguish colors, while the visual image can provide more detailed object recognition features.

[0051] The dual-branch heterogeneous multi-source feature fusion module combines the advantages of both in environmental color texture feature extraction and spatial distribution feature extraction to generate fusion features containing rich environmental information. Specifically:

[0052] Using a camera to capture RGB images Collecting laser point clouds using radar N is the number of point clouds;

[0053] S1. Extract image features F I and point cloud features F P ;

[0054] F I =FeatExtra I (I), where I is an RGB image, and H is the height of the image, W is the width of the image; FeatExtra I is an image feature extractor;

[0055] F P =FeatExtra P (P), where P is the laser point cloud, and N is the number of point clouds; FeatExtra P Point cloud feature extractor

[0056] S2, the image feature F I and point cloud features F P Serialization;

[0057] Serialized image features S I :S I = ImagetoSeq(F I ), where ImagetoSeq means converting an image into a sequence;

[0058] Serialized point cloud features S P :S P = ImagetoSeq(F P );

[0059] S3. Obtain the fused sequence feature F seq : F seq =SelfAtten(CrossAtten(S I ,S P ),CrossAtten(S I ,S P ))

[0060] Among them, SelfAtten means performing self-attention operation between sequences, and CrossAtten means performing cross-attention operation between sequences;

[0061] S4, obtain the fusion feature F of RGB image and laser point cloud: F = SeqtoImage(F seq ), where SeqtoImage means converting a sequence into an image.

[0062] After extracting the point cloud features, the point cloud feature extractor converts the point cloud features into a depth feature map, and then fuses them with the features of the RGB image to obtain the fused features. The candidate target objects are obtained by using the fusion features of RGB images and laser point clouds, and the detection object occupancy probability prediction network is adopted. The detection object occupancy probability prediction network takes the fusion feature F as input, estimates the probability that each pixel in the image and its corresponding point cloud are occupied by a specific object, and obtains the correspondence between the semantic identification and spatial distribution of the object in the image, generates candidate target objects, and encodes each candidate target object after identification and assigns a unique ID.

[0063] Exhibition Hall Robot Multimodal Environment Understanding System: It is an important component for the exhibition hall robot to achieve accurate navigation and interaction in a complex exhibition hall environment. The exhibition hall robot receives voice commands and uses a large multimodal model of visual language to understand the target object and user intention, then extracts the high-dimensional semantic features of the target object text from the system, aligns these high-dimensional semantic features with the image features of each candidate target in the current scene perceived by the exhibition hall robot multimodal scene perception system, and finds the target object from the candidate target objects;

[0064] The process of using the visual language multimodal model to understand the target object and intention is as follows:

[0065] First, the voice command is converted into command text. The visual language multimodal large model deeply understands the command text. It first parses the semantic structure of the sentence, identifies the text involving actions and the text of the target object, and then reasoning based on the context semantics to understand the user's intention.

[0066] After the visual language multimodal large model understands the target object, the system extracts high-dimensional semantic features of the target object text, which include concepts, shapes, colors, and usage information related to the target object.

[0067] Among them, the process of finding the target object from the candidate target objects is:

[0068] The target object understood by the visual language multimodal large model calculates the similarity between the high-dimensional semantic features corresponding to the target object text and the image features of the candidate target object in the current scene, and selects the candidate target object with the highest similarity as the target object.

[0069] Using examples to illustrate the understanding process, the exhibition hall robot collects the user's voice commands through the microphone on board, such as "Please move to the sofa position", and converts it into natural language text using speech recognition technology. After the speech-to-text conversion is completed, the robot uses the multimodal visual language large model to deeply understand the command text. First, it parses the semantic structure of the sentence and identifies the action "move" and the target object "sofa" involved. The multimodal large language model can not only understand the command text after speech conversion, but also reason with contextual semantics to ensure that the user's intention can be accurately understood under complex or ambiguous instructions. After identifying the target object "sofa", the system extracts high-dimensional semantic features of the text, which include information related to the concept, shape, color, purpose, etc. of "sofa". These high-dimensional features are then aligned with the image features perceived by the robot's vision system. The image features are extracted from the environment through the multimodal scene perception module, covering the visual and spatial features of each object in the exhibition hall, such as color, shape, texture, and three-dimensional position. In order to find the object that best matches the target object "sofa" in the instruction, the system calculates the similarity between the high-dimensional features of "sofa" and the high-dimensional visual features of candidate objects in the current scene (such as ①, ②, ③, etc.). The similarity calculation can be completed by vector space comparison in the deep learning model, which maps the semantic features and visual features of the object into a shared feature space. In this space, the more similar the semantic and visual features are, the smaller the distance between the vectors. Finally, the robot selects the object with the highest similarity as the target object based on the similarity results. For example, when the high-dimensional features of object ② have the highest similarity with the high-dimensional features of "sofa", the robot recognizes object ② as "sofa" and performs subsequent movement or interaction tasks according to the user's instructions.

[0070] Exhibition hall robot intelligent navigation system based on multi-modal large model:

[0071] The process of dynamically adjusting its own behavior in navigation according to the real-time changing scene provided by the exhibition hall robot's multimodal scene perception system is as follows:

[0072] Initially, the exhibition hall robot's multimodal environment understanding system generates a posture sequence based on the intention understood by the voice command, and the exhibition hall robot starts to move according to the path planning generated by the posture sequence;

[0073] During the movement of the exhibition hall robot, the exhibition hall robot's multimodal scene perception system continuously perceives the spatial distribution in the scene. The spatial distribution includes the positioning of candidate target objects and the spatial relationship with other obstacles. The exhibition hall robot adjusts its own positioning and path planning according to the scene changes.

[0074] In the natural language understanding and decision-making navigation technology, the exhibition hall robot uses the exhibition hall robot multimodal scene perception system and the exhibition hall robot multimodal environment understanding system to accurately extract the target object and intention in the user's command, such as "move" or "navigation". During the execution process, if the recognition confidence of the large language model for the target object is lower than the set threshold, the system will actively initiate multiple rounds of dialogue with the user for interactive confirmation. For example, the robot may ask "Do you mean the display cabinet near the entrance?" to further confirm the target object. Such multiple rounds of dialogue ensure that the robot can collect enough information to avoid misoperation when facing complex or ambiguous instructions. When the robot determines the mobile navigation task, it will use the multimodal scene perception system and the multimodal environment understanding system, combined with visual images and lidar point cloud data, to accurately locate the target object in space. Positioning not only includes identifying the existence of the target object, but also estimating the distance, orientation, angle and other information of the object relative to the robot. At the same time, the robot will continue to perceive the distribution of obstacles in the surrounding environment and update the spatial relationship between the target object and other objects in real time. This enables the robot to cope with the dynamic and complex exhibition hall environment, and perceive scene changes such as pedestrian movement and object position changes in real time, thereby adjusting its positioning and path planning. After completing the recognition and positioning of the target object, the robot enters the motion strategy generation stage. Based on the relative posture information between itself and the target object and the obstacle distribution in the surrounding environment, the system generates a sequence of movement direction and speed. This sequence determines how the robot approaches the target object safely and effectively. During the navigation process, the robot continuously optimizes the motion strategy through a dynamic feedback mechanism. It continuously obtains sensor data, including visual images, lidar point clouds, IMU, etc., and adjusts the movement strategy based on real-time data. For example, the robot will adjust the route or speed according to environmental changes to ensure smooth and safe completion of the navigation task.

[0075] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A large-scale model-based exhibition hall robot visual language navigation method, characterized in that: The method includes a multimodal scene perception system of an exhibition hall robot, a multimodal environment understanding system of an exhibition hall robot and an intelligent navigation system of an exhibition hall robot based on a multimodal large model; Exhibition hall robot multimodal scene perception system: perceive the current scene, where the camera is used to collect RGB images, the radar is used to collect laser point clouds, and then the candidate target objects are obtained by the fusion features of the RGB images and the laser point clouds; at the same time, the image features of all candidate target objects in the scene are extracted, and the image features include visual features and spatial features; Exhibition hall robot multimodal environment understanding system: The exhibition hall robot receives voice commands and uses a large multimodal model of visual language to understand the target object and user intention. It then extracts high-dimensional semantic features of the target object text from the system, aligns these high-dimensional semantic features with the image features of each candidate target in the current scene perceived by the exhibition hall robot multimodal scene perception system, and finds the target object from the candidate target objects. Intelligent navigation system for exhibition hall robots based on multimodal large models: The exhibition hall robot generates a posture sequence based on the intention understood by the exhibition hall robot's multimodal environment understanding system. The exhibition hall robot performs path planning and autonomous movement according to the generated posture sequence, and dynamically adjusts its own behavior according to the real-time changing scene provided by the exhibition hall robot's multimodal scene perception system.

2. According to claim 1, a large model-based exhibition hall robot visual language navigation method is characterized in that: During the navigation process, when the recognition confidence of the target object by the visual language multimodal large model is lower than the set threshold, the system actively initiates multiple rounds of dialogue with the user for interactive confirmation to adjust the posture sequence, thereby adjusting its positioning and path planning.

3. According to the large model-based exhibition hall robot visual language navigation method of claim 1, it is characterized in that: The fusion features of RGB images and laser point clouds are obtained through a multi-branch heterogeneous multi-source feature fusion module, specifically: Using a camera to capture RGB images Collecting laser point clouds using radar N is the number of point clouds; S1. Extract image features F I and point cloud features F P ; F I =FeatExtra I (I), where I is an RGB image, and H is the height of the image, W is the width of the image; FeatExtra I is an image feature extractor; F P =FeatExtra P (P), where P is the laser point cloud, and N is the number of point clouds; FeatExtra P Point cloud feature extractor S2, the image feature F I and point cloud features F P Serialization; Serialized image features S I :S I = ImagetoSeq(F I ), where ImagetoSeq means converting an image into a sequence; Serialized point cloud features S P :S P = ImagetoSeq(F P ); S3. Obtain the fused sequence feature F seq : F seq =SelfAtten(CrossAtten(S I ,S P ),CrossAtten(S I ,S P )) Among them, SelfAtten means performing self-attention operation between sequences, and CrossAtten means performing cross-attention operation between sequences; S4, obtain the fusion feature F of RGB image and laser point cloud: F = SeqtoImage(F seq ), where SeqtoImage means converting a sequence into an image.

4. According to claim 3, a large model-based exhibition hall robot visual language navigation method is characterized in that: The candidate target objects are obtained by using the fusion features of RGB images and laser point clouds, and the detection object occupancy probability prediction network is adopted. The detection object occupancy probability prediction network takes the fusion feature F as input, estimates the probability that each pixel in the image and its corresponding point cloud are occupied by a specific object, and obtains the correspondence between the semantic identification and spatial distribution of the object in the image, generates candidate target objects, and encodes each candidate target object after identification and assigns a unique ID.

5. According to the large model-based exhibition hall robot visual language navigation method of claim 1, it is characterized in that: The process of using the visual language multimodal model to understand the target object and intention is as follows: First, the voice command is converted into command text. The visual language multimodal large model deeply understands the command text. It first parses the semantic structure of the sentence, identifies the text involving actions and the text of the target object, and then reasoning based on the context semantics to understand the user's intention.

6. The large-model-based exhibition hall robot visual language navigation method according to claim 5 is characterized in that: After the visual language multimodal large model understands the target object, the system extracts high-dimensional semantic features of the target object text, which include concepts, shapes, colors, and usage information related to the target object.

7. The large-model-based exhibition hall robot visual language navigation method according to claim 1, characterized in that: The process of finding the target object from the candidate target objects is: The target object understood by the visual language multimodal large model calculates the similarity between the high-dimensional semantic features corresponding to the target object text and the image features of the candidate target object in the current scene, and selects the candidate target object with the highest similarity as the target object.

8. The large-model-based exhibition hall robot visual language navigation method according to claim 1, characterized in that: The process of dynamically adjusting its own behavior in navigation according to the real-time changing scene provided by the exhibition hall robot's multimodal scene perception system is as follows: Initially, the exhibition hall robot's multimodal environment understanding system generates a posture sequence based on the intention understood by the voice command, and the exhibition hall robot starts to move according to the path planning generated by the posture sequence; During the movement of the exhibition hall robot, the exhibition hall robot's multimodal scene perception system continuously perceives the spatial distribution in the scene. The spatial distribution includes the positioning of candidate target objects and the spatial relationship with other obstacles. The exhibition hall robot adjusts its own positioning and path planning according to the scene changes.

Images