A large model-based exhibition hall robot visual language navigation method

Abstract

The application discloses a hall robot vision language navigation method based on a large model and belongs to the field of hall robots. The application is used for solving the problems of the existing hall robots. The method comprises the following steps: a hall robot multi-modal scene perception system is used for perceiving a current scene, candidate target objects are obtained by using the fusion features of an RGB image and a laser point cloud, and the image features of all candidate target objects in the scene are extracted; a hall robot multi-modal environment understanding system is used for accepting a voice instruction, understanding a target object and a user intention by using a vision language multi-modal large model, extracting high-dimensional semantic features of the text of the target object from the system, and finding the target object from the candidate target objects; and a hall robot intelligent navigation system based on the multi-modal large model is used for path planning and autonomous movement of the hall robot according to the pose sequence generated according to the intention, and dynamically adjusting the behavior of the hall robot according to the real-time changing scene provided by the hall robot multi-modal scene perception system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot navigation technology, and belongs to the field of exhibition hall robots. BACKGROUND

[0002] Exhibition hall robots have been widely used in various exhibitions, museums and commercial places, and have become an important tool for improving operational efficiency and reducing labor costs. However, despite the significant progress made in related technologies, most existing robot systems still rely on predefined navigation tasks and path planning, resulting in certain limitations of these systems when facing complex and variable exhibition hall environments. In particular, in terms of human-robot interaction, current systems mostly use coding-based interaction methods, which not only increase the complexity of robot operation, but also require a high level of expertise from the operator, thereby limiting the widespread application of these methods in real-world scenarios. In order to overcome these limitations, in recent years, visual language navigation (VLN) technology has gradually become a promising solution with broad application prospects. VLN allows users to communicate with robots through natural language, and robots execute corresponding navigation tasks through visual understanding of the scene and analysis of language instructions. However, current VLN causes navigation uncertainty due to insufficient observation of the environment and ambiguous understanding of instructions.

[0003] Exhibition hall robots have gradually been applied to various complex and variable exhibition hall environments due to their strong flexibility and reliability. The core task of an exhibition hall robot is to perceive the environment through its own sensors, acquire information about the unknown environment, and model and analyze the environment in real time without human intervention; and then complete tasks such as guiding visitors to tour the exhibition hall, providing route guidance, introducing exhibit information, and answering visitors' questions.

[0004] Current navigation technologies can be mainly divided into two categories: traditional SLAM-based navigation methods and visual language model-based navigation methods. Simultaneous Localization And Mapping (SLAM) is a traditional technical solution for achieving the task of an exhibition hall robot. SLAM technology was first applied in the field of robotics, with the hope that a robot could construct a map of the surrounding environment in real time based on sensor information without any prior knowledge, and at the same time, infer its own position based on this map. According to the type of sensor used, SLAM can be divided into radar-based SLAM and vision-based SLAM. A complete SLAM system mainly consists of a sensor data stream, a front-end tracking module (visual odometry), a back-end optimization module, a loop closure detection module, and a map construction module.

[0005] Current SLAM includes vision-based SLAM and SLAM based on lidar point cloud data. Montemerlo et al. proposed the FastSLAM framework (A factored solution to the simultaneous localization and mapping problem[J]. Aaai / iaai, 2002, 593598:593-598.), which uses Kalman filter to estimate the position of landmarks and particle filter to estimate the pose of the system, so that the robot path and the landmark are relatively independent over time. The posterior is decomposed into the product of conditional landmark distribution and path distribution according to the principle of conditional independence in the optimization of the back end, which simplifies the complexity of the calculation process compared with EKF SLAM. GMapping improves the FastSLAM (Improved techniques for grid mapping with rao-blackwellized particle filters[J]. IEEE transactions on Robotics, 2007, 23(1):34-46.) by reducing the number of particles to solve the memory explosion problem that may exist in FastSLAM; the selective resampling method is used to sort the particles according to the importance weight, and the low-weight examples are resampled. GMapping algorithm has good real-time performance and high accuracy in indoor environment. Zhang et al. proposed the LOAM algorithm using feature point method (Lidar odometry and mapping in real-time[C] / / Robotics: Science and systems. 2014, 2(9):1-9.), which calculates the curvature of the points in the local range, and classifies the feature points into edge feature points and plane feature points according to the curvature size; the algorithm assumes that the lidar moves at a constant speed, and uses linear interpolation method for motion compensation.Shan et al. added loop detection function to Loam (Lightweight and ground-optimized lidar odometry and mapping on variable terrain [C] / / 2018IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018:4758-4765.), introduced the concept of key frames, and used key frames and data frames within their local range to form loop-submap, which can greatly reduce the amount of calculation, filter redundant data, and match the current data frame with the loop-submap to achieve the purpose of loop detection. However, traditional SLAM technology relies on a pre-established scene map and cannot cope with dynamic changes in the environment or natural interaction with humans. In addition, traditional SLAM methods mainly rely on the geometric features of the environment, ignoring the integration of semantic information and natural language, limiting the ability of robots to perform tasks in complex environments.

[0006] In view of the limitations of the above traditional navigation methods, in recent years, navigation methods based on visual language have gradually become a research hotspot. Such methods aim to integrate visual information and natural language instructions to enable robots to understand and execute more complex and humanized navigation tasks. For example, an exhibition hall robot can process image, video, point cloud and other information, combined with natural language instructions, to achieve autonomous navigation and task execution. This method not only adapts to changes in dynamic environments, but also enables more natural and intuitive interaction with humans.

[0007] Visual language navigation method enables mobile robots to navigate autonomously in unknown environments, i.e., without pre-built maps, and make decisions in real-time based on various perception modules. Peter et al. created Room-to-Room (R2R) dataset (Interpreting visually-grounded navigation instructions in real environments[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2018:3674-3683.) to provide rich natural language instructions and corresponding real-world panoramic images for visual language navigation tasks. However, this work mainly uses RGB sensors to perceive and understand the environment and does not include depth information or other advanced visual features that may help with navigation tasks. Wang et al. combined model-free and model-based reinforcement learning methods to solve visual language navigation tasks (Neural machine translation by jointly learning to align and translate[J].arXiv preprint arXiv:1409.0473,2014.). By combining a forward-looking module with a decision-making model, the best action for the next step is predicted, and an environment model is used to predict the next state and reward, and the model is verified on the R2R dataset. Like Peter et al., this study relies mainly on RGB images to perceive and understand the environment, using color information from the scene to guide the navigation task. Although RGB images provide rich visual details, this approach does not integrate depth information or other advanced visual features, which limits the mobile robot's overall understanding of the three-dimensional structure of the environment to some extent. Li et al. used large-scale pre-trained language models to learn text representations (Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation[C] / / Proceedings of the European Conference on Computer Vision (ECCV).2018:37-53.) to better generalize to instructions not learned, and proposed a random sampling scheme to reduce the error between expert actions during training and sampled actions during testing, enabling mobile robots to learn to correct their own mistakes during long sequences of action decoding.Hao et al. proposed a new pre-training method by performing multi-task learning on large-scale and diverse datasets (Towards learning a generic agent for vision-and-language navigation via pre-training[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2020:13137-13146.), which enables the model to better capture the relationship between vision and language, thereby enhancing its generalization ability. This method combines visual representation learning, language understanding, and multi-modal alignment to build an agent with generalization characteristics during the pre-training phase. To build the dataset, the authors used multiple scenarios and path generation methods to simulate the diversity in real-world environments. Meanwhile, the article proposes an improved multi-modal alignment mechanism that enables visual features and language descriptions to be more accurately matched, thereby improving navigation accuracy. Chen et al. proposed a history-aware multimodal Transformer that utilizes the historical visual, language, and action information obtained by the agent during the entire navigation process (Historyaware multimodal transformer for vision-and-language navigation[J]. Advances in neural information processing systems,2021,34:5834-5847.). This information is encoded into a context representation and then modeled using a multi-modal Transformer to capture the dependency between historical information and the current navigation state. Although these methods have made significant progress in handling visual and language navigation tasks, they still have certain shortcomings when it comes to ambiguous instructions. These shortcomings mainly manifest in the model's difficulty in effectively parsing and executing instructions that are not clear or ambiguous, thereby affecting the robustness and practicality in real-world applications. SUMMARY

[0008] In view of the problems existing in the current exhibition hall robot, the application provides a visual language navigation method for an exhibition hall robot based on a large model.

[0009] The visual language navigation method for an exhibition hall robot based on a large model comprises an exhibition hall robot multi-modal scene perception system, an exhibition hall robot multi-modal environment understanding system, and an exhibition hall robot intelligent navigation system based on a multi-modal large model.

[0010] The hall robot multi-modal scene perception system: perceive the current scene, wherein, the RGB image is collected by using the camera, the laser point cloud is collected by using the radar, and then the fusion features of the RGB image and the laser point cloud are used to obtain the candidate target object; meanwhile, the image features of all candidate target objects in the scene are extracted, and the image features include visual features and spatial features;

[0011] The hall robot multi-modal environment understanding system: the hall robot receives the voice instruction, and understands the target object and the user's intention by using the visual language multi-modal large model, and then extracts the high-dimensional semantic features of the text of the target object from the system, aligns the high-dimensional semantic features with the image features of each candidate target in the current scene perceived by the hall robot multi-modal scene perception system, and finds out the target object from the candidate target objects;

[0012] The hall robot intelligent navigation system based on the multi-modal large model: generate the pose sequence according to the intention understood by the hall robot multi-modal environment understanding system, and the hall robot plans the path and moves autonomously according to the generated pose sequence, and adjusts the behavior according to the real-time changing scene provided by the hall robot multi-modal scene perception system.

[0013] Preferably, during the navigation process, when the recognition confidence of the target object by the visual language multi-modal large model is lower than the set threshold, the system initiatively initiates the multi-round dialogue with the user for interactive confirmation to adjust the pose sequence, so as to adjust the positioning and path planning.

[0014] Preferably, the fusion features of the RGB image and the laser point cloud are obtained by a multi-branch heterogeneous multi-source feature fusion module, specifically as follows:

[0015] Collect the RGB image by using the camera Collect the laser point cloud by using the radar N is the number of point clouds;

[0016] S1, extract the image feature F I and the point cloud feature F P ;

[0017] F I = FeatExtra I (I), wherein I is the RGB image, and H is the height of the image, and W is the width of the image; FeatExtra I is an image feature extractor;

[0018] F P = FeatExtra P (P), wherein P is the laser point cloud, and N is the number of point clouds; FeatExtra PFor point cloud feature extractor

[0019] S2, image features F I and point cloud features F P are serialized;

[0020] serialized image features S I : S I = ImagetoSeq(F I ), where ImagetoSeq represents converting an image into a sequence;

[0021] serialized point cloud features S P : S P = ImagetoSeq(F P );

[0022] S3, fused sequence features F seq : F seq = SelfAtten(CrossAtten(S I , S P ), CrossAtten(S I , S P ))

[0023] where SelfAtten represents performing a self-attention operation between sequences, and CrossAtten represents performing a cross-attention operation between sequences;

[0024] S4, fused features F of the RGB image and the laser point cloud: F = SeqtoImage(F seq ), where SeqtoImage represents converting a sequence into an image.

[0025] Preferably, obtaining the candidate target object using the fused features of the RGB image and the laser point cloud is implemented using a detection object occupancy probability prediction network, which takes the fused features F as input, estimates the probability that each pixel in the image and its corresponding point cloud is occupied by a specific object, obtains the correspondence relationship between the semantic label and the spatial distribution of the object in the image, generates the candidate target object, and encodes and assigns a unique ID to each candidate target object after recognition.

[0026] Preferably, the process of understanding the target object and the intent using the visual language multimodal large model is as follows:

[0027] First, the voice instruction is converted into instruction text, and the visual language multimodal large model deeply understands the instruction text. First, the semantic structure of the sentence is analyzed, and the text related to the action and the text related to the target object are identified. Then, reasoning is performed in combination with the context semantics to understand the user's intent.

[0028] Preferably, after the visual language multimodal large model understands the target object, the system extracts high-dimensional semantic features of the target object text, which include concept, shape, color and use information related to the target object.

[0029] Preferably, the process of finding the target object from the candidate target object is:

[0030] The visual language multimodal large model understands the target object, calculates the similarity between the high-dimensional semantic features corresponding to the target object text and the image features of the candidate target object in the current scene, and selects the candidate target object with the highest similarity as the target object.

[0031] Preferably, the process of adjusting the behavior of the navigation according to the real-time changing scene provided by the multi-modal scene perception system of the exhibition hall robot is:

[0032] Initially, the multi-modal environment understanding system of the exhibition hall robot generates a pose sequence according to the intention understood by the voice instruction, and the exhibition hall robot starts to move according to the path planning generated by the pose sequence;

[0033] During the movement of the exhibition hall robot, the multi-modal scene perception system of the exhibition hall robot continuously perceives the spatial distribution in the scene, which includes the positioning of the candidate target object and the spatial relationship with other obstacles, and the exhibition hall robot adjusts its own positioning and path planning according to the changes in the scene.

[0034] The beneficial effects of the present application are: a visual language navigation method for exhibition hall robots based on large language models is proposed. This method makes full use of the powerful ability of current large language models (LLM) in language understanding and reasoning, enabling robots to accurately understand and execute complex instructions based on natural language, providing a more natural and intuitive user interaction experience. This system not only improves the navigation flexibility of the robot, but also greatly reduces the operation threshold of the user, making it easy for non-professionals to control the robot.

[0035] To address the limitations of VLN in environmental observation, first, the system utilizes a high-resolution image segmentation network to perform precise semantic segmentation of the exhibition hall scene, thereby extracting key environmental features. Next, the system fuses these semantic segmentation results with the point cloud data obtained by the laser radar through a feature matching algorithm to construct a detailed three-dimensional semantic map. This fusion process enables the robot to have a stereoscopic perception of the exhibition hall environment, thereby more accurately understanding the environment and effectively navigating. To address the ambiguity of VLN in understanding instructions, with the powerful parsing ability of large language models, a multi-label decision model is constructed to convert complex natural language into target-oriented instructions and path-oriented instructions. By generating specific pose sequences, the robot can move autonomously according to the user's instructions and dynamically adjust its behavior according to the real-time changes in the scene. This navigation method not only enhances the robot's adaptability in dynamic exhibition hall environments, but also significantly improves the robot's execution efficiency in complex tasks.

[0036] The visual language navigation has the following advantages: first, it enables the robot to more naturally understand and execute human instructions. By combining visual information and natural language processing, the robot can directly extract the core information of the navigation task from voice or text instructions, such as the target location or route requirements. This ability not only simplifies the interaction process with the robot, but also enables the robot to better meet the individual needs of visitors without the need for complex pre-programming or specific instruction formats. Second, visual language navigation enhances the robot's environmental adaptability. In a dynamically changing exhibition hall environment, traditional navigation methods often rely on pre-constructed static maps, while visual language navigation allows the robot to perceive and understand changes in the surrounding environment in real time. For example, the robot can recognize newly placed exhibits, temporarily constructed display areas, or even the distribution of people within the exhibition hall. This dynamic environmental perception capability enables the robot to flexibly adjust its navigation path, avoid obstacles, and ensure the safety and accuracy of the navigation process.

[0037] Second, visual language navigation also enhances the robot's interaction capabilities. Through natural language dialogue with humans, the robot can obtain more environmental context information or clarify uncertain instructions, thereby making more intelligent decisions in complex or ambiguous situations. This not only improves the success rate of the robot's task execution, but also provides visitors with a more friendly and interactive experience.

[0038] Finally, visual language navigation helps improve the overall service level of the exhibition hall. The robot can provide precise tour guide services to visitors through visual language navigation technology, answer questions, and recommend the most suitable visiting routes based on the visitors' interest points. This intelligent service can greatly improve the visitors' satisfaction, while also reducing the burden on exhibition hall staff, making exhibition hall management more efficient. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a principle block diagram of a hall robot visual language navigation method based on a large model according to the present application;

[0040] Figure 2 is a point cloud-semantic map feature matching flowchart. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0042] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0043] The present application will be further described below with reference to the drawings and specific embodiments, but not as a limitation of the present application.

[0044] Specific embodiment one: the present embodiment will be described below with reference to the drawings and specific embodiments. Figure 1 and Figure 2 The present embodiment describes a hall robot visual language navigation method based on a large model, which includes a hall robot multi-modal scene perception system, a hall robot multi-modal environment understanding system, and a hall robot intelligent navigation system based on a multi-modal large model.

[0045] The hall robot multi-modal scene perception system perceives the current scene, wherein the RGB image is collected by a camera, the laser point cloud is collected by a radar, and then the candidate target object is obtained by fusing the features of the RGB image and the laser point cloud. At the same time, the image features of all candidate target objects in the scene are extracted, including visual features and spatial features.

[0046] The hall robot multi-modal environment understanding system: the hall robot receives a voice instruction, and understands the target object and user intention by a visual language multi-modal large model, and then extracts the high-dimensional semantic features of the target object text from the system. Align these high-dimensional semantic features with the image features of each candidate target in the current scene perceived by the hall robot multi-modal scene perception system, and find out the target object from the candidate target objects.

[0047] The intelligent navigation system of the exhibition hall robot based on a multi-modal large model generates a pose sequence according to an intention understood by the multi-modal environment understanding system of the exhibition hall robot, the exhibition hall robot plans a path and moves autonomously according to the generated pose sequence, and adjusts its behavior according to a dynamically changing scene provided by the multi-modal scene perception system of the exhibition hall robot.

[0048] During navigation, when the visual language multi-modal large model has a recognition confidence of a target object that is lower than a set threshold, the system initiatively initiates a multi-round dialogue with the user for interactive confirmation to adjust the pose sequence, so as to adjust its positioning and path planning.

[0049] The multi-modal scene perception system of the exhibition hall robot: the exhibition hall robot can perform two types of tasks according to instructions: target type navigation tasks and route type navigation tasks. The target type navigation task requires the robot to locate a target (identify all objects in the exhibition hall, including fixed exhibits, dynamic objects (such as pedestrians), and temporary items (such as equipment boxes, display props, etc.)) in a complex exhibition hall environment using its own perception system (cameras, radars, etc.) using a scene perception module. Each identified object is assigned a unique ID, which not only identifies the object but also plays a key role in subsequent navigation to provide support for finding the best path to the target. The route type navigation task requires the robot to visit multiple locations in sequence according to a specified route sequence; for example, the robot is required to go to the No. 3 exhibition stand to confirm whether the exhibits are intact and then go to the No. 5 exhibition stand to complete the explanation task.

[0050] Different modalities provide different environmental information, RGB images can capture rich color and texture features, while laser radars can provide accurate spatial position and distance information. As shown in Figure 2 , the exhibition hall robot multi-modal scene perception fuses laser point clouds and RGB images to compensate for possible deficiencies of a single sensor. For example, performance may be unstable in poor light, while laser radars can still accurately perceive distances in such cases; on the contrary, laser radars cannot distinguish colors, while visual images can provide more detailed object recognition features.

[0051] By combining the advantages of both in environmental color and texture feature extraction and spatial distribution feature extraction through a double-branch heterogeneous multi-source feature fusion module, a fusion feature containing rich environmental information is generated Specifically:

[0052] RGB images are collected using a camera Laser point clouds are collected using a radar N is the number of point clouds;

[0053] S1, extract image features F I and point cloud features F P ;

[0054] F I =FeatExtra I (I), wherein I is an RGB image, and H is the height of the image, W is the width of the image; FeatExtra I is an image feature extractor;

[0055] F P =FeatExtra P (P), wherein P is a laser point cloud, and N is the number of point clouds; FeatExtra P is a point cloud feature extractor

[0056] S2, serializing the image feature F I and the point cloud feature F P ;

[0057] The serialized image feature S I : S I =ImagetoSeq(F I ), wherein ImagetoSeq represents converting an image into a sequence;

[0058] The serialized point cloud feature S P : S P =ImagetoSeq(F P );

[0059] S3, obtaining a fused sequence feature F seq : F seq =SelfAtten(CrossAtten(S I ,S P ), CrossAtten(S I ,S P ))

[0060] wherein SelfAtten represents performing a self-attention operation between sequences, and CrossAtten represents performing a cross-attention operation between sequences;

[0061] S4, obtaining a fusion feature F of the RGB image and the laser point cloud: F=SeqtoImage(F seq ), wherein SeqtoImage represents converting a sequence into an image.

[0062] After the point cloud feature extractor extracts the point cloud feature, the point cloud feature is converted into a deep feature map, and then fused with the feature of the RGB image to obtain a fusion feature The candidate target object is obtained by using a fusion feature of an RGB image and a laser point cloud, and a detection object occupancy probability prediction network is used to realize the detection object occupancy probability prediction network taking the fusion feature F as input, estimating the probability of each pixel in the image and its corresponding point cloud being occupied by a specific object, obtaining the correspondence relationship between the semantic label and the spatial distribution of the object in the image, generating the candidate target object, and encoding and assigning a unique ID to each candidate target object after identification.

[0063] The exhibition hall robot multi-modal environment understanding system is an important part of the precise navigation and interaction of the exhibition hall robot in a complex exhibition hall environment. The exhibition hall robot receives a voice instruction, and uses a visual language multi-modal large model to understand a target object and a user intention, and then extracts high-dimensional semantic features of the target object text from the system, aligns the high-dimensional semantic features with image features of each candidate target in the current scene perceived by the exhibition hall robot multi-modal scene perception system, and finds the target object from the candidate target objects.

[0064] The process of using the visual language multi-modal large model to understand the target object and the intention is as follows:

[0065] First, the voice instruction is converted into an instruction text, and the visual language multi-modal large model deeply understands the instruction text. First, the semantic structure of the sentence is analyzed, the text related to the action and the text of the target object are identified, and then the context semantics are combined for reasoning to understand the user's intention.

[0066] After the visual language multi-modal large model understands the target object, the system extracts high-dimensional semantic features of the target object text, and the high-dimensional semantic features include concept, shape, color and use information related to the target object.

[0067] The process of finding the target object from the candidate target object is as follows:

[0068] The visual language multi-modal large model understands the target object, calculates the similarity between the high-dimensional semantic features corresponding to the target object text and the image features of the candidate target object in the current scene, and selects the candidate target object with the highest similarity as the target object.

[0069] Using an example to illustrate the understanding process, the exhibition robot collects the user's voice instructions through the microphone it carries, such as "please move to the sofa position", and converts them into natural language text using voice recognition technology. After the voice-to-text conversion is complete, the robot uses a multi-modal visual language large model to deeply understand the instruction text. First, it analyzes the semantic structure of the sentence and identifies the action "move" and the target object "sofa". The multi-modal large language model not only understands the instruction text converted from voice, but also reasons based on the context semantics to ensure that it can accurately understand the user's intention even under complex or ambiguous instructions. After identifying the target object "sofa", the system extracts high-dimensional semantic features of the text, which include concepts, shapes, colors, purposes, and other information related to "sofa". These high-dimensional features are then aligned with the image features perceived by the robot's vision system. Image features are extracted from the environment by a multi-modal scene perception module, covering the visual and spatial features of objects in the exhibition hall, such as color, shape, texture, and three-dimensional position. To find the object that best matches the target object "sofa" in the instruction, the system calculates the similarity between the high-dimensional features of "sofa" and the high-dimensional visual features of candidate objects in the current scene (such as ①, ②, ③, etc.). Similarity calculation can be done through vector space comparison in a deep learning model. The model maps semantic and visual features of objects into a shared feature space. In this space, the more similar the semantic and visual features, the smaller the distance between vectors. Finally, the robot selects the object with the highest similarity as the target object based on the similarity results. For example, when the high-dimensional features of object ② have the highest similarity with the high-dimensional features of "sofa", the robot identifies object ② as "sofa" and performs subsequent movement or interaction tasks according to the user's instructions.

[0070] Exhibition robot intelligent navigation system based on multi-modal large model:

[0071] The process of adjusting its behavior according to the real-time changes in the scene provided by the multi-modal scene perception system of the exhibition robot during navigation is:

[0072] Initially, the multi-modal environment understanding system of the exhibition robot generates a pose sequence based on the intention understood from the voice instruction, and the exhibition robot starts moving according to the path planning generated by the pose sequence;

[0073] During the movement of the exhibition robot, the multi-modal scene perception system of the exhibition robot continuously perceives the spatial distribution in the scene, including the positioning of candidate target objects and the spatial relationship with other obstacles, and the exhibition robot adjusts its positioning and path planning according to the changes in the scene.

[0074] In natural language understanding and decision navigation technology, the exhibition robot accurately extracts the target object and intent such as "move" or "navigate" in the user's instruction through the exhibition robot multi-modal scene perception system and the exhibition robot multi-modal environment understanding system. During execution, if the large language model's recognition confidence of the target object is lower than the set threshold, the system will actively initiate multi-round dialogue with the user for interactive confirmation. For example, the robot may ask "Do you mean the display cabinet near the entrance?" to further confirm the target object. Such multi-round dialogue ensures that the robot can collect enough information when facing complex or ambiguous instructions, avoiding misoperation. After the robot determines the movement navigation task, it will use the multi-modal scene perception system and the multi-modal environment understanding system to combine visual images and laser radar point cloud data to accurately locate the target object. The positioning not only includes identifying the existence of the target object, but also includes estimating the distance, direction, angle, and other information of the object relative to the robot. At the same time, the robot will continuously perceive the obstacle distribution in the surrounding environment, and update the spatial relationship between the target object and other objects in real time. This enables the robot to respond to dynamic and complex exhibition environments, and to perceive changes in scenarios such as pedestrian movement and object position changes in real time, so as to adjust its positioning and path planning. After completing the identification and positioning of the target object, the robot enters the motion strategy generation stage. Based on the relative pose information of itself and the target object and the obstacle distribution of the surrounding environment, the system generates a sequence containing the moving direction and speed. This sequence determines how the robot safely and effectively approaches the target object. During navigation, the robot continuously optimizes the motion strategy through a dynamic feedback mechanism. It continuously acquires sensor data, including visual images, laser radar point clouds, IMU, etc., and adjusts the movement strategy according to the real-time data. For example, the robot will adjust the route or speed according to environmental changes to ensure smooth and safe completion of the navigation task.

[0075] While the application has been described with reference to particular embodiments, it will be understood that the examples are merely illustrative of the principles and applications of the present application. It will be understood that numerous modifications can be made to the illustrative embodiments, and that other arrangements can be devised without departing from the spirit and scope of the present application as defined by the appended claims. It will be understood that the features described in relation to one embodiment can be used in other embodiments.

Claims

1. A large model-based exhibition hall robot visual language navigation method, characterized in that, The method comprises an exhibition hall robot multi-modal scene perception system, an exhibition hall robot multi-modal environment understanding system, and an exhibition hall robot intelligent navigation system based on a multi-modal large model; The exhibition hall robot multi-modal scene perception system perceives a current scene, wherein an RGB image is collected by using a camera, a laser point cloud is collected by using a radar, and then a candidate target object is obtained by using fusion features of the RGB image and the laser point cloud; meanwhile, image features of all candidate target objects in the scene are extracted, and the image features include visual features and spatial features; The exhibition hall robot multi-modal environment understanding system: the exhibition hall robot receives a voice instruction, and understands a target object and a user's intention by using a visual language multi-modal large model, and then extracts high-dimensional semantic features of a text of the target object from the system, aligns the high-dimensional semantic features with image features of each candidate target in the current scene perceived by the exhibition hall robot multi-modal scene perception system, and finds the target object from the candidate target objects; The exhibition hall robot intelligent navigation system based on the multi-modal large model: a pose sequence is generated according to the intention understood by the exhibition hall robot multi-modal environment understanding system, the exhibition hall robot performs path planning and autonomous movement according to the generated pose sequence, and dynamically adjusts its behavior according to a real-time changing scene provided by the exhibition hall robot multi-modal scene perception system; The fusion features of the RGB image and the laser point cloud are obtained by using a multi-branch heterogeneous multi-source feature fusion module, and specifically are as follows: Capturing RGB images with a camera Capturing laser point clouds with a radar N is the number of point clouds; S1, extract image features F I and point cloud features F P ; F I =FeatExtra I (I), where I is an RGB image, and H is the height of the image, W is the width of the image; FeatExtra I is an image feature extractor; F P = FeatExtra P (P), where P is a laser point cloud, and N is the number of point clouds; FeatExtra P is a point cloud feature extractor S2, serialize the image features F I and the point cloud features F P serialization; serialized image feature S I : S I = ImagetoSeq(F I ), where ImagetoSeq denotes converting an image to a sequence; serialized point cloud feature S P : S P = ImagetoSeq(F P ); S3, obtain the fused sequence feature F seq : F seq = SelfAtten(CrossAtten(S I , S P ), CrossAtten(S I , S P )) Wherein, SelfAtten represents a self-attention operation between sequences, and CrossAtten represents a cross-attention operation between sequences; S4, obtaining a fusion feature F of the RGB image and the laser point cloud: F = SeqtoImage(F seq ), wherein SeqtoImage represents converting a sequence into an image; The candidate target object is obtained by using the fusion features of the RGB image and the laser point cloud, and is achieved by using a detection object occupancy probability prediction network, the detection object occupancy probability prediction network takes the fusion features F as input, estimates a probability that each pixel in the image and its corresponding point cloud is occupied by a specific object, obtains a correspondence relationship between semantic labels and spatial distributions of objects in the image, generates a candidate target object, and encodes and assigns a unique ID to each candidate target object after identification.

2. The method of claim 1, wherein the method is based on a large model. During navigation, when the visual language multi-modal large model has a recognition confidence of the target object lower than a set threshold, the system initiates a multi-round dialogue with the user for interactive confirmation to adjust the pose sequence, so as to adjust its positioning and path planning.

3. The method of claim 1, wherein the method is based on a large model. The process of understanding the target object and the intention by using the visual language multi-modal large model is as follows: First, the voice instruction is converted into an instruction text, and the visual language multi-modal large model deeply understands the instruction text, first analyzes the semantic structure of the sentence, identifies the text related to the action and the text of the target object, and then combines the context semantics to reason to understand the user's intention.

4. The method of claim 3, wherein the method further comprises: After the visual language multi-modal large model understands the target object, the system extracts high-dimensional semantic features of the target object text, and the high-dimensional semantic features include concept, shape, color and purpose information related to the target object.

5. The method of claim 1, wherein, The process of finding the target object from the candidate target objects is as follows: The target object understood by the visual language multi-modal large model is calculated for the similarity between the high-dimensional semantic features corresponding to the text of the target object and the image features of the candidate target objects in the current scene, and the candidate target object with the highest similarity is selected as the target object.

6. The method of claim 1, wherein the method is based on a large model. The process of dynamically adjusting the behavior of the navigation robot according to the real-time changing scene provided by the multi-modal scene perception system of the exhibition hall robot is as follows: Initially, the multi-modal environment understanding system of the exhibition hall robot generates a pose sequence according to the intention understood from the voice instruction, and the exhibition hall robot starts moving according to the path planning generated by the pose sequence; During the movement of the exhibition hall robot, the multi-modal scene perception system of the exhibition hall robot continuously perceives the spatial distribution in the scene, including the positioning of the candidate target objects and the spatial relationship with other obstacles, and the exhibition hall robot adjusts its own positioning and path planning according to the scene changes.

Images