Autonomous navigation methods, systems, devices, and media based on multimodal large models

Abstract

This invention belongs to the field of autonomous navigation technology, specifically relating to an autonomous navigation method, system, device, and medium based on a multimodal large model. The invention plans an initial driving path by importing a scene map and receiving task instructions. During the journey, image data is acquired in real time and identified using an image acquisition device. When a target object is identified, pose conversion is performed, and a path planning algorithm is applied to the converted target point to obtain the navigation path. If an object outside the scene map is detected, or if no target object is identified upon reaching the target point, the robot rotates to acquire image data. The image data is then bound to a direction, and multimodal large model analysis is performed on the bound image data to obtain a scene description for each image. All scene descriptions, task instructions, scene map, and the robot's current location are input into the large language model to infer the target location. Based on the inferred target as the fulcrum, a path planning algorithm is used to obtain the navigation path.

Description

Technical Field

[0001] This invention belongs to the field of autonomous navigation technology, and in particular relates to an autonomous navigation method, system, device and medium based on a multimodal large model. Background Technology

[0002] Autonomous navigation technology is a core support for realizing the intelligence and practicality of robots. Its goal is to enable robots to complete navigation tasks in dynamically disturbed environments through autonomous perception, localization, decision-making, and control. This technology has been widely applied in key areas such as service robots, industrial inspection, emergency rescue, and outdoor exploration.

[0003] Common robot navigation technologies can be broadly categorized into three types: geometric grid map-based navigation, deep reinforcement learning-based navigation, and scene graph-based navigation. Geometric grid map-based navigation searches for the optimal path from start to finish on a grid map constructed using SLAM (Simultaneous Localization and Mapping). During navigation, the grid state is updated in real-time to avoid obstacles. It boasts advantages such as high localization and path accuracy and mature, stable technology. However, it heavily relies on pre-built grid maps, requiring time-consuming map construction in unknown scenarios, exhibiting poor dynamic adaptability, lacking semantic understanding, and struggling to respond to complex commands. Deep reinforcement learning-based navigation, on the other hand, does not rely on predefined maps. Instead, it uses sensor data as state input, extracts features through deep neural networks, adjusts decision-making strategies based on a reward function, and ultimately learns an end-to-end "perception-action" mapping, directly outputting navigation control commands. It possesses advantages such as strong adaptability and fast dynamic response, but suffers from high training costs and poor interpretability. Scene graph-based navigation methods use entities within a scene as nodes and relationships between entities as edges, modeling the environment as a semantically structured model. During navigation, they leverage the semantic reasoning of large language models to plan paths, supporting natural language command interaction, exhibiting strong global cognition, and possessing semantic understanding capabilities. However, scene graph construction is complex, and graph maintenance costs are high when the environment changes. In recent years, the rise of multimodal large models has provided a new paradigm for addressing scene changes in scene graph-based navigation methods. These models, by integrating multi-source information such as vision, language, and geometry, possess powerful cross-modal understanding, contextual reasoning, and common-sense decision-making capabilities, simulating the integrated process of human perception, cognition, and decision-making. Deeply integrating multimodal large models with robot navigation technology is expected to overcome the bottlenecks of traditional methods in perceptual robustness, semantic understanding, and dynamic collaboration, enabling robots to achieve human-like autonomous navigation capabilities in dynamically changing scenes, becoming a core research direction and technological breakthrough in the current robotics field.

[0004] The prior art disclosed in CN119309580A is a visual language navigation method for exhibition hall robots based on a large model. This method includes a multimodal scene perception system for the exhibition hall robot, a multimodal environment understanding system for the exhibition hall robot, and an intelligent navigation system for the exhibition hall robot based on a multimodal large model. The multimodal scene perception system for the exhibition hall robot perceives the current scene by acquiring RGB images using a camera and laser point clouds using radar. Then, it uses the fusion features of the RGB images and laser point clouds to obtain candidate target objects. Simultaneously, it extracts image features of all candidate target objects in the scene, including visual and spatial features. The multimodal environment understanding system for the exhibition hall robot... The robot receives voice commands and uses a visual-language multimodal big data model to understand the target object and the user's intent. Then, it extracts high-dimensional semantic features of the target object's text from the system and aligns these high-dimensional semantic features with the image features of each candidate target in the current scene perceived by the exhibition robot's multimodal scene perception system to identify the target object from the candidate objects. The exhibition robot's intelligent navigation system based on the multimodal big data model generates a pose sequence based on the intent understood by the exhibition robot's multimodal environment understanding system. The exhibition robot performs path planning and autonomous movement according to the generated pose sequence and dynamically adjusts its behavior based on the real-time changing scene provided by the exhibition robot's multimodal scene perception system.

[0005] Therefore, it can be seen that traditional scene map-based navigation technology relies on pre-built static maps, which cannot adapt to objects outside the map or when the object's position is offset. It is necessary to rebuild or update the map, resulting in high maintenance costs and navigation failures. Navigation technologies based on geometric grid maps or deep reinforcement learning are also problematic. The former has poor dynamic adaptability and no semantic understanding, while the latter has high training costs and weak interpretability. Both are difficult to deal with complex dynamic scenes. Even navigation schemes that have applied multimodal large models use image acquisition and radar point cloud matching for feature extraction to select the closest target object. In this method, the range of image acquisition and the radar perception range may not be consistent, which will lead to large errors in data matching. The accuracy of the data obtained by feature analysis on the data with errors is low. Therefore, it has problems with perception robustness and inaccurate target object selection, which may require the robot to frequently replan its route during navigation. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides an autonomous navigation method, system, device, and medium based on a multimodal large model. This invention plans an initial driving path by importing a scene map and receiving task instructions. During the journey, image data is acquired in real-time and identified using an image acquisition device. When a target object is identified, pose conversion is performed. A path planning algorithm is then applied to the converted target point to obtain the navigation path. If an object outside the scene map is detected, or if no target object is identified upon reaching the target point, the robot rotates to acquire image data. The image data is then bound to a direction, and multimodal large model analysis is performed on the bound image data to obtain a scene description for each image. All scene descriptions, task instructions, scene map, and the robot's current location are input into the large language model to infer the target location. Based on the inferred target as the fulcrum, a path planning algorithm is used to obtain the navigation path.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides an autonomous navigation method based on a multimodal large model, comprising the following steps:

[0009] Step S1: The robot generates a planned navigation path S based on the scene map and task instructions using a path planning algorithm, and the robot travels according to the navigation path S.

[0010] In step S2, the robot collects image data in real time through image acquisition devices during its movement, and identifies the target object in the task command based on the collected image data. And update the navigation path S;

[0011] Step S3, the robot identifies the target object. During navigation, if an object or target object that does not exist in the scene map is identified... When not in the designated position, the robot rotates its chassis to collect visual images from different perspectives, and generates scene descriptions of the corresponding images through a multimodal large model;

[0012] Step S4: Generate all scene descriptions Scene Map Robot's current location Task instructions The input is fed into a large language model, and the target object is identified using a preset prompt template. The selection yields the predicted target location point. This point is used as the next navigation target point, and a new navigation path is generated by the path planning algorithm. The robot follows the navigation path Continue moving forward, proceeding to step S2. If the predicted target location is reached... If no target object is detected, proceed to steps S3-S4.

[0013] In a second aspect, the present invention provides an autonomous navigation system based on a multimodal large model, including a configuration module, a path planning algorithm module, an image recognition module, a pose conversion module, a multimodal large model module, and a large language module;

[0014] The configuration module parses externally imported scene graph files, accepts task instructions, and submits the task instructions and scene graphs to the path planning algorithm module.

[0015] The path planning algorithm module plans the path based on the task instructions and the scene graph, or plans the path based on the target points given by the pose conversion module and the large language model combined with the scene graph. The planned navigation path is then submitted to the robot, and the robot moves according to the navigation path.

[0016] The image recognition module identifies the target object in the image data acquired by the image acquisition device. If the target object is not identified... If the target object is detected, continue moving forward. Obtain the depth point cloud of the target object image. The depth point cloud of the target object image The data is submitted to the pose conversion module; if an object not in the scene atlas is detected, or if the target object is not detected after reaching the target point, the data will be processed. Then, the image data is acquired by rotation and submitted to the multimodal large model module for image data processing.

[0017] The pose conversion module first converts the depth point cloud of the target object image. Conversion to pose vector in camera coordinate system C For the converted pose vector Depth thresholding is applied for filtering, and the image point cloud pose vectors are used as the basis for this filtering. Find the first pose vector of the target object's center in the camera coordinate system C. , the first pose vector The transformation yields the second pose vector in robot coordinate system B. , the second pose vector Perform matrix transformations to obtain the third pose vector in the global coordinate system W. Complete the pose calculation of the target object and submit the pose information of the target object as the target point to the path planning algorithm module.

[0018] The multimodal large model module uses preset prompts within the multimodal large model to generate scene descriptions corresponding to each image data point from the image data acquired during robot rotation. The expression is defined as follows:

[0019]

[0020] in, Generate functions for describing multimodal large-scale model scenarios; For the first Visual images from various perspectives; The multimodal large model prompt is set to generate scene descriptions; the generated scene descriptions are then submitted to the large language model module.

[0021] The large language model module will generate all scene descriptions. Scene Map Robot's current location Task instructions Input into a large language model; determine the target object based on task instructions. Using a prompt template pre-set in the large language model, the scene description number that is highly relevant to the target object is selected from numerous scene descriptions. Choose the following expression:

[0022]

[0023] in, Selecting functions for large language model scenario descriptions This is a template for the prompt; based on the selected scene description number. Get the orientation of the image Then, the scene size is determined based on the depth of the image point cloud, and the movement step size is set according to the scene size. Taking the robot's current position as the origin, and traversing the direction... and step length Inferring the target location ; to infer the target location The data is submitted to the path planning algorithm module.

[0024] A third aspect of the present invention provides an electronic device including a memory 102, a processor 101, a display module 103, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps described in any of the preceding autonomous navigation methods based on a multimodal large model.

[0025] A fourth aspect of the present invention provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in any of the preceding autonomous navigation methods based on a multimodal large model.

[0026] The beneficial effects of this invention are: by implementing a closed-loop process of initial scene graph planning, dynamic scene multimodal completion, and LLM-based target point prediction, it solves the problems of object recognition outside the scene graph and target object position offset in traditional navigation. This eliminates the need to rebuild or update the scene graph, reducing maintenance costs. Furthermore, through iterative navigation logic, it improves navigation success rate and execution efficiency in dynamic scenes. It also clarifies the pose calculation and path update mechanism after target object recognition, achieving a comprehensive recognition-calculation-... The seamless integration of navigation systems prevents navigation failures caused by deviations between the real-time and planned positions of the target object, improving navigation and positioning accuracy and ensuring the robot accurately reaches the target object's location. Image data acquired via an RGBD camera achieves consistency between images and depth point clouds, ensuring accurate image data conversion. Depth thresholding eliminates depth measurement error points, guaranteeing the accuracy of target object center position calculations. Simultaneously, precise mapping of position information across different coordinate systems provides reliable global coordinate support for path planning, further enhancing the accuracy and stability of navigation control. The robot acquires multi-view images through chassis rotation and binds orientations, combining this with a multimodal large model to generate structured scene descriptions. This enables comprehensive perception of unknown areas or scene changes, avoiding information omissions caused by single perspectives and providing rich and comprehensive scene data support for subsequent LLM decisions, enhancing the navigation's environmental adaptability. Inputting multi-dimensional information into the LLM, and filtering target-related scene descriptions using customized prompt templates, accurately infers navigation direction and step size, avoiding blind exploration and shortening target search time. Furthermore, the inference-verification-iteration logic ensures continuous approach to the target in complex dynamic scenes, improving navigation efficiency. Attached Figure Description

[0027] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating the method of this invention;

[0029] Figure 2 This is a schematic diagram of the system structure of the present invention;

[0030] Figure 3 This is a schematic diagram of the device structure of the present invention.

[0031] Among them, 101 is the processor, 102 is the memory, and 103 is the display module. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0034] Example 1, such as Figure 1 The autonomous navigation method based on a multimodal large model, as shown, includes the following steps:

[0035] Step S1: The robot generates a planned navigation path S based on the scene map and task instructions using a path planning algorithm, and the robot travels according to the navigation path S.

[0036] In step S2, the robot acquires image data in real time using an RGBD camera during its movement, and identifies the target object in the task command based on the acquired image data. And update the navigation path S; the specific steps for target object recognition navigation are as follows:

[0037] Step S21: If the target recognition algorithm detects that the object in the current field of view is not the target object... If so, continue to proceed and execute step S2;

[0038] Step S22: If the target recognition algorithm detects that the object in the current field of view is the target object... Obtain the depth point cloud of the target object image. Perform robot pose calculation to obtain the pose vector of the target object in the global coordinate system W. Update the navigation target point to The path planning algorithm generates a new navigation path. The robot follows the navigation path Proceed to the target object Complete the navigation task; the specific steps are as follows:

[0039] Step S221: If the target recognition algorithm detects that the object in the current field of view is the target object... Obtain the depth point cloud of the target object image. Based on the depth point cloud of the target object Obtain the pose vector of the depth image points of the target object in the camera coordinate system C. and use depth thresholding for pose vectors Filtering yields new image point cloud pose vectors. The formula is as follows:

[0040]

[0041]

[0042] in, The total number of depth point cloud points of the target object. The depth threshold is determined based on the effective detection range set by the sensor manufacturer. Remove the points. The position on the x-axis. The position on the y-axis. The position on the z-axis;

[0043] Step S222: Based on the filtered image point cloud pose vector Find the first pose vector of the target object's center in the camera coordinate system C. The conversion formula is as follows:

[0044]

[0045] in, This represents the total number of depth point cloud points of the target object after filtering. The average value on the x-axis. This is the average value on the y-axis. This represents the average value along the z-axis.

[0046] Step S223, set the first pose vector After matrix transformation, the second pose vector in the robot chassis coordinate system B is obtained. The transformation formula is as follows:

[0047]

[0048] in, It is an extrinsic parameter matrix. and These are the rotation matrix and translation vector from the camera to the robot chassis, respectively;

[0049] Step S224, the second pose vector After matrix transformation, the third pose vector in the global coordinate system W is obtained. The transformation formula is as follows:

[0050]

[0051] in, Let be the transformation matrix of the robot chassis in the global coordinate system. and These are the rotation matrix and translation vector of the robot chassis in the global coordinate system, respectively.

[0052] Step S225, the third pose vector As the new target object point, a navigation path is generated through a path planning algorithm. The robot follows the navigation path Travel to the target object Complete the navigation task.

[0053] Step S3, the robot identifies the target object. During navigation, if an object or target object that does not exist in the scene map is identified... When not in the designated position, the robot rotates its chassis to acquire visual images from different perspectives, and generates scene descriptions corresponding to these images using a multimodal large model; the specific steps are as follows:

[0054] Step S31, the robot identifies the target object. During navigation, if an object or target object that does not exist in the scene map is identified... When not in the designated position, the robot acquires visual images from different perspectives at the current position by rotating its chassis in place, and then binds the visual images to the orientation in which they were acquired.

[0055] Step S32: After the robot rotates one revolution, the array of visual images and orientations is input into the multimodal large model.

[0056] Step S33: The multimodal large model generates a scene description for each visual image by using a scene description generation function and combining it with a pre-set prompt. The expression is defined as follows:

[0057]

[0058] in, Generate functions for describing multimodal large-scale model scenarios; For the first Visual images from various perspectives; This is a prompt set to generate a multimodal large model for scene description.

[0059] Step S4: Generate all scene descriptions Scene Map Robot's current location Task instructions The input is fed into a large language model, and the target object is identified using a preset prompt template. The selection yields the predicted target location point. This is taken as the next navigation target point. A new navigation path is generated by the path planning algorithm. The navigation path is followed, and step S2 is executed. If the predicted target location is reached... If no target object is detected, proceed to steps S3-S4; the specific steps are as follows:

[0060] Step S41: Generate all scene descriptions Scene Map Robot's current location Task instructions Input into a large language model;

[0061] Step S42: Determine the target object according to the task instructions. Using a prompt template pre-set in the large language model, the scene description number that is highly relevant to the target object is selected from numerous scene descriptions. Choose the following expression:

[0062]

[0063] in, Selecting functions for large language model scenario descriptions A template for the prompt;

[0064] The prompt template is defined as follows: it includes a role, scene graph, localization points, scene description, task instructions, and examples. The role defines the role localization of the large language model, selecting the scene description that best matches the task instructions. The scene graph is JSON-formatted text data describing the current scene information, consisting of objects, rooms, floors, and their corresponding attributes. The localization points are the robot's current positions. The task instructions are the navigation tasks. The scene description is a scene description from different perspectives. The examples list the input and output references for selecting scene descriptions. The specific style is as follows:

[0065] Agent role: Among multiple scene descriptions, the ID of the scenedescription most re lated to thetaskins tucton is output in json forrnat.

[0066] Scene Graph:Text-serialised json description of a 3D scene graph.

[0067] Locate_foor_room:The floor and room where the agent is currentlylocated

[0068] Instruction:Natral language description of the task.

[0069] Descriptions:Scene description.

[0070] Examples :

[0071] {"role": "user", "content": {"instruction": "Find an apple", "SceneGraph": {"nodes": [{"floorB": {"room": [{"objects": [{"?": "bottle"}, {"18": "bowl"}, {"21": "bowl"}, {"$5": "potted plant"}, {"56": "potted plant"}, {"57": "potted plant"}, {"67": "dining table"}, {"68": "dining table"}, {"77": "microwave"}, {"78": "oven"}, {"79": "oven"}, {"80": "oven"}, {"83": "oven"}, {"88": "sink"}, {"93": "sink"}, {"99": "refrigerator"}, {"117": "vas"}]}, "20": "kitchen"}]}}}, "Locate floor room": ["floorB", "kitchen"], "Descriptions": [{"pl": "In the office scene, one person is sitting in front of the workstation, and there are green plants, computer screens and water cups on the table."}, {"pl": "In the kitchen scene, with refrigerator and chopsticks"}, {"p2": "In the bathroom scene, with toilet and sink"}, {"p3": "In the bedroom scene, there are beds, pillows and quilts."}]}, "role": "assistant", "content": {"id": "pl1"}}

[0072] }}

[0073] Step S43, according to the selected scene description number Get the orientation of the image The scene size is determined based on the depth of the image point cloud, and the movement step size is set based on the scene size. Taking the robot's current position as the origin, and traversing the direction... and step length Inferring the target location ;

[0074] Step S44, using the predicted target location point The path planning algorithm generates a navigation path for the next navigation target point. The robot follows the navigation path During the march, step S2 is executed.

[0075] Step S45, if the robot reaches the predicted target location point After that, no target object was detected. Then proceed with steps S3-S4.

[0076] Example 2, as Figure 2 As shown, an autonomous navigation system based on a multimodal large model includes a configuration module, a path planning algorithm module, an image recognition module, a pose conversion module, a multimodal large model module, and a large language module.

[0077] The configuration module parses externally imported scene graph files, accepts task instructions, and submits the task instructions and scene graphs to the path planning algorithm module.

[0078] The path planning algorithm module plans the path based on the task instructions and the scene graph, or plans the path based on the target points given by the pose conversion module and the large language model combined with the scene graph. The planned navigation path is then submitted to the robot, and the robot moves according to the navigation path.

[0079] The image recognition module identifies the target object in the image data acquired by the image acquisition device. If the target object is not identified... Then continue moving forward and identifying the target object. If the target object is identified... Obtain the depth point cloud of the target object image. The depth point cloud of the target object image The data is submitted to the pose conversion module; if an object not in the scene atlas is detected, or if the target object is not detected after reaching the target point, the data will be processed. Then, the image data is acquired by rotation and submitted to the multimodal large model module for image data processing.

[0080] The pose conversion module first converts the depth point cloud of the target object image. Converted to target object pose vector in camera coordinate system C For the converted target object pose vector Perform depth threshold filtering based on the target object pose vector after filtering. Find the first pose vector of the target object's center in the camera coordinate system C. The conversion formula is as follows:

[0081]

[0082] in, This represents the total number of depth point cloud points of the target object after filtering. The average value on the x-axis. This is the average value on the y-axis. The average value on the z-axis; the first pose vector The transformation yields the second pose vector in robot coordinate system B. The transformation formula is as follows:

[0083]

[0084] in, It is an extrinsic parameter matrix. and These are the rotation matrix and translation vector from the camera to the robot chassis, respectively;

[0085] The second pose vector Perform matrix transformations to obtain the third pose vector in the global coordinate system W. The transformation formula is as follows:

[0086]

[0087] in, Let be the transformation matrix of the robot chassis in the global coordinate system. and These are the rotation matrix and translation vector of the robot chassis in the global coordinate system, respectively; the pose calculation of the target object is completed, and the pose information of the target object is submitted as the target point to the path planning algorithm module;

[0088] The multimodal large model module uses preset prompts within the multimodal large model to generate scene descriptions corresponding to each image data point from the image data acquired during robot rotation. The expression is defined as follows:

[0089]

[0090] in, Generate functions for describing multimodal large-scale model scenarios; For the first Visual images from various perspectives; Set a prompt for generating a multimodal large model to describe the scene; submit the generated scene description to the large language model module;

[0091] The large language model module will generate all scene descriptions. Scene Map Robot's current location Task instructions Input into a large language model; determine the target object based on task instructions. Using a prompt template pre-set in the large language model, the scene description number that is highly relevant to the target object is selected from numerous scene descriptions. Choose the following expression:

[0092]

[0093] in, Selecting functions for large language model scenario descriptions This is a template for the prompt; based on the selected scene description number. Get the direction of image binding Then, the scene size is determined based on the depth of the image point cloud, and the movement step size is set according to the scene size. Taking the robot's current position as the origin, and traversing the direction... and step length Inferring the target location ; to infer the target location The data is submitted to the path planning algorithm module.

[0094] Example 3, as Figure 3 As shown, a computer device includes a processor 101, a memory 102, a display module 103, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the autonomous navigation method based on a multimodal large model described in Embodiment 1.

[0095] Example 4: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the autonomous navigation method based on a multimodal large model described in Example 1.

[0096] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0097] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0101] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An autonomous navigation method based on a multimodal large model, characterized in that: Includes the following steps: Step S1: The robot generates a planned navigation path S based on the scene map and task instructions using a path planning algorithm, and the robot travels according to the navigation path S. In step S2, the robot collects image data in real time through image acquisition devices during its movement, and identifies the target object in the task command based on the collected image data. And update the navigation path S; Step S3, the robot identifies the target object. During navigation, if an object or target object that does not exist in the scene map is identified... When not in the designated position, the robot rotates its chassis to collect visual images from different perspectives, and generates scene descriptions of the corresponding images through a multimodal large model; The specific steps for S3 are as follows: Step S31, the robot identifies the target object. During navigation, if an object or target object that does not exist in the scene map is identified... When not in the designated position, the robot acquires visual images from different perspectives at the current position by rotating its chassis in place, and then binds the visual images to the orientation in which they were acquired. Step S32: After the robot rotates one revolution, the array of visual images and orientations is input into the multimodal large model. Step S33: The multimodal large model generates a corresponding scene description based on each visual image and a pre-set prompt using a scene description generation function. The expression is defined as follows: in, Generate functions for describing multimodal large-scale model scenarios; For the first Visual images from various perspectives; The prompt is set to generate a multimodal large model for scene description; Step S4: Generate all scene descriptions Scene Map Robot's current location Task instructions The input is fed into a large language model, and the target object is identified using a preset prompt template. The selection yields the predicted target location point. This point is used as the next navigation target point, and a new navigation path is generated by the path planning algorithm. The robot follows the navigation path Continue moving forward, proceeding to step S2. If the predicted target location is reached... If no target object is detected, proceed to steps S3-S4; The specific steps for S4 are as follows: Step S41: Generate all scene descriptions Scene Map Robot's current location Task instructions Input into a large language model; Step S42: Determine the target object according to the task instructions. Using a prompt template pre-set in the large language model, the scene description number that is highly relevant to the target object is selected from numerous scene descriptions. Choose the following expression: in, Selecting functions for large language model scenario descriptions A template for the prompt; Step S43, based on the selected scene description number Get the direction of image binding The scene size is determined based on the depth of the image point cloud, and the movement step size is set based on the scene size. Taking the robot's current position as the origin, and traversing the direction... and step length Inferring the target location ; Step S44, using the predicted target location point The path planning algorithm generates a navigation path for the next navigation target point. The robot follows the navigation path During the march, step S2 is executed. Step S45, if the robot reaches the predicted target location point After that, no target object was detected. Then specify the steps S3-S4.

2. The autonomous navigation method based on a multimodal large model according to claim 1, characterized in that: The specific steps of S2 target object recognition navigation are as follows: Step S21: If the target recognition algorithm detects that the object in the current field of view is not the target object... If so, continue to proceed and execute step S2; Step S22: If the target recognition algorithm detects that the object in the current field of view is the target object... Obtain the depth point cloud of the target object image. Perform robot pose calculation to obtain the pose vector of the target object in the global coordinate system W. Update the navigation target point to Path planning algorithms generate navigation paths The robot follows the navigation path Proceed to the target object Complete the navigation task.

3. The autonomous navigation method based on a multimodal large model according to claim 2, characterized in that: The specific steps for S22 are as follows: Step S221: If the target recognition algorithm detects that the object in the current field of view is the target object... Obtain the depth point cloud of the target object image. Based on the depth point cloud of the target object Obtain the pose vector of the depth image points of the target object in the camera coordinate system C. and use depth thresholding for pose vectors Filtering yields new image point cloud pose vectors. The formula is as follows: in, The total number of depth point cloud points of the target object. For depth threshold, Remove the points. The position on the x-axis. The position on the y-axis. The position on the z-axis; Step S222: Based on the filtered image point cloud pose vector Find the first pose vector of the target object's center in the camera coordinate system C. The conversion formula is as follows: in, This represents the total number of depth point cloud points of the target object after filtering. The average value on the x-axis. This is the average value on the y-axis. This represents the average value along the z-axis. Step S223, set the first pose vector After matrix transformation, the second pose vector in the robot chassis coordinate system B is obtained. The transformation formula is as follows: in, It is an extrinsic parameter matrix. and These are the rotation matrix and translation vector from the camera to the robot chassis, respectively; Step S224, the second pose vector After matrix transformation, the third pose vector in the global coordinate system W is obtained. The transformation formula is as follows: in, Let be the transformation matrix of the robot chassis in the global coordinate system. and These are the rotation matrix and translation vector of the robot chassis in the global coordinate system, respectively. Step S225, the third pose vector As the new target object point, a navigation path is generated through a path planning algorithm. The robot follows the navigation path Travel to the target object Complete the navigation task.

4. The autonomous navigation method based on a multimodal large model according to claim 1, characterized in that: The prompt template in step S42 is defined as follows: including role, scene graph, localization point, scene description, task instruction, and example. Among them, the role defines the role localization of the large language model and selects the scene description that best matches the task instruction; the scene graph is a JSON-formatted text data describing the current scene information, consisting of objects, rooms, floors, and their corresponding attributes; the localization point is the robot's current position; the task instruction is the task of performing navigation; and the scene description is a scene description from different perspectives.

5. An autonomous navigation system based on a multimodal large model, characterized in that: The system is used to implement the autonomous navigation method based on a multimodal large model as described in any one of claims 1 to 4; the system includes a configuration module, a path planning algorithm module, an image recognition module, a pose conversion module, a multimodal large model module, and a large language model module; The configuration module parses externally imported scene graph files, accepts task instructions, and submits the task instructions and scene graphs to the path planning algorithm module. The path planning algorithm module plans the path based on the task instructions and the scene graph, or plans the path based on the target points given by the pose conversion module and the large language model combined with the scene graph. The planned navigation path is then submitted to the robot, and the robot moves according to the navigation path. The image recognition module identifies the target object in the image data acquired by the image acquisition device. If the target object is not identified... If the target object is detected, continue moving forward. Obtain the depth point cloud of the target object image. The depth point cloud of the target object image The data is submitted to the pose conversion module; If an object not in the scene map is detected, or if the target object is not detected after reaching the target point. Then, the image data is acquired by rotation and submitted to the multimodal large model module for image data processing. The pose conversion module first converts the depth point cloud of the target object image. Conversion to pose vector in camera coordinate system C For the converted pose vector Depth thresholding is applied for filtering, and the image point cloud pose vectors are used as the basis for this filtering. Find the first pose vector of the target object's center in the camera coordinate system C. , the first pose vector The transformation yields the second pose vector in robot coordinate system B. , the second pose vector Perform matrix transformations to obtain the third pose vector in the global coordinate system W. Complete the pose calculation of the target object and submit the pose information of the target object as the target point to the path planning algorithm module. The multimodal large model module uses preset prompts within the multimodal large model to generate scene descriptions corresponding to each image data point from the image data acquired during robot rotation. The expression is defined as follows: in, Generate functions for describing multimodal large-scale model scenarios; For the first Visual images from various perspectives; The multimodal large model prompt is set to generate scene descriptions; the generated scene descriptions are submitted to the large language model module. The large language model module will generate all scene descriptions. Scene Map Robot's current location Task instructions Input into a large language model; determine the target object based on task instructions. Using a prompt template pre-set in the large language model, the scene description number that is highly relevant to the target object is selected from numerous scene descriptions. Choose the following expression: in, Selecting functions for large language model scenario descriptions This is a template for the prompt; based on the selected scene description number. Get the orientation of the image Then, the scene size is determined based on the depth of the image point cloud, and the movement step size is set according to the scene size. Taking the robot's current position as the origin, and traversing the direction... and step length Inferring the target location ; to infer the target location The data is submitted to the path planning algorithm module.

6. The autonomous navigation system based on a multimodal large model according to claim 5, characterized in that: The first pose vector in the pose conversion module The conversion formula is as follows: in, This represents the total number of depth point cloud points of the target object after filtering. The average value on the x-axis. This is the average value on the y-axis. This represents the average value along the z-axis. Second pose vector The transformation formula is as follows: in, It is an extrinsic parameter matrix. and These are the rotation matrix and translation vector from the camera to the robot chassis, respectively; Third pose vector The transformation formula is as follows: in, Let be the transformation matrix of the robot chassis in the global coordinate system. and These are the rotation matrix and translation vector of the robot chassis in the global coordinate system, respectively.

7. An electronic device comprising a memory (102), a processor (101), a display module (103), and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the autonomous navigation method based on a multimodal large model as described in any one of claims 1 to 4.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the autonomous navigation method based on a multimodal large model as described in any one of claims 1 to 4.

Images