Zero sample based self-intelligent navigation robot control method and system, medium and product

Abstract

The application discloses a zero-sample self-intelligent navigation robot control method and system, a medium and a product, relates to the field of robot control, and comprises the following steps: performing semantic segmentation on a to-be-executed instruction to obtain a to-be-executed instruction vocabulary group; inputting a first image and the to-be-executed instruction vocabulary group into a SAM model to generate a semantic map with a multi-dimensional feature channel; acquiring an exploration area based on the semantic map with the multi-dimensional feature channel and taking a robot position as a reference; performing path planning on the exploration area to obtain a planned route; and controlling the robot to move based on the planned route, so that the robot completes the to-be-executed instruction. The application realizes a control system of a zero-sample self-intelligent navigation robot.

Description

Technical Field

[0001] This invention relates to the field of robot control, and in particular to a control method, system, medium, and product for a zero-sample self-intelligent navigation robot. Background Technology

[0002] In recent years, artificial intelligence methods such as deep learning and reinforcement learning have helped people analyze hidden patterns in data, surpassing human capabilities in many specific tasks. In particular, technologies such as computer vision and natural language processing have seen explosive development in tasks such as image recognition and speech processing, and various integrated applications have long been integrated into daily life. However, intelligent agents with physical entities, multimodal real-world interaction capabilities, the ability to perceive and understand the environment like humans, and the ability to perform complex tasks effectively through autonomous learning are still in the early stages of research. Therefore, there is an urgent need to provide new methods for the research of intelligent agents. Summary of the Invention

[0003] The purpose of this invention is to provide a control method, system, medium, and product for a self-guided intelligent navigation robot based on zero-sample training, which can realize the control system of a self-guided intelligent navigation robot with zero-sample training.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A zero-shot self-navigation robot control method includes:

[0006] Semantic segmentation is performed on the instruction to be executed to obtain a vocabulary of the instruction to be executed; the instruction to be executed includes the target object;

[0007] The first image and the vocabulary of instructions to be executed are input into the SAM model to generate a semantic map with multi-dimensional feature channels; the SAM model includes an encoder and a decoder; the first image is acquired during the robot's movement.

[0008] Based on a semantic map with multi-dimensional feature channels, the exploration area is obtained with the robot's position as the reference.

[0009] Path planning is performed on the exploration area to obtain the planned route;

[0010] The robot moves according to the planned route, enabling it to complete the instructions to be executed.

[0011] A computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the zero-sample self-navigation robot control method described above.

[0012] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described zero-sample self-navigation robot control method.

[0013] A computer program product includes a computer program that, when executed by a processor, implements the steps of the zero-sample self-navigation robot control method described above.

[0014] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0015] This invention discloses a zero-shot self-training intelligent navigation robot control method, system, medium, and product. The method includes semantic segmentation of the instruction to be executed to obtain a vocabulary of the instruction to be executed; the instruction to be executed includes a target object; inputting a first image and the vocabulary of the instruction to be executed into a SAM model to generate a semantic map with multi-dimensional feature channels; the SAM model includes an encoder and a decoder; the first image is acquired during the robot's movement; based on the semantic map with multi-dimensional feature channels, an exploration area is obtained with the robot's position as a reference; path planning is performed on the exploration area to obtain a planned route; the robot is controlled to move based on the planned route, enabling the robot to complete the instruction to be executed. This invention realizes a control system for a zero-shot self-training intelligent navigation robot. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the zero-sample self-navigation intelligent robot control method of the present invention;

[0018] Figure 2 This is a logic overview diagram of the zero-sample self-intelligent navigation robot control method of the present invention;

[0019] Figure 3 This is a schematic diagram of the zero-sample self-intelligent navigation robot control method of the present invention;

[0020] Figure 4 This is a logic diagram illustrating the implementation of the decomposition of high-level natural language and open vocabulary function based on a large language model in this invention.

[0021] Figure 5 This is a diagram illustrating the local semantic map construction process of the present invention;

[0022] Figure 6 The flowchart illustrates the algorithm for selecting regions to be explored based on reinforcement learning and a large language model in long-term navigation, as described in this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The purpose of this invention is to provide a control method, system, medium, and product for a self-navigating intelligent robot based on zero-shot training, aiming to realize a control system for a self-navigating intelligent robot by utilizing zero-shot training.

[0025] Visual and semantic information play crucial roles in robot control technology, serving as the cornerstone for effective robot interaction with the real world. First, visual information allows robots to perceive external features of their surroundings, including the shape, size, and color of objects. Through visual sensors and image processing techniques, robots can acquire and process this information, thereby better understanding their environment and responding accordingly.

[0026] Meanwhile, semantic information focuses more on understanding and interpreting perceived information. This information includes the names, functions, and locations of objects, and is essential for robots to make intelligent decisions and plan their actions. Through natural language processing and semantic analysis techniques, robots can extract key semantic content from information collected from the environment and transform it into actionable knowledge.

[0027] Zero-shot training also plays a crucial role in robotics, and is particularly important in certain situations. Zero-shot learning is a machine learning method that emphasizes learning and reasoning without any labeled samples. In robotics, obtaining large amounts of training data can be extremely difficult due to privacy concerns, especially when personal privacy or proprietary information is involved. In such cases, zero-shot learning provides an effective solution, enabling robots to learn and perform tasks from limited data without direct access to large-scale training data. Applications of zero-shot learning can include various robotic tasks, such as personalized service robots, medical assistance robots, and security monitoring robots. In these scenarios, robots need to make personalized decisions and handle different individuals or environments, but cannot directly access large amounts of personal data or sensitive information. Through zero-shot learning, robots can learn enough information from a small number of samples to adapt to different individual needs or environmental changes, thereby providing personalized services or solutions. Therefore, by integrating visual and semantic information through a multimodal large model and utilizing zero-shot training, a control system for an embodied intelligent navigation robot was implemented.

[0028] Inspired by observations of real life, this invention, through a comprehensive analysis of existing markets and research, addresses the pain points of current robots' inability to understand high-level natural language and their poor navigation performance in highly dynamic and unfamiliar environments. This invention addresses these pain points by designing this robot. By comprehensively utilizing advanced models in related fields, a novel navigation algorithm is proposed to increase navigation efficiency and robustness. Numerous results demonstrate the enormous potential of multimodal large language models in robot navigation, and this project opens a new paradigm for the future development of robot navigation.

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Example 1

[0031] like Figure 1 and Figure 2 As shown, a zero-sample self-navigation intelligent robot control method in this embodiment includes:

[0032] Step 101: Perform semantic segmentation on the instruction to be executed to obtain the vocabulary of the instruction to be executed.

[0033] Based on the Large Language Model (LLM), the user's high-level language is decomposed into navigation goals for different stages, and the ambiguity of these goals is determined. Specifically, based on the required level of detail in the complex semantic information within the goals, two different modes—fast mode and fine mode—are selected to set corresponding navigation goals for the robot.

[0034] The LLM model is used to perform semantic segmentation of the instructions to be executed, resulting in a vocabulary of the instructions to be executed.

[0035] The instruction to be executed is input into the LLM model to decompose the instruction and determine the decomposition result; the decomposition result includes the navigation mode and the decomposed instruction; the navigation mode is either the first navigation mode (fine mode) or the second navigation mode (fast mode); the decomposed instruction includes the navigation target and the detection target in the instruction to be executed.

[0036] Determine whether the navigation mode is the first navigation mode to obtain a first determination result.

[0037] If the first judgment result is yes, then the detection target is matched with the robot's built-in fixed vocabulary list to obtain the fixed vocabulary corresponding to the detection target as the target object, and the detection target is replaced with the target object to obtain the vocabulary group of the instruction to be executed.

[0038] If the first judgment result is negative, then both the detection target and the navigation target are matched with the robot's built-in fixed vocabulary to obtain the fixed vocabulary corresponding to the detection target as the fixed vocabulary corresponding to the target object and the navigation target. The detection target is replaced with the target object, and the navigation target is replaced with the fixed vocabulary corresponding to the navigation target to obtain the vocabulary group of the instructions to be executed.

[0039] The navigation target is the target used in the navigation / exploration process described below. It can be an open term or a fixed term. The detection target is used in the detection part of the navigation process described below. It can only be a fixed term and uses a pre-set correspondence table to convert from the "robot vocabulary" to the "YOLO v8 vocabulary".

[0040] Using Large Language Models (LLMs) to understand human semantics for task planning has proven to be a highly effective method. LLM-Robot modifies the LLM architecture to receive high-level language from users, sequentially planning according to a specific template while outputting the target to be navigated to for each decomposed action and its ambiguity resolution.

[0041] like Figure 4 As shown, specifically, the present invention provides an input template, such as... Figure 4As shown, it consists of three parts: the example section, the receiving section, and the instruction section. The example section provides basic requirements and positive and negative examples. The basic requirements are: "You are now a robot and need to execute this instruction. You need to navigate to one or more target points. After reaching the destination, the task will be completed by other modules, so you don't need to worry about it. You only need to output the targets you plan to navigate to in sequence. The targets should faithfully retain the original semantic information in the sentence, and the targets should only contain nouns and adjectives." Correct examples include: "living room (unambiguous), yellow cell phone (ambiguous), plant (unambiguous)." Incorrect examples include: "go to the living room, throw away the trash can, water the plant." In testing, it was found that providing positive and negative examples can significantly improve the understanding ability of LLM and help to obtain better targets.

[0042] The receiving unit receives the current user's instructions, such as "bring the yellow phone from the bedroom." The instruction unit provides fixed instructions: "To execute this instruction, you now need to navigate to one or more target points. Please output the sequence of navigation target objects and ambiguity information:" A Python script then processes the output open vocabulary and ambiguity information, storing it in a dictionary. Each instance's key is the open vocabulary itself, stored as a string. Ambiguity information is processed as boolean values ​​and stored in corresponding values: "(unambiguous)" is saved as False, and "(ambiguous)" is saved as True, for example, {"kitchen": True, "yellow phone": False}. This dictionary will be further processed in the next step.

[0043] Identification and understanding of open vocabulary.

[0044] Supporting open vocabularies is an important issue in the field of natural language processing, with numerous related methods and models emerging. Considering practical effects and current model architectures, this paper implements open vocabulary recognition and understanding using a large language model understanding approach.

[0045] Specifically, the decomposed instructions will simultaneously output the target to be found for each step (hereinafter referred to as the open vocabulary), and the open vocabulary may still have relatively complex semantic information. Therefore, the robot will send the output open vocabulary and the robot's built-in fixed vocabulary into the template: "Vocabulary is: <Fixed vocabulary>, input vocabulary is: <All keys in the dictionary in 1.1>. Now we need to find the semantically closest word for each of the above input words in the vocabulary and output them on separate lines. Only the words need to be output, no other explanations are required, and all other answers will be considered invalid." This is then sent to the large language model to obtain the word that best matches each open vocabulary in the fixed vocabulary, that is, the open vocabulary is converted into the fixed vocabulary (hereinafter referred to as the fixed vocabulary). On this basis, LLM-Robot supports two navigation modes: fast mode and fine mode.

[0046] The fast mode is a semi-open vocabulary mode used when target accuracy requirements are not high, making greater use of previously trained and validated methods. Fast mode uses fixed vocabulary as both navigation and detection targets. The refined mode, on the other hand, is a fully open vocabulary mode, retaining open vocabulary in addition to the fast mode's features. It uses open vocabulary as navigation targets and fixed vocabulary as detection targets. LLM-Robot selects the mode based on the ambiguity of the preceding output; if there is no ambiguity, it selects the fast mode; otherwise, it selects the refined mode. The target information is saved in an array in the format [navigation goal, detection goal], where the former is the navigation goal and the latter is the detection goal. This array is then passed to step 102 for further navigation.

[0047] First, the robot receives the RGB-D image (first image) from its current sensor based on the Segment Anything Model (SAM). Combining information from the depth channels of the preceding RGB-D image with the obtained semantic information, a semantic map with multi-dimensional feature channels is generated through point cloud transformation. Next, the navigation step employs multiple methods to comprehensively determine the exploration area, using the Fast Marching Method (FMM) as the path planning algorithm. Clustering algorithms are used to eliminate abnormal regions, identifying several interconnected regions to be explored. A large language model is used to assign a relevant score to each region, guiding the robot's exploration. Finally, a dual-model detection method is employed, using YOLO-v8 as the basic object detection model, combined with a multimodal large model to ensure effective detection of open vocabulary.

[0048] Step 102: Input the first image and the vocabulary of instructions to be executed into the SAM model to generate a semantic map with multi-dimensional feature channels. The SAM model includes an encoder and a decoder; the first image is acquired during the robot's movement.

[0049] The first image is input into the encoder to obtain the encoding of each pixel in the first image.

[0050] A decoder is used to obtain the position of the vocabulary of instructions to be executed on the first image, and the encoding of each vocabulary in the vocabulary of instructions to be executed is determined based on the encoding of each pixel in the first image to generate a semantic map with multi-dimensional feature channels.

[0051] The Segment Anything Model (SAM) is used to receive RGB-D images from the robot's sensors and generate a semantic map with multi-dimensional feature channels through point cloud transformation. Specifically, the robot is equipped with a camera with a depth sensor, which can send RGB-D images with depth information to the processor in real time. To obtain the semantic information of the visual input at this time, SAM, which performs well on zero-shot tasks, is introduced as a semantic segmentation model.

[0052] The SAM model is a deep neural network trained on a dataset containing 11 million images and 1.1 billion semantic masks. The SAM model first feeds an RGB image, a 3D h×w×3 array `image`, into an encoder to obtain the image's encoding `p`. emb Then, the decoder obtains the positions of different semantic information on the original image. The final output of the decoder is a two-dimensional array `semantic_image` with the same width and height as the original image. `semantic_image[u][v]` stores the IDs of the semantic information corresponding to pixels u and v in the original image, thus completing semantic segmentation.

[0053] The processed semantic_image array containing semantic information will be fed into the robot, which will then initialize an H×W×K map locally and store it in a three-dimensional array local_map. Here, H×W represents the expected map size, and K represents the multi-dimensional semantic channels, which include all objects in the vocabulary and the "explored area" auxiliary channel. local_map[loc_1][loc_2][c] has only two values, 0 and 1, indicating whether there is an object in the c-th channel at coordinates loc_1 and loc_2 on the local map. 0 means that there is no object represented by the current channel at this pixel, and 1 means that there is an object.

[0054] Combining the distance information of each pixel in the depth channel of the preceding RGB-D image from its position in the actual environment to the robot sensor, with the obtained semantic information, a semantic point cloud map of the current moment is generated with the current robot camera as the origin. This point cloud map is then mapped into a two-dimensional top-down view, resulting in a newly added semantic map in the current coordinate system. Specifically, a temporary three-dimensional Cartesian coordinate system (right-handed system) is constructed, with the robot's current position as the origin, the direction of view as the z-axis, and the upward direction as the x-axis. Then, based on the pinhole camera, a spatial point [x, y, z] has the following relationship with its coordinates [u, v, d] in the RGB-D image:

[0055]

[0056]

[0057]

[0058] The above c x ,c y ,f x ,f y 's' are fixed parameters of the pinhole camera. The internal program of the pinhole camera will traverse all pixels in the current image and obtain the coordinate transformation. Each pixel is projected into the current three-dimensional space. Since each pixel also stores the semantic information output by SAM, it is equivalent to constructing a three-dimensional point cloud map with semantic information. Then, looking from above in the negative x direction, the projection of all point clouds onto the yz plane (that is, the plane parallel to the ground) is taken out to obtain a two-dimensional semantic map with the robot as the origin, which constitutes a three-dimensional array temp_local_map. Each element temp_local_map[loc_1][loc_2] stores a list that stores all semantic channel IDs of position (loc_1, loc_2). If there are no semantic channels, the list is filled with blank [].

[0059] Then, through coordinate transformation, the temporary local map with the robot as the origin and the z-axis as the positive direction is translated and rotated to the semantic map with the global origin and the north direction as the positive direction, and then superimposed.

[0060] Through the above process, the semantic map `local_map` with multi-dimensional feature channels stored locally by the robot can be updated, realizing the construction of a semantic map with multi-dimensional feature channels. All points with values ​​are updated in the obstacle map, and obstacles expand outward by one pixel, meaning that the eight pixels surrounding an obstacle are considered part of its range. It is worth noting that although the temporary local map is called the yz plane, the local map still uses the traditional xy-axis system. After the temporary map is rotated, translated, and aligned with the local map, its coordinate system also changes to that of the local map, so it is just a difference in terminology and has no essential impact.

[0061] like Figure 5 As shown, step 103: Based on the semantic map with multi-dimensional feature channels, the exploration area is obtained with the robot's position as the reference.

[0062] Step 104: Perform path planning on the exploration area to obtain the planned route.

[0063] The fast-forward method is used to plan a path in the exploration area to obtain a planned route.

[0064] The exploration area navigation employs a combination of methods to determine the exploration area, using the FastMarching Method (FMM) as the path planning algorithm. The selectable exploration area is derived from the boundaries of already explored areas. The currently known boundaries are expanded to become the exploration boundaries, consisting of connected line segments of varying sizes. A clustering algorithm is used to separate the areas and remove outlier areas (such as those with excessively small connected areas). A circular area with a preset radius r is expanded outward from the center of each cluster, resulting in several long-term exploration areas to be selected. In past map-based navigation tasks, numerous models exist that can be used to determine the exploration value of a region. To ensure sufficient robustness in long-term planning, these models are integrated and utilized for each region f. i Basic scores were assigned

[0065] like Figure 6 As shown, for each exploration area, all semantic information, i.e., object information, within that area can be obtained by traversing the semantic map generated above. These can then be organized into an object chain {o1, o2, ..., o...}. n The locally debugged and refined statement prompt template is embedded and fed into the multimodal large language model to obtain the large language model score for each object. Here, it is used to integrate the exploratory method. The exploratory method is suitable for environments where initial training is possible locally, but where the frequency of change is high and the dynamics are high, such as a home environment. The selection method is as follows:

[0066]

[0067] Here, f represents the selected exploration region. arg max represents the index of the element corresponding to the maximum value, which is a common mathematical notation.

[0068] Through multiple experiments, it was found that the higher the maximum LLM score, the better its judgment matches human common sense; conversely, if the LLM cannot provide a very definite answer, its score has little reference value. Therefore, a confidence score range b1, b2 ∈ [0, 1] was set to classify whether the current LLM answer is trustworthy: when it is above the upper bound of the range, the LLM score is fully trusted; when it is within the score range, the prior model score and the LLM score are considered together; when it is below the lower bound of the range, the LLM score has no reference value, and the pre-trained score is used.

[0069] After obtaining the selected exploration area, the Fast Marching Method (FMM), a well-established method in path planning, is applied to enable the robot to move to the selected area. It utilizes the idea of ​​wavefront expansion, obtaining the shortest path under the current wave function until the target point is reached. FMM has good scalability; by changing the wave function (e.g., reducing the velocity around obstacles), shortest path planning with specific requirements can be achieved.

[0070] Shortest path planning can be viewed as an interface evolution problem, which can be described using the Eikonal equation, i.e.:

[0071]

[0072] Here, F is a vector field, and T represents the shortest time required for the curve to travel from its starting point to every point in the computational domain. Based on solving this equation, all points (x, y) satisfying T(x, y) = t constitute the wavefront curve after time t from the starting point. The FMM provides a method for approximating the Eikonal equation in the case of map discretization.

[0073] FMM is an algorithm based on dynamic programming. First, the map is discretized, with each discrete cell having a side length of h. The starting point is set to t = 0, and all unknown points are set to t = ∞. A wavefront coordinate stack is maintained. Then, all elements in the wavefront coordinate stack are traversed. The neighbor with the smaller value in the x-direction is value T1, and the neighbor with the smaller value in the y-direction is value T2. The value to be solved is t. Thus, an approximate formula is obtained:

[0074]

[0075] Then, the t-value of each point in the discrete map can be calculated, and it can be taken out of the wavefront coordinate stack and put into its neighbor that is not in the stack. The above process is repeated until the target point is calculated.

[0076] After obtaining the shortest time for each discrete cell, based on wave theory, the velocity source direction of each cell is the direction with the most significant difference between its surrounding t-values, i.e., the velocity direction equals the negative gradient direction. Based on this, the gradient direction can be continuously calculated from the endpoint to the starting point to obtain the shortest path from the starting point to the endpoint under the wave field F. The wave field F has a velocity of 0 at obstacles, meaning it cannot propagate, while the same applies to other areas. Subsequently, the robot will proceed along the planned path to the target area, updating the semantic map in real time and performing target detection.

[0077] Step 105: Control the robot to move based on the planned route, so that the robot can complete the instructions to be executed.

[0078] The robot moves according to the planned route, and the robot determines whether the target object exists in its current field of vision to obtain a second judgment result.

[0079] Based on the planned route, the robot moves, and the image of the robot's current field of view is obtained. The image of the robot's current field of view and the target object in the instruction to be executed are input into the YOLO-v8 model to obtain the second judgment result.

[0080] If the second judgment result is yes, then control the robot to directly guide it to the target object, so that the robot can complete the instruction to be executed, such as... Figure 3 As shown, this process represents a short-term exploration of the present invention:

[0081] 1. Generate visual cues.

[0082] 2. Determine the range of motion.

[0083] 3. Receive scores for each hint.

[0084] 4. Calculate the scores comprehensively and defuzzify to obtain the exploration actions.

[0085] Short-term exploration is suitable for small exploration areas where target objects exist. In such cases, the robot will output motion commands to the robotic arm and perform infrared scanning to enable the robot to interact with the environment and complete the commands to be executed.

[0086] If the second judgment result is negative, the robot is controlled to collect the image of the current field of view as the first image, and the process returns to the step of "inputting the first image and the vocabulary of the instruction to be executed into the SAM model to generate a semantic map with multi-dimensional feature channels".

[0087] This invention proposes a novel dual-model detection method, employing YOLO-v8 as the foundational object detection model. Since the object detection problem was first proposed in the field of computer vision, numerous foundational models have emerged, but they generally suffer from high computational requirements and long computation times, hindering their widespread application in real-world scenarios. Since its initial introduction in 2015, YOLO (You Only Look Once) has achieved orders-of-magnitude improvements in computational speed compared to previous models while maintaining detection accuracy. It has been continuously updated and iterated, with the latest version being YOLO-v8 released in January 2023. YOLO's core objective is to improve classification performance while maintaining real-time performance, and this objective has been achieved more thoroughly with model upgrades. YOLO-v8 can be deployed in low-performance environments, reducing robot manufacturing costs; it supports detection in dynamic video, meeting the requirements for detection while the robot is in motion.

[0088] During the detection process, the robot will probe the current image at a frequency of 10 frames per second and feed it into the YOLO-v8 model. This model is a well-trained deep learning model that uses a deep convolutional neural network (CNN). The image is input as an array, and after several layers of neural network transformation, several outputs are finally obtained, namely: the name of the detected object, the midpoint of the object's location, and the length and width information of the bounding box.

[0089] For the above output, this invention only uses the first piece of information. A dictionary corresponding to the robot-supported vocabulary and the detection vocabulary supported by YOLO-v8 is pre-constructed, organized in the format {"yolo detectionword": "ourword list"}. After YOLO-v8 returns all detected object names, a traversal algorithm is used to convert all words back to the robot-supported vocabulary. The detection goal in both modes is a word within the robot's vocabulary. Therefore, the words returned by YOLO-v8 and the detection goal are aligned in the same vocabulary. That is, the target objects detected by the YOLO-v8 model and the target objects obtained in the fine and fast modes are all obtained by matching with the robot's built-in fixed vocabulary, ensuring that the target objects detected by the YOLO-v8 model and the target objects obtained in the fine and fast modes are comparable. String equality is used to check if the object to be searched exists in the current frame. If a match is found, the above FMM navigation ends, and the system directly guides to the target object; if no match is found, the next frame is proceeded until the end.

[0090] After the above steps are completed, a multimodal visual large model is used to ensure the detection effect of open vocabulary. The robot will save the semantic information of the target in this round and send it to the multimodal large model for inspection along with the current image input. If the inspection is successful, it will jump to the next round or end; if the inspection fails, it will force the fine mode to be activated and return to the above-mentioned planned environment to explore again.

[0091] In summary, this invention is a zero-shot learning intelligent navigation robot based on a multimodal large language model. Through a comprehensive analysis of existing markets and research, this robot was designed to address the shortcomings of current intelligent robots in understanding high-level natural language and navigating in dynamic and complex environments. By combining a large language model with advanced navigation algorithms, it achieves the ability to understand high-level natural language and navigate effectively in complex and dynamic environments. This novel approach brings entirely new possibilities to robot navigation and lays a solid foundation for the future development of intelligent robots. Extensive experimental results demonstrate the enormous potential of large language models in the field of robot navigation, and this project has opened up a new paradigm for the future of robot navigation.

[0092] Example 2

[0093] A computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the zero-sample self-navigation robot control method in Embodiment 1.

[0094] Example 3

[0095] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the zero-sample self-navigation robot control method in Embodiment 1.

[0096] Example 4

[0097] A computer program product includes a computer program that, when executed by a processor, implements the steps of the zero-sample self-intelligent navigation robot control method in Embodiment 1.

[0098] Example 5

[0099] A computer device, which may be a database, includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores pending transactions. The I / O interfaces facilitate information exchange between the processor and external devices. The communication interface enables communication with external terminals via a network connection. When executed by the processor, the computer program implements the zero-sample self-navigation robot control method described in Embodiment 1.

[0100] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0101] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided by this invention may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided by this invention may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A control method for a self-navigating intelligent robot based on zero-sample self-navigation, characterized in that, The method includes: Semantic segmentation is performed on the instructions to be executed to obtain a vocabulary of instructions to be executed, specifically including: The LLM model is used to perform semantic segmentation of the instructions to be executed, resulting in a vocabulary of the instructions to be executed. The instruction to be executed is input into the LLM model to decompose the instruction and determine the decomposition result; the decomposition result includes the navigation mode and the decomposed instruction; the navigation mode is a first navigation mode or a second navigation mode; the decomposed instruction includes the navigation target and the detection target in the instruction to be executed. Determine whether the navigation mode is the first navigation mode, and obtain a first determination result; If the first judgment result is yes, then the detection target is matched with the robot's built-in fixed vocabulary list to obtain the fixed vocabulary corresponding to the detection target as the target object, and the detection target is replaced with the target object to obtain the vocabulary group of the instruction to be executed; If the first judgment result is negative, then both the detected target and the navigation target are matched with the robot's built-in fixed vocabulary to obtain the fixed vocabulary corresponding to the detected target as the fixed vocabulary corresponding to the target object and the navigation target. The detected target is then replaced with the target object, and the navigation target is replaced with the fixed vocabulary corresponding to the navigation target to obtain the vocabulary group of the instruction to be executed. The LLM model is used to segment the instruction to be executed based on semantics. The instruction to be executed includes the target object. The first image and the vocabulary of instructions to be executed are input into the SAM model to generate a semantic map with multi-dimensional feature channels, specifically including: The first image is input into the encoder to obtain the encoding of each pixel in the first image; A decoder is used to obtain the position of the instruction vocabulary group to be executed on the first image, and the encoding corresponding to each vocabulary group in the instruction vocabulary group to be executed is determined based on the encoding of each pixel in the first image to generate a semantic map with multi-dimensional feature channels; the SAM model includes an encoder and a decoder; the first image is acquired during the robot's movement; Based on a semantic map with multi-dimensional feature channels, the exploration area is obtained with the robot's position as the reference. Path planning is performed on the exploration area to obtain the planned route; The robot moves according to the planned route, enabling it to complete the instructions to be executed.

2. The zero-sample self-navigation intelligent robot control method according to claim 1, characterized in that, The robot moves based on the planned route, enabling it to complete the instructions to be executed, specifically including: Based on the planned route, the robot moves and the robot's current field of vision is determined to determine whether the target object exists, thus obtaining a second determination result. If the second judgment result is yes, then control the robot to directly guide it to the target object, so that the robot can complete the instruction to be executed; If the second judgment result is negative, the robot is controlled to collect the image of the current field of view as the first image, and the process returns to the step of "inputting the first image and the vocabulary of the instruction to be executed into the SAM model to generate a semantic map with multi-dimensional feature channels".

3. The zero-sample self-navigation intelligent robot control method according to claim 2, characterized in that, Based on the planned route, the robot moves, and a second judgment result is obtained by determining whether the target object exists in the robot's current field of vision. This specifically includes: The robot moves according to the planned route, and the image of the robot's current field of view is obtained. The image of the robot's current field of view and the target object in the instruction to be executed are input into the YOLO-v8 model to obtain a second judgment result. The YOLO-v8 model is used to identify whether there is a target object in the image.

4. The zero-sample self-navigation intelligent robot control method according to claim 1, characterized in that, Path planning is performed on the exploration area to obtain the planned route, specifically including: The fast-forward method is used to plan a path in the exploration area to obtain a planned route.

5. A computer system, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the zero-sample self-navigation robot control method according to any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the zero-sample self-navigation robot control method according to any one of claims 1-4.

7. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the zero-sample self-navigation robot control method according to any one of claims 1-4.

Images