Navigation path planning method, device, equipment, robot, medium and product

Abstract

The present application provides a navigation path planning method, device, equipment, robot, medium and product, belonging to the field of robot technology, including: obtaining the robot's environmental perception information at a first time step; combining the environmental perception information and navigation target planning to obtain a step-by-step navigation instruction, the navigation target is determined based on the user instruction; controlling the robot to perform an operation task according to the navigation path planned by the step-by-step navigation instruction; if the robot's environmental perception information at a second time step changes compared to the environmental perception information at the first time step, then combining the changed environmental perception information and navigation target planning to obtain an updated step-by-step navigation instruction, and controlling the robot to perform an operation task according to the navigation path planned by the updated step-by-step navigation instruction, the second time step being the next time step of the first time step. The present application can improve the efficiency of the embodied intelligent system in responding to high-level short instructions for navigation path planning in complex scenarios.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a navigation path planning method, device, equipment, robot, medium and product. Background Art

[0002] With the continuous development and progress of artificial intelligence and robotics, the application prospects of artificial intelligence systems with physical bodies (referred to as "embodied intelligent systems") in daily life, industrial automation, medical assistance and other fields are becoming increasingly broad. In related technologies, embodied intelligent systems mainly rely on detailed paths and steps pre-set by humans to plan navigation paths and perform corresponding operations. However, understanding short and high-level behavioral instructions (for example, the behavioral instructions of "go to the kitchen to get milk" used by people in daily life) to plan navigation paths in complex scenarios will obviously lack flexibility and accuracy, resulting in the embodied intelligent system's efficiency in responding to high-level short instructions for navigation path planning in complex scenarios is very low. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a navigation path planning method, device, equipment, robot, medium and product, aiming to improve the efficiency of embodied intelligent systems in responding to high-level short instructions for navigation path planning in complex scenarios.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes a navigation path planning method, the method comprising:

[0005] Obtain the robot's environmental perception information at the first time step;

[0006] Combining the environmental perception information and the navigation target to perform task planning to obtain step-by-step navigation instructions for the robot, wherein the navigation target is determined based on the user instruction;

[0007] Controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the step-by-step navigation instruction;

[0008] If the environmental perception information of the robot at the second time step changes compared to the environmental perception information of the robot at the first time step, task planning is performed in combination with the changed environmental perception information and the navigation target to obtain updated step-by-step navigation instructions, and the steps of controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the updated step-by-step navigation instructions are re-executed, wherein the second time step is the next time step of the first time step.

[0009] In some embodiments, the method further comprises:

[0010] Selecting a mission planning demonstration example based on the user instruction;

[0011] The step of combining the environmental perception information and the navigation target to perform task planning to obtain step-by-step navigation instructions for the robot includes:

[0012] In combination with the environmental perception information and the navigation target, task planning is performed based on the prompt information provided by the task planning demonstration example to obtain step-by-step navigation instructions for the robot.

[0013] In some embodiments, selecting a task planning demonstration example based on the user instruction includes:

[0014] Performing similarity matching between the user instruction and a plurality of basic instructions in a preset example library, respectively, to obtain a target basic instruction having the highest similarity with the user instruction among the plurality of basic instructions;

[0015] Select a task planning demonstration example in the preset example library whose task decomposition target is the target basic instruction.

[0016] In some embodiments, obtaining the robot's environmental perception information at the first time step includes:

[0017] Identify a panoramic environment image of the environment in which the robot is located at a first time step, and obtain spatial type information and visible object information of the environment in which the robot is located;

[0018] Zero-shot classification is performed on the space type information and the visible object information to generate environment perception information of the robot at a first time step.

[0019] In some embodiments, obtaining the robot's environmental perception information at the first time step includes:

[0020] Extracting features of a panoramic environment image of the environment in which the robot is located at a first time step by using a preset multimodal large model, and obtaining environmental visual features corresponding to the panoramic environment image; and extracting features of environmental voice information of the environment in which the robot is located at the first time step by using the multimodal large model, and obtaining environmental language features corresponding to the environmental voice information;

[0021] The environmental visual features and the environmental language features are fused through the multimodal large model to generate environmental perception information of the robot at a first time step.

[0022] In some embodiments, the method further comprises:

[0023] Identifying a target object in an operation task corresponding to a user instruction through a preset large-scale language model, and determining a location of the target object through the large-scale language model using a preset knowledge base;

[0024] The position of the target object determined and output by the large-scale language model is determined as the navigation target of the robot.

[0025] In some embodiments, controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the step-by-step navigation instruction includes:

[0026] Matching the step-by-step navigation instruction with the atomic instruction library of the robot, and performing task logic verification on the step-by-step navigation instruction to obtain a processing result;

[0027] When the processing result indicates that the step-by-step navigation instruction belongs to the atomic instruction library and the step-by-step navigation instruction passes the task logic verification, a navigation path is obtained according to the step-by-step navigation instruction planning;

[0028] The robot is controlled to move along the navigation path to perform the operation task corresponding to the user instruction.

[0029] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a navigation path planning device, the device comprising:

[0030] An acquisition module is used to obtain the robot's environmental perception information at the first time step;

[0031] A dynamic planning module, used to perform task planning in combination with the environmental perception information and the navigation target to obtain step-by-step navigation instructions for the robot, wherein the navigation target is determined based on the user instructions;

[0032] A task execution module, used to control the robot to execute the operation task corresponding to the user instruction according to the navigation path planned by the step-by-step navigation instruction;

[0033] The dynamic planning module is further configured to perform task planning in combination with the changed environmental perception information and the navigation target to obtain updated step-by-step navigation instructions if the environmental perception information of the robot at the second time step changes compared to the environmental perception information of the robot at the first time step; wherein the second time step is the next time step of the first time step;

[0034] The task execution module is also used to re-execute the steps of controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the updated step-by-step navigation instruction.

[0035] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0036] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a robot, which is equipped with an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0037] To achieve the above objectives, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0038] To achieve the above-mentioned purpose, the sixth aspect of the embodiments of the present application proposes a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method provided in the first aspect above is implemented.

[0039] The navigation path planning method, device, electronic device, robot, computer-readable storage medium and computer program product proposed in the present application are as follows: obtaining environmental perception information of the robot at a first time step; performing task planning in combination with the environmental perception information and the navigation target to obtain step-by-step navigation instructions of the robot, wherein the navigation target is determined based on user instructions; controlling the robot to execute the operation task corresponding to the user instruction according to the navigation path planned by the step-by-step navigation instructions; if the environmental perception information of the robot at a second time step changes compared to the environmental perception information of the robot at the first time step, performing task planning in combination with the changed environmental perception information and the navigation target to obtain updated step-by-step navigation instructions, and re-executing the steps of controlling the robot to execute the operation task corresponding to the user instruction according to the navigation path planned by the updated step-by-step navigation instructions, wherein the second time step is the next time step of the first time step.

[0040] The embodiment of the present application determines the navigation target of the robot based on the user instructions, and obtains the robot's environmental perception information at the current first time step, and then combines the environmental perception information and the navigation target to perform task planning to obtain the robot's step-by-step navigation instructions. In addition, the robot is controlled to perform the operation task instructed by the user instructions according to the navigation path planned by the step-by-step navigation instructions at the current first time step. Afterwards, when the robot's environmental perception information at the second time step after the first time step changes compared to the robot's environmental perception information at the first time step, the task planning is performed again in combination with the changed environmental perception information and the navigation target to obtain an updated step-by-step navigation instruction, and the robot is controlled to continue to perform the operation task instructed by the user instructions according to the navigation path planned by the updated step-by-step navigation instructions at the second time step.

[0041] Thus, the embodied intelligent system using the embodiments of the present application can determine the navigation target based on high-level and brief user instructions, and then combine the robot's dynamically changing environmental perception information at different time steps and the navigation target to perform task planning to generate relatively fine-grained step-by-step navigation instructions, and plan the navigation path according to the step-by-step navigation instructions to control the robot to perform actions, so that the robot performs the operation tasks indicated by the user instructions. In this way, the embodiments of the present application can enable the embodied intelligent system to dynamically generate a reasonable navigation path according to high-level and brief instructions in a complex and changeable environment, thereby improving the adaptability and efficiency of the embodied intelligent system in responding to high-level and brief instructions for navigation path planning in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the steps of the navigation path planning method provided in some embodiments of the present application;

[0043] Figure 2 A schematic flow chart of the steps of static planning preliminary instructions involved in some embodiments of the navigation path planning method provided in the embodiments of the present application;

[0044] Figure 3 for Figure 1 A detailed step flow diagram of step S101;

[0045] Figure 4 for Figure 1 Another detailed flowchart of step S101;

[0046] Figure 5 A schematic flow chart of steps of a selected task planning demonstration example involved in some embodiments of the navigation path planning method provided in the embodiments of the present application;

[0047] Figure 6A schematic diagram of a linkage process for generating step-by-step navigation instructions for a retrieval task planning demonstration example involved in some embodiments of the navigation path planning method provided in the embodiments of the present application;

[0048] Figure 7 for Figure 1 Schematic diagram of detailed process flow of step S103;

[0049] Figure 8 A schematic diagram of the overall process involved in some embodiments of the navigation path planning method provided in the embodiments of the present application;

[0050] Fig. 9 A schematic diagram of the application flow of the navigation path planning method provided in the embodiments of the present application in some embodiments;

[0051] Fig.10 A schematic diagram of the structure of a navigation path planning device provided in an embodiment of the present application;

[0052] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] First, the professional and technical terms involved in the embodiments of the present application are explained.

[0057] Embodied intelligent systems.

[0058] Embodied intelligence systems refer to artificial intelligence systems with physical bodies. Embodied intelligence systems are able to perceive, act and interact in real physical environments. Embodied intelligence systems combine technologies from multiple fields such as computer vision, natural language processing, robotics and artificial intelligence, enabling robots and other intelligent agents to understand and adapt to their physical environment and perform complex tasks. The characteristics of embodied intelligence systems include: embodied intelligence systems can perceive information about the surrounding environment through sensors (such as cameras, microphones, etc.), embodied intelligence systems can move and manipulate objects in the environment through physical components (such as robotic arms, wheels, etc.) to perform physical actions, and embodied intelligence systems can also understand task instructions and perform corresponding tasks in the physical environment.

[0059] High-level, short instructions and step-by-step navigation instructions.

[0060] High-level short instructions usually refer to those task instructions with high concepts and concise descriptions, which usually contain fewer specific steps and details. High-level short instructions are closer to the language expression in daily human communication, and require the embodied intelligent system to have higher understanding and reasoning capabilities. For example, "Go to the kitchen to get water", "Empty the washing machine on the first floor", etc. are all high-level short instructions. Step-by-step navigation instructions can also be called "step-by-step instructions". Compared with high-level short instructions, step-by-step navigation instructions have detailed steps and details. For example, "Walk to the door, turn left, enter the kitchen, and get water" is a step-by-step navigation instruction.

[0061] Both high-level brief instructions and step-by-step navigation instructions can be task instructions issued by the user to the robot. The user instructions involved in the embodiments of the present application specifically refer to the above-mentioned high-level brief instructions.

[0062] Large Language Model (LLM)

[0063] Large-scale language model is referred to as LLM model, which generally refers to a language model based on the Transformer encoder, decoder or encoder-decoder structure that takes text sequences as input and output. Generally, the number of parameters of LLM model is more than one billion, so that LLM model can generate certain emergent capabilities under large-scale data and computing power training, and thus can produce better results in a variety of natural language processing tasks (referred to as "NLP tasks") and applications.

[0064] Atomic instruction library.

[0065] The atomic instruction library refers to the basic instruction set that the robot can execute. The basic instructions in the atomic instruction library are the smallest unit of robot operation. The basic instructions in the atomic instruction library can be such as "move to a certain position", "grab an object", "turn on a switch", etc. The atomic instruction library can be set up in the form of "predefined instruction set", that is, a set of basic operation instructions are predefined according to the function of the robot to construct the atomic instruction library of the robot.

[0066] Then, the overall concept of the navigation path planning method provided in the embodiment of the present application is explained.

[0067] The application prospects of embodied intelligent systems in daily life, industrial automation, medical assistance and other fields are becoming increasingly broad. In related technologies, embodied intelligent systems mainly rely on detailed paths and steps pre-set by humans to plan navigation paths and perform corresponding operations. However, understanding short and high-level behavioral instructions (for example, the high-level and short instructions "go to the kitchen to get milk" used by people in daily life) to plan navigation paths in complex scenarios will obviously lack flexibility and accuracy, resulting in the embodied intelligent system's efficiency in responding to high-level and short instructions for navigation path planning in complex scenarios being very low.

[0068] Based on this, the embodiments of the present application provide a navigation path planning method, device, equipment, robot, medium and product, which aim to improve the efficiency of embodied intelligent systems in responding to high-level short instructions for navigation path planning in complex scenarios.

[0069] The embodiment of the present application determines the navigation target of the robot based on the user instructions, obtains the environment perception information of the robot at the current first time step, and then performs task planning in combination with the environment perception information and the navigation target to obtain the step-by-step navigation instructions of the robot. In addition, the robot is controlled to perform the operation task instructed by the user instructions according to the navigation path planned by the step-by-step navigation instructions at the current first time step. Afterwards, when the environment perception information of the robot at the second time step after the first time step changes compared to the environment perception information of the robot at the first time step, the task planning is performed again in combination with the changed environment perception information and the navigation target to obtain the updated step-by-step navigation instructions, and the robot is controlled to continue to perform the operation task instructed by the user instructions according to the navigation path planned by the updated step-by-step navigation instructions at the second time step.

[0070] Thus, the embodied intelligent system using the embodiments of the present application can determine the navigation target based on high-level and brief user instructions, and then combine the robot's dynamically changing environmental perception information at different time steps and the navigation target to perform task planning to generate relatively fine-grained step-by-step navigation instructions, and plan the navigation path according to the step-by-step navigation instructions to control the robot to perform actions, so that the robot performs the operation tasks indicated by the user instructions. In this way, the embodiments of the present application can enable the embodied intelligent system to dynamically generate a reasonable navigation path according to high-level and brief instructions in a complex and changeable environment, thereby improving the adaptability and efficiency of the embodied intelligent system in responding to high-level and brief instructions for navigation path planning in complex scenarios.

[0071] Next, the navigation path planning method, device, equipment, robot, medium and product provided in the embodiments of the present application are specifically described through the following embodiments, and the navigation path planning method provided in the embodiments of the present application is first described in detail.

[0072] It should be noted that the navigation path planning method provided in the embodiment of the present application relates to the field of robot technology. The navigation path planning method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a terminal configured on the robot, or it can be an electronic device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. associated with the robot. The terminal and the robot are associated with each other, which means that the terminal can communicate and interact with the robot based on the network. The server side can be a background server terminal device of the robot, which can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN) and cloud servers that provide basic cloud computing services such as big data and artificial intelligence platforms. The software can be an application, a computer program, and a storage medium that carries the computer program, etc., that implements the navigation path planning method. It should be understood that based on different design requirements of actual applications, in different feasible embodiments, the terminal, server, and software that apply the navigation path planning method provided in the embodiments of the present application may of course also be in other forms not listed here, and the navigation path planning method provided in the embodiments of the present application does not specifically limit this.

[0073] In addition, the present application can also be used in many general or special computer system environments or configurations. For example: embodied intelligent systems, robots, personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, personal computers (PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] For ease of understanding and explanation, the following text takes the embodied intelligent system application of the navigation path planning method provided by the embodiment of the present application as an example to describe in detail each specific embodiment of the present application. The implementation of the navigation path planning method provided by the embodiment of the present application in any of the above-mentioned forms of the subject application can refer to the process of the embodied intelligent system application navigation path planning method described later.

[0075] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0076] Please refer to Figure 1 , Figure 1 The following is a flowchart of the steps of the navigation path planning method provided in some embodiments of the present application. Figure 1 The subsequent flowcharts of other steps show the execution order of some method steps, but based on different design requirements of actual applications, the navigation path planning method provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. That is, Figure 1The order of the steps of the method shown does not constitute a limitation on the execution logic order of the navigation path planning method provided in the embodiment of the present application. Figure 1 Reasonable changes in the order of the method steps shown should be included in the protection scope of the navigation path planning method provided in the embodiments of the present application.

[0077] like Figure 1 As shown, in some embodiments, the embodied intelligent system applies the navigation path planning method provided by the embodiments of the present application, which may include steps S101 to S104.

[0078] Step S101: Obtain the robot's environmental perception information at the first time step.

[0079] At the current first time step, the embodied intelligent system obtains the environmental perception information generated by the robot at the first time step by collecting information about the environment based on the perception module.

[0080] It should be noted that the embodied intelligent system generates the robot's step-by-step navigation instructions by performing scene-oriented dynamic programming based on the robot's environmental perception information at each time step, and controls the robot to select execution actions based on the generated step-by-step navigation instructions at each time step, until the user instruction indicates that the robot's operation task is completed or the maximum number of steps is reached. Based on this, the embodied intelligent system needs to obtain the robot's environmental perception information at each time step. In the embodiment of the present application, in addition to referring to the difference between two previous and subsequent time points, the time step can also refer to the time difference between two task nodes in the operation task performed by the robot. In addition, in the embodiment of the present application, in order to facilitate the distinction between two adjacent time steps, the current time step is defined as the first time step, and the time step after the first time step is defined as the second time step.

[0081] In some embodiments, the embodied intelligent system can control the robot to use its own configured room and object perception module to perceive information about the surrounding environment (images and / or sounds, etc.) through sensors (such as cameras, microphones, etc.), and then perform necessary processing (such as denoising, etc.) on the perceived information to generate environmental perception information.

[0082] Step S102: combining the environmental perception information and the navigation target to perform task planning to obtain step-by-step navigation instructions for the robot, wherein the navigation target is determined based on user instructions.

[0083] After the embodied intelligent system obtains the robot's environmental perception information at the current first time step, it combines the environmental perception information with the navigation target determined by the user to perform task planning, thereby obtaining step-by-step navigation instructions for the robot.

[0084] In some embodiments, the embodied intelligent system can parse and identify user instructions to obtain the target object that the user instructions instruct the robot to operate to perform the action, and infer the possible location of the target object to determine the location as the robot's navigation target.

[0085] Step S103: Control the robot to execute the operation task corresponding to the user instruction according to the navigation path planned by the step-by-step navigation instruction.

[0086] After the embodied intelligent system plans and generates the robot's step-by-step navigation instructions at the current first time step, it plans the navigation path according to the step-by-step navigation instructions, generates a reasonable navigation path for the robot in the current environment, and controls the robot to perform actions according to the navigation path at the first time step, thereby executing the operation task indicated by the user's instructions.

[0087] It should be noted that, when there are multiple step-by-step navigation instructions, the embodied intelligent system can plan a navigation path for the robot in the environment for each step-by-step navigation instruction one by one until all the step-by-step navigation instructions have planned corresponding navigation paths.

[0088] Step S104: If the environmental perception information of the robot at the second time step changes compared to the environmental perception information of the robot at the first time step, task planning is performed in combination with the changed environmental perception information and the navigation target to obtain updated step-by-step navigation instructions, and the steps of controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the updated step-by-step navigation instructions are re-executed, wherein the second time step is the next time step of the first time step.

[0089] After the embodied intelligent system controls the robot to perform an action at the first time step, if the environmental perception information of the robot at the second time step obtained by the embodied intelligent system after the first time step changes from the environmental perception information of the robot at the first time step, the embodied intelligent system will again combine the changed environmental perception information and the navigation target of the robot to perform task planning, thereby generating a step-by-step navigation instruction that is updated compared to the step-by-step navigation instruction at the first time step. In addition, the embodied intelligent system further performs navigation path planning according to the updated step-by-step navigation instruction to generate a reasonable navigation path for the robot in the current environment, and controls the robot to perform actions according to the navigation path at the second time step, thereby continuing to perform the operation task indicated by the user instruction. In this way, the embodied intelligent system obtains the step-by-step navigation instruction of the robot at each time step by combining the environmental perception information and the navigation target for dynamic task planning when the environmental perception information changes, and controls the robot to perform actions based on the generated step-by-step navigation instruction until the operation task indicated by the user instruction is completed or the maximum number of steps for the robot to perform the action is reached.

[0090] In some embodiments, the embodied intelligent system can obtain the robot's environmental perception information at each time step based on an interactive prompt navigation mechanism, thereby updating the step-by-step navigation instructions in combination with the changed environmental perception information and navigation goals, and selecting the actions for the robot to perform based on the generated step-by-step navigation instructions.

[0091] It should be noted that the embodied intelligent system can perform actions through the robot's navigation agent. In addition, the embodied intelligent system can also receive high-level short instructions (user instructions) input by the user through the navigation agent. After the embodied intelligent system performs task planning on the user instructions to generate preliminary static planning instructions (navigation goals), the navigation agent uses the perception module to extract information about the robot's environment to generate environmental perception information. Then, the embodied intelligent system combines the environmental perception information and the navigation goal to perform task planning to generate fine-grained step-by-step navigation instructions, so that the navigation agent selects the execution action according to the generated step-by-step navigation instructions at each time step until the operation task indicated by the user instruction is completed or the maximum number of steps is reached.

[0092] In the embodiment of the present application, the embodied intelligent system obtains the environmental perception information generated by the robot at the current first time step based on the information of the environment collected by the perception module. Afterwards, the embodied intelligent system combines the environmental perception information and the navigation target determined by the user to perform task planning, thereby obtaining a step-by-step navigation instruction for the robot. Afterwards, the embodied intelligent system performs navigation path planning according to the step-by-step navigation instruction to generate a reasonable navigation path for the robot in the current environment, and controls the robot to perform actions according to the navigation path at the first time step, thereby performing the operation task indicated by the user instruction. In addition, if the environmental perception information of the robot at the second time step changes from the environmental perception information of the robot at the first time step, the embodied intelligent system performs task planning again in combination with the changed environmental perception information and the navigation target of the robot, thereby generating a step-by-step navigation instruction updated compared to the step-by-step navigation instruction of the first time step, and further performs navigation path planning according to the updated step-by-step navigation instruction to generate a reasonable navigation path for the robot in the current environment, and controls the robot to perform actions according to the navigation path at the second time step, thereby continuing to perform the operation task indicated by the user instruction.

[0093] That is to say, the embodied intelligent system can determine the navigation target based on the high-level and brief user instructions, and then combine the robot's dynamically changing environmental perception information at different time steps and the navigation target to perform task planning to generate relatively fine-grained step-by-step navigation instructions, and plan the navigation path according to the step-by-step navigation instructions to control the robot to perform actions, so that the robot performs the operation tasks indicated by the user instructions. In this way, the embodiments of the present application enable the embodied intelligent system to dynamically generate a reasonable navigation path according to high-level and brief instructions in a complex and changeable environment, thereby improving the adaptability and efficiency of the embodied intelligent system in responding to high-level and brief instructions for navigation path planning in complex scenarios.

[0094] The embodied intelligent system can receive high-level, short instructions (user instructions) input by the user to the robot, and then generate preliminary static planning instructions (navigation goals) for the robot through goal-oriented static task planning. After that, the embodied intelligent system combines the robot's environmental perception information and navigation goals to perform scenario-oriented dynamic task planning to generate step-by-step navigation instructions for the robot that are more fine-grained than user instructions.

[0095] In some embodiments, after receiving the user's instructions, the embodied intelligent system can perform goal-oriented static task planning through the LLM large model to determine the robot's navigation target. Figure 2 , Figure 2A schematic flow chart of the steps of static planning preliminary instructions involved in some embodiments of the navigation path planning method provided in the embodiments of the present application. Figure 2 As shown, the navigation path planning method provided in the embodiment of the present application may also include step S201 and step S202 as shown below.

[0096] Step S201: identifying a target object in an operation task corresponding to a user instruction through a preset large-scale language model, and determining a location of the target object through the large-scale language model using a preset knowledge base.

[0097] It should be noted that the preset knowledge base can be the world knowledge base of the LLM large model plug-in.

[0098] After receiving the user's instruction, the embodied intelligent system inputs the user's instruction into the pre-built LLM large model (large-scale language model), so that the user's instruction is parsed by the LLM large model to identify the target object in the operation task that the user's instruction instructs the robot to perform, and, when the target object is identified, the LLM large model further uses the world knowledge base to determine the location of the target object. For example, when the user instruction received by the embodied intelligent system is a high-level short instruction "empty the washing machine on the first floor", after the embodied intelligent system inputs the instruction into the LLM large model, the LLM large model can identify the target object in the operation task of "emptying the washing machine on the first floor" that the user instructs the robot to perform as "the washing machine on the first floor" based on parsing "empty the washing machine on the first floor". Afterwards, the LLM large model uses the world knowledge base to query, so that it can be determined that the target object "the washing machine on the first floor" is located in the "laundry room on the first floor".

[0099] In some embodiments, the embodied intelligent system can input a prompt of "parse the instruction to identify the target object and infer the location of the target object" to the LLM big model at the same time when inputting the user instruction into the LLM big model, so that the LLM big model can automatically parse the instruction to identify the object and further use world knowledge to determine the location of the target object after inputting the user instruction.

[0100] In other embodiments, the embodied intelligent system can also set the LLM model to parse the input user instructions to identify the target object when pre-building the LLM model, and further infer the location of the target object through world knowledge. In this way, in actual applications, after the embodied intelligent system inputs the user instructions into the LLM model, the LLM model can automatically parse the instructions to identify the object and further use world knowledge to determine the location of the target object.

[0101] Step S202: The position of the target object determined and output by the large-scale language model is determined as the navigation target of the robot.

[0102] After the embodied intelligent system obtains the target object in the operation task indicated by the user's instructions through the LLM large model and infers the location of the target object, when the LLM large model outputs the location of the target object, the embodied intelligent system can directly determine the location as the navigation target that the robot needs to reach to perform the operation task.

[0103] It should be noted that the embodied intelligent system performs static task planning for user instructions through the LLM model: "The target object is identified through the LLM model, and the location of the target object is inferred and output, and the location is used as the navigation target of the robot." It can be considered that the navigation target is the preliminary static planning instruction of the robot obtained by the embodied intelligent system through the LLM model to perform static task planning for user instructions.

[0104] In this embodiment, after receiving the user's instructions, the embodied intelligent system uses the world knowledge through the LLM model to perform goal-oriented static planning, thereby obtaining the robot's preliminary static planning instructions (navigation goals). After that, the embodied intelligent system combines the navigation goal and the robot's dynamically changing environmental perception information at each time step to perform dynamic task planning to obtain fine-grained step-by-step navigation instructions for each time step. In this way, the navigation problem under high-level short instructions can be solved, so that the embodied intelligent system can effectively navigate under high-level short instructions. In addition, the embodied intelligent system first performs static task planning to generate preliminary instructions for the robot, and then uses dynamic task planning to update the step-by-step navigation instructions according to the environmental perception information, so as to plan the navigation path according to the updated step-by-step navigation instructions, so as to adjust the navigation path of the robot when performing actions at each time step, and also ensure that the embodied intelligent system can flexibly respond to complex environments.

[0105] The embodied intelligent system can obtain the robot's environmental perception information by extracting features from the panoramic environment image of the robot's environment.

[0106] Please refer to Figure 3 , Figure 3 for Figure 1 A schematic diagram of a detailed step flow chart of step S101.

[0107] like Figure 3 As shown, in some embodiments, the above step S101: obtaining the robot's environmental perception information at the first time step may include steps S301 and S302 as shown below.

[0108] Step S301: Identify the panoramic environment image of the environment where the robot is located at the first time step, and obtain the space type information and visible object information of the environment where the robot is located.

[0109] It should be noted that the space type information may be the type of the room the robot is in. For example, the space type information may specifically be a bedroom, a living room, a kitchen, a corridor, a laundry room, and the like.

[0110] The embodied intelligent system controls the robot to use the perception module to collect a panoramic environment image of the robot's environment at the first time step at the current first time step. Then, the embodied intelligent system parses and recognizes the panoramic environment graphic based on image recognition technology, thereby extracting the spatial type information and visible object information of the robot's environment at the first time step from the panoramic image.

[0111] In some embodiments, the embodied intelligent system can control the robot to use its own perception module and call the image acquisition device to visually observe the robot's surrounding environment, thereby acquiring visual observation data of the robot's environment (i.e., a panoramic environment image).

[0112] In other embodiments, the embodied intelligent system can receive a panoramic environmental image of the robot's environment uploaded by a terminal device associated with the robot by controlling the robot based on the data transmission interface provided by the robot to the outside world.

[0113] Step S302: performing zero-sample classification on the space type information and the visible object information to generate the environment perception information of the robot at the first time step.

[0114] After the embodied intelligent system identifies the panoramic environment image and obtains the spatial type information and visible object information of the environment in which the robot is located at the first time step, it further generates environmental perception information of the environment in which the robot is located at the first time step based on zero-sample classification of the spatial type information and the visible object information.

[0115] It should be noted that the embodied intelligent system can perform zero-shot classification of spatial type information and visible object information through zero-shot learning (ZSL).

[0116] The embodied intelligent system can also perform multimodal feature extraction and fusion processing on the panoramic environmental images of the robot's environment through a multimodal large model (referred to as the "CLIP model"), thereby obtaining the robot's environmental perception information.

[0117] Please refer to Figure 4 , Figure 4 for Figure 1Another detailed flowchart of step S101 is shown in FIG.

[0118] like Figure 4 As shown, in some embodiments, the above step S101: obtaining the robot's environmental perception information at the first time step may also include steps S401 and S402 as shown below.

[0119] Step S401: extract features of the panoramic environment image of the environment in which the robot is located at the first time step through a preset multimodal large model to obtain environmental visual features corresponding to the panoramic environment image; and extract features of environmental voice information of the environment in which the robot is located at the first time step through the multimodal large model to obtain environmental language features corresponding to the environmental voice information.

[0120] It should be noted that the preset multimodal large model can be an open source MINI-GEMINI multimodal large model. It should be understood that based on the different design requirements of actual applications, in other different feasible embodiments, the system can of course also use other multimodal large models different from the MINI-GEMINI multimodal large model to process the robot's environmental perception information, and the method for generating behavioral instructions provided in the embodiment of the present application is not limited to the specific type of the multimodal large model.

[0121] In addition, the environmental voice information may be voice input information of the surrounding environment (such as environmental noise data and / or user voice data, etc.) perceived by the robot through the perception module calling the microphone.

[0122] When the embodied intelligent system controls the robot to collect the panoramic environment image of the environment in the first time step at the current first time step, it can also control the robot to collect the environmental voice information of the environment in the first time step at the first time step. In this way, when the embodied intelligent system obtains the environmental perception information of the environment in which the robot is located at the first time step, it can extract features of the panoramic environment image through the preset MINI-GEMINI multimodal large model to obtain the environmental visual features corresponding to the panoramic environment image (such as space type and visible object information, etc.), and extract features of the environmental voice information through the multimodal large model to obtain the environmental language features corresponding to the environmental voice information (such as environmental language description information, etc.).

[0123] Step S402: The environment visual features and the environment language features are fused through the multimodal large model to generate the environment perception information of the robot at the first time step.

[0124] After obtaining the environmental visual features and environmental language features through the MINI-GEMINI multimodal large model, the embodied intelligent system further performs feature fusion processing on the environmental visual features and environmental language features through the MINI-GEMINI multimodal large model, thereby generating accurate and comprehensive environmental perception information of the robot's environment at the first time step.

[0125] In this embodiment, the embodied intelligent system uses the robot's perception module to collect panoramic environment images, and then recognizes the panoramic environment images to obtain space type information and visible object information. That is, the space type and visible object information are extracted based on the robot's visual observation of the surrounding environment, which can improve the accuracy of environmental perception.

[0126] In addition, the embodied intelligent system generates environmental perception information by performing zero-sample classification on spatial type information and visible object information. Based on the CLIP model, the room type and visible objects are first extracted from visual observations and the language description is extracted from the ambient speech. The room type, visible objects and language description are then fused to generate environmental perception information. This can improve the recognition ability of complex scenes and diverse objects, thereby enhancing the adaptability and perception accuracy of the embodied intelligent system in complex environments.

[0127] When the embodied intelligent system dynamically combines environmental perception information and navigation goals to perform task planning and generate fine-grained step-by-step navigation instructions, it can dynamically select the most suitable demonstration examples as prompts to improve the relevance and rationality of the generated step-by-step navigation instructions.

[0128] Based on this, in some embodiments, the present application provides a navigation path planning method, which may also include the following steps:

[0129] A mission planning demonstration example is selected based on the user instruction.

[0130] Before the embodied intelligent system combines environmental perception information and navigation goals for dynamic task planning, the system can first select, based on the received user instructions, from a preset example library the most suitable task planning demonstration example for the current dynamic task planning combined with environmental perception information and navigation goals at the first time step or the second time step.

[0131] It should be noted that the preset example library stores a series of correct task decomposition examples. For example, a high-level short instruction: "Go to the kitchen and find the apples on the table" is decomposed into instructions "1. Navigate to the kitchen; 2. Identify the apples; 3. Pick up the apples" and so on.

[0132] Please refer to Figure 5 , Figure 5A flowchart of the steps of selecting a task planning demonstration example involved in some embodiments of the navigation path planning method provided in the embodiments of the present application.

[0133] like Figure 5 As shown, in some embodiments, the above-mentioned "selecting a task planning demonstration example based on the user instruction" may include the following steps S501 and S502.

[0134] Step S501: performing similarity matching between the user instruction and a plurality of basic instructions in a preset example library, and obtaining a target basic instruction having the highest similarity with the user instruction among the plurality of basic instructions.

[0135] It should be noted that the basic instructions in the preset example library can be low-level instructions such as the above-mentioned step-by-step navigation instructions, such as walk to the door, turn left, enter the kitchen, get water, navigate to the kitchen, identify apples, pick up apples, and other fine-grained instructions.

[0136] When the embodied intelligent system selects the most appropriate task planning demonstration example from a preset example library, it can match the user instructions one by one with the multiple basic instructions in the preset example library for similarity, thereby retrieving the target basic instruction with the highest similarity to the user instruction among the multiple basic instructions.

[0137] Step S502: Select a task planning demonstration example in the preset example library whose task decomposition target is the target basic instruction.

[0138] After retrieving the target basic instruction with the highest similarity to the user instruction in the preset example library, the embodied intelligent system further determines a task decomposition example in the preset example library in which the target basic instruction is obtained as the task decomposition target, and then selects the task decomposition example as the most suitable task planning demonstration example.

[0139] In some embodiments, the embodied intelligent system can calculate the semantic similarity between the user instruction and the basic instructions in the preset example library, thereby selecting the most suitable example for dynamic task planning to generate fine-grained step-by-step navigation instructions. The embodied intelligent system can calculate the semantic similarity between the user instruction and the basic instructions in the preset example library by the formula shown below.

[0140] ;

[0141] Among them, s( ) is the semantic similarity, e( ) is the embedding function, Q is the high-level embedding representation of user instructions, and K i It is an embedded representation of low-level basic instructions in the preset example library.

[0142] In some embodiments, the above step S102: combining the environmental perception information and the navigation target to perform task planning to obtain step-by-step navigation instructions for the robot may include the following steps:

[0143] In combination with the environmental perception information and the navigation target, task planning is performed based on the prompt information provided by the task planning demonstration example to obtain step-by-step navigation instructions for the robot.

[0144] After the embodied intelligent system matches the user instructions with the basic instructions in the preset example library for similarity to find the most suitable task planning demonstration example, it inserts the task planning demonstration example into the system prompt. When dynamic task planning is performed in combination with environmental perception information and navigation goals, task planning can be performed based on the prompt information provided by the task planning demonstration example to generate fine-grained step-by-step navigation instructions.

[0145] It should be noted that the embodied intelligent system can combine environmental perception information and navigation goals through the LLM large model for dynamic task planning. In this way, by inserting task planning demonstration examples into the system prompts for the LLM large model, the embodied intelligent system can help the LLM large model complete more accurate reasoning when performing dynamic task planning, and then generate fine-grained step-by-step navigation instructions.

[0146] Please refer to Figure 6 , Figure 6 A schematic diagram of a linkage process for generating step-by-step navigation instructions in a retrieval task planning demonstration example involved in some embodiments of the navigation path planning method provided in the embodiments of the present application.

[0147] For example, Figure 6 As shown, after receiving the user's instruction, the embodied intelligent system will perform goal-oriented static planning processing on the robot based on the user's instruction, and will also match the user's instruction with multiple basic instructions in the preset example library one by one for similarity, thereby retrieving the target basic instruction with the highest similarity to the user's instruction among the multiple basic instructions, so as to select the task planning demonstration example in the preset example library whose task decomposition target is the target basic instruction. Afterwards, when the demonstration example is selected through retrieval, the embodied intelligent system will construct a system prompt based on the demonstration example, so as to insert the system prompt into the LLM large model that combines environmental perception information and navigation goals for dynamic task planning, so that the LLM large model can complete more accurate reasoning when performing dynamic task planning, and then generate fine-grained step-by-step navigation instructions that meet the scene tasks.

[0148] In this embodiment, the embodied intelligent system selects the most suitable task planning demonstration example from a preset example library based on a dynamic demonstration selection mechanism, so that when performing scene-oriented task planning in combination with environmental perception information and navigation goals, task planning is performed based on the prompt information provided by the task planning demonstration example. In this way, it can be ensured that the embodied intelligent system decomposes the task correctly enough when performing task planning, so that the generated fine-grained step-by-step navigation instructions are closer to the actual scene, thereby improving the efficiency and accuracy of navigation path planning based on step-by-step navigation instructions.

[0149] When the embodied intelligent system selects instructions to control the robot to perform actions at each time step, it can perform physical constraint verification on the step-by-step navigation instructions generated at the current time step, and then select the step-by-step navigation instructions for navigation path planning after confirming that the step-by-step navigation instructions have passed the verification, thereby controlling the robot to perform actions according to the navigation path obtained by the plan to perform the operation tasks indicated by the user instructions.

[0150] Please refer to Figure 7 , Figure 7 for Figure 1 Schematic diagram of the detailed process flow of step S103.

[0151] like Figure 7 As shown, in some embodiments, the above-mentioned step S103: controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the step-by-step navigation instruction may include steps S701 to S703 as shown below.

[0152] Step S701: Match the step-by-step navigation instruction with the atomic instruction library of the robot, and perform task logic verification on the step-by-step navigation instruction to obtain a processing result.

[0153] After the embodied intelligent system generates the robot's step-by-step navigation instructions at each time step, it further matches the step-by-step navigation instructions with the robot's atomic instruction library that has been set in advance, so as to confirm whether the step-by-step navigation instructions are basic instructions existing in the atomic instruction library. In addition, the embodied intelligent system also performs task logic verification on the step-by-step navigation instructions to confirm whether the step-by-step navigation instructions conform to the robot's current task logic. After matching the step-by-step navigation instructions with the robot's atomic instruction library and performing task logic verification on the step-by-step navigation instructions, the system obtains the corresponding processing results.

[0154] Step S702: when the processing result indicates that the step-by-step navigation instruction belongs to the atomic instruction library and the step-by-step navigation instruction passes the task logic verification, a navigation path is obtained according to the step-by-step navigation instruction planning.

[0155] It should be noted that the step-by-step navigation instruction belongs to the atomic instruction library, which means that the embodied intelligent system matches the step-by-step navigation instruction with the atomic instruction library, thereby confirming that the step-by-step navigation instruction is a basic instruction existing in the atomic instruction library.

[0156] After the system matches the step-by-step navigation instruction with the robot's atomic instruction library and performs task logic verification on the step-by-step navigation instruction to obtain the corresponding processing result, if the processing result indicates that the step-by-step navigation instruction is a basic instruction existing in the atomic instruction library, and the step-by-step navigation instruction has also passed the system's task logic verification, then in this case, the embodied intelligent system selects the step-by-step navigation instruction at the current time step for navigation path planning, thereby obtaining the robot's navigation path at the current time step.

[0157] Step S703: Control the robot to move along the navigation path to perform the operation task corresponding to the user instruction.

[0158] After planning the navigation path of the robot at the current time step, the embodied intelligent system immediately controls the robot to move along the navigation path to perform actions, thereby executing the operation tasks instructed by the user instructions to the robot.

[0159] In this embodiment, the embodied intelligent system performs physical constraint verification on the step-by-step navigation instructions generated at each time step to ensure that these instructions are both in the robot's atomic instruction library and conform to the logic of the robot's current task, thereby avoiding conflicts and erroneous operations when the robot executes the actions corresponding to these instructions. This can improve the executability and logical correctness of the embodied intelligent system controlling the robot to perform operational tasks, thereby improving the success rate of the robot executing instructions to achieve task goals.

[0160] Next, a navigation path planning method provided in an embodiment of the present application is proposed, which is a general embodiment of dynamically generating a reasonable navigation path for the robot based on an embodied intelligent navigation algorithm framework in response to high-level brief instructions.

[0161] It should be noted that the embodied intelligent navigation algorithm framework includes a goal-oriented static planning module, a room and object perception module, and a scene-oriented dynamic planning module. The embodied intelligent system can select execution actions and dynamically update instructions at each time step based on the generated static and dynamic planning instructions through an interactive prompt navigation mechanism, thereby ensuring the accuracy and efficiency of the navigation path, and then achieving effective navigation in complex environments under high-level and brief instructions.

[0162] Please refer to Figure 8 , Figure 8 A schematic diagram of the overall process involved in some embodiments of the navigation path planning method provided in the embodiments of the present application.

[0163] like Figure 8 As shown in the figure, when the embodied intelligent system generates the navigation path of the robot based on the embodied intelligent navigation algorithm framework, after receiving the high-level brief instructions (user instructions) issued by the user, the static planning module uses the world knowledge of the LLM large model to establish the navigation goal; and the room and object perception module uses the CLIP model to extract the information of the robot's environment at each time step to generate environmental perception information; finally, the dynamic planning module combines the environmental perception information and the navigation goal to perform task planning at each time step, so as to split the subtasks and output specific step-by-step navigation instructions according to the environmental state and navigation goal of the robot's environment at the current time step. The room and object perception module and the dynamic planning module alternately iterate at each time step until the embodied intelligent system controls the robot to perform actions according to the navigation path planned based on the step-by-step navigation instructions, completes the task goal, or reaches the maximum number of iteration steps.

[0164] It should be noted that in the framework of the embodied intelligent navigation algorithm, the input of the static planning module is the high-level user instructions received by the embodied intelligent system. It identifies the target object and infers its possible position through specially designed prompts (analyzing and identifying the position of the target object and the target object on the other end in the operation task indicated by the user instruction), and then outputs the robot's preliminary static planning instructions (navigation target). In addition, the input of the room and object perception module is the robot's visual observation data (panoramic environment image) at each time step, and then uses the CLIP model to extract the room type and visible object information from the visual observation data, generate environmental perception information and output it. Furthermore, the input of the dynamic planning module is the environmental perception information output by the room and object perception module at each time step, and the navigation target (or high-level user instructions) output by the static planning module, and its output is the fine-grained step-by-step navigation instructions of the robot at each time step. The dynamic planning module mainly combines the environmental perception information to dynamically generate step-by-step navigation instructions for each time step, and can select the task planning demonstration example through the dynamic demonstration selection mechanism to generate step-by-step navigation instructions to optimize the robot's navigation path.

[0165] In addition, the interactive prompt navigation mechanism in the embodied intelligent navigation algorithm framework takes the static planning instructions (navigation goals) generated by the static planning module, the step-by-step navigation instructions dynamically generated by the dynamic planning module, and the environmental perception information as input, so that at each time step, the execution action is selected and output according to the generated step-by-step navigation instructions, and the step-by-step navigation instructions are dynamically updated to ensure the accuracy and efficiency of the planned navigation path.

[0166] In some embodiments, the functional sub-modules in the embodied intelligent navigation algorithm framework can be linked according to the process shown below.

[0167] 1. Initialization: The embodied intelligent system receives high-level and brief user instructions through the navigation agent. The static planning module identifies the target object in the operation task indicated by the user instruction through the LLM large model and infers its possible position, generating the robot's preliminary static planning instructions (navigation target).

[0168] 2. Environmental perception: The embodied intelligent system uses the room and object perception module to extract room type and visible object information from visual observation data (panoramic environmental image) at each time step through the navigation agent to generate environmental perception information.

[0169] 3. Dynamic Planning: The embodied intelligent system consists of a dynamic planning module that combines environmental perception information and high-level, concise user instructions to dynamically generate fine-grained step-by-step navigation instructions, and optimizes the navigation path through a dynamic demonstration selection mechanism.

[0170] 4. Execution and update: The embodied intelligent system selects the actions to be performed by the robot according to the step-by-step navigation instructions generated by the navigation agent at the current time step. In addition, if the room and object perception module detects that the robot's environment has changed, it generates the changed environmental perception information and triggers the dynamic planning module to generate updated step-by-step navigation instructions in combination with the updated environmental perception information, thereby updating the robot's navigation path. The embodied intelligent system repeats the above process through the navigation agent until the operation task indicated by the user instruction is completed or the maximum number of steps is reached.

[0171] Next, a complete embodiment of the navigation path planning method provided by the embodiment of the present application in response to a high-level brief instruction is proposed.

[0172] Please refer to Fig. 9 , Fig. 9 A schematic diagram of the application flow of the navigation path planning method provided in some embodiments of the present application.

[0173] like Fig. 9As shown in the figure, when the embodied intelligent system receives the user instruction "empty the washing machine on the first floor", it first parses the user instruction based on the goal-oriented static planning module to recognize that the target object that the instruction indicates the robot to operate is the "washing machine", and uses the world indication of the LLM large model to infer that the location of the "washing machine" is the "laundry room on the first floor" (navigation target). After that, as time goes by, the embodied intelligent system controls the robot to collect panoramic environmental images in the current environment (bedroom), and recognizes the panoramic environmental images through the room and object perception module to extract the room type (bedroom) and visible object information (bed, lamp, etc.) to generate environmental perception information "In the bedroom, you can see the bed, lamp, etc.". In addition, the embodied intelligent system combines the environmental perception information and the navigation target for task planning through the scene-oriented dynamic planning module, and generates a step-by-step navigation instruction "exit the current room" that meets the robot's current time and environment. After that, the embodied intelligent system controls the robot to collect panoramic environmental images in the environment (corridor) in the next time step, and recognizes the panoramic environmental images through the room and object perception module to extract the room type (corridor) and visible object information (faucets, washing machines, etc. in the laundry room) to generate environmental perception information "In the corridor, you can see faucets, washing machines, etc. in the laundry room". In addition, the embodied intelligent system further uses the scene-oriented dynamic planning module to combine the environmental perception information and navigation goals that have changed at the current time to plan tasks, and further generates step-by-step navigation instructions that are consistent with the robot at the current time and environment, "Go straight and enter the laundry room". In this way, the embodied intelligent system can finally complete the task of emptying the washing machine on the first floor by combining the room and object perception module and the dynamic planning module to alternately iterate and generate updated dynamic navigation instructions.

[0174] See also Fig.10 The present application also provides a navigation path planning device, which can implement the above navigation path planning method. The device includes an acquisition module 1001, a dynamic planning module 1002 and a task execution module 1003.

[0175] An acquisition module 1001 is used to acquire the environment perception information of the robot at the first time step;

[0176] A dynamic planning module 1002 is used to perform task planning in combination with the environmental perception information and the navigation target to obtain step-by-step navigation instructions for the robot, wherein the navigation target is determined based on the user instruction;

[0177] A task execution module 1003, used to control the robot to execute the operation task corresponding to the user instruction according to the navigation path planned by the step-by-step navigation instruction;

[0178] The dynamic planning module 1002 is further configured to perform task planning in combination with the changed environmental perception information and the navigation target to obtain an updated step-by-step navigation instruction if the environmental perception information of the robot at the second time step changes compared to the environmental perception information of the robot at the first time step; wherein the second time step is the next time step of the first time step;

[0179] The task execution module 1003 is further used to re-execute the steps of controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the updated step-by-step navigation instruction.

[0180] In some embodiments, the dynamic planning module 1002 is also used to select a task planning demonstration example based on the user instructions; and, in combination with the environmental perception information and navigation goals, perform task planning based on the prompt information provided by the task planning demonstration example to obtain step-by-step navigation instructions for the robot.

[0181] In some embodiments, the dynamic planning module 1002 is also used to perform similarity matching between the user instructions and multiple basic instructions in a preset example library, respectively, to obtain a target basic instruction among the multiple basic instructions that has the highest similarity with the user instruction; and to select a task planning demonstration example in the preset example library whose task decomposition target is the target basic instruction.

[0182] In some embodiments, the acquisition module 1001 is also used to identify the panoramic environment image of the environment in which the robot is located at the first time step, and obtain the spatial type information and visible object information of the environment in which the robot is located; and perform zero-sample classification on the spatial type information and visible object information to generate the environmental perception information of the robot at the first time step.

[0183] In some embodiments, the acquisition module 1001 is also used to perform feature extraction on the panoramic environment image of the environment in which the robot is located at the first time step through a preset multimodal large model to obtain environmental visual features corresponding to the panoramic environment image; and to perform feature extraction on the environmental voice information of the environment in which the robot is located at the first time step through the multimodal large model to obtain environmental language features corresponding to the environmental voice information; and to fuse the environmental visual features and the environmental language features through the multimodal large model to generate the environmental perception information of the robot at the first time step.

[0184] In some embodiments, the dynamic programming module 1002 is also used to identify the target object in the operation task corresponding to the user instruction through a preset large-scale language model, and to determine the position of the target object using a preset knowledge base through the large-scale language model; and, determine the position of the target object determined and output by the large-scale language model as the navigation target of the robot.

[0185] In some embodiments, the task execution module 1003 is also used to match the step-by-step navigation instruction with the atomic instruction library of the robot, and to perform task logic verification on the step-by-step navigation instruction to obtain a processing result; when the processing result indicates that the step-by-step navigation instruction belongs to the atomic instruction library and the step-by-step navigation instruction passes the task logic verification, a navigation path is obtained according to the step-by-step navigation instruction planning; and, the robot is controlled to move along the navigation path to perform the operation task corresponding to the user instruction.

[0186] The specific implementation of the navigation path planning device provided in the embodiment of the present application is basically the same as the specific implementation of the navigation path planning method described above, and will not be repeated here.

[0187] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above navigation path planning method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0188] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0189] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0190] The memory 1102 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the navigation path planning method of the embodiments of this application;

[0191] Input / output interface 1103, used to implement information input and output;

[0192] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0193] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0194] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0195] An embodiment of the present application also provides a robot, which is equipped with an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned navigation path planning method when executing the computer program.

[0196] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned navigation path planning method is implemented.

[0197] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory includes a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0198] The embodiment of the present application also provides a computer program product, including a computer program. The steps implemented when the computer program is executed by a processor are basically the same as the specific embodiment of the above-mentioned navigation path planning method, and will not be repeated here.

[0199] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0200] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0201] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0203] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0204] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0205] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0206] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0207] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0208] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0209] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A navigation path planning method, characterized in that: The method comprises: Obtain the robot's environmental perception information at the first time step; Combining the environmental perception information and the navigation target to perform task planning to obtain step-by-step navigation instructions for the robot, wherein the navigation target is determined based on the user instruction; Controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the step-by-step navigation instruction; If the environment perception information of the robot at the second time step changes compared to the environment perception information of the robot at the first time step, then task planning is performed in combination with the changed environment perception information and the navigation target to obtain an updated step-by-step navigation instruction, and the steps of controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the updated step-by-step navigation instruction are re-executed, wherein the second time step is the next time step of the first time step; The method further comprises: Selecting a mission planning demonstration example based on the user instruction; The step of combining the environmental perception information and the navigation target to perform task planning to obtain step-by-step navigation instructions for the robot includes: In combination with the environmental perception information and the navigation target, task planning is performed based on the prompt information provided by the task planning demonstration example to obtain step-by-step navigation instructions for the robot.

2. The method according to claim 1, characterized in that The selecting a task planning demonstration example based on the user instruction includes: Performing similarity matching between the user instruction and a plurality of basic instructions in a preset example library, respectively, to obtain a target basic instruction having the highest similarity with the user instruction among the plurality of basic instructions; Select a task planning demonstration example in the preset example library whose task decomposition target is the target basic instruction.

3. The method according to claim 1, characterized in that The step of obtaining the robot's environmental perception information at the first time step includes: Identify a panoramic environment image of the environment in which the robot is located at a first time step, and obtain spatial type information and visible object information of the environment in which the robot is located; Zero-shot classification is performed on the space type information and the visible object information to generate environment perception information of the robot at a first time step.

4. The method according to claim 1, characterized in that: The step of obtaining the robot's environmental perception information at the first time step includes: Extracting features of a panoramic environment image of the environment in which the robot is located at a first time step by using a preset multimodal large model, and obtaining environmental visual features corresponding to the panoramic environment image; and extracting features of environmental voice information of the environment in which the robot is located at the first time step by using the multimodal large model, and obtaining environmental language features corresponding to the environmental voice information; The environmental visual features and the environmental language features are fused through the multimodal large model to generate environmental perception information of the robot at a first time step.

5. The method according to claim 1, characterized in that The method further comprises: Identifying a target object in an operation task corresponding to a user instruction through a preset large-scale language model, and determining a location of the target object through the large-scale language model using a preset knowledge base; The position of the target object determined and output by the large-scale language model is determined as the navigation target of the robot.

6. The method according to any one of claims 1 to 5, characterized in that The controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the step-by-step navigation instruction includes: Matching the step-by-step navigation instruction with the atomic instruction library of the robot, and performing task logic verification on the step-by-step navigation instruction to obtain a processing result; When the processing result indicates that the step-by-step navigation instruction belongs to the atomic instruction library and the step-by-step navigation instruction passes the task logic verification, a navigation path is obtained according to the step-by-step navigation instruction planning; The robot is controlled to move along the navigation path to perform the operation task corresponding to the user instruction.

7. A navigation path planning device, characterized in that: The device comprises: An acquisition module is used to obtain the robot's environmental perception information at the first time step; A dynamic planning module, used to perform task planning in combination with the environmental perception information and the navigation target to obtain step-by-step navigation instructions for the robot, wherein the navigation target is determined based on the user instructions; A task execution module, used to control the robot to execute the operation task corresponding to the user instruction according to the navigation path planned by the step-by-step navigation instruction; The dynamic planning module is further configured to perform task planning in combination with the changed environmental perception information and the navigation target to obtain updated step-by-step navigation instructions if the environmental perception information of the robot at the second time step changes compared to the environmental perception information of the robot at the first time step; wherein the second time step is the next time step of the first time step; The task execution module is further used to re-execute the step of controlling the robot to perform the operation task corresponding to the user instruction according to the navigation path planned by the updated step-by-step navigation instruction; The dynamic planning module is also used to select a task planning demonstration example based on the user instructions; and, in combination with the environmental perception information and the navigation target, perform task planning based on the prompt information provided by the task planning demonstration example to obtain step-by-step navigation instructions for the robot.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the navigation path planning method according to any one of claims 1 to 6 when executing the computer program.

9. A robot, characterized in that: The robot includes an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the navigation path planning method described in any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the navigation path planning method according to any one of claims 1 to 6 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the navigation path planning method described in any one of claims 1 to 6 are implemented.

Images