Service robot path planning system with self-repairing function and method thereof

Abstract

The invention relates to the field of robot path planning, and discloses a service robot path planning system with a self-repairing function and a method thereof. The voice recognition module is used for carrying out voice activity detection and transcription and realizing accurate understanding of natural language commands; the object recognition module is used for carrying out object detection and classification and simultaneously receiving text input so as to provide recognition prompts and enhance the perceptual ability of the robot in the environment; the task planning module is used for converting a natural language command into an executable format and realizing automatic execution of a complex task by executing combination of skill functions; and the semantic mapping module integrates the environment information into a 3D semantic map, carries out image and text coding, and provides rich semantic information for task planning with a self-repairing function of the robot. According to the invention, efficient path planning and task execution in a dynamic and huge service environment are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot path planning, and in particular relates to a service robot path planning system with a self-repairing function and a method thereof. Background Art

[0002] With the rapid development of robotics technology, the field of mobile robots has shown a rapid growth momentum, especially service mobile robots, which have attracted much attention in recent years. As partners working with humans and other robots, service robots play a key role in the execution of automated tasks in many occasions such as hotels, hospitals, restaurants, homes and nursing homes.

[0003] Among the core technologies of service robots, navigation systems and path planning modules are particularly important. These systems give robots the ability to operate autonomously, enabling them to move autonomously from source to destination with little or no human intervention. As an important part of the navigation system, the path planning system is responsible for determining the optimal path to ensure that the robot completes its tasks efficiently and safely.

[0004] However, the dynamic and complex nature of the service environment poses many challenges to path planning. Although the static path planning module can plan the path based on the initial environmental information, it cannot identify or adapt to changes in the environment. Therefore, the application of the dynamic path planning module becomes the key to solving this problem. The dynamic path planning module can detect environmental changes in real time and adjust the path planning accordingly to ensure that the robot can still operate efficiently in a changing environment.

[0005] In recent years, the development of artificial intelligence technology has provided new solutions for dynamic path planning. In particular, the application of technologies such as machine learning and reinforcement learning has made the path planning system more adaptable and dynamic. Reinforcement learning algorithms learn how to make optimal decisions in different states by interacting with the environment, thereby achieving dynamic adaptation to the environment. This path planning system based on reinforcement learning not only improves the autonomy of the robot, but also enables it to cope with long-term environmental changes.

[0006] However, despite these advances, general service robot systems still face challenges in generalization and adaptability when facing complex tasks and dynamic environments. The system often needs to readjust the working mode or rely on a lot of manual intervention when switching tasks, which reduces the efficiency and responsiveness of the system. In addition, when encountering failures or abnormal situations, the system often lacks sufficient robustness and cannot automatically recover to normal working state. Summary of the invention

[0007] The present invention provides a service robot path planning system with a self-repairing function, which realizes efficient path planning and task execution in a dynamic and large service environment by effectively combining voice recognition with a self-repairing function, object recognition, task planning with a self-repairing function and semantic mapping.

[0008] The present invention provides a service robot path planning method with a self-repairing function, which is used for executing complex path planning.

[0009] The present invention is achieved through the following technical solutions:

[0010] A service robot path planning system with self-repair function, the system using a RL framework with self-repair function;

[0011] The RL framework with self-repair function specifically includes a speech recognition module, an object recognition module, a task planning module, a semantic mapping module and a path planner;

[0012] The object recognition module performs object detection and classification and can simultaneously accept text input to provide recognition prompts, thereby enhancing the robot's perception ability in the environment;

[0013] The semantic mapping module integrates environmental information into a 3D semantic map, performs image and text encoding, and provides rich semantic information for the robot's task planning with self-repair capabilities;

[0014] The speech recognition module performs speech activity detection and converts natural language commands into an executable format to achieve accurate understanding of natural language commands;

[0015] The task planning module forms an LLM large language model through GPT;

[0016] The path planner combines environmental information with the LLM large language model based on the self-repairing RL algorithm of Q-Learning to output an executable path planning solution.

[0017] Furthermore, the training parameters of the path planner include states, actions and rewards;

[0018] Specifically, the state is represented by s∈S, where s is the current state of the agent, i.e., a subset of all possible states of the agent; in the work, S is the set of all nodes in the topological map; at any given time, the agent, i.e., the robot, will be in one of the nodes, i.e., a certain position in the environment;

[0019] Specifically, the action is represented by a∈A, where a is the action taken by the agent to reach the next state; since the agent is a mobile robot, A is the set of all basic movements of the robot on a two-dimensional plane, A={left, right, forward, backward};

[0020] The reward is specifically given to the agent based on the state-action feedback; when the desired goal g is reached, the highest positive reward is given, where g∈G and A small negative reward is given when any known or visited location is reached; the highest negative reward is given when an unknown state U is reached; an unknown location is any location in the environment that is not on the topological map.

[0021] Furthermore, the reward function of the path planner is:

[0022]

[0023] Among them, r(s,a) is the reward of the state-action pair (s,a); s is the current state of the agent; g is the desired goal; U is the unknown state; S is the set of all nodes in the topological map; G is the set of target nodes that we hope to reach.

[0024] Furthermore, reinforcement learning is performed based on the training parameters of the RL framework with self-repair function, specifically,

[0025] Training is performed in rounds. In each round, the agent will take multiple discrete steps. In each step, an action causes the agent's state to change. For such a transition, a reward will be given, and the agent updates the Q value according to the Bellman equation. An round will terminate based on the agent's current state s. The goal of the reinforcement learning agent is to get more rewards in each round.

[0026] During training, the agent uses the next-state table and the Q-table to learn to reach the desired goal efficiently; the next-state table is a lookup table used by the agent to transition states given the current state-action pair; the table contains all states as row elements and all actions as column elements; the components of the table are the next state based on the state-action pair; the Q-table is used to update the Q-value for a specific state-action pair; the table is used to evaluate the Bellman equation.

[0027] Furthermore, the termination condition of the robot's current action state s is:

[0028]

[0029] Among them, g is the desired goal; U is the unknown state; total states is the variable of the number of steps to reach all states; Otherwise is the continued action.

[0030] Furthermore, the self-repair function of the RL framework implements recovery M1 including insufficient information through a self-recovery mechanism. Specifically, in the case of insufficient information, the missing information used for planning will be supplemented by the common sense possessed by the planning module and additional information obtained through conversations with people.

[0031] Furthermore, the self-repair function of the RL framework is implemented through a self-recovery mechanism including recovery M2 of incorrect plan generation, specifically, in the case of incorrect plan generation, a solution is developed for command recognition and plan generation.

[0032] Furthermore, the self-repair function of the RL framework is implemented through a self-recovery mechanism, including recovery M3 of plan execution failure. Specifically, in the case of plan execution failure, the failure is recovered through skills and plans.

[0033] Furthermore, the self-repairing RL algorithm of the Q-Learning of the path planner specifically includes the following steps:

[0034] Step S1: Initialize the Q value table and the next state table Next_State_Table;

[0035] Step S2: Initialize the hyperparameters learning rate α, discount factor γ, exploration rate ε and total number of training episodes episode_count;

[0036] Step S3: Determine whether the current training set episode_count ≥ episode, if yes, proceed to step S4, if no, proceed to step S15;

[0037] Step S4: Randomly select an initial state s;

[0038] Step S5: Determine whether the initial state s is the terminal state, if so, proceed to step S6, if not, proceed to step S14;

[0039] Step S6: Generate a random number between 0 and 1;

[0040] Step S7: Determine whether the random number is less than the exploration rate ε. If so, select the action a with the maximum Q value. If not, select a random integer from 1 to 4 as the action a.

[0041] Step S8: Based on action a in step S7, obtain the next state s' according to the current state s;

[0042] Step S9: Calculate the reward r(s,a);

[0043] Step S10: Update the Q value based on the reward in step S9.

[0044] Step S11: based on the updated Q value of step S10, update the Q value table;

[0045] Step S12: Based on the Q value table updated in step S11, set s=s';

[0046] Step S13: Repeat steps S5 to S12 until s is in the terminal state;

[0047] Step S14: Current training set number = current training set number + 1;

[0048] Step S15: End.

[0049] A path planning method for a service robot with a self-repairing function, the planning method comprising the following steps:

[0050] Step 1: Use the semantic mapping module to integrate environmental information into the 3D semantic map, encode images and text, and store them;

[0051] Step 2: Use the object recognition module to detect and classify objects, and accept text input to provide recognition cues;

[0052] Step 3: Input the classification and text recognition in step 2 into the semantic mapping module and output the determined environmental information;

[0053] Step 4: Use the speech recognition module to parse the semantic information in the natural language command;

[0054] Step 5: The parsed command is input into the task planning module, where it is converted into an LLM large language model;

[0055] Step 6: Input the LLM large language model in step 5 and the environmental information in step 3 into the path planner;

[0056] Step 7: The self-healing RL algorithm based on Q-Learning in the path planner performs plan planning and condition judgment, and outputs an executable path planning solution.

[0057] The beneficial effects of the present invention are:

[0058] The present invention improves the performance of the robot in complex environments by generalizing the changes and stimulating the adaptability of each model.

[0059] The present invention is used to recover from a failure and complete a task.

[0060] The present invention provides more powerful technical support and promotes the further development and popularization of robot technology.

[0061] The present invention adopts a concise representation method of topological map to not only simplify the description of complex environment, but also greatly reduce the required training time.

[0062] Experimental results show that the present invention performs well in two different service environments, and can efficiently learn and generate paths with an accuracy rate of up to 95%. This framework enhances the adaptability of the path planning system to long-term route changes, which is crucial for the long-term operation of the service robot. This framework is used to solve the long-term adaptation problem of the service robot. Long-term changes in the environment may cause the current path to be blocked, affecting the performance of the path planning module. Therefore, it is key to design a path planning module that can be adaptively adjusted. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is an example environment of the present invention and its topology map.

[0064] Figure 2 Schematic diagrams of robot layouts of the present invention, wherein (a) is a schematic diagram of hotel layouts, and (b) is a schematic diagram of nursing home layouts.

[0065] Figure 3 Schematic diagram of the robot layout topology map of the present invention, wherein (a) is the topology map of the hotel layout and (b) is the topology map of the nursing home layout.

[0066] Figure 4 It is a schematic diagram of the training index of the hotel layout destination 'D8' of the present invention.

[0067] Figure 5 It is a line graph of the success rate of hotel layout of the present invention.

[0068] Figure 6 It is a histogram of the number of successful hotel layouts of the present invention.

[0069] Figure 7 It is a histogram of the number of successful layouts of the nursing home of the present invention.

[0070] Figure 8 It is a schematic diagram comparing the scores of the China Service Robot Competition of the present invention.

[0071] Fig. 9 These are the 21 skill functions used in the present invention.

[0072] Fig.10 It is a system block diagram of the present invention.

[0073] Fig.11 is a flow chart of the algorithm used by the path planner of the present invention.

[0074] Fig.12 It is the algorithm program diagram of the present invention. DETAILED DESCRIPTION

[0075] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0076] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0077] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0078] The following is attached to this application specification Figure 1-12 , the technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0079] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0080] Implementation Method 1

[0081] Mobile service robots, as a type of robot that works collaboratively with other robots and humans in a dynamic environment, face the challenges of complex and changing service environments. Such environments are usually large and unknown, requiring robots to operate continuously and efficiently. Static path planning systems perform well in stable environments, but the dynamic nature of service environments means that paths may change over time, old routes may be blocked, and new routes may be generated. Therefore, artificial intelligence-based path planning systems become the key to solving this problem.

[0082] This embodiment provides a service robot path planning system with self-repair function, wherein the system uses an RL framework with self-repair function;

[0083] The RL framework with self-repair function specifically includes a speech recognition module, an object recognition module, a task planning module, a semantic mapping module and a path planner;

[0084] The object recognition module performs object detection and classification and can simultaneously accept text input to provide recognition prompts, thereby enhancing the robot's perception ability in the environment;

[0085] The semantic mapping module integrates environmental information into a 3D semantic map, performs image and text encoding, and provides rich semantic information for the robot's task planning with self-repair capabilities;

[0086] The speech recognition module performs speech activity detection and converts natural language commands into an executable format to achieve accurate understanding of natural language commands;

[0087] The task planning module forms an LLM large language model through GPT;

[0088] The path planner combines environmental information with the LLM large language model based on the self-repairing RL algorithm of Q-Learning to output an executable path planning solution.

[0089] Furthermore, the training parameters of the RL framework with self-repair function include state, action and reward;

[0090] States are all the finite configurations that the agent can have in the environment; actions are the set of all possible actions that the agent can take to move from one state s to another state s′; reward is feedback for the last action taken by the agent; it can be a positive reward or a negative penalty; this reinforces the agent to take actions that lead to positive feedback and avoid negative reward actions; Q-Learning agents train to achieve the best Q-value; Q-value is defined as action a in state s, estimating how good action A is in state S;

[0091] NewQ(s,a)=Q(s,a)+α[R(s,a)+γmaxQ′(s′,a′)-Q(s,a)] (1)

[0092] The Q-learning agent uses the Bellman equation to update the Q-value; it is shown in (1), where α is the learning rate, a hyperparameter used to control the learning ability of the agent; R(s,a) is the reward for the state-action pair (s,a); Q(s,a) is the current Q-value obtained from the Q-table; maxQ′(s′,a′) is the maximum expected future reward based on the new state s′ and all possible actions; γ is the discount rate, a hyperparameter used to control the weight of future rewards; another important factor to consider is the exploration and exploitation strategy, which determines the success of the agent training process; ∈ is a hyperparameter used to control whether the agent explores or exploits; during training, the agent chooses an action based on the Q-value, or randomly selects an action based on the value of ∈;

[0093] Specifically, the state is represented by s∈S, where s is the current state of the agent, i.e., a subset of all possible states of the agent; in the work, S is the set of all nodes in the topological map; at any given time, the agent, i.e., the robot, will be in one of the nodes, i.e., a certain position in the environment;

[0094] Specifically, the action is represented by a∈A, where a is the action taken by the agent to reach the next state; since the agent is a mobile robot, A is the set of all basic movements of the robot on a two-dimensional plane, A={left, right, forward, backward};

[0095] The reward is specifically given to the agent based on the state-action feedback; the reward function of the proposed path planner is shown in formula (2); when the desired goal g is reached, the highest positive reward is given, where g∈G and A small negative reward is given when any known or visited location is reached; the highest negative reward is given when an unknown state U is reached; an unknown location is any location in the environment that is not on the topological map.

[0096] Furthermore, the reward function of the path planner is:

[0097]

[0098] Among them, r(s,a) is the reward of the state-action pair (s,a); s is the current state of the agent; g is the desired goal; U is the unknown state; S is the set of all nodes in the topological map; G is the set of target nodes that we hope to reach.

[0099] Furthermore, reinforcement learning is performed based on the training parameters of the RL framework with self-repair function, specifically,

[0100] Training is performed in episodes. In each episode, the agent will take multiple discrete steps. In each step, an action causes the agent's state to change. In the work, the state is considered to be a different physical location in the environment, and the action is the basic movement of the robot. For such a transition, a reward will be given and the agent updates the Q value according to the Bellman equation. An episode will terminate based on the agent's current state s. (3) describes all the termination conditions. The goal of the reinforcement learning agent is to get more rewards in each episode.

[0101] During training, the agent uses the next-state table and the Q-table to learn to reach the desired goal efficiently; the next-state table is a lookup table used by the agent to transition states given the current state-action pair; the table contains all states as row elements and all actions as column elements; the components of the table are the next state based on the state-action pair; the Q-table is used to update the Q-value for a specific state-action pair; the table is used to evaluate the Bellman equation.

[0102] Furthermore, the termination condition of the current state s is,

[0103]

[0104] Among them, g is the desired goal; U is the unknown state; total states is the variable of the number of steps to reach all states; Otherwise is the continued action.

[0105] Furthermore, the self-repair function of the RL framework is implemented through a self-recovery mechanism to achieve recovery M1 including insufficient information. Specifically, in the case of insufficient information (M1), the missing information used for planning will be supplemented by the common sense possessed by the planning module (for example, the food may be in the kitchen or the dining room) and additional information obtained by talking with people (for example, asking where the apples are); specifically, two recovery functions are implemented in the GPSR system; for the case where the location name (for example, dining table) is not included in the command (or conversation with a person), the system first utilizes the LLM-based planner to infer location candidates from the command and plans to visit these locations; in the case where the operator or LLM output references a location name that is not defined in the robot system, the robot asks the operator to rephrase the location name and uses the LLM to extract it from the operator's response.

[0106] Furthermore, the self-repair function of the RL framework is implemented through a self-recovery mechanism, including recovery M2 of incorrect plan generation, specifically, in the case of incorrect plan generation (M2), solutions are developed for command recognition and plan generation; regarding command recognition, a promptable self-repairing speech recognition module (such as Whisper) can be improved by updating prompts; for plan generation, prompts of the LLM-based planner are updated based on human feedback obtained by the system after completing the original plan to confirm task completion; if the task is not evaluated as completed, the plan is regenerated using the updated prompts.

[0107] Furthermore, the self-repair function of the RL framework is implemented through a self-recovery mechanism, including recovery M3 of plan execution failure. Specifically, in the case of plan execution failure (M3), the failure can be recovered through skills and plans; for skill-level recovery, two functions are developed; one is to retry skill execution in the plan (for example, retry navigation skills), and the other is to re-plan alternative skill sequences using the following prompt template instead of executing the original skills; recovery at the plan level is performed when the task is considered failed based on human feedback after the entire plan is executed, similar to the solution to the second failure mode (M2); in this case, the prompts of the LLM (language-based model) planner will be updated based on the feedback, and the entire plan will be regenerated and executed; for example, this occurs when the wrong object is identified for a specified object in the object recognition skill; after completing the plan, the system will ask the operator to provide more information about the object, especially the name and color; the prompts of the object recognition module will be updated and the task plan will be regenerated.

[0108] The self-repairing RL algorithm of the Q-Learning of the path planner specifically includes the following steps:

[0109] Step S1: Initialize the Q value table (i.e., Q_Table) and the next state table Next_State_Table;

[0110] Step S2: Initialize the hyperparameters learning rate α, discount factor γ, exploration rate ε and total number of training episodes episode_count;

[0111] Step S3: Determine whether the current training set episode_count ≥ episode, if yes, proceed to step S4, if no, proceed to step S15;

[0112] Step S4: Randomly select an initial state s;

[0113] Step S5: Determine whether the initial state s is the terminal state, if so, proceed to step S6, if not, proceed to step S14;

[0114] Step S6: Generate a random number between 0 and 1;

[0115] Step S7: Determine whether the random number is less than the exploration rate ε. If so, select the action a with the maximum Q value. If not, select a random integer from 1 to 4 as the action a.

[0116] Step S8: Based on action a in step S7, obtain the next state s' according to the current state s;

[0117] Step S9: Calculate the reward r(s,a);

[0118] Step S10: Update the Q value based on the reward in step S9.

[0119] Step S11: based on the updated Q value of step S10, update the Q value table;

[0120] Step S12: Based on the Q value table updated in step S11, set s=s';

[0121] Step S13: Repeat steps S5 to S12 until s is in the terminal state;

[0122] Step S14: Current training set number = current training set number + 1;

[0123] Step S15: End.

[0124] Implementation Method 2

[0125] This embodiment provides a path planning method for a service robot with a self-repairing function, and the planning method comprises the following steps:

[0126] Step 1: Use the semantic mapping module to integrate environmental information into the 3D semantic map, encode images and text, and store them;

[0127] Step 2: Use a speech recognition module to parse the natural language command based on the semantic information in step 1;

[0128] Step 3: Use the object recognition module to detect and classify objects, and accept text input to provide recognition hints;

[0129] Step 4: For the object in step 2, input the command parsed in step 1 into the task planning module, convert it into an executable format in the task planning module, and combine the command with the execution skill function to realize the automated execution of the task.

[0130] Specifically, the path planning module is trained and tested in two different service environments: one environment has a hotel layout with 8 destinations, and the other environment has a nursing home layout with 28 destinations. Figure 2 Two environments with destinations and paths are shown. Figure 3 The topology graphs for both environments are shown. A new node “U” is added for the agent to fully learn the environment. This node represents the unknown state of the agent. Therefore, the topology graph used for training will be a complete ternary tree with all leaf nodes being unknown nodes. This allows the agent to stop taking actions when it transits to an unknown state. The training hyperparameters selected based on the experimental trials are listed in Table 1. All parameters for all environments are the same except for the training rounds. There are more nodes in the nursing home layout, so there are more training rounds.

[0131] General Purpose Service Robot (GPSR): The concept of GPSR was introduced by Walker et al. in , where robots are expected to perform various tasks given by humans in a natural way (e.g., verbal communication). Based on the concept of GPSR, the RoboCup Alliance uses GPSR as a performance test for GPSR tasks. GPSR tasks are conducted in a real home environment, and the robot is expected to perform the tasks given verbally by the operator (referee) as perfectly as possible within the specified time. These tasks are randomly generated using a command generator. Since the rules are updated every year, the rules are adapted to the China Service Robot Competition in this paper.

[0132] Robotic base models: Base models are a set of models trained on large-scale, extensive datasets that can be adapted to a variety of downstream tasks, including large language models (LLMs), vision-language models (VLMs), and audio-language models (ALMs). A key feature of base models is their generalization and adaptability, thanks to pre-training on large and diverse datasets (often collected from the Internet). In particular, several base models, including Whisper, GPT, CLIP, and Detic, are hintable; they can improve performance by adding text descriptions (called hints) to the input, which are related to context such as detailed instructions and environmental information.

[0133] In the field of robotics, base models are used as modules for perception and planning. For perception, VLMs such as CLIP, Detic, and SAM are used for object and environment perception. Similarly, ALMs such as Whisper and AudioCLIP are used for speech and sound recognition. In addition, some robotic systems use LLMs as task planning modules with self-healing capabilities. LLMs are expected to handle the ambiguity of natural language and convert it into a machine-interpretable representation, reasoning about missing information in commands. For example, SayCan uses LLMs to generate plans based on natural language instructions (such as "I spilled my drink, can you help?"). As a variant, Code as Policies generates Python code (including calls to external perception modules) and executes these codes, while Obinata et al. proposed using LLMs to generate state machines.

[0134] The closest setting and system is by Obinata et al., who proposed a solution for the GPSR task using the base model in recognition and planning. While using LLMs for planning and VLMs for object detection like them, they further leverage modules based on the base model for speech recognition and semantic mapping with self-healing capabilities and surpass their performance in the GPSR task of the China Service Robot Competition. In addition, typical failure cases are discussed and a novel self-recovery mechanism is introduced in the base model-based robotic system.

[0135] Self-recovery robot system: In the field of robotics, the importance of the concept of self-recovery is emphasized and implemented in the motion planning of multi-legged robots and the mechanical design of aerial robots. The aim is to realize self-recovery and self-recovery task planning in GPSR systems under the framework of the prompt basic model.

[0136] Some parallel robotics research using foundational models provides solutions to deal with plan execution failures. For example, DoReMi proposed to detect skill execution failures through VLMs and re-plan when the skill fails, but it was only tested in a simulator. FindThis proposed to resolve ambiguity in object recognition through human-computer dialogue. Ren et al. proposed a framework to interactively ask humans for help if the uncertainty of the appropriate plan is high. In contrast, the system of presents a complete GPSR system in a real-world home environment, which can be prompted and has the ability to autonomously handle multiple types of failures.

[0137] Three failure modes of GPSR system:

[0138] Insufficient information (M1): In a home environment, a robot must perform tasks in a dynamic environment; for example, the locations of objects and people change frequently. In addition, it is not feasible to register all information about the environment (e.g., the name, category, and location of objects or people) into the system in advance. Even if the system has sufficient reasoning ability or recognition of human intentions, the lack of information about the environment prevents the system from immediately generating a correct plan. For example, the information required for planning may be missing in many ways, such as "I lost my watch. Can you help me find it?" (a case where even humans do not know the location of the object) or "Can you get me a cup?" (a case where people think it should be there, but it is not clearly stated in the command).

[0139] Plan Generation Errors (M2): Even if the system has enough information to complete the task (i.e., there is no insufficient information problem), the current systems described in Section 3 cannot perfectly perform the tasks. For example, the robot may pick up noise in the verbal commands and transcribe them incorrectly, resulting in the generation of incorrect plans. In addition, incorrect plans may be generated due to the planner's reasoning performance or lack of common sense. Assume that the planner simply extracts verbs in the order they appear in the command and makes them into executable skill sequences, and the command is "Can you get me an apple?", the plan may start with "bring the apple to the operator", which is a serious mistake. Instead, the ideal plan is "go to the location of the apple (estimate if necessary)", "find it", "grab it", and then "bring it back to the operator".

[0140] Plan Execution Failure (M3): Even if the plan is generated correctly with sufficient environmental information, the robotic system may fail to execute the skill in the environment. This failure mode is inevitable due to imperfect skill execution and is an intrinsic feature of real-world systems. For example, the robot may calculate the wrong manipulation pose of an object and fail to grasp the object. Failure to find the object or wrong front face are also considered execution failures. It is important to note that execution failures not only occur in the case of hardware execution errors, but can also be attributed to the service environment (i.e., the object does not exist in the hotel).

[0141] 1. Hotel layout training

[0142] 1) Single-target training: First, the RL agent is trained to reach a destination to fine-tune the hyperparameters and determine how many training rounds are needed to fully converge. D8 is selected for training in the hotel layout. Figure 4 A plot showing the training metrics for all episodes. The training success rate estimates the level of convergence of the agent throughout the training episodes. For 500 training episodes, the training success rate reached 79%.

[0143] 2) Multi-objective training: Usually, RL problems are defined as single-objective problems. The RL agent is trained to converge to one goal of maximum reward. However, for path planning problems, the robot needs to reach multiple destinations autonomously. The Q-learning agent for path planning can be trained to plan paths to multiple destinations by changing the Q-table and the next state table to n dimensions (n ​​is the total number of objectives). The training process is similar to single-objective training, where all destinations are trained sequentially. The success rate of all training rounds is shown in the figure below. Figure 5 As shown, the success rate and number of successes for each destination are Figure 6 For all goals, the agents achieved an average success rate of 86.6%.

[0144] 2. Nursing home layout training

[0145] A multi-objective Q-learning agent was configured based on the hospital layout environment. The agent was trained for 3,000 rounds per destination. Compared with the hotel layout, as more nodes are covered, more rounds are required for each destination. The training success rate for each destination is shown in Figure 7 For all goals, the agent achieved an average success rate of 70.4%.

[0146] 3. Testing

[0147] After successfully training the RL-based path planning module agents in two different environments, the two agents were tested. The test evaluated all the paths generated given a [source location, destination location pair. The source location can be any destination node, and the source node can be any other destination node. The test was conducted with all the combinations of source and destination nodes as the destination node. Table 2 gives a summary of the comparison of all the key testing and training parameters in the two environments. From the key points in the table, despite the large differences in the total number of nodes and destinations in the two environments, the Q-learning agent with the same hyperparameter settings can effectively plan paths in a shorter training time. The accuracy in both environments exceeded 95%. This proves that the proposed Q-Learning-based path planning module has been successfully modeled and tested in both environments.

[0148] 4. GPSR fault recovery test

[0149] 1) Experimental setup: In order to verify whether the system can recover from each failure mode by utilizing the proposed system, the names of objects and people and their locations are not provided before the task starts (a map with location names is provided). According to the purpose of the experiment, the commands used for testing are manually created instead of using a command generator, and it is expected that it will be very challenging to complete all the commands in Part III using the original system. Table 4 represents the prepared commands. The check mark (√) in the table indicates that the command and its settings have the characteristics of the corresponding failure mode.

[0150] 2) Results: For all tested commands, the self-recovery prompt mechanism successfully resolved the failure.

[0151] In the case of the 5th command, the robot asked Ashley the location of the apples but received no response, thus potentially causing the system to stop due to lack of information. However, the developed system overcame this potential failure point by seeking the general knowledge of the LLM at this stage (asking for the location of “apples”). In the case of the 6th command, the robot had difficulty generating a plan because the instruction contained a phrase that was difficult to transcribe (“get the apples from the staircase-shaped shelf”). The developed system overcame this failure point by asking the operator to rephrase the command. In the case of the 7th command, execution failure in the find the person stage was a possible failure point. The system recovered from it by replanning.

[0152] Table 1 Hotel layout and nursing home layout parameters

[0153] parameter Hotel Layout Nursing home layout α 0.9 0.9 β 0.9 0.9 ε 0.9 0.9

[0154] Table 2 Multi-target testing and training parameters

[0155] parameter Hotel Layout Nursing home layout Target quantity 8 28 Number of training samples / targets 500 3000 Average training time / goal 2.4s 48s Average training success rate 86.60％ 70.40％ Number of test cases 64 784 Accuracy 98.43％ 95.57％

[0156] Table 3 China Service Robot Competition Team Results

[0157] GPSR Overall Ranking RCJ2023 First Place First Place RC2023 Second Place Third Place

[0158] Table 4 Test command and self-recovery prompt process effect table

[0159]

[0160] Implementation Method 3

[0161] This embodiment provides a path planning method for a service robot with a self-repairing function. The planning method uses the system described in the first embodiment. The planning method includes the following steps:

[0162] Step 1: Use the semantic mapping module to integrate environmental information into the 3D semantic map, encode images and text, and store them;

[0163] Step 2: Use the object recognition module to detect and classify objects, and accept text input to provide recognition cues;

[0164] Step 3: Input the classification and text recognition in step 2 into the semantic mapping module and output the determined environmental information;

[0165] Step 4: Use the speech recognition module to parse the semantic information in the natural language command;

[0166] Step 5: The parsed command is input into the task planning module, where it is converted into an LLM large language model;

[0167] Step 6: Input the LLM large language model in step 5 and the environmental information in step 3 into the path planner;

[0168] Step 7: The self-healing RL algorithm based on Q-Learning in the path planner performs plan planning and condition judgment, and outputs an executable path planning solution.

[0169] Furthermore, the training parameters of the RL framework with self-repair function include state, action and reward;

[0170] States are all the finite configurations that the agent can have in the environment; actions are the set of all possible actions that the agent can take to move from one state s to another state s′; reward is feedback for the last action taken by the agent; it can be a positive reward or a negative penalty; this reinforces the agent to take actions that lead to positive feedback and avoid negative reward actions; Q-Learning agents train to achieve the best Q-value; Q-value is defined as action a in state s, estimating how good action A is in state S;

[0171] NewQ(s,a)=Q(s,a)+α[R(s,a)+γmaxQ′(s′,a′)-Q(s,a)] (1)

[0172] The Q-learning agent uses the Bellman equation to update the Q-value; it is shown in (1), where α is the learning rate, a hyperparameter used to control the learning ability of the agent; R(s,a) is the reward for the state-action pair (s,a); Q(s,a) is the current Q-value obtained from the Q-table; maxQ′(s′,a′) is the maximum expected future reward based on the new state s′ and all possible actions; γ is the discount rate, a hyperparameter used to control the weight of future rewards; another important factor to consider is the exploration and exploitation strategy, which determines the success of the agent training process; ∈ is a hyperparameter used to control whether the agent explores or exploits; during training, the agent chooses an action based on the Q-value, or randomly selects an action based on the value of ∈;

[0173] Specifically, the state is represented by s∈S, where s is the current state of the agent, i.e., a subset of all possible states of the agent; in the work, S is the set of all nodes in the topological map; at any given time, the agent, i.e., the robot, will be in one of the nodes, i.e., a certain position in the environment;

[0174] Specifically, the action is represented by a∈A, where a is the action taken by the agent to reach the next state; since the agent is a mobile robot, A is the set of all basic movements of the robot on a two-dimensional plane, A={left, right, forward, backward};

[0175] The reward is specifically given to the agent based on the state-action feedback; the reward function of the proposed path planner is shown in formula (2); when the desired goal g is reached, the highest positive reward is given, where g∈G and A small negative reward is given when any known or visited location is reached; the highest negative reward is given when an unknown state U is reached; an unknown location is any location in the environment that is not on the topological map.

[0176] Furthermore, the reward function of the path planner is:

[0177]

[0178] Among them, r(s,a) is the reward of the state-action pair (s,a); s is the current state of the agent; g is the desired goal; U is the unknown state; S is the set of all nodes in the topological map; G is the set of target nodes that we hope to reach.

[0179] Furthermore, reinforcement learning is performed based on the training parameters of the RL framework with self-repair function, specifically,

[0180] Training is performed in episodes. In each episode, the agent will take multiple discrete steps. In each step, an action causes the agent's state to change. In the work, the state is considered to be a different physical location in the environment, and the action is the basic movement of the robot. For such a transition, a reward will be given and the agent updates the Q value according to the Bellman equation. An episode will terminate based on the agent's current state s. (3) describes all the termination conditions. The goal of the reinforcement learning agent is to get more rewards in each episode.

[0181] During training, the agent uses the next-state table and the Q-table to learn to reach the desired goal efficiently; the next-state table is a lookup table used by the agent to transition states given the current state-action pair; the table contains all states as row elements and all actions as column elements; the components of the table are the next state based on the state-action pair; the Q-table is used to update the Q-value for a specific state-action pair; the table is used to evaluate the Bellman equation.

[0182] Furthermore, the termination condition of the current state s is,

[0183]

[0184] Among them, g is the desired goal; U is the unknown state; total states is the variable of the number of steps to reach all states; Otherwise is the continued action.

[0185] Furthermore, the self-repair function of the RL framework is implemented through a self-recovery mechanism to achieve recovery M1 including insufficient information. Specifically, in the case of insufficient information (M1), the missing information used for planning will be supplemented by the common sense possessed by the planning module (for example, the food may be in the kitchen or the dining room) and additional information obtained by talking with people (for example, asking where the apples are); specifically, two recovery functions are implemented in the GPSR system; for the case where the location name (for example, dining table) is not included in the command (or conversation with a person), the system first utilizes the LLM-based planner to infer location candidates from the command and plans to visit these locations; in the case where the operator or LLM output references a location name that is not defined in the robot system, the robot asks the operator to rephrase the location name and uses the LLM to extract it from the operator's response.

[0186] Furthermore, the self-repair function of the RL framework is implemented through a self-recovery mechanism, including recovery M2 of incorrect plan generation, specifically, in the case of incorrect plan generation (M2), solutions are developed for command recognition and plan generation; regarding command recognition, a promptable self-repairing speech recognition module (such as Whisper) can be improved by updating prompts; for plan generation, prompts of the LLM-based planner are updated based on human feedback obtained by the system after completing the original plan to confirm task completion; if the task is not evaluated as completed, the plan is regenerated using the updated prompts.

[0187] Furthermore, the self-repair function of the RL framework is implemented through a self-recovery mechanism, including recovery M3 of plan execution failure. Specifically, in the case of plan execution failure (M3), the failure can be recovered through skills and plans; for skill-level recovery, two functions are developed; one is to retry skill execution in the plan (for example, retry navigation skills), and the other is to re-plan alternative skill sequences using the following prompt template instead of executing the original skills; recovery at the plan level is performed when the task is considered failed based on human feedback after the entire plan is executed, similar to the solution to the second failure mode (M2); in this case, the prompts of the LLM (language-based model) planner will be updated based on the feedback, and the entire plan will be regenerated and executed; for example, this occurs when the wrong object is identified for a specified object in the object recognition skill; after completing the plan, the system will ask the operator to provide more information about the object, especially the name and color; the prompts of the object recognition module will be updated and the task plan will be regenerated.

[0188] The self-repairing RL algorithm of the Q-Learning of the path planner specifically includes the following steps:

[0189] Step S1: Initialize the Q value table (i.e., Q_Table) and the next state table Next_State_Table;

[0190] Step S2: Initialize the hyperparameters learning rate α, discount factor γ, exploration rate ε and total number of training episodes episode_count;

[0191] Step S3: Determine whether the current training set episode_count ≥ episode, if yes, proceed to step S4, if no, proceed to step S15;

[0192] Step S4: Randomly select an initial state s;

[0193] Step S5: Determine whether the initial state s is the terminal state, if so, proceed to step S6, if not, proceed to step S14;

[0194] Step S6: Generate a random number between 0 and 1;

[0195] Step S7: Determine whether the random number is less than the exploration rate ε. If so, select the action a with the maximum Q value. If not, select a random integer from 1 to 4 as the action a.

[0196] Step S8: Based on action a in step S7, obtain the next state s' according to the current state s;

[0197] Step S9: Calculate the reward r(s,a);

[0198] Step S10: Update the Q value based on the reward in step S9.

[0199] Step S11: based on the updated Q value of step S10, update the Q value table;

[0200] Step S12: Based on the Q value table updated in step S11, set s=s';

[0201] Step S13: Repeat steps S5 to S12 until s is in the terminal state;

[0202] Step S14: Current training set number = current training set number + 1;

[0203] Step S15: End.

[0204] The above-mentioned embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above-mentioned embodiments, or replace some of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A service robot path planning system with self-repair function, characterized in that: The system uses an RL framework with self-healing capabilities; The RL framework with self-repair function specifically includes a speech recognition module, an object recognition module, a task planning module, a semantic mapping module and a path planner; The object recognition module performs object detection and classification and can simultaneously accept text input to provide recognition prompts, thereby enhancing the robot's perception ability in the environment; The semantic mapping module integrates environmental information into a 3D semantic map, performs image and text encoding, and provides rich semantic information for the robot's task planning with self-repair capabilities; The speech recognition module performs speech activity detection and converts natural language commands into an executable format to achieve accurate understanding of natural language commands; The task planning module forms an LLM large language model through GPT; The path planner combines environmental information with the LLM large language model based on the self-repairing RL algorithm of Q-Learning to output an executable path planning solution.

2. A service robot path planning system with self-repair function according to claim 1, characterized in that: The training parameters of the path planner include states, actions and rewards; Specifically, the state is represented by s∈S, where s is the current state of the agent, i.e., a subset of all possible states of the agent; in the work, S is the set of all nodes in the topological map; at any given time, the agent, i.e., the robot, will be in one of the nodes, i.e., a certain position in the environment; Specifically, the action is represented by a∈A, where a is the action taken by the agent to reach the next state; since the agent is a mobile robot, A is the set of all basic movements of the robot on a two-dimensional plane, A={left, right, forward, backward}; Specifically, the reward is given to the agent based on the state-action feedback; when the desired goal g is reached, the highest positive reward is given, where g∈G and A small negative reward is given when any known or visited location is reached; the highest negative reward is given when an unknown state U is reached; an unknown location is any location in the environment that is not on the topological map.

3. A service robot path planning system with self-repair function according to claim 2, characterized in that: The reward function of the path planner is, Among them, r(s,a) is the reward of the state-action pair (s,a); s is the current state of the agent; g is the desired goal; U is the unknown state; S is the set of all nodes in the topological map; G is the set of target nodes that we hope to reach.

4. A service robot path planning system with self-repair function according to claim 2, characterized in that: Reinforcement learning based on the training parameters of the RL framework with self-healing capabilities, specifically, Training is performed in rounds. In each round, the agent will take multiple discrete steps. In each step, an action causes the agent's state to change. For such a transition, a reward will be given, and the agent updates the Q value according to the Bellman equation. An round will terminate based on the agent's current state s. The goal of the reinforcement learning agent is to get more rewards in each round. During training, the agent uses the next-state table and the Q-table to learn to reach the desired goal efficiently; The next-state table is a lookup table used by the agent to transition to a state given the current state-action pair; the table contains all states as row elements and all actions as column elements; the components of the table are the next state based on the state-action pair; the Q-table is used to update the Q-value for a particular state-action pair; this table is used to evaluate the Bellman equation.

5. A service robot path planning system with self-repair function according to claim 4, characterized in that: The termination condition of the robot's current action state s is, Among them, g is the desired goal; U is the unknown state; total states is the variable of the number of steps to reach all states; Otherwise is the continued action.

6. A service robot path planning system with self-repair function according to claim 5, characterized in that: The self-repair function of the RL framework implements the recovery M1 including insufficient information through a self-recovery mechanism. Specifically, in the case of insufficient information, the missing information used for planning will be supplemented by the common sense possessed by the planning module and the additional information obtained by talking with people.

7. A service robot path planning system with self-repair function according to claim 6, characterized in that: The self-repair function of the RL framework is implemented through a self-recovery mechanism including recovery M2 of incorrect plan generation, specifically, in the case of incorrect plan generation, a solution is developed for command recognition and plan generation.

8. A service robot path planning system with self-repair function according to claim 7, characterized in that: The self-repair function of the RL framework is implemented through a self-recovery mechanism, including recovery M3 of plan execution failure. Specifically, in the case of plan execution failure, the failure is recovered through skills and plans.

9. A service robot path planning system with self-repair function according to claim 8, characterized in that: The self-repairing RL algorithm of the Q-Learning of the path planner specifically includes the following steps: Step S1: Initialize the Q value table and the next state table Next_State_Table; Step S2: Initialize the hyperparameters learning rate α, discount factor γ, exploration rate ε and total number of training episodes episode_count; Step S3: Determine whether the current training set episode_count ≥ episode, if yes, proceed to step S4, if no, proceed to step S15; Step S4: Randomly select an initial state s; Step S5: Determine whether the initial state s is the terminal state, if so, proceed to step S6, if not, proceed to step S14; Step S6: Generate a random number between 0 and 1; Step S7: Determine whether the random number is less than the exploration rate ε. If so, select the action a with the maximum Q value. If not, select a random integer from 1 to 4 as the action a. Step S8: Based on action a in step S7, obtain the next state s' according to the current state s; Step S9: Calculate the reward r(s,a); Step S10: Update the Q value based on the reward in step S9. Step S11: based on the updated Q value of step S10, update the Q value table; Step S12: Based on the Q value table updated in step S11, set s=s'; Step S13: Repeat steps S5 to S12 until s is in the terminal state; Step S14: Current training set number = current training set number + 1; Step S15: End.

10. A path planning method for a service robot with self-repair function, characterized in that: The planning method comprises the following steps: Step 1: Use the semantic mapping module to integrate environmental information into the 3D semantic map, encode images and text, and store them; Step 2: Use the object recognition module to detect and classify objects, and accept text input to provide recognition cues; Step 3: Input the classification and text recognition in step 2 into the semantic mapping module and output the determined environmental information; Step 4: Use the speech recognition module to parse the semantic information in the natural language command; Step 5: The parsed command is input into the task planning module, where it is converted into an LLM large language model; Step 6: Input the LLM large language model in step 5 and the environmental information in step 3 into the path planner; Step 7: The self-healing RL algorithm based on Q-Learning in the path planner performs solution planning and condition judgment, and outputs an executable path planning solution.

Images