Body task planning method for family old-age-assisting robot

Abstract

The invention discloses a method for planning a task of a family old-age-assisting robot, and the method comprises the following steps: S1, carrying out the task planning modeling based on a large language model, and comprehensively evaluating the task completion probability of a skill; s2, collecting all-around visual perception in the scene; s3, generating an action plan by the large language model; s4, dynamically adjusting a task plan according to a task execution result; and S5, constructing a data set. According to the family old-age assisting robot body task planning method, dynamic planning can be achieved, planning can be adjusted in time according to the actual situation after each subtask is completed, the complex planning problem in the family environment can be more effectively solved, and the task execution success rate and efficiency are improved.

Description

Technical Field

[0001] This invention belongs to the field of robot task planning technology, specifically relating to an embodied task planning method for a home-based elderly care robot. Background Technology

[0002] Embodied Task Planning for Robots refers to the process by which robots, in a physical environment, autonomously decompose high-level tasks into executable sub-tasks through perception, reasoning, decision-making, and interaction, plan specific action sequences, and ultimately complete the task in the real world. Currently, Large Language Models (LLMs) are demonstrating remarkable emerging capabilities. Leveraging their rich accumulated world knowledge and powerful reasoning abilities, they can effectively decompose abstract tasks—a process highly similar to the human reasoning pattern of systematically outlining task steps before action. Therefore, numerous research efforts on task planning have focused on Large Language Models.

[0003] Large Language Models (LLMs) are detached from the physical world, generating content without considering the impact on real-world physical processes. To address this issue, SayCan pioneered an application method of LLMs in embodied task planning, combining the semantic knowledge of LLMs with pre-trained skill value functions. This allows robots to understand and execute high-level, long-term instructions expressed in natural language, enabling them to complete complex tasks in the real world. Translated LM proposes a zero-shot planning method using pre-trained large language models (LLMs). Through techniques such as semantic translation and autoregressive trajectory correction, it decomposes high-level tasks into executable action sequences, significantly improving the executability of action plans in interactive environments. To cope with environmental changes, Inner Monologue uses natural language as the interaction interface. Through success detection, scene description, and human feedback, LLMs can adjust their plans in real time during task execution, significantly improving the robot's task completion rate and adaptability in simulated and real environments. ReAct combines the reasoning capabilities (such as reasoning chains) of large language models (LLMs) with action capabilities (such as task-specific action generation), achieving dynamic interaction in language understanding and decision-making tasks. In the field of multi-agent collaboration, Zhang et al. proposed the ReAd framework. This framework introduces a reinforcement advantage feedback mechanism to optimize the plans generated by large language models (LLMs), reduce the number of interactions with the physical environment, and effectively improve the execution efficiency and success rate of multi-agent tasks. Furthermore, Sarch et al., Wang et al., and Sharma et al. investigated methods that abstract past successful examples into a series of skills stored in a memory bank for reference during reasoning, thereby improving the success rate of planning. Singh et al., Vemprala et al., and Liang et al. conducted separate studies, choosing to use code rather than natural language as the reasoning medium, generating task planning as code based on available application programming interface (API) libraries. In addition, to correct potential illusions in task planning, many agents based on large language models have focused on multi-turn reasoning methods. For example, Socratic Models and Socratic Planner use Socratic dialogue to derive reliable plans. These methods, based on different approaches, integrate large language models into task planning with significant results. However, research from InnerMonologue and LLM-Planner shows that planners often fail to fully consider the complexity of the real-world environment and the difficulty of task execution during task planning, leading to potential failure risks in the execution phase. Furthermore, due to a lack of visual information, planned sub-tasks may become disconnected from the actual scenario, ultimately resulting in task failure. Therefore, integrating visual information into the planning or replanning process is essential during execution.This measure can significantly improve the accuracy and feasibility of mission planning, thereby enabling us to better address various challenges in the real-world environment.

[0004] Integrating visual information into task planning (or replanning) is crucial. In this process, object labels, locations, or descriptions provided by visual input offer key references for Large Language Models (LLMs) in task decomposition and execution. Through visual information, LLMs can more accurately identify target objects and obstacles in the current environment, thereby optimizing task steps or adjusting sub-task objectives. Zeng et al., Song et al., and Wu et al., in their research on embodied task planning, used object detectors to query objects in the environment and then fed this information back to the LLM, enabling it to modify unreasonable steps in the current plan. Chen et al. further considered the different names for similar objects in the same task and proposed the RoboGPT framework, effectively avoiding planning biases caused by ambiguous object references, further improving the feasibility of the replanning stage, and making the planning scheme more suitable for complex and ever-changing real-world application scenarios. However, the information provided by labels remains limited. To provide more scene information, Rana et al. proposed the SayPlan method, which uses layered 3D scene graphs to represent the environment, effectively alleviating the challenges of task planning in large, multi-layered, and multi-room scenes. Similarly, ConceptGraphs also uses 3D scene graphs to provide environmental information to large language models. Compared to SayPlan, it offers more detailed open-world object detection capabilities and presents task planning in a code-based format, which is more efficient and better suited for complex tasks. While the aforementioned task planning methods do incorporate visual information during the planning process, improving accuracy and rationality to some extent, they have limitations. They only utilize visual information in the initial task planning stage and fail to perform secondary planning based on newly acquired visual information after each sub-task. In complex and ever-changing scenarios like a home environment, dynamic factors such as furniture changes and the addition or removal of items frequently occur, making static planning methods inadequate. Dynamic planning, adjusting the plan promptly based on the actual situation after each sub-task, would more effectively solve the complex planning challenges in a home environment, improving task execution success rate and efficiency.

[0005] A crucial prerequisite for the deployment of service robots is their ability to perform complex tasks assigned by humans in typical indoor environments. Recently emerging Large Language Models (LLMs) can effectively embed rich common sense into robot task parsing and planning. For example, if a robot detects an elderly person's fall through panoramic perception, or if the elderly person informs the robot of their fall via voice, ChatGPT can directly parse and generate a series of executable end-to-end tasks: the robot first moves to the elderly person's side, then engages its wheels, extends its robotic arm to help the elderly person stand up, and finally notifies the family. Figure 1 As shown. However, since the input of LLM is usually a single text modality, it cannot perceive information in the real world and may generate actions that cannot be performed in the current scene.

[0006] The computational problem this invention aims to solve is to overcome the limitations of traditional methods in the face of dynamic environmental changes and unexpected situations, and to resolve the problem of planning being out of sync with the actual scenario due to a lack of visual information during task execution, thereby improving the accuracy and feasibility of task planning and enhancing the intelligent agent's understanding of the environment. Summary of the Invention

[0007] To address the aforementioned problems, this invention proposes an embodied task planning method for home-based elderly care robots.

[0008] The embodied task planning method for a home-based elderly care robot of the present invention includes the following steps:

[0009] S1. Based on a large language model, task planning modeling is used to comprehensively evaluate the probability of skill completion for task completion;

[0010] S2. Collect comprehensive visual perception within the scene;

[0011] S3. Generate action plans using large language models;

[0012] S4. Dynamically adjust the task plan based on the task execution results;

[0013] S5. Construct the dataset.

[0014] The model in S1 that evaluates the probability of skill contributing to task completion is as follows:

[0015]

[0016] In the formula, The probability that a skill will progress during the actual execution of an instruction. Let π be the feasibility function of skill. Let be the probability that the skill will effectively proceed to the next step while actually completing instruction i, and let e be the environmental state. For the language description of skills, cπ and c i They are all Bernoulli random variables.

[0017] The visual perception in S2 refers to panoramic images or videos at different locations within the scene. The panoramic image acquisition strategy is defined as follows:

[0018] S={(x, y, h, θ)|(x, y)∈L(λ, Ψ)}

[0019] In the formula, (x, y, h, θ) represents the position of the cruise point and the rotation angle and lifting height of the surround-view fisheye camera at that cruise point, L(λ, Ψ) represents the selection of the cruise point, λ represents the size of the map grid, and Ψ represents the area that the robot can reach.

[0020] The scene in S2 is an indoor scene. The environmental information of the indoor scene includes human behavior and the categories and lists of all objects. The environmental information is obtained through surround-view fisheye target and human behavior detection, specifically represented as follows:

[0021]

[0022] In the formula, X L For environmental information, Rd represents the deduplication operation, which is to reduce information redundancy and avoid excessively long X values. L Interferes with the performance of large language models, O(I i ), A(J i ) represent the object and behavior categories output by the surround-view fisheye image detector, respectively.

[0023] The action plan generated by the large language model in S3 includes fine-tuning samples and inference prompts. The fine-tuning samples include a list of items and actions, instructions, completed sub-goals, and a list of sub-goals. The sub-goals are generated by the large language model based on the prompts, instructions, completed sub-goals, and the list of items and actions.

[0024] The reasoning prompts require a comprehensive analysis of the instructions, completed sub-goals, and item list. First, understand the core of the instructions, and then combine them with the completed sub-goals to ensure the continuity of the plan.

[0025] The sub-objective is represented as

[0026] X a =Task-Planer(P in X L (G, I)

[0027] In the formula, X a For sub-targets, P in As a clue for reasoning, X L This is a list of actions, where I represents the instruction and G represents the sub-target.

[0028] The S4 section describes dynamically adjusting the task plan based on the task execution result. This means that when the robot fails to complete a certain sub-task during the task execution process, the task planner performs a secondary plan for that sub-task, continuously adjusting the task strategy, and repeating this process until the task is successfully completed.

[0029] The task planner's secondary planning of sub-tasks specifically refers to the robot updating environmental information through object detectors and motion recognizers, re-invoking the task planner to generate a new task plan, and resetting the sub-target index to restart execution.

[0030] The dataset in S5 includes instructions, action plans, and egocentric images. Instructions are used to clearly define the tasks that the robot needs to complete; action plans are used to provide specific execution steps; and egocentric images, by simulating the robot's perspective, help it understand the environment.

[0031] The beneficial effects of this invention are:

[0032] (1) This application integrates panoramic object and human behavior scene information into a large language model. Based on a carefully designed prompting strategy, it explores in depth the task planning method of panoramic embodied perception for home-assisted elderly robots. The aim is to lay the foundation for the implementation of home-assisted elderly robots and promote technological innovation and industrial development in the field of intelligent elderly care.

[0033] (2) The dynamic task planning algorithm of this application enables the robot to perform secondary planning based on its own state and environmental information after each sub-task, adjust the task process in real time, effectively cope with environmental changes, and improve the reliability of task planning.

[0034] (3) Drawing inspiration from the command generator model of the RoboCup Robotics Competition's General Purpose Service Robot project, a dataset suitable for task planning of home service robots in real-world scenarios is constructed. By testing the task planning method based on a lightweight large language model in a real-world environment, the effectiveness of this method is verified using the constructed dataset, thus providing strong support and basis for efficient task planning of home service robots in complex real-world scenarios. Attached Figure Description

[0035] Figure 1 It involves embedding scenario information into the Big Prophecy model to generate executable actions.

[0036] Figure 2 This is a schematic diagram of scene visual perception.

[0037] Figure 3 It is a dynamically redesigned flowchart. Detailed Implementation

[0038] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0039] The embodied task planning method for a home-based elderly care robot of the present invention includes the following steps:

[0040] S1. Based on a large language model, task planning modeling is used to comprehensively evaluate the probability of skill completion for task completion;

[0041] The system receives a natural language instruction 'i' provided by the user, which describes the task the robot should perform. Instruction 'i' may be lengthy, highly abstract, or semantically ambiguous. The system's goal is to decompose this high-level instruction into a series of executable subtasks.

[0042] Suppose the system has a known set of skills Π, where each skill π∈Π can perform a short task (e.g., "pick up a specific object"). Each skill π is accompanied by a short verbal description. (For example, "find a sponge"), used to describe the specific content of the skill.

[0043] Each skill π is also equipped with a feasibility function. This indicates that under environmental state e, the task described was successfully completed. The probability of a skill. Where:

[0044] c π It is a Bernoulli random variable:

[0045] c π =1 indicates that skill π can be successfully executed under environmental state e;

[0046] c π =0 indicates execution failure.

[0047] The environmental state 'e' encompasses environmental information about the robot's current state, including elements such as the list of items and human actions. Intuitively, This can be understood as "under environmental state e, if the robot is required to perform..." The task described, and whether it can be successfully completed.

[0048] Large Language Model (LLM) is responsible for providing That is, under environmental state e, the skill description This is the probability that user instruction i is valid and proceeds to the next step. This probability reflects the skill description. The degree of semantic matching with user instruction i.

[0049] The system focuses on the probability that a skill will progress during the actual completion of instruction i, denoted as . in:

[0050] c i It is a Bernoulli random variable:

[0051] c i =1 indicates that skill π has made progress on instruction i;

[0052] c i =0 indicates no progress has been made.

[0053] Assuming that successful skills are based on probability (That is, the probability that it is the correct skill) progresses on instruction i, while the probability of progressing on a failed skill is zero. Therefore, the probability of progress can be decomposed as:

[0054]

[0055] This breakdown is equivalent to describing the skills. Semantic matching probability under instruction i and environment e The probability of skill feasibility in the current environmental state e Multiply these factors to comprehensively assess the contribution of skills to task completion.

[0056] S2. Collect comprehensive visual perception within the scene;

[0057] like Figure 2 As shown, for any deployed indoor scene X S Human behavior and the categories and lists of all objects constitute environmental information X. L ={fall, Creditcard, Statue, Drawer, Box, ...} can be used to represent scene information, serving as a bridge for communication with LLM. To incorporate real-world feasibility constraints into task planning, the primary goal is to accurately obtain a list of human behaviors and objects in the scene, avoiding missed or false detections. L Acquisition and panoramic image X V The acquisition situation is closely related. Although the panoramic image X under the fisheye view is viewed... V It can cover the entire scene, but it still needs to collect panoramic images or videos from different locations to perceive the visual scene, and discover existing objects and human behavior through multi-angle and multi-distance shooting. V The acquisition strategy S can be defined as:

[0058] S={(x, y, h, θ)|(x, y)∈L(λ, Ψ)}

[0059] Where (x, y, h, θ) represents the position of the cruise point and the rotation angle and elevation height of the surround-view fisheye camera at that point. L(λ, Ψ) represents the selection of the cruise point, where λ represents the size of the map grid and Ψ represents the area that the robot can reach.

[0060] X L It is obtained through panoramic fisheye target and human behavior detection, and can be specifically expressed as:

[0061]

[0062] Here, Rd represents the deduplication operation, which is mainly to reduce information redundancy and avoid excessively long X values. L Interferes with LLM performance. O(I) i ), A(J i ) represent the object and behavior categories output by the surround-view fisheye image detector, respectively.

[0063] S3. Generate action plans using large language models;

[0064] Table 1 shows a sample of fine-tuned task planning hints during large model inference, as well as our designed inference hint P. in The fine-tuning sample consists of four parts: an item and an action list X. L Instruction I: Sub-objective G and sub-objective list X completed. a When no human movement is detected, X L The action part is represented by null; G is empty at the beginning; sub-target X a Based on the prompt P by the large language model in Instruction I, completed sub-objectives G, and list of items and actions X L Generation. In the reasoning prompts of the large language model, it is necessary to comprehensively analyze instruction I, completed sub-goals G, and the item list X. L First, understand the core of instruction I, then combine it with the completed sub-goal G to ensure the plan's coherence. Organize the item list X. L The objects and actions within generate action plan X. a At that time, based on list X L It carefully selects atomic actions and strictly matches them to task requirements. In short, the Task-Planner embodied task generation framework comprehensively considers X. L G, I, using a large language model to generate executable X a :

[0065] X a =Task-Planer(P in X L (G, I)

[0066]

[0067] S4. Dynamically adjust the task plan based on the task execution results;

[0068] For details on the dynamic programming process, please refer to [link / reference]. Figure 3 At time t=0, the task planner receives the instruction, generates a subtask, and assigns it to the robot for execution. However, at time t=5, the robot encounters an inability to find the "tomato" during task execution. Fortunately, by combining the current environmental information X... L The robot discovered a refrigerator. Based on this, the task planner quickly performed a secondary planning, adjusting the task direction to find tomatoes from the refrigerator. At time t=20, the robot failed to locate a recycling bin. However, by analyzing the current environmental information X... L The robot notices a "trash can". Based on this, the task planner restarts the secondary planning process, continuously adjusting the task strategy, and repeating this cycle until the task is successfully completed.

[0069] The dynamic programming algorithm, as shown in Table 2, is based on the core idea of ​​continuously perceiving the environment, planning tasks, and executing sub-goals, while dynamically adjusting the plan according to the execution results to ensure successful task completion. First, the task planner acquires the task instruction I, the observed set of objects O, and the action set A, and combines them into environmental information X. L Simultaneously, the list of completed sub-goals G is initialized. Next, the Task Planner is invoked to generate a complete task plan X. a and initialize the subtarget index k to obtain the first subtarget a. k During the loop execution, if the current sub-target a... k If the execution fails, the robot updates the environmental information X using the Object Detector and Action Recognizer. L And re-invoke the task planner to generate a new task plan X. a Simultaneously, reset subtarget index k to 0 to restart execution; if the current subtarget a k If completed successfully, it is added to the list of completed sub-goals G, and the process continues to the next sub-goal until all sub-goals are completed.

[0070]

[0071] S5. Construct the dataset.

[0072] We will construct a multimodal dataset specifically for task planning in real-world home environments and conduct field tests on laboratory robots. Our research revealed that existing multimodal datasets, such as ALFRED, are mostly collected in simulator environments, primarily used to validate the effectiveness of task planning methods. However, simulator environments differ significantly from real-world home environments. In simulators, object physical properties and lighting conditions are simplified, unlike the rich materials and complex lighting conditions of a real home. This leads to errors in the robot's judgment of grasping force and movement path when performing tasks in a real environment after training with simulator data, affecting task execution. Therefore, referencing the RoboCup GPSR competition format, we began collecting a multimodal dataset suitable for real-world home environments. This dataset contains three parts: instructions, clearly defining the tasks the robot needs to complete; action plans, providing specific execution steps; and egocentric images, simulating the robot's perspective to aid its understanding of the environment. To comprehensively demonstrate the complexity of tasks in a home environment, we categorize tasks into four types. First, there's object recognition, such as "find an apple in the bedroom," which tests the robot's ability to recognize objects in a specific environment. Second, there's question answering, like "find someone in the hallway and ask them a question," requiring the robot to complete a series of actions including finding the person, asking the question, and understanding the answer. Third, there's human tracking, such as "find someone in the hallway and ask them a question," where the robot needs to continuously track the target person while performing the task. Fourth, there's object grasping, such as "take a lemon from the table and put it on the coffee table," which places high demands on the robot's operational and spatial awareness capabilities. Through this classification and dataset construction, we aim to improve the robot's task execution capabilities and reliability in real-world home environments, laying the foundation for its widespread application.

[0073] Although the above embodiments have been shown and described, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Any changes, modifications, substitutions and variations made to the above embodiments by those skilled in the art are within the protection scope of the present invention.

Claims

1. A method for embodied task planning for a home-based elderly care robot, characterized in that, Includes the following steps: S1. Based on a large language model, task planning modeling is used to comprehensively evaluate the probability of skill completion for task completion; S2. Collect comprehensive visual perception within the scene; S3. Generate action plans using large language models; S4. Dynamically adjust the task plan based on the task execution results; S5. Construct the dataset.

2. The embodied task planning method for home-based elderly care robots according to claim 1, characterized in that, The model in S1 that evaluates the probability of skill contributing to task completion is as follows: p(c i |i,e,l π )∝p(c π |e,l π )p(l π |i,e). In the formula, p(c i |i, e, l π p(c) represents the probability that a skill will progress during the actual execution of an instruction. π |e,l π ) is the feasibility function of skill π, p(l) π |i, e) are provided by the large language model, where i is the probability that the skill will effectively proceed to the next step after actually completing instruction i, e is the environmental state, and l π For the language description of skills, c π and c i They are all Bernoulli random variables.

3. The embodied task planning method for a home-based elderly care robot according to claim 1, characterized in that, The visual perception in S2 refers to panoramic images or videos at different locations within the scene. The panoramic image acquisition strategy is defined as follows: S={(x, y, h, θ)|(x, y)∈L(λ, Ψ)} In the formula, (x, y, h, θ) represents the position of the cruise point and the lifting height and rotation angle of the surround-view fisheye camera at that cruise point, L(λ, Ψ) represents the selection of the cruise point, λ represents the size of the map grid, and Ψ represents the area that the robot can reach.

4. The embodied task planning method for a home-based elderly care robot according to claim 1, characterized in that, The scene in S2 is an indoor scene. The environmental information of the indoor scene includes human behavior and the categories and lists of all objects. The environmental information is obtained through surround-view fisheye target and human behavior detection, specifically represented as follows: In the formula, X L For environmental information, Rd represents the deduplication operation, which is to reduce information redundancy and avoid excessively long X values. L Interferes with the performance of large language models, O(I i ), A(J i ) represent the object and behavior categories output by the surround-view fisheye image detector, respectively.

5. The embodied task planning method for a home-based elderly care robot according to claim 1, characterized in that, The action plan generated by the large language model in S3 includes fine-tuning samples and inference prompts. The fine-tuning samples include a list of items and actions, instructions, completed sub-goals, and a list of sub-goals. The sub-goals are generated by the large language model based on the prompts, instructions, completed sub-goals, and the list of items and actions.

6. The embodied task planning method for a home-based elderly care robot according to claim 5, characterized in that, The reasoning prompts require a comprehensive analysis of the instructions, completed sub-goals, and item list. First, understand the core of the instructions, and then combine them with the completed sub-goals to ensure the continuity of the plan.

7. The embodied task planning method for a home-based elderly care robot according to claim 6, characterized in that, The sub-objective is represented as X a =Task-Planer(P in , XL ,G,I) In the formula, X a For sub-targets, P in As a clue for reasoning, X L This is a list of actions, where I represents the instruction and G represents the sub-target.

8. The embodied task planning method for a home-based elderly care robot according to claim 1, characterized in that, The S4 section describes dynamically adjusting the task plan based on the task execution result. This means that when the robot fails to complete a certain sub-task during the task execution process, the task planner performs a secondary plan for that sub-task, continuously adjusting the task strategy, and repeating this process until the task is successfully completed.

9. The embodied task planning method for a home-based elderly care robot according to claim 8, characterized in that, The task planner's secondary planning of sub-tasks specifically refers to the robot updating environmental information through object detectors and motion recognizers, re-invoking the task planner to generate a new task plan, and resetting the sub-target index to restart execution.

10. The embodied task planning method for a home-based elderly care robot according to claim 1, characterized in that, The dataset in S5 includes instructions, action plans, and egocentric images. Instructions are used to clearly define the tasks that the robot needs to complete; action plans are used to provide specific execution steps; and egocentric images, by simulating the robot's perspective, help it understand the environment.

Images