Autonomous planning method based on large language model

Abstract

This invention provides an autonomous planning method based on a large language model, applicable to the field of artificial intelligence. The method includes: using a large language model to perform reasoning analysis on target environment data and user instructions to obtain a set of executable actions for the robotic arm, state change rules for the robotic arm and environmental objects in the target environment data during the execution of the executable actions, and state definition information for the robotic arm and environmental objects at the start and end of executing user instructions; generating a sequence of actions to be executed by the robotic arm based on the set of executable actions, state change rules, and state definition information; and, during the execution of the sequence of actions to be executed by the robotic arm, for each action to be executed, repeatedly performing the following operations: determining the control instructions for the current execution stage based on the updated environment data and robotic arm state data obtained after the previous execution stage, so as to control the robotic arm to implement user instructions based on the control instructions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to an autonomous planning method based on a large language model. Background Technology

[0002] In industrial production, office automation, and other scenarios, robotic arms need to efficiently and accurately complete tasks such as grasping and transporting. The operating environment is constrained by spatial layout limitations and the variety of objects to be manipulated. Therefore, the feasibility of planned movements is crucial to ensuring smooth operation of robotic arms in various scenarios.

[0003] In the process of realizing the concept of this invention, it was found that at least the following problems exist in the related technology: the motion planning method of the robotic arm in the related technology has the technical problem of low feasibility of the planned action due to insufficient consideration of the prerequisites. Summary of the Invention

[0004] In view of the above problems, the present invention provides an autonomous planning method, apparatus, device, medium and program product based on a large language model.

[0005] According to one aspect of the present invention, an autonomous planning method based on a large language model is provided, comprising: using a large language model to perform reasoning analysis on target environment data and user instructions to obtain a set of executable actions of a robotic arm, state change rules of the robotic arm and environmental objects in the target environment data during the execution of the set of executable actions, and state definition information of the robotic arm and environmental objects at the start and end of the execution of user instructions, wherein the state change rules and state definition information both include structured state information, and the state information is obtained by transforming or reasoning from environmental details indicated by the target environment data; generating a sequence of actions to be executed by the robotic arm based on the set of executable actions, the state change rules, and the state definition information; the sequence of actions to be executed includes multiple actions to be executed, each action to be executed having multiple execution stages; during the execution of the sequence of actions to be executed by the robotic arm, for each action to be executed, repeating the following operations until the sequence of actions to be executed is completed; determining the control instructions for the current execution stage based on the updated environment data after the previous execution stage and the state data of the robotic arm, so as to control the robotic arm to implement user instructions based on the control instructions.

[0006] According to an embodiment of the present invention, based on the updated environment data obtained after the previous execution stage and the robotic arm state data, the control instructions for the current execution stage are determined. This includes, in response to the current execution stage being the j-th execution stage of the i-th action to be executed in the sequence of actions to be executed, using a motion planner, determining a target decomposition action based on the updated environment data and robotic arm state data obtained in the j-th execution stage, where i and j are integers greater than or equal to 1, and when i=1 and j=1, the updated environment data is obtained based on the target environment data; and generating the control instructions for the current execution stage based on the target decomposition action.

[0007] According to an embodiment of the present invention, the motion planner is trained through the following operations: inputting sample execution actions, sample environment data, and sample robotic arm state data into the initial motion planner to obtain sample decomposed actions; simulating the sample decomposed actions using a preset simulation environment to obtain simulation results, wherein the preset simulation environment is obtained based on the structural parameters of the robotic arm and the environmental parameters of the environment; determining the reward value of the sample decomposed actions based on the simulation results and the reward function; and adjusting the parameters of the initial motion planner based on the reward value to obtain the motion planner.

[0008] According to an embodiment of the present invention, the motion planner includes a time-dependent encoder; the time-dependent encoder causes the target decomposed motion to be generated based on the historical decomposed motion of the historical execution phase; the execution phase of the historical decomposed motion precedes the target decomposed motion; the target decomposed motion includes sub-decomposed motions of at least one part of the robotic arm.

[0009] According to an embodiment of the present invention, a large language model is used to perform reasoning analysis on target environment data and user instructions to obtain a set of executable actions of the robotic arm, state change rules of the robotic arm and environmental objects in the target environment data during the execution of the set of executable actions, and state definition information of the robotic arm and environmental objects at the start and end of the execution of user instructions. This includes: using the large language model to perform the following operations: obtaining a set of executable actions based on at least one executable action determined by the user instructions and the constraints of each executable action determined by the target environment data during execution; reasoning about the states of the robotic arm and environmental objects after the execution of each executable action based on the initial state of the environmental objects included in the target environment data and the constraints of each executable action, thus obtaining state change rules; reasoning about the target state of the robotic arm and environmental objects when the user instructions are completed based on the initial state of the environmental objects and the user instructions; and obtaining state definition information based on the initial state of the environmental objects and the target state of the robotic arm and environmental objects when the user instructions are completed. The state definition information and the state information of the robotic arm and environmental objects included in the state change rules are both described using structured language.

[0010] According to an embodiment of the present invention, the target environment data is obtained by: collecting environmental data based on a camera installed at a preset position to obtain initial environment data, wherein the preset position includes a preset position in the environment and / or a preset position on a robotic arm; identifying environmental objects from the initial environment data to obtain target environment data, wherein the target environment data includes environmental objects included in the environment and the initial state of the environmental objects.

[0011] According to an embodiment of the present invention, a sequence of actions to be performed by a robotic arm is generated based on a set of executable actions, state change rules, and state definition information. This includes: parsing the set of executable actions, state change rules, and state definition information using a preset parser to obtain the sequence of actions to be performed; wherein the preset parser is used to generate a relational space between states and actions based on the set of executable actions, state change rules, and state definition information; and is used to determine the sequence of actions to be performed that matches the state definition information from the relational space based on a state space search algorithm.

[0012] According to embodiments of the present invention, the preset parser, motion planner, and large language model all subscribe to the target topic; the autonomous planning method based on the large language model further includes: in response to the camera publishing target environment data or updating environment data to the target topic, the large language model obtains target environment data or updates environment data from the target topic; in response to the large language model publishing a set of executable actions, state change rules, and state definition information to the target topic, the preset parser obtains the set of executable actions, state change rules, and state definition information from the target topic; in response to the preset parser publishing a sequence of actions to be executed to the target topic, the motion planner obtains the sequence of actions to be executed from the target topic.

[0013] According to an embodiment of the present invention, before generating the sequence of actions to be executed by the robotic arm based on the set of executable actions, state change rules, and state definition information, the method further includes: performing format verification on the set of executable actions, state change rules, and state definition information based on preset format rules to obtain a first verification result; performing execution logic verification on the set of executable actions, state change rules, and state definition information to obtain a second verification result; and, if both the first verification result and the second verification result indicate that there is no abnormality, performing the operation of generating the sequence of actions to be executed by the robotic arm based on the set of executable actions, state change rules, and state definition information.

[0014] According to an embodiment of the present invention, the autonomous planning method based on a large language model further includes: using a preset simulation environment to simulate the execution of a sequence of actions to be executed, and obtaining the execution result of at least one action to be executed included in the sequence of actions to be executed; matching the execution result of at least one action to be executed with state change rules respectively, and obtaining matching results; determining the execution evaluation of at least one execution result based on the matching results and the execution result of at least one action to be executed; and performing action correction on the action to be executed corresponding to the execution result that represents an execution anomaly in the execution evaluation.

[0015] Another aspect of the present invention provides an autonomous planning device based on a large language model, comprising: a reasoning module, used to perform reasoning analysis on target environment data and user instructions using a large language model to obtain a set of executable actions of a robotic arm, state change rules of the robotic arm and environmental objects in the target environment data during the execution of the set of executable actions, and state definition information of the robotic arm and environmental objects at the start and end of the execution of user instructions, wherein the state change rules and state definition information both include structured state information, and the state information is obtained by transforming or reasoning from environmental details indicated by the target environment data; a generation module, used to generate a sequence of actions to be executed by the robotic arm based on the set of executable actions, the state change rules, and the state definition information; and an execution module, used to determine control instructions matching each execution stage based on the updated environment data and robotic arm state data obtained at each execution stage of the actions to be executed in the sequence of actions to be executed by the robotic arm, so as to control the robotic arm to implement user instructions based on the control instructions.

[0016] Another aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0017] Another aspect of the present invention provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0018] Another aspect of the present invention provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0019] According to embodiments of the present invention, a pre-trained language model is used to understand the environmental scene and perform task reasoning based on target environment data and user instructions. The initial and target states of the robotic arm and environmental objects are defined, and the set of executable actions and the state change rules of each object are analyzed. Abstract task requirements and complex environmental information are transformed into structured data that machines can understand, thereby reducing the difficulty of describing multimodal data in natural language. Based on the above information, a sequence of actions to be executed is planned, and during execution, control instructions are dynamically generated based on the real-time updated environmental data and robotic arm state data at each execution stage. This enables immediate adjustment of the next path point and rapid adaptation to dynamic environmental changes. Therefore, it at least partially solves the technical problem of low feasibility of planned actions, supports timely response to environmental changes when the robotic arm performs complex long-term tasks, and improves the feasibility, accuracy, and environmental adaptability of robotic arm motion planning. Attached Figure Description

[0020] The above-mentioned contents, as well as other objects, features and advantages of the present invention, will become clearer from the following description of embodiments of the present invention with reference to the accompanying drawings.

[0021] Figure 1 The diagram illustrates application scenarios of an autonomous planning method, apparatus, device, medium, and program product based on a large language model according to embodiments of the present invention.

[0022] Figure 2 A flowchart of an autonomous planning method based on a large language model according to an embodiment of the present invention is shown.

[0023] Figure 3 A schematic diagram of an autonomous planning method based on a large language model according to an embodiment of the present invention is shown.

[0024] Figure 4 A schematic diagram of an autonomous planning method based on a large language model according to another embodiment of the present invention is shown.

[0025] Figure 5 A schematic diagram of a training initial motion planner according to an embodiment of the present invention is shown.

[0026] Figure 6 A schematic diagram of an application motion planner according to an embodiment of the present invention is shown.

[0027] Figure 7 A structural block diagram of an autonomous planning device based on a large language model according to an embodiment of the present invention is shown.

[0028] Figure 8 A block diagram of an electronic device suitable for implementing an autonomous planning method based on a large language model, according to an embodiment of the present invention, is shown. Detailed Implementation

[0029] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0031] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0032] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0033] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0034] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this invention offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0035] The research revealed that directly using large language models for robotic arm motion planning may lead to illusions and "thought loops" due to insufficient reasoning ability for complex problems. Furthermore, in real-time dynamic environments with complex constraints, the planned actions generated by large language models still lack feasibility because they do not consider sufficient preconditions. In addition, real-world environmental feedback is often multimodal, including images and audio, which are difficult to describe in natural language, limiting the effectiveness of large language models in handling such scenarios.

[0036] In view of this, embodiments of the present invention provide an autonomous planning method based on a large language model, comprising: using a large language model to perform reasoning analysis on target environment data and user instructions to obtain a set of executable actions of the robotic arm, state change rules of the robotic arm and environmental objects in the target environment data during the execution of the set of executable actions, and state definition information of the robotic arm and environmental objects at the start and end of the execution of user instructions, wherein the state change rules and state definition information both include structured state information, and the state information is obtained by transforming or reasoning from the environmental details indicated by the target environment data; generating a sequence of actions to be executed by the robotic arm based on the set of executable actions, the state change rules, and the state definition information, wherein the sequence of actions to be executed includes multiple actions to be executed, and each action to be executed has multiple execution stages; during the execution of the sequence of actions to be executed by the robotic arm, for each action to be executed, repeating the following operations until the sequence of actions to be executed is completed; determining the control instructions for the current execution stage based on the updated environment data and robotic arm state data obtained after the previous execution stage, so as to control the robotic arm to implement user instructions based on the control instructions.

[0037] Figure 1 The diagram illustrates application scenarios of an autonomous planning method, apparatus, device, medium, and program product based on a large language model according to embodiments of the present invention.

[0038] like Figure 1As shown, application scenario 100 according to this embodiment may include a first robotic arm 101, a second robotic arm 102, a camera 103, a terminal device 104, a network 105, and a server 106. The network 105 serves as a medium for providing a communication link between the first robotic arm 101, the second robotic arm 102, the camera 103, and the server 106. The network 105 may include various connection types, such as wired and / or wireless communication links, etc. The first robotic arm 101, the second robotic arm 102, the camera 103, and the terminal device 104 can interact with the server 106 through the network 105 to receive or send messages, etc.

[0039] The first robotic arm 101 and the second robotic arm 102 can be equipped with controllers, sensors, etc. The controller can be connected to the server 106 to control the rotation of the robotic arm's motors, so that at least one part of the robotic arm, such as a joint or gripper, can perform actions that match the control commands. The sensors can collect the robotic arm status data of the first robotic arm 101 and the second robotic arm 102.

[0040] Camera 103 can collect environmental data to obtain initial environmental data. There is no limit to the number of cameras 103; multiple cameras can be used. Furthermore, there is no limitation on the mounting location of each camera 103; it can be mounted on the environment and / or on a robotic arm.

[0041] Terminal device 104 can be various electronic devices with a display screen and web browsing support, including but not limited to smartphones, tablets, laptops, and desktop computers. Users can use terminal device 104 to interact with server 106 via network 105 to receive or send messages, etc. Various communication client applications can be installed on terminal device 104, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0042] Server 106 can be used to perform reasoning analysis on target environment data and user instructions using a large language model to obtain the set of executable actions of the first robotic arm 101 and the second robotic arm 102, the state change rules of the first robotic arm 101 and the second robotic arm 102 with the environment objects in the target environment data during the execution of the set of executable actions, and the state definition information of the first robotic arm 101 and the second robotic arm 102 with the environment objects when they start executing user instructions and when they finish executing them. Based on the set of executable actions, the state change rules, and the state definition information, server 106 generates a sequence of actions to be executed by the first robotic arm 101 and the second robotic arm 102. Based on the updated environment data and the robotic arm state data of the first robotic arm 101 and the second robotic arm 102 obtained at each execution stage of the actions to be executed in the sequence of actions to be executed, server 106 determines the control instructions that match each execution stage in sequence, so as to control the first robotic arm 101 and the second robotic arm 102 to implement user instructions.

[0043] It should be noted that the autonomous planning method based on a large language model provided in this embodiment of the invention can generally be executed by server 106. Correspondingly, the autonomous planning device based on a large language model provided in this embodiment of the invention can generally be located in server 106. The autonomous planning method based on a large language model provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 106 and can communicate with the first robotic arm 101, the second robotic arm 102, the camera 103, the terminal device 104, and / or server 106. Correspondingly, the autonomous planning device based on a large language model provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 106 and can communicate with the first robotic arm 101, the second robotic arm 102, the camera 103, the terminal device 104, and / or server 106. Alternatively, the autonomous planning method based on a large language model provided in this embodiment of the invention can also be executed by the first robotic arm 101, the second robotic arm 102, the camera 103, and the terminal device 104, or by other terminal devices different from the first robotic arm 101, the second robotic arm 102, the camera 103, and the terminal device 104. Correspondingly, the autonomous planning method based on a large language model provided in this embodiment of the invention can also be set in the first robotic arm 101, the second robotic arm 102, the camera 103, and the terminal device 104, or in other terminal devices different from the first robotic arm 101, the second robotic arm 102, the camera 103, and the terminal device 104.

[0044] It should be understood that Figure 1The number of the first robotic arm, second robotic arm, camera, terminal device, network, and server shown is merely illustrative. Depending on the implementation requirements, any number of first robotic arms, second robotic arms, cameras, networks, and servers can be included.

[0045] The following will be based on Figure 1 The described scene, through Figures 2-6 The autonomous planning method based on a large language model, according to the embodiments of the invention, is described in detail.

[0046] Figure 2 A flowchart of an autonomous planning method based on a large language model according to an embodiment of the present invention is shown.

[0047] like Figure 2 As shown, the method includes operations S210 to S230.

[0048] During operation S210, the large language model is used to perform reasoning analysis on the target environment data and user instructions to obtain the set of executable actions of the robotic arm, the state change rules of the robotic arm and the environment objects in the target environment data during the execution of the set of executable actions, and the state definition information of the robotic arm and the environment objects when they start executing user instructions and when they finish executing them. The state change rules and state definition information both include structured state information, which is obtained by transforming or reasoning from the environmental details indicated by the target environment data.

[0049] In operation S220, based on the set of executable actions, state change rules, and state definition information, a sequence of actions to be executed by the robotic arm is generated. The sequence of actions to be executed includes multiple actions to be executed, and each action to be executed has multiple execution stages.

[0050] During operation S230, in the process of the robotic arm executing the sequence of actions to be executed, for each action to be executed, the following operations are repeated until the sequence of actions to be executed is completed: based on the updated environment data and robotic arm status data obtained after the previous execution stage, the control instructions for the current execution stage are determined so as to control the robotic arm to implement the user's instructions based on the control instructions.

[0051] There are no restrictions on the implementation method of the large language model; it can be a large language model (LLM).

[0052] There are no restrictions on the form of user commands; natural language commands can be given to humans in the form of voice or text.

[0053] Target environment data can include identification and status information of multiple environmental objects. Identification information can include the object's identifier, such as its name and category. Status information can include the object's location. Environmental objects can be items within the environment; for example, in an office setting, environmental objects could be a computer, mouse, and desk.

[0054] The target environment data can be acquired using multimodal sensing. There are no restrictions on the method of acquiring the target environment data; it can be obtained by acquiring environmental data through multiple cameras and then performing target detection on the obtained initial environment data.

[0055] By using large language models to perform reasoning analysis on target environment data and user instructions, it is possible to use large language models to perform scene understanding and logical reasoning on multimodal data. This allows us to obtain the set of executable actions of the robotic arm, the state change rules of the robotic arm and environmental objects in the target environment data during the execution of the set of executable actions, and the state definition information of the robotic arm and environmental objects when they start executing user instructions and when they finish executing them. This at least partially makes up for the difficulty of describing multimodal feedback in natural language.

[0056] The format of the set of executable actions of the robotic arm, the state change rules of the robotic arm and the environmental objects in the target environment data during the execution of the set of executable actions, and the state definition information of the robotic arm and the environmental objects when they start executing user instructions and when they finish executing them are not limited. They can be in the format of Planning Domain Definition Language (PDDL).

[0057] When the output of a pre-trained language model uses PDDL, its output can be a PDDL planning file, including a domain file and a problem definition file. The set of executable actions of the robotic arm and the state change rules of the robotic arm and environmental objects in the target environment data during the execution of the executable actions can be the content of the domain file. The state definition information of the robotic arm and environmental objects at the start and end of executing user instructions can be the content of the problem definition file.

[0058] The set of executable actions can include the executable actions of the robotic arm and the constraints on the execution of these actions. There can be one or more robotic arms. When there are multiple robotic arms, the set of executable actions can include the individual executable actions of each robotic arm and the constraints on those actions.

[0059] For example, if the executable action is to grasp an object, the constraint condition for this executable action can be that before grasping, the robotic arm must be next to the object and the object must not have been grasped.

[0060] The state change rules can include information on the state changes of the environment and the robot arm after each executable action is performed. This state change information can include the changes in the state of the environment or the robot arm before and after the executable action is performed.

[0061] For example, after being grasped, the state of object A changes from not being grasped to being held by the robotic arm.

[0062] State definition information can be used to define specific scenarios, that is, to define the initial state of a task and the target state required by the task. For example, it can include the initial state information of the robotic arm and environmental objects before the user command begins to be executed, and the target state information of the robotic arm and environmental objects when the user command is completed.

[0063] When representing the state information of the robotic arm and environmental objects included in the state change rules and state definition information, a structured representation can be used. For example, predicates can be used to represent state information.

[0064] Specifically, for example, the initial state information includes robotic arm A being at position C (at armA, positionC) and object B being at position D (at objectB, positionD). The target state information includes robotic arm A being at position E (at armA, positionE), holding object B, and object B being at position E (at objectB, positionE).

[0065] Based on the set of executable actions and the definition of state change rules for the robotic arm, as well as the initial and target states of the environment defined by state definition information, a complete sequence of actions can be obtained, that is, the sequence of actions to be executed by the robotic arm.

[0066] In some embodiments, a multi-arm motion planner based on reinforcement learning can be used to execute the action sequence sequentially according to the target of the action sequence to be executed and to update the environmental data and the state data of the robotic arm in order to complete the task.

[0067] During the execution of a sequence of actions to be executed, since the sequence may include multiple actions to be executed, and the robotic arm can decompose each action to be executed during the execution of each action, an action to be executed will be divided into decomposed actions executed in multiple execution stages.

[0068] During each execution phase, updated environmental data and robotic arm status data from the previous execution phase will be acquired. Based on this data, the decomposed actions of the current execution phase will be further planned, thereby achieving real-time planning of the decomposed actions.

[0069] According to embodiments of the present invention, a pre-trained language model is used to understand the environmental scene and perform task reasoning based on target environment data and user instructions. The initial and target states of the robotic arm and environmental objects are defined, and the set of executable actions and the state change rules of each object are analyzed. Abstract task requirements and complex environmental information are transformed into structured data that machines can understand, thereby reducing the difficulty of describing multimodal data in natural language. Based on the above information, a sequence of actions to be executed is planned, and during execution, control instructions are dynamically generated based on the real-time updated environmental data and robotic arm state data at each execution stage. This enables immediate adjustment of the next path point and rapid adaptation to dynamic environmental changes. Therefore, it at least partially solves the technical problem of low feasibility of planned actions, supports timely response to environmental changes when the robotic arm performs complex long-term tasks, and improves the feasibility, accuracy, and environmental adaptability of robotic arm motion planning.

[0070] According to an embodiment of the present invention, the target environmental data is obtained in the following manner.

[0071] Environmental data is collected using cameras installed at preset locations to obtain initial environmental data. The preset locations include preset locations in the environment and / or preset locations on the robotic arm. Environmental object identification is performed on the initial environmental data to obtain target environmental data, which includes environmental objects included in the environment and the initial state of the environmental objects.

[0072] There is no limitation on the camera model; any depth camera can be used. Depth cameras can output various types of visual data, such as color image data and depth map data, which is the position data of objects in the image. The position data can be three-dimensional coordinate data in the world coordinate system.

[0073] Cameras can be mounted in the environment and / or on the robotic arm, and hand-eye calibration can be performed to ensure that the coordinates of the target object perceived by the camera can be transferred from the camera coordinate system to the world coordinate system, so as to facilitate the robotic arm's positioning and grasping. For example, in an office environment, two cameras can be placed in the robotic arm's end effector gripper and in the office scene, respectively, and hand-eye calibration can be performed.

[0074] The specific method used for hand-eye calibration is not limited; an eye-in-hand calibration method can be used. For example, for a camera located at the end effector of a robotic arm, an eye-in-hand calibration method can be used. The calibration board (April tag) is printed and placed on a table. Then, the eye-in-hand camera calibration program is started, and the robotic arm is continuously moved, accumulating 12-17 robotic arm poses. During this process, it is necessary to ensure that the camera can capture the entire calibration board. After marking all collected robotic arm poses, the transition matrix is ​​calculated using a preset calculation module to complete the calibration. Similarly, cameras located in office scenes can be calibrated using a similar process. The April tag can consist of a binary coded area enclosed by a black border.

[0075] There are no restrictions on the method for identifying environmental objects. For example, the Grounding DINO method can be used to identify and locate environmental objects, and the environmental identification information, i.e., the target environment data, can be passed to a large language model. Grounding DINO is a multimodal visual model that combines natural language understanding and object detection, and can directly locate objects in an image through text descriptions.

[0076] Because Grounding DINO uses a zero-shot open set target detection algorithm, objects in the environment can be identified and located. Grounding DINO uses bounding boxes to locate the target pixel pose, and after calibration, the camera transfer matrix can be calculated to obtain the three-dimensional coordinates in the world coordinate system.

[0077] According to embodiments of the present invention, environmental data from multiple perspectives is acquired via a camera, reducing information omissions caused by a single perspective. Furthermore, environmental object recognition transforms the original unstructured data into structured target environmental data containing object categories and initial states, reducing the processing costs of coordinate transformation and subsequent large language model inference.

[0078] According to an embodiment of the present invention, a large language model is used to perform reasoning analysis on target environment data and user instructions to obtain the set of executable actions of the robotic arm, the state change rules of the robotic arm and the environment objects in the target environment data during the execution of the set of executable actions, and the state definition information of the robotic arm and the environment objects when they start executing user instructions and when they finish executing them. The large language model can then be used to perform the following operations.

[0079] Based on at least one executable action determined by user instructions and the constraints of each executable action determined by target environment data during execution, a set of executable actions is obtained. Based on the initial state of the environment objects included in the target environment data and the constraints of each executable action, the states of the robotic arm and the environment objects after the execution of each executable action are inferred, resulting in state change rules. Based on the initial state of the environment objects and user instructions, the target state of the robotic arm and the environment objects when the user instructions are completed is inferred. Based on the initial state of the environment objects and the target state of the robotic arm and the environment objects when the user instructions are completed, state definition information is obtained. The state information of the robotic arm and the environment objects included in the state definition information and state change rules are described using structured language.

[0080] The large language model can determine multiple executable actions from a pool of candidate actions based on user commands and a pre-trained robotic arm action library. It then adds constraints to each executable action by combining user commands and target environment data.

[0081] At the same time, the large language model can deduce the update rules of state variables based on the physical logic of each executable action and the constraints between environmental objects, thereby obtaining the state change rules.

[0082] For example, based on spatial relationships in environmental data, such as object A being on table B, we can deduce the impact of an executable action on the object's pose. For instance, after a grasping action is executed, the pose of object A is updated synchronously with the gripper at the end of the robotic arm.

[0083] The large language model can combine initial observations of environmental data and the content of user instructions to deduce the target state of the robotic arm and environmental objects when the user instructions are executed. Both the target state and the initial state are represented in a structured format to obtain state definition information.

[0084] Structured languages ​​can be used to represent state information in a structured form, without limiting its specific structure. It can take the form of predicates + environment objects or robotic arms.

[0085] According to embodiments of the present invention, the state information of the robotic arm and the environmental objects is described by structured language, which more effectively makes up for the defects of ambiguity and vagueness in the description of multimodal feedback in natural language. This makes the set of executable actions, state change rules, target state and state definition information generated based on user instructions and target environment data clear and computable, thereby improving the accuracy and executableness of the subsequently generated action sequences.

[0086] According to an embodiment of the present invention, a sequence of actions to be performed by a robotic arm is generated based on a set of executable actions, state change rules, and state definition information, which may include the following operations.

[0087] A preset parser is used to parse the set of executable actions, state change rules, and state definition information to obtain a sequence of actions to be executed. The preset parser is used to generate a relation space between states and actions based on the set of executable actions, state change rules, and state definition information. It is also used to determine the sequence of actions to be executed that matches the state definition information from the relation space based on a state space search algorithm.

[0088] After reading the set of executable actions, state change rules, and state definition information, the pre-defined parser can construct a relationship space between states and actions. Specifically, state information can be used as nodes, and edges can represent the relationships between nodes, which can be determined by the state changes after the executable actions are executed. Specifically, the relationships between nodes can be established if the executable actions satisfy constraints during execution and satisfy the state change rules after execution.

[0089] Subsequently, the pre-defined parser can invoke a state space search algorithm, starting from the initial state and ending with the target state, to traverse the associated edges in the relation space and filter out action sequences that can gradually transform the initial state into the target state. Simultaneously, it verifies the logical consistency of each intermediate state through state change rules, thereby obtaining a sequence of actions to be executed that matches the state definition information.

[0090] There are no restrictions on the state space search algorithm; it can be a heuristic algorithm, such as the A algorithm (A*) or the Greedy Best-First Search (GBFS).

[0091] In some embodiments, the preset parser may also be a parsing tool, such as a Fast-Downward parser.

[0092] According to embodiments of the present invention, a preset parser parses the set of executable actions, state change rules, and state definition information, fully considering various prerequisites to generate a state-action relationship space. This reduces the problem of insufficient feasibility of planned actions due to a lack of sufficient prerequisite verification. Furthermore, by relying on a state space search algorithm to calculate the sequence of actions to be executed that matches the state definition information, the efficiency and accuracy of action sequence generation are improved.

[0093] According to an embodiment of the present invention, the preset parser, the motion planner, and the large language model all subscribe to the target topic; the autonomous planning method based on the large language model may further include the following operations.

[0094] In response to the camera publishing or updating environmental data to the target topic, the large language model obtains the target environmental data or updates the environmental data from the target topic; in response to the large language model publishing the set of executable actions, state change rules, and state definition information to the target topic, the preset parser obtains the set of executable actions, state change rules, and state definition information from the target topic; in response to the preset parser publishing the sequence of actions to be executed to the target topic, the motion planner obtains the sequence of actions to be executed from the target topic.

[0095] Initial environment data, target environment data, updated environment data, set of executable actions, state change rules, state definition information, sequence of actions to be executed, and control execution data can all be published and subscribed to in the form of topics to enable communication between the camera, server, and robotic arm.

[0096] Specifically, the camera can be launched as a node to publish target detection topic information, including initial environment data, target environment data, or updated environment data. A large language model node is launched and subscribes to initial environment data, target environment data, or updated environment data from the camera node, as well as natural language commands given by humans. It also provides a set of executable actions, state change rules, and state definition information, which are saved locally. A preset parser, acting as a parser node, reads the local set of executable actions, state change rules, and state definition information, parses and plans specific executable action sequences, and publishes them in the topic. The motion planner subscribes to the executable action sequences and target environment data, plans the motion for the robotic arm, and publishes control commands to the robotic arm. The robotic arm executes low-level control code according to the control commands to complete the task. If the environment changes, such as the target object moving, the camera will feed back or update the environmental data to the motion planner to update the target pose in a timely manner, ensuring that the robotic arm's next planned action responds to environmental changes in real time.

[0097] According to embodiments of the present invention, by publishing and subscribing to relevant information in the form of topics, not only can communication between the camera, server, and robotic arm be realized, but also timely notification and acquisition of useful information can be achieved.

[0098] According to an embodiment of the present invention, before generating the sequence of actions to be performed by the robotic arm based on the set of executable actions, state change rules, and state definition information, the following operations may also be included.

[0099] Based on preset format rules, the executable action set, state change rules, and state definition information are format-verified to obtain a first verification result; the executable action set, state change rules, and state definition information are execution logic-verified to obtain a second verification result; if both the first and second verification results indicate that there are no abnormalities, the operation of generating the robot arm's action sequence based on the executable action set, state change rules, and state definition information is performed.

[0100] Since syntax errors may occur in the set of executable actions, state change rules, and state definition information, which may cause the preset parser to plan actions inaccurately or fail to parse feasible action sequences, an external checker self-correction mechanism has been added to increase the probability that the action sequences output by the preset parser are logically correct and feasible.

[0101] Based on preset formatting rules, it can check whether the format of the set of executable actions, state change rules, and state definition information is correct, such as basic syntax checks, predicate definition checks, and action definition checks. Basic syntax checks can include checks such as bracket correspondence checks and keyword spelling checks. Predicate definition checks can include whether the state information includes a predicate and whether the predicate is in the correct position. Action definition checks can check whether the action definition format meets the formatting requirements.

[0102] It can also perform execution logic verification on the set of executable actions, state change rules, and state definition information, such as whether there are unrealizable actions or contradictory state change rules.

[0103] Figure 3 A schematic diagram of an autonomous planning method based on a large language model according to an embodiment of the present invention is shown.

[0104] like Figure 3 As shown, the human natural language instruction is: Sort all boxes in ascending order of their serial numbers and place them on the assembly line, remembering not to touch anything else.

[0105] User commands and environmental state observations can be input into a large language model (LLM) for scene understanding and reasoning, thereby outputting a set of executable actions, state change rules, and state definition information. Environmental state observations can include state information of scene objects, such as target environment data or target environment data and robotic arm state data. Target environment data can include state information of multiple environmental objects and the robotic arm, such as the position and shape of target object A, the pose of robotic arm A, and the gripper.

[0106] A semantic checker can be used to perform format verification and execution logic verification on the above information, resulting in a first verification result and a second verification result. If both the first and second verification results indicate the absence of anomalies, a sequence of actions to be executed is generated based on a preset parser. This sequence of actions is then converted into control commands for the robotic arm. Simultaneously, the robotic arm's state data after executing the decomposed actions in each execution stage is acquired. Furthermore, updated environmental data is obtained through multimodal perception, and the environmental state observations for the next execution stage are summarized.

[0107] If it is determined that either the first verification result or the second verification result indicates an anomaly, the two verification results can be sent to the large language model, and the large language model can output the corrected set of executable actions, state change rules, and state definition information based on the two verification results.

[0108] According to embodiments of the present invention, by pre-checking the set of executable actions, state change rules, and state definition information, the problem of low availability of the sequence of actions to be executed output by the preset parser can be reduced.

[0109] According to embodiments of the present invention, the autonomous planning method based on a large language model may further include the following operations.

[0110] Using a preset simulation environment, the sequence of actions to be executed is simulated and executed to obtain the execution result of at least one action included in the sequence of actions to be executed; the execution result of at least one action to be executed is matched with state change rules to obtain matching results; based on the matching results and the execution result of at least one action to be executed, the execution evaluation of at least one execution result is determined; and the action to be executed corresponding to the execution result that represents an execution anomaly in the execution evaluation is corrected.

[0111] In a preset simulation environment, each action to be executed can be simulated and executed sequentially according to time steps, and the execution result corresponding to each action can be output, including the robotic arm status data after execution, whether abnormal events such as collisions or joint over-limits are triggered, and the time taken to complete the action, which can reflect the actual execution effect of the action.

[0112] Then, the execution result of each action to be executed can be compared with the preset state change rules to verify whether the actual post-action state after the action is executed is consistent with the expected post-action state in the rules. At the same time, it is checked whether the action execution process meets the constraints, thereby obtaining the matching result.

[0113] In constructing an execution evaluation based on the matching and execution results, different levels of evaluation can be assigned based on both. For example, actions that match successfully and have no abnormal events can be assigned a positive evaluation of normal execution. For partially matched actions, a slight abnormality evaluation can be given based on the magnitude of parameter deviations, such as joint angle deviations and coordinate offsets. Actions that fail to match or trigger serious abnormal events such as collisions or joint overruns can be assigned a negative evaluation of serious abnormality. Thus, the evaluation can pinpoint the type and degree of problem in the action execution.

[0114] For actions to be executed that exhibit abnormalities in performance evaluation, action correction can be initiated. During implementation, the root cause of the anomaly can be analyzed by combining simulation results and matching results. If the anomaly is caused by parameters exceeding the rule's allowable range, the action parameters can be adjusted to the rule's range. If the anomaly is caused by a conflict between the action execution order and state change rules, the execution sequence of the action can be replanned. If the anomaly is caused by a conflict with environmental constraints, such as a collision triggered by the action path, the action trajectory can be optimized based on the physical feedback of the simulation environment, such as by adding obstacle avoidance actions.

[0115] According to an embodiment of the present invention, by simulating the execution of the sequence of actions to be performed in a preset simulation environment, the execution process of the actions can be simulated before the actual operation of the robotic arm, unreasonable actions to be performed can be identified in advance, reducing equipment damage or task interruption caused by actual execution failures and reducing trial and error costs.

[0116] Figure 4 A schematic diagram of an autonomous planning method based on a large language model according to another embodiment of the present invention is shown.

[0117] like Figure 4 As shown, embodiments of the present invention can be applied to cooking scenarios, object handling scenarios, open environments, etc. During the task description phase, the user can issue user commands, such as: "Find me a wooden spatula and put it in the bowl on the table." Simultaneously, the camera can be used for visual perception to obtain initial environmental data, and the Grounding DINO method can be used to identify and locate environmental targets, thereby obtaining target environmental data. At this point, because the camera can acquire comprehensive initial environmental data, even if the user does not know the location of the wooden spatula, they can still complete the user command normally.

[0118] In the reasoning and planning phase of the Large Language Model (LLM), target environment data and user instructions can be input into the LLM for scene understanding and reasoning. Combined with environmental constraints and planning, this yields the PDDL domain file and PDDL problem file. A pre-defined parser then generates the sequence of actions to be executed, including action 1, action 2, ..., action N. An external checker can be used to implement a self-correcting mechanism to examine the sequence of actions to be executed, thereby generating a new plan.

[0119] During the inspection process, the external inspector can perform pre-execution. Based on the detection results of the external inspector, feedback can be provided on the sequence of actions to be executed. Specifically, a preset simulation environment can be used to simulate the execution of the sequence of actions to be executed, obtaining the execution result of at least one action included in the sequence. The execution result of at least one action to be executed is then matched with state change rules to obtain matching results. Based on the matching results and the execution results of at least one action to be executed, an execution evaluation of at least one execution result is determined. Action correction is then performed on the action to be executed corresponding to the execution result that indicates an execution anomaly in the execution evaluation.

[0120] Once the external inspector confirms that the plan is error-free, control commands can be generated based on the sequence of actions to be executed, and the robotic arm can then perform the underlying execution of the action sequence. The application scenarios for this invention are not limited and can include industrial, office, entertainment, and cooking scenarios, among others.

[0121] According to an embodiment of the present invention, the control instructions for the current execution stage are determined based on the updated environment data obtained after the previous execution stage and the robotic arm status data, which may include the following operations.

[0122] In response to the current execution stage being the j-th execution stage of the i-th action in the sequence of actions to be executed, the motion planner is used to determine the target decomposition action based on the updated environmental data and the robotic arm state data obtained in the j-th execution stage, where i and j are integers greater than or equal to 1. When i=1 and j=1, the updated environmental data is obtained based on the target environmental data; control instructions for the current execution stage are generated based on the target decomposition action.

[0123] During the execution of the sequence of actions to be executed, the motion planner breaks down each action according to motion logic, determining multiple consecutive execution stages. When a new execution stage is reached, the motion planner initiates real-time calculations to further analyze the updated environmental data obtained after the execution of the decomposed actions in the previous execution stage, as well as the robotic arm's state data, thereby generating control instructions for the corresponding execution stage.

[0124] The updated environment data can be the updated target environment data or the updated target environment data after filtering by environment objects. For example, the updated environment data can include only environment objects that match the task object indicated by the user command.

[0125] For example, if the user command indicates a task object that is a water cup placed on the desk, then the updated environmental data can include only the status information of the water cup.

[0126] The status data of a robotic arm can include status information of multiple parts of the robotic arm, such as joint angles, end effector gripper posture, and gripper force.

[0127] Based on the motion parameters indicated by the target decomposition action, control commands for the corresponding execution stage can be generated. These commands are then used to drive the joints of the robotic arm through its controller. Simultaneously, the execution status is fed back to the motion planner in real time, providing a basis for calculating the decomposition action in the next stage.

[0128] According to embodiments of the present invention, by updating environmental data and robotic arm status data during the phased execution of the sequence of actions to be executed, dynamic changes in the environment can be captured more accurately, and target decomposition actions or path points adapted to the current scene can be dynamically determined accordingly. That is, by adjusting the next decomposition action in real time, the control commands of each execution stage are matched with the immediate environmental state. This reduces problems such as path failure and execution deviation caused by dynamic environmental changes, achieves real-time response to dynamic environmental changes, and improves the accuracy and robustness of the robotic arm's action execution in dynamic scenes.

[0129] According to an embodiment of the present invention, the motion planner is trained through the following steps.

[0130] The sample execution action, sample environment data, and sample robotic arm state data are input into the initial motion planner to obtain the sample decomposed action. The sample decomposed action is simulated using a preset simulation environment, which is obtained based on the structural parameters of the robotic arm and the environmental parameters of the environment. Based on the simulation results and the reward function, the reward value of the sample decomposed action is determined. Based on the reward value, the parameters of the initial motion planner are adjusted to obtain the motion planner.

[0131] According to an embodiment of the present invention, the motion planner includes a time-dependent encoder; the time-dependent encoder causes the target decomposed motion to be generated based on the historical decomposed motion of the historical execution phase; the execution phase of the historical decomposed motion precedes the target decomposed motion; the target decomposed motion includes sub-decomposed motions of at least one part of the robotic arm.

[0132] The motion planner may include a temporally dependent encoder, a fully connected layer, and an activation function layer. The activation function used in the activation function layer is not limited and can be a hyperbolic tangent function (tanh). The temporally dependent encoder can be a Long Short-Term Memory (LSTM) encoder.

[0133] At least one sub-part can be a joint, base, link, or end effector of the robotic arm.

[0134] Before training the initial motion planner, training samples, a preset simulation environment, a reward function, and an evaluation network can be pre-built.

[0135] Training samples can include sample execution actions, sample environment data, and sample robotic arm state data. Sample execution actions can correspond to actions to be executed, sample environment data can correspond to updated environment data, and sample robotic arm state data can correspond to robotic arm state data. The specific contents they include are similar and will not be repeated here.

[0136] The preset simulation environment can be constructed based on the actual structural parameters of the robotic arm, such as the joint range of motion, link length, and maximum load, as well as environmental physical parameters, such as the environmental objects and their positions, and the coefficient of friction of the objects.

[0137] There are no restrictions on the implementation method of the preset simulation environment; PyBULLET can be used. PyBULLET is a lightweight, efficient, open-source physics simulation tool. The PyBULLET physics engine can be used to build a model of the robotic arm, and a dataset of 1 million tasks can be constructed within PyBULLET. The target positions in the data are generated in the simulation environment, ensuring the solvability of the tasks. There are no restrictions on the computational framework used during training; the Ray framework, a distributed computing framework, can be used. Multi-process training using the Ray framework can improve training efficiency and speed.

[0138] Without limiting the reward function, positive rewards can be given to effective behaviors such as satisfying joint constraints, avoiding collisions, and moving closer to the target, while negative penalties can be given to ineffective behaviors such as exceeding joint limits, colliding, and deviating from the target, thereby quantifying the feasibility of the action and its contribution to the task.

[0139] During the initial training of the motion planner, sample execution actions, sample environmental data, and sample robotic arm state data can be used as inputs. The mean and variance of the joint motion values ​​at the current time step are calculated forward. The mean represents the predicted optimal motion direction, and the variance defines the exploration range of the motion; together, they constitute a Gaussian distribution model. By randomly sampling from this Gaussian distribution model, specific joint motion values ​​of the robotic arm, i.e., sample decomposed motions, can be obtained.

[0140] In the preset simulation environment, after the robotic arm executes the sample decomposed action, the robotic arm's state data will be updated, such as the joint angles, gripper position, and relative relationship with environmental objects. This updated result will be used as the simulation result. Simultaneously, based on the reward function and the simulation results, the reward value corresponding to the sample decomposed action will be calculated, reflecting the action's contribution to task progress.

[0141] In some embodiments, the sample decomposition actions, corresponding reward values, sample environment data, sample robotic arm state data, and simulation results can be stored in an experience replay pool. When the initial motion planner needs to be updated after several time steps, data is retrieved from the experience replay pool, and reward values ​​for multiple time steps are calculated to update the initial motion planner. Simultaneously, reward values ​​can also be used to update the evaluation network, causing the evaluation network to converge to the true value.

[0142] Therefore, based on the reward value obtained in each iteration, the parameters of the initial motion planner can be adjusted using a reinforcement learning optimization algorithm, enabling the initial motion planner to gradually output more accurate sample decomposition actions. Through continuous iteration, the Gaussian distribution of the initial motion planner will continuously converge towards the high-reward action region, thereby generating a continuous action sequence that conforms to the physical characteristics of the robotic arm, adapts to environmental constraints, and is executable.

[0143] Figure 5 A schematic diagram of a training initial motion planner according to an embodiment of the present invention is shown.

[0144] like Figure 5 As shown, the fully connected layers of the initial motion planner can be implemented using a multilayer perceptron (MLP). The LSTM encoder includes multiple memory cells. During training, the state observations of the robotic arm, i.e., the sample robotic arm state data, can be input into the initial motion planner. The initial motion planner combines this with updated environmental data to obtain the sample robotic arm action value at the current time step t, i.e., the sample decomposed action. The sample decomposition action can be executed in a preset simulation environment, and the reward value can be calculated. Then, the current time step state value is... Sample decomposition action Reward value and state value at the next time step Stored in the experience replay pool, state value for the next time step. The reward value is used as input for the next execution phase. It can be used to update the parameters of the initial motion planner to obtain the final motion planner. For example, an actor strategy can be used to update the parameters of the initial motion planner.

[0145] The current time step state value is the sample robotic arm state data and the sample environment data. The next time step state value is the updated sample robotic arm state data after the sample decomposition action is performed.

[0146] The sample robotic arm state data is not limited and can include base position, joint configuration, current end effector pose, link pose, and target end effector pose.

[0147] According to embodiments of the present invention, by inputting sample execution actions, sample environmental data, and sample robotic arm state data into an initial motion planner, it learns action decomposition logic in diverse scene samples. The output target decomposed actions are then simulated in a preset simulation environment. The reward value is determined through simulation results and a reward function to train the parameters of the initial motion planner. This quantifies the contribution of decomposed actions to task progress, forming a clearer optimization direction. The trained motion planner can then decompose executable actions into fine-grained decomposed actions that conform to the physical characteristics of the robotic arm and adapt to environmental constraints, reducing problems such as route deviation during actual execution and improving the accuracy and feasibility of action decomposition.

[0148] Figure 6 A schematic diagram of an application motion planner according to an embodiment of the present invention is shown.

[0149] like Figure 6 As shown, the initial motion planner can be trained offline on an offline training platform using a preset simulation environment and its associated dynamic model, thus obtaining the final motion planner. In practical applications, the motion planner can be used to calculate the decomposed actions for each execution stage, such as joint motion values, and the corresponding control commands and task objectives can be published in a topic. This allows the robotic arm controller to subscribe to this information from the topic and control the robotic arm actuators to execute the commands. Upon completion, the controller returns robotic arm status data, execution results, and other information.

[0150] For example, the environment includes environmental objects such as books, boxes, drawers, and remote controls. Robotic arms A and B can manipulate these environmental objects based on control commands.

[0151] In some embodiments, the trained motion planner can be tested. The test tasks are divided into two types. One is a long-term complex dynamic task, which is to command the robotic arm to find unknown objects. This type of task is a multi-step task that requires the robotic arm to continuously obtain updated environmental information by interacting with the environment, that is, to explore blind spots or closed containers. After finding the object, the subsequent search action is updated to a grasping action. The robotic arm needs to have the ability to adjust in real time according to environmental feedback.

[0152] Another type is high-frequency dynamic tasks, where the object grasped by the robotic arm can move at any time. The camera needs to provide real-time feedback of the object's pose to the motion planner to ensure that the motion planner can adjust the planning results of the next path point in real time.

[0153] The testing process can be divided into two parts: finding the mouse and placing the remote control into the box. The first part requires the robotic arm to open drawers sequentially until it finds the mouse and places it on the cabinet. The second part involves grabbing the remote control, but the remote control may be moved during the task, so the robotic arm must adjust its planned path in real time to ensure it can be grabbed. Both tasks require real-time feedback from the camera regarding environmental information.

[0154] Specifically, in test process 1, the user command could be "Please find the mouse and place it on the cabinet." The camera can provide feedback on the target environment data. This target environment data can include the identification and status information of Robot A, Robot B, the drawer, the camera, and the book. The execution of the user command could include Robot A opening drawer A, Robot A opening drawer B, and Robot B picking up the mouse, waiting to perform the action. Robot A and Robot B can be equipped with robotic arms.

[0155] Furthermore, during the process, the camera continuously acquires and updates environmental data. For example, after opening drawer D, the camera did not detect a mouse, but only scissors. However, by opening drawer F, the camera detected a mouse.

[0156] During the execution of each action to be executed, the motion planner outputs target decomposed actions at different execution stages. For example, during the execution of action 1 to be executed, decomposed actions such as opening the drawer ("starting position 1", "ending position 1"), closing the gripper ("drawer handle"), and closing the drawer ("starting position 2", "ending position 2") can be executed.

[0157] Action 2 to be performed may include decomposed actions such as opening the drawer (“starting position 3”, “ending position 4”), closing the gripper (“drawer handle”), and closing the drawer (“starting position 5”, “ending position 6”).

[0158] It should be noted that the sequence of actions to be executed also included the action of opening drawer 3 of robot B, but during the execution process, it was found that this step could be shortened and the action of picking up the mouse, which is the action to be executed, could be executed directly, thus skipping this action.

[0159] In test procedure 2, the user instruction could be: "Please grab the remote control and place it in the box on the table." Target environment data can include the identification and status information of: Robot C, Robot D, Drawer, Remote, Box, and Books. The remote control's position will change during the planning process; for example, the remote control's position may change from position 1 to position N.

[0160] The sequence of actions to be executed in test process 2 includes action 1 (robot D picking up the remote control) and action 4 (placing the remote control and waiting to be executed). When robot D picks up the remote control, it continuously changes the planned picking position to adapt to the constantly changing remote control. For example, from the planned picking position ("remote control position 1") to the planned picking position ("remote control position 2") and then to the planned picking position ("remote control position N").

[0161] During implementation, if action 1 (picking up the remote control) is difficult to complete, the sequence of actions to be executed can be modified, such as: first execute action 4 (placing the remote control).

[0162] According to the embodiments of the present invention, the above tests show that the present invention has completed the practical tests of long-term complex tasks of blind zone exploration and dynamic tasks of real-time grasping of moving objects.

[0163] Based on the above-mentioned autonomous planning method based on a large language model, this invention also provides an autonomous planning device based on a large language model. The following will combine... Figure 7 The device is described in detail.

[0164] Figure 7 A structural block diagram of an autonomous planning device based on a large language model according to an embodiment of the present invention is shown.

[0165] like Figure 7 As shown, the autonomous planning device 700 based on a large language model in this embodiment includes a reasoning module 710, a generation module 720, and an execution module 730.

[0166] The reasoning module 710 is used to perform reasoning analysis on the target environment data and user instructions using a large language model to obtain the set of executable actions of the robotic arm, the state change rules of the robotic arm and environmental objects in the target environment data during the execution of the set of executable actions, and the state definition information of the robotic arm and environmental objects at the start and end of the execution of user instructions. Both the state change rules and the state definition information include structured state information, which is obtained by transforming or reasoning from the environmental details indicated by the target environment data. In one embodiment, the reasoning module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0167] The generation module 720 is used to generate a sequence of actions to be executed by the robotic arm based on the set of executable actions, state change rules, and state definition information. The sequence of actions to be executed includes multiple actions, and each action has multiple execution stages. In one embodiment, the generation module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0168] The execution module 730 is used to repeatedly perform the following operations for each action to be executed during the execution of the sequence of actions by the robotic arm, until the sequence of actions is completed: Based on the updated environment data obtained after the previous execution stage and the robotic arm state data, the control command for the current execution stage is determined, so as to control the robotic arm to implement the user command based on the control command. In one embodiment, the execution module 730 can be used to execute the operation S230 described above, which will not be repeated here.

[0169] According to embodiments of the present invention, any plurality of modules among the inference module 710, generation module 720, and execution module 730 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the inference module 710, generation module 720, and execution module 730 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the inference module 710, generation module 720, and execution module 730 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0170] Figure 8 A block diagram of an electronic device suitable for implementing an autonomous planning method based on a large language model, according to an embodiment of the present invention, is shown.

[0171] like Figure 8As shown, an electronic device 800 according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0172] Random access memory (RAM) 803 stores various programs and data required for the operation of electronic device 800. Processor 801, read-only memory (ROM) 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present invention by executing programs in read-only memory (ROM) 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than read-only memory (ROM) 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0173] According to an embodiment of the present invention, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0174] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0175] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include one or more memories other than the read-only memory (ROM) 802 and / or random access memory (RAM) 803 described above.

[0176] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the autonomous planning method based on a large language model provided in the embodiments of the present invention.

[0177] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0178] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0179] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0180] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0182] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0183] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A self-planning method based on a large language model, characterized in that, The method includes: By using a large language model to perform reasoning analysis on target environment data and user instructions, the following results are obtained: a set of executable actions for the robotic arm, state change rules for the robotic arm and environmental objects in the target environment data during the execution of the set of executable actions, and state definition information for the robotic arm and environmental objects at the start and end of the execution of the user instructions. The state change rules and the state definition information both include structured state information, which is obtained by transforming or reasoning from environmental details indicated by the target environment data. The set of executable actions includes the executable actions of the robotic arm and the constraints of the executable actions during execution. The state change rules include the state change information of the environmental objects and the robotic arm after the robotic arm executes the executable actions. Based on the set of executable actions, the state change rules, and the state definition information, a sequence of actions to be executed by the robotic arm is generated; the sequence of actions to be executed includes multiple actions to be executed, and each action to be executed has multiple execution stages; During the execution of the sequence of actions to be performed by the robotic arm, for each action to be performed, the following operations are repeated until the sequence of actions to be performed is completed: based on the updated environment data and robotic arm status data obtained after the previous execution stage, the control command for the current execution stage is determined so as to control the robotic arm to implement the user command based on the control command; The step of generating the sequence of actions to be executed by the robotic arm based on the set of executable actions, the state change rules, and the state definition information includes: The set of executable actions, the state change rules, and the state definition information are parsed using a preset parser to obtain the sequence of actions to be executed. The preset parser is used to generate a relationship space between states and actions based on the set of executable actions, the state change rules, and the state definition information. It is also used to determine the sequence of actions to be executed that matches the state definition information from the relationship space based on a state space search algorithm.

2. The autonomous planning method according to claim 1, characterized in that, The step of determining the control commands for the current execution phase based on the updated environment data obtained after the previous execution phase and the robotic arm status data includes: In response to the current execution stage being the j-th execution stage of the i-th action in the sequence of actions to be executed, the motion planner is used to determine the target decomposed action based on the updated environmental data and the robotic arm state data obtained in the j-th execution stage, where i and j are integers greater than or equal to 1, and the updated environmental data is obtained based on the target environmental data when i=1 and j=1. The control instructions for the current execution phase are generated based on the target decomposition actions.

3. The autonomous planning method according to claim 2, characterized in that, The motion planner was trained through the following operations: The sample execution action, sample environment data, and sample robotic arm state data are input into the initial motion planner to obtain the sample decomposed action; The sample decomposition action is simulated using a preset simulation environment to obtain simulation results. The preset simulation environment is obtained based on the structural parameters of the robotic arm and the environmental parameters of the environment. Based on the simulation results and the reward function, the reward value of the sample decomposition action is determined; The parameters of the initial motion planner are adjusted based on the reward value to obtain the motion planner.

4. The autonomous planning method according to claim 2, characterized in that, The motion planner includes a time-dependent encoder; the time-dependent encoder causes the target decomposed motion to be generated based on the historical decomposed motion of the historical execution phase; the execution phase of the historical decomposed motion is prior to the target decomposed motion; the target decomposed motion includes sub-decomposed motions of at least one part of the robotic arm.

5. The autonomous planning method according to claim 1, characterized in that, The step of using a large language model to perform reasoning analysis on target environment data and user instructions to obtain the set of executable actions for the robotic arm, the state change rules of the robotic arm and environmental objects in the target environment data during the execution of the set of executable actions, and the state definition information of the robotic arm and environmental objects at the start and end of the execution of the user instructions, includes: performing the following operations using the large language model: Based on at least one executable action determined by the user instruction and the constraints of each executable action during execution determined by the target environment data, an executable action set is obtained. Based on the initial state of the environmental object and the constraints of each executable action included in the target environment data, the states of the robotic arm and the environmental object after each executable action are inferred, and the state change rules are obtained. Based on the initial state of the environmental object and the user instruction, the target state of the robotic arm and the environmental object when the user instruction is completed is deduced. The state definition information is obtained based on the initial state of the environmental object and the target state of the robotic arm and the environmental object when the user instruction is completed; The state definition information and the state change rules, including the state information of the robotic arm and the environmental objects, are described using structured language.

6. The autonomous planning method according to claim 1 or 2, characterized in that, The target environment data was obtained through the following methods: Initial environmental data is obtained by collecting environmental data using cameras installed at preset locations, where preset locations include preset locations in the environment and / or preset locations on the robotic arm. Environmental object identification is performed on the initial environmental data to obtain the target environmental data, wherein the target environmental data includes the environmental objects included in the environment and the initial state of the environmental objects.

7. The autonomous planning method according to claim 1, characterized in that, The preset parser, motion planner, and large language model all subscribe to the target topic; the method further includes: In response to the camera publishing target environment data or updated environment data to the target topic, the large language model obtains the target environment data or the updated environment data from the target topic; In response to the large language model publishing the set of executable actions, the state change rules, and the state definition information to the target topic, the preset parser obtains the set of executable actions, the state change rules, and the state definition information from the target topic; In response to the preset parser publishing a sequence of actions to be executed in the target topic, the motion planner obtains the sequence of actions to be executed from the target topic.

8. The autonomous planning method according to any one of claims 1 to 3, characterized in that, Before generating the sequence of actions to be executed by the robotic arm based on the set of executable actions, the state change rules, and the state definition information, the method further includes: Based on preset format rules, the executable action set, the state change rules, and the state definition information are format-validated to obtain a first validation result. The execution logic is verified on the set of executable actions, the state change rules, and the state definition information to obtain a second verification result. If both the first verification result and the second verification result indicate that there is no abnormality, the operation of generating the sequence of actions to be executed by the robotic arm based on the set of executable actions, the state change rules and the state definition information is performed.

9. The autonomous planning method according to any one of claims 1 to 3, characterized in that, The method further includes: Using a preset simulation environment, the sequence of actions to be executed is simulated and executed to obtain the execution result of at least one action included in the sequence of actions to be executed; The execution result of at least one action to be executed is matched with the state change rule to obtain the matching result; Based on the matching results and the execution results of at least one action to be executed, determine the execution evaluation of at least one execution result; Action correction is performed on the action to be executed corresponding to the execution result that represents an execution anomaly in the execution evaluation.

Images