Task motion planning method and device of intelligent agent and intelligent agent

Abstract

The invention discloses a task motion planning method and device of an intelligent agent and the intelligent agent, and belongs to the field of intelligent agents. The task motion planning method of the intelligent agent comprises the steps of obtaining a target kinematic model between the intelligent agent and each to-be-interacted object in a task scene; based on the target task corresponding to the intelligent agent, a task sequence is generated, the task sequence comprises at least one sub-sequence arranged based on a time sequence, and the sub-sequence comprises target interaction object information, action information and first pose information between the execution mechanism and the to-be-interacted object; the target interaction object information is used for representing a target to-be-interacted object in the to-be-interacted objects; and based on the task sequence and the target kinematic model, generating a task motion plan of the intelligent agent. According to the task motion planning method of the intelligent agent, a scene with rich tasks is effectively coped with, the task completion rate of the intelligent agent is guaranteed, the flexibility of the intelligent agent control process is improved, and the use experience of a user is improved.

Description

Technical Field

[0001] The present application belongs to the field of intelligent agents, and in particular, relates to a task motion planning method, device and intelligent agent. Background Art

[0002] It is becoming increasingly important for intelligent agents to operate autonomously in task-rich environments and assist users in various daily life tasks. In related technologies, the challenges of high complexity of the contact pattern between the end effector and the object and high dimensionality of the state space in sequential mobile operation planning problems are mainly solved in a hierarchical manner. However, the above method simplifies the complexity of motion planning and can only solve simple operations. It cannot effectively solve the actual challenges in task-rich scenarios, limits the deployment of intelligent agents in actual environments, affects the flexibility of planning and the completion rate of tasks, and affects the user experience. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a task motion planning method, device and intelligent agent, which can effectively cope with task-rich scenarios, ensure the task completion rate of the intelligent agent, and improve the flexibility of the intelligent agent control process and improve the user experience.

[0004] In a first aspect, the present application provides a method for task motion planning of an intelligent agent, the method comprising:

[0005] Acquire a target kinematic model between the agent and each object to be interacted in the task scene, wherein the target kinematic model is constructed based on the kinematic information of the actuator of the agent, the kinematic information of the object to be interacted, and the kinematic information of the mobile mechanism corresponding to the agent;

[0006] Based on the target task corresponding to the agent, a task sequence is generated, wherein the task sequence includes at least one subsequence arranged in chronological order, and the subsequence includes target interaction object information, action information, and first posture information between the actuator and the object to be interacted; the target interaction object information is used to characterize the target object to be interacted among the objects to be interacted;

[0007] Based on the task sequence and the target kinematic model, a task motion plan of the agent is generated.

[0008] According to the task motion planning method of the intelligent agent of the present application, by simplifying complex actions into task sequences including action information, target interaction object information and posture information, for complex tasks, the complex tasks can be decomposed into multiple sub-sequences including action information, target interaction object information and posture information, and based on the task sequence and the target kinematic model, the task motion planning of the intelligent agent to perform the complex task is generated, so as to control the intelligent agent to perform the task based on the task motion planning, effectively cope with task-rich scenarios, ensure the task completion rate of the intelligent agent, and improve the flexibility of the intelligent agent control process, thereby improving the user experience.

[0009] According to the task motion planning method of the intelligent agent of the present application, the action information includes: one of a placing action, a picking action and a moving action.

[0010] According to the task motion planning method of the intelligent agent of the present application, the task motion planning of the intelligent agent is generated based on the task sequence and the target kinematic model, including:

[0011] Based on at least two of the degrees of freedom corresponding to the moving mechanism of the intelligent agent, the degrees of freedom corresponding to the actuator of the intelligent agent, and the degrees of freedom corresponding to the target object to be interacted with included in the first target subsequence in the at least one subsequence, and the first target subsequence, a task motion plan of the intelligent agent is generated.

[0012] According to the task motion planning method of the intelligent agent of the present application, the task motion planning of the intelligent agent is generated based on at least two of the degrees of freedom corresponding to the mobile mechanism of the intelligent agent, the degrees of freedom corresponding to the actuator of the intelligent agent, and the degrees of freedom corresponding to the target to-be-interacted object included in the first target subsequence in the at least one subsequence, and the first target subsequence, including:

[0013] In a case where the action information included in the first target subsequence is a picking action, determining second posture information of the agent when executing the first target subsequence based on the first target subsequence, the degrees of freedom corresponding to the mobile device of the agent, and the degrees of freedom corresponding to the agent;

[0014] Based on the second posture information, a task motion plan of the agent is generated.

[0015] According to the task motion planning method of the intelligent agent of the present application, the task motion planning of the intelligent agent is generated based on at least two of the degrees of freedom corresponding to the mobile mechanism of the intelligent agent, the degrees of freedom corresponding to the actuator of the intelligent agent, and the degrees of freedom corresponding to the target to-be-interacted object included in the first target subsequence in the at least one subsequence, and the first target subsequence, including:

[0016] In a case where the action information included in the first target subsequence is a placement action, determining third posture information of the actuator when executing the first target subsequence based on the first target subsequence, the degrees of freedom corresponding to the mobile device of the agent, the degrees of freedom corresponding to the agent, and the degrees of freedom corresponding to the target to-be-interacted object;

[0017] Based on the third posture information, a task motion plan of the agent is generated.

[0018] According to the task motion planning method of the intelligent agent of the present application, based on the task sequence and the target kinematic model, the task motion planning of the intelligent agent is generated, including:

[0019] After controlling the agent to execute a second target subsequence in the at least one subsequence arranged in time order, when the posture of the target object to be interacted with included in the second target subsequence changes, updating the task sequence and the target kinematic model based on the changed posture;

[0020] Based on the updated task sequence and the updated target kinematic model, a task motion plan of the agent is generated.

[0021] In a second aspect, the present application provides a task motion planning device for an intelligent agent, the device comprising:

[0022] A first processing module is used to obtain a target kinematic model between the agent and each object to be interacted in the task scene, wherein the target kinematic model is constructed based on the kinematic information of the actuator of the agent, the kinematic information of the object to be interacted, and the kinematic information of the mobile mechanism corresponding to the agent;

[0023] A second processing module is used to generate a task sequence based on the target task corresponding to the intelligent agent, wherein the task sequence includes at least one subsequence arranged in chronological order, and the subsequence includes target interaction object information, action information, and first posture information between the actuator and the object to be interacted; the target interaction object information is used to characterize the target object to be interacted among the objects to be interacted;

[0024] The third processing module is used to generate the task motion plan of the intelligent agent based on the task sequence and the target kinematic model.

[0025] According to the task motion planning device of the intelligent agent of the present application, by simplifying complex actions into task sequences including action information, target interaction object information and posture information, for complex tasks, the complex tasks can be decomposed into multiple sub-sequences including action information, target interaction object information and posture information, and based on the task sequence and the target kinematic model, the task motion planning of the intelligent agent to perform the complex task is generated, so as to control the intelligent agent to perform the task based on the task motion planning, effectively cope with task-rich scenarios, ensure the task completion rate of the intelligent agent, and improve the flexibility of the intelligent agent control process, thereby improving the user experience.

[0026] In a third aspect, the present application provides an intelligent agent, comprising:

[0027] sensor;

[0028] Executive body;

[0029] A moving mechanism, wherein the actuator is arranged on the moving mechanism;

[0030] As described in the second aspect, the task motion planning device of the intelligent body is electrically connected to the sensor, the actuator and the moving mechanism respectively.

[0031] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the task motion planning method for an intelligent agent as described in the first aspect above.

[0032] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the task motion planning method of an intelligent agent as described in the first aspect above.

[0033] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0034] By simplifying complex actions into task sequences including action information, target interaction object information and posture information, for complex tasks, the complex tasks can be decomposed into multiple sub-sequences including action information, target interaction object information and posture information. Based on the task sequence and the target kinematic model, the task motion plan for the intelligent agent to perform the complex task is generated, so as to control the intelligent agent to perform the task based on the task motion plan, effectively cope with task-rich scenarios, ensure the task completion rate of the intelligent agent, improve the flexibility of the intelligent agent control process, and improve the user experience.

[0035] Furthermore, by determining that in the process of the intelligent agent executing the task sequence corresponding to the target task, the action corresponding to the subsequence that has been completed by the intelligent agent has been artificially changed, according to the characteristics that the task sequence only includes action information, target interaction object information and the final relative posture between the intelligent agent and the actuator and the object to be interacted, and no intermediate parameters are set to force the intermediate posture of the moving structure and the actuator, the task sequence can be updated based on the changed posture of the target object to be interacted, thereby improving the flexibility of executing the target task; in addition, the action information includes three actions, and when the environment changes, the task sequence can be quickly updated without the need for re-planning after the original task sequence is executed, which effectively saves time, saves labor costs, and improves user experience.

[0036] Furthermore, by determining multiple degrees of freedom corresponding to the moving mechanism of the intelligent agent, the degrees of freedom corresponding to the actuator of the intelligent agent, and the degrees of freedom corresponding to the target object to be interacted with, the trajectory of the intelligent agent in executing the action corresponding to the subsequence can be effectively determined, so as to control the intelligent agent based on the action information corresponding to each subsequence and the trajectory corresponding to the subsequence to complete the target task. The intelligent agent can handle actions corresponding to multiple subsequences, thereby handling complex scenarios, improving task completion rate, and improving user experience.

[0037] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0039] Figure 1 It is one of the flowcharts of the task motion planning method of the intelligent agent provided in the embodiment of the present application;

[0040] Figure 2 It is one of the principle schematic diagrams of the task motion planning method of the intelligent agent provided in the embodiment of the present application;

[0041] Figure 3 This is the second principle schematic diagram of the task motion planning method of the intelligent agent provided in the embodiment of the present application;

[0042] Figure 4 It is a structural diagram of a task motion planning device for an intelligent agent provided in an embodiment of the present application;

[0043] Figure 5 This is the third principle schematic diagram of the task motion planning method of the intelligent agent provided in the embodiment of the present application;

[0044] Figure 6This is the fourth principle schematic diagram of the task motion planning method of the intelligent agent provided in the embodiment of the present application;

[0045] Figure 7 It is a structural diagram of a task motion planning system for an intelligent agent provided in an embodiment of the present application;

[0046] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0048] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0049] In combination with the accompanying drawings, the task motion planning method of the intelligent agent, the task motion planning device of the intelligent agent, the intelligent agent and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0050] Among them, the task motion planning method of the intelligent agent can be applied to the terminal, and can be specifically executed by hardware or software in the terminal.

[0051] The task motion planning method for an intelligent agent provided in an embodiment of the present application may be executed by an intelligent agent or an electronic device connected to the intelligent agent for communication, or a functional module or functional entity in the electronic device that can implement the task motion planning method for the intelligent agent. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The task motion planning method for an intelligent agent provided in an embodiment of the present application is described below using an intelligent agent as an example of an execution subject.

[0052] like Figure 1As shown, the task motion planning method of the intelligent agent includes: step 110, step 120 and step 130.

[0053] It should be noted that the task motion planning method of the intelligent agent can be applied to intelligent agent control scenarios, wherein the intelligent agent can be a robot or other structure that can achieve the same function. The robot can include but is not limited to household agents, production agents, service agents, mobile robotic arms, aerial work agents and other agents.

[0054] like Figure 3 As shown, the intelligent body may include an actuator 310 and a moving mechanism 320 . The actuator 310 is disposed on the moving mechanism 320 , and the moving mechanism 320 drives the actuator 310 to move.

[0055] The actuator 310 is used to perform operations such as grabbing, pushing, pulling, and placing, and may be a robotic arm, for example.

[0056] The moving mechanism 320 may be a roller or a track, etc.

[0057] Step 110: obtaining a target kinematic model between the agent and each object to be interacted in the task scene, wherein the target kinematic model is constructed based on the kinematic information of the actuator of the agent, the kinematic information of the object to be interacted with, and the kinematic information of the mobile mechanism corresponding to the agent;

[0058] In this step, the task scenario is the scenario in which the agent is located.

[0059] Mission scenarios include but are not limited to: shopping malls, libraries, and residential areas.

[0060] The objects to be interacted with are objects that can interact with the agent in the task scenario.

[0061] Objects to be interacted with include but are not limited to: cups, cabinets, bottles, chairs and other objects.

[0062] The actuator kinematic information is used to characterize the relevant information of the actuator, including but not limited to: the structure, size, properties and motion characteristics of the actuator.

[0063] The kinematic information of the object to be interacted with is used to represent relevant information of the object to be interacted with, including but not limited to: information such as the structure, size, shape, and attributes of the object to be interacted with.

[0064] The kinematic information of the mobile mechanism is used to characterize the relevant information of the mobile mechanism, including but not limited to: the structure, dimensional attributes, relative relationship with the external environment, and movement characteristics of the mobile mechanism.

[0065] The kinematic information of the mobile mechanism can reflect the movement possibility of the mobile mechanism.

[0066] In actual implementation, the planar motion of the mobile mechanism can be simulated through a three-degree-of-freedom kinematic chain.

[0067] The target kinematics model is a model used to characterize the serial motion relationship between the mobile mechanism, the actuator and the object to be interacted.

[0068] In some embodiments, there may be one or more objects to be interacted with, and the categories of the multiple objects to be interacted with may be different.

[0069] In some embodiments, when there are multiple objects to be interacted with, step 110 may also include: constructing a target kinematic model corresponding to the target first object to be interacted with based on the kinematic information of the object to be interacted with, the kinematic information of the actuator, and the kinematic information of the moving mechanism corresponding to the target first object to be interacted with among the multiple objects to be interacted with; and then fusing the target kinematic models corresponding to each object to be interacted with to obtain the final target kinematic model.

[0070] The target first object to be interacted with is any object among the multiple objects to be interacted with.

[0071] Step 120: Generate a task sequence based on the target task corresponding to the agent;

[0072] In this step, the target task is the task that the target agent needs to perform.

[0073] The task sequence includes at least one subsequence arranged in time order.

[0074] The subsequence includes target interaction object information, action information, and the first position information between the actuator and the object to be interacted.

[0075] The first pose information is used to characterize the final relative pose between the actuator and the object to be interacted with.

[0076] The target interaction object information is used to represent the target to-be-interacted object among the to-be-interacted objects.

[0077] In some embodiments, the action information may include: one of a placement action, a pickup action, and a move action.

[0078] In this embodiment, the placement action is that the actuator places the target object to be interacted with at a desired position, which can be expressed as place.

[0079] The picking action is that the actuator grabs the object to be interacted with, which can be expressed as pick.

[0080] The moving operation is to move the mobile mechanism to the desired position, which can be represented as goto.

[0081] During the actual execution process, the action information corresponding to each subsequence can be determined based on the actual situation of the target task.

[0082] In the actual execution process, the picking action moves the virtual kinematic chain to the target object to be interacted with and extends its kinematics by adding virtual additional joints of the target object to be interacted with and the actuator.

[0083] The picking action includes the task of moving the actuator to interact with the environment, for example, picking up the target object to be interacted with or grasping the target object to be interacted with.

[0084] The placement action moves the target object to be interacted with, which is already connected to the current virtual kinematic chain, to the desired position.

[0085] The target object to be interacted with needs to be incorporated into the virtual kinematic chain to impose kinematic constraints on the target kinematic model.

[0086] After the target object to be interacted with reaches the desired posture, the virtual kinematic chain at the virtual attachment joint can be disconnected to separate the actuator from the target object to be interacted with.

[0087] It can be appreciated that the place and pick actions are able to represent a wide range of mobile manipulation tasks, as well as help simplify the subsequent planning domain by eliminating unnecessary operations and intermediate state predicates.

[0088] According to the task motion planning method of the intelligent agent provided in the embodiment of the present application, by determining the actions that the intelligent agent can perform, the target task can be effectively completed based on different action information in the future to meet the user's usage needs, and the actions included in the action information can represent a wide range of operational tasks, which can effectively simplify the subsequent planning areas.

[0089] During the actual execution process, the target task can be disassembled to obtain the task sequence corresponding to the target task.

[0090] Taking the target task of putting the bottle on the coffee table and then putting it in the cabinet as an example, in the actual execution process, the agent needs to pick up the bottle first, that is, pick up the bottle, then place the bottle on the coffee table, and then pick up the bottle again and place the bottle on the cabinet, so the corresponding task sequence is: {pick bottle pose1, place bottle pose2, pickbottle pose3, place bottle pose4}.

[0091] It can be understood that pose1 and pose3 are used to characterize the relative position between the bottle and the actuator when the bottle is clamped, and pose2 and pose4 are used to characterize the relative position between the bottle and the actuator after the bottle is placed on the coffee table or cabinet.

[0092] Step 130: Generate a task motion plan for the agent based on the task sequence and the target kinematic model.

[0093] In this step, the target kinematic model can be adjusted based on the task sequence to obtain the kinematic characteristics of the agent performing the target task in the task scenario, and the task motion plan of the agent is generated based on the obtained kinematic characteristics.

[0094] It is understandable that the corresponding task sequences may be different depending on the task scenarios or the target tasks being performed. By simply inputting the task sequence into the target kinematic model, the relevant parameters in the target kinematic model can be adjusted accordingly, thereby achieving flexible control of the intelligent agent.

[0095] In the actual execution process, the task sequence can represent the motion constraints and spatial relationships between the intelligent agent and the target object to be interacted with.

[0096] In some embodiments, when the target object to be interacted with in the task sequence is articulated, the target kinematic model needs to be inverted so that the constructed kinematic chain remains continuous.

[0097] In some embodiments, the state of the virtual kinematic chain can be represented by a state vector, which can be expressed as:

[0098] Among them, q B =[x B ,y B ,θ B ] is the vector of the mobile mechanism, is the joint position vector of the actuator, q o is the joint position vector of the target object to be interacted with.

[0099] is the collision-free configuration space of the virtual kinematic chain, where n is the total degree of freedom.

[0100] In the actual execution process, motion planning can be performed based on the task sequence and the target kinematic model, that is, the t-step path is obtained: The subscript [k] represents the variable at step length k, which can be solved by the cabinet optimization formula to obtain the task motion plan of the intelligent agent, so as to control the intelligent agent to perform the target task based on the task motion plan.

[0101] During the research and development process, the inventors discovered that in the related art, the method of solving the challenges of the high complexity of the contact pattern between the end effector and the object and the high dimensionality of the state space in the sequential mobile operation planning problem in a hierarchical manner affects the flexibility of planning and the completion rate of the task, and affects the user experience; in addition, the above method is time-consuming and labor-intensive during the planning process.

[0102] In the present application, by converting the target task that the intelligent agent needs to perform into a task sequence including information of the target object to be interacted with, action information, and posture information between the actuator and the object to be interacted with, the planning field of controlling the intelligent agent to perform the target task is effectively simplified, and the task motion planning of the intelligent agent to perform the target task is generated based on multiple subsequences in the task sequence, which effectively reduces the time consumption in the task motion planning process and reduces labor costs; it can execute multiple actions based on subsequences, effectively cope with task-rich scenarios, so that the intelligent agent can complete the task in complex scenarios and ensure the completion rate of the task; in addition, the action information and posture information in the task sequence do not set intermediate parameters to force the intermediate posture of the mobile mechanism or the actuator, thereby improving the flexibility of the task motion planning process of the intelligent agent and improving the user experience.

[0103] According to the task motion planning method for an intelligent agent provided in an embodiment of the present application, by simplifying complex actions into task sequences including action information, target interaction object information and posture information, for complex tasks, the complex tasks can be decomposed into multiple sub-sequences including action information, target interaction object information and posture information, and based on the task sequence and the target kinematic model, a task motion plan for the intelligent agent to perform the complex task is generated, so that the intelligent agent can be controlled to perform the task based on the task motion plan, effectively coping with task-rich scenarios, ensuring the task completion rate of the intelligent agent, and improving the flexibility of the intelligent agent control process, thereby improving the user experience.

[0104] In some embodiments, step 130 may further include:

[0105] Based on at least two of the degrees of freedom corresponding to the agent's mobile mechanism, the degrees of freedom corresponding to the agent's actuator, and the degrees of freedom corresponding to the target object to be interacted with included in the first target subsequence in at least one subsequence, and the first target subsequence, a task motion plan for the agent is generated.

[0106] In some embodiments, the degrees of freedom corresponding to the moving mechanism may include the ability of the moving mechanism to move in the horizontal plane and in the vertical direction.

[0107] The degrees of freedom corresponding to the actuator may include: the translational degree of freedom of the actuator and the rotational degree of freedom of the actuator.

[0108] Among them, the translational degree of freedom is the ability of the actuator to move in the horizontal and vertical directions.

[0109] The rotational degree of freedom is the ability of the actuator to rotate around the axis corresponding to the horizontal plane and the axis corresponding to the vertical direction.

[0110] The target object to be interacted with is the object to be interacted with indicated in the first target subsequence.

[0111] The degree of freedom corresponding to the target object to be interacted with is the ability of the target object to be interacted with to move or rotate.

[0112] In the actual execution process, after obtaining the task sequence, the whole-body trajectory of the actuator, the mobile mechanism and the target object to be interacted with can be constructed based on each subsequence in the task sequence, that is, in the process of executing the actions corresponding to each subsequence, the possible values ​​of the degrees of freedom of the actuator, the mobile mechanism and the target object to be interacted with are obtained, thereby obtaining the whole-body trajectory of the intelligent agent, so as to control the intelligent agent to perform the target task based on the whole-body trajectory in the future.

[0113] The whole body trajectory is used to characterize the trajectory of each joint, that is, the task motion planning of each joint.

[0114] In some embodiments, the trajectory may include waypoints for each joint.

[0115] It can be understood that, for each subsequence, the whole body trajectory corresponding to each subsequence can be obtained in the above manner.

[0116] It is understandable that different target tasks correspond to different task sequences, and different task sequences may correspond to different whole-body trajectories.

[0117] According to the task motion planning method of the intelligent agent provided in the embodiment of the present application, by determining multiple degrees of freedom corresponding to the mobile mechanism of the intelligent agent, the degrees of freedom corresponding to the actuator of the intelligent agent, and the degrees of freedom corresponding to the target object to be interacted with, the trajectory of the intelligent agent in executing the action corresponding to the subsequence is effectively determined, so that based on the action information corresponding to each subsequence and the trajectory corresponding to the subsequence, the intelligent agent is controlled to complete the target task. The intelligent agent can process actions corresponding to multiple subsequences, thereby handling complex scenarios, improving the task completion rate, and improving the user experience.

[0118] In some embodiments, generating a task motion plan of the agent based on at least two of the degrees of freedom corresponding to the agent's mobile mechanism, the degrees of freedom corresponding to the agent's actuator, and the degrees of freedom corresponding to the target to-be-interacted object included in the first target subsequence in at least one subsequence, and the first target subsequence, may further include:

[0119] In a case where the action information included in the first target subsequence is a picking action, determining second posture information of the agent when executing the first target subsequence based on the first target subsequence, the degrees of freedom corresponding to the mobile device of the agent, and the degrees of freedom corresponding to the agent;

[0120] Based on the second pose information, the task motion plan of the intelligent agent is generated.

[0121] In this embodiment, the second posture information is the posture information of the moving mechanism and the actuator that needs to be determined when the intelligent agent performs a picking action.

[0122] In the actual execution process, taking the example of the agent's actuator reaching a position and grabbing a bottle, the second posture information of the agent when executing the first target subsequence can be determined by determining the three degrees of freedom of the mobile mechanism and the six degrees of freedom of the actuator.

[0123] In some embodiments, the degrees of freedom of an actuator may also be determined based on the number of axes of the actuator.

[0124] It can be understood that the greater the number of axes of an actuator, the greater the degree of freedom of the actuator.

[0125] After determining the second posture information, a task motion plan corresponding to the target task of the intelligent agent is generated based on the second posture information, so as to subsequently control the intelligent agent to complete the target task based on the task motion plan.

[0126] According to the task motion planning method for an intelligent agent provided in an embodiment of the present application, by determining that the action information is a picking action, the posture information of the intelligent agent in the process of executing the picking action is effectively determined through the degrees of freedom corresponding to the moving mechanism of the intelligent agent and the degrees of freedom corresponding to the executing mechanism of the intelligent agent, so as to generate a task motion plan corresponding to the execution target task of the intelligent agent based on the posture information of the intelligent agent, so that the intelligent agent can execute corresponding actions based on the task motion plan, accurately control the intelligent agent, and improve the user experience.

[0127] In some embodiments, generating a task motion plan of the agent based on at least two of the degrees of freedom corresponding to the agent's mobile mechanism, the degrees of freedom corresponding to the agent's actuator, and the degrees of freedom corresponding to the target to-be-interacted object included in the first target subsequence in at least one subsequence, and the first target subsequence, may further include:

[0128] In the case where the action information included in the first target subsequence is a placement action, determining third posture information of the agent when executing the first target subsequence based on the first target subsequence, the degrees of freedom corresponding to the mobile device of the agent, the degrees of freedom corresponding to the agent, and the degrees of freedom corresponding to the target to-be-interacted object;

[0129] Based on the third pose information, the task motion plan of the intelligent agent is generated.

[0130] In this embodiment, the third posture information is the posture information of the moving mechanism, the actuator, and the target object to be interacted with that needs to be determined when the intelligent agent performs a placement action.

[0131] In the actual implementation process, Figure 2 As shown, the agent opens the cabinet door, and the target object to be interacted with is the cabinet door. The agent's mobile mechanism needs to move in front of the cabinet door, and then the actuator opens the cabinet door. The cabinet door is hinged. At this time, it is also necessary to further determine the third posture information of the agent when executing the first target subsequence based on the degree of freedom of the cabinet door.

[0132] After determining the third posture information, a task motion plan corresponding to the target task of the intelligent agent is generated based on the third posture information, so as to subsequently control the mobile mechanism, actuator and target object to be interacted with of the intelligent agent to reach the required posture based on the task motion plan, thereby controlling the intelligent agent to complete the target task.

[0133] According to the task motion planning method of the intelligent agent provided in the embodiment of the present application, by determining that the action information is a placement action, the posture information of the intelligent agent and the posture information of the target object to be interacted with are effectively determined during the placement action through the degrees of freedom corresponding to the mobile mechanism of the intelligent agent, the degrees of freedom corresponding to the actuator mechanism of the intelligent agent, and the degrees of freedom corresponding to the target object to be interacted with, so as to generate a task motion plan corresponding to the execution of the target task of the intelligent agent based on the posture information, so that the intelligent agent can execute the corresponding action based on the task motion plan, accurately control the intelligent agent, and improve the user experience.

[0134] In some embodiments, step 130 may further include:

[0135] After the control agent completes the execution of the second target subsequence in at least one subsequence arranged in time order, when the posture of the target object to be interacted with included in the second target subsequence changes, the task sequence and the target kinematic model are updated based on the changed posture;

[0136] Based on the updated task sequence and the updated target kinematic model, the agent's task motion plan is generated.

[0137] In this embodiment, the second target subsequence is one of a plurality of subsequences included in the task sequence.

[0138] The changed posture is a new posture of the target object to be interacted with that is different from the posture information included in the second target subsequence after the agent executes the second target subsequence.

[0139] In the actual execution process, when the agent executes multiple subsequences in the task sequence, external force information may appear, causing the position and posture of the target object to be interacted with to change.

[0140] During the actual execution process, the sensor can be used to determine whether the position and posture of the target object to be interacted with has changed.

[0141] For example, the task sequence is the sequence corresponding to closing the cabinet door and placing the chair in the specified position, such as Figure 5 As shown in , after the agent completes the subsequence corresponding to closing the cabinet door, in the process of executing the subsequence corresponding to moving the chair, as shown in Figure 6 As shown, the cabinet door is opened manually again. At this time, the sensor needs to send the cabinet door opening situation to the intelligent agent to update the task sequence and execute the subsequent process.

[0142] For another example, the task sequence is to first take the bottle to the coffee table and then put it in the cabinet. For the corresponding task sequence, the agent needs to execute {pick bottle pose1, place bottle pose2} twice. However, after the agent puts the bottle on the coffee table, there may be a situation where someone manually takes the bottle to the table.

[0143] At this time, the position of the target object to be interacted with, the bottle, has changed. The task sequence needs to be updated based on the new position of the target object to be interacted with. For this target task, the agent still needs to perform pick and place operations, but pose1 and pose2 in the two subsequences have changed. Pose1 and pose2 need to be updated to obtain a new task sequence.

[0144] It can be understood that after the task sequence changes, the target kinematics model will also be updated accordingly, so that based on the updated task sequence and the updated target kinematics model, the task motion planning of the intelligent agent in the new task scenario can be generated to subsequently control the intelligent agent to perform the target task corresponding to the updated task sequence.

[0145] According to the task motion planning method of the intelligent agent provided in the embodiment of the present application, by determining that in the process of the intelligent agent executing the task sequence corresponding to the target task, the action corresponding to the subsequence that has been completed by the intelligent agent has been artificially changed, based on the characteristics that the task sequence only includes action information, target interaction object information and the final relative posture between the intelligent agent and the actuator and the object to be interacted, and no intermediate parameters are set to force the intermediate posture of the moving structure and the actuator, the task sequence can be updated based on the changed posture of the target object to be interacted, thereby improving the flexibility of executing the target task; in addition, the action information includes three actions, and when the environment changes, the task sequence can be quickly updated without the need for re-planning after the original task sequence is executed, which effectively saves time, saves labor costs, and improves user experience.

[0146] The task motion planning method of the intelligent agent provided in the embodiment of the present application can be performed by the task motion planning device of the intelligent agent. In the embodiment of the present application, the task motion planning device of the intelligent agent performing the task motion planning method of the intelligent agent is taken as an example to illustrate the task motion planning device of the intelligent agent provided in the embodiment of the present application.

[0147] An embodiment of the present application also provides a task motion planning device for an intelligent agent.

[0148] like Figure 4 As shown, the task motion planning device of the intelligent agent includes: a first processing module 410, a second processing module 420 and a third processing module 430.

[0149] The first processing module 410 is used to obtain a target kinematic model between the agent and each object to be interacted in the task scene, where the target kinematic model is constructed based on the kinematic information of the actuator of the agent, the kinematic information of the object to be interacted with, and the kinematic information of the mobile mechanism corresponding to the agent;

[0150] The second processing module 420 is used to generate a task sequence based on the target task corresponding to the agent, the task sequence includes at least one subsequence arranged in chronological order, the subsequence includes target interaction object information, action information and first posture information between the actuator and the object to be interacted; the target interaction object information is used to characterize the target object to be interacted among the objects to be interacted;

[0151] The third processing module 430 is used to generate a task motion plan of the agent based on the task sequence and the target kinematic model.

[0152] According to the task motion planning device of the intelligent agent provided in the embodiment of the present application, by simplifying complex actions into task sequences including action information, target interaction object information and posture information, for complex tasks, the complex tasks can be decomposed into multiple sub-sequences including action information, target interaction object information and posture information, and based on the task sequence and the target kinematic model, the task motion planning of the intelligent agent to perform the complex task is generated, so as to control the intelligent agent to perform the task based on the task motion planning, effectively cope with task-rich scenarios, ensure the task completion rate of the intelligent agent, and improve the flexibility of the intelligent agent control process, thereby improving the user experience.

[0153] In some embodiments, the third processing module 430 may also be used to:

[0154] Based on at least two of the degrees of freedom corresponding to the agent's mobile mechanism, the degrees of freedom corresponding to the agent's actuator, and the degrees of freedom corresponding to the target object to be interacted with included in the first target subsequence in at least one subsequence, and the first target subsequence, a task motion plan for the agent is generated.

[0155] In some embodiments, the third processing module 430 may also be used to:

[0156] In a case where the action information included in the first target subsequence is a picking action, determining second posture information of the agent when executing the first target subsequence based on the first target subsequence, the degrees of freedom corresponding to the mobile device of the agent, and the degrees of freedom corresponding to the agent;

[0157] Based on the second pose information, the task motion plan of the intelligent agent is generated.

[0158] In some embodiments, the third processing module 430 may also be used to:

[0159] In the case where the action information included in the first target subsequence is a placement action, based on the first target subsequence, the degrees of freedom corresponding to the mobile device of the agent, the degrees of freedom corresponding to the agent, and the degrees of freedom corresponding to the target to-be-interacted object, determining the third posture information of the actuator when executing the first target subsequence;

[0160] Based on the third pose information, the task motion plan of the intelligent agent is generated.

[0161] In some embodiments, the third processing module 430 may also be used to:

[0162] After the control agent completes the execution of the second target subsequence in at least one subsequence arranged in time order, when the posture of the target object to be interacted with included in the second target subsequence changes, the task sequence and the target kinematic model are updated based on the changed posture;

[0163] Based on the updated task sequence and the updated target kinematic model, the agent's task motion plan is generated.

[0164] The task motion planning device of the agent in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, an agent, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0165] The task motion planning device of the agent in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0166] The task motion planning device of the intelligent body provided in the embodiment of the present application can achieve Figures 1 to 3 and Figures 5 and 6 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0167] like Figure 3 As shown, an embodiment of the present application further provides an intelligent agent, including: a sensor, an actuator 310, a mobile mechanism 320 and a task motion planning device of the intelligent agent as described in any of the above embodiments.

[0168] The actuator 310 is used to perform operations such as grabbing, pushing, pulling, and placing.

[0169] The actuator 310 is disposed on the moving mechanism 320 , and the moving mechanism 320 drives the actuator 310 to move.

[0170] The task motion planning device of the intelligent agent is electrically connected to the sensor, the actuator 310 and the moving mechanism 320 respectively.

[0171] The device for task motion planning of an intelligent agent is used to execute the task motion planning method of an intelligent agent as described in any of the above embodiments.

[0172] According to the intelligent agent provided in the embodiment of the present application, by simplifying complex actions into task sequences including action information, target interaction object information and posture information, for complex tasks, the complex tasks can be decomposed into multiple sub-sequences including action information, target interaction object information and posture information, and based on the task sequence and the target kinematic model, a task motion plan for the intelligent agent to perform the complex task is generated, so as to control the intelligent agent to perform the task based on the task motion plan, effectively cope with task-rich scenarios, ensure the task completion rate of the intelligent agent, and improve the flexibility of the intelligent agent control process, thereby improving the user experience.

[0173] In some embodiments, Figure 8 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, each process of the above-mentioned intelligent agent task motion planning method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0174] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0175] like Figure 7 As shown, the present application also provides a task motion planning system for an intelligent agent, the system comprising: a task planning module, a motion planning module and a state monitoring module.

[0176] In some embodiments, the task planning module decomposes the target task into a task sequence represented by a virtual kinematic chain based on symbolic actions in a planning domain of a planning domain definition language.

[0177] For example, opening a cabinet door can be expressed as: Action1: Pick, Object1: Handle Attach1: the relative position of the actuator and the door handle, VKC1: from the root coordinate system of the mobile mechanism to the coordinate system of the actuator; Action2: Place, Object2: Handle, Attach2: the fixed connection position of the actuator and the door handle, VKC2: from the root coordinate system of the mobile mechanism to the root coordinate system of the cabinet.

[0178] The motion planning module is used to obtain the trajectory (i.e., posture information) of all joints of the intelligent agent by considering the execution objectives and self-constraints of the target task based on the motion planning of the virtual kinematic chain after obtaining the task sequence.

[0179] The state monitoring module is used to monitor changes in the task environment.

[0180] For example, environmental information corresponding to task scenarios such as the opening and closing of cabinet doors and the position of chairs.

[0181] Taking the target task of closing the cabinet door and placing the chair in the specified position as an example, after the intelligent agent closes the cabinet door, the cabinet door is opened manually during the process of moving the chair. The state monitoring module needs to send the information that the cabinet door is opened to the task planning module so that the task planning module can quickly respond to environmental changes, generate new task planning results, and execute subsequent processes.

[0182] According to the task motion planning system of the intelligent agent provided in the embodiment of the present application, by simplifying complex actions into task sequences including action information, target interaction object information and posture information, for complex tasks, the complex tasks can be decomposed into multiple sub-sequences including action information, target interaction object information and posture information, and based on the task sequence and the target kinematic model, the task motion planning of the intelligent agent to perform the complex task is generated, so as to control the intelligent agent to perform the task based on the task motion planning, effectively cope with task-rich scenarios, ensure the task completion rate of the intelligent agent, and improve the flexibility of the intelligent agent control process, thereby improving the user experience.

[0183] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned intelligent agent task motion planning method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0184] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0185] An embodiment of the present application also provides a computer program product, including a computer program, which implements the task motion planning method of the above-mentioned intelligent agent when executed by a processor.

[0186] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0187] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned intelligent agent task motion planning method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0188] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0189] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0190] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0191] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0192] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0193] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A task motion planning method for an intelligent agent, characterized in that: include: Acquire a target kinematic model between the agent and each object to be interacted in the task scene, wherein the target kinematic model is constructed based on the kinematic information of the actuator of the agent, the kinematic information of the object to be interacted, and the kinematic information of the mobile mechanism corresponding to the agent; Based on the target task corresponding to the agent, a task sequence is generated, wherein the task sequence includes at least one subsequence arranged in time order, and the subsequence includes target interaction object information, action information, and first position information between the actuator and the object to be interacted; The target interaction object information is used to represent the target to-be-interacted object among the to-be-interacted objects; Based on the task sequence and the target kinematic model, a task motion plan of the agent is generated.

2. The task motion planning method of an intelligent agent according to claim 1, characterized in that: The action information includes: one of a placing action, a picking action, and a moving action.

3. The task motion planning method of an intelligent agent according to claim 1, characterized in that: The step of generating the task motion plan of the agent based on the task sequence and the target kinematic model includes: Based on at least two of the degrees of freedom corresponding to the moving mechanism of the intelligent agent, the degrees of freedom corresponding to the actuator of the intelligent agent, and the degrees of freedom corresponding to the target object to be interacted with included in the first target subsequence in the at least one subsequence, and the first target subsequence, a task motion plan of the intelligent agent is generated.

4. The task motion planning method of an intelligent agent according to claim 3, characterized in that: The step of generating a task motion plan of the agent based on at least two of the degrees of freedom corresponding to the moving mechanism of the agent, the degrees of freedom corresponding to the actuator of the agent, and the degrees of freedom corresponding to the target to-be-interacted object included in the first target subsequence in the at least one subsequence, and the first target subsequence, comprises: In a case where the action information included in the first target subsequence is a picking action, determining second posture information of the agent when executing the first target subsequence based on the first target subsequence, the degrees of freedom corresponding to the mobile device of the agent, and the degrees of freedom corresponding to the agent; Based on the second posture information, a task motion plan of the agent is generated.

5. The task motion planning method of an intelligent agent according to claim 3, characterized in that: The step of generating a task motion plan of the agent based on at least two of the degrees of freedom corresponding to the moving mechanism of the agent, the degrees of freedom corresponding to the actuator of the agent, and the degrees of freedom corresponding to the target to-be-interacted object included in the first target subsequence in the at least one subsequence, and the first target subsequence, comprises: In a case where the action information included in the first target subsequence is a placement action, determining third posture information of the actuator when executing the first target subsequence based on the first target subsequence, the degrees of freedom corresponding to the mobile device of the agent, the degrees of freedom corresponding to the agent, and the degrees of freedom corresponding to the target to-be-interacted object; Based on the third posture information, a task motion plan of the agent is generated.

6. The task motion planning method of an intelligent agent according to any one of claims 1 to 5, characterized in that: Generating a task motion plan of the agent based on the task sequence and the target kinematic model includes: After controlling the agent to execute a second target subsequence in the at least one subsequence arranged in time order, when the posture of the target object to be interacted with included in the second target subsequence changes, updating the task sequence and the target kinematic model based on the changed posture; Based on the updated task sequence and the updated target kinematic model, a task motion plan of the agent is generated.

7. A task motion planning device for an intelligent agent, characterized in that: include: A first processing module is used to obtain a target kinematic model between the agent and each object to be interacted in the task scene, wherein the target kinematic model is constructed based on the kinematic information of the actuator of the agent, the kinematic information of the object to be interacted, and the kinematic information of the mobile mechanism corresponding to the agent; A second processing module is used to generate a task sequence based on the target task corresponding to the agent, wherein the task sequence includes at least one subsequence arranged in chronological order, and the subsequence includes target interaction object information, action information, and first position information between the actuator and the object to be interacted; The target interaction object information is used to represent the target to-be-interacted object among the to-be-interacted objects; The third processing module is used to generate the task motion plan of the intelligent agent based on the task sequence and the target kinematic model.

8. An intelligent agent, characterized in that: include: sensor; Executive body; A moving mechanism, wherein the actuator is arranged on the moving mechanism; According to the task motion planning device of the intelligent body as described in claim 7, the task motion planning device of the intelligent body is electrically connected to the sensor, the actuator and the moving mechanism respectively.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task motion planning method of an intelligent agent as described in any one of claims 1-6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the task motion planning method of an intelligent agent as described in any one of claims 1-6 is implemented.

Images