Intelligent agent control method and device and intelligent agent

Abstract

The invention discloses an agent control method and device and an agent, and belongs to the field of agents. The intelligent agent control method comprises the following steps: constructing a target kinematic model based on execution mechanism information of an intelligent agent, executed object information of an executed object and moving mechanism information corresponding to the intelligent agent; based on the environment information and the target task information in the target scene, determining a target action sequence and a motion constraint corresponding to the intelligent agent; and controlling the intelligent agent based on the target action sequence, the motion constraint and the target kinematics model. According to the intelligent agent control method, efficient coordinated movement of the intelligent agent can be achieved, the movement posture of the intelligent agent can be correspondingly adjusted without modifying source codes of the intelligent agent under the condition that an actual application scene or an execution task changes, effective control over the intelligent agent under any scene is achieved, high universality is achieved, and the intelligent agent control method is suitable for popularization and application. The method is suitable for any complex execution environment, the control precision is high, and the intelligent agent can execute complex operation tasks.

Description

Technical Field

[0001] The present application belongs to the field of intelligent agents, and in particular, relates to an intelligent agent control method, device and intelligent agent. Background Art

[0002] With the development of science and technology, intelligent agents are widely used in people's daily work and life. In related technologies, sensor data is mainly collected by sensors set on the intelligent agent, so as to control the intelligent agent to move or grasp based on the sensor data. However, this method can only control the intelligent agent to perform relatively simple operations, such as grasping or placing, and cannot be applied to complex environments and the execution of complex tasks. For example, when executing long task sequences, the control accuracy is low and the versatility is poor, which affects the user experience. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes an intelligent agent control method, device and intelligent agent, which have high versatility, are applicable to any complex execution environment, and have high control accuracy, so that the intelligent agent can perform complex operation tasks.

[0004] In a first aspect, the present application provides an intelligent agent control method, the method comprising:

[0005] Constructing a target kinematic model based on the actuator information of the agent, the executed object information of the executed object, and the mobile mechanism information corresponding to the agent;

[0006] Determine the target action sequence and motion constraints corresponding to the agent based on the environment information and target task information in the target scene;

[0007] The agent is controlled based on the target action sequence, the motion constraints and the target kinematic model.

[0008] According to the intelligent agent control method of the present application, the target kinematic model is obtained by integrating the kinematic models of the different motion units of the intelligent agent itself and the executed object, and then the target action sequence and motion constraints obtained based on the environmental information and target task information in the target scene are added to the target kinematic model, so that the target kinematic model generates control instructions in a specific scene to control the intelligent agent to perform corresponding actions, thereby realizing efficient and coordinated movement of the intelligent agent. When the actual application scene or the execution task changes, the motion posture of the intelligent agent can be adjusted accordingly without modifying the source code of the intelligent agent to realize effective control of the intelligent agent in any scene. The method has high versatility, is suitable for any complex execution environment, and has high control accuracy, so that the intelligent agent can perform complex operation tasks.

[0009] According to an embodiment of the present application, determining the target action sequence and motion constraints corresponding to the agent based on the environmental information and target task information in the target scene includes:

[0010] Based on the target task information, determining at least one of the agent displacement action information, the agent grasping action information, and the executed object displacement information;

[0011] The target action sequence is determined based on at least one of the intelligent agent displacement action information, the intelligent agent grasping action information, the executed object displacement information, and the environmental information.

[0012] According to an embodiment of the present application, determining the target action sequence and motion constraints corresponding to the agent based on the environmental information and target task information in the target scene includes:

[0013] Based on the environment information and the target task information, determining a plurality of candidate action sequences of the agent within a next target time period;

[0014] An optimal candidate action sequence among the multiple candidate action sequences is determined as the target action sequence in the next target time period.

[0015] According to an embodiment of the present application, the target kinematic model is constructed based on the actuator information of the agent, the executed object information of the executed object, and the mobile mechanism information corresponding to the agent, including:

[0016] Constructing a first kinematic model based on the actuator information, constructing a second kinematic model based on the executed object information, and constructing a third kinematic model based on the moving mechanism information;

[0017] performing inversion processing on the second kinematic model to obtain a fourth kinematic model;

[0018] The target kinematic model is constructed based on the first kinematic model, the fourth kinematic model and the third kinematic model.

[0019] According to an embodiment of the present application, performing inversion processing on the second kinematic model to obtain a fourth kinematic model includes:

[0020] Based on the second kinematic model, acquiring a plurality of nodes of the executed object and a plurality of connection paths between the plurality of nodes;

[0021] Determine the path between the root node and the grabbing node in the multiple connection paths as a trunk path;

[0022] Kinematic inversion is performed on the trunk path, and other connection paths among the multiple connection paths except the trunk path are updated to obtain the fourth kinematic model.

[0023] According to an embodiment of the present application, constructing the target kinematic model based on the first kinematic model, the fourth kinematic model and the third kinematic model includes:

[0024] Connecting the first kinematic model and the fourth kinematic model via a virtual joint;

[0025] The third kinematic model is connected to the first kinematic model to construct the target kinematic model.

[0026] According to an embodiment of the present application, determining the target action sequence and motion constraints corresponding to the agent based on the environmental information and target task information in the target scene includes:

[0027] At least one of a collision constraint, an environmental constraint, and a task constraint is determined based on the environmental information and the target task information.

[0028] According to an embodiment of the present application, determining at least one of a collision constraint, an environmental constraint, and a task constraint based on the environmental information and the target task information includes:

[0029] Constructing a signed distance field function based on the obstacle information in the environmental information, wherein the signed distance field function is used to characterize the distance and gradient between each position point in the target scene and the closed surface of the obstacle;

[0030] The collision constraint is constructed based on the signed distance field function.

[0031] According to an embodiment of the present application, determining at least one of a collision constraint, an environmental constraint, and a task constraint based on the environmental information and the target task information includes:

[0032] Mapping the joint space of the agent at the target position to the task target space of the object being executed to obtain a real-time state value;

[0033] The task constraint is constructed based on the real-time state value and the target task information.

[0034] According to one embodiment of the present application, controlling the agent based on the target action sequence, the motion constraints and the target kinematic model includes:

[0035] Adjust the target kinematic model based on the target action sequence and the motion constraint, and obtain the motion trajectory corresponding to the agent and the pose information at the target position in the motion trajectory;

[0036] Based on the motion trajectory and the posture information, the intelligent agent is controlled.

[0037] In a second aspect, the present application provides an intelligent body control device, the device comprising:

[0038] A first processing module is used to construct a target kinematic model based on the actuator information of the agent, the executed object information of the executed object and the mobile mechanism information corresponding to the agent;

[0039] A second processing module is used to determine the target action sequence and motion constraints corresponding to the agent based on the environmental information and target task information in the target scene;

[0040] The third processing module is used to control the intelligent agent based on the target action sequence, the motion constraints and the target kinematic model.

[0041] According to the intelligent agent control device of the present application, the target kinematic model is obtained by integrating the different motion units of the intelligent agent itself and the motion structure of the executed object, and then the target action sequence and motion constraints obtained based on the environmental information and target task information in the target scene are added to the target kinematic model, so that the target kinematic model generates control instructions in a specific scene to control the intelligent agent to perform corresponding actions, thereby realizing efficient and coordinated movement of the intelligent agent. When the actual application scene or the execution task changes, the motion posture of the intelligent agent can be adjusted accordingly without modifying the source code of the intelligent agent to realize effective control of the intelligent agent in any scene. It has high versatility, is suitable for any complex execution environment, and has high control accuracy, so that the intelligent agent can perform complex operation tasks.

[0042] In a third aspect, the present application provides an intelligent agent, comprising:

[0043] sensor;

[0044] Executive body;

[0045] A moving mechanism, wherein the actuator is arranged on the moving mechanism;

[0046] As described in the second aspect, the intelligent body control device is electrically connected to the sensor, the actuator and the moving mechanism respectively.

[0047] In a fourth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the intelligent body control method as described in the first aspect above is implemented.

[0048] In a fifth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the intelligent agent control method as described in the first aspect above is implemented.

[0049] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the intelligent agent control method as described in the first aspect above.

[0050] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0051] The target kinematic model is obtained by integrating the different motion units of the intelligent agent and the motion structure of the executed object, and then the target action sequence and motion constraints obtained based on the environmental information and target task information in the target scene are added to the target kinematic model, so that the target kinematic model generates control instructions in a specific scene to control the intelligent agent to perform corresponding actions, thereby realizing efficient coordinated movement of the intelligent agent. When the actual application scenario or execution task changes, the motion posture of the intelligent agent can be adjusted accordingly without modifying the source code of the intelligent agent to achieve effective control of the intelligent agent in any scene. It has high versatility and is suitable for any complex execution environment, and has high control accuracy, so that the intelligent agent can perform complex operation tasks.

[0052] Furthermore, the kinematic model is decomposed into multiple action states based on the target task information, which can effectively cover most mobile operation tasks, thereby effectively describing tasks of various kinematic states; on this basis, the action states are connected based on the time sequence to obtain the target action sequence, which has high accuracy and precision, and is especially suitable for complex kinematic models, and thus suitable for complex execution actions.

[0053] Furthermore, by constructing collision constraints through signed distance field functions, the closed surface of the collision volume can be accurately described, which is particularly suitable for obstacle avoidance control in narrow and crowded environments, thereby further improving the accuracy and precision of the agent's path optimization.

[0054] Furthermore, intelligently planning the target action sequence within the next target period through environmental information and target task information can further improve the rationality and accuracy of the obtained target action sequence, which is conducive to improving the control effect of the intelligent body.

[0055] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0057] Figure 1 It is one of the flow charts of the intelligent agent control method provided in the embodiment of the present application;

[0058] Figure 2 This is the second flow chart of the intelligent agent control method provided in the embodiment of the present application;

[0059] Figure 3 This is the third flow chart of the intelligent agent control method provided in the embodiment of the present application;

[0060] Figure 4 It is a schematic diagram of the principle of the intelligent agent control method provided in the embodiment of the present application;

[0061] Figure 5 is a schematic diagram of the structure of the intelligent body control device provided in an embodiment of the present application;

[0062] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0064] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0065] The following, in conjunction with the accompanying drawings, describes in detail the intelligent agent control method, intelligent agent control device, intelligent agent and readable storage medium provided in the embodiments of the present application through specific embodiments and their application scenarios.

[0066] Among them, the intelligent agent control method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0067] The intelligent body control method provided in the embodiment of the present application may be executed by an intelligent body, or an electronic device connected to the intelligent body for communication, or a functional module or functional entity in the electronic device that can implement the intelligent body control method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The intelligent body control method provided in the embodiment of the present application is described below using an intelligent body as an example of an execution subject.

[0068] like Figure 1 As shown, the intelligent agent control method includes: step 110, step 120 and step 130.

[0069] It should be noted that the intelligent agent control method can be applied to intelligent agent control scenarios, wherein the intelligent agent can be a robot or other structure that can achieve the same function. The robot can include but is not limited to household agents, production agents, service agents, mobile robotic arms, aerial work agents and other intelligent agents.

[0070] like Figure 4 As shown, the intelligent body may include an actuator 410 and a moving mechanism 420 . The actuator 410 is disposed on the moving mechanism 420 , and the moving mechanism 420 drives the actuator 410 to move.

[0071] The actuator 410 is used to perform operations such as grabbing, pushing, pulling, and placing, and may be a robotic arm, for example.

[0072] The moving mechanism 420 may be a roller or a track, etc.

[0073] The executed object 430 is an execution object of the execution mechanism 410 .

[0074] Step 110, constructing a target kinematic model based on the actuator information of the agent, the executed object information of the executed object 430 and the mobile mechanism information corresponding to the agent;

[0075] In this step, the actuator information is used to characterize relevant information of the actuator 410 , including but not limited to: information such as the structure, size, attributes, and motion characteristics of the actuator 410 .

[0076] The mobile mechanism information is used to characterize relevant information of the mobile mechanism 420 , including but not limited to: information such as the structure, dimensional attributes, relative relationship with the external environment, and movement characteristics of the mobile mechanism 420 .

[0077] The executed object 430 is an object to be operated. For example, in a grasping scenario, the executed object 430 may be an object to be grasped by the agent; in a door opening and closing scenario, the executed object 430 may be a door that needs to be opened or closed.

[0078] The executed object information is used to represent relevant information of the executed object 430 , including but not limited to: information such as the structure, size, shape, and attributes of the executed object 430 .

[0079] The target kinematics model is a model used to characterize the serial motion relationship between the mobile mechanism 420 , the actuator 410 , and the object 430 .

[0080] In some embodiments, there may be one or more executed objects 430 , and the categories of the multiple executed objects 430 may be different.

[0081] In some embodiments, when there are multiple executed objects 430, step 110 may also include: constructing a target kinematic model corresponding to the target executed object based on the executed object information, actuator information, and moving mechanism information corresponding to the target executed object in the multiple executed objects 430; and then fusing the target kinematic models corresponding to each executed object 430 to obtain the final target kinematic model.

[0082] The target executed object may be any object among the multiple executed objects 430 .

[0083] In some embodiments, step 110 may include:

[0084] Constructing a first kinematic model based on the actuator information, constructing a second kinematic model based on the executed object information, and constructing a third kinematic model based on the moving mechanism information;

[0085] Performing inversion processing on the second kinematic model to obtain a fourth kinematic model;

[0086] A target kinematic model is constructed based on the first kinematic model, the fourth kinematic model and the third kinematic model.

[0087] In this embodiment, the first kinematic model is used to characterize the kinematic characteristics of the actuator 410 ; the second kinematic model is used to characterize the kinematic characteristics of the object 430 ; and the third kinematic model is used to characterize the kinematic characteristics of the moving mechanism 420 .

[0088] The fourth kinematic model is obtained by performing kinematic structural inversion on some nodes in the kinematic chain formed by the second kinematic model.

[0089] By inverting the second kinematic model, the grasping node of the object 430 to be executed can be directly connected to the end effector of the intelligent body, ensuring the establishment of a serial motion chain, thereby realizing subsequent motion planning.

[0090] In the actual implementation process, the first kinematic model can be determined from the agent DH parameters, such as Figure 2 (a) As shown; the second kinematic model can be determined from the agent visual parameters, such as Figure 2 (a) shown.

[0091] In some embodiments, constructing the third kinematic model based on the mobile mechanism information may include:

[0092] Determining a motion mode of the mobile mechanism 420 based on a motion characteristic of the mobile mechanism 420;

[0093] Based on the motion pattern, a third kinematic model is constructed.

[0094] In this embodiment, the moving characteristics of the moving mechanism 420 are different, and the corresponding motion patterns may also be different.

[0095] Taking the movement of the mobile mechanism 420 on the plane through the roller as an example, based on the movement characteristics, the kinematic abstraction of the mobile mechanism 420 can be obtained, including: a movement mode including two translations and one rotational degree of freedom. Based on the movement mode, the third kinematic model can be constructed through the PPR series structure, such as Figure 2 (c) As shown, the P end is connected to a fixed node (usually a world system node), and the other end is connected to a mobile mechanism 420.

[0096] In some embodiments, performing inversion processing on the second kinematic model to obtain the fourth kinematic model may include:

[0097] Based on the second kinematic model, a plurality of nodes of the executed object 430 and a plurality of connection paths between the plurality of nodes are acquired;

[0098] Determine the path between the root node and the grab node in the multiple connection paths as the main path;

[0099] Kinematic inversion is performed on the trunk path, and other connection paths except the trunk path among the multiple connection paths are updated to obtain a fourth kinematic model.

[0100] In this embodiment, the grasping node is the execution node when the execution mechanism 410 executes the executed object 430 (that is, the executed node of the executed object 430), and the grasping node corresponds to the end effector of the intelligent body.

[0101] Taking the executed object 430 as a door as an example, the multiple nodes may include a door frame, a door, and other nodes.

[0102] For the second kinematic model, its root node is the door frame, the grasping node of the actuator 410 is the door, and the trunk path is from the door frame to the door.

[0103] By inverting the main rod path, the inversion path from the door to the door frame can be obtained: from the door to the door frame.

[0104] At the same time, based on the inversion path, other paths except the main path are updated to obtain the fourth kinematic model, such as Figure 2 (b) shown.

[0105] It should be noted that the joints under the tree representation describe the relative position of the child node relative to the parent node and the motion constraints. Therefore, kinematic inversion is not equivalent to simply exchanging the parent-child relationship between two adjacent rigid bodies. The transformation between them must also be updated appropriately.

[0106] In the actual execution process, we can first use the depth-first algorithm to find the branch path connecting the root node and the grasping node as the trunk path; then perform kinematic inversion on the joints on the trunk path, reverse their parent-child relationship, and update the parent-child node space transformation matrix to reverse the positive and negative upper and lower limits; finally, update the kinematic relationship of other branches to ensure the correctness of the geometric structure.

[0107] After kinematic inversion, the original grab node becomes the root node of the executed object 430 .

[0108] It can be understood that, when there are additional constraints between the operated object and the environment, it will not affect the construction of the fourth kinematic model.

[0109] Continue to refer Figure 2 In some embodiments, constructing a target kinematic model based on the first kinematic model, the fourth kinematic model, and the third kinematic model may include:

[0110] Connecting the first kinematic model and the fourth kinematic model via a virtual joint;

[0111] The third kinematic model is connected to the first kinematic model to construct a target kinematic model.

[0112] In this embodiment, the virtual joint is used to connect the kinematic structure corresponding to the actuator 410 and the kinematic structure corresponding to the executed object 430 .

[0113] It can be understood that by inverting the second kinematic model, the root node of the fourth kinematic model becomes the grasping node of the agent, that is, the correspondence between the end effector of the agent and the executed node of the executed object 430 is realized.

[0114] The virtual joint is used to simulate the motion constraints between the executed node on the executed object 430 and the end effector of the intelligent agent when the intelligent agent performs the target operation task, such as Figure 2 (c) as shown.

[0115] Among them, the motion constraints are determined based on the target operation task, such as based on the grasping posture.

[0116] For example, the motion constraint may include a fixed constraint; that is, when the end effector grasps the object 430, the relative position between the two is not allowed to change during the mobile operation task, which is most suitable for maintaining the initial grasping posture.

[0117] In some embodiments, the motion constraints in the virtual grasping joints may be reduced, and the degree of freedom of the kinematic model may be increased to obtain a more flexible motion space.

[0118] In the actual implementation process, continue to refer to Figure 2 (c) The third kinematic model Access to the first kinematic model Connecting the first kinematic model via virtual joints The end effector node and the fourth kinematic model , thus obtaining a complete serial kinematic chain.

[0119] It can be understood that the target kinematic model constructed through the embodiments of the present application is a universal kinematic model. The executed objects and agent structures with different motion structures do not lose generality. Various objects can be operated on multiple agent configurations. There is no need to design planning strategies separately for different operation tasks. It can be applied to any type of agent to perform any task and is suitable for any scenario, which can significantly reduce the cost of planning strategy design.

[0120] Step 120: Determine the target action sequence and motion constraints corresponding to the agent based on the environment information and target task information in the target scene;

[0121] In this step, the target scene can be any scene, including but not limited to: home intelligent scene, factory scene, shopping mall scene, etc.

[0122] For example, it can be used for food delivery scenarios in restaurants, household cleaning scenarios, and material transportation scenarios in factories.

[0123] Environmental information is used to represent scene-related information in the target scene, including but not limited to the scene structure, the position coordinates of each object in the scene, and the relationship between objects.

[0124] In some embodiments, the environmental information may include actual environmental information and virtual environmental information; wherein the actual environmental information is environmental information under a real scene, and the virtual environmental information is simulated environmental information used to simulate a real scene.

[0125] The target task information is the relevant information of the target task that the agent needs to perform in the target scene, including but not limited to: grasping, placing, pushing and pulling, cleaning, and navigation.

[0126] Taking the target scene as "home agent scene" and the target task information as "the agent moves the pencil placed in the living room to the drawer in the bedroom" as an example, the environmental information may include: the structural relationship between the living room and the bedroom, the dimensions of the living room and the bedroom, the shape, size, and position information of the door in the bedroom, the structure, size, and position information of the drawer, and the obstacle information in the living room and the bedroom, such as the bed, sofa, and wall.

[0127] Environmental information can be obtained through sensors, such as through image sensors or radar measurements, or can also be obtained through drawings, which is not limited in this application.

[0128] The acquired environmental information can be stored in a local or cloud database and retrieved when needed later.

[0129] The target action sequence is a series of continuous action states required for the agent to perform the target task information.

[0130] Continuing with the example of "the agent moves the pencil placed in the living room to the drawer in the bedroom", the target action sequence may include "navigating to the pencil - operating the actuator 410 and picking up the pencil - navigating to the bedroom drawer and placing the pencil in the drawer using the actuator 410."

[0131] The motion constraint is a constraint on the target kinematic model. The target kinematic model is adjusted through the motion constraint so that the target kinematic model can achieve an action state that matches the target task information.

[0132] In some embodiments, when there are additional constraints between the executed object 430 and the environment, such as when the executed object 430 is fixed to the ground, the motion constraint can be determined based on the connection relationship between the executed object 430 and the ground.

[0133] In some embodiments, step 120 may include: determining at least one of a collision constraint, an environmental constraint, and a task constraint based on the environmental information and the target task information.

[0134] In this embodiment, the collision constraint is an obstacle constraint in the target scene to prevent the intelligent agent from colliding with obstacles.

[0135] The environmental constraints are the constraints between the actual environment and the target scene, including its forward kinematics and the physical constraints between the executed object 430 and the environment.

[0136] Task constraints are constraints between the target task and the current posture of the agent.

[0137] It is understandable that the target scene and target task information are different, and the corresponding motion constraints may also be different.

[0138] In the actual execution process, the motion constraints for executing the target task in the scene may be determined based on the environmental information and target task information of the target scene.

[0139] After the motion constraint is obtained, the motion constraint is added to the target kinematics model to adjust specific parameters of the target kinematics model, thereby controlling the actuator 410 and the moving mechanism 420 to adjust to corresponding postures.

[0140] In some embodiments, motion constraints may also include: joint range of motion constraints, speed constraints, and upper and lower limit constraints on acceleration, etc., which may be added based on actual conditions and are not limited in this application.

[0141] The following is an explanation of how to determine the target action sequence.

[0142] In some embodiments, step 120 may include:

[0143] Based on the target task information, determining at least one of the intelligent agent displacement action information, the intelligent agent grasping action information and the executed object displacement information;

[0144] A target action sequence is determined based on at least one of the intelligent agent displacement action information, the intelligent agent grasping action information, the displacement information of the executed object, and the environmental information.

[0145] In this embodiment, the target action sequence may include action states arranged in sequence based on time order.

[0146] The intelligent agent displacement action information is GoTo(chain,q1,q2), which moves the actuator 410chain from the current position q1 to the position q2, and this action does not need to interact with the environment; GoTo can represent tasks that do not require changes to the kinematic model, such as the intelligent agent navigation task, which is a typical task of this type.

[0147] The agent's grasping action information is Pick(object, state, chain), which moves the actuator 410chain to the target state state and grasps the executed object 430object, and expands the current chain structure by adding virtual joints after kinematic inversion; Pick can ensure the tasks of the agent's interaction with the environment, such as picking up objects, grabbing door handles, or picking up tools.

[0148] The displacement information of the executed object is Place (object, state, chain), which moves the executed object 430object connected to the actuator 410chain to the target posture state and disconnects the current chain at the virtual joint; Place represents the task of the agent stopping interacting with the environment, such as putting down an object, opening the door to a certain angle and releasing the gripper, and using tools to operate other objects.

[0149] It can be understood that through the agent's displacement action information, the agent's grasping action information and the executed object's displacement information, the changes in the kinematic model can be regarded as different action states, thereby covering most mobile operation tasks and effectively describing most tasks that change the environment's kinematic state, providing a basis for tasks and motion planning methods for diverse operation objects.

[0150] In the actual execution process, the kinematic model of the intelligent agent can be decomposed into multiple different action states (such as GoTo, Pick, and Place, etc.) based on the actual target task information, and then the different action states can be connected based on the time sequence to obtain the target action sequence.

[0151] It can be understood that the target action sequence changes accordingly with the change of the target task information.

[0152] According to the intelligent body control method provided in the embodiment of the present application, the kinematic model is decomposed into multiple action states based on the target task information, which can effectively cover most mobile operation tasks, thereby effectively describing the tasks of various kinematic states; on this basis, the action states are connected based on the time sequence to obtain the target action sequence, which has high accuracy and precision, and is particularly suitable for complex kinematic models, and thus suitable for complex execution actions.

[0153] like Figure 3 As shown, in some embodiments, step 120 may include:

[0154] Based on the environment information and the target task information, multiple candidate action sequences of the agent in the next target period are determined;

[0155] The best candidate action sequence among the multiple candidate action sequences is determined as the target action sequence in the next target time period.

[0156] In this embodiment, the target action sequence may include symbolic actions arranged in sequence based on a time sequence.

[0157] The next target time period may include a time period corresponding to when the agent performs the next step operation; or, the next target time period may also include multiple sub-time periods corresponding to when the agent performs the subsequent multiple steps of operation.

[0158] The multiple candidate action sequences are different.

[0159] The target action sequence is the entire action sequence corresponding to the next target period.

[0160] In this embodiment, multiple candidate action sequences of the agent in the next target period can be obtained based on the environmental information and the target task information, and the optimal sequence can be selected from the multiple candidate action sequences as the target action sequence.

[0161] It can be understood that the optimal candidate action sequence can be the action sequence with the highest efficiency, such as the candidate action sequence with the shortest completion time; or the candidate action sequence with the shortest path; or the candidate action sequence with the least action states involved. The specific selection can be based on actual conditions and is not limited in this application.

[0162] According to the intelligent body control method provided in the embodiment of the present application, the target action sequence within the next target time period is intelligently planned through environmental information and target task information, which can further improve the rationality and accuracy of the acquired target action sequence, realize advance control of the intelligent body, and help improve the control effect of the intelligent body.

[0163] In some embodiments, determining the best candidate action sequence among multiple candidate action sequences as the target action sequence in the next target period may include:

[0164] Receiving a first input from a user; the first input is used to determine an optimal candidate action sequence;

[0165] In response to the first input, an optimal candidate action sequence is determined as a target action sequence in a next target time period.

[0166] In this embodiment, the first input is used to determine a target action sequence.

[0167] The first input may be in at least one of the following ways:

[0168] First, the first input may be a touch operation, including but not limited to a click operation, a slide operation, and a press operation.

[0169] In this implementation, receiving the first input from the user may be receiving a touch operation of the user in the display area of ​​the terminal display screen.

[0170] Secondly, the first input may be a physical key input.

[0171] In this embodiment, the terminal body is provided with corresponding physical buttons, and receiving the first input from the user may be receiving the operation of the user pressing the corresponding physical button; the first input may also be a combined operation of pressing multiple physical buttons at the same time.

[0172] Third, the first input may be voice input.

[0173] In this implementation, upon receiving a voice message such as "select sequence X", the terminal may determine sequence X as the target action sequence.

[0174] Of course, in other embodiments, the first input may also be in other forms, including but not limited to character input, etc., which can be determined according to actual needs and is not limited in this embodiment of the present application.

[0175] According to the intelligent agent control method provided in the embodiment of the present application, multiple candidate action sequences of the intelligent agent within the next target time period are generated based on environmental information and target task information. The user can select the best action sequence from them based on the first input, allowing the user to perform customized and personalized operations, with high flexibility and personalization.

[0176] In some embodiments, the next target period may include a plurality of consecutive sub-target periods;

[0177] Determining multiple candidate action sequences of the agent in the next target period based on the environment information and the target task information may include: determining multiple sub-candidate action sequences corresponding to each sub-target period of the agent based on the environment information and the target task information;

[0178] Determining the best candidate action sequence among the multiple candidate action sequences as the target action sequence in the next target period may include:

[0179] Determine the best sub-candidate action sequence among the multiple sub-candidate action sequences corresponding to each sub-goal period as the sub-goal action sequence corresponding to the sub-goal period;

[0180] Based on the arrangement order of multiple consecutive sub-goal time periods, the sub-goal action sequences corresponding to each sub-goal time period are spliced ​​to obtain the target action sequence in the next target time period.

[0181] In this embodiment, the next target period may include a plurality of consecutive sub-target periods, each of which corresponds to an execution step.

[0182] In the actual execution process, based on the environmental information and target task information, multiple sub-candidate action sequences corresponding to each step when the intelligent agent performs subsequent multi-step operations can be obtained; then the best sub-candidate action sequence in the sub-target time period is obtained from each step, and the best sub-candidate action sequence in each sub-target time period is combined to obtain the target action sequence.

[0183] According to the intelligent agent control method provided in the embodiment of the present application, the optimal sub-action sequence corresponding to each step of the intelligent agent when performing subsequent multi-step operations is intelligently planned in advance through environmental information and target task information, which can achieve rapid advance planning of multi-step paths, help improve control efficiency, and thus improve the execution efficiency of the intelligent agent.

[0184] In some embodiments, determining the best candidate action sequence among multiple candidate action sequences as the target action sequence in the next target period may include:

[0185] Conduct feasibility judgment on each candidate action sequence to obtain at least one feasible action sequence;

[0186] Based on the motion input costs corresponding to each feasible action sequence, a target action sequence is obtained from at least one feasible action sequence.

[0187] In this embodiment, the feasibility judgment is used to judge whether the candidate action sequence is feasible.

[0188] In the actual execution process, the candidate action sequence can be input into the constructed target kinematic model to obtain path planning, and the feasibility judgment can be made based on the path planning, so as to in turn screen the candidate action sequence based on the judgment result.

[0189] Continue to refer Figure 4 , taking the target task "the agent is located at the side of the refrigerator door or in front of the refrigerator and then opens the refrigerator door to take the cup from the refrigerator" as an example, when the agent is in front of the refrigerator door, the candidate action sequence is opening the refrigerator door in front of the refrigerator door - picking up the cup from the refrigerator, Figure 4It can be seen that there is no obstacle between the candidate action sequences, so the candidate action sequence is preliminarily considered feasible; when the agent is located on the side of the refrigerator door, the candidate action sequence is opening the refrigerator door on the side - picking up the cup from the refrigerator. There is an open refrigerator door (i.e., an obstacle) between the candidate action sequences, which prevents the agent from executing the next action state "picking up the cup from the refrigerator", so the candidate action sequence is considered infeasible.

[0190] The motion input cost is used to characterize the cost that an agent needs to incur when performing a task based on each possible action sequence.

[0191] It can be understood that the smaller the movement input cost is, the lower the cost required to execute the task based on the feasible action sequence is, and the more reasonable the sub-candidate movement path is approximately considered to be.

[0192] For example, in the actual execution process, a search-based path planning algorithm can be used to obtain candidate action sequences, which are then used for motion planning. In the mobile operation task, the moving distance of the mobile mechanism 420 is much greater than the moving distance of the actuator 410. If you want to avoid falling into the local optimum, you can consider using a search path planning algorithm to provide initial values ​​only for the mobile platform. Through search path planning (such as A*), the space is discretized and dimensionally reduced in advance. The mobile mechanism 420 has only three degrees of freedom, which can greatly reduce the computational cost of obtaining the initial value. Since the mobile mechanism 420 has a large range of movement, it can help the actuator 410 escape from the local optimum while escaping from the local optimum.

[0193] According to the intelligent agent control method provided in the embodiment of the present application, by screening candidate action sequences based on feasibility and input cost, the target action sequence finally obtained can be optimal, further improving the precision and accuracy of control.

[0194] Step 130: Control the agent based on the target action sequence, motion constraints and target kinematic model.

[0195] In this step, the target kinematic model can be adjusted based on the target action sequence and motion constraints to obtain the kinematic characteristics of the agent performing the target task in the target scene, and the agent can be controlled based on the obtained kinematic characteristics.

[0196] It is understandable that the target action sequence and motion constraints corresponding to different scenes or tasks may also be different. By simply inputting the target action sequence and motion constraints into the target kinematic model, the relevant parameters in the target kinematic model can be adjusted accordingly, thereby achieving flexible control of the intelligent agent.

[0197] In some embodiments, step 130 may further include:

[0198] Adjust the target kinematic model based on the target action sequence and motion constraints to obtain the motion trajectory corresponding to the agent and the position information of the target position in the motion trajectory;

[0199] Control the agent based on motion trajectory and posture information.

[0200] In this embodiment, the position information includes posture information.

[0201] Continue to refer Figure 3 After obtaining the target kinematic model (i.e., completing VTKC modeling) based on steps 110 and 120, as well as the motion constraints and target action sequences for executing the target task information in the target scene, the motion constraints and the target action sequences are respectively input into the target kinematic model for motion planning, thereby generating a motion trajectory and the pose information at the target position in the motion trajectory.

[0202] In some embodiments, the motion planning problem can be described as: Find a collision-free path q 1:T ∈Q free , from the initial pose q1∈Q of the kinematic chain free Reach the target state q T ∈Q free ; where Q describes the joint space of the kinematic chain, and the dimension of Q is the degree of freedom; its subspace It is a space that the agent and the executed object 430 can freely traverse without colliding with the environment or themselves.

[0203] It should be noted that for motion planning, usually use To describe the state of the environment ( and Q are different).

[0204] The action space defines the transition rules between environment states: The action Change the environment status from Changes to

[0205] Motion planning is used to find a task sequence a 1:N , to change the environment from the initial state Transition to target state in It indicates the set where the target is located.

[0206] In task and motion planning, the motion planning problem needs to be generated sequentially according to the actions in the task sequence (i.e., action sequence).

[0207] For example, for a mobile operation task, the environmental information that needs to be provided in the planning domain includes: 1) the state of the kinematic chain in its joint space, which describes the position of the mobile mechanism 420, the posture of the actuator 410, or the state of the object 430 to be executed; 2) the posture of the object 430 to be executed or the task target posture, which includes the posture or the state of the articulated object in its joint space according to the designed task constraints. This information also implies the forward kinematics of the mobile actuator 410. task 3) whether the agent end effector can grasp freely, or which executed objects 430 the agent end effector is connected to. This information implies the state of the kinematic chain structure.

[0208] In the actual execution process, we can first define the actions that the agent can perform and the possible results based on the planning domain of the logical language, where the problem description expresses the target scenario and target task information of the agent;

[0209] Through intelligent planning, the target action sequence that the agent needs to execute is obtained; it can be understood that the actions in the target action sequence themselves determine the construction mode of the motion chain (GoTo, Pick, Place) and its own constraints, and the parameters in the actions guide the construction of task constraints in the motion planning problem;

[0210] Then, through the integrated modeling method and the target kinematic model of the intelligent agent and the executed object 430, the target action sequence is converted into multiple motion planning problems for the motion chain. These motion planning problems are solved according to the action sequence using the motion planning algorithm to obtain the motion trajectory of the intelligent agent in the multi-step operation task.

[0211] After obtaining the motion trajectory and posture information, the intelligent agent is controlled to move according to the motion trajectory, and when it moves to the target position, the intelligent agent is controlled to perform corresponding actions based on the posture information corresponding to the target position to perform the corresponding execution task.

[0212] During the research and development process, the inventors discovered that in related technologies, the control of intelligent agents can often only be applied to specific scenarios, or can only perform specific operations; when the application scenario or execution task changes, the algorithm in the intelligent agent model and the control source code need to be changed accordingly to adjust the intelligent agent accordingly, which places high requirements on the operator and has poor versatility.

[0213] In the present application, by constructing a joint kinematic relationship between the actuator 410-the moving mechanism 420 and the executed object 430, the linkage modeling among the three can be achieved; on this basis, the motion constraints and motion sequences determined based on the actual execution task and the actual environment are added to the target kinematic model, and the motion posture of the intelligent body can be changed accordingly with the change of the scene or task without changing the source code. The operation is simple and convenient, the user threshold is low, and it has high versatility and universality.

[0214] In addition, by constructing a joint kinematic relationship between the actuator 410, the mobile mechanism 420 and the executed object 430, the linkage modeling between the three can be realized to obtain the target kinematic model. The model has high accuracy and is suitable for intelligent body control in long tasks or complex environments, and has high control precision.

[0215] The target kinematic model provided by the embodiment of the present application can provide a complete serial motion chain, which can freely extend or change its own structure according to the task sequence corresponding to the target task, interact with various different objects, and even realize the use of intelligent tools, that is, to regard the operated object as a part of itself to operate other objects. By uniformly converting the target task into a continuous motion planning problem, the mapping mechanism between the task sequence and the motion trajectory is effectively simplified, and the versatility and scalability of the intelligent autonomous planning method are improved.

[0216] According to the intelligent agent control method provided in the embodiment of the present application, the target kinematic model is obtained by integrating the different motion units of the intelligent agent itself and the motion structure of the executed object 430, and then the target action sequence and motion constraints obtained based on the environmental information and target task information in the target scene are added to the target kinematic model, so that the target kinematic model generates control instructions in a specific scene to control the intelligent agent to perform corresponding actions, thereby realizing efficient and coordinated movement of the intelligent agent. In the case of changes in actual application scenarios or execution tasks, the motion posture of the intelligent agent can be adjusted accordingly without modifying the source code of the intelligent agent to achieve effective control of the intelligent agent in any scene. The method has high versatility, is suitable for any complex execution environment, and has high control accuracy, so that the intelligent agent can perform complex operation tasks.

[0217] Continue to refer Figure 3 In some embodiments, adjusting the target kinematic model based on the target action sequence and the motion constraint, obtaining the motion trajectory corresponding to the agent and the pose information at the target position in the motion trajectory may also include:

[0218] The target kinematic model is adjusted based on the target action sequence and motion constraints, and the initial motion trajectory corresponding to the intelligent agent is obtained using the sampling planning algorithm;

[0219] The initial motion trajectory is optimized to obtain the final motion trajectory.

[0220] In this embodiment, the result of the sampling-based planning algorithm can be input as an initial value into the trajectory optimization method, so that the motion trajectory finally obtained has the completeness of the sampling-based planning method and can effectively ensure the smoothness and constraint retention of the motion trajectory.

[0221] The following describes how to construct collision constraints, environmental constraints, and task constraints.

[0222] 1. Collision Constraints

[0223] In some embodiments, determining at least one of a collision constraint, an environmental constraint, and a task constraint based on the environmental information and the target task information may include:

[0224] A signed distance field function is constructed based on obstacle information in the environment information. The signed distance field function is used to characterize the distance and gradient between each position point in the target scene and the closed surface of the obstacle.

[0225] Construct collision constraints based on the signed distance field function.

[0226] In this embodiment, motion planning may be performed in a trajectory optimization manner.

[0227] The signed distance field function is used to characterize the distance and gradient between each position point in the target scene and the closed surface of the obstacle.

[0228] Signed Distance Field (SDF) is an implicit description of a three-dimensional surface. For a closed surface including a collision volume, an SDF function SDF() is established. SDF() can calculate the distance between a coordinate point x in space and the surface. The sign of success or failure is used to indicate whether the coordinate point x is inside the surface. The gradient of the field at coordinate point x is calculated.

[0229] In this embodiment, the distance and gradient between any objects can be calculated through SDF, which provides a basis for trajectory optimization.

[0230] During the research and development process, the inventors also discovered that in the related technology, there is a method of motion planning by approximating the collision volume with a ball, thereby achieving obstacle avoidance in the optimization process. However, this method is not optimal for volume approximation, especially in narrow and crowded environments, which affects the obstacle avoidance effect.

[0231] In the present application, by adopting collision constraints based on signed distance field (SDF), motion planning in narrow and crowded environments can be achieved, thereby improving obstacle avoidance effects.

[0232] In some other embodiments, the motion plan may also be determined based on sampling.

[0233] In this embodiment, Q free It can be obtained by sampling in the joint space through rejection sampling. For sampling in a highly restricted environment, it is possible to focus on sampling near small spaces and reduce sampling in wide spaces to balance computing resources.

[0234] According to the intelligent agent control method provided in the embodiment of the present application, the collision constraint is constructed through the signed distance field function, which can accurately describe the closed surface of the collision volume, and is particularly suitable for obstacle avoidance control in narrow and crowded environments, thereby further improving the accuracy and precision of the intelligent agent path optimization.

[0235] 2. Task Constraints

[0236] In some embodiments, determining at least one of a collision constraint, an environmental constraint, and a task constraint based on the environmental information and the target task information may include:

[0237] Mapping the joint space of the agent at the target position to the task target space of the executed object 430 to obtain the real-time state value;

[0238] Construct task constraints based on real-time status values ​​and target task information.

[0239] In this embodiment, the task constraint may be an inequality constraint.

[0240] For example, based on the real-time state value and target task information, task constraints can be constructed in the following constraint form:

[0241]

[0242] Among them, f task (q T ) is the real-time state value; g is the target task information; ξ goal The target accuracy can be customized by the user and is not limited in this application.

[0243] It is understandable that the real-time state value f task (q T ) is task-specific and is constrained by mapping the joint space into the task goal space.

[0244] Taking the pick-and-place task performed by the agent as an example, through the function f task (·) Map the joint space to the end effector pose of the agent, f task That is, the forward kinematics of the mobile actuator 410 .

[0245] Take the example of an agent manipulating an articulated object, such as an agent opening a door, f task (·) The joint space is mapped into its sub-joint space, i.e., the joints of the door rotation axis, rather than the joint poses in the task space generated by the planner.

[0246] 3. Environmental Constraints

[0247] Environmental constraints can be expressed as equality constraints, which look like this:

[0248]

[0249] Among them, h chain (q t ) is the environmental constraint.

[0250] The constraints describe the kinematic structure, including its forward kinematics and the physical constraints between the executed object 430 and the environment.

[0251] For example, the object to be executed 430 is fixed on the ground; the end effector grasps the cylindrical handle to form a rotation joint at a connectable position; the box is pushed on the ground and the differential constraints of the non-omnidirectional wheeled mobile mechanism 420 are applied.

[0252] Violation of environmental constraints may damage the executed object 430 or the agent and cause execution failure.

[0253] According to the intelligent body control method provided in the embodiment of the present application, the motion constraints in specific scenarios and specific tasks are determined based on the environmental information and target task information in the target scenario, and the motion constraints are added to the general target kinematic model to achieve control of the intelligent body in specific scenarios and specific tasks. During each application, there is no need to rebuild the target kinematic model, only the motion constraints need to be adjusted based on the actual situation. The operation is simple and convenient, the usage threshold is low, and it is suitable for a wide range of application scenarios, with high control efficiency and versatility.

[0254] In this application, by introducing motion constraints from multiple different angles to constrain the target kinematic model, the model precision and accuracy of the target kinematic model can be significantly improved, which is suitable for any scenario, especially for precise control of intelligent bodies in narrow scenarios.

[0255] The intelligent agent control method provided in the embodiment of the present application can be executed by an intelligent agent control device. In the embodiment of the present application, the intelligent agent control device provided in the embodiment of the present application is described by taking the intelligent agent control method executed by the intelligent agent control device as an example.

[0256] The embodiment of the present application also provides an intelligent body control device.

[0257] like Figure 5 As shown, the intelligent agent control device includes: a first processing module 510 , a second processing module 520 and a third processing module 530 .

[0258] The first processing module 510 is used to construct a target kinematic model based on the actuator information of the agent, the executed object information of the executed object and the mobile mechanism information corresponding to the agent;

[0259] The second processing module 520 is used to determine the target action sequence and motion constraints corresponding to the agent based on the environmental information and target task information in the target scene;

[0260] The third processing module 530 is used to control the intelligent agent based on the target action sequence, motion constraints and target kinematic model.

[0261] According to the intelligent agent control device provided in the embodiment of the present application, the target kinematic model is obtained by integrating the different motion units of the intelligent agent itself and the motion structure of the executed object, and then the target action sequence and motion constraints obtained based on the environmental information and target task information in the target scene are added to the target kinematic model, so that the target kinematic model generates control instructions in a specific scene to control the intelligent agent to perform corresponding actions, thereby realizing efficient coordinated movement of the intelligent agent. When the actual application scene or the execution task changes, the motion posture of the intelligent agent can be adjusted accordingly without modifying the source code of the intelligent agent to realize effective control of the intelligent agent in any scene. The intelligent agent has high versatility, is suitable for any complex execution environment, and has high control accuracy, so that the intelligent agent can perform complex operation tasks.

[0262] In some embodiments, the first processing module 510 may also be used to:

[0263] Constructing a first kinematic model based on the actuator information, constructing a second kinematic model based on the executed object information, and constructing a third kinematic model based on the moving mechanism information;

[0264] Performing inversion processing on the second kinematic model to obtain a fourth kinematic model;

[0265] A target kinematic model is constructed based on the first kinematic model, the fourth kinematic model and the third kinematic model.

[0266] In some embodiments, the first processing module 510 may also be used to:

[0267] Based on the second kinematic model, a plurality of nodes of the executed object and a plurality of connection paths between the plurality of nodes are acquired;

[0268] Determine the path between the root node and the grab node in the multiple connection paths as the main path;

[0269] Kinematic inversion is performed on the trunk path, and other connection paths except the trunk path among the multiple connection paths are updated to obtain a fourth kinematic model.

[0270] In some embodiments, the first processing module 510 may also be used to:

[0271] Connecting the first kinematic model and the fourth kinematic model via a virtual joint;

[0272] The third kinematic model is connected to the first kinematic model to construct a target kinematic model.

[0273] In some embodiments, the second processing module 520 may also be used to:

[0274] At least one of a collision constraint, an environmental constraint, and a task constraint is determined based on the environmental information and the target task information.

[0275] In some embodiments, the second processing module 520 may also be used to:

[0276] A signed distance field function is constructed based on obstacle information in the environment information. The signed distance field function is used to characterize the distance and gradient between each position point in the target scene and the closed surface of the obstacle.

[0277] Construct collision constraints based on the signed distance field function.

[0278] In some embodiments, the second processing module 520 may also be used to:

[0279] Map the joint space of the agent at the target position to the task target space of the object being executed to obtain the real-time state value;

[0280] Construct task constraints based on real-time status values ​​and target task information.

[0281] In some embodiments, the second processing module 520 may also be used to:

[0282] Based on the target task information, determining at least one of the intelligent agent displacement action information, the intelligent agent grasping action information and the executed object displacement information;

[0283] A target action sequence is determined based on at least one of the intelligent agent displacement action information, the intelligent agent grasping action information, the displacement information of the executed object, and the environmental information.

[0284] In some embodiments, the second processing module 520 may also be used to:

[0285] Based on the environment information and the target task information, multiple candidate action sequences of the agent in the next target period are determined;

[0286] The best candidate action sequence among the multiple candidate action sequences is determined as the target action sequence in the next target time period.

[0287] In some embodiments, the third processing module 530 may also be used to:

[0288] Adjust the target kinematic model based on the target action sequence and motion constraints to obtain the motion trajectory corresponding to the agent and the position information of the target position in the motion trajectory;

[0289] Control the agent based on motion trajectory and posture information.

[0290] The intelligent body control device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, an intelligent body, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0291] The intelligent body control device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0292] The intelligent control device provided in the embodiment of the present application can achieve Figures 1 to 5 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0293] like Figure 4 As shown, an embodiment of the present application further provides an intelligent body, comprising: a sensor, an actuator 410, a moving mechanism 420 and an intelligent body control device as described in any of the above embodiments.

[0294] The actuator 410 is used to perform operations such as grabbing, pushing, pulling, and placing.

[0295] The actuator 410 is disposed on the moving mechanism 420 , and the moving mechanism 420 drives the actuator 410 to move.

[0296] The intelligent body control device is electrically connected to the sensor, the actuator 410 and the moving mechanism 420 respectively.

[0297] The intelligent agent control device is used to execute the intelligent agent control method as described in any of the above embodiments.

[0298] According to the intelligent agent provided in the embodiment of the present application, a target kinematic model is obtained by integrating the different motion units of the intelligent agent itself and the motion structure of the executed object, and then the target action sequence and motion constraints obtained based on the environmental information and target task information in the target scene are added to the target kinematic model, so that the target kinematic model generates control instructions in a specific scene to control the intelligent agent to perform corresponding actions, thereby realizing efficient coordinated movement of the intelligent agent. When the actual application scene or the execution task changes, the motion posture of the intelligent agent can be adjusted accordingly without modifying the source code of the intelligent agent to realize effective control of the intelligent agent in any scene. The intelligent agent has high versatility, is suitable for any complex execution environment, and has high control accuracy, so that the intelligent agent can perform complex operation tasks.

[0299] In some embodiments, Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, each process of the above-mentioned intelligent body control method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0300] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0301] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned intelligent body control method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0302] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0303] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned intelligent agent control method when executed by a processor.

[0304] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0305] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned intelligent body control method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0306] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0307] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0308] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0309] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0310] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0311] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. An intelligent agent control method, characterized in that: include: Constructing a target kinematic model based on the actuator information of the agent, the executed object information of the executed object, and the mobile mechanism information corresponding to the agent; Determine the target action sequence and motion constraints corresponding to the agent based on the environment information and target task information in the target scene; The agent is controlled based on the target action sequence, the motion constraints and the target kinematic model.

2. The intelligent agent control method according to claim 1, characterized in that: The step of determining the target action sequence and motion constraints corresponding to the agent based on the environmental information and target task information in the target scene includes: Based on the target task information, determining at least one of the agent displacement action information, the agent grasping action information, and the executed object displacement information; The target action sequence is determined based on at least one of the intelligent agent displacement action information, the intelligent agent grasping action information, the executed object displacement information, and the environmental information.

3. The intelligent agent control method according to claim 1, characterized in that: The step of determining the target action sequence and motion constraints corresponding to the agent based on the environmental information and target task information in the target scene includes: Based on the environment information and the target task information, determining a plurality of candidate action sequences of the agent within a next target time period; An optimal candidate action sequence among the multiple candidate action sequences is determined as the target action sequence in the next target time period.

4. The intelligent agent control method according to any one of claims 1 to 3, characterized in that: The target kinematic model is constructed based on the actuator information of the agent, the executed object information of the executed object and the mobile mechanism information corresponding to the agent, including: Constructing a first kinematic model based on the actuator information, constructing a second kinematic model based on the executed object information, and constructing a third kinematic model based on the moving mechanism information; performing inversion processing on the second kinematic model to obtain a fourth kinematic model; The target kinematic model is constructed based on the first kinematic model, the fourth kinematic model and the third kinematic model.

5. The intelligent agent control method according to claim 4, characterized in that: The performing inversion processing on the second kinematic model to obtain a fourth kinematic model includes: Based on the second kinematic model, acquiring a plurality of nodes of the executed object and a plurality of connection paths between the plurality of nodes; Determine the path between the root node and the grabbing node in the multiple connection paths as a trunk path; Kinematic inversion is performed on the trunk path, and other connection paths among the multiple connection paths except the trunk path are updated to obtain the fourth kinematic model.

6. The intelligent agent control method according to claim 4, characterized in that: The constructing the target kinematic model based on the first kinematic model, the fourth kinematic model and the third kinematic model includes: Connecting the first kinematic model and the fourth kinematic model via a virtual joint; The third kinematic model is connected to the first kinematic model to construct the target kinematic model.

7. The intelligent agent control method according to any one of claims 1 to 3, characterized in that: The step of determining the target action sequence and motion constraints corresponding to the agent based on the environmental information and target task information in the target scene includes: At least one of a collision constraint, an environmental constraint, and a task constraint is determined based on the environmental information and the target task information.

8. The intelligent agent control method according to claim 7, characterized in that: The determining, based on the environmental information and the target task information, at least one of a collision constraint, an environmental constraint, and a task constraint comprises: Constructing a signed distance field function based on the obstacle information in the environmental information, wherein the signed distance field function is used to characterize the distance and gradient between each position point in the target scene and the closed surface of the obstacle; The collision constraint is constructed based on the signed distance field function.

9. The intelligent agent control method according to claim 7, characterized in that: The determining, based on the environmental information and the target task information, at least one of a collision constraint, an environmental constraint, and a task constraint comprises: Mapping the joint space of the agent at the target position to the task target space of the object being executed to obtain a real-time state value; The task constraint is constructed based on the real-time state value and the target task information.

10. The intelligent agent control method according to any one of claims 1 to 3, characterized in that: The controlling the agent based on the target action sequence, the motion constraints and the target kinematic model comprises: Adjust the target kinematic model based on the target action sequence and the motion constraint, and obtain the motion trajectory corresponding to the agent and the pose information at the target position in the motion trajectory; Based on the motion trajectory and the posture information, the intelligent agent is controlled.

11. An intelligent agent control device, characterized in that: include: A first processing module is used to construct a target kinematic model based on the actuator information of the agent, the executed object information of the executed object and the mobile mechanism information corresponding to the agent; A second processing module is used to determine the target action sequence and motion constraints corresponding to the agent based on the environmental information and target task information in the target scene; The third processing module is used to control the intelligent agent based on the target action sequence, the motion constraints and the target kinematic model.

12. An intelligent agent, characterized in that: include: sensor; Executive body; A moving mechanism, wherein the actuator is arranged on the moving mechanism; The intelligent body control device as described in claim 11, wherein the intelligent body control device is electrically connected to the sensor, the actuator and the moving mechanism respectively.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the intelligent agent control method as described in any one of claims 1-10.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intelligent agent control method as described in any one of claims 1-10 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the intelligent agent control method as described in any one of claims 1-10 is implemented.

Images