Robot reference trajectory generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a robot reference trajectory generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target video which is a video that a target object demonstrates a target action; for each video frame in the target video, at least one robot task of the video frame is acquired, joint reverse solution is performed on the at least one robot task, a target joint position of a time node corresponding to the video frame is acquired, and the robot task is generated according to information of a target object in the video frame; the robot task is a function of a robot joint position; and generating a reference trajectory of the target action according to the target joint position of each time node. According to the embodiment of the invention, the robot reference trajectory can be directly obtained from the video of the demonstration action, and the cost of obtaining the robot reference trajectory is reduced.

Description

Technical Field

[0001] The present application belongs to the field of robotics technology, and in particular relates to a method, device, electronic device and storage medium for generating a robot reference trajectory. Background Art

[0002] With the continuous development of robotics technology, the actions that robots can perform are becoming more and more complex.

[0003] At present, robots usually need to rely on the reference trajectory of the action to achieve complex actions. For example, if a humanoid robot wants to achieve a flip, it needs to first obtain the reference trajectory of the flip, and then control the robot's entire body joints based on the reference trajectory to achieve the flip.

[0004] In the prior art, motion capture equipment is usually used to capture the movements of real people to obtain the initial reference trajectory of the target movement; then the initial reference trajectory is optimized to obtain a reference trajectory that satisfies the robot dynamics. Based on the reference trajectory, the robot can control the motion of the whole body joints to perform the target movement. However, this method of obtaining the robot reference trajectory requires real people to demonstrate the movements and requires the use of expensive motion capture equipment for motion capture, which is very costly. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device and storage medium for generating a robot reference trajectory, which can reduce the cost of obtaining the robot reference trajectory.

[0006] In the first aspect, the embodiment of the present application provides a method for generating a reference trajectory of a robot. In this method, a target video is first obtained, which is a video of a target object demonstrating a target action; then, for each video frame in the target video, at least one robot task of the video frame is obtained, and the joint inverse solution of at least one robot task is performed to obtain the target joint position of the time node corresponding to the video frame, and the robot task is generated according to the information of the target object in the video frame, and the robot task is a function of the robot joint position; finally, according to the target joint position of each time node, a reference trajectory of the target action is generated.

[0007] As can be seen from the above, the embodiment of the present application performs joint inverse solution on the robot task of each video frame, and after obtaining the robot joint positions at each time node, the robot reference trajectory of the target action can be generated according to the joint positions at each time node, thereby realizing the acquisition of the robot reference trajectory directly from the video of the demonstration action. There is no need to use motion capture equipment to capture the actions demonstrated by real people to obtain the robot reference trajectory, which is less costly.

[0008] In a possible implementation of the first aspect, in the process of performing joint inverse solution on at least one robot task to obtain the target joint position of the time node corresponding to the video frame, the Jacobian matrix and velocity residual value of each robot task in the mth iteration cycle can be determined; the joint position of the mth iteration cycle is determined according to the Jacobian matrix, the velocity residual value and the joint position of the previous iteration cycle; the error between the actual value and the expected value of each robot task is determined according to the joint position of the mth iteration cycle; if the sum of the errors of each robot task is less than or equal to the preset error, the joint position of the mth iteration cycle is determined as the target joint position; if the sum of the errors of each robot task is greater than the preset error, m=m+1 and then return to the step of determining the Jacobian matrix and velocity residual value of each robot task in the mth iteration cycle until the number of iterations reaches the preset number of iterations.

[0009] In a possible implementation of the first aspect, in the process of determining the joint position of the m-th iteration cycle based on the Jacobian matrix, the velocity residual value and the joint position of the previous iteration cycle, the joint velocity of the m-th iteration cycle can be determined based on the Jacobian matrix and the velocity residual value; and the joint position of the m-th iteration cycle can be determined based on the joint velocity and the joint position of the previous iteration cycle.

[0010] In a possible implementation of the first aspect, in the process of determining the joint position of the mth iteration cycle based on the joint velocity, the target coefficient value of the mth iteration cycle can be determined by line search based on the joint velocity and the joint position of the previous iteration cycle; the target change of the joint position can be determined based on the target coefficient value and the joint velocity; the joint position of the mth iteration cycle can be determined based on the target change of the joint position and the joint position of the previous iteration cycle.

[0011] In this implementation, the target coefficient value is searched by line search, which improves the search speed and further improves the speed of solving the joint position.

[0012] In a possible implementation of the first aspect, in the process of determining the joint velocity of the mth iteration cycle based on the Jacobian matrix and the velocity residuals, a quadratic programming problem can be established based on the Jacobian matrix and the velocity residuals, and the joint velocity of the mth iteration cycle can be obtained by solving the optimization problem of the quadratic programming problem.

[0013] In a possible implementation of the first aspect, in the process of generating a reference trajectory of the target action according to the target joint position at each time node, two adjacent time nodes can be interpolated to obtain the interpolation results of the two adjacent time nodes; based on the interpolation results and the target joint position, the reference trajectory of the target action is obtained.

[0014] In this implementation, a more detailed reference trajectory can be generated by interpolation.

[0015] In a possible implementation of the first aspect, in the process of acquiring at least one robot task of a video frame, information of a target object can be extracted from the video frame; based on the information of the target object, an expected value and a task expression of the robot task are determined to generate the robot task.

[0016] In this implementation, a robot task may be automatically generated according to information of the target object.

[0017] In a second aspect, an embodiment of the present application provides a robot reference trajectory generating device, comprising:

[0018] A video acquisition module is used to acquire a target video, which is a video of a target object demonstrating a target action;

[0019] an inverse solution module, for obtaining at least one robot task of each video frame in the target video, performing inverse dynamics solution on the at least one robot task, and obtaining the target joint position of the time node corresponding to the video frame, wherein the robot task is generated according to the information of the target object in the video frame, and the robot task is a function of the robot joint position;

[0020] The reference trajectory generation module is used to generate the reference trajectory of the target action according to the target joint position at each time node.

[0021] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any method of the first aspect when executing the computer program.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method of any one of the above-mentioned first aspects is implemented.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0024] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A schematic flow chart of a method for generating a reference trajectory for a robot provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of the joint reverse solution process provided in an embodiment of the present application;

[0028] Figure 3 A structural block diagram of a robot reference trajectory generating device provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0031] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0032] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0033] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0034] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0035] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0036] See also Figure 1 , is a schematic flow chart of a method for generating a robot reference trajectory provided in an embodiment of the present application, and the method may include the following steps:

[0037] Step S101: Obtain a target video, where the target video is a video of a target object demonstrating a target action.

[0038] The target object is usually a person, the target action is the action that the robot needs to achieve, and the target video is usually a video of a person demonstrating the target action.

[0039] For example, the robot needs to perform a dance, that is, the action that the robot needs to perform (i.e., the target action) is the action of the dance, and the target video is a video of a person performing the dance. In other words, the person can be seen performing the dance in the target video.

[0040] In some embodiments, the target video can be downloaded from the Internet. For example, if the robot needs to perform a somersault, a video of a human body demonstrating the somersault can be downloaded from the Internet. In other embodiments, the video can also be obtained from local storage. For example, a video of a human body demonstrating the target action can be captured by a camera, and the captured video can be stored locally.

[0041] Step S102: for each video frame in the target video, obtain at least one robot task of the video frame, perform joint inverse solution on at least one robot task, and obtain the target joint position of the time node corresponding to the video frame. The robot task is generated according to the information of the target object in the video frame, and the robot task is a function of the robot joint position.

[0042] The target video is composed of multiple video frames with a time sequence. For example, the frame rate of the target video is 60fps, that is, there are 60 video frames per second. At this time, the time point (i.e. the corresponding time node) of the first video frame is 0ms (milliseconds), the time point of the second video frame is 16.67ms, the time point of the third video frame is 33.34ms, and so on.

[0043] For each video frame, at least one robot task for the video frame is designed according to information of a target object in the video frame to obtain at least one robot task for the video frame.

[0044] Among them, the robot task is a function of the robot joint position, that is, any robot task can be written as a function of the robot joint position F(q), where q is the robot joint position and F is the robot task. For example, the robot task F is the position of the robot's center of gravity or the direction of the trunk tilt, which can be written as a function of the robot's whole body joint position.

[0045] Exemplarily, robot tasks may include but are not limited to the following types:

[0046] Joint space trajectory tasks: specify the desired joint position of a joint or multiple joints.

[0047] Endpoint position and posture tasks: You can specify the position and posture of multiple endpoints at the same time. Endpoints can include palms, soles, and heads.

[0048] Center of gravity task: the position of the center of gravity of the whole body.

[0049] Floating-Base Position and Posture Task: The floating base of a humanoid robot corresponds to the position and posture of the torso.

[0050] Stability-related tasks: For example, calculating the location of the zero moment point (ZMP) or the center of gravity, and then calculating the projection of the ZMP or center of gravity to the sole of the foot to maintain balance.

[0051] Custom tasks: The task can be written as a function of the joint positions, and the Jacobian matrix of the task relative to the joint positions can be calculated using automatic symbolic differentiation.

[0052] The robot task is generated based on the information of the target object in the video frame. The information of the target object may include, for example, the center of gravity position and the end position (such as the position of the end such as the hands and feet) of the target object.

[0053] For example, for a certain video frame, the skeleton of the human body can be extracted from the video frame, the image coordinates of the center of gravity of the skeleton (i.e., the coordinates of the center of gravity in the image) are calculated, and the image coordinates of the center of gravity are mapped to the real three-dimensional space to obtain the three-dimensional coordinates of the center of gravity. Similarly, the image coordinates of the hands and feet can also be obtained based on the skeleton, and then the image coordinates of the hands and feet are mapped to the real three-dimensional space to obtain the three-dimensional coordinates of the hands and feet.

[0054] When designing a robot task for the video frame, it is necessary to design a center of gravity task F1 and an end position posture task F2. At this time, for the robot task F1, the expression of the task can be expressed by the robot's whole body joint position, that is, F1(q); the three-dimensional coordinates of the center of gravity are used as the expected value of the robot task F1 (that is, the expected center of gravity position). At this time, the robot task F1 can be: requiring the robot's center of gravity position to reach the expected position (that is, the three-dimensional coordinates of the center of gravity mentioned above).

[0055] Similarly, for robot task F2, the expression of this task can be expressed by the robot's whole body joint positions, i.e., F2(q); the three-dimensional coordinates of the hands and feet are used as the expected values ​​of robot task F2 (i.e., the expected positions of the hands and feet). Robot task F2 can be: requiring the robot's hands to reach the expected positions (i.e., the three-dimensional coordinates of the hands above), and the robot's feet to reach the expected positions (i.e., the three-dimensional coordinates of the feet above).

[0056] In a specific application, when designing a robot task for each video frame, the robot task can be manually designed or automatically generated.

[0057] When manually designing a robotic task, information such as joint angles and center of gravity locations can be manually specified.

[0058] For example, after identifying the image coordinates of the hand from the video frame, the image coordinates of the hand can be manually completed to obtain the three-dimensional coordinates of the hand; based on the three-dimensional coordinates of the hand, the expected value of the robot task is designed, and the task expression of the robot task (i.e., the function F(q) of the joint position) is obtained.

[0059] In a specific application, when manually designing a robot task, the graphical user interface (GUI) may include the robot task types mentioned above, and the user may select a corresponding robot task type from the GUI as needed.

[0060] If the robot task is a stability-related task or a custom task, symbolic calculation can be used to solve the corresponding Jacobian. Since the pinocchio library supports symbolic calculation, designers can easily add custom tasks.

[0061] If the robot task is a joint space trajectory task, end position and posture task, center of gravity task, or floating base position and posture task, the corresponding Jacobian can be solved using an open source dynamics library (such as the pinocchio library).

[0062] When automatically generating robot tasks, the target object information can be extracted from the video frame, and the expected value and task expression of the robot task can be determined based on the target object information to generate the robot task. For example, the human skeleton image can be automatically extracted from the video frame, and the three-dimensional coordinates of the center of gravity can be automatically calculated based on the human skeleton image to determine the expected value of the robot task.

[0063] If the robot task of the video frame is manually designed, after the manual design is completed, at least one video frame task of the video frame can be obtained; if the machine task is automatically generated, the information of the target object can be automatically extracted, and the robot task can be automatically generated according to the information of the target object to obtain at least one robot task of the video frame.

[0064] After acquiring at least one robot task of the video frame, a joint inverse solution may be performed on the at least one robot task to acquire the target joint position at each time node.

[0065] The correct solution of robot kinematics means that the position of the robot's end effector is solved based on the joint positions, given the joint positions of each joint of the robot.

[0066] The inverse solution of robot joints means: given the position of the robot's end effector, the position of each joint of the robot can be solved based on the position of the robot's end effector.

[0067] In the embodiment of the present application, the expected value of the robot task is known, for example, the expected position of the end effector such as the hand, foot and head, or the expected position of the center of gravity or ZMP of the robot; according to the expected value of the robot task, the joint position of each joint of the robot is solved. In this case, the center of gravity or ZMP of the robot can also be understood as a special end effector.

[0068] For each video frame, when performing joint inverse solution for at least one robot task of the video frame, the joint positions can be changed iteratively so that the actual value of the robot task (i.e., the position of the end, center of gravity or ZMP, etc.) gradually approaches the expected value.

[0069] Among them, in each iteration cycle, after the robot joint position is changed, the actual value of the robot task is calculated according to the changed joint position; the error of each robot task is calculated according to the actual value and expected value of each robot task; if the sum of the errors of all robot tasks is less than or equal to the preset error, the changed joint position in the current iteration cycle can be determined as the target joint position; if the sum of the errors of all robot tasks is greater than the preset error, continue to the next iteration, that is, continue to change the robot joint position.

[0070] The time node corresponding to a video frame refers to the time point where the video frame is located. For example, the target video is a 10s (seconds) video, and there is a video frame every 1s. For the first video frame, its corresponding time node is 0s, and for the second video frame, its corresponding time node is 1s.

[0071] In simple terms, the target video is a video that demonstrates the target action, so the action posture of the target object in each video frame is different. For example, in one video frame, the target object's hands are in the first position, the feet are in the second position, and the center of gravity is in the third position; while in another video frame, the target object's hands are in the fourth position, the feet are in the fifth position, and the center of gravity is in the sixth position.

[0072] For each video frame, the position of the robot's end, center of gravity or ZMP can be determined based on the movement posture of the target object in the video frame (such as the position of hands and feet, etc.), and the target joint position of the robot in the video frame can be calculated through the inverse solution of the joints based on the position of the robot's end, center of gravity or ZMP.

[0073] Step S103: Generate a reference trajectory of the target action according to the target joint position at each time node.

[0074] In specific applications, the target joint positions at each time node can be connected to obtain a reference trajectory. The reference trajectory can include the joint positions at each time node, as well as the position and posture of the floating base. For example, the target video is a 10s (second) video, and each second corresponds to a video frame. In the time period of 0 to 10 seconds, all the joints of the robot correspond to a joint angle every second (i.e., time node).

[0075] In some embodiments, if a more detailed reference trajectory needs to be generated, two adjacent time nodes may be interpolated to obtain the interpolation results of the two adjacent time nodes, and the reference trajectory of the target action may be obtained based on the interpolation results and the target joint position.

[0076] The interpolation may be performed by linear interpolation or according to the smoothness between two time nodes, which is not limited here.

[0077] For example, in the time period of 0 to 10 seconds, all joints have a corresponding target joint position every 1 second. Linear interpolation is performed between 0 and 1 seconds, that is, the corresponding joint position (i.e., interpolation result) is inserted between 0 and 1 second, and the corresponding joint position is inserted between 1 and 2 seconds. And so on, the corresponding joint position is inserted between all two adjacent time nodes. Finally, a more detailed reference trajectory is generated based on the inserted joint position and the target joint position.

[0078] Of course, in other embodiments, interpolation may not be used, but the reference trajectory may be generated directly according to the target joint position at each time point.

[0079] In some embodiments, after generating the reference trajectory of the target action, the reference trajectory can be optimized according to the robot dynamics to obtain an optimized reference trajectory. The robot controls all joints of the whole body according to the optimized reference trajectory to achieve the target action.

[0080] As can be seen from the above, the embodiment of the present application can directly obtain the reference trajectory of the target action from the video demonstrating the target action. Compared with the method of using expensive motion capture equipment to capture the real person's demonstration action to obtain the reference trajectory, the embodiment of the present application does not need to use expensive motion capture equipment, and can directly obtain the reference trajectory from the video, thereby reducing the cost of obtaining the reference trajectory.

[0081] Based on the above embodiments, see Figure 2 The schematic diagram of the joint inverse solution process provided by the embodiment of the present application is shown. The process of performing joint inverse solution on at least one robot task to obtain the target joint position of the time node corresponding to the video frame may include the following steps:

[0082] Step S201, determining the Jacobian matrix and velocity residual value of each robot task in the mth iteration cycle.

[0083] Wherein, m is the number of iterations, which is a positive integer greater than or equal to 1. Usually, the maximum number of iterations can be preset, that is, the maximum value of m can be preset.

[0084] In the embodiment of the present application, any robot task can be written as a function of the joint position, i.e., F(q). In this case, the Jacobian matrix of the robot task relative to the joint position is: The residual value of speed is: Where F* is the expected value of the robot task, and F(q) is the current value of the robot task. In the joint inverse solution process, the joint position is changed by v*δt in each iteration so that the current value (or actual value) of the robot task gradually approaches the expected value. Where v*δt is the change in joint position in each iteration.

[0085] Step S202: Determine the joint position of the mth iteration cycle according to the Jacobian matrix, the velocity residual value and the joint position of the previous iteration cycle.

[0086] It can be understood that if m is equal to 1, that is, in the first iteration cycle, the joint position of the previous iteration cycle can be regarded as the target joint position output by the previous video frame, that is, the target joint position of the previous video frame is used as the joint starting position of the current video frame.

[0087] In the mth iteration cycle, the joint position of the previous iteration cycle and the change in the joint position of the current iteration cycle are added together to obtain the joint position of the mth iteration cycle.

[0088] The change in joint position in the current iteration cycle can be obtained based on the joint velocity. The joint velocity refers to the first derivative of the joint position. At this time, the joint velocity of the mth iteration cycle can be determined based on the Jacobian matrix and velocity residual of each machine task, and then the joint position of the mth iteration cycle can be determined based on the joint velocity and the joint position of the previous iteration cycle.

[0089] In some embodiments, a quadratic programming problem can be established based on the Jacobian matrix and velocity residuals of each robot task, and the joint velocity of the mth iteration cycle can be obtained by solving the optimization problem of the quadratic programming problem.

[0090] For example, assuming that a video frame has N robot tasks, in the mth iteration cycle, the objective function of the quadratic programming can be as follows:

[0091]

[0092] in, is the joint velocity, which is the first-order derivative of the joint position q. task i is the i-th task, w i is the weight of the i-th robot task, J i is the Jacobian matrix of the ith robot task, v i is the residual value of the speed of the i-th robot task, K i is the gain of the ith robot task, which is used to control the speed of the approach.

[0093] In order to ensure that the joint position obtained by the solution of the above objective function is within the limit range, a velocity constraint can be added to the above objective function. For example, for the jth joint, the joint position of the joint needs to be limited to the interval [q min ,q max] If the function is within the range of , two linear constraints need to be added to the objective function. The two linear constraints are: and q j,0 is the current position of the jth joint.

[0094] By solving the objective function of the above quadratic programming (i.e., the quadratic programming problem), that is, by solving the optimization problem of the established quadratic programming problem, the optimal joint speed in the mth iteration cycle can be obtained.

[0095] It should be noted that if the number of robot joints is large, the number of variables in the objective function of the above quadratic programming will also be large. At this time, in order to make the robot task converge, it is necessary to solve the objective function of the above quadratic programming multiple times. In some embodiments, in order to increase the solution speed, a line search method can be used to calculate the joint position of the mth iteration cycle. Specifically, first determine the target coefficient value of the mth iteration cycle through line search based on the joint speed; then determine the target change of the joint position based on the target coefficient value and the joint speed; finally, determine the joint position of the mth iteration cycle based on the target change of the joint position and the joint position of the previous iteration cycle.

[0096] Among them, each time the objective function of the above quadratic programming is solved to obtain the joint velocity Afterwards, you can Calculate the joint position of the current iteration cycle, where k is the coefficient value, q(n) is the joint position of the previous iteration cycle, and q(n+1) is the joint position of the current iteration cycle. At this time, k is unknown.

[0097] After calculating q(n+1), we can calculate the error between the actual value and the expected value of all robot tasks based on the joint position q(n+1). The error will be different for different k values. Select the k value with the minimum error as the target coefficient value. Substitute this target coefficient value into the formula Calculate the joint positions of the current iteration cycle.

[0098] Step S203: Determine the error between the actual value and the expected value of each robot task according to the joint position of the mth iteration cycle.

[0099] Step S204: Determine whether the sum of the errors of each robot task is less than or equal to the preset error. If yes, proceed to step S205; if no, proceed to step S206.

[0100] After obtaining the joint positions of the mth iteration cycle, the actual values ​​of each robot task can be calculated based on the joint positions of the current iteration cycle; for each robot task, the error of the robot task can be determined based on the actual value and expectation of the robot task.

[0101] The sum of the errors of each robot task may be obtained by multiplying the error of each robot task by the weight of the robot task and then summing the results, or the errors of each robot task may be directly added together.

[0102] Step S205: determine the joint position of the mth iteration cycle as the target joint position.

[0103] Step S206, determine whether the number of iterations reaches the preset number of iterations. If yes, proceed to step S207, if not, m=m+1 and then return to the step of determining the Jacobian matrix and velocity residual value of each robot task in the mth iteration cycle.

[0104] For each video frame, starting from the starting position of the joint, the joint position is changed iteratively one by one so that the actual value of the robot task gradually approaches the expected value, and then the target joint position of the video frame is solved.

[0105] For example, a video frame includes three robot tasks, namely F1, F2 and F3. Among them, F1 is the robot's hand to reach position R1, F2 is the robot's foot to reach position R2, and F3 is the robot's center of gravity to reach position R3. That is, the expected value of F1 is R1, the expected value of F2 is R2, and the expected value of F3 is R3.

[0106] In the first iteration cycle, the joint position of the current iteration cycle is determined to be q1 according to the joint starting position and the joint position change of the current iteration cycle, that is, the robot joint moves from the starting position to q1. It can be understood that each joint in the robot's body joints has a starting position, and each joint has a joint position change. According to the starting position and the joint position change, the position of each joint after the change can be determined.

[0107] After obtaining the joint position q1 of the first iteration cycle, the actual value of F1 is determined according to the joint position q1, that is, the actual position of the hand at the current iteration cycle is determined. The error of F1 is calculated according to the actual value of F1 and the expected value of F1.

[0108] Similarly, according to the joint position q1, the actual value of F2 is determined, that is, the actual position of the foot in the current iteration cycle is determined; according to the actual value of F2 and the expected value of F2, the error of F2 can be calculated.

[0109] According to the joint position q1, the actual value of F3 is determined, that is, the actual position of the center of gravity in the current iteration cycle is determined; according to the actual value of F3 and the expected value of F3, the error of F3 can be calculated.

[0110] According to the error of F1, the error of F2 and the error of F3, it is determined whether the error of each robot task is less than or equal to the preset error. If so, it is considered that after the first iteration of the joint position, the robot's hands, feet and center of gravity can be close to the expected position, so the joint position of the first iteration is determined as the target joint position of the video frame; if not, m = m + 1, that is, m = 1 + 1, and the second iteration is performed. At this time, the number of iterations is less than the preset number of iterations.

[0111] In the second iteration cycle, according to the joint position of the previous iteration cycle (ie, q1) and the joint position change of the current iteration cycle, the joint position of the current iteration cycle is determined to be q2, that is, the robot joint moves from q1 to q2.

[0112] According to the joint position q1, the errors between the actual values ​​and the expected values ​​of F1, F2 and F3 are calculated, and the sum of the errors of F1, F2 and F3 is calculated, and it is determined whether the sum of the errors of the three robot tasks is less than or equal to the preset error.

[0113] And so on, until the number of iterations reaches the preset number of iterations, or the sum of the errors of each robot task is less than or equal to the preset error.

[0114] It should be noted that in a specific application, each video frame can be calculated sequentially, that is, after calculating the target joint position of the current video frame, the target joint position of the next video frame is calculated.

[0115] Step S207: redesign the robot task.

[0116] If the robot task has not converged after exceeding the preset number of iterations during the iteration process, the robot task of the video frame needs to be re-set. For example, for a certain video frame, the arm task space task can be changed to the joint space task, and the center of gravity task can be changed to the waist posture task.

[0117] After redesigning the robot task of the video frame, the process may return to step S201 to restart the calculation of the target joint position of the video frame.

[0118] As can be seen from the above, the embodiment of the present application iteratively solves the target joint position of each video frame through the Jacobian matrix and velocity residual value of the robot task, thereby obtaining a reference trajectory from the target video.

[0119] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0120] Corresponding to the robot reference trajectory generation method described in the above embodiment, Figure 3 A structural block diagram of a robot reference trajectory generating device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0121] Reference Figure 3 , the device comprises:

[0122] The video acquisition module 31 is used to acquire a target video, which is a video of a target object demonstrating a target action;

[0123] an inverse solution module 32, for obtaining at least one robot task of each video frame in the target video, performing joint inverse solution on at least one robot task, and obtaining a target joint position at a time node corresponding to the video frame, wherein the robot task is generated according to information of the target object in the video frame, and the robot task is a function of the robot joint position;

[0124] The reference trajectory generating module 33 is used to generate a reference trajectory of the target action according to the target joint position at each time node.

[0125] In one possible implementation, the inverse solution module 32 is specifically used to: determine the Jacobian matrix and velocity residuals of each robot task in the mth iteration cycle; determine the joint position of the mth iteration cycle based on the Jacobian matrix, the velocity residuals and the joint positions of the previous iteration cycle; determine the error between the actual value and the expected value of each robot task based on the joint positions of the mth iteration cycle; if the sum of the errors of each robot task is less than or equal to the preset error, determine the joint position of the mth iteration cycle as the target joint position; if the sum of the errors of each robot task is greater than the preset error, m=m+1 and then return to the step of determining the Jacobian matrix and velocity residuals of each robot task in the mth iteration cycle until the number of iterations reaches the preset number of iterations.

[0126] In a possible implementation, the inverse solution module 32 is specifically used to: determine the joint velocity of the mth iteration cycle according to the Jacobian matrix and the velocity residual; determine the joint position of the mth iteration cycle according to the joint velocity and the joint position of the previous iteration cycle.

[0127] In one possible implementation, the inverse solution module 32 is specifically used to: determine the target coefficient value of the mth iteration cycle through line search based on the joint velocity; determine the target change of the joint position based on the target coefficient value and the joint velocity; determine the joint position of the mth iteration cycle based on the target change of the joint position and the joint position of the previous iteration cycle.

[0128] In a possible implementation, the inverse solution module 32 is specifically used to establish a quadratic programming problem according to the Jacobian matrix and the velocity residual value, and obtain the joint velocity of the mth iteration cycle by solving the optimization problem of the quadratic programming problem.

[0129] In a possible implementation, the reference trajectory generating module 33 is specifically used to: interpolate two adjacent time nodes to obtain interpolation results of the two adjacent time nodes; and obtain a reference trajectory of the target action according to the interpolation result and the target joint position.

[0130] In a possible implementation, the inverse solution module 32 is specifically used to: extract information of the target object from the video frame; determine the expected value and task expression of the robot task according to the information of the target object to generate the robot task.

[0131] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0132] Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 4 As shown, the electronic device 4 of this embodiment includes: at least one processor 40 ( Figure 4 Only one is shown in the figure), a memory 41, and a computer program 42 stored in the memory 41 and executable on the at least one processor 40, wherein the processor 40 implements the steps of any of the above-mentioned method embodiments when executing the computer program 42.

[0133] The electronic device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will appreciate that Figure 4 It is only an example of the electronic device 4 and does not constitute a limitation on the electronic device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0134] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0135] In some embodiments, the memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. In other embodiments, the memory 41 may also be an external storage device of the electronic device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 4. Further, the memory 41 may also include both an internal storage unit of the electronic device 4 and an external storage device. The memory 41 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 41 may also be used to temporarily store data that has been output or is to be output.

[0136] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0137] In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the above method embodiment, which will not be repeated here.

[0138] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0139] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0140] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0142] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0143] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0144] In the embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating a robot reference trajectory, characterized in that: include: Acquire a target video, wherein the target video is a video of a target object demonstrating a target action; For each video frame in the target video, obtaining at least one robot task of the video frame, performing joint inverse solution on the at least one robot task, and obtaining a target joint position at a time node corresponding to the video frame, wherein the robot task is generated according to information of the target object in the video frame, and the robot task is a function of a robot joint position; A reference trajectory of the target action is generated according to the target joint position at each of the time nodes.

2. The method according to claim 1, characterized in that Performing joint inverse solution on the at least one robot task to obtain a target joint position at a time node corresponding to the video frame includes: Determine the Jacobian matrix and velocity residual value of each robot task in the mth iteration cycle; Determine the joint position of the mth iteration cycle according to the Jacobian matrix, the velocity residual value and the joint position of the previous iteration cycle; Determining the error between the actual value and the expected value of each of the robot tasks according to the joint position of the mth iteration cycle; If the sum of the errors of the robot tasks is less than or equal to the preset error, determining the joint position of the mth iteration cycle as the target joint position; If the sum of the errors of each robot task is greater than the preset error, after m=m+1, return to the step of determining the Jacobian matrix and velocity residual value of each robot task in the mth iteration cycle until the number of iterations reaches the preset number of iterations.

3. The method according to claim 2, characterized in that Determining the joint position of the mth iteration cycle according to the Jacobian matrix, the velocity residual value and the joint position of the previous iteration cycle includes: Determine the joint velocity of the mth iteration cycle according to the Jacobian matrix and the velocity residual value; The joint position of the mth iteration cycle is determined according to the joint velocity and the joint position of the previous iteration cycle.

4. The method according to claim 3, characterized in that Determining the joint position of the mth iteration cycle according to the joint velocity and the joint position of the previous iteration cycle includes: According to the joint velocity, determining the target coefficient value of the mth iteration cycle by line search; Determining a target change in joint position according to the target coefficient value and the joint velocity; The joint position of the mth iteration cycle is determined according to the joint position target change and the joint position of the previous iteration cycle.

5. The method according to claim 3, characterized in that Determining the joint velocity of the mth iteration cycle according to the Jacobian matrix and the velocity residual value includes: A quadratic programming problem is established according to the Jacobian matrix and the velocity residual value, and the joint velocity of the mth iteration cycle is obtained by solving the optimization problem of the quadratic programming problem.

6. The method according to any one of claims 1 to 5, characterized in that: Generating a reference trajectory of the target action according to the target joint position at each of the time nodes includes: Interpolate two adjacent time nodes to obtain interpolation results of the two adjacent time nodes; A reference trajectory of the target action is obtained according to the interpolation result and the target joint position.

7. The method according to claim 1, characterized in that Acquiring at least one robot task of the video frame, comprising: Extracting information of the target object from the video frame; According to the information of the target object, the expected value and task expression of the robot task are determined to generate the robot task.

8. A robot reference trajectory generating device, characterized in that: include: A video acquisition module is used to acquire a target video, wherein the target video is a video of a target object demonstrating a target action; an inverse solution module, for obtaining, for each video frame in the target video, at least one robot task of the video frame, performing joint inverse solution on the at least one robot task, and obtaining a target joint position at a time node corresponding to the video frame, wherein the robot task is generated according to information of the target object in the video frame, and the robot task is a function of a robot joint position; The reference trajectory generation module is used to generate a reference trajectory of the target action according to the target joint position at each of the time nodes.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images