Robot reference trajectory generation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a robot reference trajectory generation method and device, electronic equipment and a storage medium. The method comprises: obtaining a target video, the target video being a video in which a target object demonstrates a target action; for each video frame in the target video, obtaining at least one robot task of the video frame, performing inverse kinematics on the at least one robot task to obtain a target joint position of a time node corresponding to the video frame, the robot task being generated according to information of the target object in the video frame, the robot task being a function of a robot joint position; and generating a reference trajectory of the target action according to the target joint position of each time node. The embodiments of the present application can directly obtain a robot reference trajectory from a video in which an action is demonstrated, thereby reducing the cost of obtaining the robot reference trajectory.

Description

Technical Field

[0001] This application belongs to the field of robotics technology, and in particular relates to a method, apparatus, electronic device and storage medium for generating robot reference trajectories. Background Technology

[0002] As robotics technology continues to develop, the actions that robots can perform are becoming increasingly complex.

[0003] Currently, robots typically rely on a reference trajectory to perform complex movements. For example, for a humanoid robot to perform a somersault, it first needs to obtain a reference trajectory for the somersault, and then control all the joints of the robot's body based on that trajectory to achieve the somersault.

[0004] In existing technologies, motion capture equipment is typically used to capture the movements of a live human demonstration to obtain an initial reference trajectory for the target movement. This initial reference trajectory is then optimized to obtain a reference trajectory that satisfies the robot's dynamics. Based on this reference trajectory, the robot can perform full-body joint motion control to execute the target movement. However, this method of obtaining the robot's reference trajectory requires a live human demonstration and the use of expensive motion capture equipment, resulting in high costs. Summary of the Invention

[0005] This application provides a robot reference trajectory generation method, apparatus, electronic device, and storage medium, which can reduce the cost of acquiring robot reference trajectories.

[0006] Firstly, embodiments of this application provide a method for generating a robot reference trajectory. In this method, a target video is first acquired, which is a video of a target object demonstrating a target action. Then, for each video frame in the target video, at least one robot task for each frame is acquired. Inverse joint solving is performed on the at least one robot task to obtain the target joint positions at the corresponding time points of the video frames. The robot tasks are generated based on information about the target object in the video frames, and the robot tasks are functions of the robot joint positions. Finally, a reference trajectory for the target action is generated based on the target joint positions at each time point.

[0007] As can be seen from the above, the embodiments of this application obtain the robot joint positions at each time point by performing inverse joint solving on the robot task of each video frame. Based on the joint positions at each time point, the robot reference trajectory of the target action can be generated, thereby realizing the direct acquisition of the robot reference trajectory from the video of the demonstration action. It is not necessary to use motion capture equipment to capture the action of the real person to obtain the robot reference trajectory, which is more cost-effective.

[0008] In one possible implementation of the first aspect, during the process of performing inverse joint solving on at least one robot task to obtain the target joint position of the time node corresponding to the video frame, the Jacobian matrix and velocity residual of each robot task in the m-th iteration cycle can be determined; based on the Jacobian matrix, velocity residual, and joint position of the previous iteration cycle, the joint position of the m-th iteration cycle is determined; based on the joint position of the m-th iteration cycle, the error between the actual value and the expected value of each robot task is determined; if the sum of the errors of each robot task is less than or equal to a preset error, the joint position of the m-th iteration cycle is determined as the target joint position; if the sum of the errors of each robot task is greater than the preset error, after m = m + 1, the process returns to the step of determining the Jacobian matrix and velocity residual of each robot task in the m-th iteration cycle, until the number of iterations reaches the preset number of iterations.

[0009] In one possible implementation of the first aspect, in the process of determining the joint position of the m-th iteration cycle based on the Jacobian matrix, the velocity residual, and the joint position of the previous iteration cycle, the joint velocity of the m-th iteration cycle can be determined based on the Jacobian matrix and the velocity residual; and the joint position of the m-th iteration cycle can be determined based on the joint velocity and the joint position of the previous iteration cycle.

[0010] In one possible implementation of the first aspect, in the process of determining the joint position of the m-th iteration cycle based on the joint velocity, the target coefficient value of the m-th iteration cycle can be determined by line search based on the joint velocity and the joint position of the previous iteration cycle; the target change amount of the joint position can be determined based on the target coefficient value and the joint velocity; and the joint position of the m-th iteration cycle can be determined based on the target change amount of the joint position and the joint position of the previous iteration cycle.

[0011] In this implementation, the target coefficient value is searched by line search, which improves the search speed and thus improves the solution speed of joint position.

[0012] In one possible implementation of the first aspect, in the process of determining the joint velocity of the m-th iteration cycle based on the Jacobian matrix and the velocity residual, a quadratic programming problem can be established based on the Jacobian matrix and the velocity residual, and the joint velocity of the m-th iteration cycle can be obtained by solving the optimization problem of the quadratic programming problem.

[0013] In one possible implementation of the first aspect, during the process of generating a reference trajectory of the target action based on the target joint position at each time node, interpolation can be performed on two adjacent time nodes to obtain the interpolation result of the two adjacent time nodes; based on the interpolation result and the target joint position, the reference trajectory of the target action is obtained.

[0014] In this implementation, more detailed reference trajectories can be generated through interpolation.

[0015] In one possible implementation of the first aspect, during the process of acquiring at least one robot task from video frames, information about a target object can be extracted from the video frames; based on the information about the target object, the expected value and task expression of the robot task are determined to generate the robot task.

[0016] In this implementation, robot tasks can be automatically generated based on the information of the target object.

[0017] Secondly, embodiments of this application provide a robot reference trajectory generation device, comprising:

[0018] The video acquisition module is used to acquire the target video, which is a video of the target object demonstrating the target action;

[0019] The inverse kinematics module is used to obtain at least one robot task for each video frame in the target video, perform inverse kinematics on at least one robot task, and obtain the target joint position of the time node corresponding to the video frame. The robot task is generated based on the information of the target object in the video frame and is a function of the robot joint position.

[0020] The reference trajectory generation module is used to generate a reference trajectory for the target motion based on the target joint position at each time point.

[0021] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any of the first aspects above.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0023] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in any one of the first aspects above.

[0024] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic flowchart of a robot reference trajectory generation method provided in an embodiment of this application;

[0027] Figure 2 This is a schematic diagram of the joint reverse engineering process provided in an embodiment of this application;

[0028] Figure 3 This is a structural block diagram of the robot reference trajectory generation device provided in the embodiments of this application;

[0029] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0031] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0032] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0033] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0034] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0035] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0036] Please see Figure 1 This is a schematic flowchart of a robot reference trajectory generation method provided in an embodiment of this application. The method may include the following steps:

[0037] Step S101: Obtain the target video. The target video is a video of the target object demonstrating the target action.

[0038] The target object is usually a person, and the target action is the action that the robot needs to perform. The target video is usually a video of a human demonstrating the target action.

[0039] For example, a robot needs to perform a dance; that is, the action the robot needs to perform (i.e., the target action) is the action of that dance, and the target video is a video of a person demonstrating that dance. In other words, the target video shows a person demonstrating that dance.

[0040] In some embodiments, the target video can be downloaded from the Internet. For example, if a robot needs to perform a somersault, a video of a human demonstrating the somersault can be downloaded from the Internet. In other embodiments, it can also be obtained from local storage. For example, a video of a human demonstrating the target action can be captured by a camera and stored locally.

[0041] Step S102: For each video frame in the target video, obtain at least one robot task for the video frame, perform inverse joint solving on at least one robot task, and obtain the target joint position of the time node corresponding to the video frame. The robot task is generated based on the information of the target object in the video frame, and the robot task is a function of the robot joint position.

[0042] The target video is composed of multiple video frames in a time sequence. For example, the frame rate of the target video is 60fps, meaning there are 60 video frames per second. In this case, the time point (i.e., the corresponding time node) of the first video frame is 0ms (milliseconds), the time point of the second video frame is 16.67ms, the time point of the third video frame is 33.34ms, and so on.

[0043] For each video frame, based on the information of the target object in the video frame, design at least one robot task for that video frame to obtain at least one robot task for that video frame.

[0044] Here, the robot task is a function of the robot's joint positions; that is, any robot task can be written as a function F(q) of the robot's joint positions, where q is the robot's joint position and F is the robot task. For example, the robot task F is the position of the robot's center of gravity or the direction of its torso tilt, which can be written as a function of the positions of all the robot's joints.

[0045] For example, robot tasks may include, but are not limited to, the following types:

[0046] Joint space trajectory task: Specify the desired joint position for one or more joints.

[0047] End-effector position and pose task: Multiple end-effector poses and orientations can be specified simultaneously. Examples of end-effectors include hands, feet, and heads.

[0048] Central focus: The position of the body's center of gravity.

[0049] The task of determining the position and orientation of the floating base: The floating base of the humanoid robot corresponds to the position and orientation of the torso.

[0050] Stability-related tasks include calculating the position of the Zero Moment Point (ZMP) or the center of gravity, and then calculating the projection of the ZMP or center of gravity onto the foot to maintain balance.

[0051] Custom tasks: Tasks can be written as functions of joint positions, and the Jacobian matrix of the task relative to the joint positions can be calculated using automatic sign differentiation.

[0052] Robot tasks are generated based on information about target objects in video frames. This information may include, for example, the target object's center of gravity and end-effector positions (e.g., the positions of hands and feet).

[0053] For example, for a specific video frame, a human skeleton can be extracted from the frame. The coordinates of the center of gravity (i.e., the coordinates of the center of gravity in the image) can be calculated, and these coordinates can be mapped to real 3D space to obtain the 3D coordinates of the center of gravity. Similarly, the coordinates of the hands and feet can be obtained from the skeleton image, and then mapped to real 3D space to obtain the 3D coordinates of the hands and feet.

[0054] When designing a robot task for this video frame, it is necessary to design a center of gravity task F1 and an end-effector pose task F2. For robot task F1, the expression can be represented by the positions of all the robot's joints, i.e., F1(q); the three-dimensional coordinates of the center of gravity are used as the expected value of robot task F1 (i.e., the expected center of gravity position). Therefore, robot task F1 can be: requiring the robot's center of gravity to reach the expected position (i.e., the aforementioned three-dimensional coordinates of the center of gravity).

[0055] Similarly, for robot task F2, the expression for this task can be represented by the positions of all the robot's joints, i.e., F2(q); the three-dimensional coordinates of the hands and feet are used as the expected values ​​of robot task F2 (i.e., the expected positions of the hands and feet). Robot task F2 can be: requiring the robot's hands to reach the expected positions (i.e., the three-dimensional coordinates of the hands mentioned above), and the robot's feet to reach the expected positions (i.e., the three-dimensional coordinates of the feet mentioned above).

[0056] In practical applications, when designing robot tasks for each video frame, robot tasks can be designed manually or generated automatically.

[0057] When manually designing robot tasks, you can manually specify information such as joint angles and center of gravity position.

[0058] For example, after identifying the image coordinates of the hand from a video frame, the image coordinates of the hand can be completed by human intervention to obtain the three-dimensional coordinates of the hand; based on the three-dimensional coordinates of the hand, the expected value of the robot task can be designed, and the task expression of the robot task (i.e., the function F(q) of the joint position) can be obtained.

[0059] In practical applications, when manually designing robot tasks, the graphical user interface (GUI) can include the robot task types mentioned above, and users can select the appropriate robot task type from the GUI as needed.

[0060] If the robot task is a stability-related task or a custom task, symbolic computation can be used to solve the corresponding Jacobian problem. Since the pinocchio library supports symbolic computation, designers can easily add custom tasks.

[0061] If the robot task is a joint space trajectory task, an end-effector position and orientation task, a center of gravity task, or a floating base position and orientation task, the corresponding Jacobian can be solved using an open-source dynamics library (such as the pinocchio library).

[0062] When automatically generating robot tasks, information about the target object can be extracted from video frames. Based on this information, the expected value and task expression of the robot task can be determined to generate the robot task. For example, a human skeleton image can be automatically extracted from video frames, and the three-dimensional coordinates of the center of gravity can be automatically calculated based on the human skeleton image to determine the expected value of the robot task.

[0063] If the robot task is designed manually using video frames, at least one video frame task can be obtained after the manual design is completed. If the robot task is generated automatically, the information of the target object can be extracted automatically, and the robot task can be generated automatically based on the information of the target object, so as to obtain at least one robot task with video frames.

[0064] After acquiring at least one robot task from video frames, inverse joint solving can be performed on at least one robot task to obtain the target joint positions at each time point.

[0065] Among them, the forward kinematics of a robot refers to: knowing the joint positions of each joint of the robot, and solving for the position of the robot's end effector based on the joint positions.

[0066] Inverse joint solving of a robot refers to: given the position of the robot's end effector, solving for the position of each joint of the robot.

[0067] In this embodiment, the expected value of the robot task is known, such as the expected position of the end effectors (hands, feet, and head), or the expected position of the robot's center of gravity or ZMP. Based on the expected value of the robot task, the joint positions of each joint of the robot are solved. In this case, the robot's center of gravity or ZMP can also be understood as a special type of end effector.

[0068] For each video frame, when performing joint inverse solving for at least one robot task in that video frame, the joint positions can be iteratively changed to allow the actual values ​​of the robot task (i.e., the position of the end effector, center of gravity, or ZMP) to gradually approach the desired values.

[0069] In each iteration cycle, after changing the robot joint position, the actual value of the robot task is calculated based on the changed joint position; the error of each robot task is calculated based on the actual value and the expected value of each robot task; if the sum of the errors of all robot tasks is less than or equal to the preset error, the changed joint position in the current iteration cycle can be determined as the target joint position; if the sum of the errors of all robot tasks is greater than the preset error, the next iteration continues, that is, the robot joint position is changed again.

[0070] The time point corresponding to a video frame refers to the time at which the video frame is located. For example, if the target video is a 10-second video, with one video frame every second, the time point corresponding to the first video frame is 0 seconds, and the time point corresponding to the second video frame is 1 second.

[0071] Simply put, a target video is a video demonstrating the target's actions, so the target object's posture and movements are different in each video frame. For example, in one video frame, the target object's hand is in position one, its foot is in position two, and its center of gravity is in position three; while in another video frame, the target object's hand is in position four, its foot is in position five, and its center of gravity is in position six.

[0072] For each video frame, based on the action posture of the target object in the video frame (such as the position of the hands and feet), the position of the robot's end effector, center of gravity, or ZMP can be determined. Based on the position of the robot's end effector, center of gravity, or ZMP, the position of the robot's target joint in that video frame can be calculated through inverse joint solving.

[0073] Step S103: Generate a reference trajectory for the target motion based on the target joint position at each time node.

[0074] In practical applications, the target joint positions at various time points can be connected to obtain a reference trajectory. The reference trajectory can include the joint positions at each time point, as well as the position and orientation of the floating base. For example, the target video is a 10-second video, with each second corresponding to one video frame. Within the time interval of 0 to 10 seconds, all the robot's joints correspond to a joint angle every second (i.e., at each time point).

[0075] In some embodiments, if it is necessary to generate a more detailed reference trajectory, interpolation can be performed on two adjacent time nodes to obtain the interpolation results of the two adjacent time nodes, and the reference trajectory of the target action can be obtained based on the interpolation results and the target joint position.

[0076] Interpolation can be performed using linear interpolation or based on the smoothness between two time points; no specific method is required.

[0077] For example, within a time interval of 0 to 10 seconds, each joint corresponds to a target joint position every second. Linear interpolation is performed between 0 and 1 seconds, i.e., the corresponding joint position is inserted between 0 and 1 second (i.e., the interpolation result), and between 1 and 2 seconds. This process continues, inserting corresponding joint positions between all adjacent time points. Finally, based on the inserted joint positions and the target joint positions, a more detailed reference trajectory is generated.

[0078] Of course, in other embodiments, interpolation can be omitted, and the reference trajectory can be generated directly based on the target joint position at each time point.

[0079] In some embodiments, after generating a reference trajectory for the target action, the reference trajectory can be optimized based on robot dynamics to obtain an optimized reference trajectory. The robot then controls all joints throughout its body according to the optimized reference trajectory to achieve the target action.

[0080] As can be seen from the above, the embodiments of this application can directly obtain the reference trajectory of the target action from the video demonstrating the target action. Compared with using expensive motion capture equipment to capture the real person's demonstration action to obtain the reference trajectory, the embodiments of this application do not require the use of expensive motion capture equipment and can directly obtain the reference trajectory from the video, thus reducing the cost of obtaining the reference trajectory.

[0081] Based on the above embodiments, see Figure 2 The schematic diagram shown in this application illustrates the joint inverse solving process. The process of performing joint inverse solving on at least one robot task to obtain the target joint position at the time node corresponding to the video frame may include the following steps:

[0082] Step S201: Determine the Jacobian matrix and velocity residual for each robot task in the m-th iteration period.

[0083] Here, m is the number of iterations, which is a positive integer greater than or equal to 1. Typically, the maximum number of iterations can be preset, i.e., the maximum value of m can be preset.

[0084] In this embodiment, any robot task can be written as a function of joint positions, i.e., F(q). In this case, the Jacobian matrix of the robot task relative to the joint positions is: The velocity residual value is: Where F* is the expected value of the robot task, and F(q) is the current value of the robot task. During the inverse joint solution process, the joint position is iteratively changed by v*δt to gradually bring the current value (or actual value) of the robot task closer to the expected value. Here, v*δt is the change in joint position in each iteration.

[0085] Step S202: Determine the joint position of the m-th iteration based on the Jacobian matrix, the velocity residual, and the joint position of the previous iteration cycle.

[0086] It is understandable that if m equals 1, that is, in the first iteration cycle, the joint position of the previous iteration cycle can be regarded as the target joint position output by the previous video frame. In other words, the target joint position of the previous video frame is used as the joint starting position of the current video frame.

[0087] In the m-th iteration cycle, the joint position of the previous iteration cycle and the change in the joint position of the current iteration cycle are added together to obtain the joint position of the m-th iteration cycle.

[0088] The change in joint position during each iteration cycle can be obtained from the joint velocity. The joint velocity is the first derivative of the joint position. Therefore, we can first determine the joint velocity for the m-th iteration cycle based on the Jacobian matrix and velocity residuals for each machine task, and then determine the joint position for the m-th iteration cycle based on the joint velocity and the joint position from the previous iteration cycle.

[0089] In some embodiments, a quadratic programming problem can be established based on the Jacobian matrix and velocity residuals of each robot task, and the joint velocity in the m-th iteration period can be obtained by solving the quadratic programming problem through optimization.

[0090] For example, assuming a video frame has N robot tasks, the objective function of the quadratic programming in the m-th iteration period can be as follows:

[0091]

[0092] in, It is the joint velocity, which is the first derivative of the joint position q. task i is the i-th task, w i J is the weight of the i-th robot task. i It is the Jacobian matrix of the i-th robot task, v i K is the velocity residual for the i-th robot task. i It is the gain of the i-th robot task, used to control the approximation speed.

[0093] To ensure that the joint positions obtained from the solution of the above objective function are within a specified range, velocity constraints can be added to the objective function. For example, for the j-th joint, the joint position needs to be limited to the interval [q...]. min ,q max] Therefore, two linear constraints need to be added to the objective function. These two linear constraints are: and q j,0 It is the current position of the j-th joint.

[0094] By solving the objective function of the quadratic programming problem (i.e., the quadratic programming problem itself), that is, by optimizing the established quadratic programming problem, the optimal joint velocity in the m-th iteration period can be obtained.

[0095] It should be noted that if the number of robot joints is large, the number of variables in the objective function of the above quadratic programming will also be large. In this case, to ensure the robot task converges, the objective function of the above quadratic programming needs to be solved multiple times. In some embodiments, to improve the solution speed, a line search method can be used to calculate the joint position in the m-th iteration cycle. Specifically, first, based on the joint velocity, the target coefficient value for the m-th iteration cycle is determined using a line search; then, based on the target coefficient value and the joint velocity, the target change in joint position is determined; finally, based on the target change in joint position and the joint position in the previous iteration cycle, the joint position for the m-th iteration cycle is determined.

[0096] In each step, the objective function of the quadratic programming problem is solved to obtain the joint velocity. After that, it can be based on Calculate the joint position in the current iteration cycle, where k is the coefficient value, q(n) is the joint position in the previous iteration cycle, and q(n+1) is the joint position in the current iteration cycle. At this point, k is unknown.

[0097] After calculating q(n+1), the error between the actual and expected values ​​of all robot tasks can be calculated based on the joint positions q(n+1). Different values ​​of k will result in different errors. The k value corresponding to the smallest error is selected as the target coefficient value. This target coefficient value is then substituted into the formula. Calculate the joint position for the current iteration cycle.

[0098] Step S203: Determine the error between the actual value and the expected value of each robot task based on the joint position in the m-th iteration cycle.

[0099] Step S204: Determine whether the sum of the errors of each robot task is less than or equal to the preset error. If yes, proceed to step S205; otherwise, proceed to step S206.

[0100] After obtaining the joint position in the m-th iteration cycle, the actual value of each robot task can be calculated based on the joint position in the current iteration cycle; for each robot task, the error of the robot task is determined based on the difference between the actual value and the expected value.

[0101] The sum of the errors of each robot task can be obtained by multiplying the error of each robot task by its weight and then summing the results, or by directly adding the errors of each robot task together.

[0102] Step S205: Determine the joint position of the m-th iteration cycle as the target joint position.

[0103] Step S206: Determine whether the number of iterations has reached the preset number of iterations. If yes, proceed to step S207; otherwise, after m = m + 1, return to the step of determining the Jacobian matrix and velocity residuals of each robot task in the m-th iteration period.

[0104] For each video frame, starting from the joint's initial position, the joint position is iteratively changed to gradually bring the actual value of the robot's task closer to the expected value, thereby solving for the target joint position of the video frame.

[0105] For example, a video frame might include three robot tasks: F1, F2, and F3. F1 represents the robot's hand reaching position R1, F2 represents the robot's foot reaching position R2, and F3 represents the robot's center of gravity reaching position R3. That is, the expected value of F1 is R1, the expected value of F2 is R2, and the expected value of F3 is R3.

[0106] In the first iteration cycle, the joint position for the current iteration cycle is determined as q1 based on the joint's initial position and the change in joint position. This means the robot joint moves from its initial position to q1. It's understandable that each joint in the robot's entire body has an initial position and a change in joint position. Based on the initial position and the change in joint position, the changed position of each joint can be determined.

[0107] After obtaining the joint position q1 in the first iteration cycle, the actual value of F1 is determined based on the joint position q1, which determines the actual starting position for the current iteration cycle. The error of F1 is calculated based on the actual value and the expected value of F1.

[0108] Similarly, based on the joint position q1, the actual value of F2 is determined, which means the actual position of the foot in the current iteration cycle is determined; based on the actual value of F2 and the expected value of F2, the error of F2 can be calculated.

[0109] Based on the joint position q1, the actual value of F3 is determined, which means the actual position of the center of gravity in the current iteration cycle is determined; based on the actual value of F3 and the expected value of F3, the error of F3 can be calculated.

[0110] Based on the errors of F1, F2, and F3, determine whether the error of each robot task is less than or equal to the preset error. If yes, it is considered that after the first iteration of joint position adjustments, the robot's hands, feet, and center of gravity have approached the desired position, and the joint position of the first iteration is determined as the target joint position for that video frame; if not, m = m + 1, i.e., m = 1 + 1, and then proceed to the second iteration. At this time, the number of iterations is less than the preset number of iterations.

[0111] In the second iteration cycle, based on the joint position (i.e., q1) of the previous iteration cycle and the change in joint position of the current iteration cycle, the joint position of the current iteration cycle is determined to be q2, that is, the robot joint moves from q1 to q2.

[0112] Based on the joint position q1, calculate the error between the actual and expected values ​​of F1, F2 and F3, calculate the sum of the errors of F1, F2 and F3, and determine whether the sum of the errors of the three robot tasks is less than or equal to the preset error.

[0113] This process continues until the number of iterations reaches the preset number of iterations, or the sum of the errors of all robot tasks is less than or equal to the preset error.

[0114] It should be noted that in practical applications, the calculation can be performed on each video frame sequentially. That is, after calculating the target joint position of the current video frame, the target joint position of the next video frame is then calculated.

[0115] Step S207: Redesign the robot task.

[0116] If the robot task fails to converge after exceeding the preset number of iterations, the robot task for each video frame needs to be restarted. For example, for a specific video frame, the arm spatial task can be changed to a joint spatial task, and the center of gravity task can be changed to a waist pose task, etc.

[0117] After redesigning the robot task for the video frame, you can return to step S201, which is to start calculating the target joint position for the video frame again.

[0118] As can be seen from the above, the embodiments of this application solve the target joint position of each video frame by iteratively solving the Jacobian matrix and velocity residual of the robot task, thereby realizing the acquisition of reference trajectory from the target video.

[0119] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0120] Corresponding to the robot reference trajectory generation method described in the above embodiments, Figure 3 A structural block diagram of a robot reference trajectory generation device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0121] Reference Figure 3 The device includes:

[0122] Video acquisition module 31 is used to acquire target video, which is a video of the target object demonstrating the target action;

[0123] The inverse solution module 32 is used to obtain at least one robot task for each video frame in the target video, perform inverse joint solution on at least one robot task, and obtain the target joint position of the time node corresponding to the video frame. The robot task is generated based on the information of the target object in the video frame and is a function of the robot joint position.

[0124] The reference trajectory generation module 33 is used to generate a reference trajectory of the target motion based on the target joint position at each time node.

[0125] In one possible implementation, the inverse kinematics module 32 is specifically used to: determine the Jacobian matrix and velocity residual of each robot task in the m-th iteration cycle; determine the joint position in the m-th iteration cycle based on the Jacobian matrix, velocity residual, and joint position of the previous iteration cycle; determine the error between the actual value and the expected value of each robot task based on the joint position in the m-th iteration cycle; if the sum of the errors of each robot task is less than or equal to a preset error, then the joint position in the m-th iteration cycle is determined as the target joint position; if the sum of the errors of each robot task is greater than the preset error, after m = m + 1, return to the step of determining the Jacobian matrix and velocity residual of each robot task in the m-th iteration cycle, until the number of iterations reaches the preset number of iterations.

[0126] In one possible implementation, the inverse solution module 32 is specifically used to: determine the joint velocity of the m-th iteration cycle based on the Jacobian matrix and the velocity residual; and determine the joint position of the m-th iteration cycle based on the joint velocity and the joint position of the previous iteration cycle.

[0127] In one possible implementation, the inverse kinematics module 32 is specifically used to: determine the target coefficient value of the m-th iteration period by line search based on the joint velocity; determine the target change amount of the joint position based on the target coefficient value and the joint velocity; and determine the joint position of the m-th iteration period based on the target change amount of the joint position and the joint position of the previous iteration period.

[0128] In one possible implementation, the inverse solution module 32 is specifically used to: establish a quadratic programming problem based on the Jacobian matrix and the velocity residual, and obtain the joint velocity of the m-th iteration period by solving the optimization problem of the quadratic programming problem.

[0129] In one possible implementation, the reference trajectory generation module 33 is specifically used to: interpolate two adjacent time nodes to obtain the interpolation result of the two adjacent time nodes; and obtain the reference trajectory of the target action based on the interpolation result and the target joint position.

[0130] In one possible implementation, the inverse kinematics module 32 is specifically used to: extract information about the target object from the video frame; and determine the expected value and task expression of the robot task based on the information about the target object, so as to generate the robot task.

[0131] It should be noted that the information interaction and execution process between the above-mentioned devices / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0132] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 4 of this embodiment includes: at least one processor 40 ( Figure 4 (Only one is shown in the diagram), memory 41, and computer program 42 stored in said memory 41 and executable on said at least one processor 40, which, when executed, implements the steps in any of the above method embodiments.

[0133] The electronic device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0134] The processor 40 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0135] In some embodiments, the memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. In other embodiments, the memory 41 may be an external storage device of the electronic device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 4. Furthermore, the memory 41 may include both internal and external storage units of the electronic device 4. The memory 41 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0137] Furthermore, the specific names of each functional unit and module are merely for ease of differentiation and are not intended to limit the scope of protection of this application. The specific working processes of the units and modules in the above system can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0138] This application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0139] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0140] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0142] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0144] In the embodiments provided in this application, it should be understood that the disclosed devices, electronic devices, and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for generating a robot reference trajectory, characterized in that, include: Acquire the target video, which is a video of the target object demonstrating the target action; For each video frame in the target video, at least one robot task of the video frame is obtained, and the joint inverse solution is performed on the at least one robot task to obtain the target joint position of the time node corresponding to the video frame. The robot task is generated based on the information of the target object in the video frame, and the robot task is a function of the robot joint position. Based on the target joint position at each of the aforementioned time points, a reference trajectory for the target motion is generated; Specifically, performing inverse joint solving on the at least one robot task to obtain the target joint position at the time node corresponding to the video frame includes: Determine the Jacobian matrix and velocity residual for each robot task in the m-th iteration period; The joint position of the m-th iteration is determined based on the Jacobian matrix, the velocity residual, and the joint position of the previous iteration cycle. Based on the joint position in the m-th iteration cycle, determine the error between the actual value and the expected value of each robot task; If the sum of the errors of each robot task is less than or equal to the preset error, then the joint position in the m-th iteration period is determined as the target joint position. If the sum of the errors of each robot task is greater than the preset error, after m=m+1, return to the step of determining the Jacobian matrix and velocity residual of each robot task in the m-th iteration period, until the number of iterations reaches the preset number of iterations.

2. The method as described in claim 1, characterized in that, Determining the joint position of the m-th iteration cycle based on the Jacobian matrix, the velocity residual, and the joint position of the previous iteration cycle includes: Based on the Jacobian matrix and the velocity residual, determine the joint velocity in the m-th iteration period; The joint position for the m-th iteration is determined based on the joint velocity and the joint position of the previous iteration.

3. The method as described in claim 2, characterized in that, Determining the joint position for the m-th iteration based on the joint velocity and the joint position of the previous iteration cycle includes: Based on the joint velocity, the target coefficient value for the m-th iteration period is determined by line search. Based on the target coefficient value and the joint velocity, determine the target change in joint position; The joint position for the m-th iteration is determined based on the target change in joint position and the joint position of the previous iteration.

4. The method as described in claim 3, characterized in that, Based on the Jacobian matrix and the velocity residual, the joint velocity for the m-th iteration period is determined, including: Based on the Jacobian matrix and the velocity residual, a quadratic programming problem is established, and the joint velocity in the m-th iteration period is obtained by solving the quadratic programming problem through optimization.

5. The method according to any one of claims 1 to 4, characterized in that, Based on the target joint position at each of the aforementioned time points, a reference trajectory for the target motion is generated, including: Interpolate between two adjacent time points to obtain the interpolation result between the two adjacent time points; Based on the interpolation result and the target joint position, a reference trajectory for the target motion is obtained.

6. The method as described in claim 1, characterized in that, The robot task of acquiring at least one video frame includes: Extract information about the target object from the video frame; Based on the information of the target object, the expected value and task expression of the robot task are determined to generate the robot task.

7. A robot reference trajectory generation device, characterized in that, include: The video acquisition module is used to acquire the target video, which is a video of the target object demonstrating the target action; The inverse kinematics module is used to obtain at least one robot task for each video frame in the target video, perform inverse joint solving on the at least one robot task, and obtain the target joint position of the time node corresponding to the video frame. The robot task is generated based on the information of the target object in the video frame, and the robot task is a function of the robot joint position. The reference trajectory generation module is used to generate a reference trajectory of the target action based on the target joint position at each of the time nodes. Specifically, the inverse kinematics module is used to: determine the Jacobian matrix and velocity residual of each robot task in the m-th iteration cycle; determine the joint position in the m-th iteration cycle based on the Jacobian matrix, the velocity residual, and the joint position of the previous iteration cycle; determine the error between the actual value and the expected value of each robot task based on the joint position in the m-th iteration cycle; if the sum of the errors of each robot task is less than or equal to a preset error, then the joint position in the m-th iteration cycle is determined as the target joint position; if the sum of the errors of each robot task is greater than the preset error, after m=m+1, return to the step of determining the Jacobian matrix and velocity residual of each robot task in the m-th iteration cycle, until the number of iterations reaches the preset number of iterations.

8. The apparatus as claimed in claim 7, characterized in that, The inverse solution module is specifically used for: Based on the Jacobian matrix and the velocity residual, determine the joint velocity in the m-th iteration period; The joint position for the m-th iteration is determined based on the joint velocity and the joint position of the previous iteration.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images